org.owasp.dependencycheck.data.lucene

Class UrlTokenizingFilter

java.lang.Object

org.apache.lucene.util.AttributeSource

org.apache.lucene.analysis.TokenStream

org.apache.lucene.analysis.TokenFilter

org.owasp.dependencycheck.data.lucene.AbstractTokenizingFilter

org.owasp.dependencycheck.data.lucene.UrlTokenizingFilter

All Implemented Interfaces:

Closeable, AutoCloseable

@NotThreadSafe
public final class UrlTokenizingFilter
extends AbstractTokenizingFilter

Takes a TokenStream, looks for URLs, and breaks them into separate tokens.

Author:

Jeremy Long

Nested Class Summary

Nested classes/interfaces inherited from class org.apache.lucene.util.AttributeSource

org.apache.lucene.util.AttributeSource.State

Field Summary

Fields inherited from class org.apache.lucene.analysis.TokenFilter

input

Fields inherited from class org.apache.lucene.analysis.TokenStream

DEFAULT_TOKEN_ATTRIBUTE_FACTORY

Constructor Summary

Constructors

Constructor and Description
UrlTokenizingFilter(org.apache.lucene.analysis.TokenStream stream) Constructs a new UrlTokenizingFilter.

Method Summary

Modifier and Type	Method and Description
boolean	incrementToken() Increments the underlying TokenStream and sets CharTermAttributes to construct an expanded set of tokens by concatenating tokens with the previous token.

Methods inherited from class org.owasp.dependencycheck.data.lucene.AbstractTokenizingFilter

addTerm, getTermAtt, getTokens, reset

Methods inherited from class org.apache.lucene.analysis.TokenFilter

close, end

Methods inherited from class org.apache.lucene.util.AttributeSource

addAttribute, addAttributeImpl, captureState, clearAttributes, cloneAttributes, copyTo, endAttributes, equals, getAttribute, getAttributeClassesIterator, getAttributeFactory, getAttributeImplsIterator, hasAttribute, hasAttributes, hashCode, reflectAsString, reflectWith, removeAllAttributes, restoreState, toString

Methods inherited from class java.lang.Object

clone, finalize, getClass, notify, notifyAll, wait, wait, wait

Constructor Detail

UrlTokenizingFilter

public UrlTokenizingFilter(org.apache.lucene.analysis.TokenStream stream)

Constructs a new UrlTokenizingFilter.

Parameters:

stream - the TokenStream that this filter will process

Method Detail

incrementToken

public boolean incrementToken()
                       throws IOException

Increments the underlying TokenStream and sets CharTermAttributes to construct an expanded set of tokens by concatenating tokens with the previous token.

Specified by:

incrementToken in class org.apache.lucene.analysis.TokenStream

Returns:

whether or not we have hit the end of the TokenStream

Throws:

IOException - is thrown when an IOException occurs