org.owasp.dependencycheck.data.lucene

Class TokenPairConcatenatingFilter

java.lang.Object

org.apache.lucene.util.AttributeSource

org.apache.lucene.analysis.TokenStream

org.apache.lucene.analysis.TokenFilter

org.owasp.dependencycheck.data.lucene.TokenPairConcatenatingFilter

All Implemented Interfaces:

Closeable, AutoCloseable

@NotThreadSafe
public final class TokenPairConcatenatingFilter
extends org.apache.lucene.analysis.TokenFilter

Takes a TokenStream and adds additional tokens by concatenating pairs of words.

Example: "Spring Framework Core" -> "Spring SpringFramework Framework FrameworkCore Core".

Author:

Jeremy Long

Nested Class Summary

Nested classes/interfaces inherited from class org.apache.lucene.util.AttributeSource

org.apache.lucene.util.AttributeSource.State

Field Summary

Fields inherited from class org.apache.lucene.analysis.TokenFilter

input

Fields inherited from class org.apache.lucene.analysis.TokenStream

DEFAULT_TOKEN_ATTRIBUTE_FACTORY

Constructor Summary

Constructors

Constructor and Description
TokenPairConcatenatingFilter(org.apache.lucene.analysis.TokenStream stream) Constructs a new TokenPairConcatenatingFilter.

Method Summary

Modifier and Type	Method and Description
void	clear() Resets the filter.
boolean	equals(Object obj)
int	hashCode()
boolean	incrementToken() Increments the underlying TokenStream and sets CharTermAttributes to construct an expanded set of tokens by concatenating tokens with the previous token.

Methods inherited from class org.apache.lucene.analysis.TokenFilter

close, end, reset

Methods inherited from class org.apache.lucene.util.AttributeSource

addAttribute, addAttributeImpl, captureState, clearAttributes, cloneAttributes, copyTo, endAttributes, getAttribute, getAttributeClassesIterator, getAttributeFactory, getAttributeImplsIterator, hasAttribute, hasAttributes, reflectAsString, reflectWith, removeAllAttributes, restoreState, toString

Methods inherited from class java.lang.Object

clone, finalize, getClass, notify, notifyAll, wait, wait, wait

Constructor Detail

TokenPairConcatenatingFilter

public TokenPairConcatenatingFilter(org.apache.lucene.analysis.TokenStream stream)

Constructs a new TokenPairConcatenatingFilter.

Parameters:

stream - the TokenStream that this filter will process

Method Detail

incrementToken

public boolean incrementToken()
                       throws IOException

Increments the underlying TokenStream and sets CharTermAttributes to construct an expanded set of tokens by concatenating tokens with the previous token.

Specified by:

incrementToken in class org.apache.lucene.analysis.TokenStream

Returns:

whether or not we have hit the end of the TokenStream

Throws:

IOException - is thrown when an IOException occurs

clear

public void clear()
           throws IOException

Resets the filter. This must be manually called between searching and indexing. Unable to rely on `reset` as it appears to be called between terms.

Throws:

IOException - thrown if there is an error reseting the tokenizer

hashCode

public int hashCode()

Overrides:

hashCode in class org.apache.lucene.util.AttributeSource

equals

public boolean equals(Object obj)

Overrides:

equals in class org.apache.lucene.util.AttributeSource