org.apache.parquet.column.values.bloomfilter

Class AdaptiveBlockSplitBloomFilter

java.lang.Object

org.apache.parquet.column.values.bloomfilter.AdaptiveBlockSplitBloomFilter

All Implemented Interfaces:

BloomFilter

public class AdaptiveBlockSplitBloomFilter
extends Object
implements BloomFilter

The purpose of this is to finally generate a bloom filter with the optimal bit size according to the number of real data distinct values. `AdaptiveBlockSplitBloomFilter` contains multiple `BlockSplitBloomFilter` as candidates and inserts values in the candidates at the same time. Some candidates that are too small will be eliminated during the insertion process. Finally we will choose the most appropriate size candidate to write out.

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
protected class	AdaptiveBlockSplitBloomFilter.BloomFilterCandidate

Nested classes/interfaces inherited from interface org.apache.parquet.column.values.bloomfilter.BloomFilter

BloomFilter.Algorithm, BloomFilter.Compression, BloomFilter.HashStrategy

Constructor Summary

Constructors

Constructor and Description
AdaptiveBlockSplitBloomFilter(int maximumBytes, BloomFilter.HashStrategy hashStrategy, double fpp, int numCandidates, ColumnDescriptor column)
AdaptiveBlockSplitBloomFilter(int maximumBytes, int numCandidates, double fpp, ColumnDescriptor column)

Method Summary

Modifier and Type	Method and Description
boolean	findHash(long hash) Determine whether an element is in set or not.
BloomFilter.Algorithm	getAlgorithm() Return the algorithm that the bloom filter apply.
int	getBitsetSize() Get the number of bytes for bitset in this Bloom filter.
protected List<AdaptiveBlockSplitBloomFilter.BloomFilterCandidate>	getCandidates()
BloomFilter.Compression	getCompression() Return the compress algorithm that the bloom filter apply.
BloomFilter.HashStrategy	getHashStrategy() Return the hash strategy that the bloom filter apply.
long	hash(Binary value) Compute hash for Binary value by using its plain encoding result.
long	hash(double value) Compute hash for double value by using its plain encoding result.
long	hash(float value) Compute hash for float value by using its plain encoding result.
long	hash(int value) Compute hash for int value by using its plain encoding result.
long	hash(long value) Compute hash for long value by using its plain encoding result.
long	hash(Object value) Compute hash for Object value by using its plain encoding result.
void	insertHash(long hash) Insert an element to the multiple bloom filter candidates and remove the bad candidate if the number of distinct values exceeds its expected size.
protected AdaptiveBlockSplitBloomFilter.BloomFilterCandidate	optimalCandidate() Used at the end of the insertion, select the candidate of the smallest size.
void	writeTo(OutputStream out) Write the Bloom filter to an output stream.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.apache.parquet.column.values.bloomfilter.BloomFilter

canMergeFrom, equals, merge

Constructor Detail

AdaptiveBlockSplitBloomFilter

public AdaptiveBlockSplitBloomFilter(int maximumBytes,
                                     int numCandidates,
                                     double fpp,
                                     ColumnDescriptor column)

Parameters:

maximumBytes - the maximum bytes size of candidate

numCandidates - the number of candidates

fpp - the false positive probability

AdaptiveBlockSplitBloomFilter

public AdaptiveBlockSplitBloomFilter(int maximumBytes,
                                     BloomFilter.HashStrategy hashStrategy,
                                     double fpp,
                                     int numCandidates,
                                     ColumnDescriptor column)

Method Detail

optimalCandidate

protected AdaptiveBlockSplitBloomFilter.BloomFilterCandidate optimalCandidate()

Used at the end of the insertion, select the candidate of the smallest size. At least one of the largest candidates will be kept when inserting data.

Returns:

the smallest and optimal candidate

getCandidates

protected List<AdaptiveBlockSplitBloomFilter.BloomFilterCandidate> getCandidates()

writeTo

public void writeTo(OutputStream out)
             throws IOException

Description copied from interface: BloomFilter

Write the Bloom filter to an output stream. It writes the Bloom filter header including the bitset's length in bytes, the hash strategy, the algorithm, and the bitset.

Specified by:

writeTo in interface BloomFilter

Parameters:

out - the output stream to write

Throws:

IOException

insertHash

public void insertHash(long hash)

Insert an element to the multiple bloom filter candidates and remove the bad candidate if the number of distinct values exceeds its expected size.

Specified by:

insertHash in interface BloomFilter

Parameters:

hash - the hash result of element.

getBitsetSize

public int getBitsetSize()

Description copied from interface: BloomFilter

Get the number of bytes for bitset in this Bloom filter.

Specified by:

getBitsetSize in interface BloomFilter

Returns:

The number of bytes for bitset in this Bloom filter.

findHash

public boolean findHash(long hash)

Description copied from interface: BloomFilter

Determine whether an element is in set or not.

Specified by:

findHash in interface BloomFilter

Parameters:

hash - the hash value of element plain encoding result.

Returns:

false if element is must not in set, true if element probably in set.

hash

public long hash(Object value)

Description copied from interface: BloomFilter

Compute hash for Object value by using its plain encoding result.

Specified by:

hash in interface BloomFilter

Parameters:

value - the value to hash

Returns:

hash result

getHashStrategy

public BloomFilter.HashStrategy getHashStrategy()

Description copied from interface: BloomFilter

Return the hash strategy that the bloom filter apply.

Specified by:

getHashStrategy in interface BloomFilter

Returns:

hash strategy that the bloom filter apply

getAlgorithm

public BloomFilter.Algorithm getAlgorithm()

Description copied from interface: BloomFilter

Return the algorithm that the bloom filter apply.

Specified by:

getAlgorithm in interface BloomFilter

Returns:

algorithm that the bloom filter apply

getCompression

public BloomFilter.Compression getCompression()

Description copied from interface: BloomFilter

Return the compress algorithm that the bloom filter apply.

Specified by:

getCompression in interface BloomFilter

Returns:

compress algorithm that the bloom filter apply

hash

public long hash(int value)

Description copied from interface: BloomFilter

Compute hash for int value by using its plain encoding result.

Specified by:

hash in interface BloomFilter

Parameters:

value - the value to hash

Returns:

hash result

hash

public long hash(long value)

Description copied from interface: BloomFilter

Compute hash for long value by using its plain encoding result.

Specified by:

hash in interface BloomFilter

Parameters:

value - the value to hash

Returns:

hash result

hash

public long hash(double value)

Description copied from interface: BloomFilter

Compute hash for double value by using its plain encoding result.

Specified by:

hash in interface BloomFilter

Parameters:

value - the value to hash

Returns:

hash result

hash

public long hash(float value)

Description copied from interface: BloomFilter

Compute hash for float value by using its plain encoding result.

Specified by:

hash in interface BloomFilter

Parameters:

value - the value to hash

Returns:

hash result

hash

public long hash(Binary value)

Description copied from interface: BloomFilter

Compute hash for Binary value by using its plain encoding result.

Specified by:

hash in interface BloomFilter

Parameters:

value - the value to hash

Returns:

hash result