org.apache.parquet.column.values.delta

Class DeltaBinaryPackingValuesWriter

java.lang.Object

org.apache.parquet.column.values.ValuesWriter

org.apache.parquet.column.values.delta.DeltaBinaryPackingValuesWriter

All Implemented Interfaces:

AutoCloseable

Direct Known Subclasses:

DeltaBinaryPackingValuesWriterForInteger, DeltaBinaryPackingValuesWriterForLong

public abstract class DeltaBinaryPackingValuesWriter
extends ValuesWriter

Write integers with delta encoding and binary packing The format is as follows:

   
     delta-binary-packing: <page-header> <block>*
     page-header := <block size in values> <number of miniblocks in a block> <total value count> <first value>
     block := <min delta> <list of bitwidths of miniblocks> <miniblocks>

     min delta : zig-zag var int encoded
     bitWidthsOfMiniBlock : 1 byte little endian
     blockSizeInValues,blockSizeInValues,totalValueCount,firstValue : unsigned varint
   
 

The algorithm and format is inspired by D. Lemire's paper: http://lemire.me/blog/archives/2012/09/12/fast-integer-compression-decoding-billions-of-integers-per-second/

Field Summary

Fields

Modifier and Type	Field and Description
protected CapacityByteArrayOutputStream	baos
protected int[]	bitWidths bit width for each mini block, reused between flushes
protected org.apache.parquet.column.values.delta.DeltaBinaryPackingConfig	config stores blockSizeInValues, miniBlockNumInABlock and miniBlockSizeInValues
static int	DEFAULT_NUM_BLOCK_VALUES
static int	DEFAULT_NUM_MINIBLOCKS
protected int	deltaValuesToFlush a pointer to deltaBlockBuffer indicating the end of deltaBlockBuffer the number of values in the deltaBlockBuffer that haven't flushed to baos it will be reset after each flush
protected byte[]	miniBlockByteBuffer bytes buffer for a mini block, it is reused for each mini block.
protected int	totalValueCount

Constructor Summary

Constructors

Constructor and Description
DeltaBinaryPackingValuesWriter(int slabSize, int pageSize, ByteBufferAllocator allocator)
DeltaBinaryPackingValuesWriter(int blockSizeInValues, int miniBlockNum, int slabSize, int pageSize, ByteBufferAllocator allocator)

Method Summary

Modifier and Type	Method and Description
void	close() Called to close the values writer.
long	getAllocatedSize() ( > ValuesWriter.getBufferedSize() )
long	getBufferedSize() used to decide if we want to work to the next page
Encoding	getEncoding() called after getBytes() and before reset()
protected int	getMiniBlockCountToFlush(double numberCount)
String	memUsageString(String prefix)
void	reset() called after getBytes() to reset the current buffer and start writing the next page
protected void	writeBitWidthForMiniBlock(int i)

Methods inherited from class org.apache.parquet.column.values.ValuesWriter

getBytes, resetDictionary, toDictPageAndClose, writeBoolean, writeByte, writeBytes, writeDouble, writeFloat, writeInteger, writeLong

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

DEFAULT_NUM_BLOCK_VALUES

public static final int DEFAULT_NUM_BLOCK_VALUES

See Also:

Constant Field Values

DEFAULT_NUM_MINIBLOCKS

public static final int DEFAULT_NUM_MINIBLOCKS

See Also:

Constant Field Values

baos

protected final CapacityByteArrayOutputStream baos

config

protected final org.apache.parquet.column.values.delta.DeltaBinaryPackingConfig config

stores blockSizeInValues, miniBlockNumInABlock and miniBlockSizeInValues

bitWidths

protected final int[] bitWidths

bit width for each mini block, reused between flushes

totalValueCount

protected int totalValueCount

deltaValuesToFlush

protected int deltaValuesToFlush

a pointer to deltaBlockBuffer indicating the end of deltaBlockBuffer the number of values in the deltaBlockBuffer that haven't flushed to baos it will be reset after each flush

miniBlockByteBuffer

protected byte[] miniBlockByteBuffer

bytes buffer for a mini block, it is reused for each mini block. Therefore the size of biggest miniblock with bitwidth of MAX_BITWIDTH is allocated

Constructor Detail

DeltaBinaryPackingValuesWriter

public DeltaBinaryPackingValuesWriter(int slabSize,
                                      int pageSize,
                                      ByteBufferAllocator allocator)

DeltaBinaryPackingValuesWriter

public DeltaBinaryPackingValuesWriter(int blockSizeInValues,
                                      int miniBlockNum,
                                      int slabSize,
                                      int pageSize,
                                      ByteBufferAllocator allocator)

Method Detail

getBufferedSize

public long getBufferedSize()

Description copied from class: ValuesWriter

used to decide if we want to work to the next page

Specified by:

getBufferedSize in class ValuesWriter

Returns:

the size of the currently buffered data (in bytes)

writeBitWidthForMiniBlock

protected void writeBitWidthForMiniBlock(int i)

getMiniBlockCountToFlush

protected int getMiniBlockCountToFlush(double numberCount)

getEncoding

public Encoding getEncoding()

Description copied from class: ValuesWriter

called after getBytes() and before reset()

Specified by:

getEncoding in class ValuesWriter

Returns:

the encoding that was used to encode the bytes

reset

public void reset()

Description copied from class: ValuesWriter

called after getBytes() to reset the current buffer and start writing the next page

Specified by:

reset in class ValuesWriter

close

public void close()

Description copied from class: ValuesWriter

Called to close the values writer. Any output stream is closed and can no longer be used. All resources are released.

Specified by:

close in interface AutoCloseable

Overrides:

close in class ValuesWriter

getAllocatedSize

public long getAllocatedSize()

Description copied from class: ValuesWriter

( > ValuesWriter.getBufferedSize() )

Specified by:

getAllocatedSize in class ValuesWriter

Returns:

the allocated size of the buffer

memUsageString

public String memUsageString(String prefix)

Specified by:

memUsageString in class ValuesWriter