org.apache.parquet.column.statistics

Class SizeStatistics

java.lang.Object

org.apache.parquet.column.statistics.SizeStatistics

public class SizeStatistics
extends Object

A structure for capturing metadata for estimating the unencoded, uncompressed size of data written. This is useful for readers to estimate how much memory is needed to reconstruct data in their memory model and for fine-grained filter push down on nested structures (the histograms contained in this structure can help determine the number of nulls at a particular nesting level and maximum length of lists).

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	SizeStatistics.Builder Builder to create a SizeStatistics.

Constructor Summary

Constructors

Constructor and Description
SizeStatistics(PrimitiveType type, long unencodedByteArrayDataBytes, List<Long> repetitionLevelHistogram, List<Long> definitionLevelHistogram) Create a SizeStatistics.

Method Summary

Modifier and Type	Method and Description
SizeStatistics	copy()
List<Long>	getDefinitionLevelHistogram() Same as repetition_level_histogram except for definition levels.
List<Long>	getRepetitionLevelHistogram() When present, there is expected to be one element corresponding to each repetition (i.e.
PrimitiveType	getType()
Optional<Long>	getUnencodedByteArrayDataBytes() The number of physical bytes stored for BYTE_ARRAY data values assuming no encoding.
boolean	isValid()
void	mergeStatistics(SizeStatistics other) Merge two SizeStatistics of the same column.
static SizeStatistics.Builder	newBuilder(PrimitiveType type, int maxRepetitionLevel, int maxDefinitionLevel) Create a builder to create a SizeStatistics.
static SizeStatistics.Builder	noopBuilder(PrimitiveType type, int maxRepetitionLevel, int maxDefinitionLevel) Creates a builder that doesn't collect any statistics.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

SizeStatistics

public SizeStatistics(PrimitiveType type,
                      long unencodedByteArrayDataBytes,
                      List<Long> repetitionLevelHistogram,
                      List<Long> definitionLevelHistogram)

Create a SizeStatistics.

Parameters:

type - physical type of the column associated with this statistics

unencodedByteArrayDataBytes - number of physical bytes stored for BYTE_ARRAY data values assuming no encoding

repetitionLevelHistogram - histogram for all repetition levels if non-empty

definitionLevelHistogram - histogram for all definition levels if non-empty

Method Detail

newBuilder

public static SizeStatistics.Builder newBuilder(PrimitiveType type,
                                                int maxRepetitionLevel,
                                                int maxDefinitionLevel)

Create a builder to create a SizeStatistics.

Parameters:

type - physical type of the column associated with this statistics

maxRepetitionLevel - maximum repetition level of the column

maxDefinitionLevel - maximum definition level of the column

mergeStatistics

public void mergeStatistics(SizeStatistics other)

Merge two SizeStatistics of the same column. It is used to merge size statistics from all pages of the same column chunk.

getType

public PrimitiveType getType()

getUnencodedByteArrayDataBytes

public Optional<Long> getUnencodedByteArrayDataBytes()

The number of physical bytes stored for BYTE_ARRAY data values assuming no encoding. This is exclusive of the bytes needed to store the length of each byte array. In other words, this field is equivalent to the `(size of PLAIN-ENCODING the byte array values) - (4 bytes * number of values written)`. To determine unencoded sizes of other types readers can use schema information multiplied by the number of non-null and null values. The number of null/non-null values can be inferred from the histograms below. For example, if a column chunk is dictionary-encoded with dictionary ["a", "bc", "cde"], and a data page contains the indices [0, 0, 1, 2], then this value for that data page should be 7 (1 + 1 + 2 + 3). This field should only be set for types that use BYTE_ARRAY as their physical type. It represents the field `unencoded_byte_array_data_bytes` in the SizeStatistics

Returns:

unencoded and uncompressed byte size of the BYTE_ARRAY column, or empty for other types.

getRepetitionLevelHistogram

public List<Long> getRepetitionLevelHistogram()

When present, there is expected to be one element corresponding to each repetition (i.e. size=max repetition_level+1) where each element represents the number of times the repetition level was observed in the data. This field may be omitted if max_repetition_level is 0 without loss of information. It represents the field `repetition_level_histogram` in the SizeStatistics

Returns:

repetition level histogram of all levels if not empty.

getDefinitionLevelHistogram

public List<Long> getDefinitionLevelHistogram()

Same as repetition_level_histogram except for definition levels. This field may be omitted if max_definition_level is 0 or 1 without loss of information. It represents the field `definition_level_histogram` in the SizeStatistics

Returns:

definition level histogram of all levels if not empty.

copy

public SizeStatistics copy()

Returns:

a new independent statistics instance of this class.

isValid

public boolean isValid()

Returns:

whether the statistics has valid value.

noopBuilder

public static SizeStatistics.Builder noopBuilder(PrimitiveType type,
                                                 int maxRepetitionLevel,
                                                 int maxDefinitionLevel)

Creates a builder that doesn't collect any statistics.