org.jwat.warc

Class WarcReaderCompressed

java.lang.Object

org.jwat.warc.WarcReader

org.jwat.warc.WarcReaderCompressed

All Implemented Interfaces:

Closeable, AutoCloseable

public class WarcReaderCompressed
extends WarcReader

WARC Reader implementation for reading GZip compressed files. Use WarcReaderFactory to get an instance of this class.

Author:

nicl

Field Summary

Fields

Modifier and Type	Field and Description
protected int	bufferSize Buffer size, if any, to use on GZip entry InputStream.
protected GzipEntry	currentEntry GZip entry for the current record, if random access methods used.
protected GzipReader	currentReader GZip reader used for the current record, if random access methods used.
static int	PUSHBACK_BUFFER_SIZE Buffer size used by PushbackInputStream.
protected GzipReader	reader WARC file InputStream.
protected long	startOffset Cached start offset used after the reader is closed.

Fields inherited from class org.jwat.warc.WarcReader

bBlockDigest, bIsCompliant, blockDigestAlgorithm, blockDigestEncoding, bPayloadDigest, consumed, currentRecord, diagnostics, errors, fieldParsers, headerLineReader, iteratorExceptionThrown, lineReader, payloadDigestAlgorithm, payloadDigestEncoding, payloadHeaderMaxSize, recordHeaderMaxSize, records, uriProfile, warcTargetUriProfile, warnings

Constructor Summary

Constructors

Constructor and Description
WarcReaderCompressed() This constructor is used to get random access to records.
WarcReaderCompressed(GzipReader reader) Construct reader using the supplied input stream.
WarcReaderCompressed(GzipReader reader, int buffer_size) Construct object using supplied GzipInputStream.

Method Summary

Modifier and Type	Method and Description
void	close() Close current record resource(s) and input stream(s).
long	getConsumed() Get number of bytes consumed by the WARC GzipReader.
WarcRecord	getNextRecord() Parses and gets the next record.
WarcRecord	getNextRecordFrom(InputStream rin, long offset) Parses and gets the next record from an Inputstream.
WarcRecord	getNextRecordFrom(InputStream rin, long offset, int buffer_size) Parses and gets the next record from an Inputstream wrapped by a BufferedInputStream.
long	getOffset() Get the current offset in the WARC GzipReader.
long	getStartOffset() Get the offset of the current WARC record from the GZip entry or -1 if no records have been read yet.
boolean	isCompressed() Is this reader assuming GZip compressed input.
protected void	recordClosed() Callback method called when the payload has been processed.

Methods inherited from class org.jwat.warc.WarcReader

getBlockDigestAlgorithm, getBlockDigestEnabled, getBlockDigestEncoding, getIteratorExceptionThrown, getPayloadDigestAlgorithm, getPayloadDigestEnabled, getPayloadDigestEncoding, getPayloadHeaderMaxSize, getRecordHeaderMaxSize, getUriProfile, getWarcTargetUriProfile, init, isCompliant, iterator, reset, setBlockDigestAlgorithm, setBlockDigestEnabled, setBlockDigestEncoding, setPayloadDigestAlgorithm, setPayloadDigestEnabled, setPayloadDigestEncoding, setPayloadHeaderMaxSize, setRecordHeaderMaxSize, setUriProfile, setWarcTargetUriProfile

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

PUSHBACK_BUFFER_SIZE

public static final int PUSHBACK_BUFFER_SIZE

Buffer size used by PushbackInputStream.

See Also:

Constant Field Values

reader

protected GzipReader reader

WARC file InputStream.

bufferSize

protected int bufferSize

Buffer size, if any, to use on GZip entry InputStream.

currentReader

protected GzipReader currentReader

GZip reader used for the current record, if random access methods used.

currentEntry

protected GzipEntry currentEntry

GZip entry for the current record, if random access methods used.

startOffset

protected long startOffset

Cached start offset used after the reader is closed.

Constructor Detail

WarcReaderCompressed

public WarcReaderCompressed()

This constructor is used to get random access to records. The records are then accessed using the getNextRecordFrom methods using a supplied input stream for each record.

WarcReaderCompressed

public WarcReaderCompressed(GzipReader reader)

Construct reader using the supplied input stream. This method is primarily for sequential access to records.

Parameters:

reader - GZip reader

WarcReaderCompressed

public WarcReaderCompressed(GzipReader reader,
                            int buffer_size)

Construct object using supplied GzipInputStream. This method is primarily for sequential access to records.

Parameters:

reader - GZip reader

buffer_size - buffer size used on entries

Method Detail

isCompressed

public boolean isCompressed()

Description copied from class: WarcReader

Is this reader assuming GZip compressed input.

Specified by:

isCompressed in class WarcReader

Returns:

boolean indicating the assumption of GZip compressed input

close

public void close()

Description copied from class: WarcReader

Close current record resource(s) and input stream(s).

Specified by:

close in interface Closeable

Specified by:

close in interface AutoCloseable

Specified by:

close in class WarcReader

recordClosed

protected void recordClosed()

Description copied from class: WarcReader

Callback method called when the payload has been processed.

Specified by:

recordClosed in class WarcReader

getStartOffset

public long getStartOffset()

Get the offset of the current WARC record from the GZip entry or -1 if no records have been read yet.

Specified by:

getStartOffset in class WarcReader

Returns:

offset of the current WARC record from the GZip entry or -1

getOffset

public long getOffset()

Get the current offset in the WARC GzipReader.

Specified by:

getOffset in class WarcReader

Returns:

offset in WARC InputStream

getConsumed

public long getConsumed()

Get number of bytes consumed by the WARC GzipReader.

Specified by:

getConsumed in class WarcReader

Returns:

number of bytes consumed by the WARC GzipReader

getNextRecord

public WarcRecord getNextRecord()
                         throws IOException

Description copied from class: WarcReader

Parses and gets the next record. This method is for linear access to records.

Specified by:

getNextRecord in class WarcReader

Returns:

the next record

Throws:

IOException - i/o exception in parsing process

getNextRecordFrom

public WarcRecord getNextRecordFrom(InputStream rin,
                                    long offset)
                             throws IOException

Description copied from class: WarcReader

Parses and gets the next record from an Inputstream. This method is mainly for random access use since there are serious side-effects involved in using multiple PushBackInputStream instances.

Specified by:

getNextRecordFrom in class WarcReader

Parameters:

rin - InputStream used to read next record

offset - offset provided by caller

Returns:

the next record

Throws:

IOException - i/o exception in parsing process

getNextRecordFrom

public WarcRecord getNextRecordFrom(InputStream rin,
                                    long offset,
                                    int buffer_size)
                             throws IOException

Description copied from class: WarcReader

Parses and gets the next record from an Inputstream wrapped by a BufferedInputStream. This method is mainly for random access use since there are serious side-effects involved in using multiple PushBackInputStream instances.

Specified by:

getNextRecordFrom in class WarcReader

Parameters:

rin - InputStream used to read next record

offset - offset provided by caller

buffer_size - buffer size to use

Returns:

the next record

Throws:

IOException - i/o exception in parsing process