org.jwat.warc

Class WarcReaderUncompressed

java.lang.Object

org.jwat.warc.WarcReader

org.jwat.warc.WarcReaderUncompressed

All Implemented Interfaces:

Closeable, AutoCloseable

public class WarcReaderUncompressed
extends WarcReader

WARC Reader implementation for reading uncompressed files. Use WarcReaderFactory to get an instance of this class.

Author:

nicl

Field Summary

Fields

Modifier and Type	Field and Description
protected ByteCountingPushBackInputStream	in WARC file InputStream.
static int	PUSHBACK_BUFFER_SIZE Buffer size used by PushbackInputStream.
protected long	startOffset Start offset of current or next valid record.

Fields inherited from class org.jwat.warc.WarcReader

bBlockDigest, bIsCompliant, blockDigestAlgorithm, blockDigestEncoding, bPayloadDigest, consumed, currentRecord, diagnostics, errors, fieldParsers, headerLineReader, iteratorExceptionThrown, lineReader, payloadDigestAlgorithm, payloadDigestEncoding, payloadHeaderMaxSize, recordHeaderMaxSize, records, uriProfile, warcTargetUriProfile, warnings

Constructor Summary

Constructors

Constructor and Description
WarcReaderUncompressed() This constructor is used to get random access to records.
WarcReaderUncompressed(ByteCountingPushBackInputStream in) Construct reader using the supplied input stream.

Method Summary

Modifier and Type	Method and Description
void	close() Close current record resource(s) and input stream(s).
long	getConsumed() Get number of bytes consumed by this reader.
WarcRecord	getNextRecord() Parses and gets the next record.
WarcRecord	getNextRecordFrom(InputStream rin, long offset) Parses and gets the next record from an Inputstream.
WarcRecord	getNextRecordFrom(InputStream rin, long offset, int buffer_size) Parses and gets the next record from an Inputstream wrapped by a BufferedInputStream.
long	getOffset() Get the current offset in the WARC InputStream.
long	getStartOffset() Get the offset of the current WARC record or -1 if none have been read.
boolean	isCompressed() Is this reader assuming GZip compressed input.
protected void	recordClosed() Callback method called when the payload has been processed.

Methods inherited from class org.jwat.warc.WarcReader

getBlockDigestAlgorithm, getBlockDigestEnabled, getBlockDigestEncoding, getIteratorExceptionThrown, getPayloadDigestAlgorithm, getPayloadDigestEnabled, getPayloadDigestEncoding, getPayloadHeaderMaxSize, getRecordHeaderMaxSize, getUriProfile, getWarcTargetUriProfile, init, isCompliant, iterator, reset, setBlockDigestAlgorithm, setBlockDigestEnabled, setBlockDigestEncoding, setPayloadDigestAlgorithm, setPayloadDigestEnabled, setPayloadDigestEncoding, setPayloadHeaderMaxSize, setRecordHeaderMaxSize, setUriProfile, setWarcTargetUriProfile

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

PUSHBACK_BUFFER_SIZE

public static final int PUSHBACK_BUFFER_SIZE

Buffer size used by PushbackInputStream.

See Also:

Constant Field Values

in

protected ByteCountingPushBackInputStream in

WARC file InputStream.

startOffset

protected long startOffset

Start offset of current or next valid record.

Constructor Detail

WarcReaderUncompressed

public WarcReaderUncompressed()

This constructor is used to get random access to records. The records are then accessed using the getNextRecordFrom methods using a supplied input stream for each record.

WarcReaderUncompressed

public WarcReaderUncompressed(ByteCountingPushBackInputStream in)

Construct reader using the supplied input stream. This method is primarily for sequential access to records.

Parameters:

in - WARC file input stream

Method Detail

isCompressed

public boolean isCompressed()

Description copied from class: WarcReader

Is this reader assuming GZip compressed input.

Specified by:

isCompressed in class WarcReader

Returns:

boolean indicating the assumption of GZip compressed input

close

public void close()

Description copied from class: WarcReader

Close current record resource(s) and input stream(s).

Specified by:

close in interface Closeable

Specified by:

close in interface AutoCloseable

Specified by:

close in class WarcReader

recordClosed

protected void recordClosed()

Description copied from class: WarcReader

Callback method called when the payload has been processed.

Specified by:

recordClosed in class WarcReader

getStartOffset

public long getStartOffset()

Description copied from class: WarcReader

Get the offset of the current WARC record or -1 if none have been read.

Specified by:

getStartOffset in class WarcReader

Returns:

offset of the current WARC record or -1

getOffset

public long getOffset()

Description copied from class: WarcReader

Get the current offset in the WARC InputStream.

Specified by:

getOffset in class WarcReader

Returns:

offset in WARC InputStream

getConsumed

public long getConsumed()

Description copied from class: WarcReader

Get number of bytes consumed by this reader.

Specified by:

getConsumed in class WarcReader

Returns:

number of bytes consumed by this reader

getNextRecord

public WarcRecord getNextRecord()
                         throws IOException

Description copied from class: WarcReader

Parses and gets the next record. This method is for linear access to records.

Specified by:

getNextRecord in class WarcReader

Returns:

the next record

Throws:

IOException - i/o exception in parsing process

getNextRecordFrom

public WarcRecord getNextRecordFrom(InputStream rin,
                                    long offset)
                             throws IOException

Description copied from class: WarcReader

Parses and gets the next record from an Inputstream. This method is mainly for random access use since there are serious side-effects involved in using multiple PushBackInputStream instances.

Specified by:

getNextRecordFrom in class WarcReader

Parameters:

rin - InputStream used to read next record

offset - offset provided by caller

Returns:

the next record

Throws:

IOException - i/o exception in parsing process

getNextRecordFrom

public WarcRecord getNextRecordFrom(InputStream rin,
                                    long offset,
                                    int buffer_size)
                             throws IOException

Description copied from class: WarcReader

Parses and gets the next record from an Inputstream wrapped by a BufferedInputStream. This method is mainly for random access use since there are serious side-effects involved in using multiple PushBackInputStream instances.

Specified by:

getNextRecordFrom in class WarcReader

Parameters:

rin - InputStream used to read next record

offset - offset provided by caller

buffer_size - buffer size to use

Returns:

the next record

Throws:

IOException - i/o exception in parsing process