org.jwat.warc

Class WarcReaderFactory

java.lang.Object

org.jwat.archive.common.ReaderFactoryAbstract

org.jwat.warc.WarcReaderFactory

public class WarcReaderFactory
extends ReaderFactoryAbstract

Factory used for creating WarcReader instances. The general getReader methods will auto-detect Gzip'ed data and return the appropriate WarcReader instances. The other factory methods can be used to return specific WarcReader instances for compressed or uncompressed records. Readers are available for both sequential and random reading of records. Use of buffered methods and/or buffering speeds up the reader considerably.

Author:

nicl

Field Summary

Fields

Modifier and Type	Field and Description
static int	PUSHBACK_BUFFER_SIZE Buffer size used by PushbackInputStream.

Fields inherited from class org.jwat.archive.common.ReaderFactoryAbstract

ARC_MAGIC_HEADER, GZIP_MAGIC, WARC_MAGIC_HEADER

Constructor Summary

Constructors

Modifier	Constructor and Description
protected	WarcReaderFactory() Private constructor to enforce factory methods.

Method Summary

Modifier and Type	Method and Description
static WarcReader	getReader(InputStream in) Creates a new WarcReader from an InputStream.
static WarcReader	getReader(InputStream in, int buffer_size) Creates a new WarcReader from an InputStream wrapped by a BufferedInputStream.
static WarcReaderCompressed	getReaderCompressed() Creates a new WarcReader without any associated InputStream for random access to GZip compressed records.
static WarcReaderCompressed	getReaderCompressed(InputStream in) Creates a new WarcReader from an InputStream primarily for random access to GZip compressed records.
static WarcReaderCompressed	getReaderCompressed(InputStream in, int buffer_size) Creates a new WarcReader from an InputStream wrapped by a BufferedInputStream primarily for random access to GZip compressed records.
static WarcReaderUncompressed	getReaderUncompressed() Creates a new WarcReader without any associated InputStream for random access to uncompressed records.
static WarcReaderUncompressed	getReaderUncompressed(InputStream in) Creates a new WarcReader from an InputStream primarily for random access to uncompressed records.
static WarcReaderUncompressed	getReaderUncompressed(InputStream in, int buffer_size) Creates a new WarcReader from an InputStream wrapped by a BufferedInputStream primarily for random access to uncompressed records.
static boolean	isWarcFile(ByteCountingPushBackInputStream pbin) Check head of PushBackInputStream for a WARC file identifier.
static boolean	isWarcRecord(ByteCountingPushBackInputStream pbin) Check head of PushBackInputStream for a WARC record identifier.

Methods inherited from class org.jwat.archive.common.ReaderFactoryAbstract

isArcFile, isArcRecord, isGzipped

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

PUSHBACK_BUFFER_SIZE

public static final int PUSHBACK_BUFFER_SIZE

Buffer size used by PushbackInputStream.

See Also:

Constant Field Values

Constructor Detail

WarcReaderFactory

protected WarcReaderFactory()

Private constructor to enforce factory methods.

Method Detail

isWarcFile

public static boolean isWarcFile(ByteCountingPushBackInputStream pbin)
                          throws IOException

Check head of PushBackInputStream for a WARC file identifier. The identifier for WARC files is "WARC/" in the beginning.

Parameters:

pbin - PushBackInputStream with WARC records

Returns:

boolean indicating presence of a WARC file identifier

Throws:

IOException - if an i/o error occurs while examining head of stream

isWarcRecord

public static boolean isWarcRecord(ByteCountingPushBackInputStream pbin)
                            throws IOException

Check head of PushBackInputStream for a WARC record identifier. The identifier for WARC records is "WARC/" in the beginning.

Parameters:

pbin - PushBackInputStream with WARC records

Returns:

boolean indicating presence of a WARC magic number

Throws:

IOException - if an i/o error occurs while examining head of stream

getReader

public static WarcReader getReader(InputStream in,
                                   int buffer_size)
                            throws IOException

Creates a new WarcReader from an InputStream wrapped by a BufferedInputStream. The WarcReader implementation returned is chosen based on GZip auto detection.

Parameters:

in - WARC File represented as InputStream

buffer_size - buffer size to use

Returns:

appropriate WarcReader based on data read from InputStream

Throws:

IOException - if an i/o exception occurs during initialization

getReader

public static WarcReader getReader(InputStream in)
                            throws IOException

Creates a new WarcReader from an InputStream. The WarcReader implementation returned is chosen based on GZip auto detection.

Parameters:

in - WARC File represented as InputStream

Returns:

appropriate WarcReader based on data read from InputStream

Throws:

IOException - if an i/o exception occurs during initialization

getReaderUncompressed

public static WarcReaderUncompressed getReaderUncompressed()

Creates a new WarcReader without any associated InputStream for random access to uncompressed records.

Returns:

WarcReader for uncompressed records read from InputStream

getReaderUncompressed

public static WarcReaderUncompressed getReaderUncompressed(InputStream in)
                                                    throws IOException

Creates a new WarcReader from an InputStream primarily for random access to uncompressed records.

Parameters:

in - WARC File represented as InputStream

Returns:

WarcReader for uncompressed records read from InputStream

Throws:

IOException - i/o exception while initializing reader

getReaderUncompressed

public static WarcReaderUncompressed getReaderUncompressed(InputStream in,
                                                           int buffer_size)
                                                    throws IOException

Creates a new WarcReader from an InputStream wrapped by a BufferedInputStream primarily for random access to uncompressed records.

Parameters:

in - WARC File represented as InputStream

buffer_size - buffer size to use

Returns:

WarcReader for uncompressed records read from InputStream

Throws:

IOException - i/o exception while initializing reader

getReaderCompressed

public static WarcReaderCompressed getReaderCompressed()

Creates a new WarcReader without any associated InputStream for random access to GZip compressed records.

Returns:

WarcReader for GZip compressed records read from InputStream

getReaderCompressed

public static WarcReaderCompressed getReaderCompressed(InputStream in)
                                                throws IOException

Creates a new WarcReader from an InputStream primarily for random access to GZip compressed records.

Parameters:

in - WARC File represented as InputStream

Returns:

WarcReader for GZip compressed records read from InputStream

Throws:

IOException - i/o exception while initializing reader

getReaderCompressed

public static WarcReaderCompressed getReaderCompressed(InputStream in,
                                                       int buffer_size)
                                                throws IOException

Creates a new WarcReader from an InputStream wrapped by a BufferedInputStream primarily for random access to GZip compressed records.

Parameters:

in - WARC File represented as InputStream

buffer_size - buffer size to use

Returns:

WarcReader for GZip compressed records read from InputStream

Throws:

IOException - i/o exception while initializing reader