WarcReader (jwat-warc 1.0.3 API)

java.lang.Object
- org.jwat.warc.WarcReader

All Implemented Interfaces:

Closeable, AutoCloseable

Direct Known Subclasses:

WarcReaderCompressed, WarcReaderUncompressed
```
public abstract class WarcReader
extends Object
implements Closeable
```
Base class for WARC reader implementations.

Author:

nicl

Field Summary

Fields
Modifier and Type	Field and Description
`protected boolean`	`bBlockDigest` Block Digest enabled/disabled.
`protected boolean`	`bIsCompliant` Compliance status for records parsed up to now.
`protected String`	`blockDigestAlgorithm` Default block digest algorithm to use if none is present in the record.
`protected String`	`blockDigestEncoding` Default encoding scheme used to encode block digest into a string, if none is detected from the record.
`protected boolean`	`bPayloadDigest` Payload Digest enabled/disabled.
`protected long`	`consumed` Number of bytes consumed by this reader.
`protected WarcRecord`	`currentRecord` Current WARC record object.
`Diagnostics<Diagnosis>`	`diagnostics` Reader level errors and warnings or when no record is available.
`protected int`	`errors` Aggregated number of errors encountered while parsing.
`protected WarcFieldParsers`	`fieldParsers` WARC field parser used.
`protected HeaderLineReader`	`headerLineReader` Header line reader used to read the WARC headers.
`protected Exception`	`iteratorExceptionThrown` Exception thrown while using the iterator.
`protected HeaderLineReader`	`lineReader` Line reader used to read version lines.
`protected String`	`payloadDigestAlgorithm` Default payload digest algorithm to use if none is present in the record.
`protected String`	`payloadDigestEncoding` Default encoding scheme used to encode payload digest into a string, if none is detected from the record.
`protected int`	`payloadHeaderMaxSize` Max size allowed for a payload header.
`protected int`	`recordHeaderMaxSize` Max size allowed for a record header.
`protected int`	`records` Records parsed.
`protected UriProfile`	`uriProfile` URI profile.
`protected UriProfile`	`warcTargetUriProfile` WARC-Target-URI profile.
`protected int`	`warnings` Aggregate number of warnings encountered while parsing.

Constructor Summary

Constructors
Constructor and Description

WarcReader()

Constructors
Constructor and Description
`WarcReader()`

Method Summary

All Methods Instance Methods Abstract Methods Concrete Methods
Modifier and Type	Method and Description
`abstract void`	`close()` Close current record resource(s) and input stream(s).
`String`	`getBlockDigestAlgorithm()` Get the default block digest algorithm.
`boolean`	`getBlockDigestEnabled()` Get the readers block digest on/off status.
`String`	`getBlockDigestEncoding()` Get the default block digest encoding scheme.
`abstract long`	`getConsumed()` Get number of bytes consumed by this reader.
`Exception`	`getIteratorExceptionThrown()` Gets an exception thrown in the iterator if any or null.
`abstract WarcRecord`	`getNextRecord()` Parses and gets the next record.
`abstract WarcRecord`	`getNextRecordFrom(InputStream in, long offset)` Parses and gets the next record from an `Inputstream`.
`abstract WarcRecord`	`getNextRecordFrom(InputStream in, long offset, int buffer_size)` Parses and gets the next record from an `Inputstream` wrapped by a `BufferedInputStream`.
`abstract long`	`getOffset()` Get the current offset in the WARC `InputStream`.
`String`	`getPayloadDigestAlgorithm()` Get the default payload digest algorithm.
`boolean`	`getPayloadDigestEnabled()` Get the readers payload digest on/off status.
`String`	`getPayloadDigestEncoding()` Get the default payload digest encoding scheme.
`int`	`getPayloadHeaderMaxSize()` Get the max size allowed for a payload header.
`int`	`getRecordHeaderMaxSize()` Get the max size allowed for a record header.
`abstract long`	`getStartOffset()` Get the offset of the current WARC record or -1 if none have been read.
`UriProfile`	`getUriProfile()` Get the URI profile used to validate URIs.
`UriProfile`	`getWarcTargetUriProfile()` Get the URI profile used to validate WARC-Target URIs.
`protected void`	`init()` Method used to initialize a readers internal state.
`boolean`	`isCompliant()` Returns a boolean indicating if all records parsed so far are compliant.
`abstract boolean`	`isCompressed()` Is this reader assuming GZip compressed input.
`Iterator<WarcRecord>`	`iterator()` Returns an `Iterator` over the records as they are being parsed.
`protected abstract void`	`recordClosed()` Callback method called when the payload has been processed.
`void`	`reset()` Reset reader for reuse.
`boolean`	`setBlockDigestAlgorithm(String digestAlgorithm)` Tries to set the default block digest algorithm and returns a boolean indicating whether the algorithm was accepted or not.
`void`	`setBlockDigestEnabled(boolean enabled)` Set the readers block digest on/off status.
`void`	`setBlockDigestEncoding(String encodingScheme)` Set the default block digest encoding scheme.
`boolean`	`setPayloadDigestAlgorithm(String digestAlgorithm)` Tries to set the default payload digest algorithm and returns a boolean indicating whether the algorithm was accepted or not.
`void`	`setPayloadDigestEnabled(boolean enabled)` Set the readers payload digest on/off status.
`void`	`setPayloadDigestEncoding(String encodingScheme)` Set the default payload digest encoding scheme.
`void`	`setPayloadHeaderMaxSize(int size)` Set the max size allowed for a payload header.
`void`	`setRecordHeaderMaxSize(int size)` Set the max size allowed for a record header.
`void`	`setUriProfile(UriProfile uriProfile)` Set the URI profile used to validate URIs.
`void`	`setWarcTargetUriProfile(UriProfile uriProfile)` Set the URI profile used to validate WARC-Target URIs.

Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

- Field Detail
  - warcTargetUriProfile
```
protected UriProfile warcTargetUriProfile
```
    WARC-Target-URI profile.
  - uriProfile
```
protected UriProfile uriProfile
```
    URI profile.
  - blockDigestAlgorithm
```
protected String blockDigestAlgorithm
```
    Default block digest algorithm to use if none is present in the record.
  - blockDigestEncoding
```
protected String blockDigestEncoding
```
    Default encoding scheme used to encode block digest into a string, if none is detected from the record.
  - bPayloadDigest
```
protected boolean bPayloadDigest
```
    Payload Digest enabled/disabled.
  - payloadDigestAlgorithm
```
protected String payloadDigestAlgorithm
```
    Default payload digest algorithm to use if none is present in the record.
  - payloadDigestEncoding
```
protected String payloadDigestEncoding
```
    Default encoding scheme used to encode payload digest into a string, if none is detected from the record.
  - bBlockDigest
```
protected boolean bBlockDigest
```
    Block Digest enabled/disabled.
  - recordHeaderMaxSize
```
protected int recordHeaderMaxSize
```
    Max size allowed for a record header.
  - payloadHeaderMaxSize
```
protected int payloadHeaderMaxSize
```
    Max size allowed for a payload header.
  - lineReader
```
protected HeaderLineReader lineReader
```
    Line reader used to read version lines.
  - headerLineReader
```
protected HeaderLineReader headerLineReader
```
    Header line reader used to read the WARC headers.
  - fieldParsers
```
protected WarcFieldParsers fieldParsers
```
    WARC field parser used.
  - diagnostics
```
public final Diagnostics<Diagnosis> diagnostics
```
    Reader level errors and warnings or when no record is available.
  - bIsCompliant
```
protected boolean bIsCompliant
```
    Compliance status for records parsed up to now.
  - consumed
```
protected long consumed
```
    Number of bytes consumed by this reader.
  - records
```
protected int records
```
    Records parsed.
  - errors
```
protected int errors
```
    Aggregated number of errors encountered while parsing.
  - warnings
```
protected int warnings
```
    Aggregate number of warnings encountered while parsing.
  - currentRecord
```
protected WarcRecord currentRecord
```
    Current WARC record object.
  - iteratorExceptionThrown
```
protected Exception iteratorExceptionThrown
```
    Exception thrown while using the iterator.
- Constructor Detail
  - WarcReader
```
public WarcReader()
```
- Method Detail
  - init
```
protected void init()
```
    Method used to initialize a readers internal state. Must be called by all constructors.
  - reset
```
public void reset()
```
    Reset reader for reuse.
  - isCompliant
```
public boolean isCompliant()
```
    Returns a boolean indicating if all records parsed so far are compliant.
    
    Returns:
    
    a boolean indicating if all records parsed so far are compliant
  - isCompressed
```
public abstract boolean isCompressed()
```
    Is this reader assuming GZip compressed input.
    
    Returns:
    
    boolean indicating the assumption of GZip compressed input
  - setWarcTargetUriProfile
```
public void setWarcTargetUriProfile(UriProfile uriProfile)
```
    Set the URI profile used to validate WARC-Target URIs. If null, the uriProfile is set to RCF3986.
    
    Parameters:
    
    uriProfile - URI profile to use
  - getWarcTargetUriProfile
```
public UriProfile getWarcTargetUriProfile()
```
    Get the URI profile used to validate WARC-Target URIs.
    
    Returns:
    
    the URI profile used to validate WARC-Target URIs
  - setUriProfile
```
public void setUriProfile(UriProfile uriProfile)
```
    Set the URI profile used to validate URIs. If null, the uriProfile is set to RCF3986.
    
    Parameters:
    
    uriProfile - URI profile to use
  - getUriProfile
```
public UriProfile getUriProfile()
```
    Get the URI profile used to validate URIs.
    
    Returns:
    
    the URI profile used to validate URIs
  - getBlockDigestEnabled
```
public boolean getBlockDigestEnabled()
```
    Get the readers block digest on/off status.
    
    Returns:
    
    boolean indicating block digest on/off
  - setBlockDigestEnabled
```
public void setBlockDigestEnabled(boolean enabled)
```
    Set the readers block digest on/off status. Digest, however, will only be computed if either a Warc-Block-Digest header is present or an optional algorithm has been chosen. The Warc-Block-Digest always takes precedence.
    
    Parameters:
    
    enabled - boolean indicating block digest on/off
  - getPayloadDigestEnabled
```
public boolean getPayloadDigestEnabled()
```
    Get the readers payload digest on/off status.
    
    Returns:
    
    boolean indicating payload digest on/off
  - setPayloadDigestEnabled
```
public void setPayloadDigestEnabled(boolean enabled)
```
    Set the readers payload digest on/off status. Digest, however, will only be computed if either a Warc-Payload-Digest header is present or an optional algorithm has been chosen. The Warc-Payload-Digest always takes precedence.
    
    Parameters:
    
    enabled - boolean indicating payload digest on/off
  - getBlockDigestAlgorithm
```
public String getBlockDigestAlgorithm()
```
    Get the default block digest algorithm.
    
    Returns:
    
    default block digest algorithm
  - setBlockDigestAlgorithm
```
public boolean setBlockDigestAlgorithm(String digestAlgorithm)
```
    Tries to set the default block digest algorithm and returns a boolean indicating whether the algorithm was accepted or not. This algorithm is only used in case no WARC payload digest header is present in the record.
    
    Parameters:
    
    digestAlgorithm - block digest algorithm (null means no default block digest algorithm is selected)
    
    Returns:
    
    boolean indicating the validity of the algorithm supplied
  - getPayloadDigestAlgorithm
```
public String getPayloadDigestAlgorithm()
```
    Get the default payload digest algorithm.
    
    Returns:
    
    default payload digest algorithm
  - setPayloadDigestAlgorithm
```
public boolean setPayloadDigestAlgorithm(String digestAlgorithm)
```
    Tries to set the default payload digest algorithm and returns a boolean indicating whether the algorithm was accepted or not. This algorithm is only used in case no WARC payload digest header is present in the record.
    
    Parameters:
    
    digestAlgorithm - payload digest algorithm (null means no default payload digest algorithm is selected)
    
    Returns:
    
    boolean indicating the validity of the algorithm supplied
  - getBlockDigestEncoding
```
public String getBlockDigestEncoding()
```
    Get the default block digest encoding scheme.
    
    Returns:
    
    default block digest encoding scheme
  - setBlockDigestEncoding
```
public void setBlockDigestEncoding(String encodingScheme)
```
    Set the default block digest encoding scheme. This scheme is only used if none can be inferred from an existing block digest header.
    
    Parameters:
    
    encodingScheme - encoding scheme (null means default block digest is not encoded)
  - getPayloadDigestEncoding
```
public String getPayloadDigestEncoding()
```
    Get the default payload digest encoding scheme.
    
    Returns:
    
    default payload digest encoding scheme
  - setPayloadDigestEncoding
```
public void setPayloadDigestEncoding(String encodingScheme)
```
    Set the default payload digest encoding scheme. This scheme is only used if none can be inferred from an existing payload digest header.
    
    Parameters:
    
    encodingScheme - encoding scheme (null means default payload digest is not encoded)
  - getRecordHeaderMaxSize
```
public int getRecordHeaderMaxSize()
```
    Get the max size allowed for a record header.
    
    Returns:
    
    max size allowed for a record header
  - setRecordHeaderMaxSize
```
public void setRecordHeaderMaxSize(int size)
```
    Set the max size allowed for a record header.
    
    Parameters:
    
    size - max size allowed
  - getPayloadHeaderMaxSize
```
public int getPayloadHeaderMaxSize()
```
    Get the max size allowed for a payload header.
    
    Returns:
    
    max size allowed for a payload header
  - setPayloadHeaderMaxSize
```
public void setPayloadHeaderMaxSize(int size)
```
    Set the max size allowed for a payload header.
    
    Parameters:
    
    size - max size allowed
  - close
```
public abstract void close()
```
    Close current record resource(s) and input stream(s).
    
    Specified by:
    
    close in interface Closeable
    
    Specified by:
    
    close in interface AutoCloseable
  - recordClosed
```
protected abstract void recordClosed()
```
    Callback method called when the payload has been processed.
  - getStartOffset
```
public abstract long getStartOffset()
```
    Get the offset of the current WARC record or -1 if none have been read.
    
    Returns:
    
    offset of the current WARC record or -1
  - getOffset
```
public abstract long getOffset()
```
    Get the current offset in the WARC InputStream.
    
    Returns:
    
    offset in WARC InputStream
  - getConsumed
```
public abstract long getConsumed()
```
    Get number of bytes consumed by this reader.
    
    Returns:
    
    number of bytes consumed by this reader
  - getNextRecord
```
public abstract WarcRecord getNextRecord()
                                  throws IOException
```
    Parses and gets the next record. This method is for linear access to records.
    
    Returns:
    
    the next record
    
    Throws:
    
    IOException - i/o exception in parsing process
  - getNextRecordFrom
```
public abstract WarcRecord getNextRecordFrom(InputStream in,
                                             long offset)
                                      throws IOException
```
    Parses and gets the next record from an Inputstream. This method is mainly for random access use since there are serious side-effects involved in using multiple PushBackInputStream instances.
    
    Parameters:
    
    in - InputStream used to read next record
    
    offset - offset provided by caller
    
    Returns:
    
    the next record
    
    Throws:
    
    IOException - i/o exception in parsing process
  - getNextRecordFrom
```
public abstract WarcRecord getNextRecordFrom(InputStream in,
                                             long offset,
                                             int buffer_size)
                                      throws IOException
```
    Parses and gets the next record from an Inputstream wrapped by a BufferedInputStream. This method is mainly for random access use since there are serious side-effects involved in using multiple PushBackInputStream instances.
    
    Parameters:
    
    in - InputStream used to read next record
    
    offset - offset provided by caller
    
    buffer_size - buffer size to use
    
    Returns:
    
    the next record
    
    Throws:
    
    IOException - i/o exception in parsing process
  - getIteratorExceptionThrown
```
public Exception getIteratorExceptionThrown()
```
    Gets an exception thrown in the iterator if any or null.
    
    Returns:
    
    exception thrown in the iterator if any or null
  - iterator
```
public Iterator<WarcRecord> iterator()
```
    Returns an Iterator over the records as they are being parsed. Any exception thrown during parsing is accessible through the getIteratorExceptionThrown method.
    
    Returns:
    
    Iterator over the records

Class WarcReader

Field Summary

Constructor Summary

Method Summary

Methods inherited from class java.lang.Object

Field Detail

warcTargetUriProfile

uriProfile

blockDigestAlgorithm

blockDigestEncoding

bPayloadDigest

payloadDigestAlgorithm

payloadDigestEncoding

bBlockDigest

recordHeaderMaxSize

payloadHeaderMaxSize

lineReader

headerLineReader

fieldParsers

diagnostics

bIsCompliant

consumed

records

errors

warnings

currentRecord

iteratorExceptionThrown

Constructor Detail

WarcReader

Method Detail

init

reset

isCompliant

isCompressed

setWarcTargetUriProfile

getWarcTargetUriProfile

setUriProfile

getUriProfile

getBlockDigestEnabled

setBlockDigestEnabled

getPayloadDigestEnabled

setPayloadDigestEnabled

getBlockDigestAlgorithm

setBlockDigestAlgorithm

getPayloadDigestAlgorithm

setPayloadDigestAlgorithm

getBlockDigestEncoding

setBlockDigestEncoding

getPayloadDigestEncoding

setPayloadDigestEncoding

getRecordHeaderMaxSize

setRecordHeaderMaxSize

getPayloadHeaderMaxSize

setPayloadHeaderMaxSize

close

recordClosed

getStartOffset

getOffset

getConsumed

getNextRecord

getNextRecordFrom

getNextRecordFrom

getIteratorExceptionThrown

iterator