WarcRecord (jwat-warc 1.0.3 API)

java.lang.Object
- org.jwat.warc.WarcRecord

All Implemented Interfaces:

Closeable, AutoCloseable, PayloadOnClosedHandler
```
public class WarcRecord
extends Object
implements PayloadOnClosedHandler, Closeable
```
This class represents a parsed WARC record header block including possible validation and format warnings/errors encountered in the process. The payload of the WARC record is accessible through a wrapped payload object.

Author:

nicl

Field Summary

Fields
Modifier and Type	Field and Description
`protected boolean`	`bClosed` Has record been closed before.
`protected boolean`	`bIsCompliant` Is this record compliant ie.
`protected boolean`	`bPayloadClosed` Has payload been closed before.
`WarcDigest`	`computedBlockDigest` Computed block digest.
`WarcDigest`	`computedPayloadDigest` Computed payload digest.
`protected long`	`consumed` Uncompressed bytes consumed while validating this record.
`Diagnostics<Diagnosis>`	`diagnostics` Validation errors and warnings.
`WarcHeader`	`header` WARC header.
`protected HttpHeader`	`httpHeader` HTTP header content parsed from payload.
`protected ByteCountingPushBackInputStream`	`in` Input stream used to read this record.
`Boolean`	`isValidBlockDigest` Is Warc-Block-Digest valid.
`Boolean`	`isValidPayloadDigest` Is Warc-Payload-Digest valid.
`NewlineParser`	`nlp` Newline parser for counting/validating trailing newlines.
`protected Payload`	`payload` Payload object if any exists.
`protected WarcReader`	`reader` Reader instance used, required for file compliance.
`protected long`	`startOffset` WARC record parsing start offset relative to the source WARC file input stream.
`int`	`trailingNewlines` Number of trailing newlines after record.

Constructor Summary

Constructors
Modifier Constructor and Description

protected WarcRecord()
Non public constructor to allow unit testing.

Constructors
Modifier	Constructor and Description
`protected`	`WarcRecord()` Non public constructor to allow unit testing.

Method Summary

All Methods Static Methods Instance Methods Concrete Methods
Modifier and Type	Method and Description
`protected void`	`addErrorDiagnosis(DiagnosisType type, String entity, String... information)` Add an error diagnosis of the given type on a specific entity with optional extra information.
`void`	`close()` Close resources associated with the WARC record.
`static WarcRecord`	`createRecord(WarcWriter writer)` Create a `WarcRecord` and prepare it for writing.
`long`	`getConsumed()` Return number of uncompressed bytes consumed validating this record.
`HeaderLine`	`getHeader(String field)` Get a non-standard WARC header or null, if nothing is stored for this header name.
`List<HeaderLine>`	`getHeaderList()` Get a `List` of all the non-standard WARC headers found during parsing.
`HttpHeader`	`getHttpHeader()` Returns the `HttpHeader` object if identified in the payload, or null.
`Payload`	`getPayload()` Return Payload object.
`InputStream`	`getPayloadContent()` Payload content `InputStream` getter.
`long`	`getStartOffset()` Get the record offset relative to the start of the WARC file `InputStream`.
`boolean`	`hasPayload()` Specifies whether this record has a payload or not.
`boolean`	`isClosed()` Check to see if the record has been closed.
`boolean`	`isCompliant()` Returns a boolean indicating the ISO compliance status of this record.
`static WarcRecord`	`parseRecord(ByteCountingPushBackInputStream in, WarcReader reader)` Given an `InputStream` it tries to read and validate a WARC header block.
`void`	`payloadClosed()` Called when the payload object is closed and final steps in the validation process can be performed.
`protected void`	`processComputedDigest(WarcDigest computedDigest, String digestAlgorithm, String digestEncoding, String digestName)` Adjust algorithm and encoding information about computed block digest.
`protected Boolean`	`processWarcDigest(WarcDigest warcDigest, WarcDigest computedDigest, String digestName)` Auto-detect encoding used in WARC digest header and compare it to the internal one, if it has been computed.

Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

- Field Detail
  - reader
```
protected WarcReader reader
```
    Reader instance used, required for file compliance.
  - in
```
protected ByteCountingPushBackInputStream in
```
    Input stream used to read this record.
  - bIsCompliant
```
protected boolean bIsCompliant
```
    Is this record compliant ie. error free.
  - startOffset
```
protected long startOffset
```
    WARC record parsing start offset relative to the source WARC file input stream. Used to keep track of the uncompressed amount of bytes consumed.
  - consumed
```
protected long consumed
```
    Uncompressed bytes consumed while validating this record.
  - diagnostics
```
public final Diagnostics<Diagnosis> diagnostics
```
    Validation errors and warnings.
  - nlp
```
public NewlineParser nlp
```
    Newline parser for counting/validating trailing newlines.
  - isValidBlockDigest
```
public Boolean isValidBlockDigest
```
    Is Warc-Block-Digest valid. (Null is equal to not tested)
  - isValidPayloadDigest
```
public Boolean isValidPayloadDigest
```
    Is Warc-Payload-Digest valid. (Null is equal to not tested)
  - trailingNewlines
```
public int trailingNewlines
```
    Number of trailing newlines after record.
  - header
```
public WarcHeader header
```
    WARC header.
  - bPayloadClosed
```
protected boolean bPayloadClosed
```
    Has payload been closed before.
  - bClosed
```
protected boolean bClosed
```
    Has record been closed before.
  - payload
```
protected Payload payload
```
    Payload object if any exists.
  - httpHeader
```
protected HttpHeader httpHeader
```
    HTTP header content parsed from payload.
  - computedBlockDigest
```
public WarcDigest computedBlockDigest
```
    Computed block digest.
  - computedPayloadDigest
```
public WarcDigest computedPayloadDigest
```
    Computed payload digest.
- Constructor Detail
  - WarcRecord
```
protected WarcRecord()
```
    Non public constructor to allow unit testing.
- Method Detail
  - createRecord
```
public static WarcRecord createRecord(WarcWriter writer)
```
    Create a WarcRecord and prepare it for writing.
    
    Parameters:
    
    writer - writer which will be used to write the record
    
    Returns:
    
    a WarcRecord ready to be changed and then written
  - parseRecord
```
public static WarcRecord parseRecord(ByteCountingPushBackInputStream in,
                                     WarcReader reader)
                              throws IOException
```
    Given an InputStream it tries to read and validate a WARC header block.
    
    Parameters:
    
    in - InputStream containing WARC record data
    
    reader - WarcReader used, with access to user defined options
    
    Returns:
    
    WarcRecord or null
    
    Throws:
    
    IOException - i/o exception in the process of reading record
  - payloadClosed
```
public void payloadClosed()
                   throws IOException
```
    Called when the payload object is closed and final steps in the validation process can be performed.
    
    Specified by:
    
    payloadClosed in interface PayloadOnClosedHandler
    
    Throws:
    
    IOException - i/o exception in final validation processing
  - processWarcDigest
```
protected Boolean processWarcDigest(WarcDigest warcDigest,
                                    WarcDigest computedDigest,
                                    String digestName)
```
    Auto-detect encoding used in WARC digest header and compare it to the internal one, if it has been computed.
    
    Parameters:
    
    warcDigest - digest from WARC header
    
    computedDigest - internally compute digest
    
    digestName - used to identify the digest ("block" or "payload")
    
    Returns:
    
    WARC digest validity indication
  - processComputedDigest
```
protected void processComputedDigest(WarcDigest computedDigest,
                                     String digestAlgorithm,
                                     String digestEncoding,
                                     String digestName)
```
    Adjust algorithm and encoding information about computed block digest.
    
    Parameters:
    
    computedDigest - internally compute digest
    
    digestAlgorithm - default algorithm
    
    digestEncoding - default encoding
    
    digestName - used to identify the digest ("block" or "payload")
  - isClosed
```
public boolean isClosed()
```
    Check to see if the record has been closed.
    
    Returns:
    
    boolean indicating whether this record is closed or not
  - close
```
public void close()
           throws IOException
```
    Close resources associated with the WARC record. Mainly payload stream if any.
    
    Specified by:
    
    close in interface Closeable
    
    Specified by:
    
    close in interface AutoCloseable
    
    Throws:
    
    IOException - if unable to close resources
  - isCompliant
```
public boolean isCompliant()
```
    Returns a boolean indicating the ISO compliance status of this record.
    
    Returns:
    
    a boolean indicating the ISO compliance status of this record
  - getStartOffset
```
public long getStartOffset()
```
    Get the record offset relative to the start of the WARC file InputStream.
    
    Returns:
    
    the record offset relative to the start of the WARC file
  - getConsumed
```
public long getConsumed()
```
    Return number of uncompressed bytes consumed validating this record.
    
    Returns:
    
    number of uncompressed bytes consumed validating this record
  - getHeaderList
```
public List<HeaderLine> getHeaderList()
```
    Get a List of all the non-standard WARC headers found during parsing.
    
    Returns:
    
    List of HeaderLine
  - getHeader
```
public HeaderLine getHeader(String field)
```
    Get a non-standard WARC header or null, if nothing is stored for this header name.
    
    Parameters:
    
    field - header name
    
    Returns:
    
    HeaderLine structure or null
  - hasPayload
```
public boolean hasPayload()
```
    Specifies whether this record has a payload or not.
    
    Returns:
    
    true/false whether the ARC record has a payload
  - getPayload
```
public Payload getPayload()
```
    Return Payload object.
    
    Returns:
    
    payload or null
  - getPayloadContent
```
public InputStream getPayloadContent()
```
    Payload content InputStream getter.
    
    Returns:
    
    Payload content InputStream
  - getHttpHeader
```
public HttpHeader getHttpHeader()
```
    Returns the HttpHeader object if identified in the payload, or null.
    
    Returns:
    
    the HttpHeader object if identified or null
  - addErrorDiagnosis
```
protected void addErrorDiagnosis(DiagnosisType type,
                                 String entity,
                                 String... information)
```
    Add an error diagnosis of the given type on a specific entity with optional extra information. The information varies according to the diagnosis type.
    
    Parameters:
    
    type - diagnosis type
    
    entity - entity examined
    
    information - optional extra information

Class WarcRecord

Field Summary

Constructor Summary

Method Summary

Methods inherited from class java.lang.Object

Field Detail

reader

in

bIsCompliant

startOffset

consumed

diagnostics

nlp

isValidBlockDigest

isValidPayloadDigest

trailingNewlines

header

bPayloadClosed

bClosed

payload

httpHeader

computedBlockDigest

computedPayloadDigest

Constructor Detail

WarcRecord

Method Detail

createRecord

parseRecord

payloadClosed

processWarcDigest

processComputedDigest

isClosed

close

isCompliant

getStartOffset

getConsumed

getHeaderList

getHeader

hasPayload

getPayload

getPayloadContent

getHttpHeader

addErrorDiagnosis