WarcWriter (jwat-warc 1.0.3 API)

java.lang.Object
- org.jwat.warc.WarcWriter

All Implemented Interfaces:

Closeable, AutoCloseable

Direct Known Subclasses:

WarcWriterCompressed, WarcWriterUncompressed
```
public abstract class WarcWriter
extends Object
implements Closeable
```
Base class for WARC writer implementations.

Author:

nicl

Field Summary

Fields
Modifier and Type	Field and Description
`protected boolean`	`bExceptionOnContentLengthMismatch` Configuration for throwing exception on content-length mismatch.
`Diagnostics<Diagnosis>`	`diagnostics` Writer level errors and warnings or when writing byte headers.
`protected WarcFieldParsers`	`fieldParsers` WARC field parser used.
`protected WarcHeader`	`header` Current WARC header written.
`protected Long`	`headerContentLength` Content-Length from the WARC header.
`protected OutputStream`	`out` Outputstream used to write WARC records.
`protected long`	`payloadWrittenTotal` Total bytes written for current record payload.
`protected static int`	`S_HEADER_WRITTEN` State after header has been written.
`protected static int`	`S_INIT` State after writer has been constructed and before records have been written.
`protected static int`	`S_PAYLOAD_WRITTEN` State after payload has been written.
`protected static int`	`S_RECORD_CLOSED` State after record has been closed.
`protected int`	`state` Current state of writer.
`protected byte[]`	`stream_copy_buffer` Buffer used by streamPayload() to copy from one stream to another.
`protected UriProfile`	`uriProfile` URI profile.
`protected DateFormat`	`warcDateFormat` WARC `DateFormat` as specified by the WARC ISO standard.
`protected UriProfile`	`warcTargetUriProfile` WARC-Target-URI profile.

Constructor Summary

Constructors
Constructor and Description

WarcWriter()

Constructors
Constructor and Description
`WarcWriter()`

Method Summary

All Methods Instance Methods Abstract Methods Concrete Methods
Modifier and Type	Method and Description
`abstract void`	`close()` Close WARC writer and free its resources.
`protected void`	`closeRecord_impl()` Closes the WARC record by writing two newlines and comparing the amount of payload data streamed with the content-length supplied with the header.
`abstract void`	`closeRecord()` Close the WARC record in an implementation specific way.
`boolean`	`exceptionOnContentLengthMismatch()` Does this writer throw an exception if the content-length does not match the payload amount written.
`UriProfile`	`getUriProfile()` Get the URI profile used to validate URIs.
`UriProfile`	`getWarcTargetUriProfile()` Get the URI profile used to validate WARC-Target URIs.
`protected void`	`init()` Method used to initialize a readers internal state.
`abstract boolean`	`isCompressed()` Is this writer producing compressed output.
`void`	`setExceptionOnContentLengthMismatch(boolean enabled)` Tell the writer what to do in case of mismatch between content-length and amount payload written.
`void`	`setUriProfile(UriProfile uriProfile)` Set the URI profile used to validate URIs.
`void`	`setWarcTargetUriProfile(UriProfile uriProfile)` Set the URI profile used to validate WARC-Target URIs.
`long`	`streamPayload(InputStream in)` Stream the content of an input stream to the payload content.
`protected byte[]`	`writeHeader_impl(WarcRecord record)` Write a WARC header to the WARC output stream.
`abstract byte[]`	`writeHeader(WarcRecord record)` Write a WARC header to the WARC output stream.
`long`	`writePayload(byte[] b)` Append the content of a byte array to the payload content.
`long`	`writePayload(byte[] b, int offset, int len)` Append the partial content of a byte array to the payload content.
`void`	`writeRawHeader(byte[] header_bytes, Long contentLength)` Write a raw WARC header to the WARC output stream.

Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

- Field Detail
  - S_INIT
```
protected static final int S_INIT
```
    State after writer has been constructed and before records have been written.
    
    See Also:
    
    Constant Field Values
  - S_HEADER_WRITTEN
```
protected static final int S_HEADER_WRITTEN
```
    State after header has been written.
    
    See Also:
    
    Constant Field Values
  - S_PAYLOAD_WRITTEN
```
protected static final int S_PAYLOAD_WRITTEN
```
    State after payload has been written.
    
    See Also:
    
    Constant Field Values
  - S_RECORD_CLOSED
```
protected static final int S_RECORD_CLOSED
```
    State after record has been closed.
    
    See Also:
    
    Constant Field Values
  - warcTargetUriProfile
```
protected UriProfile warcTargetUriProfile
```
    WARC-Target-URI profile.
  - uriProfile
```
protected UriProfile uriProfile
```
    URI profile.
  - warcDateFormat
```
protected DateFormat warcDateFormat
```
    WARC DateFormat as specified by the WARC ISO standard.
  - fieldParsers
```
protected WarcFieldParsers fieldParsers
```
    WARC field parser used.
  - stream_copy_buffer
```
protected byte[] stream_copy_buffer
```
    Buffer used by streamPayload() to copy from one stream to another.
  - bExceptionOnContentLengthMismatch
```
protected boolean bExceptionOnContentLengthMismatch
```
    Configuration for throwing exception on content-length mismatch. (Default is true)
  - diagnostics
```
public final Diagnostics<Diagnosis> diagnostics
```
    Writer level errors and warnings or when writing byte headers.
  - state
```
protected int state
```
    Current state of writer.
  - out
```
protected OutputStream out
```
    Outputstream used to write WARC records.
  - header
```
protected WarcHeader header
```
    Current WARC header written.
  - headerContentLength
```
protected Long headerContentLength
```
    Content-Length from the WARC header.
  - payloadWrittenTotal
```
protected long payloadWrittenTotal
```
    Total bytes written for current record payload.
- Constructor Detail
  - WarcWriter
```
public WarcWriter()
```
- Method Detail
  - init
```
protected void init()
```
    Method used to initialize a readers internal state. Must be called by all constructors.
  - isCompressed
```
public abstract boolean isCompressed()
```
    Is this writer producing compressed output.
    
    Returns:
    
    boolean indicating whether compressed output is produced
  - setWarcTargetUriProfile
```
public void setWarcTargetUriProfile(UriProfile uriProfile)
```
    Set the URI profile used to validate WARC-Target URIs. If null, the uriProfile is set to RCF3986.
    
    Parameters:
    
    uriProfile - URI profile to use
  - getWarcTargetUriProfile
```
public UriProfile getWarcTargetUriProfile()
```
    Get the URI profile used to validate WARC-Target URIs.
    
    Returns:
    
    the URI profile used to validate WARC-Target URIs
  - setUriProfile
```
public void setUriProfile(UriProfile uriProfile)
```
    Set the URI profile used to validate URIs. If null, the uriProfile is set to RCF3986.
    
    Parameters:
    
    uriProfile - URI profile to use
  - getUriProfile
```
public UriProfile getUriProfile()
```
    Get the URI profile used to validate URIs.
    
    Returns:
    
    the URI profile used to validate URIs
  - exceptionOnContentLengthMismatch
```
public boolean exceptionOnContentLengthMismatch()
```
    Does this writer throw an exception if the content-length does not match the payload amount written.
    
    Returns:
    
    boolean indicating if an exception is thrown or not
  - setExceptionOnContentLengthMismatch
```
public void setExceptionOnContentLengthMismatch(boolean enabled)
```
    Tell the writer what to do in case of mismatch between content-length and amount payload written.
    
    Parameters:
    
    enabled - boolean indicating exception throwing on/off
  - close
```
public abstract void close()
                    throws IOException
```
    Close WARC writer and free its resources.
    
    Specified by:
    
    close in interface Closeable
    
    Specified by:
    
    close in interface AutoCloseable
    
    Throws:
    
    IOException - if an i/o exception occurs while closing the writer
  - closeRecord
```
public abstract void closeRecord()
                          throws IOException
```
    Close the WARC record in an implementation specific way.
    
    Throws:
    
    IOException - if an i/o exception occurs while closing the record
  - closeRecord_impl
```
protected void closeRecord_impl()
                         throws IOException
```
    Closes the WARC record by writing two newlines and comparing the amount of payload data streamed with the content-length supplied with the header.
    
    Throws:
    
    IOException - if an i/o exception occurs while closing the record
  - writeRawHeader
```
public void writeRawHeader(byte[] header_bytes,
                           Long contentLength)
                    throws IOException
```
    Write a raw WARC header to the WARC output stream. Closes any previously written record that has not been closed prior to this call. Errors and warnings are reported on the writers diagnostics object.
    
    Parameters:
    
    header_bytes - raw WARC header to output
    
    contentLength - the expected content-length to be written and validated
    
    Throws:
    
    IOException - if an i/o exception occurs while writing header data
  - writeHeader
```
public abstract byte[] writeHeader(WarcRecord record)
                            throws IOException
```
    Write a WARC header to the WARC output stream. Errors and warnings are reported on the records diagnostics object.
    
    Parameters:
    
    record - WARC record to output
    
    Returns:
    
    byte array version of header as it was written
    
    Throws:
    
    IOException - if an i/o exception occurs while writing header data
  - writeHeader_impl
```
protected byte[] writeHeader_impl(WarcRecord record)
                           throws IOException
```
    Write a WARC header to the WARC output stream. The WARC header is not required to be valid. Errors and warnings are reported on the records diagnostics object.
    
    Parameters:
    
    record - WARC record to output
    
    Returns:
    
    byte array version of header as it was written
    
    Throws:
    
    IOException - if an i/o exception occurs while writing header data
  - streamPayload
```
public long streamPayload(InputStream in)
                   throws IOException
```
    Stream the content of an input stream to the payload content.
    
    Parameters:
    
    in - input stream containing payload data
    
    Returns:
    
    number of bytes written during method invocation
    
    Throws:
    
    IOException - if an i/o exception occurs while writing payload data
  - writePayload
```
public long writePayload(byte[] b)
                  throws IOException
```
    Append the content of a byte array to the payload content.
    
    Parameters:
    
    b - byte array with data to be written
    
    Returns:
    
    number of bytes written during method invocation
    
    Throws:
    
    IOException - if an i/o exception occurs while writing payload data
  - writePayload
```
public long writePayload(byte[] b,
                         int offset,
                         int len)
                  throws IOException
```
    Append the partial content of a byte array to the payload content.
    
    Parameters:
    
    b - byte array with partial data to be written
    
    offset - offset to data to be written
    
    len - length of data to be written
    
    Returns:
    
    number of bytes written during method invocation
    
    Throws:
    
    IOException - if an i/o exception occurs while writing payload data

Class WarcWriter

Field Summary

Constructor Summary

Method Summary

Methods inherited from class java.lang.Object

Field Detail

S_INIT

S_HEADER_WRITTEN

S_PAYLOAD_WRITTEN

S_RECORD_CLOSED

warcTargetUriProfile

uriProfile

warcDateFormat

fieldParsers

stream_copy_buffer

bExceptionOnContentLengthMismatch

diagnostics

state

out

header

headerContentLength

payloadWrittenTotal

Constructor Detail

WarcWriter

Method Detail

init

isCompressed

setWarcTargetUriProfile

getWarcTargetUriProfile

setUriProfile

getUriProfile

exceptionOnContentLengthMismatch

setExceptionOnContentLengthMismatch

close

closeRecord

closeRecord_impl

writeRawHeader

writeHeader

writeHeader_impl

streamPayload

writePayload

writePayload