org.nd4j.api.loader

Class FileBatch

java.lang.Object

org.nd4j.api.loader.FileBatch

All Implemented Interfaces:

java.io.Serializable

public class FileBatch
extends java.lang.Object
implements java.io.Serializable

FileBatch: stores the raw contents of multiple files in byte arrays (one per file) along with their original paths. FileBatch can be stored to disk (or any output stream) in zip format. Typical use cases include creating batches of data in their raw format for distributed training. This can reduce the number of disk reads required (fewer files) and network transfers when reading from remote storage, due to the zip format used for compression.

Author:

Alex Black

See Also:

Serialized Form

Field Summary

Fields

Modifier and Type	Field and Description
static java.lang.String	ORIGINAL_PATHS_FILENAME Name of the file in the zip file that contains the original paths/filenames

Constructor Summary

Constructors

Constructor and Description
FileBatch()

Method Summary

Modifier and Type	Method and Description
static FileBatch	forFiles(java.io.File... files) Create a FileBatch from the specified files
static FileBatch	forFiles(java.util.List<java.io.File> files) Create a FileBatch from the specified files
static FileBatch	readFromZip(java.io.File file) Read a FileBatch from the specified file.
static FileBatch	readFromZip(java.io.InputStream is) Read a FileBatch from the specified input stream.
void	writeAsZip(java.io.File f) Write the FileBatch to the specified File, in zip file format
void	writeAsZip(java.io.OutputStream os)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

ORIGINAL_PATHS_FILENAME

public static final java.lang.String ORIGINAL_PATHS_FILENAME

Name of the file in the zip file that contains the original paths/filenames

See Also:

Constant Field Values

Constructor Detail

FileBatch

public FileBatch()

Method Detail

readFromZip

public static FileBatch readFromZip(java.io.File file)
                             throws java.io.IOException

Read a FileBatch from the specified file. This method assumes the FileBatch was previously saved to zip format using writeAsZip(File) or writeAsZip(OutputStream)

Parameters:

file - File to read from

Returns:

The loaded FileBatch

Throws:

java.io.IOException - If an error occurs during reading

readFromZip

public static FileBatch readFromZip(java.io.InputStream is)
                             throws java.io.IOException

Read a FileBatch from the specified input stream. This method assumes the FileBatch was previously saved to zip format using writeAsZip(File) or writeAsZip(OutputStream)

Parameters:

is - Input stream to read from

Returns:

The loaded FileBatch

Throws:

java.io.IOException - If an error occurs during reading

forFiles

public static FileBatch forFiles(java.io.File... files)
                          throws java.io.IOException

Create a FileBatch from the specified files

Parameters:

files - Files to create the FileBatch from

Returns:

The created FileBatch

Throws:

java.io.IOException - If an error occurs during reading of the file content

forFiles

public static FileBatch forFiles(java.util.List<java.io.File> files)
                          throws java.io.IOException

Create a FileBatch from the specified files

Parameters:

files - Files to create the FileBatch from

Returns:

The created FileBatch

Throws:

java.io.IOException - If an error occurs during reading of the file content

writeAsZip

public void writeAsZip(java.io.File f)
                throws java.io.IOException

Write the FileBatch to the specified File, in zip file format

Parameters:

f - File to write to

Throws:

java.io.IOException - If an error occurs during writing

writeAsZip

public void writeAsZip(java.io.OutputStream os)
                throws java.io.IOException

Parameters:

os - Write the FileBatch to the specified output stream, in zip file format

Throws:

java.io.IOException - If an error occurs during writing