net.snowflake.client.jdbc

Class SnowflakeFileTransferAgent

java.lang.Object

net.snowflake.client.jdbc.SnowflakeFileTransferAgent

All Implemented Interfaces:

SnowflakeFixedView

public class SnowflakeFileTransferAgent
extends Object
implements SnowflakeFixedView

Class for uploading/downloading files

Author:

jhuang

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	SnowflakeFileTransferAgent.CommandType
class	SnowflakeFileTransferAgent.DownloadCommandEncryptionFacade
class	SnowflakeFileTransferAgent.DownloadCommandFacade A class for encapsulating the columns to return for the download command
static class	SnowflakeFileTransferAgent.FileCompressionType
static class	SnowflakeFileTransferAgent.ResultStatus Result status enum
static class	SnowflakeFileTransferAgent.UploadColumns A class for encapsulating the columns to return for the upload command
class	SnowflakeFileTransferAgent.UploadCommandEncryptionFacade
class	SnowflakeFileTransferAgent.UploadCommandFacade

Field Summary

Fields

Modifier and Type	Field and Description
static String	SRC_FILE_NAME_FOR_STREAM

Constructor Summary

Constructors

Constructor and Description
SnowflakeFileTransferAgent(String command, SFSession connection, SFStatement statement)

Method Summary

Modifier and Type	Method and Description
void	cancel()
List<SnowflakeColumnMetadata>	describeColumns() Describe the metadata of a fixed view.
boolean	execute()
static Set<String>	expandFileNames(String[] filePathList) process a list of file paths separated by "," and expand the wildcards if any to generate the list of paths for all files matched by the wildcards
static net.snowflake.client.jdbc.SnowflakeFileTransferAgent.S3Location	extractBucketNameAndPath(String stageLocation) A small helper for extracting bucket name and path from stage location.
SnowflakeFileTransferAgent.CommandType	getCommandType()
static Callable<Void>	getDownloadFileCallable(String stageLocationType, String stageLocation, Map credentials, String srcFilePath, String localLocation, Map<String,net.snowflake.client.jdbc.SnowflakeFileTransferAgent.FileMetadata> fileMetadataMap, SnowflakeS3Client s3Client, SFSession connection, String command, int parallel, net.snowflake.common.core.S3FileEncryptionMaterial encMat, String stageRegion) A callable that can be executed in a separate thread using exeuctor service.
List<net.snowflake.common.core.S3FileEncryptionMaterial>	getEncryptionMaterial()
List<Object>	getNextRow()
Object	getResultSet()
Map<String,net.snowflake.common.core.S3FileEncryptionMaterial>	getSrcToMaterialsMap()
Map	getStageCredentials()
String	getStageLocation()
static Callable<Void>	getUploadFileCallable(String stageLocationType, String stageLocation, String srcFilePath, boolean requireCompress, Map<String,net.snowflake.client.jdbc.SnowflakeFileTransferAgent.FileMetadata> fileMetadataMap, SnowflakeS3Client s3Client, SFSession connection, String command, InputStream inputStream, boolean sourceFromStream, long sourceDataSize, int parallel, File srcFile, net.snowflake.common.core.S3FileEncryptionMaterial encMat, String stageRegion) A callable that can be executed in a separate thread using exeuctor service.
void	setCompressSourceFromStream(boolean compressSourceFromStream)
void	setDestFileNameForStreamSource(String destFileNameForStreamSource)
void	setOverwrite(boolean overwrite)
void	setSourceStream(InputStream sourceStream)
void	setSourceStreamSize(long sourceStreamSize)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

SRC_FILE_NAME_FOR_STREAM

public static final String SRC_FILE_NAME_FOR_STREAM

See Also:

Constant Field Values

Constructor Detail

SnowflakeFileTransferAgent

public SnowflakeFileTransferAgent(String command,
                                  SFSession connection,
                                  SFStatement statement)
                           throws SnowflakeSQLException

Throws:

SnowflakeSQLException

Method Detail

getStageCredentials

public Map getStageCredentials()

getEncryptionMaterial

public List<net.snowflake.common.core.S3FileEncryptionMaterial> getEncryptionMaterial()

getSrcToMaterialsMap

public Map<String,net.snowflake.common.core.S3FileEncryptionMaterial> getSrcToMaterialsMap()

getStageLocation

public String getStageLocation()

getUploadFileCallable

public static Callable<Void> getUploadFileCallable(String stageLocationType,
                                                   String stageLocation,
                                                   String srcFilePath,
                                                   boolean requireCompress,
                                                   Map<String,net.snowflake.client.jdbc.SnowflakeFileTransferAgent.FileMetadata> fileMetadataMap,
                                                   SnowflakeS3Client s3Client,
                                                   SFSession connection,
                                                   String command,
                                                   InputStream inputStream,
                                                   boolean sourceFromStream,
                                                   long sourceDataSize,
                                                   int parallel,
                                                   File srcFile,
                                                   net.snowflake.common.core.S3FileEncryptionMaterial encMat,
                                                   String stageRegion)

A callable that can be executed in a separate thread using exeuctor service. The callable does compression if needed and upload the result to the table's staging area.

Parameters:

stageLocationType - stage location type

stageLocation - stage location

srcFilePath - source file path

requireCompress - boolean indicating if requiring compression

fileMetadataMap - map where key is file name and value is metadata

s3Client - client object used to communicate with c3

connection - connection object

command - command string

inputStream - null if upload source is file

sourceFromStream - whether upload source is file or stream

sourceDataSize - upload data size

parallel - number of threads for parallel uploading

srcFile - source file name

encMat - not null if encryption is required

stageRegion - region name where stage persists

Returns:

a callable that uploading file to s3

getDownloadFileCallable

public static Callable<Void> getDownloadFileCallable(String stageLocationType,
                                                     String stageLocation,
                                                     Map credentials,
                                                     String srcFilePath,
                                                     String localLocation,
                                                     Map<String,net.snowflake.client.jdbc.SnowflakeFileTransferAgent.FileMetadata> fileMetadataMap,
                                                     SnowflakeS3Client s3Client,
                                                     SFSession connection,
                                                     String command,
                                                     int parallel,
                                                     net.snowflake.common.core.S3FileEncryptionMaterial encMat,
                                                     String stageRegion)

A callable that can be executed in a separate thread using exeuctor service. The callable download files from a stage location to a local location

Parameters:

stageLocationType - stage location type

stageLocation - stage location

credentials - credentials

srcFilePath - path that stores the downloaded file

localLocation - local location

fileMetadataMap - file metadata map

s3Client - s3 client

connection - connection object

command - command string

encMat - s3 encryption material

parallel - number of parallel threads for downloading

stageRegion - region name where the stage persists

Returns:

a callable responsible for downloading files

execute

public boolean execute()
                throws SQLException

Throws:

SQLException

cancel

public void cancel()

expandFileNames

public static Set<String> expandFileNames(String[] filePathList)
                                   throws SnowflakeSQLException

process a list of file paths separated by "," and expand the wildcards if any to generate the list of paths for all files matched by the wildcards

Parameters:

filePathList - file path list

Returns:

a set of file names that is matched

Throws:

SnowflakeSQLException - if cannot find the file

extractBucketNameAndPath

public static net.snowflake.client.jdbc.SnowflakeFileTransferAgent.S3Location extractBucketNameAndPath(String stageLocation)

A small helper for extracting bucket name and path from stage location.

Parameters:

stageLocation - stage location

Returns:

s3 location

describeColumns

public List<SnowflakeColumnMetadata> describeColumns()
                                              throws Exception

Describe the metadata of a fixed view.

Specified by:

describeColumns in interface SnowflakeFixedView

Returns:

list of column meta data

Throws:

Exception - failed to construct list

getNextRow

public List<Object> getNextRow()
                        throws Exception

Specified by:

getNextRow in interface SnowflakeFixedView

Throws:

Exception

getResultSet

public Object getResultSet()
                    throws SnowflakeSQLException

Throws:

SnowflakeSQLException

getCommandType

public SnowflakeFileTransferAgent.CommandType getCommandType()

setSourceStream

public void setSourceStream(InputStream sourceStream)

setDestFileNameForStreamSource

public void setDestFileNameForStreamSource(String destFileNameForStreamSource)

setSourceStreamSize

public void setSourceStreamSize(long sourceStreamSize)

setCompressSourceFromStream

public void setCompressSourceFromStream(boolean compressSourceFromStream)

setOverwrite

public void setOverwrite(boolean overwrite)