Flat file DataFlow connection reference

Learn about the fields used to create a Flat file connection with ThoughtSpot DataFlow.

Here is a list of the fields for a Flat file connection in ThoughtSpot DataFlow. You need specific information to establish a seamless and secure connection.

Connection properties

Connection name

Name your connection.

Mandatory field.

Example:

FilesConnection

Connection type

Choose the Files connection type.

Mandatory field.

Example:

Files

File location

Specify the base location of the file on the server.

Mandatory field.

Example:

/home/admin/dataflow/data

Files on remote location

Specify If the files on remote server.

Mandatory field.

Default:

No

Other notes:

Advanced configuration

Protocol

Select the required remote server connection.

Mandatory field.

For remote location files only.

Example:

FTP

Valid Values:

FTP, SFTP

Default:

FTP

Authentication type

Specify the authentication type for SFTP Protocol.

Mandatory field.

For SFTP protocol only

Example:

Password

Valid Values:

Key, Password

Default:

Password

Host

Specify the Hostname or the IP address of the remote server.

Mandatory field.

For remote location files only.

Example:

www.example.com

Port

Specify the Port to connect the remote server.

Mandatory field.

For remote location files only.

Example:

1234

User

Specify the user to connect to remote server. This user must have data access privileges.

Mandatory field.

For remote location files only.

Example:

userdi

Password

Specify the password.

Optional field.

For remote location files only, when using password authentication.

Example:

pswrd234%!

Key file

Specify the key file and its fully qualified path.

Mandatory field.

For remote location files only, when using key authentication.

Example:

/app/key/sftp.key

Passphrase for key file

Specify the passphrase for the key file.

Optional field.

For remote location files only, when using key authentication.

Example:

!hello

Sync properties

File name

Specify name of the file.

Mandatory field.

Example:

CUSTOMER.csv

Valid Values:

Any string literal

Default:

The name of the file

Other notes:

To specify wildcard, use '*' character in the file name

Column delimiter

Specify the column delimiter character.

Mandatory field.

Example:

,

Valid Values:

Any printable ASCII character or decimal value for ASCII character

Default:

The delimiter specified in sync

Enable archive on success

Specify if system is to archive file after successful sync.

Optional field.

Example:

No

Valid Values:

Yes, No

Default:

No

Delete file on success

Specify if system to delete file after successful sync.

Optional field.

Example:

No

Valid Values:

Yes, No

Default:

No

Compression

Specify if file uses compression, and compression type.

Optional field.

Example:

gzip

Valid Values:

None, gzip

Default:

None

Skip header rows

Specify the number of header rows to skip when loading data.

Optional field.

Example:

5

Valid Values:

Any numeric value

Default:

0

Enclosing character

Specify if the text columns in the source data needs to be enclosed in quotes.

Optional field.

Example:

Single Valid Values:

Single, Double, Empty

Default:

Empty

Escape character

Specify the escape character if using a text qualifier in the source data.

Optional field.

Example:

\\

Valid Values:

Any ASCII character

Default:

Empty

Null value

Specifies the string literal indicates the null value for a column. During the data load, the column value matching this string will be loaded as null in the target.

Optional field.

Example:

+ NULL

+ Valid Values:

+ Any string literal

+ Default:

+ NULL

Date style

Specifies how to interpret the date format.

Optional field.

Example:

YMD

Valid Values:

YMD, MDY, DMY, DMONY, MONDY, Y2MD, MDY2, DMY2, DMONY2, and MONDY2

Default:

YMD

Other notes:

MDY: 2-digit month, 2-digit day, 4-digit year

DMY: 2-digit month, 2-digit day, 4-digit year

DMONY: 2-digit day, 3-character month name, 4-digit year

MONDY: 3-character month name, 2-digit day, 4-digit year

Y2MD: 2-digit year, 2-digit month, 2-digit day

MDY2: 2-digit month, 2-digit day, 2-digit year

DMY2: 2-digit day, 2-digit month, 2-digit year

DMONY2: 2-digit day, 3-character month name, 2-digit year

MONDY2: 3-character month name, 2-digit day, 2-digit year

Date delimiter

Specifies the separator used in the date format.

Optional field.

Example:

  • Valid Values:

    Any printable ASCII character

    Default:

    -

Time style

Specifies the format of the time portion in the data.

Optional field.

Example:

24HOUR

Valid Values:

12 Hour, 24 Hour

Default:

24HOUR

Time delimiter

Specifies the character used as separate the time components.

Optional field.

Example:

:

Valid Values:

Any printable ASCII character

Default:

:

Statistics

Specify if the file source count needs to be displayed in Monitor statistics.

Optional field.

Example:

ON

Valid Values:

OFF

Default:

ON

Skip trailer rows

Skip trailer row while processing the data.

Optional field.

Example:

5

Valid Values:

Any numeric value

Default:

0

Reader encoder

Specify the encoding format of the source system.

Optional field.

Example:

UFT-8

Valid Values:

ISO-8859-1, UTF-16, UTF-16BE, WINDOWS-1252

Default:

UFT-8

TS load options

Specifies the parameters passed with the tsload command, in addition to the commands already included by the application. The format for these parameters is:

--<param_1_name> <optional_param_1_value>

--<param_2_name> <optional_param_2_value>

Optional field.

Example:

--max_ignored_rows 0 Valid Values:

--user "dbuser"

--password "$DIWD"

--target_database "ditest"

--target_schema "falcon_schema"

Default:

--max_ignored_rows 0

Boolean representation

Specifies the representation of data in the boolean field.

Optional field.

Example:

true_false

Valid Values:

true_false, T_F, 1_0, T_NULL

Default:

true_false


Related information