DataFlow tips

Include Epoch date and timestamp

Epoch date and time options in the TQL expression editor let you configure pre-sync and post-sync scripts with greater control.

$$CurrentTimestampEpochAdd

Example of pre-sync script:

delete from "DataFlow_BD"."falcon_defaut_schema"."test_tql" where C_DATE > $$CurrentDateEpochAdd(1)

$$CurrentDateEpochAdd

Example of post-sync script:

delete from "DataFlow_BD"."falcon_defaut_schema"."test_tql" where C_TIMESTAMP > $$CurrentTimestampEpochAdd(1)

Track last sync start time

When setting up syncing, the Advance setup column mapping between external data source and ThoughtSpot offers a new system parameter, $$LastSyncStartTime. Use it in the Add new formula interface to manage sync updates.

$$LastSyncStartTime

Example:

"LAST_ACTIVITY_DATE"= $$LastSyncStartTime

After you apply the new formula, it appears in the Map columns listing.

Specify tsload datetime format

To avoid datetime conversion errors when loading csv files through Dataflow, specify the datetime format in the sync connector properties window. Use tsload options to override default settings in the UI, for example the default "%H:%M:%S" setting for time format.

To specify your datetime format within DataFlow, follow these steps:

  1. Open the Advanced setup interface by selecting the toggle.

  2. Choose the Sync properties tab.

  3. Scroll to Sync connector properties and select the toggle to expand.

    For this example, our data is in the format "%d/%m/%Y %H:%M", or "13/05/2021 10:30", with no data for the seconds value.

  4. Specify the datetime format as follows:

    Date style

    "DMY"

    Date delimiter

    "/"

    Time style:

    "24HOUR"

    Time delimiter

    ":"

  5. To override the default datetime format, "%H:%M:%S", enter --date_time_format '%d/%m/%Y %H:%M' under tsload options.

  6. Select Save. Alternatively, select Save and sync now to save your work and sync data at the same time.

Was this page helpful?
×