DataFlow Tips

Truncate table

Use the --empty_target tsload option to truncate the table, and remove delete from the target query.

In "tsload options" sync parameter, the default is --empty_target false. You can change this by explicitly adding --empty_target true.

Include Epoch date and timestamp

Epoch date and time options in the TQL expression editor let you configure pre-sync and post-sync scripts with greater control.

$$CurrentTimestampEpochAdd

Example of pre-sync script:

delete from "DataFlow_BD"."falcon_defaut_schema"."test_tql" where C_DATE > $$CurrentDateEpochAdd(1)

$$CurrentDateEpochAdd

Example of post-sync script:

delete from "DataFlow_BD"."falcon_defaut_schema"."test_tql" where C_TIMESTAMP > $$CurrentTimestampEpochAdd(1)

Track last sync start time

When setting up syncing, the Advance setup column mapping between external data source and ThoughtSpot offers a new system parameter, $$LastSyncStartTime. Use it in the Add new formula interface to manage sync updates.

$$LastSyncStartTime

Example:

"LAST_ACTIVITY_DATE"= $$LastSyncStartTime

After you apply the new formula, it appears in the Map columns listing.

Specify tsload datetime format

To avoid datetime conversion errors when loading csv files through Dataflow, specify the datetime format in the sync connector properties window. Use tsload options to override default settings in the UI, for example the default "%H:%M:%S" setting for time format.

To specify your datetime format within DataFlow, follow these steps:

  1. Open the Advanced setup interface by clicking the toggle.

  2. Choose the Sync properties tab.

  3. Scroll to Sync connector properties and click the toggle to expand.

    For this example, our data is in the format "%d/%m/%Y %H:%M", or "13/05/2021 10:30", with no data for the seconds value.

  4. Specify the datetime format as follows:

    Date style

    "DMY"

    Date delimiter

    "/"

    Time style:

    "24HOUR"

    Time delimiter

    ":"

  5. To override the default datetime format, "%H:%M:%S", enter --date_time_format '%d/%m/%Y %H:%M' under tsload options.

  6. Click Save. Alternatively, click Save and sync now to save your work and sync data at the same time.

Was this page helpful?
×