Sync data from an API

Step-by-step guide on how to continuously sync data from an API (using a sync template).

Pre-requisite: creation of an integration and at least one connection.

Activate a sync template

Terapi uses syncs to read data from APIs continuously. For common use cases, templates are available to let you get started fast.

Select your integration in the Integrations tab, and navigate to the Endpoints tab. Available sync templates will appear in the endpoint list. Select the relevant one and enable it with the toggle.

Terapi will automatically sync the corresponding records in the background for each relevant connection.

Is there no template for your API? Or none matching your exact use case?

Learn more about how to build a custom integration, extend a template or request custom integrations from Terapi experts.

Listen for webhooks from Terapi

Terapi sends webhook notifications to your backend whenever new data is available for a connection & sync combination.

To set this up, go to the Environment Settings tab and configure a Webhook URL to which Terapi will send notifications.

The webhook from Terapi is a POST request with the following body:

,
    "syncType": "INITIAL" | "INCREMENTAL",
    "modifiedAfter": ""
}

Webhooks with non-2xx responses are retried with exponential backoff.

Before using webhooks in production, verify their origin.

Fetch the latest data

After receiving a Terapi webhook, fetch the latest records using the backend SDK or API:

curl -G https://api.terapi.dev/records \
  --header 'Authorization: Bearer ' \
  --header 'Provider-Config-Key: ' \
  --header 'Connection-Id: ' \
  --data-urlencode 'model=' \
  --data-urlencode 'modified_after=' \

import   from '@terapihq/node';

const terapi = new Terapi();

const result = await terapi.listRecords();

This returns an array of records conforming to the specified data model.

Each record contains useful metadata automatically generated by Terapi:

            },
            ...
        ],
    next_cursor: "Y3JlYXRlZF9hdF4yMDIzLTExLTE3VDExOjQ3OjE0LjQ0NyswMjowMHxpZF4xYTE2MTYwMS0yMzk5LTQ4MzYtYWFiMi1mNjk1ZWI2YTZhYzI"
}

Cursor-based synchronization

In practice, webhook notifications can be missed, and relying solely on the webhook payload to fetch modified records can cause you to miss some updates.

A more reliable way of keeping track of how far you've synced records (for each connection & sync combination) is to rely on record cursors.

Each record comes with a synchronization cursor in _terapi_metadata.cursor. Terapi uses cursors internally to keep a chronological list of record modifications.

Each time you fetch records, you should store the cursor of the last record you fetched to remember how far you've synced (for each connection & sync combination).

The next time you fetch records, pass in the cursor of the last-fetched record to only receive records modified after that record:

curl -G https://api.terapi.dev/records \
  --header 'Authorization: Bearer ' \
  --header 'Provider-Config-Key: ' \
  --header 'Connection-Id: ' \
  --data-urlencode 'model=' \
  --data-urlencode 'cursor=' \

import   from '@terapihq/node';

const terapi = new Terapi();

const result = await terapi.listRecords();

So, the overall logic for cursor-based synchronization should be:

1. Receive a webhook notification from Terapi.

2. Query your database for the cursor of the last-fetched record.

3. Fetch the modified records (passing the cursor).

4. Store the modified records.

5. Store the last-fetched record cursor.

Write back to APIs (2-way syncing)

Write back to APIs with actions or proxy requests.

Troubleshoot errors & monitor

Navigate to the Logs tab to inspect potential errors & monitor sync executions.

Questions, problems, feedback? Please reach out in the Slack community.

This document has been revised according to your instructions. Let me know if further changes are required.