Guide / 00 Overview

Overview

The Provision API exports the data your teams capture in Provision as CSV: form submissions and their verification steps, the rows of grids inside them, the deviations raised on them, and your resources (suppliers, customers, products and the rest). Use it to keep a copy in your own warehouse, BI tool or pipeline, and to keep that copy current without downloading everything again.

Base URL and versioning

Every endpoint that returns your data lives under https://api.provision.io/v1. The few open, data-free endpoints in Authentication sit outside that prefix. /v1 is part of the contract: Versioning and deprecation sets out what may and may not change within it.

That host is production, and a key issued for it reads your live data. To build against something else first, ask your Provision contact what is available for your organization. Non-production access is arranged with us, not self-served, and a key only works in the environment it was issued for.

The two endpoints

Those two are everything that returns your records. The rest of these pages are about using them well.

Quickstart

You need an API key; see Authentication. Treat it like a password.

Each entry has a key (use it in URLs) and a columns array describing its columns. Most also have a displayName, but not all, so treat it as optional.

You get CSV with a header row, oldest change first.

3. Keep it current. Every row carries its own paging position in its _cursor column. Send the last row's _cursor value back as cursor, unchanged, and repeat until a page has fewer rows than the limit you sent. Incremental sync covers the loop properly, including the original modifiedSince/after parameters, which still work.

The OpenAPI document

A machine-readable OpenAPI 3 description of both endpoints is served at /swagger/v1/swagger.json. Point a client generator, Postman or your API gateway at it.

The spec cannot describe your columns: those come from your forms, and you read them from GET /v1/datasets at runtime.

Pointing an AI assistant at these docs? Give it /docs.md, which holds every page in one Markdown file, or /llms.txt, a short index of what is where.

Guide / 01 Checklist

Integration checklist

The rules a correct integration follows. Each links to the detail. If you build nothing else from these pages, build these.

Requests

• Authenticate with Authorization: Bearer ak_…. A 401 usually means fix the key; retrying does not help. See Authentication.
• Read GET /v1/datasets on every sync. Store each dataset's key, and alert when a key you sync disappears: the form or grid was renamed. See Datasets.
• Page with the _cursor column. Send the last row's _cursor value back as cursor, unchanged and untruncated. Do not parse it, and do not send it together with modifiedSince/after — that is a 400. See The loop.
• Treat a INVALID_CURSOR 400 as a real error, never as a quiet restart: it is never answered with the first page. Read the detail — it says whether to fix how you store the cursor, send each dataset its own, or re-export that dataset from the first page.
• Stop when a page has fewer rows than the limit you sent — compare against your own limit, never against a number you wrote down. A client whose end-of-data test is len(rows) < 5000 stops early and reports success the day that maximum moves. Keep limit at or below 5000, and read that maximum from these pages rather than hardcoding it. A page may come back with a few rows more than limit; that is not the end of the data and not an error.
• Run one request at a time per dataset, and sync datasets one after another. Concurrency is capped per organization. See Rate limits.

Parsing

• Read columns by header name, never by position, on every page: columns are added as forms change, even mid-backfill. See CSV columns.
• Decode as UTF-8 with a byte-order mark (utf-8-sig in Python).
• Raise your parser's field-size cap. A reference field holds a whole record as JSON, in one field, and can run to hundreds of thousands of characters; Python's csv raises at 131,072 of them unless you call csv.field_size_limit. See Reading the values.
• Parse timestamps with an explicit format. 2026-07-14 08:12:44.518+00 has a space, not a T. The envelope's _local columns have no offset. Booleans are t and f. See Reading the values.
• Map date to a date, dateTime to a timestamp with a time zone, and localDateTime to one without. A localDateTime is already in the site's time zone and carries no offset, so converting it again shifts it. Take the type from dataType, not from the column name: the envelope and verification _local columns and those of deviations are localDateTime, but a form field or custom attribute whose own key happens to end in _local is an ordinary UTC instant. See Column metadata.

Loading

• Treat an interrupted download as a failed page, never as a short one. Retry it from the same cursor, a bounded number of times.
• Upsert, do not append. Key on id, plus row_index on grid datasets, and trim grid rows beyond the highest row_index after each sync. See Upsert, do not append.
• Save the cursor after every page, once that page's rows are committed. Store it as text with no length cap — a truncated cursor is refused rather than silently resumed, but it still stops your sync. On the original modifiedSince parameters, rewind by a few minutes before each run: a row can be stamped just before it becomes visible.
• Re-read datasets in full on a schedule. An incremental sync never shows a deletion, a renamed related record, or a form edit that rewrites old rows' exported text, such as relabelling an option. See Deleted records.

Errors

• Branch on the status code and the errorCode, never on title or detail. Every error body carries an errorCode; the prose beside it may be reworded at any time. See Error responses.
• On a 429, wait Retry-After seconds and repeat the same request. Read RateLimit-Limit to see which limit you hit: the per-minute one means run slower overall, and Retry-After really clears it; on the concurrency one Retry-After is only a polling interval, because a slot frees when one of your own in-flight requests finishes — which for a large export takes minutes — so the fix is fewer requests in flight. See The 429 contract.
• On a 503, read the errorCode. Only DATASET_NOT_EXPORTABLE is not worth retrying. See The three 503s.
• Log the traceId from error bodies, and quote it to support. Only the 429 has none, so read the field defensively.

Guide / 02 Authentication

Authentication

Every request that returns your data needs a Bearer token in the Authorization header. There is no other auth mode and no browser login.

A few endpoints carry no customer data and need no key:

• GET /health: liveness. Returns 200 whenever the service is up.
• GET /health/ready: readiness. Returns 503 while the service's own database dependency is unhealthy. It is a coarse "is the service up" check, not proof that exports will work: it does not probe the datastore exports read from, or any individual dataset. The export endpoint's own 503 is the authority on that.
• GET /build: the deployed build (image tag, commit, build date). Quote it when reporting a problem.
• These documentation pages, and the OpenAPI document.

None of these are in the OpenAPI document, so a generated client will not know about them.

The header

The token is the API key secret issued to your organization. It begins with ak_; a credential that starts with anything else is the wrong one.

A missing, malformed, revoked or expired token gets 401. Retrying changes nothing, so alert on the first one.

It is almost always the key. The exception: a valid, current key that is not tied to an organization we know about also gets 401, and replacing the key will not fix that. If a newly issued key has never worked, tell us rather than rotating it.

That differs from 404 Organization not configured, a later stage: your key is accepted and your organization recognised, but it has not been set up for data export yet. 401 means we could not tell who you are; 404 means we could, and have nothing to export to you yet. Only we can fix the 404.

If we cannot verify your key because our identity provider is unreachable, you get 503 Authentication temporarily unavailable with a Retry-After header, not a 401. Your key is fine: wait, retry, and alert only if it persists.

Two timing details can make a correct key look broken, or a revoked one look live:

• A rejected key stays rejected for a short while (tens of seconds), even if you retry it. The rejection is remembered per secret, so deploying a different, correct key works on its very next request.
• A revoked key can keep working for about a minute. A successful check is cached for 60 seconds, and a check already in flight when we revoke the key can start its own 60 seconds, so the worst case is a minute plus one round trip to our identity provider. Using the key does not extend that. A key is checked once, when a request arrives, so an export accepted just before the cache expires streams its whole page. Revoke a leaked key, but do not treat revocation as instant: if a key is being abused, tell us, and assume the last minute of reads got through, plus any export still streaming.

Keys are scoped to your organization

A key belongs to exactly one Provision organization, and every response is filtered to it. No parameter widens that: you cannot reach another organization's data, and you never pass an organization id.

If your organization has not been enabled for API access, you get 404 Organization not configured. That is a setup step on our side; contact support.

Getting a key

Keys are minted by Provision on request. There is no self-serve key management in the product today. Ask your Provision contact or support for one, and say which integration it is for: that description is stored with the key and is how we identify it later.

You receive the secret once. We cannot recover it; if it is lost, we revoke it and issue a new one.

A key can be issued with an expiry, enforced separately from revocation: when it passes, the key starts returning 401 with nothing changed on your side. Ask what expiry your key has and note the renewal date. We do not warn you as it approaches.

Handling the secret

• Keep it in a secret manager or environment variable. Never commit it, and never ship it to a browser or mobile app: it reads your whole organization's data.
• Ask for one key per integration, so revoking a leaked warehouse key does not also break your BI refresh.
• To rotate, take the new key, deploy it, then ask us to revoke the old one. Both work during the overlap, so there is no downtime.
• Rate limits count per organization, across all your keys. More keys do not buy more throughput; see Rate limits.

Guide / 03 Datasets

Datasets

A dataset is one exportable table. GET /v1/datasets lists every dataset your organization has. Treat that response as the authoritative description of your data and re-read it regularly, rather than hardcoding what you found on day one.

Datasets come in three families:

• Form datasets, built from your published forms. They differ between customers and change when your forms change. A form produces a submission dataset (one row per submission) and a grid dataset for each repeating table on it, at any depth. How your forms become CSV explains them, with a worked example.
• Resource datasets, one per kind of resource: suppliers, customers, products, product categories, equipment, chemicals, employees, areas, lots, recipes and documents. Their keys begin resources_, and their core columns are the same for every customer. All but resources_documents add one column per custom attribute you define. Resources lists every column.
• deviations: one row per value flagged as out of spec on a submission. Its columns are the same for every customer; see Deviations.

All three behave the same way in this API: every dataset has id and last_modified, appears in the same listing, and pages through the same endpoint under the same cursor rules — with the third cursor value a grid dataset needs on the original parameters.

The listing separates resource datasets from the other two: they are the ones with no displayName, and their keys begin resources_. It does not separate a form dataset from deviations — neither has a prefix to go by, and both have a displayName. So match on the keys you expect and ignore the rest, rather than assuming every entry is a form.

The response

columns above is abridged, so do not size a schema from it. A real submission dataset also lists the rest of its envelope (site_id, site_path, status, is_archived, started_at and completed_at with their _local siblings, and last_modified_local) before one entry per form field, and ends with _cursor, which every dataset has and which the export appends after the dataset's own columns. The envelope columns is the full list; GET /v1/datasets is authoritative for your own forms.

Column metadata

Each entry in columns describes one CSV column:

On a form dataset, a column's key comes from the form field's key, which survives wording changes, and its label from the field's label, which changes freely. Build against key; show label. (A dataset's own key is different: it comes from the form name and grid labels, so relabelling a form or grid renames it; see When someone edits the form.)

A label is often composed, not copied. That plain rule holds for a field that produces exactly one column. Every other column gets a label built from the field's label and what the column is: Verification: <label> for a verification column, <label> - Name and <label> - Date for the two halves of a signature, <label> - ID for the id column of a resource select, <label> - <question> for one row of a survey, <label> - <option> for one option of a checkbox group, and <grid label> - Row Index for the ancestor row-index columns of a nested grid. Every form but the verification one joins its parts with - , and a field's own label may contain that too, so do not split a label back into its parts — match on key.

dataType tells you how to parse a value, not the exact database type. Three things to know before you generate a warehouse schema from it:

• A date without a time is date, not dateTime. It has no time and no offset, so it names a calendar day, not an instant. resources_lots has three (expiry_date, date_received, date_produced), and every Date custom attribute adds one. The CSV agrees: a date column is written 2026-07-14, while a dateTime always writes a time, even at midnight (2026-07-14 00:00:00+00). Store date as a date: a timestamp column invents a midnight the source never stated.
• dateTime is an instant in UTC; localDateTime is a wall-clock time at the site. A dateTime (last_modified, started_at, completed_at) names a moment, and the CSV writes it with an offset (2026-07-14 00:00:00+00). A localDateTime has already been converted to the site's time zone and has no offset (2026-07-14 12:00:00): the _local envelope and verification columns, and those of deviations. Store a localDateTime as a local (timezone-less) timestamp; converting it again attaches an offset the source never stated. A form field or custom attribute whose own key ends in _local is still a UTC instant.
• text is also the fallback for anything with no narrower type, so it means "nothing more specific to say", not "free text".

The values above are what /v1 publishes today, and the vocabulary is additive: we may add a value to it, but no existing value changes meaning and none is removed; see Versioning and deprecation. So do not close an enum over them. Treat a dataType you do not recognise as text — that is what the API itself does with a type it has no narrower name for, and it is always a safe reading.

Dataset and column naming rules

Dataset names are generated from form names and grid labels, and column names from field keys: lowercased, spaces and dashes turned into underscores, anything else removed. What follows from that:

• Runs of underscores are collapsed within a name, so a double underscore only ever separates parts: parent__grid for grid datasets, and field__option / field__question for the multi-column field types in How each field type becomes columns.
• Identifiers are capped at 63 characters. A name taken straight from one form or field name is simply truncated. A composed name (anything built with a separator or suffix: parent__grid, field__option, field__question, verification_<field>, the <field>_id, <field>_name and <field>_date companion columns, and resource custom-attribute columns) instead gets an 8-character hash of the full name appended when it overflows (…_3f9ac1b2). The hash eats into the visible part, so a grid dataset's key does not always contain its parent's.
• When two column names in one dataset come out equal, a numeric suffix (_2, _3) keeps both. Which one keeps the plain name follows the order the fields are processed in, which does not change between rebuilds. Two long field names that truncate to the same 63 characters are separated the same way.
• When two dataset names collide, an 8-character hash separates them, so both still export. The hash is of the form's id, plus, for a grid dataset, the keys of the grid and the grids around it; in the rare case the hashed name collides too, a numeric suffix is added. Which one keeps the plain name depends on which was generated first, and stays put after that.

So never derive a long identifier by eye. Read names from GET /v1/datasets and match on what you receive. A dataset with a hash suffix pages and exports like any other.

What happens when a form is deleted

The dataset stays. It remains in GET /v1/datasets, keeps its columns, and keeps exporting what it held when the form was last generated. Nothing is added to it again, and nothing removes it. This is permanent, not a delay before a cleanup.

The listing does not mark such a dataset, so expect to meet one:

• A dataset that stops changing is not necessarily a broken sync. Its last_modified values stop advancing and every incremental pull returns nothing, which looks exactly like a broken sync. A deleted form is at least as likely an explanation.
• Its rows are still real history. Deleting a form does not retract the submissions made against it, so keep them unless you have your own reason not to.

Nothing in the API says whether a dataset is still live. Ask the technical contact on your account; see How you will hear about a change.

Terms these pages use

Four words, one meaning each. A submission dataset is the top of a form's chain: one row per submission, never a grid. A grid dataset comes from a repeating table, at any depth. Parent always means one level up, as parentDataset reports: the submission dataset for a top-level grid, the enclosing grid dataset for a nested one. Form dataset means both kinds; anything said of every form dataset holds for grid datasets too.

Two words, one thing. A filled-in form is a submission on most of these pages, and a record where its human-readable code is concerned (record_id, and what Provision users call it).

Record is looser only where a rule covers every dataset at once: "a deleted record stops being exported" covers a customer or product as much as a submission. Next to record_id or a record code, it always means a submission.

Our parameter descriptions and one error message say "grid/child dataset". That is a grid dataset; "child" is its position under a parent, not a third kind. afterRowIndex follows that family and nothing else: required on a grid dataset you page with after, and rejected on every other dataset. A grid dataset is one with a parentDataset, and it is the only kind with a row_index column — row_index is reserved against form fields and custom attributes everywhere else, so no other dataset can acquire one. You do not have to classify anything to get this right: both mistakes are a 400 whose detail names the fix, and cursor needs none of it.

Guide / 04 Incremental sync

Incremental sync

The CSV endpoint returns about 5000 rows per call, so every export is a loop. The same loop does the first backfill and every later update: there is no separate full-export mode, and running the loop with no cursor is a full export.

Rows come back in modification order, and the cursor is a position in that order. That is what makes the loop both resumable and incremental.

Every row carries its own cursor, in the _cursor column. Send the last row's back as cursor and the next call resumes immediately after it. That is the whole mechanism: you never build a cursor, you echo one back.

The loop cannot tell you a record was deleted: a deletion leaves no row behind. If that matters, run the loop with no cursor on a schedule and reconcile; see Deleted records stop being exported.

Parameters

The loop

1. Call with no cursor. You get the oldest-modified rows first.
2. Take the last row of the response and read its _cursor column.
3. Call again with cursor = that value, sent back exactly as it arrived.
4. Repeat until a response has fewer rows than your limit. That is the end of the data.
5. Later, resume from the cursor you stored. Anything created or edited since comes back.

Compare against the limit you sent, not against a constant. rows < limit is the test; rows < 5000 only looks like it. If the maximum ever changes, the first breaks nothing and the second stops a backfill part-way through and reports success. Read the maximum from this page instead of hardcoding it, and prefer a limit you chose yourself over relying on the default.

Cursor problems are always a 400, never a quiet restart. A cursor that is empty, truncated, altered, taken from a different dataset, or issued before the dataset was rebuilt into a different shape is refused with a message saying which of those it is. It is never answered with the first page — that would look like a successful resume and silently re-ingest the whole dataset. So if you get a INVALID_CURSOR 400, read the detail: it tells you whether to fix your storage or start the dataset again. Omit cursor entirely to start from the beginning; sending an empty one is refused.

The original cursor parameters still work. modifiedSince, after and afterRowIndex page the same data, and a client using them needs no change. They describe a position that several rows can share, which is why a page served for them can run on past your limit to the end of that group. Prefer cursor all the same: it carries the whole position in one value, so a grid dataset needs no third parameter, there is no timestamp for your storage to round, and it will not have to grow a fourth parameter the next time the ordering does.

That end-of-data rule is safe because a limit above 5000 is refused, not reduced. If it were quietly reduced, a client asking for 10000 would get 5000 rows on its first call, read "fewer than I asked for" as the end, and stop partway through its first backfill without any sign of it. Compare against the limit you sent, and keep it at or below 5000.

Grid datasets need a third cursor value — on the original parameters only. Their rows share the last_modified and id of their submission, so those two alone cannot move past a submission's rows. Send afterRowIndex = the last row's row_index as well. Leaving it out is a 400, not a silent loop, but only from the second page, so test that your first pull of a grid gets past its second page. cursor needs none of this: it already carries the row's position, so a grid pages exactly like a submission dataset.

afterRowIndex is part of the cursor, not a parameter of its own. Send it only with modifiedSince and after, and only on a grid dataset. It is at least 1: row_index counts from 1, so afterRowIndex=0 is refused rather than read as "from the start". Once it parses as an integer, four things make it a 400: a value below 1; sending it without modifiedSince and after; sending it on a dataset that is not a grid dataset; and leaving it out when paging one that is. A value that is not an integer at all is refused earlier, before the dataset is looked at; see When more than one thing is wrong.

An interrupted download is a failed page, never a short one. The body is streamed, so a failure after the response has started cannot change the 200 that was already sent. Instead of finishing a partial CSV that looks complete, the export aborts the connection: command-line tools exit non-zero, and HTTP libraries raise their usual interrupted-stream error. That is your only completeness signal: there is no Content-Length and no error marker inside the CSV.

Retry the page from the same cursor, but a bounded number of times. Most aborts are transient. One is not: a single stored value the export cannot render. When that row lies beyond the first 64 KiB of the response, the failure is found too late to send an error, so the connection aborts at the same point on every attempt. If retries keep failing at the same cursor, stop and contact support. It is the same condition the Dataset cannot be exported 503 reports when it is caught before the response starts. Whatever you do, do not swallow the error: a reader that keeps whatever arrived commits a partial page, and if it then advances the cursor, the missing rows are never offered again. A failure caught before the first bytes go out is an ordinary error response instead: one of the three 503s or a 500, each with its own retry rule in Errors, limits and versioning.

A row can arrive just behind your cursor. A submission's last_modified is set a moment before it becomes visible to the export, so a row can appear with a timestamp slightly older than a cursor you already hold, and a strict cursor never offers it. On the original parameters, rewind your stored modifiedSince by a few minutes before each run. A cursor cannot be rewound — it is opaque, and there is nothing in it to move back — so cover this by re-reading the dataset in full on a schedule instead. Rows you have already loaded come back either way, which costs nothing because you upsert them.

The header row can change between pages of a long backfill. Publishing a form rebuilds its dataset, so page 5 can have columns page 1 did not. Read the header of every response, treat unknown columns as new, and treat missing columns as empty.

Skipping datasets that have not changed

The loop above syncs one dataset. If you have many, running it against every one of them on every cycle means one CSV call per dataset just to discover that most of them have nothing new — and if you have a thousand datasets, that is a thousand calls to learn almost nothing.

Two things in GET /v1/datasets let you avoid that. Poll the catalogue first, and use it to decide what to export.

Per dataset, compare changeToken. Store the token alongside the cursor. On the next cycle, a dataset whose token is unchanged has nothing new, so skip its CSV call entirely. The token tells you whether to export; the cursor still tells you where to resume. Keep both.

For the whole catalogue, use the ETag. Every response carries one. Send it back as If-None-Match and you get 304 Not Modified with no body when nothing at all has changed — no dataset moved, and none was added, renamed or reshaped. A cycle that finds nothing then costs exactly one request.

So a cycle is: conditional call to GET /v1/datasets; on 304 nothing in the catalogue moved, so there is nothing to export this cycle; on 200, export only the datasets whose token changed, then store the new tokens and the new ETag. That is the fast path, not the whole job — read the warning below before treating a 304, or an unchanged token, as "nothing to do".

Responses are marked Cache-Control: private, no-cache: the catalogue is specific to your organization, so store the ETag and revalidate rather than serving it from a shared cache.

Worked example

First page:

curl -s "https://api.provision.io/v1/datasets/daily_sanitation_check/csv?limit=1000" \
  -H "Authorization: Bearer $PROVISION_API_KEY"

Suppose the last row of that response is:

id                    , record_id   , last_modified
0191f0c2-...-a41c     , 26-195-0042 , 2026-07-14 08:12:44.518+00

Next page — the last row's _cursor, sent straight back:

curl -s -G "https://api.provision.io/v1/datasets/daily_sanitation_check/csv" \
  --data-urlencode "cursor=pvc1.AQyZ8fUAAAYwtOA3ARnwwkNhQRyZ8f..." \
  --data-urlencode "limit=1000" \
  -H "Authorization: Bearer $PROVISION_API_KEY"

The same page on the original parameters, which still work and need no _cursor:

curl -s -G "https://api.provision.io/v1/datasets/daily_sanitation_check/csv" \
  --data-urlencode "modifiedSince=2026-07-14 08:12:44.518+00" \
  --data-urlencode "after=0191f0c2-...-a41c" \
  --data-urlencode "limit=1000" \
  -H "Authorization: Bearer $PROVISION_API_KEY"

For a grid dataset, that second form also needs the last row's row_index:

--data-urlencode "afterRowIndex=7" \

Upsert, do not append

Rows come back in modification order, so an edited submission comes back again in a later page with its new last_modified. That is what makes incremental sync work, and it means your loader must upsert, not insert:

• Submission datasets: upsert on id, which is unique there whatever the form's shape.
• Grid datasets: upsert on id and row_index (id alone is not unique), then trim the tail as described below.

Grids need the extra step because row_index is a position, not an identity: it is renumbered from 1 on every export. Reorder a grid's rows, or delete one from the middle, and every row below it moves up, so what now arrives as row_index 3 is not what you stored as row_index 3. If a submission loses rows, upserting never overwrites the surplus rows you already stored.

Do not fix that by deleting an id's rows as each page arrives. Pages are cut from one ordered stream regardless of submission boundaries, so a submission's rows can straddle two pages, and delete-then-insert per page would drop the ones from the earlier page. Instead:

1. Upsert each row on (id, row_index) as it arrives.
2. Track the highest row_index seen for each id during the sync.
3. After the sync finishes, delete, for every id you touched, its rows with a higher row_index than that. They are left over from a shorter version of the grid.

To avoid tracking state, buffer rows by id instead, and replace a submission's rows once you know you have all of them: when a row with a different id arrives, or when the sync ends.

One kind of change the cursor never shows. last_modified follows the row's own record, but some columns read from records beside it: the comma-joined list columns on resource datasets, and any column showing a related record's name. Rename a product and every supplier and customer row listing it exports a different products value, while none of their last_modified values move. An incremental sync never offers those rows again, and nothing signals the change. If those columns matter, re-read the affected resource datasets in full on a schedule; they are small next to submission data.

Reference implementation

A complete sync for one dataset, in Python with requests. It saves the cursor after every page, retries only what is worth retrying, and gives up after a bounded number of attempts.

session is a requests.Session whose Authorization header carries your key. load_cursor, save_cursor and upsert are yours. Commit each page's upserts before save_cursor stores the cursor past them. Store the cursor as text, with no length limit that could truncate it: a truncated cursor is refused with a 400 rather than silently resuming somewhere else, but that still stops your sync. len(rows) < PAGE_SIZE is the end-of-data test, and it stays correct even though a page can come back slightly larger than PAGE_SIZE. For grid datasets, trim the tail after sync returns, as Upsert, do not append describes.

Operational advice

• Store the cursor durably, not in memory. It is the only state your integration needs, and it makes a crash mid-backfill cheap.
• One sync at a time per dataset. Concurrency is capped per organization (see Rate limits), and a streaming export holds a database connection while it runs. Fanning out earns 429s and finishes later than running datasets in turn.
• Retrying a page is safe. The cursor only advances once you have the rows, so a retry fetches the same page again rather than skipping one. Bound the retries all the same.
• Use a generous client timeout and a modest limit. A 5000-row page of a wide form is a large response. Prefer a smaller limit to a short timeout.
• Reuse one connection for the whole sync. A page fetched on a fresh connection pays for the TCP and TLS handshake as well as the data, which is on the order of 150 ms of pure latency — comfortably more than a small page costs to serve. Over a backfill of thousands of pages that is most of your wall-clock time. Use a session or connection-pooling client (requests.Session, a shared HttpClient) rather than a new client per call, and keep limit large enough that you are not paying the round trip for a handful of rows.

Guide / 05 Errors & limits

Errors, limits and versioning

Everything that is not a 200: what can go wrong, which failures are worth retrying, how throttling works, and what may change without notice.

Error responses

Errors are application/problem+json with a status code, a title, and an errorCode. Most also have a detail describing the specific case; a 401 has none. A parameter that cannot be parsed at all (a malformed timestamp, a non-numeric limit) is rejected before the checks below, with the same fields plus an errors object naming each parameter at fault. A path that matches no endpoint at all (for example /v1/v1/... from resolving links.csv against the base URL) gets a plain 404 titled Not Found, not Dataset not found.

Branch on the status code and the errorCode. errorCode is a short upper-case token — INVALID_CURSOR, RATE_LIMITED, DATASET_NOT_FOUND, AUTH_UNAVAILABLE, DATABASE_UNAVAILABLE, DATASET_NOT_EXPORTABLE and a handful more — and it is on every error this API returns, including the ones the framework writes for a bad route or method. It is the one field in the body written for a program: within /v1 a code never changes meaning and never disappears, and the full set is published in the OpenAPI document. New codes may be added, so treat one you do not recognise by its status code alone rather than failing on it.

Do not branch on title or detail. Both are prose, written for whoever reads your support ticket, and either may be reworded at any time — a reworded title turns a failure your client used to recognise into one it does not. The same goes for matching a title in a log alert. Where the table below names a title, it is naming the case for you to read, not a string to compare.

The table below covers the two endpoints. One more 503 can answer any request to the API, including these two: 503 Database unavailable (DATABASE_UNAVAILABLE). It is transient, and The three 503s covers all three together.

Errors carry a traceId. Quote it. It identifies that one request in our logs, and a support ticket with one can be answered. Log it with your own failure. One body has none — the 429 — so read the field defensively; without one, quote the time and your organization instead.

Which to retry:

When more than one thing is wrong

Two rules decide which error you get. Neither depends on the order of your query string.

A parameter that cannot be parsed is rejected first, before the dataset is looked at. limit=abc, after=DSC-1041, modifiedSince=yesterday and afterRowIndex=1.5 all fail while the request is read, so you get a 400 naming every parameter at fault, even for a dataset you cannot export or do not have.

After that, a dataset that cannot be served wins over a parameter that is merely wrong. If the dataset is not yours (404), or is already known to be out of date (503), that is the answer, whatever else is wrong: a 400 about your cursor would send you debugging your own code when the dataset needs attention. A 503 found only while reading rows (a value that cannot be exported) comes after the parameter checks, so fix any 400 first.

Beyond those two rules, do not depend on which error you get. Break two rules with parameters that do parse and you hear about one of them, and which one is not part of the contract. Fix it and call again.

The three 503s: tell them apart by errorCode

The data endpoints send a 503 for three unrelated reasons. Two are transient and worth retrying. The third, DATASET_NOT_EXPORTABLE, is the one retrying cannot clear. Only one of the three has Retry-After, so the header alone cannot tell them apart — and neither can the title, which is prose we may reword. Switch on errorCode. (/health/ready has its own transient 503, outside this rule; see Authentication.)

503 AUTH_UNAVAILABLE (Authentication temporarily unavailable), with Retry-After. We could not verify your key because our identity provider did not respond. It is a blip on our side, deliberately not a 401: a 401 would tell you to replace a key that is fine. It affects every request while it lasts, including /v1/datasets. Wait, retry, and alert only if it keeps happening.

503 DATABASE_UNAVAILABLE (Database unavailable), without Retry-After. A database operation behind the request failed: a statement timeout, a dropped connection, the database briefly out of resources. It is transient and worth retrying, as its detail says: "The request could not be completed because a database operation failed. Please try again." No interval is advertised, so back off exponentially yourself, and alert only if it persists across several attempts.

503 DATASET_NOT_EXPORTABLE (Dataset cannot be exported), without Retry-After. The dataset cannot be exported as it stands, and repeating the request changes nothing, which is why no Retry-After is sent. It has two causes:

• The dataset is out of date or unreadable on our side. It predates a change to what exports require and has not been rebuilt since, or it cannot be read for a structural reason.
• One stored value cannot be exported. One cell in one row defeats its column. The rest of the dataset is fine, but the export stops at that row. You see this 503 only when that row comes early enough to still send an error; further into a large export, the download aborts instead, as The loop describes.

What clears it depends on the dataset. A form dataset is stored as a pre-built copy that a background job keeps up to date as submissions and the form change. A rebuild replaces the whole dataset or leaves the previous one untouched, so a rebuild in progress never causes this 503, and republishing the form can clear the first cause. A resource dataset and deviations are read live from your data at the moment you export them, so there is nothing to republish for those — but a stored value they cannot render shows up here rather than in a failed rebuild, which is the second cause. For either family, correcting the offending value in Provision clears it; anything else needs a fix from us. Treat it as an alert, not a backoff: contact support, naming the dataset from the detail.

The response only ever names one dataset, but the cause can be wider than one. Keep syncing the others rather than stopping, and then check them: a configuration problem that stops one form dataset being rebuilt usually stops the rest of that organization's too, and they will go stale without returning an error at all. Record and deviation codes has the case we meet most, a site whose time zone is blank.

The errorCode matters most for the last two: both are 503s without Retry-After, so a client that stops retrying on "503 without Retry-After" gives up on a database blip that would have cleared.

In all three cases your cursor is unaffected and nothing is lost: resume from where you were.

Rate limits

Limits apply per organization, across every key you hold, so more keys do not raise them. There are two:

• a concurrency cap on requests in flight at once. You will meet this one first: a streaming export holds its connection until it finishes, so a handful of parallel pulls reaches it.
• a requests-per-minute backstop, high enough that a well-behaved sequential client never gets near it.

Current values:

The limits are the same for every customer: counting is per organization, but we cannot raise the limits for one customer. They are current operating values, not guarantees, so do not hardcode them.

You only see them on a 429. Successful responses carry no rate-limit headers, so there is no way to check your remaining budget in advance: design for the 429.

A 429 from the concurrency cap does not mean a slot frees in Retry-After seconds. There, the Retry-After and RateLimit-Reset values in the table below are a short polling interval rather than a real reset; a slot frees only when one of your organization's in-flight requests finishes, which for a large export can take minutes. Repeated 429s usually mean your own parallel requests are still running, so reduce them rather than retrying harder.

There is also a limit per source IP address in front of the service, far above the organization limits. It counts all traffic from one IP, so an integration that syncs many organizations from one address shares it between them.

The 429 contract

A 429 response carries these headers; successful responses do not:

The three RateLimit-* headers appear only on a 429. Retry-After also appears on the Authentication temporarily unavailable 503, with the same meaning: wait that long, then retry. The other two 503s — Database unavailable and Dataset cannot be exported — never send it, so its absence does not tell you whether retrying is worthwhile (see The three 503s).

RateLimit-Limit is what tells the two limits apart, and they need different responses. Both are 429s with the same errorCode and the same headers; only the value differs. Compare it with the two values in Rate limits:

• The concurrency cap. You have too many requests open at once. A slot frees only when one of your own in-flight requests finishes, and an export holds its slot for as long as it takes to stream — seconds to minutes. So Retry-After is a polling interval here, not a reset: waiting it and retrying is right, but expect to poll, and if it keeps happening the fix is fewer requests in flight, not more patience. Retrying harder cannot help: you are competing with yourself.
• The per-minute budget. You are sending too many requests over time, not too many at once. Waiting Retry-After clears this one properly, and the fix is to slow the whole sync down — a longer gap between requests, or a larger limit so the same rows cost fewer calls.

A well-behaved client:

1. Sleeps for Retry-After seconds, and does not retry sooner.
2. Resumes from the same cursor. Nothing was consumed, so nothing is lost.
3. Keeps one request in flight per dataset, and syncs datasets one after another.

Paging is resumable, so throttling costs you time and nothing else: no partial state to reconcile, no backfill to restart.

Versioning and deprecation

The /v1 in every data path is a compatibility promise about the API surface: the endpoints, their query parameters, the JSON field names and values /v1/datasets returns, the status codes, and the CSV dialect. It is not a promise about your data: your CSV columns follow the forms and resources your organization configures, and move when you change them.

Within /v1 we may:

• add endpoints, and add fields to JSON responses;
• add optional query parameters that default to today's behaviour;
• add a value to the dataType vocabulary, when a column gains a type we can describe more precisely than the one it reports today;
• add datasets to the listing. Deleting a form does not remove its dataset; see What happens when a form is deleted. A key can still leave the listing one way: renaming a form or grid moves its dataset to a new key and retires the old one. That follows your own configuration, not ours, and it is the usual reason a key you sync starts returning 404; see When someone edits the form;
• add CSV columns. Those follow your forms, not our API: a column appears when someone adds a field. Removing a field does not remove its column, and relabelling a field does not rename it, though retyping one changes which columns it exports; see When someone edits the form;
• retune the rate limits, which are published as current values, not guarantees.

Within /v1 we will not:

• remove or rename an endpoint, a query parameter or a JSON field;
• change the meaning or type of an existing JSON field;
• change the meaning of a dataType value, or remove one. The vocabulary in Column metadata only ever grows, and anything without a narrower type is text;
• change the CSV dialect: delimiter, quoting, encoding and byte-order mark, line endings, timestamp, date or boolean rendering, or null versus empty string;
• change the ordering or the cursor semantics Incremental sync depends on;
• change which organization a key belongs to, or what it can reach.

localDateTime joined the dataType vocabulary inside /v1, and another value may follow the same way: appended, never replacing one. Do not treat the list as exhaustive — a generated client with a closed enum over it breaks on the next addition. Read a dataType you do not recognise as text; see Column metadata.

Write your client against the first list. Ignore JSON fields you do not recognise, read CSV columns by header name, and expect the column set to grow. A client that does those three things survives everything we have reserved the right to do.

What counts as a breaking change

The lists above are about names and shapes. A third kind of change breaks integrations without touching either, and we have made one before.

A value can change format inside a column whose name and type stay put. Record and deviation codes were renumbered part-way through the life of the data. The columns are still record_id and code, reported and exported the same way, but rows from before and after the change have different formats, old rows were never rewritten, and nothing marks which is which. A parser that learned the old format on its first sync met the new one without warning. Record and deviation codes documents both formats and what to key on instead.

No signature moves when that happens, which makes it the easiest kind to get wrong, so we count it as breaking. In full, a change is breaking if it:

• removes or renames anything in the will not list above;
• changes the format of a value in an existing column or field, even with the type unchanged: an identifier layout, a code scheme, a timestamp rendering;
• changes what an existing column means while its name and values look the same;
• tightens validation, so a request that used to succeed now gets a 400;
• changes which rows an export returns for a cursor you already hold.

A change that only adds is not breaking: a column because someone edited a form, an optional query parameter, a JSON field, a dataset in the listing. Those follow your own configuration, as When someone edits the form sets out. The reverse does not happen: a dataset is never withdrawn when its form is deleted; see What happens when a form is deleted.

How you will hear about a change

Today, from us directly, or not at all. The API sends no deprecation signal of its own: no Deprecation or Sunset headers, no changelog endpoint, no version field in any response.

Two things are self-service, and both are narrower than they look:

• GET /v1/datasets, compared with what you saw last time. The useful one. It shows a new dataset, a new column, a changed dataType, and a key disappearing because a form or grid was renamed: the changes most likely to affect you. A column disappears only in the three cases in When someone edits the form: a retyped field, an emptied grid or survey, and a nested grid's <outer grid>_row_index renamed with its outer grid. It does not show a dataset going out of service: a deleted form's dataset stays listed (What happens when a form is deleted). Worth doing on every sync.
• The OpenAPI document, compared the same way. It shows endpoint, parameter and status-code changes. It cannot show a change to the CSV dialect, the cursor rules, or what a value means, because it does not describe them.

Neither tells you that a value's format changed, the case in What counts as a breaking change, and nothing else will either. For that, the technical contact on your account is the channel, so keep that contact current. If you need more than an email before you go live, say so early.

Retiring a version

A breaking change ships under a new prefix, /v2, served alongside /v1. /v1 keeps its behaviour for the whole overlap: no v2 change is backported into it, and it is not retired without telling you first. Moving becomes a path change you schedule, not an outage you discover.

In short: assume nothing about your columns, and rely freely on the endpoint shapes.