Overview

The Provision External API is a read-only export API. It lets your systems pull the data your teams capture in Provision — form submissions, their verification steps, the rows of any grids inside them, and your resource master data — as CSV, on your own schedule.

It is built for one job: keeping a copy of your Provision data in your own warehouse, BI tool, or ETL pipeline, and keeping it current without re-downloading everything each time.

The one idea to take away: most of your datasets are your forms, and their CSV columns are that form's fields. We do not publish a fixed schema, because there isn't one — your forms *are* the schema. Alongside them sit a set of standing datasets for resource master data: a few have the same columns for everyone, and most also carry your own custom attributes. So apart from a handful of fixed ones, no dataset here has a shape we can print in advance — read the current columns from the API rather than assuming them, and you will be right either way.

Base URL and versioning

Every endpoint that returns your data lives under https://api.provision.io/v1. (The few open, data-free endpoints in Authentication sit outside that prefix.) The /v1 prefix is part of the contract — see Errors, limits and versioning for exactly what we may and may not change without a new version.

That host is production, and a key issued for it reads your live data. If you would rather build and test against something else first, ask your Provision contact what is available for your organization — non-production access is arranged with us rather than self-served, and a key is only ever valid for the environment it was issued for.

The two endpoints

That is the entire data surface — everything that returns your records. There are a few open, data-free endpoints besides these (see Authentication); the rest of this page is about using these two well.

Quickstart

You need an API key — see Authentication. It is a secret; treat it like a password.

1. See what you can pull.

export PROVISION_API_KEY="ak_your_key_here"

curl -s "https://api.provision.io/v1/datasets" \
  -H "Authorization: Bearer $PROVISION_API_KEY"

Each entry has a key (use it in URLs) and a properties array describing the columns you will get. Most also have a human displayName, but not all — treat it as optional.

2. Pull the first page of one.

curl -s "https://api.provision.io/v1/datasets/daily_sanitation_check/csv?limit=500" \
  -H "Authorization: Bearer $PROVISION_API_KEY"

You get CSV with a header row, oldest-modified first.

3. Keep it current. Take the last row's last_modified and id, send them back as modifiedSince and after, repeat. That is the whole sync loop — Incremental sync covers it properly, including the extra parameter grid datasets need.

The OpenAPI document

A machine-readable OpenAPI 3 description of both endpoints is served at /swagger/v1/swagger.json. Point a client generator, Postman, or your API gateway at it.

The two are for different jobs, and the spec is the narrower one:

The spec describes the HTTP surface. It cannot describe your columns — those come from your forms, and you read them from GET /v1/datasets at runtime.

Authentication

Every request that returns your data needs a Bearer token in the Authorization header. There is no other auth mode and no browser login flow.

A few endpoints are deliberately open because they carry no customer data. Everything under /v1/datasets requires a key; these do not:

• GET /health — liveness. Returns 200 whenever the service is up.
• GET /health/ready — readiness. Returns 503 while the service's own database dependency is unhealthy. Useful as a coarse "is the service up" monitor, but it does not prove exports will succeed: it does not probe the datastore the exports read from, and it says nothing about any individual dataset. The export endpoint's own 503 is the authority on whether a dataset can be exported.
• GET /build — the deployed build (image tag, commit, build date). Useful to quote when reporting a problem.
• This documentation page, and the OpenAPI document.

None of these appear in the OpenAPI document, so a generated client will not know about them.

The header

Authorization: Bearer ak_xxxxxxxxxxxxxxxxxxxxxxxx

The token is the API key secret issued to your organization, and it begins with ak_. If the credential you were handed starts with anything else, it is the wrong one — check with us before building against it.

A missing, malformed, revoked, or expired token is rejected with 401. Retrying a 401 changes nothing, so alert on the first one rather than backing off.

Almost always that means the credential: fix the key. One case is not, and it is worth recognising because replacing the key will not clear it — a valid, current key still gets a 401 if the organization it belongs to has not been linked on our side yet. That is a provisioning gap for us to close, not something you can fix. If a key you have just been issued has never worked, tell us rather than rotating it.

If we cannot verify your key because our identity provider is unreachable, that is not a 401: you get a 503 titled *Authentication temporarily unavailable* carrying a Retry-After header. Nothing is wrong with your credential — honour the header and retry, and only alert if it persists.

Two timing details, because they make a correct key look broken and a revoked one look live:

• Retrying the same rejected key will not clear it immediately. We remember a rejection for a short period (tens of seconds) so a misconfigured client cannot hammer our identity provider, so hammering a bad secret gets you the same 401 regardless. Deploying a *different*, correct key takes effect on its very next request — the rejection is remembered per secret, not per organization, so a fix is never held up by it.
• Revocation is not immediate. A successful verification is cached for a few minutes per replica, so a revoked key can keep returning 200 and reading your data until those caches expire. Revocation is the right move for a leaked key, but it is not instant containment — tell us if you need a key killed urgently.

Keys are scoped to your organization

A key resolves to exactly one Provision organization and every response is filtered to it. No parameter widens that scope — you cannot reach another organization's data, and you never pass your organization id anywhere.

If your organization has not been enabled for API access yet you will get 404 *Organization not configured*. That is a setup step on our side, not something a different request can fix — contact support.

Getting a key

Keys are minted by Provision on request. There is no self-serve key management in the product today. Ask your Provision contact or support for a key, and say what integration it is for — that description is stored with the key and is how we identify it later.

You receive the secret once, at creation. We cannot recover it afterwards; if it is lost we revoke it and issue a new one.

A key can also be issued with an expiry, which is enforced independently of revocation — an expired key starts returning 401 on its own, with nothing having changed on your side. Ask what expiry your key carries when you receive it, and diarise the renewal; we do not currently warn you as it approaches.

Handling the secret

• Keep it in a secret manager or environment variable. Never commit it, and never ship it to a browser or mobile app — it grants read access to your whole organization's export data.
• Ask for one key per integration, not one shared key. Revoking a leaked warehouse key should not also break your BI refresh.
• Rotation is revoke-and-reissue: take the new key, deploy it, then ask us to revoke the old one. Both work during the overlap, so there is no downtime.
• Rate limits are counted per organization, aggregated across all your keys. More keys do not buy more throughput — see Errors, limits and versioning.

Datasets

A dataset is one exportable table. GET /v1/datasets returns every dataset your organization has. Treat that response as the authoritative description of your data and re-read it periodically, rather than hardcoding what you found on day one.

Datasets come in two families, and it is worth knowing which you are looking at:

• Form datasets, derived from your published forms. These differ between customers and change when your forms change.
• Resource datasets, covering your master data rather than your forms. Their keys begin resources_ — suppliers, customers, products, product categories, equipment, chemicals, employees, areas, lots, recipes and documents — plus a deviations dataset. Their core columns are the same for every customer, but they are not fixed: each one also carries one extra column per custom attribute your organization has defined for that entity, so the exact column list is still yours alone.

Resource datasets have their own shape. They behave identically as far as *this API* is concerned: they have id and last_modified, they appear in the same listing, and they export and page through the same endpoint with the same cursor. Beyond those two columns, assume nothing from the CSV columns section carries over. That section describes how a form's fields and its submission envelope become columns — including the verification columns and the UTC/*_local timestamp pairing — and none of it is a rule for resource datasets. Which columns each one actually carries differs by entity, so read them from GET /v1/datasets.

Watch the names in particular. Some resource datasets have columns *called* status, record_id or site_name — those are the entity's own fields, meaning whatever they mean for that entity, and are not the submission columns of the same name. For any resource dataset, read its columns from GET /v1/datasets and take them at face value rather than mapping them onto form semantics.

Nothing in the listing labels which family a dataset belongs to. The resources_ prefix is a good hint but not a rule — deviations is one of this family and carries no prefix. So the honest rule is the same as everywhere else on this page: match on the keys you expect and ignore the rest, rather than assuming every entry is a form.

How forms become datasets

• Each published form becomes one dataset — the *parent*. One row per filled-in recording step of a submission, which for most forms means one row per submission.
• Each grid (repeating table) inside that form becomes its own *child* dataset, named parent__grid. One row per grid row.

Grids get their own dataset because a repeating section cannot be flattened into a single CSV row. If a form has a grid, expect two datasets and join them yourself.

Child rows carry the parent submission's id, so id is your join key — and it is not unique within a child dataset. Children also carry row_index, which distinguishes rows within one submission and is required when paging them.

A submission also only appears once its recording step has been saved. A submission that has been created but never filled in contributes no row at all — not a row with empty fields — so your row count will not match the record count your staff see in the app, and Pending is a status you will rarely if ever encounter in an export.

Two limits to be aware of:

• A form with more than one recording step emits more than one row per submission. Most forms have a single recording step, and for those id is unique in the parent dataset. If yours has two, you get one row per step, each carrying only its own step's field values with the other step's columns empty — and id repeats. That breaks two things this page otherwise tells you to rely on: upserting on id alone keeps only whichever row you loaded last, and because parent datasets have no row_index to break the tie, a page boundary falling between two rows of the same submission drops the second one for good. If any of your forms has multiple recording steps, tell us before you build against its dataset — we are fixing this, and we would rather size the work knowing you are affected.
• A grid nested inside another grid becomes its own dataset too. Nesting is followed to any depth, and each level's dataset hangs off the one directly above it — so a grid inside a grid has the *outer grid's* dataset as its parentDataset, not the form's. Read the hierarchy from parentDataset rather than from the key.

The response

[
  {
    "key": "daily_sanitation_check",
    "displayName": "Daily Sanitation Check",
    "childDatasets": ["daily_sanitation_check__equipment_checks"],
    "properties": [
      { "key": "id",            "dataType": "uuid",     "displayName": "ID" },
      { "key": "record_id",     "dataType": "text",     "displayName": "Record ID" },
      { "key": "site_name",     "dataType": "text",     "displayName": "Site" },
      { "key": "last_modified", "dataType": "dateTime", "displayName": "Last Modified (UTC)" },
      { "key": "water_temp_c",  "dataType": "number",   "displayName": "Water Temperature (C)" }
    ],
    "links": { "csv": "/v1/datasets/daily_sanitation_check/csv" }
  }
]

Null fields are omitted, not sent as null. The example above is a parent dataset, so it has no parentDataset key at all. Test with "parentDataset" in obj, not obj.parentDataset === null, and do not deserialize into a schema that requires the property to be present.

Column metadata

Each entry in properties describes one CSV column:

The key/displayName split is the important part. key derives from the form field's key, which survives wording changes. displayName is the field's label, which changes freely. Build against key; show displayName to humans.

dataType is deliberately coarse — it tells you how to parse, not the exact database type.

Dataset and column naming rules

Names are generated from your form and field names: lowercased, spaces and dashes turned into underscores, anything else removed. Consequences worth knowing:

• Runs of underscores are collapsed within a name, so a double underscore only ever appears as a structural separator: parent__grid for grid datasets, and field__option / field__question for the multi-column field types described in CSV columns.
• Identifiers are capped at 63 characters. A name taken straight from one form or field name is plainly truncated to fit, with no hash. A composed name — parent__grid, field__option, verification_<field>, and the resource custom-attribute columns — instead gets an 8-character hash of the full name appended when the composition overflows (…_3f9ac1b2). The hash eats into the visible part, so a grid dataset's key does not always still contain its parent's — read the hierarchy from parentDataset, never by parsing the key.
• Where two column names within one dataset end up equal, a numeric suffix (_2, _3) separates them, so both survive. Which of the two gets the suffix follows processing order rather than anything stable, so a rebuild can swap which one is x and which is x_2. Two over-long field names that truncate to the same 63 characters are separated this way too.
• Dataset names that collide are separated by an 8-character hash of the form's id rather than by a numeric suffix, so two forms whose names reduce to the same identifier still produce two datasets and both export. Which of the two keeps the plain name depends on which form is generated first, and stays put after that because the assignment is recorded.

The practical rule is the same in every case: do not derive a long identifier by eye. Read the names from GET /v1/datasets and match on what you actually receive — a dataset whose name carries a hash suffix is still perfectly ordinary to page and export.

Because of truncation and collision suffixes, a column name is not always predictable from the field name by eye. That is exactly why GET /v1/datasets exists — read the names, do not derive them.

CSV columns

Your CSV is your form, flattened.

In a form dataset every column is either part of a small fixed envelope describing the submission, or it comes from a field your organization put on the form. That is the whole model. This section covers which columns are which, how each kind of field turns into columns, how to read the values, and what happens when someone edits the form.

It describes form datasets only. Resource datasets carry the master-data fields of the entity instead, plus your custom attributes, and none of the submission envelope below applies to them — read their columns from GET /v1/datasets.

Read the header row, never the position

The export emits every column of a dataset in the order that dataset happens to define, so column order is not part of the contract. Adding a field to your form can move existing columns.

Parse by header name. A positional or fixed-width parser will break the first time someone edits the form — and it will break by shifting values into the wrong columns, which is far worse than failing outright.

The envelope columns

These are not form fields — they come from the submission itself, and every parent dataset has them:

Every timestamp appears twice: once in UTC and once as *_local, converted to the site's own time zone. Compute on the UTC column; use the local one only for display — *_local values carry no offset, so they are ambiguous across daylight-saving transitions.

Three things about this envelope are worth knowing before you build on it.

status is one of five values: Pending, InProgress, Completed, Cancelled, Rejected. Note what is *not* in that list: Archived never appears in status. Archiving is tracked separately, so status carries the status underneath an archived submission — usually Completed — and the is_archived boolean is what tells you it was archived. The app shows such a record as *Archived* on screen, so read the two columns together if you want to match what your staff see.

site_name is a name, not a key — join on site_id. Site names change, and a join on the name breaks silently when one does: your history keeps the old spelling while new rows arrive under the new one. site_id is stable for the life of the site, so key your site dimension on it and treat site_name as a label to display.

site_path gives you the hierarchy without a second lookup. It lists the row's site preceded by each of its ancestors, dot-separated and root first — so a row recorded at a site two levels down carries all three ids. That makes "everything at or beneath this site" a prefix or substring match on one column: filter on an ancestor's site_id and you get that site plus every site under it, without having to enumerate the descendants yourself. The last segment is always the row's own site_id.

Deleted records stop being exported — silently. This holds for *every* dataset, not just form datasets: deleting a submission removes it from its form dataset, its grid datasets and deviations alike, and deleting a customer, supplier, product, employee or any other resource removes it from that resource dataset. In every case the row simply stops appearing in later pages, and nothing tells you it went.

That has one consequence worth designing for. Because you only ever receive rows that still exist, a deletion is invisible to the cursor — so a record you already loaded stays in your warehouse forever unless you go looking. If deletions matter to your reporting, re-read the dataset in full on a schedule (the same loop, started from no cursor) and treat ids you no longer receive as deleted. Incremental pulls alone cannot tell you, and this applies with particular force to resource datasets, because live submissions keep referencing resources Provision no longer has.

If the form has a verification step with fields on it, these columns appear as well, and each verification field also becomes a column prefixed verification_:

The trigger is the verification *fields*, not the step: a verification step that contributes no fields produces none of these columns. Forms without a verification step have none of them either.

Only one verification is reflected per row, and an in-progress verification takes precedence over a completed one — a step that has been saved but not finished will show with an empty completion timestamp and the in-progress editor's name, hiding a verification that did complete. If you report on verification, treat a row with a verifier but no completion timestamp as *not yet verified* rather than as missing data.

Child (grid) datasets carry a smaller envelope:

A grid nested inside another grid is a dataset in its own right. Its own row_index counts every one of its rows within the submission in a single unbroken sequence rather than restarting inside each parent row, so id plus row_index stays unique and it pages exactly like every other dataset.

It also carries one extra column per grid above it — <outer grid>_row_index — holding that grid's position inside *its* parent. For a grid sitting directly inside a top-level grid, that is the same number the outer grid's dataset publishes as its row_index, so you can reattach a nested row to the row it was captured under by joining on the same id and <outer grid>_row_index = outer row_index. That join holds even when some outer rows have an empty nested grid.

Do not use it to join more than one level up. In a chain three or more grids deep, the columns for the grids above the immediate one are still positions-within-parent, while those grids publish row_index as a sequence over the whole submission — the two do not correspond, and a join on them attaches rows to the wrong parent without failing. If you need to reassemble a hierarchy that deep, tell us: reconstructing it needs a key we do not publish yet.

A parent row whose nested grid is empty contributes no rows to the nested dataset at all — the same way a submission with an empty grid contributes none to that grid's dataset. Do not read a missing nested row as a missing parent row.

How each field type becomes columns

Most fields become exactly one column. These are the ones that do not — and they are where integrations usually go wrong, so check your form for them before assuming a one-to-one mapping.

The Cell value column above describes what the value *means*. How it is written in the CSV is a separate question, answered by Reading the values — in particular, a boolean arrives as t or f, never as true or false.

In those patterns <field>, <option> and <question> are the slugified key of the field, the option, and the survey question.

Two things worth calling out:

• Choice fields store a value but export the label. For single-select and survey fields the option's display label is substituted for the stored value, so the CSV reads the way your staff see it on screen. Exactly one label is exported per stored value: options from every published version are merged on the stored value, and the newest published label for that value wins, applied from the next rebuild onwards. An option that only older versions had keeps its entry, so historic rows still read correctly and deleting an option does not stop rows that already hold it from resolving. Only a value that was never an option in any published version — imported or migrated data, typically — passes through unchanged. Note the consequence for your warehouse: because the label is what lands in the CSV, renaming an option changes the exported text for historical rows too, so key on something else if you need a value that never moves.
• Multi-select becomes one boolean column per option. Adding an option to a checkbox group adds a column.

Reading the values

Every successful export also carries two headers worth knowing about: Content-Type: text/csv and Content-Disposition: attachment; filename="<dataset>.csv". The attachment disposition means anything browser-like will treat the response as a download rather than an inline body. The body is streamed, so there is no Content-Length to size a buffer from or to verify completeness against.

The CSV is produced by PostgreSQL's own CSV writer, which pins the details down precisely:

Four of these bite people regularly:

1. There is a byte-order mark. Decode as utf-8-sig (Python) or strip the leading EF BB BF yourself. If you decode as plain UTF-8, your first header will come back as \ufeffid rather than id and every lookup on it will miss.
2. Timestamps are not RFC 3339. Date and time are separated by a space, so strict ISO 8601 parsers reject them. Parse with an explicit format.
3. Booleans are t and f. A naive value == "true" check reads every boolean as false, silently.
4. Null and empty string are technically distinguishable (bare vs quoted) but most CSV libraries collapse them. If that distinction matters to you, confirm your parser preserves it before relying on it.

When someone edits the form

Your columns are your form's fields, so editing the form changes your columns. Changes are eventual, not immediate: exports are rebuilt in the background shortly after a form is published, so expect a short lag before a new column appears.

Row data can lag slightly too. Exports may be served from a read replica rather than the live database, so a submission saved seconds ago is not guaranteed to be in the very next export. This costs you nothing if you sync on a schedule — the row arrives in a later pull with its own last_modified, and the cursor picks it up — but do not treat an export as a real-time read.

The governing rule is that columns accumulate. A dataset's columns are the union of every version of the form you have ever published, not just the current one. Publishing cannot be undone, so in practice columns are added and effectively never removed:

Five consequences worth planning for:

• Your column count only grows. A long-lived form accumulates columns for every field it has ever had. Select the columns you need by header name; do not assume the set is minimal.
• Renaming a grid silently forks it. You end up with two datasets: the new one, and the old one, still listed and still returning 200. Nothing 404s and nothing disappears. The old one is not a snapshot — it keeps returning current rows, including submissions made after the rename — so nothing about the data looks wrong. What stops moving is its column set: it is no longer regenerated, so any field added to the grid afterwards never appears in it. A consumer left on the old key therefore keeps syncing plausible, current rows while silently missing every new column. If you sync grid datasets, re-read GET /v1/datasets on a schedule and alert on a *new* key appearing, not on an old one vanishing.
• Renaming an option label rewrites history. The label is what lands in the CSV, and options are merged on their stored value, so the current label wins for every row — including rows submitted long before the rename. If you need a choice that never moves under you, key on something other than the exported text.
• Editing a field in place reaches the export, but only from the next rebuild. A field's label and type come from your newest published version, so renaming a field updates displayName and retyping one changes the column's type. Neither is synchronous with publishing, so expect a window in which the export still shows the previous label or type. Retyping is the disruptive one: the column's values change shape under a consumer that has already inferred a type from them. Prefer adding a new field over retyping an existing one.
• A field's history is never dropped. Fields and options removed in a later version keep their columns and their positions, because historical submissions still have data behind them. This is the other half of why your column count only grows.

Build your consumer to tolerate all of this: select columns you know by header name, ignore columns you do not recognise, and treat a missing column as empty rather than fatal.

Incremental sync

The CSV endpoint returns at most 5000 rows per call, so every export is a loop. The same loop does your initial backfill and your ongoing updates — there is no separate "full export" mode, and running it from no cursor *is* a full export.

Rows come back in modification order, and the cursor is a position in that order. That is what makes the loop both resumable and incremental.

One thing the loop cannot tell you is that a record was deleted — deletions leave no row behind, so they are invisible to a cursor. If that matters to your reporting, run the same loop from no cursor on a schedule and reconcile: see Deleted records stop being exported.

Parameters

The loop

1. Call with no cursor. You get the oldest-modified rows first.
2. Take the last row of the response. Its last_modified and id are your next cursor.
3. Call again with modifiedSince = that last_modified and after = that id.
4. Repeat until a response has fewer rows than your limit. That is the end of the data.
5. Later, resume from the cursor you stored. Anything created or edited since comes back.

modifiedSince and after are one cursor and must be sent together — one without the other is a 400. Send neither for the first page.

That end-of-data rule is safe because an over-cap limit is refused, not reduced. 5000 is the maximum, and asking for more is a 400 rather than a quietly smaller page. This matters more than it sounds: were an over-cap limit clamped instead, a client asking for 10000 would get 5000 rows on its very first call, read "fewer rows than I asked for" as the end of the data, and stop partway through its first backfill with nothing to indicate it had. Compare against the limit you sent, and keep it at or below 5000.

afterRowIndex is part of the cursor, not an independent parameter. Send it only alongside modifiedSince and after, and only for a dataset that has a row_index column (see "Grid datasets need a third component" below). On its own, or on a dataset without that column, it is a 400 — it cannot be applied there, and being told so beats being handed page one again.

The cursor value contains a +, so it must be URL-encoded. last_modified looks like 2026-07-14 08:12:44.518+00. In a query string an unencoded + means a space, so the offset arrives as 00 and the request fails as a malformed timestamp — a 400 naming modifiedSince in its errors object. Let your HTTP client encode the parameters (that is what --data-urlencode is doing in the examples), or encode it yourself: + becomes %2B and the space between date and time becomes %20.

Grid datasets need a third component. Their rows share the parent submission's last_modified and id, so those two alone cannot advance past a submission's rows. Pass afterRowIndex = the last row's row_index alongside the other two. Omitting it on a grid dataset is a 400 rather than a silent loop — but only once you start paging, so make sure your first pull of a grid handles its second page correctly.

A failed export breaks the transfer, so never read a short body as a short page. The body is streamed, which means a failure *after* the response has started cannot be reported as an error status — the 200 and the headers are already sent. Rather than finish the response and leave you holding a partial CSV that looks complete, the export aborts the connection: your client sees a broken transfer, not a clean end of data. Command-line tools exit non-zero and HTTP libraries raise their usual stream-interrupted error rather than returning a body. That is your completeness signal, and it is the only one — there is no Content-Length to compare against, and no error marker inside the CSV itself. Treat it as a retryable failure and re-request from the same cursor. What matters is that you do not suppress it: a reader that swallows stream errors and keeps whatever arrived will commit a partial page, and if it advances the cursor past those rows they are never re-offered. A failure caught *before* the first bytes go out still answers with an ordinary error response — a 503 if the dataset's view cannot satisfy the export, a 500 otherwise (see Errors, limits and versioning) — so it needs no special handling beyond what you already do with a failed request.

A long backfill can see the header row change between pages. Publishing a form rebuilds the dataset, and a rebuild that lands mid-backfill means page 5 can have columns that page 1 did not. Read the header row of every response rather than the first one, and treat unknown columns as new and missing columns as empty.

Worked example

First page:

curl -s "https://api.provision.io/v1/datasets/daily_sanitation_check/csv?limit=1000" \
  -H "Authorization: Bearer $PROVISION_API_KEY"

Suppose the last row of that response is:

id                    , record_id , last_modified
0191f0c2-...-a41c     , DSC-1041  , 2026-07-14 08:12:44.518+00

Next page:

curl -s -G "https://api.provision.io/v1/datasets/daily_sanitation_check/csv" \
  --data-urlencode "modifiedSince=2026-07-14 08:12:44.518+00" \
  --data-urlencode "after=0191f0c2-...-a41c" \
  --data-urlencode "limit=1000" \
  -H "Authorization: Bearer $PROVISION_API_KEY"

For a grid dataset, add the last row's row_index:

--data-urlencode "afterRowIndex=7" \

Upsert, do not append

Because rows are ordered by modification time, an edited submission comes back again in a later page with its new last_modified. That is the mechanism that makes incremental sync work, and it means your loader must upsert rather than insert.

• Parent datasets: upsert on id — unless the form has more than one recording step, in which case id is not unique and upserting on it loses data (see How forms become datasets).
• Child (grid) datasets: upsert on id and row_index — id alone is not unique — and then trim the tail, as described below.

Grids need that extra step because row_index is positional, not an identity: it is the row's place in the grid, renumbered from 1 on every export. Reorder a grid's rows, or delete one from the middle, and every row below it shifts up — so the values now arriving under row_index 3 are not the values you previously stored under row_index 3. Upserting alone is therefore not enough: if a submission loses rows, the surplus rows you already stored are never overwritten and linger forever.

Do not fix that by deleting all rows for an id as each page arrives. Pages are cut from a flat ordered stream with no regard for submission boundaries, so one submission's rows can straddle two pages. Delete-then-insert per page would drop the rows that arrived in the earlier page. Instead:

1. Upsert each row on (id, row_index) as it arrives.
2. Track the highest row_index you have seen for each id during the sync.
3. After the sync finishes, for every id you touched, delete its rows whose row_index is greater than that maximum. Those are the leftovers from a shorter version of the grid.

If you would rather not track state, the alternative is to buffer rows by id and only replace a submission's rows once you are certain you have all of them — which you know when a later row arrives with a different id, or when the sync ends.

Reference implementation

import csv, io, time, requests

BASE = "https://api.provision.io/v1"
PAGE_SIZE = 1000          # well under the 5000-row cap; smaller pages, shorter requests

def cursor_from(row, has_row_index=False):
    """The cursor that resumes immediately after this row."""
    cursor = {"last_modified": row["last_modified"], "id": row["id"]}
    if has_row_index:
        cursor["row_index"] = row["row_index"]
    return cursor


def sync(dataset, key, cursor=None, has_row_index=False):
    """Yield every row from 'cursor' onward."""
    session = requests.Session()
    session.headers["Authorization"] = f"Bearer {key}"

    while True:
        params = {"limit": PAGE_SIZE}
        if cursor:
            params["modifiedSince"] = cursor["last_modified"]
            params["after"] = cursor["id"]
            if has_row_index:
                params["afterRowIndex"] = cursor["row_index"]

        r = session.get(f"{BASE}/datasets/{dataset}/csv", params=params, timeout=300)

        # Throttled: the cursor has not advanced, so wait out Retry-After and repeat the same call.
        if r.status_code == 429:
            time.sleep(int(r.headers.get("Retry-After", "60")))
            continue

        # Two different 503s, told apart by Retry-After. With it: a transient blip verifying the
        # API key — wait it out and repeat the same call. Without it: this dataset cannot be
        # exported until we fix it at our end. Alert a human; retrying only loops against a fixed
        # failure.
        if r.status_code == 503:
            if "Retry-After" in r.headers:
                time.sleep(int(r.headers["Retry-After"]))
                continue
            raise RuntimeError(f"{dataset} cannot be exported: {r.text}")

        r.raise_for_status()

        # utf-8-sig, not utf-8: the stream starts with a byte-order mark.
        rows = list(csv.DictReader(io.StringIO(r.content.decode("utf-8-sig"))))
        for row in rows:
            yield row

        if len(rows) < PAGE_SIZE:
            return                # caught up

        cursor = cursor_from(rows[-1], has_row_index)


# Persist the cursor derived from the last row you actually handled. Do not try to take it
# from sync() — it is a generator, so a value it returns never reaches this for loop.
cursor = load_cursor()            # None on the very first run
last_row = None

for row in sync("daily_sanitation_check", API_KEY, cursor):
    upsert(row)
    last_row = row

if last_row is not None:
    save_cursor(cursor_from(last_row))

Set has_row_index from the dataset's properties: if a row_index column is present, it is a grid dataset.

Operational advice

• Store the cursor durably, not in memory. It is the only state your integration needs, and it makes a crash mid-backfill cost nothing.
• One sync at a time per dataset. Concurrency is capped per organization (see Errors, limits and versioning), and a streaming export holds a database connection for its whole duration. Fanning out earns you 429s and finishes later than running datasets in sequence.
• A retry is always safe. The cursor does not advance until you have the rows, so a retried call re-fetches the same page rather than skipping it.
• Use a generous client timeout, and a modest limit. A 5000-row page of a wide form is a large streaming response. Prefer a smaller limit over a short timeout.

Errors, limits and versioning

This is the contract for everything that is not a 200: what can go wrong, which failures are worth retrying, how throttling behaves, and what we may change without telling you.

Error responses

Errors are returned as application/problem+json carrying a title and the status code. Most also carry a detail naming the specific case — a 401 is the exception, and is title-only. A parameter that cannot be parsed at all (a malformed timestamp, a non-numeric limit) is rejected before the checks below run, and answers with the same title and detail as everything else plus a field-keyed errors object naming each parameter at fault. Branch on the status code, not on detail.

Errors carry a traceId. Quote it. It identifies that single request in our logs — a support ticket that includes it is answerable, and one that does not usually is not. Log it alongside your own failures rather than discarding the body. The one exception is the 429, whose body carries type, title, status and detail but no traceId; read it defensively rather than assuming the field is there, and quote the time and your organization instead if you need to raise one with us.

Which to retry:

The two 503s: read the Retry-After header

The data endpoints under /v1 send a 503 for two unrelated conditions, and the Retry-After header is how you tell them apart. With the header, wait and retry. Without it, alert a human. (The /health/ready probe has its own 503 — transient, self-clearing, and outside this rule; see Authentication.)

**503 *Authentication temporarily unavailable*, with Retry-After** — we could not verify your API key because our identity provider did not respond. This is a transient blip on our side, not a credential problem, and it is deliberately not a 401: a 401 tells you to fix the key, which is exactly the wrong response here. It affects every request while it lasts, including /v1/datasets. Honour the header, retry, and only alert if it keeps happening.

**503 *Dataset cannot be exported*, without Retry-After — the dataset's export view cannot serve an export in its current shape. Each dataset is backed by an export view that we rebuild in the background whenever the form behind it changes. A rebuild is all-or-nothing: it either replaces the view completely or leaves the previous one untouched, so you always read a whole view and a rebuild in progress never causes this. What causes it is a view that predates a change to what the export requires and has not been regenerated since, or one the export cannot read for a structural reason on our side. Either way, nothing you can do moves it along and no amount of waiting clears it — it persists until we ship a rebuild for that dataset. That is why no Retry-After is sent: there is no interval at which retrying would start working, and advertising one would only turn a fixed failure into an unbounded retry loop. Treat it as an alert, not a backoff: contact support**, naming the dataset from the detail. It is also always about one dataset; the others keep exporting normally, so a stuck dataset should not stop the rest of your sync.

In both cases your cursor is unaffected and nothing was lost — when the condition clears, resume from exactly where you were.

Rate limits

Limits are enforced per organization, aggregated across every key you hold — rotating or adding keys does not increase throughput. Two limits apply:

• a concurrency cap on simultaneous in-flight requests. This is the one you will meet first: a streaming export holds a connection for as long as it takes to send, so a handful of parallel pulls is enough to reach it.
• a requests-per-minute backstop, set high enough that a well-behaved sequential client never approaches it.

Current defaults:

These values are deployment-wide, not per-customer: the counting is partitioned per organization, but the limits themselves are the same for everyone and we cannot raise them for one customer alone. They are also not a guarantee — treat them as current operating values that may be retuned, and do not hardcode them.

They are only reported back to you on a 429; a successful response carries no rate-limit headers, so there is no way to check your remaining budget without being throttled. Design for the 429 rather than trying to stay under a number you can observe.

The 429 contract

A 429 response carries these headers, and no successful response does:

The three RateLimit-* headers are specific to a 429. Retry-After also appears on one other response — the *Authentication temporarily unavailable* 503, where it means the same thing: wait that long, then retry. The dataset-unavailable 503 never sends it (see The two 503s: read the Retry-After header).

A well-behaved client:

1. Sleeps for Retry-After seconds. Do not retry sooner.
2. Resumes from the same cursor. Nothing was consumed, so nothing is lost.
3. Keeps one request in flight per dataset and syncs datasets sequentially rather than fanning out.

Because paging is resumable, throttling costs you time and nothing else. There is no partial state to reconcile and no need to restart a backfill.

Versioning and deprecation

The /v1 prefix is a compatibility promise about the API surface: endpoints, parameters, JSON field names, status codes, and the CSV dialect.

Within /v1 we may:

• add endpoints, and add fields to JSON responses;
• add query parameters that are optional and default to today's behaviour;
• add, remove, rename, or retype CSV columns — those follow your forms, not our API. Column changes are driven by your own configuration; see CSV columns.

Within /v1 we will not:

• remove or rename an endpoint, parameter, or JSON field;
• change the meaning or type of an existing JSON field;
• change the CSV dialect (delimiter, quoting, encoding, timestamp or boolean rendering);
• change the ordering guarantee or the cursor semantics that incremental sync depends on.

Deprecation. A breaking change ships as a new prefix (/v2) alongside /v1. We notify the technical contact on your account before /v1 is retired, and keep both live during the overlap.

Not yet ratified. The notice period for retiring a version is still being agreed internally, so it is deliberately omitted here rather than guessed at. Treat the above as the intended policy and confirm the notice window with your Provision contact before it matters contractually.

The practical takeaway: pin nothing about columns, and depend freely on the endpoint shapes.