# Provision API documentation

Every page of the Provision API documentation in one file. The HTML pages are docs.html, forms.html, resources.html; the OpenAPI document is at /swagger/v1/swagger.json.

---

# Page: Guide (docs.html)

How to export your Provision data as CSV: authentication, datasets, incremental sync, errors and limits.

## Overview

The Provision API exports the data your teams capture in Provision as CSV: form submissions and their
verification steps, the rows of grids inside them, the deviations raised on them, and your resources
(suppliers, customers, products and the rest). Use it to keep a copy in your own warehouse, BI tool
or pipeline, and to keep that copy current without downloading everything again.

> [!NOTE]
> **The one idea to take away:** most of your datasets are your forms, and their CSV columns are that
> form's fields. There is no fixed schema for those, because your forms *are* the schema. Beside them
> sit datasets for your [resources](resources.html#resources) and [deviations](resources.html#deviations), whose
> core columns are the same for every customer. Most resources add a column per custom attribute you
> define, so read the current columns from the API rather than assuming them.

### Base URL and versioning

Every endpoint that returns your data lives under `https://api.provision.io/v1`. The few open, data-free
endpoints in [Authentication](docs.html#authentication) sit outside that prefix. `/v1` is part of the
contract: [Versioning and deprecation](docs.html#errors-limits-and-versioning--versioning-and-deprecation)
sets out what may and may not change within it.

That host is production, and a key issued for it reads your live data. To build against something
else first, ask your Provision contact what is available for your organization. Non-production access
is arranged with us, not self-served, and a key only works in the environment it was issued for.

### The two endpoints

| Endpoint | Purpose |
|---|---|
| `GET /v1/datasets` | List your datasets and the current columns of each. Start here. |
| `GET /v1/datasets/{key}/csv` | Stream one dataset as CSV, paged by a resumable cursor. `{key}` is the dataset's `key` from the listing. |

Those two are everything that returns your records. The rest of these pages are about using them well.

### Quickstart

You need an API key; see [Authentication](docs.html#authentication). Treat it like a password.

**1. See what you can pull.**

```bash
export PROVISION_API_KEY="ak_your_key_here"

curl -s "https://api.provision.io/v1/datasets" \
  -H "Authorization: Bearer $PROVISION_API_KEY"
```

Each entry has a `key` (use it in URLs) and a `columns` array describing its columns. Most also
have a `displayName`, but not all, so treat it as optional.

**2. Pull the first page of one.**

```bash
curl -s "https://api.provision.io/v1/datasets/daily_sanitation_check/csv?limit=500" \
  -H "Authorization: Bearer $PROVISION_API_KEY"
```

You get CSV with a header row, oldest change first.

**3. Keep it current.** Every row carries its own paging position in its `_cursor` column. Send the
last row's `_cursor` value back as `cursor`, unchanged, and repeat until a page has fewer rows than
the `limit` you sent. [Incremental sync](docs.html#incremental-sync) covers the loop properly, including
the original `modifiedSince`/`after` parameters, which still work.

### The OpenAPI document

A machine-readable OpenAPI 3 description of both endpoints is served at
[`/swagger/v1/swagger.json`](/swagger/v1/swagger.json). Point a client generator, Postman or your API gateway at it.

| To find | Go to |
|---|---|
| A typed client, or an import into Postman or an API gateway | The OpenAPI document |
| Paths, parameter names and types, status codes **for the two data endpoints** | Either — they agree, and the spec names the path parameter `datasetKey` to match the listing |
| The open, data-free endpoints (`/health`, `/health/ready`, `/build`) | These pages. The spec covers only the two data endpoints, so a generated client will not have them |
| What a dataset is, how forms become columns, how to build the sync loop | These pages |
| CSV dialect, cursor rules, retry behaviour, rate limits | These pages |

The spec cannot describe your columns: those come from your forms, and you read them from
`GET /v1/datasets` at runtime.

**Pointing an AI assistant at these docs?** Give it [`/docs.md`](/docs.md), which holds every page
in one Markdown file, or [`/llms.txt`](/llms.txt), a short index of what is where.

## Integration checklist

The rules a correct integration follows. Each links to the detail. If you build nothing else from
these pages, build these.

### Requests

- **Authenticate with `Authorization: Bearer ak_…`.** A 401 usually means fix the key; retrying
  does not help. See [Authentication](docs.html#authentication).
- **Read `GET /v1/datasets` on every sync.** Store each dataset's `key`, and alert when a key you
  sync disappears: the form or grid was renamed. See [Datasets](docs.html#datasets).
- **Page with the `_cursor` column.** Send the last row's `_cursor` value back as `cursor`,
  unchanged and untruncated. Do not parse it, and do not send it together with
  `modifiedSince`/`after` — that is a 400. See [The loop](docs.html#incremental-sync--the-loop).
- **Treat a `INVALID_CURSOR` 400 as a real error**, never as a quiet restart: it is never answered
  with the first page. Read the `detail` — it says whether to fix how you store the cursor, send each
  dataset its own, or re-export that dataset from the first page.
- **Stop when a page has fewer rows than the `limit` you sent** — compare against your own `limit`,
  never against a number you wrote down. A client whose end-of-data test is `len(rows) < 5000` stops
  early and reports success the day that maximum moves. Keep `limit` at or below 5000, and read
  that maximum from these pages rather than hardcoding it. A page may come back with a few rows *more*
  than `limit`; that is not the end of the data and not an error.
- **Run one request at a time per dataset**, and sync datasets one after another. Concurrency is
  capped per organization. See [Rate limits](docs.html#errors-limits-and-versioning--rate-limits).

### Parsing

- **Read columns by header name, never by position**, on every page: columns are added as forms
  change, even mid-backfill. See [CSV columns](forms.html#csv-columns).
- **Decode as UTF-8 with a byte-order mark** (`utf-8-sig` in Python).
- **Raise your parser's field-size cap.** A reference field holds a whole record as JSON, in one
  field, and can run to hundreds of thousands of characters; Python's `csv` raises at 131,072 of them
  unless you call `csv.field_size_limit`. See [Reading the values](forms.html#csv-columns--reading-the-values).
- **Parse timestamps with an explicit format.** `2026-07-14 08:12:44.518+00` has a space, not a
  `T`. The envelope's `_local` columns have no offset. Booleans are `t` and `f`. See
  [Reading the values](forms.html#csv-columns--reading-the-values).
- **Map `date` to a date, `dateTime` to a timestamp with a time zone, and `localDateTime` to one
  *without*.** A `localDateTime` is already in the site's time zone and carries no offset, so
  converting it again shifts it. Take the type from `dataType`, not from the column name: the envelope
  and verification `_local` columns and those of `deviations` are `localDateTime`, but a form field
  or custom attribute whose own key happens to end in `_local` is an ordinary UTC instant. See
  [Column metadata](docs.html#datasets--column-metadata).

### Loading

- **Treat an interrupted download as a failed page**, never as a short one. Retry it from the same
  cursor, a bounded number of times.
- **Upsert, do not append.** Key on `id`, plus `row_index` on grid datasets, and trim grid rows
  beyond the highest `row_index` after each sync. See
  [Upsert, do not append](docs.html#incremental-sync--upsert-do-not-append).
- **Save the cursor after every page**, once that page's rows are committed. Store it as text with no
  length cap — a truncated cursor is refused rather than silently resumed, but it still stops your sync.
  On the original `modifiedSince` parameters, rewind by a few minutes before each run: a row can be
  stamped just before it becomes visible.
- **Re-read datasets in full on a schedule.** An incremental sync never shows a deletion, a renamed
  related record, or a form edit that rewrites old rows' exported text, such as relabelling an option.
  See [Deleted records](forms.html#csv-columns--envelope-columns).

### Errors

- **Branch on the status code and the `errorCode`, never on `title` or `detail`.** Every error body
  carries an `errorCode`; the prose beside it may be reworded at any time. See [Error
  responses](docs.html#errors-limits-and-versioning--error-responses).
- **On a 429, wait `Retry-After` seconds** and repeat the same request. Read `RateLimit-Limit` to see
  which limit you hit: the per-minute one means run slower overall, and `Retry-After` really clears it;
  on the concurrency one `Retry-After` is only a polling interval, because a slot frees when one of
  your own in-flight requests finishes — which for a large export takes minutes — so the fix is fewer
  requests in flight. See [The 429 contract](docs.html#errors-limits-and-versioning--the-429-contract).
- **On a 503, read the `errorCode`.** Only `DATASET_NOT_EXPORTABLE` is not worth retrying. See
  [The three 503s](docs.html#errors-limits-and-versioning--the-three-503s-tell-them-apart-by-errorcode).
- **Log the `traceId`** from error bodies, and quote it to support. Only the 429 has none, so read the
  field defensively.

## Authentication

Every request that returns your data needs a Bearer token in the `Authorization` header. There is
no other auth mode and no browser login.

A few endpoints carry no customer data and need no key:

- `GET /health`: liveness. Returns 200 whenever the service is up.
- `GET /health/ready`: readiness. Returns 503 while the service's own database dependency is
  unhealthy. It is a coarse "is the service up" check, not proof that exports will work: it does not
  probe the datastore exports read from, or any individual dataset. The export endpoint's own 503 is
  the authority on that.
- `GET /build`: the deployed build (image tag, commit, build date). Quote it when reporting a problem.
- These documentation pages, and the OpenAPI document.

None of these are in the OpenAPI document, so a generated client will not know about them.

### The header

```http
Authorization: Bearer ak_xxxxxxxxxxxxxxxxxxxxxxxx
```

The token is the **API key secret** issued to your organization. It begins with `ak_`; a
credential that starts with anything else is the wrong one.

> [!WARNING]
> **A key has two values that both begin `ak_`, and only one of them works.** The key has an
> **id**, which is what you see when the key is listed, and a **secret**, which is the credential. The
> prefix does not tell them apart. Sending the id gets you a **401** that looks exactly like a wrong
> or revoked key, so it is easy to spend an afternoon rotating a key that was never the problem. The
> secret is shown to you **once**, when the key is issued — that is the value to store, and the one that
> goes in this header.

A missing, malformed, revoked or expired token gets **401**. Retrying changes nothing, so alert on
the first one.

It is almost always the key. The exception: a valid, current key that is not tied to an organization
we know about also gets 401, and replacing the key will not fix that. If a newly issued key has never
worked, tell us rather than rotating it.

That differs from **404** *Organization not configured*, a later stage: your key is accepted and your
organization recognised, but it has not been set up for data export yet. 401 means we could not tell
who you are; 404 means we could, and have nothing to export to you yet. Only we can fix the 404.

If we cannot verify your key because our identity provider is unreachable, you get
**503** *Authentication temporarily unavailable* with a `Retry-After` header, not a 401. Your key
is fine: wait, retry, and alert only if it persists.

Two timing details can make a correct key look broken, or a revoked one look live:

- **A rejected key stays rejected for a short while** (tens of seconds), even if you retry it. The
  rejection is remembered per secret, so deploying a *different*, correct key works on its very next
  request.
- **A revoked key can keep working for about a minute.** A successful check is cached for 60 seconds,
  and a check already in flight when we revoke the key can start its own 60 seconds, so the worst case
  is a minute plus one round trip to our identity provider. Using the key does not extend
  that. A key is checked once, when a request arrives, so an export accepted just before the cache
  expires streams its whole page. Revoke a leaked key, but do not treat revocation as instant: if a
  key is being abused, tell us, and assume the last minute of reads got through, plus any export
  still streaming.

### Keys are scoped to your organization

A key belongs to exactly one Provision organization, and every response is filtered to it. No
parameter widens that: you cannot reach another organization's data, and you never pass an
organization id.

If your organization has not been enabled for API access, you get **404** *Organization not
configured*. That is a setup step on our side; contact support.

### Getting a key

Keys are minted by Provision on request. **There is no self-serve key management in the product
today.** Ask your Provision contact or support for one, and say which integration it is for: that
description is stored with the key and is how we identify it later.

You receive the secret **once**. We cannot recover it; if it is lost, we revoke it and issue a new one.

A key can be issued with an **expiry**, enforced separately from revocation: when it passes, the key
starts returning 401 with nothing changed on your side. Ask what expiry your key has and note the
renewal date. We do not warn you as it approaches.

### Handling the secret

- Keep it in a secret manager or environment variable. Never commit it, and never ship it to a
  browser or mobile app: it reads your whole organization's data.
- Ask for **one key per integration**, so revoking a leaked warehouse key does not also break your BI
  refresh.
- To rotate, take the new key, deploy it, then ask us to revoke the old one. Both work during the
  overlap, so there is no downtime.
- Rate limits count **per organization, across all your keys**. More keys do not buy more throughput;
  see [Rate limits](docs.html#errors-limits-and-versioning--rate-limits).

## Datasets

A **dataset** is one exportable table. `GET /v1/datasets` lists every dataset your organization has.
Treat that response as the authoritative description of your data and re-read it regularly, rather
than hardcoding what you found on day one.

Datasets come in three families:

- **Form datasets**, built from your published forms. They differ between customers and change when
  your forms change. A form produces a **submission dataset** (one row per submission) and a **grid
  dataset** for each repeating table on it, at any depth. [How your forms become
  CSV](forms.html#forms-to-csv) explains them, with a worked example.
- **Resource datasets**, one per kind of resource: suppliers, customers, products, product
  categories, equipment, chemicals, employees, areas, lots, recipes and documents. Their keys begin
  `resources_`, and their core columns are the same for every customer. All but
  `resources_documents` add one column per custom attribute you define. [Resources](resources.html#resources)
  lists every column.
- **`deviations`**: one row per value flagged as out of spec on a submission. Its columns are the same
  for every customer; see [Deviations](resources.html#deviations).

All three behave the same way in this API: every dataset has `id` and `last_modified`, appears in the
same listing, and pages through the same endpoint under the same cursor rules — with the third cursor
value a grid dataset needs on the original parameters.

The listing separates **resource** datasets from the other two: they are the ones with no
`displayName`, and their keys begin `resources_`. It does not separate a form dataset from
`deviations` — neither has a prefix to go by, and both have a `displayName`. So match on the keys you
expect and ignore the rest, rather than assuming every entry is a form.

### The response

```json
[
  {
    "key": "daily_sanitation_check",
    "displayName": "Daily Sanitation Check",
    "childDatasets": ["daily_sanitation_check__equipment_checks"],
    "columns": [
      { "key": "id",            "dataType": "uuid",     "label": "ID" },
      { "key": "record_id",     "dataType": "text",     "label": "Record ID" },
      { "key": "site_name",     "dataType": "text",     "label": "Site" },
      { "key": "last_modified", "dataType": "dateTime", "label": "Last Modified (UTC)" },
      { "key": "water_temp_c",  "dataType": "number",   "label": "Water Temperature (C)" }
    ],
    "links": { "csv": "/v1/datasets/daily_sanitation_check/csv" },
    "changeToken": "ee39296d056e6e33"
  }
]
```

**`columns` above is abridged**, so do not size a schema from it. A real submission dataset also
lists the rest of its envelope (`site_id`, `site_path`, `status`, `is_archived`, `started_at` and
`completed_at` with their `_local` siblings, and `last_modified_local`) before one entry per form
field, and ends with `_cursor`, which every dataset has and which the export appends after the
dataset's own columns. [The envelope
columns](forms.html#csv-columns--envelope-columns) is the full list; `GET /v1/datasets` is authoritative for
your own forms.

| Field | Meaning |
|---|---|
| `key` | Dataset identifier: the value you put in the CSV URL path. Always present. Where these pages say a dataset *name*, they mean this. |
| `displayName` | Human label, for display only. **Absent on every `resources_` dataset**: form datasets and `deviations` have one, resource datasets never do. Fall back to `key`. |
| `parentDataset` | On a grid dataset, the key of the dataset one level up: the submission dataset for a top-level grid, the outer grid dataset for a nested one. **Absent**, not `null`, on top-level datasets. Also absent when the dataset above is not itself in this listing: we would rather report a grid as top-level than point at a key you cannot fetch. This field is the hierarchy; never derive it from the key. |
| `childDatasets` | Keys of the datasets one level below this one, if any. The inverse of `parentDataset`, so a grid containing a grid is both a child and a parent. |
| `columns` | The current columns, in the order the CSV happens to use now. The order of the dataset's own columns is not a contract — match on `key`; the appended `_cursor` comes last. |
| `links.csv` | Path to this dataset's CSV endpoint, rooted at the host. It already contains `/v1`, so resolve it against `https://api.provision.io`, not the `/v1` base URL, or you get `/v1/v1/...`. |
| `changeToken` | An opaque marker of this dataset's contents. Store it, and on your next call compare it with the value you stored: **the same token means none of the dataset's own records changed**, so you can skip the CSV call. It does not cover a value the dataset merely displays from a related record, which can change without moving the token or any row's `last_modified`. See [Skipping datasets that have not changed](docs.html#incremental-sync--skipping-datasets-that-have-not-changed). |

> [!CONTRACT]
> **`changeToken` is opaque: compare it only with a previous value of itself.** It is not a time, it
> is not ordered, and tokens from two different datasets have nothing to do with each other. In
> particular it is **not** comparable with `last_modified` or with your export cursor — those are
> record timestamps, while the token is a hash of our own bookkeeping, and what goes into it differs
> between form and resource datasets. Comparing the two makes every dataset look permanently out of
> date.

> [!CONTRACT]
> **Null fields are omitted, not sent as `null`.** The example above is a submission dataset, so it has
> no `parentDataset` key at all. Test with `"parentDataset" in obj`, not `obj.parentDataset === null`,
> and do not deserialize into a schema that requires the property.

### Column metadata

Each entry in `columns` describes one CSV column:

| Field | Meaning |
|---|---|
| `key` | The exact CSV header for this column. Match on this. |
| `dataType` | Coarse type: one of `text`, `number`, `boolean`, `dateTime`, `localDateTime`, `date`, `uuid`, spelled exactly as shown. |
| `label` | Human label, for your UI, not for matching. **Optional, and often missing.** On a form dataset it is the field's label and is effectively always there. On a resource dataset only some of the *fixed* columns have one — `resources_customers` labels `id`, `products` and `last_modified`, and not `name`, `city` or the rest — but every **custom attribute** column does, carrying the attribute's own name. Fall back to `key`. |

On a form dataset, a column's `key` comes from the form field's **key**, which survives wording
changes, and its `label` from the field's **label**, which changes freely. Build against
`key`; show `label`. (A dataset's own `key` is different: it comes from the form name and
grid labels, so relabelling a form or grid renames it; see [When someone edits the
form](forms.html#form-changes).)

**A `label` is often composed, not copied.** That plain rule holds for a field that produces exactly
one column. Every other column gets a label built from the field's label and what the column *is*:
`Verification: <label>` for a verification column, `<label> - Name` and `<label> - Date` for the
two halves of a signature, `<label> - ID` for the id column of a resource select,
`<label> - <question>` for one row of a survey, `<label> - <option>` for one option of a checkbox
group, and `<grid label> - Row Index` for the ancestor row-index columns of a nested grid. Every form
but the verification one joins its parts with ` - `, and a field's own label may contain that too, so
**do not split a label back into its parts** — match on `key`.

> [!WARNING]
> **`displayName` moves when someone relabels a form or a grid.** A submission dataset's is the bare
> form name; a grid dataset's is `<form name> - <grid label>`, one segment per level of nesting. Both
> are rewritten on the next rebuild after a rename, exactly as the dataset `key` is. It is a display
> string, not an identifier — do not key a dashboard, a mapping table or a warehouse schema on it.

`dataType` tells you how to parse a value, not the exact database type. Three things to know before
you generate a warehouse schema from it:

- **A date without a time is `date`, not `dateTime`.** It has no time and no offset, so it names a
  calendar day, not an instant. `resources_lots` has three (`expiry_date`, `date_received`,
  `date_produced`), and every Date custom attribute adds one. The CSV agrees: a `date` column is
  written `2026-07-14`, while a `dateTime` always writes a time, even at midnight
  (`2026-07-14 00:00:00+00`). Store `date` as a date: a timestamp column invents a midnight the
  source never stated.
- **`dateTime` is an instant in UTC; `localDateTime` is a wall-clock time at the site.** A
  `dateTime` (`last_modified`, `started_at`, `completed_at`) names a moment, and the CSV writes it
  with an offset (`2026-07-14 00:00:00+00`). A `localDateTime` has already been converted to the
  site's time zone and has no offset (`2026-07-14 12:00:00`): the `_local` envelope and verification
  columns, and those of `deviations`. Store a `localDateTime` as a local (timezone-less) timestamp;
  converting it again attaches an offset the source never stated. A form field or custom attribute
  whose own key ends in `_local` is still a UTC instant.
- **`text` is also the fallback** for anything with no narrower type, so it means "nothing more
  specific to say", not "free text".

The values above are what `/v1` publishes today, and the vocabulary is **additive**: we may add a
value to it, but no existing value changes meaning and none is removed; see [Versioning and
deprecation](docs.html#errors-limits-and-versioning--versioning-and-deprecation). So do not close an enum
over them. **Treat a `dataType` you do not recognise as `text`** — that is what the API itself does
with a type it has no narrower name for, and it is always a safe reading.

### Dataset and column naming rules

Dataset names are generated from form names and grid labels, and column names from field keys:
lowercased, spaces and dashes turned into underscores, anything else removed. What follows from that:

- Runs of underscores are collapsed **within** a name, so a double underscore only ever separates
  parts: `parent__grid` for grid datasets, and `field__option` / `field__question` for the
  multi-column field types in [How each field type becomes
  columns](forms.html#csv-columns--field-types).
- Identifiers are capped at 63 characters. A name taken straight from one form or field name is simply
  truncated. A **composed** name (anything built with a separator or suffix: `parent__grid`,
  `field__option`, `field__question`, `verification_<field>`, the `<field>_id`, `<field>_name` and
  `<field>_date` companion columns, and resource custom-attribute columns) instead gets an
  8-character hash of the full name appended when it overflows (`…_3f9ac1b2`). The hash eats into
  the visible part, so a grid dataset's key does not always contain its parent's.
- When two **column** names in one dataset come out equal, a numeric suffix (`_2`, `_3`) keeps both.
  Which one keeps the plain name follows the order the fields are processed in, which does not change
  between rebuilds. Two long field names that truncate to the same 63 characters are separated the
  same way.
- When two **dataset** names collide, an 8-character hash separates them, so both still export. The
  hash is of the form's id, plus, for a grid dataset, the keys of the grid and the grids around it; in the rare case
  the hashed name collides too, a numeric suffix is added. Which one keeps the plain name depends on
  which was generated first, and stays put after that.

So **never derive a long identifier by eye.** Read names from `GET /v1/datasets` and match on what
you receive. A dataset with a hash suffix pages and exports like any other.

### What happens when a form is deleted

**The dataset stays.** It remains in `GET /v1/datasets`, keeps its columns, and keeps exporting what
it held when the form was last generated. Nothing is added to it again, and nothing removes it. This
is permanent, not a delay before a cleanup.

The listing does not mark such a dataset, so expect to meet one:

- **A dataset that stops changing is not necessarily a broken sync.** Its `last_modified` values stop
  advancing and every incremental pull returns nothing, which looks exactly like a broken sync. A
  deleted form is at least as likely an explanation.
- **Its rows are still real history.** Deleting a form does not retract the submissions made against
  it, so keep them unless you have your own reason not to.

Nothing in the API says whether a dataset is still live. Ask the technical contact on your account; see
[How you will hear about a change](docs.html#errors-limits-and-versioning--how-you-will-hear-about-a-change).

### Terms these pages use

**Four words, one meaning each.** A **submission dataset** is the top of a form's chain: one row per
submission, never a grid. A **grid dataset** comes from a repeating table, at any depth. **Parent**
always means *one level up*, as `parentDataset` reports: the submission dataset for a top-level grid,
the enclosing grid dataset for a nested one. **Form dataset** means both kinds; anything said of every
form dataset holds for grid datasets too.

**Two words, one thing.** A filled-in form is a *submission* on most of these pages, and a *record*
where its human-readable code is concerned (`record_id`, and what Provision users call it).

*Record* is looser only where a rule covers every dataset at once: "a deleted record stops being
exported" covers a customer or product as much as a submission. Next to `record_id` or a record code,
it always means a submission.

Our parameter descriptions and one error message say **"grid/child dataset"**. That is a grid dataset;
"child" is its position under a parent, not a third kind. `afterRowIndex` follows that family and
nothing else: **required on a grid dataset you page with `after`, and rejected on every other
dataset.** A grid dataset is one with a `parentDataset`, and it is the only kind with a `row_index`
column — `row_index` is reserved against form fields and custom attributes everywhere else, so no
other dataset can acquire one. You do not have to classify anything to get this right: both mistakes
are a 400 whose `detail` names the fix, and `cursor` needs none of it.

## Incremental sync

The CSV endpoint returns about 5000 rows per call, so every export is a loop. The same loop
does the first backfill and every later update: there is no separate full-export mode, and running
the loop with no cursor *is* a full export.

Rows come back in **modification order**, and the cursor is a position in that order. That is what
makes the loop both resumable and incremental.

Every row carries its own cursor, in the `_cursor` column. Send the last row's back as `cursor` and
the next call resumes immediately after it. That is the whole mechanism: you never build a cursor, you
echo one back.

> [!CONTRACT]
> **A page can hold a few rows more than the `limit` you asked for.** Several rows can share a
> modification position, and a page is never cut through the middle of such a group — it runs on to the
> end of it. That holds for every dataset, on either set of paging parameters, so treat `limit` as
> "about this many", and keep using the rule below to find the end of the data: **fewer** rows than your
> `limit` still means the end, and that never fires early.

The loop cannot tell you a record was **deleted**: a deletion leaves no row behind. If that matters,
run the loop with no cursor on a schedule and reconcile; see [Deleted records stop being
exported](forms.html#csv-columns--envelope-columns).

### Parameters

| Parameter | Type | Required | Meaning |
|---|---|---|---|
| `cursor` | string | No | Opaque page cursor: the _cursor value of the last row processed. Omit for page one. Not combinable with the three parameters below. |
| `modifiedSince` | string (date-time) | No; required together with after when paging without cursor | Superseded by cursor. Keyset cursor: the last_modified of the last row of the previous page. |
| `after` | string (uuid) | No; required together with modifiedSince | Keyset cursor tiebreaker: the id of the last row of the previous page. |
| `afterRowIndex` | integer (int64) | Only alongside `after`: required there on a dataset that has `row_index`, rejected on one that does not | Further tiebreaker: the row_index of the last row of the previous page; at least 1. Only valid alongside modifiedSince and after. |
| `limit` | integer (int32) | No | Page size between 1 and 5000, defaults to 5000. A larger value is rejected rather than reduced. |

### The loop

1. Call with no cursor. You get the oldest-modified rows first.
2. Take the **last row** of the response and read its `_cursor` column.
3. Call again with `cursor` = that value, sent back **exactly** as it arrived.
4. Repeat until a response has **fewer rows than your `limit`**. That is the end of the data.
5. Later, resume from the cursor you stored. Anything created or edited since comes back.

**Compare against the `limit` you sent, not against a constant.** `rows < limit` is the test; `rows
< 5000` only looks like it. If the maximum ever changes, the first breaks nothing and the second
stops a backfill part-way through and reports success. Read the maximum from this page instead of
hardcoding it, and prefer a `limit` you chose yourself over relying on the default.

> [!CONTRACT]
> **`cursor` is opaque.** Store it as a string, send it back unchanged, and do not parse, shorten,
> rebuild or compare it. It is versioned, so the ordering behind it can change without changing this
> loop — which is the point of it being opaque, and the reason not to build one yourself.

> [!NOTE]
> The cursor on **any** row is valid, not just the last one. If you commit rows one at a time, store the
> `_cursor` of the last row you committed and a crash costs you nothing rather than a page.

**Cursor problems are always a 400, never a quiet restart.** A cursor that is empty, truncated,
altered, taken from a different dataset, or issued before the dataset was rebuilt into a different shape
is refused with a message saying which of those it is. It is never answered with the first page —
that would look like a successful resume and silently re-ingest the whole dataset. So if you get a
`INVALID_CURSOR` 400, read the `detail`: it tells you whether to fix your storage or start the
dataset again. Omit `cursor` entirely to start from the beginning; sending an empty one is refused.

**The original cursor parameters still work.** `modifiedSince`, `after` and `afterRowIndex` page
the same data, and a client using them needs no change. They describe a position that several rows can
share, which is why a page served for them can run on past your `limit` to the end of that group.
Prefer `cursor` all the same: it carries the whole position in one value, so a grid dataset needs no
third parameter, there is no timestamp for your storage to round, and it will not have to grow a fourth
parameter the next time the ordering does.

> [!CONTRACT]
> Send **either** `cursor` **or** `modifiedSince`/`after`/`afterRowIndex` — sending both is a
> 400, because they are two different answers to "where was I". `modifiedSince` and `after` are one
> cursor and must be sent **together**; one without the other is also a 400. Send none of them for the
> first page.

> [!WARNING]
> **Unknown query parameters are ignored, not rejected.** A misspelled name (`modified_since`,
> `cursor_token`, `offset`) gets a 200 with the first page again, so a loop built on one never ends.
> Use exactly `cursor`, `modifiedSince`, `after`, `afterRowIndex` and `limit`.

> [!WARNING]
> **On the original parameters, send `last_modified` back exactly as you received it.** It has
> microsecond precision, and the cursor compares strictly. Parsed into a type that keeps only
> milliseconds or seconds (a JavaScript `Date`, many ORMs and warehouse loaders) and written back, it
> moves the cursor earlier and pages repeat rows. Store it as the original text. `cursor` has no such
> hazard — there is nothing in it for a type to round.

**That end-of-data rule is safe because a `limit` above 5000 is refused, not reduced.** If it
were quietly reduced, a client asking for 10000 would get 5000 rows on its first call, read
"fewer than I asked for" as the end, and stop **partway through its first backfill** without any sign
of it. Compare against the `limit` you sent, and keep it at or below 5000.

**Grid datasets need a third cursor value — on the original parameters only.** Their rows share the
`last_modified` and `id` of their submission, so those two alone cannot move past a submission's rows.
Send `afterRowIndex` = the last row's `row_index` as well. Leaving it out is a 400, not a silent
loop, but only from the second page, so test that your first pull of a grid gets past its second page.
`cursor` needs none of this: it already carries the row's position, so a grid pages exactly like a
submission dataset.

**`afterRowIndex` is part of the cursor, not a parameter of its own.** Send it only with
`modifiedSince` and `after`, and only on a grid dataset. It is at least 1:
`row_index` counts from 1, so `afterRowIndex=0` is refused rather than read as "from the start".
Once it parses as an integer, four things make it a 400: a value below 1; sending it without
`modifiedSince` and `after`; sending it on a dataset that is not a grid dataset; and leaving it
out when paging one that is. A value that is not an integer at all is refused earlier,
before the dataset is looked at; see [When more than one thing is
wrong](docs.html#errors-limits-and-versioning--when-more-than-one-thing-is-wrong).

> [!WARNING]
> **The cursor contains a `+`, so it must be URL-encoded.** `last_modified` looks like
> `2026-07-14 08:12:44.518+00`. Unencoded in a query string, `+` means a space, the offset arrives
> as ` 00`, and the request fails with a 400 naming `modifiedSince` in its `errors` object. Let
> your HTTP client encode parameters (as `--data-urlencode` does in the examples), or encode `+` as
> `%2B` and the space as `%20` yourself.

**An interrupted download is a failed page, never a short one.** The body is streamed, so a failure
*after* the response has started cannot change the 200 that was already sent. Instead of finishing a
partial CSV that looks complete, the export **aborts the connection**: command-line tools exit
non-zero, and HTTP libraries raise their usual interrupted-stream error. That is your only
completeness signal: there is no `Content-Length` and no error marker inside the CSV.

Retry the page from the same cursor, but a **bounded** number of times. Most aborts are transient.
One is not: a single stored value the export cannot render. When that row lies beyond the first 64 KiB
of the response, the failure is found too late to send an error, so the connection aborts at the same
point on every attempt. **If retries keep failing at the same cursor, stop and contact support.** It is
the same condition the *Dataset cannot be exported* 503 reports when it is caught before the response
starts. Whatever you do, do not swallow the error: a reader that keeps whatever arrived commits a
partial page, and if it then advances the cursor, the missing rows are never offered again. A failure
caught *before* the first bytes go out is an ordinary error response instead: one of the three
**503**s or a **500**, each with its own retry rule in [Errors, limits and
versioning](docs.html#errors-limits-and-versioning).

**A row can arrive just behind your cursor.** A submission's `last_modified` is set a moment before
it becomes visible to the export, so a row can appear with a timestamp slightly older than a cursor you
already hold, and a strict cursor never offers it. On the original parameters, rewind your stored
`modifiedSince` by a few minutes before each run. A `cursor` cannot be rewound — it is opaque, and
there is nothing in it to move back — so cover this by re-reading the dataset in full on a schedule
instead. Rows you have already loaded come back either way, which costs nothing because you upsert them.

**The header row can change between pages of a long backfill.** Publishing a form rebuilds its dataset,
so page 5 can have columns page 1 did not. Read the header of **every** response, treat unknown
columns as new, and treat missing columns as empty.

### Skipping datasets that have not changed

The loop above syncs one dataset. If you have many, running it against every one of them on every
cycle means one CSV call per dataset just to discover that most of them have nothing new — and if you
have a thousand datasets, that is a thousand calls to learn almost nothing.

Two things in `GET /v1/datasets` let you avoid that. Poll the catalogue first, and use it to decide
what to export.

**Per dataset, compare `changeToken`.** Store the token alongside the cursor. On the next cycle, a
dataset whose token is unchanged has nothing new, so skip its CSV call entirely. The token tells you
*whether* to export; the cursor still tells you *where to resume*. Keep both.

**For the whole catalogue, use the `ETag`.** Every response carries one. Send it back as
`If-None-Match` and you get **304 Not Modified** with no body when nothing at all has changed — no
dataset moved, and none was added, renamed or reshaped. A cycle that finds nothing then costs exactly
one request.

```bash
curl -s -D- -o /dev/null "https://api.provision.io/v1/datasets" \
  -H "Authorization: Bearer $PROVISION_API_KEY" \
  -H 'If-None-Match: "02e856e3c8009965435b30c16efa0bee"'

HTTP/1.1 304 Not Modified
ETag: "02e856e3c8009965435b30c16efa0bee"
```

So a cycle is: conditional call to `GET /v1/datasets`; on 304 nothing in the catalogue moved, so there
is nothing to export this cycle; on 200, export only the datasets whose token changed, then store the
new tokens and the new ETag. That is the fast path, not the whole job — read the warning below before
treating a 304, or an unchanged token, as "nothing to do".

> [!CONTRACT]
> **A changed token means "worth exporting", not "there are new rows."** A form dataset's token follows
> our refresh pipeline, so a rebuild moves it even when nothing in the dataset changed, and an export
> that comes back with no rows is a normal outcome rather than a sign of a broken cursor. It is
> deliberately conservative in that direction.

> [!WARNING]
> **The token does not make polling complete, and what escapes it depends on the dataset.** A resource
> dataset's token follows its own base table and nothing else, so a change to a value the dataset
> *shows but does not own* — the product names on `resources_customers`, the site name on
> `resources_lots` — moves neither the token nor any row's `last_modified`. Your cursor will not
> offer those rows again either, so nothing in the API will tell you they changed; see [List
> columns](resources.html#resources--list-columns). If you maintain a mirror, re-read resource datasets in full on
> a schedule and reconcile. **That re-read sits outside the cycle above**: no token and no 304 will
> ever ask you for it, so schedule it on your own clock rather than waiting for a signal that cannot
> come.

> [!NOTE]
> Poll the catalogue about as often as you would have polled the datasets — it is now the cheap call.
> It is also the authoritative description of your data, so polling it keeps you current with new and
> renamed datasets rather than discovering them late.

Responses are marked `Cache-Control: private, no-cache`: the catalogue is specific to your
organization, so store the ETag and revalidate rather than serving it from a shared cache.

### Worked example

First page:

```bash
curl -s "https://api.provision.io/v1/datasets/daily_sanitation_check/csv?limit=1000" \
  -H "Authorization: Bearer $PROVISION_API_KEY"
```

Suppose the last row of that response is:

```text
id                    , record_id   , last_modified
0191f0c2-...-a41c     , 26-195-0042 , 2026-07-14 08:12:44.518+00
```

Next page — the last row's `_cursor`, sent straight back:

```bash
curl -s -G "https://api.provision.io/v1/datasets/daily_sanitation_check/csv" \
  --data-urlencode "cursor=pvc1.AQyZ8fUAAAYwtOA3ARnwwkNhQRyZ8f..." \
  --data-urlencode "limit=1000" \
  -H "Authorization: Bearer $PROVISION_API_KEY"
```

The same page on the original parameters, which still work and need no `_cursor`:

```bash
curl -s -G "https://api.provision.io/v1/datasets/daily_sanitation_check/csv" \
  --data-urlencode "modifiedSince=2026-07-14 08:12:44.518+00" \
  --data-urlencode "after=0191f0c2-...-a41c" \
  --data-urlencode "limit=1000" \
  -H "Authorization: Bearer $PROVISION_API_KEY"
```

For a grid dataset, that second form also needs the last row's `row_index`:

```bash
--data-urlencode "afterRowIndex=7" \
```

### Upsert, do not append

Rows come back in modification order, so **an edited submission comes back again** in a later page
with its new `last_modified`. That is what makes incremental sync work, and it means your loader must
upsert, not insert:

- Submission datasets: upsert on `id`, which is unique there whatever the form's shape.
- Grid datasets: upsert on `id` **and** `row_index` (`id` alone is not unique), then **trim the
  tail** as described below.

Grids need the extra step because `row_index` is a **position**, not an identity: it is renumbered
from 1 on every export. Reorder a grid's rows, or delete one from the middle, and every row below it
moves up, so what now arrives as `row_index` 3 is not what you stored as `row_index` 3. If a
submission loses rows, upserting never overwrites the surplus rows you already stored.

**Do not fix that by deleting an `id`'s rows as each page arrives.** Pages are cut from one ordered
stream regardless of submission boundaries, so a submission's rows can straddle two pages, and
delete-then-insert per page would drop the ones from the earlier page. Instead:

1. Upsert each row on `(id, row_index)` as it arrives.
2. Track the highest `row_index` seen for each `id` during the sync.
3. **After the sync finishes**, delete, for every `id` you touched, its rows with a higher
   `row_index` than that. They are left over from a shorter version of the grid.

To avoid tracking state, buffer rows by `id` instead, and replace a submission's rows once you know
you have all of them: when a row with a different `id` arrives, or when the sync ends.

**One kind of change the cursor never shows.** `last_modified` follows the row's *own* record, but
some columns read from records beside it: the comma-joined list columns on resource datasets, and any
column showing a related record's name. Rename a product and every supplier and customer row listing
it exports a different `products` value, while none of their `last_modified` values move. An
incremental sync never offers those rows again, and nothing signals the change. If those columns
matter, **re-read the affected resource datasets in full on a schedule**; they are small next to
submission data.

### Reference implementation

A complete sync for one dataset, in Python with `requests`. It saves the cursor after every page,
retries only what is worth retrying, and gives up after a bounded number of attempts.

```python
import csv, io, sys, time, requests

BASE = "https://api.provision.io/v1"
PAGE_SIZE = 1000        # well under the 5000-row cap: smaller pages, shorter requests
MAX_ATTEMPTS = 6

# A reference field carries a whole record as JSON, past the module's 131,072-character default.
# The cap is a C long, so a bare sys.maxsize raises on Windows.
csv.field_size_limit(min(sys.maxsize, 2**31 - 1))

class SyncError(Exception):
    pass

def fetch_page(session, dataset, cursor):
    """One page of rows. Retries transient failures; raises on everything else."""
    params = {"limit": PAGE_SIZE}
    if cursor:
        # Opaque: stored and sent back as it arrived, never parsed or rebuilt.
        params["cursor"] = cursor

    for attempt in range(MAX_ATTEMPTS):
        backoff = min(60, 2 ** attempt)
        try:
            # requests reads the whole body here, and raises if the download is cut off.
            r = session.get(f"{BASE}/datasets/{dataset}/csv", params=params, timeout=300)
        except (requests.ConnectionError, requests.Timeout, requests.exceptions.ChunkedEncodingError):
            time.sleep(backoff)
            continue

        if r.status_code == 200:
            # utf-8-sig, not utf-8: the body starts with a byte-order mark.
            return list(csv.DictReader(io.StringIO(r.content.decode("utf-8-sig"))))

        if r.status_code == 429:
            time.sleep(int(r.headers.get("Retry-After", backoff)))
            continue

        if r.status_code >= 500:
            code = ""
            try:
                code = r.json().get("errorCode", "")     # stable; "title" beside it is prose
            except ValueError:
                pass
            if code == "DATASET_NOT_EXPORTABLE":
                raise SyncError(f"{dataset} cannot be exported: {r.text}")
            time.sleep(int(r.headers.get("Retry-After", backoff)))
            continue

        raise SyncError(f"{dataset}: HTTP {r.status_code}: {r.text}")    # 400, 401, 404

    raise SyncError(f"{dataset}: still failing after {MAX_ATTEMPTS} attempts")

def sync(dataset, session, load_cursor, save_cursor, upsert):
    cursor = load_cursor(dataset)            # None on the very first run

    while True:
        rows = fetch_page(session, dataset, cursor)
        for row in rows:
            upsert(dataset, row)
        if rows:
            cursor = rows[-1]["_cursor"]     # the last row's own position
            save_cursor(dataset, cursor)     # after every page: a crash repeats at most one page
        if len(rows) < PAGE_SIZE:
            return                           # caught up
```

`session` is a `requests.Session` whose `Authorization` header carries your key.
`load_cursor`, `save_cursor` and `upsert` are yours. Commit each page's upserts before
`save_cursor` stores the cursor past them. Store the cursor as text, with no length limit that could
truncate it: a truncated cursor is refused with a 400 rather than silently resuming somewhere else, but
that still stops your sync. `len(rows) < PAGE_SIZE` is the end-of-data test, and it stays correct even
though a page can come back slightly *larger* than `PAGE_SIZE`. For grid datasets, trim the tail after
`sync` returns, as [Upsert, do not append](docs.html#incremental-sync--upsert-do-not-append) describes.

And the cycle around it, which decides what to call `sync` for at all:

```python
def poll(session, load_cursor, save_cursor, load_state, save_state, upsert):
    """One cycle. Returns the datasets it exported."""
    etag, tokens = load_state()          # etag None and tokens {} on the first run

    headers = {"If-None-Match": etag} if etag else {}
    r = session.get(f"{BASE}/datasets", headers=headers, timeout=60)

    if r.status_code == 304:
        return []                        # nothing changed anywhere: one request, done
    r.raise_for_status()

    datasets = r.json()
    exported = []

    for d in datasets:
        key, token = d["key"], d["changeToken"]
        if tokens.get(key) == token:
            continue                     # unchanged: skip the export entirely

        has_row_index = any(p["key"] == "row_index" for p in d["columns"])
        sync(key, session, load_cursor, save_cursor, upsert)

        # Only after the export succeeds. Storing it first would mark the dataset
        # caught up when it is not, and nothing would revisit it.
        tokens[key] = token
        exported.append(key)

    save_state(r.headers.get("ETag"), tokens)
    return exported
```

> [!WARNING]
> **Store a dataset's token only after its export succeeded**, as above. Storing the whole catalogue's
> tokens up front means a dataset whose export then fails is recorded as caught up, and the next cycle
> skips it.

> [!WARNING]
> **Store the ETag only if the whole cycle succeeded.** Above, a failed export raises out of `poll`
> before `save_state` runs, which is what makes it safe. If instead you catch per-dataset failures and
> carry on, you must not store the ETag: the next cycle would send it, get a 304, and never come back
> for the dataset that failed — its unstored token would never even be looked at.

### Operational advice

- **Store the cursor durably**, not in memory. It is the only state your integration needs, and it
  makes a crash mid-backfill cheap.
- **One sync at a time per dataset.** Concurrency is capped per organization (see [Rate
  limits](docs.html#errors-limits-and-versioning--rate-limits)), and a streaming export holds a database
  connection while it runs. Fanning out earns 429s and finishes later than running datasets in turn.
- **Retrying a page is safe.** The cursor only advances once you have the rows, so a retry fetches the
  same page again rather than skipping one. Bound the retries all the same.
- **Use a generous client timeout and a modest `limit`.** A 5000-row page of a wide form is a
  large response. Prefer a smaller `limit` to a short timeout.
- **Reuse one connection for the whole sync.** A page fetched on a fresh connection pays for the
  TCP and TLS handshake as well as the data, which is on the order of 150 ms of pure latency —
  comfortably more than a small page costs to serve. Over a backfill of thousands of pages that is most
  of your wall-clock time. Use a session or connection-pooling client (`requests.Session`, a shared
  `HttpClient`) rather than a new client per call, and keep `limit` large enough that you are not
  paying the round trip for a handful of rows.

## Errors, limits and versioning

Everything that is not a 200: what can go wrong, which failures are worth retrying, how throttling
works, and what may change without notice.

### Error responses

Errors are `application/problem+json` with a status code, a `title`, and an `errorCode`. Most also have a `detail` describing the specific case; a `401` has none. A parameter that cannot be parsed at all (a malformed timestamp, a non-numeric `limit`) is rejected before the checks below, with the same fields plus an `errors` object naming each parameter at fault. A path that matches no endpoint at all (for example `/v1/v1/...` from resolving `links.csv` against the base URL) gets a plain 404 titled *Not Found*, not *Dataset not found*.

**Branch on the status code and the `errorCode`.** `errorCode` is a short upper-case token — `INVALID_CURSOR`, `RATE_LIMITED`, `DATASET_NOT_FOUND`, `AUTH_UNAVAILABLE`, `DATABASE_UNAVAILABLE`, `DATASET_NOT_EXPORTABLE` and a handful more — and it is on **every** error this API returns, including the ones the framework writes for a bad route or method. It is the one field in the body written for a program: within `/v1` a code never changes meaning and never disappears, and the full set is published in the [OpenAPI document](docs.html#overview--the-openapi-document). New codes may be added, so treat one you do not recognise by its status code alone rather than failing on it.

**Do not branch on `title` or `detail`.** Both are prose, written for whoever reads your support ticket, and either may be reworded at any time — a reworded title turns a failure your client used to recognise into one it does not. The same goes for matching a title in a log alert. Where the table below names a title, it is naming the case for you to read, not a string to compare.

The table below covers the two endpoints. One more 503 can answer any request to the API, including these two: **503 *Database unavailable*** (`DATABASE_UNAVAILABLE`). It is transient, and [The three 503s](docs.html#errors-limits-and-versioning--the-three-503s-tell-them-apart-by-errorcode) covers all three together.

| Status | Title | When |
|---|---|---|
| 400 | Invalid parameter | limit is below 1 or above 5000 (rejected, not reduced). |
| 400 | Invalid parameter | afterRowIndex is less than 1. |
| 400 | Invalid parameter | afterRowIndex is sent for a dataset that has no row_index, i.e. not a grid/child dataset. |
| 400 | Invalid cursor | cursor is sent together with any of modifiedSince, after or afterRowIndex. |
| 400 | Invalid cursor | The cursor is the right shape but does not check out: truncated or altered before being sent back. |
| 400 | Invalid cursor | The cursor was issued by a version of the export this API no longer reads. |
| 400 | Invalid cursor | The cursor was issued for a different dataset than the one being exported. |
| 400 | Invalid cursor | The dataset has been regenerated with a different row shape since the cursor was issued. |
| 400 | Invalid cursor | The cursor value is not a Provision export cursor at all, including an empty cursor parameter. |
| 400 | Invalid cursor | Only one of modifiedSince and after is sent; they go together or are both omitted. |
| 400 | Invalid cursor | afterRowIndex is sent without after. |
| 400 | Missing required parameter | A grid/child dataset is paged with modifiedSince and after but without afterRowIndex. |
| 404 | Organization not configured | The calling organization has not been set up for API integration, so no datasets resolve for it. |
| 404 | Dataset not found | The requested dataset key does not exist for the caller's organization. |
| 503 | Dataset cannot be exported | The dataset cannot serve an export in its current shape; retrying does not clear it. |
| 503 | Authentication temporarily unavailable | Your key could not be verified because our identity provider did not respond. Not a credential problem — honour `Retry-After` and retry. |

**Errors carry a `traceId`. Quote it.** It identifies that one request in our logs, and a support
ticket with one can be answered. Log it with your own failure. One body has none — the **429** — so
read the field defensively; without one, quote the time and your organization instead.

**Which to retry:**

| Status | Retry? | What to do |
|---|---|---|
| 400 | No | Malformed request. Fix the caller; cursor rules are in [Incremental sync](docs.html#incremental-sync). |
| 401 | No | A bad, revoked or expired credential, or a key not tied to a known organization. Never transient: alert on the first one. Usually the key; see [Authentication](docs.html#authentication) for the case a new key will not fix. A transient verification problem on our side is a 503, never a 401. |
| 404 on an export, while `/v1/datasets` still answers | No | That dataset key no longer exists, usually because it was renamed. Re-read `/v1/datasets` to find the new key, and start it from the first page: a stored `cursor` is bound to the old key and will be refused. See [Renames](forms.html#form-changes--renames). |
| 404 on `/v1/datasets` itself | No | Your organization is not set up for API access. Contact support; nothing on your side will clear it. |
| 429 `RATE_LIMITED` | Yes | Throttled. Wait `Retry-After` seconds and repeat the same request. Which of the two limits you hit is in `RateLimit-Limit`; see [The 429 contract](docs.html#errors-limits-and-versioning--the-429-contract). |
| 503 `AUTH_UNAVAILABLE`, with `Retry-After` | Yes | We could not verify your key because our identity provider did not respond. Your key is fine: wait `Retry-After` seconds and retry. |
| 503 `DATABASE_UNAVAILABLE`, no `Retry-After` | Yes, with backoff | A database operation failed momentarily. Back off and retry; alert only if it persists. |
| 503 `DATASET_NOT_EXPORTABLE`, no `Retry-After` | No | The dataset is out of date on our side, or one stored value in it cannot be exported. Retrying does not change either. Republishing the form rebuilds a form dataset and may clear the first; correcting the value in Provision clears the second; otherwise contact support, naming the dataset. |
| 5xx (other) | Yes, with backoff | Treat as transient: back off exponentially, and alert if it persists. |

### When more than one thing is wrong

Two rules decide which error you get. Neither depends on the order of your query string.

**A parameter that cannot be parsed is rejected first, before the dataset is looked at.** `limit=abc`,
`after=DSC-1041`, `modifiedSince=yesterday` and `afterRowIndex=1.5` all fail while the request is
read, so you get a 400 naming every parameter at fault, even for a dataset you cannot export or do
not have.

**After that, a dataset that cannot be served wins over a parameter that is merely wrong.** If the
dataset is not yours (404), or is already known to be out of date (503), that is the answer, whatever
else is wrong: a 400 about your cursor would send you debugging your own code when the dataset needs
attention. A 503 found only while reading rows (a value that cannot be exported) comes after the
parameter checks, so fix any 400 first.

Beyond those two rules, **do not depend on which error you get.** Break two rules with parameters
that do parse and you hear about one of them, and which one is not part of the contract. Fix it and
call again.

### The three 503s: tell them apart by errorCode

The data endpoints send a 503 for three unrelated reasons. **Two are transient and worth retrying. The
third, `DATASET_NOT_EXPORTABLE`, is the one retrying cannot clear.** Only one of the three has
`Retry-After`, so the header alone cannot tell them apart — and neither can the `title`, which is
prose we may reword. **Switch on `errorCode`.** (`/health/ready` has its own transient 503, outside
this rule; see [Authentication](docs.html#authentication).)

**503 `AUTH_UNAVAILABLE` (*Authentication temporarily unavailable*), with `Retry-After`.** We could not verify your key
because our identity provider did not respond. It is a blip on our side, deliberately not a 401: a
401 would tell you to replace a key that is fine. It affects every request while it lasts,
including `/v1/datasets`. Wait, retry, and alert only if it keeps happening.

**503 `DATABASE_UNAVAILABLE` (*Database unavailable*), without `Retry-After`.** A database operation behind the request failed:
a statement timeout, a dropped connection, the database briefly out of resources. It is **transient
and worth retrying**, as its `detail` says: *"The request could not be completed because a database
operation failed. Please try again."* No interval is advertised, so back off exponentially yourself,
and alert only if it persists across several attempts.

**503 `DATASET_NOT_EXPORTABLE` (*Dataset cannot be exported*), without `Retry-After`.** The dataset cannot be exported as it
stands, and repeating the request changes nothing, which is why no `Retry-After` is sent. It has two
causes:

- **The dataset is out of date or unreadable on our side.** It predates a change to what exports
  require and has not been rebuilt since, or it cannot be read for a structural reason.
- **One stored value cannot be exported.** One cell in one row defeats its column. The rest of the
  dataset is fine, but the export stops at that row. You see this 503 only when that row comes early
  enough to still send an error; further into a large export, the download aborts instead, as
  [The loop](docs.html#incremental-sync--the-loop) describes.

What clears it depends on the dataset. A **form dataset** is stored as a pre-built copy that a
background job keeps up to date as submissions and the form change. A rebuild replaces the whole dataset or leaves the previous one
untouched, so a rebuild in progress never causes this 503, and republishing the form can clear the
first cause. A **resource dataset** and `deviations` are read live from your data at the moment you
export them, so there is nothing to republish for those — but a stored value they cannot render shows
up here rather than in a failed rebuild, which is the second cause. For either family, correcting the
offending value in Provision clears it; anything else needs a fix from us. Treat it as an alert, not a
backoff: **contact support**, naming the dataset from the `detail`.

**The response only ever names one dataset, but the cause can be wider than one.** Keep syncing the
others rather than stopping, and then check them: a configuration problem that stops one form dataset
being rebuilt usually stops the rest of that organization's too, and they will go stale without
returning an error at all. [Record and deviation
codes](forms.html#csv-columns--record-and-deviation-codes) has the case we meet most, a site whose time zone
is blank.

The `errorCode` matters most for the last two: both are 503s without `Retry-After`, so a client that
stops retrying on "503 without `Retry-After`" gives up on a database blip that would have cleared.

In all three cases your cursor is unaffected and nothing is lost: resume from where you were.

### Rate limits

Limits apply **per organization**, across every key you hold, so more keys do not raise them. There
are two:

- a **concurrency cap** on requests in flight at once. You will meet this one first: a streaming export
  holds its connection until it finishes, so a handful of parallel pulls reaches it.
- a **requests-per-minute** backstop, high enough that a well-behaved sequential client never gets near
  it.

Current values:

| Limit | Value |
|---|---|
| The most requests you may have in flight at once, per organization. | `5` |
| The most requests your organization may make in any 60-second window. | `600` |

The limits are the same for every customer: counting is per organization, but we cannot raise the
limits for one customer. They are current operating values, not guarantees, so do not hardcode them.

You only see them on a **429**. Successful responses carry no rate-limit headers, so there is no way
to check your remaining budget in advance: design for the 429.

**A 429 from the concurrency cap does not mean a slot frees in `Retry-After` seconds.** There, the
`Retry-After` and `RateLimit-Reset` values in the table below are a short polling interval rather
than a real reset; a slot frees only when one of your organization's
in-flight requests finishes, which for a large export can take minutes. Repeated 429s usually mean
your own parallel requests are still running, so reduce them rather than retrying harder.

There is also a limit per source IP address in front of the service, far above the organization limits.
It counts all traffic from one IP, so an integration that syncs many organizations from one address
shares it between them.

### The 429 contract

A 429 response carries these headers; successful responses do not:

| Header | Meaning |
|---|---|
| `Retry-After` | Seconds to wait before retrying; paging is resumable, so continue from the same cursor. |
| `RateLimit-Limit` | The limit you reached, so you can size your client to stay under it. |
| `RateLimit-Remaining` | Requests still allowed right now; zero on a rejection, so wait before sending more. |
| `RateLimit-Reset` | Seconds until capacity is expected to be available again; treat it as the earliest retry time. |

The three `RateLimit-*` headers appear only on a 429. `Retry-After` also appears on the *Authentication temporarily unavailable* 503, with the same meaning: wait that long, then retry. The other two **503**s — *Database unavailable* and *Dataset cannot be exported* — never send it, so its absence does not tell you whether retrying is worthwhile (see [The three 503s](docs.html#errors-limits-and-versioning--the-three-503s-tell-them-apart-by-errorcode)).

**`RateLimit-Limit` is what tells the two limits apart, and they need different responses.** Both are
429s with the same `errorCode` and the same headers; only the value differs. Compare it with the two
values in [Rate limits](docs.html#errors-limits-and-versioning--rate-limits):

- **The concurrency cap.** You have too many requests open at once. A slot frees only when one of your
  own in-flight requests finishes, and an export holds its slot for as long as it takes to stream —
  seconds to minutes. So `Retry-After` is a polling interval here, not a reset: waiting it and
  retrying is right, but expect to poll, and if it keeps happening the fix is **fewer requests in
  flight**, not more patience. Retrying harder cannot help: you are competing with yourself.
- **The per-minute budget.** You are sending too many requests over time, not too many at once. Waiting
  `Retry-After` clears this one properly, and the fix is to **slow the whole sync down** — a longer
  gap between requests, or a larger `limit` so the same rows cost fewer calls.

A well-behaved client:

1. Sleeps for `Retry-After` seconds, and does not retry sooner.
2. Resumes from the **same cursor**. Nothing was consumed, so nothing is lost.
3. Keeps one request in flight per dataset, and syncs datasets one after another.

Paging is resumable, so throttling costs you time and nothing else: no partial state to reconcile, no
backfill to restart.

### Versioning and deprecation

The `/v1` in every data path is a compatibility promise about the **API surface**: the endpoints,
their query parameters, the JSON field names and values `/v1/datasets` returns, the status codes, and
the CSV dialect. It is not a promise about your *data*: your CSV columns follow the forms and resources
your organization configures, and move when you change them.

**Within `/v1` we may:**

- add endpoints, and add fields to JSON responses;
- add optional query parameters that default to today's behaviour;
- add a value to the `dataType` vocabulary, when a column gains a type we can describe more
  precisely than the one it reports today;
- add datasets to the listing. Deleting a form does **not** remove its dataset; see [What happens when a
  form is deleted](docs.html#datasets--what-happens-when-a-form-is-deleted). A key can still leave the
  listing one way: **renaming** a form or grid moves its dataset to a new key and retires the old one.
  That follows your own configuration, not ours, and it is the usual reason a key you sync starts
  returning 404; see [When someone edits the form](forms.html#form-changes);
- add **CSV columns**. Those follow your forms, not our API: a column appears when someone adds a field.
  Removing a field does *not* remove its column, and relabelling a field does not rename it, though
  retyping one changes which columns it exports; see [When someone edits the
  form](forms.html#form-changes);
- retune the rate limits, which are published as current values, not guarantees.

**Within `/v1` we will not:**

- remove or rename an endpoint, a query parameter or a JSON field;
- change the meaning or type of an existing JSON field;
- change the meaning of a `dataType` value, or remove one. The vocabulary in [Column
  metadata](docs.html#datasets--column-metadata) only ever grows, and anything without a narrower type is
  `text`;
- change the CSV dialect: delimiter, quoting, encoding and byte-order mark, line endings, timestamp,
  date or boolean rendering, or null versus empty string;
- change the ordering or the cursor semantics [Incremental sync](docs.html#incremental-sync) depends on;
- change which organization a key belongs to, or what it can reach.

`localDateTime` joined the `dataType` vocabulary inside `/v1`, and another value may follow the same
way: appended, never replacing one. **Do not treat the list as exhaustive** — a generated client with
a closed enum over it breaks on the next addition. Read a `dataType` you do not recognise as
`text`; see [Column metadata](docs.html#datasets--column-metadata).

**Write your client against the first list.** Ignore JSON fields you do not recognise, read CSV
columns by header name, and expect the column set to grow. A client that does those three things
survives everything we have reserved the right to do.

### What counts as a breaking change

The lists above are about names and shapes. A third kind of change breaks integrations without
touching either, and we have made one before.

**A value can change format inside a column whose name and type stay put.** Record and deviation codes
were renumbered part-way through the life of the data. The columns are still `record_id` and `code`,
reported and exported the same way, but rows from before and after the change have different formats, old rows were never
rewritten, and nothing marks which is which. A parser that learned the old format on its first sync
met the new one without warning. [Record and deviation codes](forms.html#csv-columns--record-and-deviation-codes)
documents both formats and what to key on instead.

No signature moves when that happens, which makes it the easiest kind to get wrong, so we count it as
breaking. In full, a change is breaking if it:

- removes or renames anything in the *will not* list above;
- changes the **format of a value** in an existing column or field, even with the type unchanged: an
  identifier layout, a code scheme, a timestamp rendering;
- changes what an existing column **means** while its name and values look the same;
- tightens validation, so a request that used to succeed now gets a 400;
- changes which rows an export returns for a cursor you already hold.

A change that only **adds** is **not** breaking: a column because someone edited a form, an optional
query parameter, a JSON field, a dataset in the listing. Those follow your own configuration, as
[When someone edits the form](forms.html#form-changes) sets out. The reverse does not happen: a dataset is never withdrawn when
its form is deleted; see [What happens when a form is
deleted](docs.html#datasets--what-happens-when-a-form-is-deleted).

### How you will hear about a change

**Today, from us directly, or not at all.** The API sends no deprecation signal of its own: no
`Deprecation` or `Sunset` headers, no changelog endpoint, no version field in any response.

Two things are self-service, and both are narrower than they look:

- **`GET /v1/datasets`, compared with what you saw last time.** The useful one. It shows a new dataset,
  a new column, a changed `dataType`, and a key disappearing because a form or grid was renamed: the
  changes most likely to affect you. A column disappears only in the three cases in [When someone edits
  the form](forms.html#form-changes): a retyped field, an emptied grid or survey, and a nested grid's
  `<outer grid>_row_index` renamed with its outer grid. It does **not** show a dataset going out of service: a deleted form's dataset stays listed
  ([What happens when a form is deleted](docs.html#datasets--what-happens-when-a-form-is-deleted)). Worth
  doing on every sync.
- **The [OpenAPI document](docs.html#overview--the-openapi-document), compared the same way.** It shows endpoint,
  parameter and status-code changes. It cannot show a change to the CSV dialect, the cursor rules, or
  what a value means, because it does not describe them.

Neither tells you that a value's *format* changed, the case in [What counts as a breaking
change](docs.html#errors-limits-and-versioning--what-counts-as-a-breaking-change), and nothing else will
either. For that, **the technical contact on your account is the channel**, so keep that contact
current. If you need more than an email before you go live, say so early.

### Retiring a version

A breaking change ships under a **new prefix**, `/v2`, served alongside `/v1`. `/v1` keeps its
behaviour for the whole overlap: no v2 change is backported into it, and it is not retired without
telling you first. Moving becomes a path change you schedule, not an outage you discover.

> [!WARNING]
> **Two numbers are not set yet:** how long `/v1` keeps running once `/v2` exists, and how much
> notice you get before it stops. We would rather leave them blank than publish figures that are not
> agreed, because they are the part you would put in a contract. Confirm both with your Provision
> contact before they matter contractually; this page will carry them once they are settled.

In short: assume nothing about your columns, and rely freely on the endpoint shapes.

---

# Page: Forms to CSV (forms.html)

How the forms your organization builds in Provision become datasets and CSV columns.

## How your forms become CSV

Every form your organization publishes becomes datasets in this API. Its fields become columns, and a
repeating table inside it becomes a dataset of its own. Nobody at Provision defines those columns: they
follow the form as your team built it, and change when the form changes.

### A worked example

The example beside this text is a small form and the CSV it exports. The numbers match each field to
the columns it produces:

1. A **number** field becomes one column, named from the field's **key** (`water_temp_c`), not its
   label.
2. A **chemical select** becomes two columns: the chemical's name, and its id in `_id` for joining to
   `resources_chemicals`. Every resource select works this way.
3. A **checkbox group** (field type `selectboxes`) becomes one `t`/`f` column per option, named
   `field__option` — where `option` is the option's stored **value**, which is not always the label
   you see. Here they match; read the real names from `GET /v1/datasets`.
4. A **signature** becomes the signer's name in `_name` and the date stored with it in `_date`.
5. A **grid** (field type `datagrid`, `editgrid` or `provision-editgrid`) becomes a separate
   dataset, named from the form name and the grid's **label** (`form__grid`), with one row per grid
   row. Each row carries its submission's `id` to join on, and a `row_index`.
Every row also carries a fixed **envelope** of columns that come from the submission, not from a field.
A submission row has `id`, `record_id`, its status, the site and the timestamps; a grid row has a
smaller set: `id`, `row_index`, the site columns, the `last_modified` columns, and one
`<outer grid>_row_index` column per grid it sits inside. [CSV columns](forms.html#csv-columns) lists every
envelope column and every field type.
**Example: the form "Daily Sanitation Check", submitted at site Plant 2 · Packing.**

| # | Field label | Field key | Field type | Answer |
|---|---|---|---|---|
| 1 | Water temperature (C) | `water_temp_c` | number | 82.4 |
| 2 | Sanitizer | `sanitizer` | chemicalSelect | Quat 200 |
| 3 | Areas cleaned | `areas_cleaned` | selectboxes | ☑ Floor   ☐ Drains |
| 4 | Signed off by | `signed_off` | signature | Maria Lopez |
| 5 | Equipment checks | `equipment_checks` | editgrid | Slicer 3: Yes · Conveyor B: No |

It exports as two datasets. Columns are abridged (…), and their order is illustrative.

`daily_sanitation_check.csv`:

```csv
id,record_id,site_name,…,water_temp_c (1),sanitizer (2),sanitizer_id (2),areas_cleaned__floor (3),areas_cleaned__drains (3),signed_off_name (4),signed_off_date (4),last_modified
0191…a41c,26-195-0042,Plant 2 · Packing,…,82.4,Quat 200,7a1e…09bd,t,f,Maria Lopez,2026-07-14 08:12:40+00,2026-07-14 08:12:44.518+00
```

`daily_sanitation_check__equipment_checks.csv`:

```csv
id (5),row_index (5),site_name,…,equipment (5),clean (5),last_modified
0191…a41c,1,Plant 2 · Packing,…,Slicer 3,Yes,2026-07-14 08:12:44.518+00
0191…a41c,2,Plant 2 · Packing,…,Conveyor B,No,2026-07-14 08:12:44.518+00
```

### Submissions and grids

- Each **published form** becomes one **submission dataset**, with **one row per submission**, so `id`
  is unique within it. A form with several recording steps still has one row per submission: every
  step's fields are columns of that row. If the *same* field key appears on more than one step, the
  column holds one value, chosen in this order: a filled value beats an empty one; then the step
  completed most recently wins. So a field answered on one step and left blank on another exports the
  answer. **If that still leaves a tie**, the surviving value is **stable but arbitrary** — it is
  settled by the steps' internal row ids, which are generated together when the submission is created
  and carry no step order. It does not change between rebuilds, but you cannot predict which step it
  came from, so do not design around "the later step wins". Use one recording step per field key when
  it matters which answer you get.
- Each **grid** (repeating table) in the form becomes its own **grid dataset**, named `parent__grid`,
  with one row per grid row.

A grid cannot be flattened into one CSV row, which is why it gets its own dataset. A form with a grid
means two datasets for you to join.

Grid rows carry the `id` of the submission they were captured in, at every depth, so `id` is the
join key, and it is **not** unique within a grid dataset. Grid rows also carry `row_index`, which tells
rows of one submission apart and is needed to page them.

A submission contributes no grid rows when its grid is empty. Older or unusual submissions can also
store something other than a list of rows under a grid, and rather than fail the dataset the export
makes the best of it: rows stored as an object keyed by number export as normal rows, in number order;
a single object exports as one row, with values wherever its keys match the grid's fields; and a plain
value exports no rows. The submission itself exports normally, so a submission with nothing in the grid
dataset is expected, not a lost page.

A submission appears only **once a recording step has been saved**. One that was created but never
filled in has no row at all, not a row of empty fields. So your row count will not match the record
count staff see in the app, and `Pending` is a status you will rarely, if ever, see in an export.

### Nested grids

**A grid inside another grid becomes its own dataset too**, at any depth. Each level hangs off the one
directly above it, so a nested grid's `parentDataset` is the *outer grid's* dataset, not the form's.
Read the hierarchy from `parentDataset`, not from the key.

A nested grid's `row_index` counts all its rows within
the submission in one unbroken sequence, rather than restarting in each parent row, so `id` plus
`row_index` stays unique and it pages like any other dataset.

It also has **one extra column per grid above it**, `<outer grid>_row_index`, holding that grid's
position inside *its* parent. For a grid directly inside a top-level grid, that equals the outer grid
dataset's `row_index`, so join a nested row to the row it was captured under on the same `id` and
`<outer grid>_row_index` = outer `row_index`. That join holds even when some outer rows have an empty
nested grid.

**Only the join to the top-level grid holds.** An ancestor column is that ancestor's position inside
*its own* parent, while every dataset publishes `row_index` as a sequence across the whole submission.
Those agree only for a top-level grid. In a chain three or more deep, joining to an intermediate grid
attaches rows to the wrong parent without any error. If you need to rebuild a hierarchy that deep, tell
us: it needs a key we do not publish yet.

A parent row with an empty nested grid adds no rows to the nested dataset, just as a submission with an
empty grid adds none to that grid's dataset. A missing nested row is not a missing parent row.

## CSV columns

Your CSV is your form, flattened.

Every column of a form dataset is either part of a small fixed **envelope** (which submission, which
site, when it last changed) or comes from a field your organization put on the form. A submission
dataset and a grid dataset have different envelopes; fields work the same way in both.

This model is for **form datasets only**. [Resources](resources.html#resources) and
[deviations](resources.html#deviations) have fixed columns of their own, listed on their page.

Three things here apply to every dataset, not just form datasets: the [CSV
dialect](forms.html#csv-columns--reading-the-values), the rule that deleted records stop being exported, and,
for `deviations`, the [record and deviation codes](forms.html#csv-columns--record-and-deviation-codes).

**Every CSV also ends with a `_cursor` column**, on every dataset including resources and deviations.
It is added by the export rather than stored on the dataset, and it holds this row's paging position —
see [The loop](docs.html#incremental-sync--the-loop). It is listed in `GET /v1/datasets` alongside the real
columns, so a loader built from that listing has somewhere to put it, and its value is opaque: store the
string, do not model it. Nothing else in the export starts with an underscore, so you can drop it on
that alone if you would rather not keep it.

The order of the dataset's own columns is not part of the contract, and editing a form changes them; see
[When someone edits the form](forms.html#form-changes). `_cursor` is appended after all of them, which is why
it comes last — read it, like every other column, by header name rather than by position.

### Envelope columns

These come from the submission, not from form fields. They are the envelope of a **submission dataset**; a grid dataset has the smaller set in [Grid columns](forms.html#csv-columns--grid-columns), however deeply it nests. Beyond these and the form fields, every CSV ends with one more column, `_cursor`, which the export appends and no dataset stores.

| Column | Label | Type | Meaning |
|---|---|---|---|
| `id` | ID | `uuid` | Unique identifier of this submission. |
| `record_id` | Record ID | `text` | The record code identifying the submission, unique within your organization. |
| `site_name` | Site | `text` | Name of the site the submission was recorded against. |
| `site_id` | Site - ID | `uuid` | Unique identifier of the site the submission was recorded against. |
| `site_path` | Site - Path | `text` | The site's full ancestor chain as text, so matching one ancestor selects that whole site subtree. |
| `status` | Status | `text` | Status of the submission. |
| `is_archived` | Archived | `boolean` | Whether the submission has been archived. Stored separately from status. |
| `started_at` | Started At (UTC) | `dateTime` | When the submission was started, in UTC. |
| `started_at_local` | Started At (Site Local) | `localDateTime` | When the submission was started, in the site's own time zone; UTC when the site has none set. |
| `completed_at` | Completed At (UTC) | `dateTime` | When the submission was completed, in UTC. |
| `completed_at_local` | Completed At (Site Local) | `localDateTime` | When the submission was completed, in the site's own time zone; UTC when the site has none set. |
| `last_modified` | Last Modified (UTC) | `dateTime` | The later of when the submission itself changed and when the entered data changed, in UTC. |
| `last_modified_local` | Last Modified (Site Local) | `localDateTime` | The same last-change time in the site's own time zone; UTC when the site has none set. |

Each envelope timestamp appears twice: in UTC, and as `*_local` in the site's own time zone. Compute
on the UTC column and use the local one for display: `*_local` values have no offset, so they are
ambiguous around daylight-saving changes. Timestamps that come from form fields have no `_local`
twin; they are UTC instants.

**`status` is one of five values:** `Pending`, `InProgress`, `Completed`, `Cancelled`,
`Rejected`. **`Archived` is never a `status`.** Archiving is tracked separately: `status` keeps
the status underneath (usually `Completed`), and `is_archived` says whether it was archived. The app
shows such a record as *Archived*, so read both columns to match what staff see.

**`site_path` is a chain of site *ids*, not names.** It lists the row's site and every site above it,
outermost first, joined by dots, ending with the row's own `site_id`. Matching one id anywhere in it
selects that site and everything beneath, which makes it useful for rolling up a subtree. Split on `.`
for the ancestors; it is not meant to be read by people.

**Key your site dimension on `site_id`, not `site_name`.** `site_id` never changes; `site_name` is
the current name, so a join on it splits a renamed site's history in two, silently.

Every form dataset, submission and grid alike, has `site_id`, `site_name` and `site_path`, and so
does `deviations`. Most resource datasets do not; see [Site, status and record
columns](resources.html#resources--site-status-and-record-columns).

> [!WARNING]
> **Deleted records stop being exported, silently.** This holds for *every* dataset: deleting a
> submission removes it from its submission dataset, its grid datasets and `deviations`, and deleting a
> customer, supplier, product, employee or any other resource removes it from that resource dataset.
> The row just stops appearing, and nothing says it went.

Because you only receive rows that still exist, a deletion is invisible to the cursor, and a record you
already loaded stays in your warehouse unless you look for it. If deletions matter, re-read the dataset
in full on a schedule (the same loop, with no cursor) and treat ids you no longer receive as deleted.
This matters most for resource datasets, because live submissions keep referring to resources that no
longer exist.

### Verification columns

If the form has a **verification step with fields on it**, these columns appear too, and each
verification field becomes a column prefixed `verification_`:

| Column | Label | Type | Meaning |
|---|---|---|---|
| `verified_at` | Verified At (UTC) | `dateTime` | When the submission's most recent verification was completed, in UTC. |
| `verified_at_local` | Verified At (Site Local) | `localDateTime` | The same verification time in the site's own time zone; UTC when the site has none set. |
| `verified_by` | Verified By | `text` | Display name of the person who last changed that verification. |

They depend on verification *fields*, not the step: a verification step with no fields adds none of
these columns, and neither does a form without one.

A verification can also change without the row's `last_modified` moving, which an incremental sync
will not show: re-read in full if you report on these columns.

Each row reflects only one verification, and an **in-progress** one takes precedence over a completed
one. A step saved but not finished shows an empty completion time and the in-progress editor's name,
hiding a verification that did complete. When reporting on verification, treat a verifier without a
completion time as *not yet verified*, not as missing data.

### Grid columns

Every grid dataset, at any depth, has this smaller envelope, plus one `<outer grid>_row_index` column per grid it sits inside; see [Nested grids](forms.html#forms-to-csv--nested-grids).

| Column | Label | Type | Meaning |
|---|---|---|---|
| `id` | ID | `uuid` | Unique identifier of the submission this row belongs to. |
| `row_index` | Row Index | `number` | Position of this row within its submission, starting at 1. |
| `site_name` | Site | `text` | Name of the site the submission was recorded against. |
| `site_id` | Site - ID | `uuid` | Unique identifier of the site the submission was recorded against. |
| `site_path` | Site - Path | `text` | The site's full ancestor chain as text, so matching one ancestor selects that whole site subtree. |
| `last_modified` | Last Modified (UTC) | `dateTime` | The later of when the submission itself changed and when the entered data changed, in UTC. |
| `last_modified_local` | Last Modified (Site Local) | `localDateTime` | The same last-change time in the site's own time zone; UTC when the site has none set. |
A grid row carries no `record_id`, status or start and completion times: reach them through the `id`
it shares with its submission.

### Field types

Most fields become exactly one column. These are the ones that do not, and they are where
integrations usually go wrong, so check your forms for them.

| Field type | Column name | Columns | Cell value | Type |
|---|---|---|---|---|
| `areaSelect` | `<field>` | 1 | The selected area's name, as snapshotted on the submission. | `text` |
| `areaSelect` | `<field>_id` | 1 | The selected area's id, for joining to resources_areas.id; blank when no valid id was stored. | `uuid` |
| `checkbox` | `<field>` | 1 | True or false as stored; blank when the stored value is neither. | `boolean` |
| `chemicalSelect` | `<field>` | 1 | The selected chemical's name, as snapshotted on the submission. | `text` |
| `chemicalSelect` | `<field>_id` | 1 | The selected chemical's id, for joining to resources_chemicals.id; blank when no valid id was stored. | `uuid` |
| `customSignature` | `<field>_name` | 1 | The signer's first and last name joined by a space; blank on both sides when nothing was signed. | `text` |
| `customSignature` | `<field>_date` | 1 | The date stored with the signature; blank unless it is written ISO-ordered and fully padded. | `dateTime` |
| `customerSelect` | `<field>` | 1 | The selected customer's name, as snapshotted on the submission. | `text` |
| `customerSelect` | `<field>_id` | 1 | The selected customer's id, for joining to resources_customers.id; blank when no valid id was stored. | `uuid` |
| `datetime` | `<field>` | 1 | The date and time entered; blank unless it is ISO-ordered and fully padded, starting with a four-digit year. | `dateTime` |
| `datetimepicker` | `<field>` | 1 | The date and time entered; blank unless it is ISO-ordered and fully padded, starting with a four-digit year. | `dateTime` |
| `documentSelect` | `<field>` | 1 | The selected document's name, as snapshotted on the submission. | `text` |
| `documentSelect` | `<field>_id` | 1 | The selected document's id, for joining to resources_documents.id; blank when no valid id was stored. | `uuid` |
| `employeeSelect` | `<field>` | 1 | The selected employee's first and last name joined by a space, as snapshotted on the submission. | `text` |
| `employeeSelect` | `<field>_id` | 1 | The selected employee's id, for joining to resources_employees.id; blank when no valid id was stored. | `uuid` |
| `equipmentSelect` | `<field>` | 1 | The selected equipment's name, as snapshotted on the submission. | `text` |
| `equipmentSelect` | `<field>_id` | 1 | The selected equipment's id, for joining to resources_equipment.id; blank when no valid id was stored. | `uuid` |
| `lotSelect` | `<field>` | 1 | The selected lot's code, not its name — lots carry a code rather than a name. | `text` |
| `lotSelect` | `<field>_id` | 1 | The selected lot's id, for joining to resources_lots.id; blank when no valid id was stored. | `uuid` |
| `number` | `<field>` | 1 | The number entered; blank when the stored text is not a plain signed decimal, optionally with an exponent. | `number` |
| `productCategorySelect` | `<field>` | 1 | The selected product category's name, as snapshotted on the submission. | `text` |
| `productCategorySelect` | `<field>_id` | 1 | The category's id, for joining to resources_product_categories.id; blank when no valid id was stored. | `uuid` |
| `productSelect` | `<field>` | 1 | The selected product's name, as snapshotted on the submission. | `text` |
| `productSelect` | `<field>_id` | 1 | The selected product's id, for joining to resources_products.id; blank when no valid id was stored. | `uuid` |
| `radio` | `<field>` | 1 | The chosen option's label, or the stored value when no option matches it. | `text` |
| `recipeSelect` | `<field>` | 1 | The selected recipe's name, as snapshotted on the submission. | `text` |
| `recipeSelect` | `<field>_id` | 1 | The selected recipe's id, for joining to resources_recipes.id; blank when no valid id was stored. | `uuid` |
| `select` | `<field>` | 1 | The chosen option's label, or the stored value when no option matches it. | `text` |
| `selectboxes` | `<field>__<option>` | one per option | True when that option was ticked, false when it was not, blank when the stored value is not true or false. | `boolean` |
| `signature` | `<field>_name` | 1 | The signer's first and last name joined by a space; blank on both sides when nothing was signed. | `text` |
| `signature` | `<field>_date` | 1 | The date stored with the signature; blank unless it is written ISO-ordered and fully padded. | `dateTime` |
| `supplierSelect` | `<field>` | 1 | The selected supplier's name, as snapshotted on the submission. | `text` |
| `supplierSelect` | `<field>_id` | 1 | The selected supplier's id, for joining to resources_suppliers.id; blank when no valid id was stored. | `uuid` |
| `survey` | `<field>__<question>` | one per question | That question's answer: the option's label, or the stored value when no option matches it or none are configured. | `text` |
| anything not listed above — the default branch | `<field>` | 1 | The stored value as plain text. Also where a survey with no questions, and a radio, select or selectboxes with no configured options, land. | `text` |

**Cell value** says what a value *means*. How it is written in the CSV is covered in [Reading the
values](forms.html#csv-columns--reading-the-values): for example, a boolean is `t` or `f`, never `true` or
`false`.

In the column names, `<field>` is the field's slugified key and `<question>` a survey question's.
`<option>` is different: it is the option's **stored value**, slugified — not its label. The two are
set separately in the form builder and often differ, so an option labelled "Floor" with the value
`opt1` gives you `areas_cleaned__opt1`, not `areas_cleaned__floor`. The option's label reaches the
export only as the column's `label`. **Take these column names from `GET /v1/datasets`**, never from
the labels you can see on the form.

A **grid** on a recording step is not in the table because it produces no column of its own: it becomes
its own dataset, at any depth. See [Nested grids](forms.html#forms-to-csv--nested-grids). A grid on a
verification step is not split out that way, so avoid putting one there.

- **"Blank" in the table above means nothing was recorded, not an empty cell.** A `_name` column joins a
  first and last name with a space, so an unsigned signature or an empty employee field exports a
  **single space**; see [Reading the values](forms.html#csv-columns--reading-the-values). Trim before testing
  for emptiness.
- **Choice fields store a value but export its label**, so the CSV reads the way staff see the form.
  One label is exported per stored value: options from every published version are merged on the
  **stored value**, and the newest published label for it wins, from the next rebuild on. An option
  only older versions had keeps its entry, so old rows still read correctly. Only a value that was
  never an option in any published version (usually imported or migrated data) passes through as it
  is. Because the label is what lands in the CSV, renaming an option changes the text of **old** rows
  too; key on something else if you need a value that never changes.
- **A checkbox group (`selectboxes`) becomes one boolean column per option**, so adding an option adds
  a column. A single `checkbox` is one boolean column.
- **A reference field exports the selected record, not a scalar.** A `select` that draws its options
  from a data source instead of a configured list stores the record that was picked, and unless the
  form narrows it to a single property, that whole record is what lands in the cell, as a JSON
  document: an object with the record's own fields, nested objects and all. Nothing summarises or
  shortens it, so these are typically the widest cells an export has — a single one can run to
  hundreds of thousands of characters. Read them as JSON rather than as text, and see
  [Reading the values](forms.html#csv-columns--reading-the-values) for what that costs your parser. The
  `*Select` types above are the ones that are not like this: they are listed because they export a
  name and an id instead.

### Reading the values

The CSV dialect is fixed, for every dataset:

| Aspect | Behaviour |
|---|---|
| Encoding | UTF-8, **with** a byte-order mark (`EF BB BF`) before the header row |
| Header row | Always present, always the first line |
| Delimiter | Comma |
| Quoting | Double quote; an embedded quote is doubled (`""`) |
| Line endings | LF (`\n`), not CRLF |
| Embedded newlines | Allowed inside quoted fields, so your parser must handle multi-line rows |
| Maximum field size | None. No cap is enforced on a field, and a single one can run to hundreds of thousands of characters: a reference field exports the whole selected record as JSON, in one cell. See [Field types](forms.html#csv-columns--field-types) |
| Timestamps | UTC, `YYYY-MM-DD HH:MM:SS[.ffffff]+00`, with a **space** between date and time, not the `T` of RFC 3339, and the offset written `+00`, not `+00:00`. The fraction has variable width: trailing zeros are trimmed, and it is left out on a whole second |
| `*_local` envelope timestamps | Same layout in site-local time, with **no** offset. These columns report `dataType` `localDateTime`; see [Column metadata](docs.html#datasets--column-metadata) |
| Dates | `YYYY-MM-DD`, never with a time or offset. These columns report `dataType` `date`; see [Column metadata](docs.html#datasets--column-metadata) |
| Booleans | `t` / `f`, not `true` / `false` |
| Numbers | Plain decimal, `.` separator, no thousands grouping |
| Null vs empty string | A null is a bare empty field; an empty string is a quoted `""` |

Five of these catch people out regularly:

1. **There is a byte-order mark.** Decode as `utf-8-sig` (Python) or strip the leading `EF BB BF`.
   Decoded as plain UTF-8, your first header becomes `\ufeffid` instead of `id`, and every lookup
   on it misses. Inside a paging loop that reads like a paging bug rather than a decoding one: the
   first column is `id`, so the cursor you build from the last row comes out empty and the next call
   is a 400. Fix the decoding, not the loop.
2. **Timestamps are not strict ISO 8601.** Three things about them defeat a strict parser: a space
   instead of the `T`, an offset written `+00` rather than `+00:00`, and a fractional part whose
   width varies from row to row and is absent on a whole second. Parse with an explicit format that
   allows all three, or with a lenient parser.
3. **Booleans are `t` and `f`.** A check for `"true"` reads every boolean as false, silently.
4. **Null and empty string differ** (bare versus quoted), but most CSV libraries treat them the same.
   If the difference matters, check that your parser keeps it.
5. **A field can be larger than your parser allows.** Python's `csv` raises at 131,072 characters,
   so raise the cap before the first read: `csv.field_size_limit(min(sys.maxsize, 2**31 - 1))`. The
   cap is a C long, so a bare `sys.maxsize` is itself an error on Windows. A spreadsheet is the
   quieter version of the same problem: Excel keeps 32,767 characters of a cell and drops the rest
   without saying so, so load these datasets with something else.

**A name column built from two parts is never empty.** The `<field>_name` column of a `signature` or
`customSignature`, and the name column of an `employeeSelect`, join a first and last name with a
space. The space stays even when both halves are missing, so an unsigned signature or empty employee
field exports as a **single space**: not blank, not `NULL`. Trim before testing for emptiness;
`WHERE signed_off_name <> ''` does not filter these out, and a "signed" count built that way is wrong.
The same goes for `contact_name` and employee `name` on resource datasets.

Every successful export also has `Content-Type: text/csv` and
`Content-Disposition: attachment; filename="<dataset>.csv"`, so a browser treats it as a download.
The body is streamed, so there is **no** `Content-Length` to size a buffer from or check completeness
against.

### Record and deviation codes

`record_id` is the short code staff see on a record in the app. It is on every **submission** dataset
(grid datasets do not have it; reach it through the `id` they share with their submission) and on
`deviations`, which also has `code`, the deviation's own code, and `linked_record_code`, the code
of the record linked to resolve it. These are the codes to quote to Provision users, and the columns
integrations most often mistake for keys. A resource dataset can have an unrelated `record_id` column
from a custom attribute; nothing in this section applies to that one.

**They are not join keys.** Every dataset's `id` is a uuid, and `id` (plus `row_index` on a grid
dataset) is what you upsert on; see [Incremental sync](docs.html#incremental-sync). Use codes for display
and for talking to your Provision users.

**A `linked_record_code` does not mean the deviation was resolved.** It is filled in as soon as a
record is linked as the intended resolution, before that record has resolved anything: the row shows
`status` `InProgress` and `is_resolved` `f`, and may stay that way indefinitely.
**`is_resolved` is the column that answers the question**; treat `linked_record_code` as "where to
look", not "done".

**The current formats** are a date part and a daily counter, with deviations prefixed `D-`:

| Column | Where it appears | Current format | Example |
|---|---|---|---|
| `record_id` | Every submission dataset, and `deviations` | `YY-DDD-NNNN` | `26-256-0001` |
| `linked_record_code` | `deviations` | `YY-DDD-NNNN` | `26-256-0037` |
| `code` | `deviations` | `D-YY-DDD-NNNN` | `D-26-256-0001` |

`YY` is the last two digits of the year and `DDD` the day of the year (`001` to `366`), both in the
time zone of the record's site, or in **UTC** when the site has no time zone set (a per-site setting
that is not always filled in). Near midnight those give different days, so a code's date part is the
site's calendar day, not yours, and not a reliable way to date a record: use `last_modified` or the
record's own date columns. A site with no time zone at all gets UTC in its `*_local` columns as well.
`NNNN` counts within your organization and restarts at 1 each day. Record
codes and deviation codes count separately, so the same number can appear in both on one day.

> [!WARNING]
> **A site whose time zone is set to blank space takes out the whole organization's export, not one
> column.** A code's date part falls back to UTC when the time zone is missing *or* blank, but the
> `*_local` conversion falls back only when it is empty — so a whitespace value is carried through as
> though it were a real time zone name and the conversion fails. That aborts the rebuild of every form
> dataset for that organization at once, so all of them stop advancing — and one that had never been
> built does not appear in the listing at all. `deviations` shares the conversion but is read live, so
> it fails when you export it instead, with a `DATASET_NOT_EXPORTABLE` 503 or an aborted download. The
> column is a per-site setting with nothing validating it, so if datasets stop advancing, or
> `deviations` stops exporting, check the time zone on your sites first — and tell us.

Four things follow, and each breaks a parser built from the examples above:

**Older rows have a different format, permanently.** Before these codes, record and deviation codes
were six-character strings of digits and capital letters without dashes, like `H7K3RQ`, from an
alphabet that leaves out the lookalikes `0`, `1`, `5`, `I`, `O` and `S` (nothing should
depend on that). They were never rewritten, so records and deviations from before the change keep them,
and a full sync returns both formats in the same column. An old code has no `-`, so **branch on the
format, not on a date**: treat a code without `-` as an opaque identifier. Record codes and deviation
codes changed at different times, so a record in the current format can have deviations in the old one,
and the other way round.

**A current code is not fixed width.** The counter is padded to four digits only while it fits. When a
day's counter passes 9,999 in one organization, the next record code is `26-256-10000`, and deviation
codes cross the same boundary on their own counter (`D-26-256-10000`). Anything assuming a fixed length
(11 characters for a record code, 13 for a deviation code) or slicing by position breaks that day,
without warning. Once you know a code has a `-`, split on `-` and take the last part as the counter.

**Codes are unique within your organization, not globally.** If one warehouse table holds several
organizations, put the organization in the key, or use `id`, which is unique everywhere.

**The numbering has gaps, and is not a sort order.** A day's counter can skip values, so a missing
number is not a missing record, and counts cannot check completeness. And because the counter is not
fixed width, sorting codes is not sorting by time: as strings, `26-256-10000` sorts **before**
`26-256-9999`. Order by `last_modified` instead.

## When someone edits the form

Your columns are your form's fields, so editing the form changes your columns. The change is
**eventual, not immediate**: exports are rebuilt in the background shortly after a form is published,
so expect a short delay before a new column appears.

Row data can lag the app slightly too: a submission saved seconds ago may not be in the very next
export. It keeps the `last_modified` it was saved with, not the time it became exportable, so it can
surface just behind a cursor you already hold — and a strict cursor never offers it again. Cover it as
[A row can arrive just behind your cursor](docs.html#incremental-sync--the-loop) describes, and do not treat
an export as a real-time read.

Build your consumer to cope with everything below: select known columns by header name, ignore columns
you do not recognise, and treat a missing column as empty rather than fatal.

### Read the header row

Columns come out in whatever order the dataset defines, so **column order is not part of the
contract**: adding a field to a form can move existing columns.

Parse by header name. A positional parser breaks the first time someone edits the form, and it breaks
by putting values in the wrong columns, which is far worse than failing outright.

**A long backfill can see the header change between pages**, when a form is published mid-way. Read the
header of every response; see [The loop](docs.html#incremental-sync--the-loop).

### Columns accumulate

**Columns accumulate.** A dataset's columns are the union of the fields of every version of the form ever
published, and publishing cannot be undone, so columns are added and in practice never removed. The one
exception is a retyped field: it exports only the columns its newest type produces. Two smaller ones
are below: emptying a grid or survey, and relabelling a grid that a nested grid joins on.

| Change to the form | Effect on the CSV |
|---|---|
| Add a field | A new column appears after the next rebuild. Existing rows have it empty. |
| Remove a field | The column **stays**, empty for new rows. |
| Rename a field's **label** | Column name unchanged. `label` follows the new label after the next rebuild. |
| Change a field's **key** | A new column appears and the old one **stays**. You get both. |
| Rename an option label | The exported text changes, for old rows too. See below. |
| Add a checkbox-group option | A new boolean column appears, at top level and inside grids alike. |
| Rename a **grid** | The dataset is **renamed in place**, keeping its rows. The old name is gone: it leaves `GET /v1/datasets` and its export returns 404. |
| Change a field's type | After the next rebuild the field exports as its new type: a single column changes type, and a type that exports different columns (a signature, a resource select, a checkbox group, a survey) replaces the old columns with new ones. See below. |

- **The column count only grows.** A long-lived form has a column for every field it ever had. Select
  the columns you need by header name.
- **A removed field's history is never dropped.** Fields and options removed in a later version keep
  their columns, because old submissions still have data in them. The exception is emptying a grid or
  survey completely: a published version with no fields or questions left in it drops the columns the
  older versions had. A *retyped* field is different; see
  [Changing a field type](forms.html#form-changes--changing-a-field-type).
- **Two field keys can share a column name**, and the second gets `_2`; see the [naming
  rules](docs.html#datasets--dataset-and-column-naming-rules).

### Renames

**Renaming a grid moves its dataset, and the old name stops working.** Rows, numbering and access
all carry over to the new name; nothing is copied or left behind. The old name disappears from
`GET /v1/datasets`, and its export returns **404**, not an empty result, so a loader pinned to it
stops with an error rather than going quietly stale.

**Dataset** names come from the form's name and grid *labels*: renaming a form or relabelling a grid
renames its dataset, and because a grid's key is built from its parent's, renaming a form moves every
grid dataset under it too. **Column** names come from field *keys*, so relabelling a field only changes its
`label`. The exception is a nested grid's `<outer grid>_row_index` column, named from the
outer grid's *label*: relabelling the outer grid renames the column you join on. Store the `key`
from `GET /v1/datasets`, re-read the list on a schedule, and alert when a key **vanishes**: that is
a rename.

> [!WARNING]
> **A stored `cursor` does not survive the rename.** A cursor is bound to the dataset it was issued
> for, so sending it to the new key is a 400 rather than a resume — even though the rows and their
> numbering carried over unchanged. Re-export the new key from the first page, or page it with
> `modifiedSince` and `after`, which carry no such binding and do survive a rename. Whichever you
> choose, you upsert on `id`, so re-reading costs time and nothing else.

**Renaming an option label rewrites history.** Options are merged on their stored value and the
current label is exported, so every row gets the new label, however old. If you need a choice that
never changes, key on something other than the exported text.

**A rewrite like that moves no row's `last_modified`**, so an incremental sync never offers the
changed rows again. Re-read the dataset in full after a form edit that changes what old rows export.

### Changing a field type

**Editing a field in place reaches the export from the next rebuild.** A field's label and type come
from the newest published version, so renaming it updates `label`, and retyping it changes the
column's type, not at the moment of publishing but shortly after. Retyping is the disruptive one:
values change shape under a consumer that already inferred a type. Prefer adding a new field.

**A retype that changes a field's column shape drops its old columns.** A field exports only the columns
its newest type produces. Turn a checkbox group into a radio and its `<key>__<option>` columns go,
along with their historical values, replaced by one `<key>` column; a resource select turned into
text loses its `<key>_id`. A field whose old and new types both export a single `<key>` column
(text, number, date and time, checkbox, radio, select) keeps its column.

**Keeping the column does not mean keeping the values.** A `radio` or `select` with options exports the
option's current label, while a text field exports the stored value, so retyping one into the other
rewrites every cell — old rows included — while the column's name and its published `dataType` both
stay `text`. Nothing in `GET /v1/datasets` marks that, so re-read the dataset in full after such a
retype.

**A retype empties cells that do not fit the new type.** The new type applies to every row, including
rows recorded under the old one, and a stored value it cannot represent exports as empty, not as
text. The usual case is a text field holding `n/a` or `TBC` becoming a number. Watch date and time
fields, where a plausible value can still come out empty: they read only ISO-style values with the
four-digit year first, `2026-02-13`, optionally followed by a time (`T` or a space, seconds and
fractions optional) and optionally an offset (such as `Z`, `+02:00`, `+0200` or `+02`). So
`02/13/2026`, `13/02/2026`, `2026-2-3` and `20260213` all export empty. That is deliberate:
`02/13/2026` is ambiguous, and an empty cell is better than a wrong guess. The submission itself is untouched and
still shows in the app; only the exported cell is empty. If a retyped column matters, check old rows
for empty cells before relying on it.

---

# Page: Resources (resources.html)

The resource and deviation datasets every Provision organization has, column by column.

## Resources

**Resources** are the reference data your organization keeps in Provision: suppliers, customers,
products, product categories, equipment, chemicals, employees, areas, lots, recipes and documents. Each
kind is one dataset in `GET /v1/datasets`, with a key that begins `resources_`, and each exports and
pages through the same endpoint and cursor rules as every other dataset.

Unlike form datasets, **their core columns are the same for every customer**, so this page lists them.
On top of those, every resource dataset except `resources_documents` adds a column for each custom
attribute your organization defines for that kind of resource; see [Custom attribute
columns](resources.html#resources--custom-attribute-columns). Read the exact list from `GET /v1/datasets`, and
treat this page as the meaning of each core column.

### What every resource dataset shares

- **`id`** is the resource's uuid, and the key to upsert on. The resource select columns in form
  datasets (`<field>_id`) join to it.
- **`last_modified`** is when the resource itself last changed, in UTC. It is the cursor column, as
  everywhere. No resource dataset has `*_local` timestamps.
- **The dataset has no `displayName`**, and only some of its fixed columns have a `label` — though
  every custom attribute column has one, the attribute's name. Fall back to `key`.
- **Deleting a resource removes its row**, silently; see [Deleted records stop being
  exported](forms.html#csv-columns--envelope-columns). Archiving does not remove a document; see
  [Documents](resources.html#resources--documents).
- **A name can be blank while the id beside it is filled.** `site_name`, `site_path`, `product_name`
  and the other columns naming a related record are read from that record as it stands now, so deleting
  it empties the name and leaves the id in place — deliberately, so the row itself is not dropped. A
  roll-up keyed on `site_path` silently loses those rows, and the id no longer joins to anything in the
  related dataset.
- **The CSV dialect is the same** as for every dataset; see [Reading the
  values](forms.html#csv-columns--reading-the-values).
- **Two columns join a first and last name with a space** (`contact_name` on `resources_customers`,
  `name` on `resources_employees`). When both parts are missing they export as a **single space**,
  not `NULL` or an empty string, so trim before testing for emptiness.

### Custom attribute columns

Every resource dataset except `resources_documents` adds one column per custom attribute defined for
that kind of resource, after the core columns. The column is named from the attribute's key, following
the [naming rules](docs.html#datasets--dataset-and-column-naming-rules), and a name that clashes with a core
column gets a numeric suffix. Its `label` is the attribute's name.

| Custom attribute type | Column type | Value |
|---|---|---|
| Number | `number` | The number |
| Date | `date` | The date |
| DateTime | `dateTime` | The date and time |
| Document, SingleDocument | `text` | The names of the linked documents, joined by a comma and a space |
| Select | `text` | The option's stored **key**, not the label users see in Provision, and not affected by renaming the option |
| MultiSelect | `text` | A JSON array of the stored option keys, sorted: `["gmp","haccp"]`. Not a comma-joined list — parse it as JSON |
| Any other type | `text` | The stored value |

**A value that does not fit its type exports as empty.** A Number attribute holding `N/A` or
`12 kg`, or a Date that is not written year first, exports as nothing, not as its original text. Only
that cell is emptied; the rest of the row and the export are unaffected. The value is usually still
intact in Provision and worth correcting there.

**A custom attribute can take a name that means something elsewhere.** What is reserved is the
dataset's own core columns **plus the four names the export owns** — `id`, `last_modified`,
`row_index` and `row_seq`, which is internal numbering the cursor is built on and never appears in a
CSV — and those four are reserved whether or not the dataset publishes a column of that name.
Everything else is fair game, so an attribute whose key is "Site Name" or "Record ID" on a resource
without those columns becomes a `site_name` or `record_id` column holding the attribute's
value, while one keyed "Row Index" gets a numeric suffix instead. See [Site, status and record
columns](resources.html#resources--site-status-and-record-columns).

### List columns

Where a resource links to many of something, the export flattens the links into one `text` column of
names joined by a comma and a space:

| Dataset | List columns |
|---|---|
| `resources_suppliers` | `products`, `parent_names` |
| `resources_customers` | `products` |
| `resources_products` | `customers`, `sites`, `suppliers`, `allergens` |
| `resources_chemicals` | `sites` |
| `resources_employees` | `teams`, `sites` |
| `resources_documents` | `sites`, `teams` |
| Every dataset with custom attributes | each **Document** and **SingleDocument** custom attribute |

They hold names, not ids, so use them for display, not joins. A name containing a comma looks like two
names, so splitting on `, ` is unreliable. To model the relationships themselves, use the datasets on
both sides.

**A `sites` list is not a site column.** It lists site *names*, and has nothing to do with the
`site_id` / `site_name` / `site_path` columns that identify the one site a form row belongs to. Do
not join it to a site dimension.

**List columns go stale in an incremental sync.** Renaming a product changes the `products` value of
every supplier and customer listing it, without moving their `last_modified`, so the cursor never
offers those rows again. Renaming a supplier does the same to the `parent_names` of every supplier
beneath it. Re-read resource datasets in full on a schedule if these columns matter; see
[Upsert, do not append](docs.html#incremental-sync--upsert-do-not-append).

### Site, status and record columns

Some column names mean one thing on form datasets and something else, or nothing reserved, here.

**Site columns.** Only three resource datasets have a site: `resources_lots`, `resources_areas` and
`resources_equipment`. Each has `site_id`, `site_name` and `site_path`, read the same way as a form
dataset's. On every other resource dataset those names are not reserved, so a column called
`site_name` or `site_path` there is a custom attribute, and joining it to your sites produces
nonsense.

**`status`.** On `resources_suppliers`, `resources_lots` and `resources_recipes` it is
that resource's own status, unrelated to a submission's `status`.

**`record_id`.** No resource dataset has one, so a `record_id` column on a resource dataset is a
custom attribute, and joining it to submission record codes produces nonsense.

**`row_index`.** Reserved here too. A custom attribute that slugifies to `row_index` does not take
the name: it gets a numeric suffix (`row_index_2`) instead, so no resource dataset publishes a
`row_index` column and `afterRowIndex` is rejected on all of them. Only grid datasets have one; see
[Terms these pages use](docs.html#datasets--terms-these-pages-use).

Check the columns `GET /v1/datasets` reports for the handful you rely on, rather than assuming either
way.

### Suppliers

Dataset `resources_suppliers`. Adds custom attribute columns.

| Column | Type | Meaning |
|---|---|---|
| `id` | `uuid` | Unique identifier of the supplier. |
| `name` | `text` | Name of the supplier. |
| `description` | `text` | Free-text description of the supplier. |
| `website_url` | `text` | Supplier's website address. |
| `status` | `text` | Status recorded for the supplier. |
| `market` | `text` | Market recorded for the supplier. |
| `type` | `text` | Type recorded for the supplier. |
| `street1` | `text` | First street line of the supplier's address. |
| `street2` | `text` | Second street line of the supplier's address. |
| `city` | `text` | City of the supplier's address. |
| `region` | `text` | State or region of the supplier's address. |
| `zip_code` | `text` | Postal code of the supplier's address. |
| `country` | `text` | Country of the supplier's address. |
| `products` | `text` | Products linked to this supplier, as names joined by a comma and a space. |
| `parent_names` | `text` | Parent suppliers this supplier sits under, as names joined by a comma and a space. |
| `last_modified` | `dateTime` | When this supplier was last changed. |
| `_cursor` | `text` | This row’s paging position, added by the export rather than stored. Opaque: send it back as `cursor` and do not parse it. See [the sync loop](docs.html#incremental-sync--the-loop) |

### Customers

Dataset `resources_customers`. Adds custom attribute columns.

| Column | Type | Meaning |
|---|---|---|
| `id` | `uuid` | Unique identifier of the customer. |
| `name` | `text` | Name of the customer. |
| `description` | `text` | Free-text description of the customer. |
| `website_url` | `text` | Customer's website address. |
| `phone_number` | `text` | Customer's phone number. |
| `street1` | `text` | First street line of the customer's address. |
| `street2` | `text` | Second street line of the customer's address. |
| `city` | `text` | City of the customer's address. |
| `region` | `text` | State or region of the customer's address. |
| `zip_code` | `text` | Postal code of the customer's address. |
| `country` | `text` | Country of the customer's address. |
| `contact_name` | `text` | Full name of the customer's contact, first and last name together. |
| `contact_email` | `text` | Email address of the customer's contact. |
| `contact_phone` | `text` | Phone number of the customer's contact. |
| `products` | `text` | Products linked to this customer, as names joined by a comma and a space. |
| `last_modified` | `dateTime` | When this customer was last changed. |
| `_cursor` | `text` | This row’s paging position, added by the export rather than stored. Opaque: send it back as `cursor` and do not parse it. See [the sync loop](docs.html#incremental-sync--the-loop) |

### Products

Dataset `resources_products`. Adds custom attribute columns.

| Column | Type | Meaning |
|---|---|---|
| `id` | `uuid` | Unique identifier of the product. |
| `name` | `text` | Name of the product. |
| `description` | `text` | Free-text description of the product. |
| `product_code` | `text` | Code identifying the product. |
| `product_type` | `text` | Product type: RawMaterial, WorkInProgress, FinishedGood or Packaging; anything else appears as its stored number. |
| `product_category_name` | `text` | Name of the category the product is in. |
| `allergen_status` | `text` | Allergen status recorded for the product. |
| `allergens` | `text` | Allergens recorded for the product, joined by a comma and a space; empty when none are recorded. |
| `default_shelf_life_quantity` | `number` | Default shelf life of the product, as a number counted in the unit below. |
| `default_shelf_life_unit` | `text` | Unit the default shelf life is counted in. |
| `customers` | `text` | Customers linked to this product, as names joined by a comma and a space. |
| `sites` | `text` | Sites linked to this product, as names joined by a comma and a space. |
| `suppliers` | `text` | Suppliers linked to this product, as names joined by a comma and a space. |
| `last_modified` | `dateTime` | When this product was last changed. |
| `_cursor` | `text` | This row’s paging position, added by the export rather than stored. Opaque: send it back as `cursor` and do not parse it. See [the sync loop](docs.html#incremental-sync--the-loop) |

### Product categories

Dataset `resources_product_categories`. Adds custom attribute columns.

| Column | Type | Meaning |
|---|---|---|
| `id` | `uuid` | Unique identifier of the product category. |
| `name` | `text` | Name of the product category. |
| `description` | `text` | Free-text description of the product category. |
| `parent_name` | `text` | Name of the category this category sits under. |
| `last_modified` | `dateTime` | When this product category was last changed. |
| `_cursor` | `text` | This row’s paging position, added by the export rather than stored. Opaque: send it back as `cursor` and do not parse it. See [the sync loop](docs.html#incremental-sync--the-loop) |

### Equipment

Dataset `resources_equipment`. Adds custom attribute columns.

| Column | Type | Meaning |
|---|---|---|
| `id` | `uuid` | Unique identifier of the equipment. |
| `name` | `text` | Name of the equipment. |
| `make` | `text` | Make of the equipment. |
| `model` | `text` | Model of the equipment. |
| `serial_number` | `text` | Serial number of the equipment. |
| `site_name` | `text` | Name of the site the equipment belongs to. |
| `site_id` | `uuid` | Unique identifier of the site the equipment belongs to. |
| `site_path` | `text` | Full position of that site in the site hierarchy, from the top level down. |
| `area_name` | `text` | Name of the area the equipment is in. |
| `last_modified` | `dateTime` | When this equipment was last changed. |
| `_cursor` | `text` | This row’s paging position, added by the export rather than stored. Opaque: send it back as `cursor` and do not parse it. See [the sync loop](docs.html#incremental-sync--the-loop) |

### Chemicals

Dataset `resources_chemicals`. Adds custom attribute columns.

| Column | Type | Meaning |
|---|---|---|
| `id` | `uuid` | Unique identifier of the chemical. |
| `name` | `text` | Name of the chemical. |
| `type` | `text` | Type recorded for the chemical. |
| `grade` | `text` | Grade recorded for the chemical. |
| `production_method` | `text` | Production method recorded for the chemical. |
| `sites` | `text` | Sites linked to this chemical, as names joined by a comma and a space. |
| `last_modified` | `dateTime` | When this chemical was last changed. |
| `_cursor` | `text` | This row’s paging position, added by the export rather than stored. Opaque: send it back as `cursor` and do not parse it. See [the sync loop](docs.html#incremental-sync--the-loop) |

### Employees

Dataset `resources_employees`. Adds custom attribute columns.

| Column | Type | Meaning |
|---|---|---|
| `id` | `uuid` | Unique identifier of the employee. |
| `name` | `text` | Full name of the employee, first and last name together. |
| `job_title` | `text` | Job title of the employee. |
| `access_level` | `text` | The employee's access level in the organization. |
| `teams` | `text` | Teams the employee belongs to, as names joined by a comma and a space. |
| `sites` | `text` | Sites the employee is assigned to, as names joined by a comma and a space. |
| `last_modified` | `dateTime` | When this employee was last changed. |
| `_cursor` | `text` | This row’s paging position, added by the export rather than stored. Opaque: send it back as `cursor` and do not parse it. See [the sync loop](docs.html#incremental-sync--the-loop) |

### Areas

Dataset `resources_areas`. Adds custom attribute columns.

| Column | Type | Meaning |
|---|---|---|
| `id` | `uuid` | Unique identifier of the area. |
| `name` | `text` | Name of the area. |
| `description` | `text` | Free-text description of the area. |
| `type` | `text` | Type recorded for the area. |
| `size` | `number` | Size of the area, counted in the unit below. |
| `size_unit` | `text` | Unit the area size is counted in. |
| `area_path` | `text` | Full position of the area in the area hierarchy, from the top level down. |
| `parent_name` | `text` | Name of the area this area sits under. |
| `site_id` | `uuid` | Unique identifier of the site the area belongs to. |
| `site_name` | `text` | Name of the site the area belongs to. |
| `site_path` | `text` | Full position of that site in the site hierarchy, from the top level down. |
| `last_modified` | `dateTime` | When this area was last changed. |
| `_cursor` | `text` | This row’s paging position, added by the export rather than stored. Opaque: send it back as `cursor` and do not parse it. See [the sync loop](docs.html#incremental-sync--the-loop) |

`area_path` is the **area** hierarchy, not the site one. It is built by the same rule as
`site_path` — the area's own id and every area above it, outermost first, joined by dots — so
whatever parses one parses the other. This dataset carries both: read `area_path` for the area tree
and `site_path` for the site tree, and do not join one to the other.

### Lots

Dataset `resources_lots`. Adds custom attribute columns.

| Column | Type | Meaning |
|---|---|---|
| `id` | `uuid` | Unique identifier of the lot. |
| `code` | `text` | Code identifying the lot. |
| `description` | `text` | Free-text description of the lot. |
| `status` | `text` | Status recorded for the lot. |
| `quantity` | `number` | Quantity held in the lot, counted in the unit below. |
| `unit` | `text` | Unit the lot quantity is counted in. |
| `expiry_date` | `date` | Date the lot expires. |
| `date_received` | `date` | Date the lot was received. |
| `date_produced` | `date` | Date the lot was produced. |
| `product_id` | `uuid` | Unique identifier of the product this lot is of. |
| `product_name` | `text` | Name of the product this lot is of. |
| `site_id` | `uuid` | Unique identifier of the site the lot belongs to. |
| `site_name` | `text` | Name of the site the lot belongs to. |
| `site_path` | `text` | Full position of that site in the site hierarchy, from the top level down. |
| `last_modified` | `dateTime` | When this lot was last changed. |
| `_cursor` | `text` | This row’s paging position, added by the export rather than stored. Opaque: send it back as `cursor` and do not parse it. See [the sync loop](docs.html#incremental-sync--the-loop) |

`expiry_date`, `date_received` and `date_produced` are `date` columns: a calendar day, with no time
or offset.

### Recipes

Dataset `resources_recipes`. Adds custom attribute columns.

| Column | Type | Meaning |
|---|---|---|
| `id` | `uuid` | Unique identifier of the recipe. |
| `name` | `text` | Name of the recipe. |
| `description` | `text` | Free-text description of the recipe. |
| `status` | `text` | Status recorded for the recipe. |
| `output_quantity` | `number` | Quantity the recipe produces, counted in the unit below. |
| `output_unit` | `text` | Unit the produced quantity is counted in. |
| `has_unpublished_changes` | `boolean` | True when the recipe has changes that have not been published. |
| `output_product_id` | `uuid` | Unique identifier of the product the recipe produces. |
| `output_product_name` | `text` | Name of the product the recipe produces. |
| `last_modified` | `dateTime` | When this recipe was last changed. |
| `_cursor` | `text` | This row’s paging position, added by the export rather than stored. Opaque: send it back as `cursor` and do not parse it. See [the sync loop](docs.html#incremental-sync--the-loop) |

### Documents

Dataset `resources_documents`. Adds **no** custom attribute columns, so its columns are the same for
every customer.

| Column | Type | Meaning |
|---|---|---|
| `id` | `uuid` | Unique identifier of the document. |
| `name` | `text` | Name of the document. |
| `type` | `text` | Type recorded for the document. |
| `about` | `text` | Short note on what the document is about. |
| `version` | `text` | Version label of the document's currently published version. |
| `is_archived` | `boolean` | True when the document is archived. |
| `sites` | `text` | Sites linked to this document, as names joined by a comma and a space. |
| `teams` | `text` | Teams linked to this document, as names joined by a comma and a space. |
| `last_modified` | `dateTime` | When this document was last changed. |
| `_cursor` | `text` | This row’s paging position, added by the export rather than stored. Opaque: send it back as `cursor` and do not parse it. See [the sync loop](docs.html#incremental-sync--the-loop) |

**Archived documents keep exporting.** Archiving a document leaves its row in place with
`is_archived` `t`, so filter on that column for a list of current documents. It says only whether the
document is archived: a draft, one in review and a rejected one all export with `is_archived` `f`,
like a published document.

`resources_documents` is the only resource dataset with such a column. Others export their rows
whatever state they are in — an archived employee is in `resources_employees` with nothing marking it
— and lose a row only when the resource is deleted.

## Deviations

A **deviation** is a value flagged as out of spec on one field of one submission. The `deviations`
dataset has one row per deviation: the field's key and label, the recorded value, a severity, and how,
or whether, it was resolved, alongside the submission, step and site it came from.

Its columns are the same for every customer: it takes no custom attributes, and has no `resources_`
prefix. Deviations come from your forms, so the dataset moves with your submissions, and deleting a
submission removes its deviations.

### Columns

| Column | Type | Meaning |
|---|---|---|
| `id` | `uuid` | Unique identifier of the deviation. |
| `code` | `text` | Code identifying the deviation. |
| `severity` | `text` | Severity recorded for the deviation. |
| `status` | `text` | Deviation status: Resolved, InProgress or Open. |
| `is_resolved` | `boolean` | True when the deviation has been resolved. |
| `resolution_type` | `text` | How it was resolved: LinkedResolutionRecord or Comment; empty while unresolved. |
| `field_key` | `text` | Identifier of the form field the deviation was raised on. |
| `field_label` | `text` | Label of the form field the deviation was raised on. |
| `display_value` | `text` | The value recorded in that field. |
| `row_id` | `text` | Identifier of the grid row the deviation was raised on, for fields inside a grid. |
| `workflow_name` | `text` | Name of the form; kept for existing dashboards and superseded by form_name. |
| `form_name` | `text` | Name of the form the deviation was raised on. |
| `submission_id` | `uuid` | Unique identifier of the submission the deviation was raised on. |
| `record_id` | `text` | Code identifying the record the deviation was raised on. |
| `submission_status` | `text` | Status of the submission the deviation was raised on. |
| `is_archived` | `boolean` | True when the record holding the deviation is archived. |
| `step_name` | `text` | Name of the form step the deviation was raised on. |
| `step_status` | `text` | Status of that step. |
| `site_name` | `text` | Name of the site the submission belongs to. |
| `site_id` | `uuid` | Unique identifier of the site the submission belongs to. |
| `site_path` | `text` | Full position of that site in the site hierarchy, from the top level down. |
| `assigned_to` | `text` | Who the deviation is assigned to: the assigned person's name, or the assigned team's name when no person is set. |
| `assigned_user_name` | `text` | Name of the person the deviation is assigned to. |
| `assigned_team_name` | `text` | Name of the team the deviation is assigned to. |
| `supplier_name` | `text` | Name of the supplier the deviation is attributed to. |
| `comment` | `text` | Comment recorded with the deviation. |
| `resolution_comment` | `text` | Comment recorded when the deviation was resolved. |
| `resolved_by` | `text` | Name of the person who resolved the deviation. |
| `resolved_at` | `dateTime` | When the deviation was resolved, in UTC. |
| `resolved_at_local` | `localDateTime` | When the deviation was resolved, in the site's time zone. |
| `signed_by` | `text` | Name of the person who signed the resolution. |
| `signed_at` | `dateTime` | When the resolution was signed, in UTC. |
| `signed_at_local` | `localDateTime` | When the resolution was signed, in the site's time zone. |
| `linked_record_code` | `text` | Code of the record linked as the deviation's resolution. |
| `linked_record_status` | `text` | Status of the record linked as the deviation's resolution. |
| `linked_record_source` | `text` | Source of the record linked as the deviation's resolution. |
| `linked_resolution_comment` | `text` | Comment recorded on the deviation about its linked resolution record. |
| `created_at` | `dateTime` | When the deviation was created, in UTC. |
| `created_at_local` | `localDateTime` | When the deviation was created, in the site's time zone. |
| `last_modified` | `dateTime` | When the deviation was last changed, in UTC. |
| `last_modified_local` | `localDateTime` | When the deviation was last changed, in the site's time zone. |
| `_cursor` | `text` | This row’s paging position, added by the export rather than stored. Opaque: send it back as `cursor` and do not parse it. See [the sync loop](docs.html#incremental-sync--the-loop) |

It borrows some columns from the submission, with the meanings [the envelope
columns](forms.html#csv-columns--envelope-columns) give them: `record_id`, `site_id`, `site_name`,
`site_path` and `is_archived`. Its `*_local` timestamps are its **own**, describing the deviation,
read the same way as a form dataset's. `id` is the deviation's own id.

`step_name` and `step_status` are the **recording** step the flagged field sits on. A deviation is
only ever raised on a recording step, so a verification step never appears here.

`form_name` and `workflow_name` are the **same value** — the name of the form the submission was made
against. *Workflow* is our older word for a form, and `workflow_name` is kept only so dashboards built
on it keep working; the two are filled from one expression and cannot differ. Build on `form_name`,
and expect `workflow_name` to be retired in a later version.

`comment` is a standalone note on the deviation, which can be added, changed or removed at any time —
not a record of what was said when the deviation was raised. `linked_record_source` says where the
linked resolution record came from; treat its values as opaque unless we have told you otherwise.

### Joining a deviation to its submission

**Join on `submission_id`.** It holds the same uuid as the submission dataset's `id`, and it
is the only column that makes that join. `record_id` beside it is the human-readable record *code*:
good for display, not the submission dataset's key.

**`row_id` cannot reach a grid row.** It is set when the deviation was raised on a field inside a
grid, but **no grid dataset publishes a matching id**: a grid row is identified by `id` plus
`row_index`, and `row_index` is a position. So `row_id` tells you *that* the deviation came from a
grid row, not *which* one. Treat it as an opaque marker.

### Status and resolution

- **`status` is the deviation's state:** `Open`, `InProgress` or `Resolved`. The submission's own
  status is `submission_status`. **Every change to it moves `last_modified`**, so an incremental sync
  offers the row again: linking a record sets a column on the deviation row, and resolving writes its
  resolution to a table of its own *and* restamps the deviation, so `Open` → `InProgress` →
  `Resolved` all reach you. That holds however it was resolved — with a comment, by editing that
  comment, or by completing the linked record. Other columns are less well behaved; see [Keeping
  deviations current](resources.html#deviations--keeping-deviations-current).
- **`is_resolved` is the column that says whether it was resolved.** A `linked_record_code` is set as
  soon as a record is linked as the intended resolution, before it resolves anything: such a row is
  `InProgress` with `is_resolved` `f`, possibly indefinitely. See [Record and deviation
  codes](forms.html#csv-columns--record-and-deviation-codes).
- **A resolution can carry a signature**, given when the deviation is resolved with a comment. That is
  where `signed_by`, `signed_at` and `signed_at_local` come from. They are empty for a resolution
  without a signature, including one completed through a linked record, and an unresolved deviation
  never has them.
- `code` is the deviation's own code, in its own format and on its own counter; see [Record and
  deviation codes](forms.html#csv-columns--record-and-deviation-codes).

### Keeping deviations current

**A deviation's `last_modified` only follows the deviation row itself.** Resolution is the exception
worth knowing about, and it is handled: resolving writes to a table of its own *and* restamps the
deviation, so `status`, `is_resolved`, `resolution_type`, `resolution_comment`, `resolved_by` and
the `resolved_at` and `signed_*` timestamps all reach an incremental consumer. A deviation does not
keep showing as open after someone resolved it.

What still escapes the cursor are the columns the deviation only *displays*: the submission, step and
site columns, the linked record's `linked_record_status`, the names of the employee, team and
supplier, and the standalone `comment`, which is stored beside the deviation rather than on it. Those
belong to other records, and changing one of those records moves its own `last_modified`, not the
deviation's. If you report on them, re-read `deviations` in full on a
schedule, as [Upsert, do not append](docs.html#incremental-sync--upsert-do-not-append) describes for resource
list columns.

**A person's name can also be missing where a person really is set.** `assigned_user_name`,
`resolved_by` and `signed_by` are read from your employee directory, so someone with no live employee
record there — a Provision Admin, or an employee since deleted — leaves them empty on a deviation that
was assigned, resolved or signed. `assigned_to` then falls back to the **team's** name, so such a row
reads as team-assigned although a person is assigned. Read `assigned_user_name` and
`assigned_team_name` rather than inferring which of the two `assigned_to` holds, and treat an empty
`resolved_by` on a resolved deviation as an unresolvable person, not as an unresolved deviation.