Forms to CSV / 00 Overview

How your forms become CSV

Every form your organization publishes becomes datasets in this API. Its fields become columns, and a repeating table inside it becomes a dataset of its own. Nobody at Provision defines those columns: they follow the form as your team built it, and change when the form changes.

A worked example

The example beside this text is a small form and the CSV it exports. The numbers match each field to the columns it produces:

1. A number field becomes one column, named from the field's key (water_temp_c), not its label.
2. A chemical select becomes two columns: the chemical's name, and its id in _id for joining to resources_chemicals. Every resource select works this way.
3. A checkbox group (field type selectboxes) becomes one t/f column per option, named field__option — where option is the option's stored value, which is not always the label you see. Here they match; read the real names from GET /v1/datasets.
4. A signature becomes the signer's name in _name and the date stored with it in _date.
5. A grid (field type datagrid, editgrid or provision-editgrid) becomes a separate dataset, named from the form name and the grid's label (form__grid), with one row per grid row. Each row carries its submission's id to join on, and a row_index.

Every row also carries a fixed envelope of columns that come from the submission, not from a field. A submission row has id, record_id, its status, the site and the timestamps; a grid row has a smaller set: id, row_index, the site columns, the last_modified columns, and one <outer grid>_row_index column per grid it sits inside. CSV columns lists every envelope column and every field type.

Submissions and grids

• Each published form becomes one submission dataset, with one row per submission, so id is unique within it. A form with several recording steps still has one row per submission: every step's fields are columns of that row. If the same field key appears on more than one step, the column holds one value, chosen in this order: a filled value beats an empty one; then the step completed most recently wins. So a field answered on one step and left blank on another exports the answer. If that still leaves a tie, the surviving value is stable but arbitrary — it is settled by the steps' internal row ids, which are generated together when the submission is created and carry no step order. It does not change between rebuilds, but you cannot predict which step it came from, so do not design around "the later step wins". Use one recording step per field key when it matters which answer you get.
• Each grid (repeating table) in the form becomes its own grid dataset, named parent__grid, with one row per grid row.

A grid cannot be flattened into one CSV row, which is why it gets its own dataset. A form with a grid means two datasets for you to join.

Grid rows carry the id of the submission they were captured in, at every depth, so id is the join key, and it is not unique within a grid dataset. Grid rows also carry row_index, which tells rows of one submission apart and is needed to page them.

A submission contributes no grid rows when its grid is empty. Older or unusual submissions can also store something other than a list of rows under a grid, and rather than fail the dataset the export makes the best of it: rows stored as an object keyed by number export as normal rows, in number order; a single object exports as one row, with values wherever its keys match the grid's fields; and a plain value exports no rows. The submission itself exports normally, so a submission with nothing in the grid dataset is expected, not a lost page.

A submission appears only once a recording step has been saved. One that was created but never filled in has no row at all, not a row of empty fields. So your row count will not match the record count staff see in the app, and Pending is a status you will rarely, if ever, see in an export.

Nested grids

A grid inside another grid becomes its own dataset too, at any depth. Each level hangs off the one directly above it, so a nested grid's parentDataset is the outer grid's dataset, not the form's. Read the hierarchy from parentDataset, not from the key.

A nested grid's row_index counts all its rows within the submission in one unbroken sequence, rather than restarting in each parent row, so id plus row_index stays unique and it pages like any other dataset.

It also has one extra column per grid above it, <outer grid>_row_index, holding that grid's position inside its parent. For a grid directly inside a top-level grid, that equals the outer grid dataset's row_index, so join a nested row to the row it was captured under on the same id and <outer grid>_row_index = outer row_index. That join holds even when some outer rows have an empty nested grid.

Only the join to the top-level grid holds. An ancestor column is that ancestor's position inside its own parent, while every dataset publishes row_index as a sequence across the whole submission. Those agree only for a top-level grid. In a chain three or more deep, joining to an intermediate grid attaches rows to the wrong parent without any error. If you need to rebuild a hierarchy that deep, tell us: it needs a key we do not publish yet.

A parent row with an empty nested grid adds no rows to the nested dataset, just as a submission with an empty grid adds none to that grid's dataset. A missing nested row is not a missing parent row.

Forms to CSV / 01 Columns

CSV columns

Your CSV is your form, flattened.

Every column of a form dataset is either part of a small fixed envelope (which submission, which site, when it last changed) or comes from a field your organization put on the form. A submission dataset and a grid dataset have different envelopes; fields work the same way in both.

This model is for form datasets only. Resources and deviations have fixed columns of their own, listed on their page.

Three things here apply to every dataset, not just form datasets: the CSV dialect, the rule that deleted records stop being exported, and, for deviations, the record and deviation codes.

Every CSV also ends with a _cursor column, on every dataset including resources and deviations. It is added by the export rather than stored on the dataset, and it holds this row's paging position — see The loop. It is listed in GET /v1/datasets alongside the real columns, so a loader built from that listing has somewhere to put it, and its value is opaque: store the string, do not model it. Nothing else in the export starts with an underscore, so you can drop it on that alone if you would rather not keep it.

The order of the dataset's own columns is not part of the contract, and editing a form changes them; see When someone edits the form. _cursor is appended after all of them, which is why it comes last — read it, like every other column, by header name rather than by position.

Envelope columns

These come from the submission, not from form fields. They are the envelope of a submission dataset; a grid dataset has the smaller set in Grid columns, however deeply it nests. Beyond these and the form fields, every CSV ends with one more column, _cursor, which the export appends and no dataset stores.

Each envelope timestamp appears twice: in UTC, and as *_local in the site's own time zone. Compute on the UTC column and use the local one for display: *_local values have no offset, so they are ambiguous around daylight-saving changes. Timestamps that come from form fields have no _local twin; they are UTC instants.

status is one of five values: Pending, InProgress, Completed, Cancelled, Rejected. Archived is never a status. Archiving is tracked separately: status keeps the status underneath (usually Completed), and is_archived says whether it was archived. The app shows such a record as Archived, so read both columns to match what staff see.

site_path is a chain of site ids, not names. It lists the row's site and every site above it, outermost first, joined by dots, ending with the row's own site_id. Matching one id anywhere in it selects that site and everything beneath, which makes it useful for rolling up a subtree. Split on . for the ancestors; it is not meant to be read by people.

Key your site dimension on site_id, not site_name. site_id never changes; site_name is the current name, so a join on it splits a renamed site's history in two, silently.

Every form dataset, submission and grid alike, has site_id, site_name and site_path, and so does deviations. Most resource datasets do not; see Site, status and record columns.

Because you only receive rows that still exist, a deletion is invisible to the cursor, and a record you already loaded stays in your warehouse unless you look for it. If deletions matter, re-read the dataset in full on a schedule (the same loop, with no cursor) and treat ids you no longer receive as deleted. This matters most for resource datasets, because live submissions keep referring to resources that no longer exist.

Verification columns

If the form has a verification step with fields on it, these columns appear too, and each verification field becomes a column prefixed verification_:

They depend on verification fields, not the step: a verification step with no fields adds none of these columns, and neither does a form without one.

A verification can also change without the row's last_modified moving, which an incremental sync will not show: re-read in full if you report on these columns.

Each row reflects only one verification, and an in-progress one takes precedence over a completed one. A step saved but not finished shows an empty completion time and the in-progress editor's name, hiding a verification that did complete. When reporting on verification, treat a verifier without a completion time as not yet verified, not as missing data.

Grid columns

Every grid dataset, at any depth, has this smaller envelope, plus one <outer grid>_row_index column per grid it sits inside; see Nested grids.

A grid row carries no record_id, status or start and completion times: reach them through the id it shares with its submission.

Field types

Most fields become exactly one column. These are the ones that do not, and they are where integrations usually go wrong, so check your forms for them.

Cell value says what a value means. How it is written in the CSV is covered in Reading the values: for example, a boolean is t or f, never true or false.

In the column names, <field> is the field's slugified key and <question> a survey question's. <option> is different: it is the option's stored value, slugified — not its label. The two are set separately in the form builder and often differ, so an option labelled "Floor" with the value opt1 gives you areas_cleaned__opt1, not areas_cleaned__floor. The option's label reaches the export only as the column's label. Take these column names from GET /v1/datasets, never from the labels you can see on the form.

A grid on a recording step is not in the table because it produces no column of its own: it becomes its own dataset, at any depth. See Nested grids. A grid on a verification step is not split out that way, so avoid putting one there.

• "Blank" in the table above means nothing was recorded, not an empty cell. A _name column joins a first and last name with a space, so an unsigned signature or an empty employee field exports a single space; see Reading the values. Trim before testing for emptiness.
• Choice fields store a value but export its label, so the CSV reads the way staff see the form. One label is exported per stored value: options from every published version are merged on the stored value, and the newest published label for it wins, from the next rebuild on. An option only older versions had keeps its entry, so old rows still read correctly. Only a value that was never an option in any published version (usually imported or migrated data) passes through as it is. Because the label is what lands in the CSV, renaming an option changes the text of old rows too; key on something else if you need a value that never changes.
• A checkbox group (selectboxes) becomes one boolean column per option, so adding an option adds a column. A single checkbox is one boolean column.
• A reference field exports the selected record, not a scalar. A select that draws its options from a data source instead of a configured list stores the record that was picked, and unless the form narrows it to a single property, that whole record is what lands in the cell, as a JSON document: an object with the record's own fields, nested objects and all. Nothing summarises or shortens it, so these are typically the widest cells an export has — a single one can run to hundreds of thousands of characters. Read them as JSON rather than as text, and see Reading the values for what that costs your parser. The *Select types above are the ones that are not like this: they are listed because they export a name and an id instead.

Reading the values

The CSV dialect is fixed, for every dataset:

Five of these catch people out regularly:

1. There is a byte-order mark. Decode as utf-8-sig (Python) or strip the leading EF BB BF. Decoded as plain UTF-8, your first header becomes \ufeffid instead of id, and every lookup on it misses. Inside a paging loop that reads like a paging bug rather than a decoding one: the first column is id, so the cursor you build from the last row comes out empty and the next call is a 400. Fix the decoding, not the loop.
2. Timestamps are not strict ISO 8601. Three things about them defeat a strict parser: a space instead of the T, an offset written +00 rather than +00:00, and a fractional part whose width varies from row to row and is absent on a whole second. Parse with an explicit format that allows all three, or with a lenient parser.
3. Booleans are t and f. A check for "true" reads every boolean as false, silently.
4. Null and empty string differ (bare versus quoted), but most CSV libraries treat them the same. If the difference matters, check that your parser keeps it.
5. A field can be larger than your parser allows. Python's csv raises at 131,072 characters, so raise the cap before the first read: csv.field_size_limit(min(sys.maxsize, 2**31 - 1)). The cap is a C long, so a bare sys.maxsize is itself an error on Windows. A spreadsheet is the quieter version of the same problem: Excel keeps 32,767 characters of a cell and drops the rest without saying so, so load these datasets with something else.

A name column built from two parts is never empty. The <field>_name column of a signature or customSignature, and the name column of an employeeSelect, join a first and last name with a space. The space stays even when both halves are missing, so an unsigned signature or empty employee field exports as a single space: not blank, not NULL. Trim before testing for emptiness; WHERE signed_off_name <> '' does not filter these out, and a "signed" count built that way is wrong. The same goes for contact_name and employee name on resource datasets.

Every successful export also has Content-Type: text/csv and Content-Disposition: attachment; filename="<dataset>.csv", so a browser treats it as a download. The body is streamed, so there is no Content-Length to size a buffer from or check completeness against.

Record and deviation codes

record_id is the short code staff see on a record in the app. It is on every submission dataset (grid datasets do not have it; reach it through the id they share with their submission) and on deviations, which also has code, the deviation's own code, and linked_record_code, the code of the record linked to resolve it. These are the codes to quote to Provision users, and the columns integrations most often mistake for keys. A resource dataset can have an unrelated record_id column from a custom attribute; nothing in this section applies to that one.

They are not join keys. Every dataset's id is a uuid, and id (plus row_index on a grid dataset) is what you upsert on; see Incremental sync. Use codes for display and for talking to your Provision users.

A linked_record_code does not mean the deviation was resolved. It is filled in as soon as a record is linked as the intended resolution, before that record has resolved anything: the row shows status InProgress and is_resolved f, and may stay that way indefinitely. is_resolved is the column that answers the question; treat linked_record_code as "where to look", not "done".

The current formats are a date part and a daily counter, with deviations prefixed D-:

YY is the last two digits of the year and DDD the day of the year (001 to 366), both in the time zone of the record's site, or in UTC when the site has no time zone set (a per-site setting that is not always filled in). Near midnight those give different days, so a code's date part is the site's calendar day, not yours, and not a reliable way to date a record: use last_modified or the record's own date columns. A site with no time zone at all gets UTC in its *_local columns as well. NNNN counts within your organization and restarts at 1 each day. Record codes and deviation codes count separately, so the same number can appear in both on one day.

Four things follow, and each breaks a parser built from the examples above:

Older rows have a different format, permanently. Before these codes, record and deviation codes were six-character strings of digits and capital letters without dashes, like H7K3RQ, from an alphabet that leaves out the lookalikes 0, 1, 5, I, O and S (nothing should depend on that). They were never rewritten, so records and deviations from before the change keep them, and a full sync returns both formats in the same column. An old code has no -, so branch on the format, not on a date: treat a code without - as an opaque identifier. Record codes and deviation codes changed at different times, so a record in the current format can have deviations in the old one, and the other way round.

A current code is not fixed width. The counter is padded to four digits only while it fits. When a day's counter passes 9,999 in one organization, the next record code is 26-256-10000, and deviation codes cross the same boundary on their own counter (D-26-256-10000). Anything assuming a fixed length (11 characters for a record code, 13 for a deviation code) or slicing by position breaks that day, without warning. Once you know a code has a -, split on - and take the last part as the counter.

Codes are unique within your organization, not globally. If one warehouse table holds several organizations, put the organization in the key, or use id, which is unique everywhere.

The numbering has gaps, and is not a sort order. A day's counter can skip values, so a missing number is not a missing record, and counts cannot check completeness. And because the counter is not fixed width, sorting codes is not sorting by time: as strings, 26-256-10000 sorts before 26-256-9999. Order by last_modified instead.

Forms to CSV / 02 Form changes

When someone edits the form

Your columns are your form's fields, so editing the form changes your columns. The change is eventual, not immediate: exports are rebuilt in the background shortly after a form is published, so expect a short delay before a new column appears.

Row data can lag the app slightly too: a submission saved seconds ago may not be in the very next export. It keeps the last_modified it was saved with, not the time it became exportable, so it can surface just behind a cursor you already hold — and a strict cursor never offers it again. Cover it as A row can arrive just behind your cursor describes, and do not treat an export as a real-time read.

Build your consumer to cope with everything below: select known columns by header name, ignore columns you do not recognise, and treat a missing column as empty rather than fatal.

Read the header row

Columns come out in whatever order the dataset defines, so column order is not part of the contract: adding a field to a form can move existing columns.

Parse by header name. A positional parser breaks the first time someone edits the form, and it breaks by putting values in the wrong columns, which is far worse than failing outright.

A long backfill can see the header change between pages, when a form is published mid-way. Read the header of every response; see The loop.

Columns accumulate

Columns accumulate. A dataset's columns are the union of the fields of every version of the form ever published, and publishing cannot be undone, so columns are added and in practice never removed. The one exception is a retyped field: it exports only the columns its newest type produces. Two smaller ones are below: emptying a grid or survey, and relabelling a grid that a nested grid joins on.

• The column count only grows. A long-lived form has a column for every field it ever had. Select the columns you need by header name.
• A removed field's history is never dropped. Fields and options removed in a later version keep their columns, because old submissions still have data in them. The exception is emptying a grid or survey completely: a published version with no fields or questions left in it drops the columns the older versions had. A retyped field is different; see Changing a field type.
• Two field keys can share a column name, and the second gets _2; see the naming rules.

Renames

Renaming a grid moves its dataset, and the old name stops working. Rows, numbering and access all carry over to the new name; nothing is copied or left behind. The old name disappears from GET /v1/datasets, and its export returns 404, not an empty result, so a loader pinned to it stops with an error rather than going quietly stale.

Dataset names come from the form's name and grid labels: renaming a form or relabelling a grid renames its dataset, and because a grid's key is built from its parent's, renaming a form moves every grid dataset under it too. Column names come from field keys, so relabelling a field only changes its label. The exception is a nested grid's <outer grid>_row_index column, named from the outer grid's label: relabelling the outer grid renames the column you join on. Store the key from GET /v1/datasets, re-read the list on a schedule, and alert when a key vanishes: that is a rename.

Renaming an option label rewrites history. Options are merged on their stored value and the current label is exported, so every row gets the new label, however old. If you need a choice that never changes, key on something other than the exported text.

A rewrite like that moves no row's last_modified, so an incremental sync never offers the changed rows again. Re-read the dataset in full after a form edit that changes what old rows export.

Changing a field type

Editing a field in place reaches the export from the next rebuild. A field's label and type come from the newest published version, so renaming it updates label, and retyping it changes the column's type, not at the moment of publishing but shortly after. Retyping is the disruptive one: values change shape under a consumer that already inferred a type. Prefer adding a new field.

A retype that changes a field's column shape drops its old columns. A field exports only the columns its newest type produces. Turn a checkbox group into a radio and its <key>__<option> columns go, along with their historical values, replaced by one <key> column; a resource select turned into text loses its <key>_id. A field whose old and new types both export a single <key> column (text, number, date and time, checkbox, radio, select) keeps its column.

Keeping the column does not mean keeping the values. A radio or select with options exports the option's current label, while a text field exports the stored value, so retyping one into the other rewrites every cell — old rows included — while the column's name and its published dataType both stay text. Nothing in GET /v1/datasets marks that, so re-read the dataset in full after such a retype.

A retype empties cells that do not fit the new type. The new type applies to every row, including rows recorded under the old one, and a stored value it cannot represent exports as empty, not as text. The usual case is a text field holding n/a or TBC becoming a number. Watch date and time fields, where a plausible value can still come out empty: they read only ISO-style values with the four-digit year first, 2026-02-13, optionally followed by a time (T or a space, seconds and fractions optional) and optionally an offset (such as Z, +02:00, +0200 or +02). So 02/13/2026, 13/02/2026, 2026-2-3 and 20260213 all export empty. That is deliberate: 02/13/2026 is ambiguous, and an empty cell is better than a wrong guess. The submission itself is untouched and still shows in the app; only the exported cell is empty. If a retyped column matters, check old rows for empty cells before relying on it.