WritingCompliance
A form builder whose old submissions survive every schema change
How to build a compliance form builder where published schemas are immutable, each submission pins its version, and a reviewer can reopen any old case.
A vendor opens your due diligence questionnaire at 09:02, version 7 of the form, with 41 questions. At 09:20 someone in the compliance team publishes version 8, which adds a required yes or no question with the key pep_declaration and reorders two sections. At 09:41 the vendor presses submit. What the system does in the next 200 milliseconds decides whether that vendor sees a confirmation page or an error about a question that was never on their screen, and whether a reviewer can still read those 41 answers correctly in three years' time.
The system is a form builder for compliance data: ops staff design questionnaires, external parties fill them in, reviewers read the answers, and the answers feed risk rules and exports. The one constraint that shapes it is that a published schema is never edited and every submission carries the id of the exact version it was captured against. Everything else follows: drafts are the only mutable thing, validation and rendering take a version id rather than a form id, and nothing ever migrates an old submission to a new shape.
flowchart TD D[Form designer] --> P[Publish step] P --> R[Schema registry]:::accent R --> F[Form renderer] F --> V[Validator] V --> S[Submission store] S --> X[Review console] R --> X S --> E[Export job] R --> E
Drafts are mutable, versions are not
key, a type (text, choice, boolean, money, file), a label, options and validation rules. Publish is a synchronous call that passes the draft id and nothing else. The publish step compares the draft with the current version and rejects, with a list of field keys, any key whose type changed and any key that was removed without being marked retired; on rejection nothing is written and the draft stays open.The registry assigns the version and freezes it
schema_version row (form id, version number, the schema JSON, a SHA-256 of the canonical JSON, published_at) and moves the form's current_version pointer in the same transaction. The table has a trigger that raises on any UPDATE, so even a hotfix by hand cannot change a published version; the only state change allowed is withdrawn_at, which stops new submissions without touching the content. If the insert hits the unique index on form id plus version number, another publish won, and this one is reported as a conflict rather than retried with the next number.The version is pinned when the form opens
schema_version_id: 7 into the page. The submit payload carries that id back, and the validator loads version 7 by id, never the form's current version. If the row has a withdrawn_at, the validator rejects with a message that names the form and the page reloads onto the current version with the answers that still have matching keys kept.Answers are stored by field key, with the version id
schema_version_id, submitted_at, a superseded_on_submit flag, and an answers JSON keyed by field key, never by label or position. The write is synchronous; if it fails the renderer keeps the answers in the browser and shows a retry rather than clearing the page. A key that does not exist in version 7 never reaches the store, because the validator dropped the payload a step earlier.The reviewer sees what the vendor saw
Exports read across versions by key
ubo_count holds values from version 3 to version 9 in one type. A retired key still gets a column, populated only where it existed. If a referenced version is missing from the registry, the job stops for that form and emits nothing, rather than a file with blank columns that looks complete.A publish in the middle of a session
Back to the vendor at 09:41. Their payload carries schema_version_id: 7 and 41 answers; pep_declaration is not among them, because it did not exist when the page rendered. A validator that loads the form's current version sees version 8, finds a required field missing, and returns a 422 naming a question the vendor cannot find on the page. The vendor sees an error with no field highlighted, submits twice more, then emails support, and the compliance team has three identical attempts and no submission.
I built the Form Builder for a compliance product at Cocomply, and the rule I apply is that the version a person saw is the version their answers are judged against. With the version pinned, the validator loads version 7, accepts the 41 answers, and sets superseded_on_submit because version 8 was current at submit time. The review console shows a banner: answered on version 7, version 8 added pep_declaration. The reviewer sends a one-question follow-up instead of making the vendor start again.
Trade-offs
| Choice | What it buys | What it costs |
|---|---|---|
| A new immutable row per publish | Any submission renders exactly as it was collected, years later | A typo fix is a full version; ops need a diff view to see what changed between 7 and 8 |
| Pin the version at open, not at submit | Nobody is failed on a question they never saw | Some submissions land on a superseded version and need a follow-up for new required fields |
| Stable field keys, type changes forbidden | Risk rules and export columns refer to one key across every version | A type change means a new key, a retired old key, and a rule that reads both until the old one drains |
| Answers keyed by field key | Reordering and relabelling cost nothing | A renamed key is a new field; nothing ever maps the old key to the new one automatically |
| Render from the stored version, never migrate | No migration job, no lossy upcast of old answers | Cross-version reporting needs the union-of-keys export rather than one flat table per form |
Failure modes
- A published row is edited by hand. Someone fixes a label with an UPDATE in a maintenance window. What you observe: the stored SHA-256 no longer matches the canonical JSON, and the nightly integrity check lists the version. What contains it: the trigger that raises on UPDATE, and the review console recomputing the hash on load and warning when it differs.
- The validator falls back to the current version. A refactor gives the version id parameter a default of "current". What you observe: a rise in 422 responses naming fields absent from the submitted payloads, starting at the minute of a publish. What contains it: a validator signature with no default for the version id, and a contract test that publishes version n and then submits a version n minus 1 payload.
- Two publishes race. Two ops users press publish within a second; both read current as 7. What you observe: the second insert fails on the unique index. What contains it: the index, and a publish step that reports the conflict and shows the other user's changes rather than bumping to 9.
- A stale tab submits days later. A vendor left the form open for three days and version 7 was withdrawn after a question turned out to be legally wrong. What you observe: a payload pinned to 7 arriving when current is 9. What contains it: the
withdrawn_atcheck in the validator, which rejects with the reason, and the renderer reloading onto version 9 with the still-valid keys prefilled. - The registry is unreachable from the console. What you observe: reviewers report submissions opening blank. What contains it: the raw key and value fallback with a visible warning, so nobody is shown a current label against an old answer.
Build the registry first: a schema_version table with a unique index on form id and version, a trigger that refuses updates, and a current_version pointer on the form. Then make the renderer put the version id in the page and the validator take it as a required argument. The review console, the export job and the withdrawn state can follow a week later without reworking a single stored submission.