Code generation & versioning

Repository: github.com/observertc/schemas

The pipeline#

Avro schema definitions in sources/samples/ are the single source of truth. A TypeScript CLI reads them, validates them, and writes every other artifact.

git clone https://github.com/observertc/schemas.git
cd schemas
npm install
npm run generate

That one command:

1. Reads and validates the Avro schema files in sources/samples/.
2. Writes flattened Avro schemas — all chunks expanded — to outputs/avsc/.
3. Writes TypeScript type definitions to outputs/typescript/.
4. Writes Protocol Buffer definitions to outputs/proto/ and compiles them with buf.
5. Assembles the three npm packages — the sample types and the two codecs — and bumps their versions.
6. Writes the Markdown reference to schemaList.md and the schema library README.

Useful shortcuts#

npm run generate -- --help  # every option
npm run generate:check      # fail if the committed outputs are stale (CI)
npm run generate:dry-run    # show what would change, write nothing
npm run generate:types      # regenerate only TypeScript and Avro
npm run generate:proto      # regenerate only protobuf
npm run generate:packages   # proto, samples-lib, both codecs
npm run generate:codec      # the protobuf and JSON codec packages
npm run generate:docs       # markdown + samples-lib
npm run schemas:list        # list the discovered schemas and chunks
npm run schemas:validate    # validate, and check every field is documented
npm run schemas:clean       # remove generated outputs
npm run verify              # typecheck + generate:check

Generated outputs#

npm packages#

All three are versioned in lockstep with sources/version.txt by the generator, so a package version is the schema version it speaks. Each exports schemaVersion.

Project structure#

├── sources/                 # Source Avro schema definitions
│   ├── samples/             # Sample schemas (.avsc) and reusable chunks
│   ├── w3c/                 # W3C stats identifiers, copied into the library
│   └── version.txt          # Current schema version
├── src/                     # The generator (TypeScript)
├── outputs/                 # Generated outputs
│   ├── typescript/
│   ├── proto/
│   └── avsc/
├── npm-samples-lib/             # Generated core TypeScript library
├── npm-samples-protobuf-codec/  # Protobuf delta codec (encode + decode)
├── npm-samples-json-codec/      # JSON delta codec
├── CHANGELOG.md             # Schema change history
└── docs/GENERATOR.md        # How the generator works

Adding a field#

1. Edit the appropriate .avsc file in sources/samples/.
2. Document the field — npm run schemas:validate fails if any field lacks a doc.
3. Run npm run generate to regenerate all outputs.
4. Check the diff of outputs/proto/ before merging.
5. Update sources/version.txt and add a CHANGELOG.md entry.

Schema guidelines#

• Every field needs clear documentation.
• Use null unions for optional fields, with "default": null.
• camelCase field names.
• Add an attachments field for extensibility where appropriate.

Versioning#

Semantic versioning, with these conventions:

The 3.x line has had three type changes and no removals: scoreReasons in 3.4.0 and again in 3.6.0, and the four payload fields in 3.5.0 and again in 3.7.0 — the last of which did not move either wire format at all.

• PATCH — library bugfixes and improvements
• MINOR — new fields and schema updates
• MAJOR — breaking changes to schema structure

The current version lives in sources/version.txt and is stamped into every generated artifact.

Current: 3.7.0. See the version history for the full 3.x record, or CHANGELOG.md for everything back to 2.0.0.

Compatibility model#

• Forward compatibility — newer readers process older data.
• Backward compatibility — older readers may not see newer optional fields, and will misread a record whose protobuf numbering shifted.
• Schema evolution — new optional fields appended at the end of their group maintain compatibility.

Migrating between versions#

1. Review the release page for every version you are skipping.
2. Upgrade both halves of your codec together — they are generated in lockstep for a reason.
3. Decode stored binary samples with a version matching what wrote them, or re-encode.
4. Update processing for new fields you want to consume.
5. Deploy incrementally and watch for decode errors before switching writers over.

Publishing#

The three npm packages are released by GitHub Actions using npm trusted publishing (OIDC) — there is no NPM_TOKEN. Each package needs a Trusted Publisher registered once on npmjs.com pointing at its workflow file; the exact settings are in docs/GENERATOR.md.

Contributing#

1. Fork the repository.
2. Create a feature branch.
3. Make your changes in sources/.
4. Run npm run generate.
5. Run npm run verify and check the generated outputs.
6. Open a pull request.