(text bjodat)

Overview

Records that read and write themselves as bjodat: declare one, get its decoder and encoder.

(import (text bjodat))

(def/bjodat-type
  (: Dep (Record (: name string)
                 (: version string #:default "*")
                 (: note (Option string) #:optional))))

(bjodat-parse-Dep "{:name \"http\" :version \"1.2\"}")  ; (Ok Dep)
(bjodat->string (Dep->bjodat d))                        ; "{:name \"http\" ...}"

bjodat is Bjolang’s shapes read as data, and is described with (text bjodat-core), which holds the value, the reader and the writer. This module declares records that are read from it and written to it. bjo/manifest.bjo is a worked example of everything here: tagged forms, #:rest, a union dispatched on its head symbol, and hand-written instances for types the generator never heard of. It is an ordinary user of this module and gets no special case anywhere in it.

bjodat-parse-<Name> reads the text straight into the record. There is no intermediate tree: the key is compared against the field names in the reader’s own buffer, so a record costs no key string, no interned Keyword, no node and no map. That is the interface to reach for.

The two modules.

`(text bjodat)`	`def/bjodat-type`, the traits, and what they generate
`(text bjodat-core)`	the `Bjodat` value, the reader and the writer

(text bjodat) is enough for everything but the tree: the generator, the traits, bjodat->string, and hand-written instances naming BjoInt and the other cases. The code def/bjodat-type writes is spliced into your module and names Bjodat and Reader in its signatures, and this module re-exports both: one spelling of bjodat-core’s declarations, not a copy of them, so a Bjodat built either side of the boundary is one type. Re-exporting a union re-exports its cases.

Import (text bjodat-core) as well when you want the tree: parse-bjodat, read-bjodat, bjodat-ref and the entry accessors, for data whose shape is not known until run time.

These were once always imported together, on the belief that a type could not cross a facade. What cannot cross is a type declared twice; what a re-export publishes is the declaring module’s declaration under a spelling of the facade’s. See re-export in Docs/MODULES.org.

Instances for a type of your own. The generated decoders reach a field’s type through its bjodat-> instance, so a type the macro never heard of works as soon as it has one:

(type (: Port (Record (: number int))))

(impl (->bjodat Port)
  (defun (->bjodat p) (BjoInt (cast System.Int64 (record-ref p number)))))

(impl (bjodat-> Port)
  (defun (bjodat-> v)
    (match v
      ((BjoInt n) (Ok (Port (number (cast System.Int32 n)))))
      (_ (Err "expected a port number")))))

(def/bjodat-type (: Server (Record (: host string) (: port Port))))

Errors. Every decoder answers (Result string _), and the message names the path:

Package: the map has no :version entry
Package.name: expected a symbol
Package.deps: Dep: the map has no :version entry
line 1, column 12: Package: this map is written flat, so a '(' here is an entry of the other form

and, for a tagged form:

Package: there is no (version ...) clause
Package.name: expected a string
Package.name: a clause holds a field name and one value
Package: a clause begins with a field name
line 1, column 8: Package: expected (package ...), and this is (banana ...)

The one-pass and the tree decoders give the same message for the same document, word for word, which is asserted rather than assumed: the codec tests check each refusal through both paths. A structural refusal carries a line and column, because the reader knows one and a decoder walking a tree does not; the sentence under it is the same. TestFiles/218_bjodat_forms.bjo asserts both halves: the agreement, and the two cases where only the position differs.

What to watch out for.

An unknown key is read and ignored, and so is an unknown clause, at any arity. It has to be read: finding where a value ends means parsing it. Whether a manifest format should instead refuse a misspelt :verison is still open: refusing is safer, ignoring is what makes a file forward-compatible with a newer tool, and it may want to be a marker rather than a decision taken once for everyone.
A record holds its fields and nothing else, so a manifest that round-trips through one comes back without whatever clauses that version of the tool did not know. A tool that edits a manifest in place has to work on the tree. This is the cost of the item above, and the two are the same fact.
A key that is not a keyword can never name a field, so it is ignored too. The tree decoder agrees, for its own reason: bjodat-ref only matches a BjoKey.
A duplicate key: the first wins, in both decoders. The later one is still read, and then dropped. The same holds for a repeated clause.
An (Option (List %a)) holding (Some Nil) writes () and reads back as None, because None is written as the empty list. Nothing else collides.
The one-pass decoder does not track nesting depth across records. Each generated decoder starts a fresh budget, so a chain of nested record types is bounded by the types rather than by bjodat-max-depth. The values inside a field are bounded normally.

Speed. 20 000 records, 2.36 MB, best of 20 on one machine. Five scalar fields:

text to records, one pass	376–383 ns/rec	about 310 MB/s
text to records, via tree	478–492 ns/rec	about 245 MB/s
the same through `(text json)` + `(text json-codec)`	582–599 ns/rec	about 215 MB/s

A record whose fields are sequences gains nothing. The same corpus with a (Vec string) and an (Option string) field measures 725–737 ns/rec one-pass against 708–711 through the tree: a wash, and on some runs marginally worse.

That is the design, not a defect. The one-pass decoder saves the entry: the key’s string, its interned Keyword, the BjoKey node, the vec slot it lived in, and the scan over it. It does not save the value: each still becomes a Bjodat and goes through its field type’s own instance, which is what keeps a hand-written instance working and every message identical. Where the values dominate, there is nothing left for it to take away.

So reach for bjodat-parse-<Name> by default, and do not expect it to be the thing that makes a sequence-heavy document fast. If it matters, measure both: they are both there and they agree. Numbers, and what did not move, are in bench/BASELINE.md.

Reference

Functions

`->bjodat`

function

(: ->bjodat (-> %a Bjodat))

Writing a value as bjodat: a type with an instance can be a field.

%a: The type written.
v: The value.
returns: It, as a Bjodat.

def/bjodat-type writes an instance for each record it declares. Instances ship for long, int, double, bool, string, char, Symbol, Keyword and Bjodat itself, and for (Option %a), (Vec %a) and (List %a) when the element type has one. None is written as the empty list (); a Vec is a [...] and a List a (...).

`bjodat->`

function

(: bjodat-> (-> Bjodat (Result string %a)))

Reading a value from bjodat: a type with an instance can be a field.

%a: The type read.
v: The value read.
returns: (Ok x), or (Err message) when the value is not of the type's shape.

Instances ship for the same types as ->bjodat’s. Each takes only its own shape: a symbol and a keyword are different values, and a field asking for one does not take the other; a Vec wants a [...] and a List a (...). The one widening is double, which takes an integer too. An (Option %a) reads the empty list as None and anything else as Some. A refusal is a message such as expected a string, which the generated decoders prefix with the path to the field.

`bjodat-map`

function

(: bjodat-map (-> string Bjodat (Result string (Vec Bjodat))))
(bjodat-map owner v)

A value’s map entries, or a refusal. Generated code calls it; not an API for people.

owner: The record's name, for the message.
v: The value.
returns: The entries of a BjoMap, or owner: expected a map.

`bjodat-field`

function

(: bjodat-field (-> string (Vec Bjodat) Keyword (Result string %a)))
(bjodat-field owner entries key)

A required field, from a map’s entries. Generated code calls it; not an API for people.

owner: The record's name, for messages.
entries: The map's entries.
key: The field's key.
returns: The first value under the key, converted by its instance; a refusal names the record and the key.

`bjodat-field/default`

function

(: bjodat-field/default (-> string (Vec Bjodat) Keyword %a (Result string %a)))
(bjodat-field/default owner entries key fallback)

A field with a fallback, from a map’s entries. Generated code calls it; not an API for people.

owner: The record's name, for messages.
entries: The map's entries.
key: The field's key.
fallback: What an absent field is.
returns: The first value under the key, converted by its instance, or the fallback when there is none.

`bjodat-read-field`

function

(: bjodat-read-field (-> string Keyword Reader (Result string %a)))
(bjodat-read-field owner key rd)

Reads a field’s value off the reader and converts it. Generated code calls it; not an API for people.

owner: The record's name, for messages.
key: The field's key, for messages.
rd: The reader, standing at the value.
returns: The value, converted by its instance; a refusal names the record and the key.

`bjodat-need-field`

function

(: bjodat-need-field (-> (Option %a) string Keyword (Result string %a)))
(bjodat-need-field slot owner key)

A required field the one-pass decoder may not have met. Generated code calls it; not an API for people.

slot: What the decoder found, if anything.
owner: The record's name, for the message.
key: The field's key, for the message.
returns: The value, or bjodat-field's refusal word for word, so a report does not reveal which decoder read the document.

`bjodat-key`

function

(: bjodat-key (-> Keyword Bjodat))
(bjodat-key k)

A keyword as a BjoKey, for a generated encoder. Not an API for people.

k: The keyword.
returns: (BjoKey k).

`bjodat-entries`

function

(: bjodat-entries (-> (List Bjodat) Bjodat))
(bjodat-entries acc)

A map from entries pushed in reverse, for a generated encoder. Not an API for people.

acc: Value, key, value, key, ..., last entry first.
returns: The BjoMap, entries in order.

`bjodat-clauses`

function

(: bjodat-clauses (-> string string Bjodat (Result string (List Bjodat))))
(bjodat-clauses owner tag v)

A tagged form’s clauses, with its tag checked. Generated code calls it; not an API for people.

owner: The record's name, for messages.
tag: The head the form must have.
v: The value.
returns: The clauses, or a refusal when the head is not the tag or a clause does not begin with a field name.

`bjodat-clause-field`

function

(: bjodat-clause-field (-> string (List Bjodat) string (Result string %a)))
(bjodat-clause-field owner cs name)

A required field, from a tagged form’s clauses. Generated code calls it; not an API for people.

owner: The record's name, for messages.
cs: The clauses.
name: The field's name.
returns: The one value of the first clause under the name, converted; a refusal when there is none or it holds more than one.

`bjodat-clause-field/default`

function

(: bjodat-clause-field/default (-> string (List Bjodat) string %a (Result string %a)))
(bjodat-clause-field/default owner cs name fallback)

A field with a fallback, from a tagged form’s clauses. Generated code calls it; not an API for people.

owner: The record's name, for messages.
cs: The clauses.
name: The field's name.
fallback: What an absent field is.
returns: The one value of the first clause under the name, converted, or the fallback when there is none.

`bjodat-clause-rest`

function

(: bjodat-clause-rest (-> string (List Bjodat) string (Result string (List %a))))
(bjodat-clause-rest owner cs name)

A required #:rest field, from a tagged form’s clauses. Generated code calls it; not an API for people.

owner: The record's name, for messages.
cs: The clauses.
name: The field's name.
returns: Every value of the first clause under the name, each converted; a refusal when there is none.

`bjodat-clause-rest/default`

function

(: bjodat-clause-rest/default (-> string (List Bjodat) string (List %a) (Result string (List %a))))
(bjodat-clause-rest/default owner cs name fallback)

A #:rest field with a fallback, from a tagged form’s clauses. Generated code calls it; not an API for people.

owner: The record's name, for messages.
cs: The clauses.
name: The field's name.
fallback: What an absent field is.
returns: Every value of the first clause under the name, each converted, or the fallback when there is none.

`bjodat-read-clause`

function

(: bjodat-read-clause (-> string string Reader (Result string %a)))
(bjodat-read-clause owner name rd)

Reads a clause’s one value off the reader, and its ). Generated code calls it; not an API for people.

owner: The record's name, for messages.
name: The field's name, for messages.
rd: The reader, standing at the value.
returns: The value, converted; a refusal names the record and the field.

`bjodat-read-rest`

function

(: bjodat-read-rest (-> string string Reader (Result string (List %a))))
(bjodat-read-rest owner name rd)

Reads every value of a clause off the reader, and its ). Generated code calls it; not an API for people.

owner: The record's name, for messages.
name: The field's name, for messages.
rd: The reader, standing at the first value.
returns: The values, each converted; a refusal names the record and the field.

`bjodat-need-clause`

function

(: bjodat-need-clause (-> (Option %a) string string (Result string %a)))
(bjodat-need-clause slot owner name)

A required clause the one-pass decoder may not have met. Generated code calls it; not an API for people.

slot: What the decoder found, if anything.
owner: The record's name, for the message.
name: The field's name, for the message.
returns: The value, or bjodat-clause-field's refusal word for word.

`bjodat-clause`

function

(: bjodat-clause (-> string Bjodat Bjodat))
(bjodat-clause name v)

A clause holding one value, for a generated encoder. Not an API for people.

name: The field's name.
v: The value.
returns: (name v).

`bjodat-clause-many`

function

(: bjodat-clause-many (-> string (List Bjodat) Bjodat))
(bjodat-clause-many name vs)

A clause holding many values, for a generated encoder. Not an API for people.

name: The field's name.
vs: The values.
returns: (name v ...).

`bjodat-clause-values`

function

(: bjodat-clause-values (-> (List %a) (List Bjodat)))
(bjodat-clause-values xs)

Each element written as bjodat, for a #:rest field. Not an API for people.

xs: The field's elements.
returns: Each converted by its instance.

`bjodat-form-value`

function

(: bjodat-form-value (-> string (List Bjodat) Bjodat))
(bjodat-form-value tag acc)

A tagged form from clauses pushed in reverse, for a generated encoder. Not an API for people.

tag: The head.
acc: The clauses, last first.
returns: (tag clause ...), clauses in order.

Macros

`def/bjodat-type`

macro

(def/bjodat-type declaration)

Declares a record and the codecs that read it from bjodat and write it back.

declaration: A record's declaration, (: Name (Record field ...)), or (: Name (Record #:tag tag field ...)) for a record written as a tagged form. Name may be (Name %a ...) for a generic record, and then each generated instance wants the same trait of every type variable. A field is (: name type), then any of the markers below.

(def/bjodat-type
  (: Package (Record (: name Symbol)
                     (: version string)
                     (: description string #:key :desc)
                     (: authors (Vec string))
                     (: entry string #:default "src/main.bjo")
                     (: private bool #:default #f)
                     (: note (Option string) #:optional)
                     (: deps (Vec Dep)))))

(def/bjodat-type
  (: Package (Record #:tag package
                     (: name (List Symbol))
                     (: version Version)
                     (: entry string #:default "src/main.bjo")
                     (: depends (List Dep) #:rest #:default Nil))))

Declares the record and its codecs. Without #:tag, a record is a map whose keys are keywords naming its fields. The keys are keyword literals in the generated code, so decoding a field through a tree compares interned references and never interns anything at run time.

Field markers.

`#:key :other-name`	written and read under `:other-name` rather than the field’s own
`#:optional`	may be absent; absent is `None`
`#:default <expr>`	may be absent; absent is that expression, evaluated at decode
`#:rest`	takes every value of its clause; a tagged form only
`#:mutable`	passed through to the record declaration, and means nothing here

#:optional is exactly #:default None, and the field’s type must be an (Option %a) to hold it. An optional field that is None is not written at all.

There is no null. A key that may be missing is #:optional, and absence is the whole of it: a data format whose values are the language’s values has no separate empty. Where an (Option %a) is written out, None is the empty list ().

A record as a tagged form. With #:tag, the record has a second spelling, and a manifest is what it is for. The second example reads and writes

(package
  (name (bjolang http))
  (version "1.2.3")
  (depends
    (package (name (bjolang cml)) (version (version-at-least "2.0.1")))
    (package (name (bjo core)))))

The head symbol names the type and each clause names a field. It diffs a line at a time, it nests without a bracket per level, and it is what a person writes. The format did not change to allow this: (package (name ...) ...) was already a BjoList headed by a BjoSym. What #:tag changes is the record’s encoding, which shape its generated codec reads and writes, and nothing about the reader. A type has one encoding: a value that read as {...} writes back as {...}, and one that read as (tag ...) writes back as that.

The type picks the shape; the shape never picks the type. A BjoList headed by a symbol is a tagged record where a tagged record is expected, a union case where a union is, and a plain list where a (List Symbol) is, which is why a module name can be written (std run), the way import writes it, with no ambiguity against (git ...) in the next clause. It is the rule {...} has always run on: a map is a Package in one field and a lookup table in the next, and the field’s declared type says which. Nothing in the reader commits, and nothing in the reader could.

#:tag is explicit, with no default. Package spelled package would be a guess about capital letters, and which tag a type is written under belongs in the file’s vocabulary. Two types may share a tag: bjo/manifest.bjo has both Package and Dep under package, because a dependency is a package, named and constrained rather than described, and which one a form decodes into is the field’s to say.

#:rest says which clauses may repeat. A tagged list says nothing about whether a clause may appear with many values, so it is answered per field, by the type, which is the only place that knows: version holds one and depends holds many.

(depends (package ...) (package ...))   ;; (: depends (List Dep) #:rest)
(version "1.2.3")                       ;; (: version Version)

A field is marked; a clause is not repeated. (depends a) (depends b) does not accumulate: the first clause wins, as the first entry under a key does. Accumulating would have meant one rule for maps and another for forms. A #:rest field reads each of its values through the element type’s own instance, which keeps the depth budget honest: a hostile (package (depends (package (depends ...)))) nested five hundred deep is refused by bjodat-max-depth rather than by the stack. An empty #:rest field with a default is not written.

An unknown clause is read and ignored, at any arity. A map’s unknown key is one value and done; a clause may hold none, one or five, and a reader that insisted on one would make every new clause a breaking change for every older tool. So a manifest written for a newer bjo still builds under an older one.

What a tagged form costs. Nothing new. A clause is the parenthesised map entry with a bare symbol where the keyword was, so the tag and every field name are matched against the reader’s own buffer: no string is built and nothing is interned for any of them.

What it generates. For (: Package (Record ...)):

`(bjodat-parse-Package text)`	`(-> string (Result string Package))`: the one to use
`(bjodat-read-Package rd)`	`(-> Reader (Result string Package))`: from a port
`(Package->bjodat p)`	`(-> Package Bjodat)`
`(bjodat->Package v)`	`(-> Bjodat (Result string Package))`

and the ->bjodat and bjodat-> instances the last two stand for. The named pair exists because bjodat-> alone is ambiguous at a call site; the signature is what fixes the instance.

bjodat-read-Package is what to hand read-one-pass-all or parse-one-pass-all for a document that is a [...] of records:

(parse-one-pass-all text bjodat-read-Dep)   ;; -> (Result string (List Dep))

Pass it by name, not wrapped in a (fun (rd) ...). A lambda body is a member of its own and is sealed to the sync copy of what it calls, so the wrapped form makes a bjoroutine park its thread. The blocking lint says so.

See also: ->bjodat, bjodat->, parse-one-pass-all