(text bjodat-core)

Overview

The bjodat value, its reader and its writer: Bjolang’s shapes read as data.

(import (text bjodat-core))

(parse-bjodat "{:a 1 :b [2 3]}")     ; (Ok (BjoMap ...))
(parse-bjodat-all "1 2 3")           ; (Ok (list (BjoInt 1) ...)), a file is a stream
(bjodat->string v)                   ; the canonical spelling, maps flat

bjodat is Bjolang’s shapes read as data: no evaluation, no macro expansion, no environment. A file is what it says it is. It is what a package manifest, a lockfile or a config file is written in.

This module is the machinery, not the interface. A program that declares records imports (text bjodat): its def/bjodat-type reads the text straight into a record, with no Bjodat value in between, and it re-exports Bjodat, its cases, Reader, the one-pass driver and bjodat->string, so that one import covers the generator, the traits and the writer. Import this module by name when you want the tree: parse-bjodat, read-bjodat, bjodat-ref and the entry accessors, for data whose shape is not known until run time.

The format. Two differences from source, and they are the whole of it:

A bare identifier is a Symbol. It is already quoted, because there is nothing here to evaluate and so nothing for a quote to protect it from. 'foo is refused: the mark would be a second spelling of one value.
{...} is a map. In source a brace opens a comprehension, and a comprehension is a loop, and a loop is not data.

42  -100  3.14159  1e10  0x2A  0b101010   ;; integer or double, and nothing else
+inf.0  -inf.0  +nan.0                    ;; the doubles that are not numbers, as Scheme spells them
#t  #f                                    ;; the booleans; `true` is a symbol
#\a  #\newline  #\space                   ;; characters, as in source
"a \"quoted\" word\n"                     ;; strings, escaped as source escapes
"""a "raw" word"""                        ;; raw strings, as source writes them
:key   #:key                              ;; a keyword, either spelling
a-symbol   bjolang/http   net8.0          ;; symbols; `.` and `/` are in a name
(a list "of" :anything)                   ;; a list
["a" "vec" "of" "strings"]                ;; a vec, the one sequence type
{:name "http" :version "1.2"}             ;; a map, flat
{(:name "http") (:version "1.2")}         ;; a map, parenthesised
;; a comment, to the end of the line

The escapes are those source has: \n, \t, \r, \\, \", \0 and \uXXXX. The named characters are #\newline, #\space, #\tab, #\return and #\nul. A byte order mark at the start of the text is skipped.

A raw string follows the rules in Syntax.org, “Raw strings”, margin and all. The writer still writes every string escaped and on one line: a raw string written back out would need a margin to fit into the text around it, and bjodat->string writes a value on its own.

A number has no width suffix. 21uy and 21L name a .NET type, and which type a field holds is the record’s to say, not the file’s. A number is a long unless it has a fraction or an exponent, or is too wide for a long, and then it is a double. - is a symbol of its own, and the start of a number when a digit follows it. +inf.0, -inf.0 and +nan.0 are the infinities and NaN, spelled as Scheme spells them; -nan.0 reads as NaN too.

There is no #[...]. Source splits vec from array because one is immutable and the other is a .NET array; data is neither, so a second spelling would read the same and mean nothing.

A comma is not whitespace. Source lexes , as unquote, so a stray one is a token rather than a decoration, and it is refused. Separate with spaces.

Maps. Two spellings, and no third. Which one a map is, is decided by the character after the brace: a ( means the parenthesised form, anything else means the flat one. They do not mix, and a map that starts one way and continues the other is refused.

That rule is why a list may not be a key. If it could, {(a b) 1} would be both a flat map keyed by the list (a b) and a parenthesised map whose first entry is a -> b. A key is a name, and a name is a keyword, a symbol, a string, a number or a char, never a list. A flat map with an odd number of forms is the other refusal.

Entries keep the order they were written in, and writing one back out writes them in that order. A manifest that round-trips through a record comes back looking like what a person wrote, not sorted by hash.

A file is a stream. Nothing requires a file to hold exactly one value, which is what makes a log or a lockfile writable by appending. parse-bjodat asks for exactly one and refuses trailing text; parse-bjodat-all takes them all, and an empty file is an empty list rather than an error.

The tree. A BjoMap holds a flat (Vec Bjodat) of alternating key and value, in written order, not a trie. Building a trie was about seventy percent of what reading a keyed document cost, and a record decoded from one reads every field once and throws it away. Reach into it with bjodat-entry-count, bjodat-entry-key, bjodat-entry-value and bjodat-ref. If you want a dictionary, build one from the entries.

Refusals. Every reader answers (Result string _). A refusal while reading carries the line and column where it was found:

line 1, column 12: Package: this map is written flat, so a '(' here is an entry of the other form

The one-pass machinery. Reader and the functions from open-map! to parse-one-pass-all are what a decoder generated by def/bjodat-type drives to read a record without a Bjodat tree standing between the text and the fields. They are small and awkward on purpose: the caller is generated code that already knows its field names, and drives the loop itself rather than handing in a table, because a table would be the thing this is trying not to build. They are not an API for people, but the entry points (parse-one-pass and the rest) are how a generated bjodat-read-<Name> is run.

Reference

Types

`Bjodat`

union

A bjodat value: what a document reads as, as a tree.

BjoBool: #t or #f.
BjoInt: An integer, always a long.
BjoFloat: A number with a fraction or an exponent, or one too wide for a long; or +inf.0, -inf.0, +nan.0.
BjoStr: A string, raw or escaped.
BjoChar: A character, #\a or #\newline.
BjoSym: A bare identifier. It is data, so it needs no quote.
BjoKey: A keyword, written :key or #:key.
BjoList: A (...).
BjoVec: A [...].
BjoMap: A {...}: key, value, key, value, an even length, in written order, whichever spelling it was written in.

(match (parse-bjodat "{:name \"http\"}")
  ((Ok (BjoMap entries)) (bjodat-ref entries :name))   ; (Some (BjoStr "http"))
  (_ None))

A BjoMap is not a trie or a dictionary but a flat vec of alternating key and value. Read it with bjodat-entry-count, bjodat-entry-key, bjodat-entry-value and bjodat-ref.

`Reader`

record

A reader over a port, as a generated one-pass decoder drives it. Not an API for people.

port: What it reads from.
held: The one character of lookahead, as a code; -1 at the end of the port.
row: The line the held character is on, from 1.
col: Its column, from 1.
scratch: One buffer for every string, symbol, number and key name in the document.
stack: One array for every list, vec and map in the document.
top: How much of the stack is in use.
names: A small cache of the names read, so that a repeated key does not build a new string.

Made by read-one-pass and its neighbours and handed to the decoder they are given. A program has no reason to touch its fields.

Functions

`read-bjodat`

function

(: read-bjodat (-> TextInputPort (Result string Bjodat)))
(read-bjodat port)

Reads exactly one value from a port, and then the end.

port: The port to read from, to its end.
returns: (Ok value), or (Err message) naming the line and column. Text after the value is refused.

`parse-bjodat`

function

(: parse-bjodat (-> string (Result string Bjodat)))
(parse-bjodat text)

Reads exactly one value from a string.

text: The document.
returns: (Ok value), or (Err message) naming the line and column. Text after the value is refused.

(parse-bjodat "{:a 1 :b [2 3]}")   ; (Ok (BjoMap ...))
(parse-bjodat "1 2")               ; (Err "line 1, column 3: the value ended here, ...")

`read-bjodat-all`

function

(: read-bjodat-all (-> TextInputPort (Result string (List Bjodat))))
(read-bjodat-all port)

Reads every value from a port, in order.

port: The port to read from, to its end.
returns: (Ok values), or the first refusal. An empty document is the empty list.

A file is a stream of values, which is what makes a log or a lockfile writable by appending.

`parse-bjodat-all`

function

(: parse-bjodat-all (-> string (Result string (List Bjodat))))
(parse-bjodat-all text)

Reads every value in a string, in order.

text: The document.
returns: (Ok values), or the first refusal. An empty document is the empty list.

`bjodat->string`

function

(: bjodat->string (-> Bjodat string))
(bjodat->string v)

Writes a value in the canonical spelling.

v: The value.
returns: Its text: maps flat, entries in the order they are held, strings escaped and on one line.

(bjodat->string (BjoVec [(BjoInt 1) (BjoSym 'a)]))   ; "[1 a]"

Strings are always written escaped, never raw: a raw string written back out would need a margin to fit into the text around it. A double is written in the shortest form that reads back as the same double, and always with a fraction or an exponent, so that it reads back as a BjoFloat: 2.0 is written 2.0, and 1e20 1E+20. NaN and the infinities are written +nan.0, +inf.0 and -inf.0. Symbols and keywords are written as their names, unchecked.

`bjodat-entry-count`

function

(: bjodat-entry-count (-> (Vec Bjodat) int))
(bjodat-entry-count entries)

How many entries a map’s vec holds.

entries: What a BjoMap holds.
returns: Half its length.

`bjodat-entry-key`

function

(: bjodat-entry-key (-> (Vec Bjodat) int Bjodat))
(bjodat-entry-key entries i)

The key of a map’s i-th entry.

entries: What a BjoMap holds.
i: The entry, from 0.
returns: Its key, which may be any value but a list.

`bjodat-entry-value`

function

(: bjodat-entry-value (-> (Vec Bjodat) int Bjodat))
(bjodat-entry-value entries i)

The value of a map’s i-th entry.

entries: What a BjoMap holds.
i: The entry, from 0.
returns: Its value.

`bjodat-ref`

function

(: bjodat-ref (-> (Vec Bjodat) Keyword (Option Bjodat)))
(bjodat-ref entries k)

The first value under a keyword in a map.

entries: What a BjoMap holds.
k: The keyword.
returns: (Some value) for the first entry under k, in written order, or None.

(bjodat-ref entries :version)

A linear scan comparing interned keywords by reference. That is the right shape for a record’s handful of fields and the wrong one for a map with thousands of keys used as a lookup table: if you want a dictionary, build one from the entries. Only a BjoKey key can match, so an entry under a symbol or a string is never found. A duplicate key: the first wins.

`open-map!`

function

(: open-map! (-> Reader string (Result string bool)))
(open-map! rd owner)

Steps over the brace that opens a map, and says which spelling it is in. Not an API for people.

rd: The reader.
owner: The record's name, for messages.
returns: (Ok #t) when the entries are parenthesised, (Ok #f) when flat; a refusal when there is no map here.

Decided by the first thing inside the brace, as the tree reader decides it; every entry after has to agree.

`next-key!`

function

(: next-key! (-> Reader string bool (Result string bool)))
(next-key! rd owner paired)

Takes the next key’s name into the buffer, without interning it. Not an API for people.

rd: The reader.
owner: The record's name, for messages.
paired: What open-map! answered.
returns: (Ok #t) when a key is in the buffer and the reader stands at its value, (Ok #f) when the map closed.

A key that is not a keyword is read and dropped, and leaves the buffer empty, which no field name can equal: it takes the path an unknown key takes, as in the tree decoder, where bjodat-ref only matches a BjoKey.

See also: key=?, end-entry!, skip-value!

`end-entry!`

function

(: end-entry! (-> Reader string bool (Result string bool)))
(end-entry! rd owner paired)

Steps over the ) of a parenthesised entry; nothing in the flat form. Not an API for people.

rd: The reader.
owner: The record's name, for messages.
paired: What open-map! answered.
returns: (Ok #t), or a refusal when the entry holds more than a key and a value.

`key=?`

function

(: key=? (-> Reader string bool))
(key=? rd name)

Whether the name in the reader’s buffer is this one. Not an API for people.

rd: The reader.
name: A field name or tag, without its colon.
returns: #t when they are equal.

The comparison the one-pass decoder exists for: no string is built and nothing is interned, for a key, a clause name or a tag.

`field-value`

function

(: field-value (-> Reader (Result string Bjodat)))
(field-value rd)

Reads a field’s value, as a Bjodat for its type’s instance to convert. Not an API for people.

rd: The reader, standing at the value.
returns: The value, or a refusal.

The value is still read as a Bjodat and converted by the field type’s own bjodat-> instance. That keeps every instance, including one a user wrote, working, and every message identical to the tree decoder’s. The entry is what the one-pass decoder saves, not the value.

`skip-value!`

function

(: skip-value! (-> Reader (Result string bool)))
(skip-value! rd)

Reads an unknown key’s value and drops it. Not an API for people.

rd: The reader, standing at the value.
returns: (Ok #t), or a refusal.

It has to be read: a value is not a token, and finding where it ends means parsing it.

`open-form!`

function

(: open-form! (-> Reader string string (Result string bool)))
(open-form! rd owner tag)

Steps over ( and a tagged form’s head, refusing any head but the tag. Not an API for people.

rd: The reader.
owner: The record's name, for messages.
tag: The head the record is written under.
returns: (Ok #t), or a refusal such as expected (package ...), and this is (banana ...).

The tag is matched against the buffer, so a form costs no string for its own name; one is built only on the way to refusing.

`next-field!`

function

(: next-field! (-> Reader string (Result string bool)))
(next-field! rd owner)

Takes the next clause’s field name into the buffer. Not an API for people.

rd: The reader.
owner: The record's name, for messages.
returns: (Ok #t) when a name is in the buffer and the reader stands at the clause's first value, (Ok #f) when the form closed.

`end-clause!`

function

(: end-clause! (-> Reader string string (Result string bool)))
(end-clause! rd owner name)

Steps over the ) of a clause holding one value. Not an API for people.

rd: The reader.
owner: The record's name, for messages.
name: The field's name, for messages.
returns: (Ok #t), or a refusal when the clause holds more than one value.

The owner and the field name arrive apart and are joined only on the way to refusing: joined by the caller they would be a string built per field per record.

`skip-clause!`

function

(: skip-clause! (-> Reader (Result string bool)))
(skip-clause! rd)

Reads an unknown clause to its ), whatever it holds, and drops it. Not an API for people.

rd: The reader, after the clause's name.
returns: (Ok #t), or a refusal.

Any arity, and that is the point of it. A map’s unknown key is one value and done, but a clause may hold none, one or five, and a reader that insisted on one would make every new clause a breaking change for every older tool. This is what keeps a manifest forward-compatible.

`clause-values!`

function

(: clause-values! (-> Reader (Result string (List Bjodat))))
(clause-values! rd)

Reads every value of a clause, for a #:rest field, and its ). Not an API for people.

rd: The reader, after the clause's name.
returns: The values in written order, or a refusal.

The values are read as Bjodat and converted by the element type’s own instance. read-value carries the depth budget, so a manifest nesting (package (depends (package (depends ...)))) five hundred deep is refused by bjodat-max-depth rather than by the stack. A #:rest that recursed straight back into a generated decoder would have no budget at all.

`read-one-pass`

function

(: read-one-pass (-> TextInputPort (-?-> Reader (Result string %a)) (Result string %a)))
(read-one-pass port decode)

Runs a one-pass decoder over a port: one record, and then the end.

port: The port to read from, to its end.
decode: A generated bjodat-read-<Name>.
returns: What the decoder answered; text after the record is refused.

The decoder is typed -?->, so that a bjoroutine calling this gets the decoder’s suspending copy rather than the one that parks the thread. Pass it by name, not wrapped in a (fun (rd) ...): a lambda body is sealed to the sync copy of what it calls.

`parse-one-pass`

function

(: parse-one-pass (-> string (-?-> Reader (Result string %a)) (Result string %a)))
(parse-one-pass text decode)

Runs a one-pass decoder over a string: one record, and then the end.

text: The document.
decode: A generated bjodat-read-<Name>, passed by name.
returns: What the decoder answered; text after the record is refused.

A generated bjodat-parse-<Name> is exactly this with its own decoder.

`read-one-pass-all`

function

(: read-one-pass-all (-> TextInputPort (-?-> Reader (Result string %a)) (Result string (List %a))))
(read-one-pass-all port decode)

Runs a one-pass decoder over each record of a […] read from a port.

port: The port to read from.
decode: A generated bjodat-read-<Name>, passed by name.
returns: The records in order, or the first refusal. The document has to be a vec.

A List rather than a Vec: it is built by consing and turned round once, and a caller that folds over it once, which is most of them, should not pay for a tree it does not index into.

`parse-one-pass-all`

function

(: parse-one-pass-all (-> string (-?-> Reader (Result string %a)) (Result string (List %a))))
(parse-one-pass-all text decode)

Runs a one-pass decoder over each record of a […] in a string.

text: The document, a vec of records.
decode: A generated bjodat-read-<Name>, passed by name.
returns: The records in order, or the first refusal.

(parse-one-pass-all text bjodat-read-Dep)   ; (Result string (List Dep))

Values

`bjodat-max-depth`

value

(: bjodat-max-depth int)

How deep a value may nest before the reader refuses it: 200.

It applies to nesting in the reader, so a hostile document nested a thousand deep is refused with a message rather than by the stack. Each record a generated one-pass decoder reads starts a fresh budget, so a chain of nested record types is bounded by the types rather than by this; the values inside a field are bounded normally.