(std http)

Overview

HTTP requests: a request is a value you build, then send blocking, suspending or as an event.

(import (std http))

(defun (main args)
  (match (fetch (get "https://example.com/fruit"))
    ((Err e) 1)
    ((Ok r)
     (println (response-status r))
     (println (response-text r))
     (response-close r)
     0)))

;; Streaming a body a line at a time, without holding it in memory.
;; fetch, port-eof? and read-line all suspend because this is a defbjo;
;; the same source blocks in a defun.
(defbjo (count-lines url)
  (match (fetch (get url))
    ((Err e) 0)
    ((Ok r)
     (let ((port (response-port r)))
       (let count ((n 0))
         (if (port-eof? port)
             (let () (response-close r) n)
             (let () (ignore (read-line port)) (count (+ n 1)))))))))

;; JSON is two modules rather than one: this one has no serializer.
(import (text json))

(defbjo (fetch-json url)
  (match (fetch (get url #:headers [("Accept" . "application/json")]))
    ((Err e) (Err "the request failed"))
    ((Ok r)
     (let ((parsed (read-json (response-port r))))
       (response-close r)
       parsed))))

A request is a value you build and then send. How it is sent, blocking, suspending, or as an event, is the only thing that differs between fetch and fetching.

There is no C# in this module: System.Net.Http is reached with import/extern and the #:async machinery, and the response body is an ordinary TextInputPort.

Two things are deliberately not what .NET does. A status code is a value, never an exception: 404 is an Ok holding a 404. And HttpClient.Timeout is switched off, because a deadline belongs to the event system (see below).

The ambient cancellation token reaches .NET by itself, so with-cancel and with-deadline stop a request in flight without anything being passed along.

Do not drain a body with port->seq. A seq body cannot hold an await, so its reads park the fiber’s thread, and the compiler says so. port->vec, port->chan and a loop like the one in the example are the suspending ways to drain a port.

What to watch out for.

Close the response, or leak the connection. It is read as headers only, so it owns a live socket until closed. ignore closes it, with-response closes it, and dropping it silently is a compile error.
response-port once. A second call hands out a second reader over a stream the first one is already draining.
A status is not an error. Nothing here throws on 4xx or 5xx. Check response-ok? yourself.
Timeouts are yours. HttpClient.Timeout is switched off, because it cannot be varied per request and arrives as a cancellation that cannot be told from a real one. Race a (timeout ms) branch, or use with-deadline.
A request that has to be losable is fetching, not fetch. fetch waits for one thing; only an event can be a branch of a choose.
A branch that loses after the headers have arrived leaks that response. The window is small, since losing cancels the token, but the value is dropped where nothing can close it. Do not race a request you cannot afford to lose twice.
Repeated headers are folded with ", ". That is wrong for Set-Cookie alone, and nothing here reads cookies yet.
UTF-8 is assumed when the server names no charset, names one that is not installed, or names nonsense.
Bodies are text. There is no bytes story yet, so an image or a zip has no type to arrive as.

Reference

Types

`Method`

union

An HTTP method: GET, POST, PUT, PATCH, DELETE or HEAD.

`Body`

union

What a request carries: nothing, text, JSON text, a form, or a body of your own.

Empty: No body.
Text: Plain text, sent as text/plain; charset=utf-8.
JsonText: Text that is already serialized JSON, sent as application/json; charset=utf-8.
Form: Fields, percent-encoded and sent as application/x-www-form-urlencoded.
Other: A Custom body: your own content type and text.

JsonText takes text that is already serialized; this module has no JSON writer. It is spelled that way rather than Json because (text json) has a type called Json.

Every body is sent as UTF-8.

`Custom`

record

A body with a content type of your own.

content-type: The Content-Type header to send, as written.
text: The body, sent as UTF-8.

(request PUT "https://example.com/note"
         #:body (Other (Custom (content-type "text/csv") (text "a,b\n1,2"))))

`Pair`

type alias

A name and a value: a header, a query parameter or a form field.

("Accept" . "application/json")

`Request`

record

A request, not yet sent.

method: The HTTP method.
url: The url, which must be http or https. It may already carry a query.
query: Query parameters, percent-encoded and appended to the url.
headers: Headers, for the request and for its body alike.
body: The body, or Empty.

See also: request, get, post, fetch

`Response`

record

An answer: a status, headers, and a body not yet read.

Opaque: the accessors are the whole of it. The CLR reply inside owns a live connection, so a response has to be closed, with response-close, ignore or with-response.

`Client`

type alias

A System.Net.Http.HttpClient: connections and their settings.

Functions

`request`

function

(: request (-> Method string (#:query (Vec Pair)) (#:headers (Vec Pair)) (#:body Body) Request))
(request method url #:query (vec-empty) #:headers (vec-empty) #:body Empty)

A request with any method.

method: The method.
url: Where to send it.
#:query: Query parameters, appended to the url.
#:headers: Headers, for the request or its body.
#:body: The body.
returns: The request, not yet sent.

(request PUT "https://example.com/note"
         #:body (Other (Custom (content-type "text/csv") (text "a,b\n1,2"))))

Query parameters are percent-encoded and appended; a url that already has a query gets & rather than ?.

A header goes on the request or on the body depending on which it is, Accept on the one and Content-Type on the other, and both are written in the same vector. Naming a body header on a request with no body is an Err when it is sent, as is a url that does not parse or is not http or https.

`get`

function

(: get (-> string (#:query (Vec Pair)) (#:headers (Vec Pair)) Request))
(get url #:query (vec-empty) #:headers (vec-empty))

A GET request.

url: Where to send it.
#:query: Query parameters, appended to the url.
#:headers: Headers.
returns: The request, not yet sent.

(get "https://example.com/search"
     #:query   [("q" . "banana split") ("limit" . "10")]
     #:headers [("Accept" . "application/json")])

`post`

function

(: post (-> string Body (#:query (Vec Pair)) (#:headers (Vec Pair)) Request))
(post url body #:query (vec-empty) #:headers (vec-empty))

A POST request with a body.

url: Where to send it.
body: The body.
#:query: Query parameters, appended to the url.
#:headers: Headers, for the request or its body.
returns: The request, not yet sent.

(post "https://example.com/basket"
      (JsonText "{\"fruit\":\"banana\"}")
      #:headers [("Authorization" . "Bearer hunter2")])

(post "https://example.com/login" (Form [("user" . "bjoli") ("pass" . "banan")]))

`fetch`

function

(: fetch (-> Request (Result Exception Response)))
(fetch req)

Sends a request and waits for the answer’s headers.

req: The request.
returns: Ok with the response, whatever its status, or Err if it could not be built or sent.

;; A deadline over a whole request, including the body. A bjoroutine,
;; because a scope waits for its children before it returns, and waiting
;; is a yield point.
(defbjo (main args)
  (with-deadline 5000
    (match (fetch (get "https://example.com/slow"))
      ((Ok r) (println (response-text r)) (response-close r) 0)
      ((Err e) (println "gave up") 1))))

Colourless: inside a bjoroutine it suspends and the thread goes back to the pool, outside one it parks the calling thread. There is no second name, and a function that calls it needs no colour of its own.

Failure is a value. A status code is not a failure: 404 is an Ok holding a 404. The response is closed by whoever receives it.

`fetching`

function

(: fetching (-> Request (Event (Result Exception Response))))
(fetching req)

A request as an event: sent when the branch is reached, cancelled if it loses.

req: The request.
returns: An event answering what fetch would.

;; Two mirrors, whichever answers first. The loser's request is
;; cancelled, not merely ignored.
(type (: Answer (Union (: Got (Result Exception Response)) TimedOut)))

(defbjo (from-a-mirror)
  (match (sync (choose (wrap (fetching (get "https://eu.example.com/fruit")) (fun (a) (Got a)))
                       (wrap (fetching (get "https://us.example.com/fruit")) (fun (a) (Got a)))
                       (wrap (timeout 2000) (fun (fired) TimedOut))))
    ((Got (Ok r))  (let ((body (response-text r))) (response-close r) body))
    ((Got (Err e)) "failed")
    (TimedOut      "no mirror answered")))

;; Retrying. A fresh request is built at every sync, so a retry loop is
;; just a loop.
(defbjo (with-retries req tries)
  (match (sync (fetching req))
    ((Ok r) (Ok r))
    ((Err e) (if (= tries 0)
                 (Err e)
                 (let () (sync (timeout 250)) (with-retries req (- tries 1)))))))

What to reach for when the request has to be a branch of a choose. Nothing is sent until the branch is reached.

The message is built afresh at every sync: a .NET request message may be sent only once, and an event may be synced any number of times, so the same event can be retried.

A branch that loses after the headers have arrived leaks that response, since nothing is left to close it.

`response-status`

function

(: response-status (-> Response int))
(response-status r)

The status code.

r: The response.
returns: The code, such as 200 or 404.

`response-ok?`

function

(: response-ok? (-> Response bool))
(response-ok? r)

Whether the status is 2xx.

r: The response.
returns: #t for a status from 200 to 299.

Nothing here throws on a 4xx or 5xx: which codes are failures is the caller’s business.

`response-reason`

function

(: response-reason (-> Response string))
(response-reason r)

The reason phrase, such as “Not Found”.

r: The response.
returns: The phrase, or the empty string over HTTP/2, which has none. Do not parse it.

`response-header`

function

(: response-header (-> Response string (Option string)))
(response-header r name)

One header of the answer.

r: The response.
name: The header's name, in any case.
returns: Its value, or None. Repeated headers are folded with ", ".

(response-header r "Content-Type")   ; the same as "content-type"

`response-headers`

function

(: response-headers (-> Response (Map string string)))
(response-headers r)

Every header of the answer, its body’s included.

r: The response.
returns: A map from lower-cased names to values. Repeated headers are folded with ", ".

Folding is wrong for Set-Cookie alone: folded cookies cannot be taken apart again.

`response-text`

function

(: response-text (-> Response string))
(response-text r)

The whole body, as text.

r: The response.
returns: The body, decoded by the charset the server named, or UTF-8.

Reads it through response-port, so it may be called once, and not after response-port. Colourless: reading suspends inside a bjoroutine and blocks outside one, exactly as reading a file does.

`response-port`

function

(: response-port (-> Response TextInputPort))
(response-port r)

The body as a port, to read as it arrives.

r: The response.
returns: A TextInputPort over the body.

So read-line, port->chan and read-json all work on it. Reading suspends inside a bjoroutine and blocks outside one, exactly as reading a file does.

Once per response: a second call hands out a second reader over a stream the first one is already draining.

The text is decoded by the charset the Content-Type names. UTF-8 is assumed when it names none, names one that is not installed, or names nonsense.

`response-close`

function

(: response-close (-> Response void))
(response-close r)

Releases the response’s connection.

r: The response.

A response holds a connection until it is closed. (ignore r) does the same, so a response dropped in statement position releases it rather than leaking it.

`make-client`

function

(: make-client (-> (#:connection-seconds double) (#:redirects bool) Client))
(make-client #:connection-seconds 120.0 #:redirects #t)

A new client, with settings of its own.

#:connection-seconds: How long a pooled connection lives before it is replaced.
#:redirects: Whether redirects are followed.
returns: The client. Responses are decompressed, and it has no timeout.

A pooled connection is recycled after connection-seconds: one that lived forever would outlast a DNS change and keep talking to an address that stopped being right.

HttpClient.Timeout is off on purpose. It applies to every call made through the client, cannot be varied per request, and arrives as a cancellation that cannot be told from a real one. Race a (timeout ms) branch, or use with-deadline.

Values

`current-http-client`

value

(: current-http-client (Param Client))

The client every send uses.

(parameterize ((current-http-client (make-client #:redirects #f)))
  (fetch (get "https://example.com/redirects-somewhere")))

A (Param Client). Parameterize another one in for a proxy, a test double or a second set of connection settings.

Macros

`with-response`

macro

(with-response (name form) body ...)

Runs body with a response, and closes it on the way out.

name: Bound to the (Result Exception Response).
form: What produces it, such as (fetch req).
body ...: What to run. At least one form.

(with-response (r (fetch (get "https://example.com/fruit")))
  (match r
    ((Ok got) (response-text got))
    ((Err e) "")))

The response is closed however the body leaves, by returning or by raising. An Err has nothing to close.