Skip to content

http.md

Latest commit

 

History

History
259 lines (212 loc) · 15 KB

http.md

File metadata and controls

259 lines (212 loc) · 15 KB
Raw

http - an HTTP/1.1 client

Import with import "http.j" as http;. An HTTP/1.1 client over the net system library: build a request (method, URL, headers, body), send it, and read the response back into a Response (status, headers, body). http:// connects in the clear; https:// connects with TLS (net.connectTLS). Because it uses net, this module needs the default jennifer binary.

On jennifer-tiny: "needs the default jennifer binary" refers to the stock tiny build, which ships without a network driver - not a TinyGo limitation. A jennifer-tiny rebuilt with a network stack runs this module too; see the note on net and TinyGo.

import "http.j" as http;

def r as http.Response init http.get("http://example.com/", {});
io.printf("status %d\n%s\n", $r.status, $r.body);

def sent as http.Response init http.post("https://api.example.com/items",
    "application/json", "{\"name\":\"ada\"}", {"Authorization": "Bearer xyz"});

Runnable: examples/modules/http_demo.j.

Surface

headers is a map of string to string (pass {} for none); a body is a string ("" for none).

Call / type Notes
http.Response status (int), statusText, headers (lowercased keys), body (string, UTF-8).
http.BytesResponse Same, but body is raw bytes - the byte-safe response for binary downloads.
http.TlsOptions skipVerify (bool), caCert (bytes, PEM). Zero value = full verification (what request uses).
http.request(method, url, headers, body) The general request (default idle timeout); returns a Response.
http.requestWith(method, url, headers, body, timeoutMs, maxBytes) As request, with an explicit per-read idle timeout (0 = none) and body cap (0 = 64 MiB default, negative = unlimited, positive = exact ceiling).
http.requestTls(method, url, headers, body, tls) As request, with explicit TlsOptions for an https:// server (self-signed / private CA).
http.requestWithTls(method, url, headers, body, timeoutMs, maxBytes, tls) As requestWith, with explicit TlsOptions.
http.requestBytes(method, url, headers, body) Byte-safe request: returns a BytesResponse (raw bytes body) - for binary content.
http.requestWithBytes(method, url, headers, body, timeoutMs, maxBytes, tls) As requestBytes, with explicit timeout / cap / TlsOptions (pass negative maxBytes for a large download).
http.requestRawBody(method, url, headers, body, timeoutMs, maxBytes) Send a raw bytes body byte-for-byte (a multipart/form-data file upload, a protobuf), not UTF-8-encoded; returns a text Response. Set your own Content-Type.
http.requestRawBodyTls(method, url, headers, body, timeoutMs, maxBytes, tls) As requestRawBody, with explicit TlsOptions.
http.getBytes(url, headers) GET returning a raw-bytes body (the download shortcut).
http.get(url, headers) GET.
http.post(url, contentType, body, headers) POST; sets Content-Type.
http.put(url, contentType, body, headers) PUT; sets Content-Type.
http.patch(url, contentType, body, headers) PATCH (partial update); sets Content-Type.
http.delete(url, headers) DELETE.
http.head(url, headers) HEAD (status + headers, no body).
http.options(url, headers) OPTIONS (capability probe; read the Allow header).
http.header(resp, name) Read a response header case-insensitively, or "" if absent.
http.basic(user, pass) Build a Basic <base64> Authorization value (mirrors rest.basic).
http.Options Request policy: timeoutMs, maxBytes, maxRedirects, maxRetries, backoffMs, tls, allowCrossOriginRedirect. Zero value = defaults, no redirects, no retries, and credentials dropped on a cross-origin redirect.
http.defaultOptions() The zero Options, for inline use.
http.send(method, url, headers, body, options) One request with policy: follow up to maxRedirects 3xx, retry a 429 / 5xx up to maxRetries with backoff, carry cookies across the redirect chain. A redirect to a different origin drops Authorization / Cookie and the cookie jar (the browser "sensitive headers" rule) unless allowCrossOriginRedirect is set. Returns a Response.
http.Session A persistent (keep-alive) connection to one origin, with a cookie jar.
http.Exchange response (Response) + session (Session) to thread onward.
http.connect(url, options) Open a persistent Session to the origin of url.
http.exchange(session, method, path, headers, body) One request over the reused socket; returns an Exchange.
http.close(session) Close a Session's connection.

The response body is bounded by default: http reads at most 64 MiB (MAX_BODY_BYTES) and raises a catchable error beyond it. The per-read timeout only bounds a stalled server, so this size cap is what keeps a hostile server (e.g. behind an untrusted URL a feed reader follows) from streaming an unbounded body to OOM. A caller that needs a larger (or unbounded) body passes an explicit maxBytes to http.requestWith - 0 keeps the 64 MiB default, a negative value lifts the cap for a trusted large download, a positive value sets an exact ceiling. The s3 module (object storage) already lifts it; the verb shortcuts (get / post / ...) and request keep the default.

The shortcuts are thin wrappers over request, which is method-agnostic - it sends whatever method string you pass. So a method without a shortcut still works: http.request("TRACE", url, {}, "") and the like. The one method that is not supported is CONNECT: it is the HTTP tunneling primitive (after a 200 the socket becomes a raw bidirectional tunnel), which needs a connection hand-off this request/response-then-close client does not do.

Request policy: redirects, retries, cookies

http.send is the one-shot verbs plus a policy, configured by an http.Options. The zero Options (http.defaultOptions()) behaves exactly like request; set a field to opt in:

def o as http.Options;
$o.maxRedirects = 5;   # follow up to 5 3xx redirects
$o.maxRetries = 3;     # retry a 429 / 5xx up to 3 times
$o.backoffMs = 250;    # first backoff (doubles per attempt), honours Retry-After
def r as http.Response init http.send("GET", "https://example.com/", {}, "", $o);
  • Redirects. A 301 / 302 / 303 / 307 / 308 with a Location is followed up to maxRedirects hops (0 returns the 3xx as-is). A 303 (and a 301/302 on a POST) becomes a bodyless GET; 307 / 308 preserve the method and body. Cookies set along the chain are replayed on later hops.
  • Retries. A 429 or 5xx is retried up to maxRetries with exponential backoff (backoffMs * 2^attempt), raised to a numeric Retry-After when the server sends a larger one, and clamped to 30s.

For Basic auth, http.basic(user, pass) builds the header value:

def r as http.Response init http.get(url, {"Authorization": http.basic("ada", "s3cret")});

Persistent connections (keep-alive)

http.connect opens a Session to one origin (scheme + host + port) and reuses the socket across exchange calls, so a request loop pays a single handshake instead of one per request. The Session is value-semantic, but its underlying net.Conn is a shared handle - thread the returned session forward:

def s as http.Session init http.connect("https://api.example.com", http.defaultOptions());
def x1 as http.Exchange init http.exchange($s, "GET", "/items?page=1", {}, "");
$s = $x1.session;                 # carry the reused socket + any cookies
def x2 as http.Exchange init http.exchange($s, "GET", "/items?page=2", {}, "");
$s = $x2.session;
http.close($s);

exchange maintains a small cookie jar on the session (name -> value; it preserves multiple Set-Cookie lines and replays them as one Cookie header) and reconnects transparently if the server closed the connection. It does not follow redirects (a redirect can cross origins and break the socket) - use http.send for that; exchange returns the 3xx.

The jar is deliberately small: no domain / path scoping and no expiry (a session-scoped name -> value store). A full RFC 6265 jar is a follow-up.

URLs and headers

A URL is parsed into scheme / host / port / path: http:// defaults to port 80, https:// to 443, an explicit :port overrides, and the path (with any query string) defaults to /. The Host header is set automatically (with the port when non-default), along with Connection: close and a default User-Agent (overridable by supplying your own).

Response header names are lowercased (HTTP header names are case-insensitive), so $r.headers["content-type"] works regardless of how the server cased it; http.header($r, "Content-Type") does the case-folding for you.

Response body and framing

The client frames the response from its own headers (RFC 9112 6.3), so a request returns as soon as the body is complete - it does not wait for the connection to close, even though it sends Connection: close. A server that ignores that token and holds the socket open (some appliances and proxies) works the same as one that closes. Both framings are handled:

  • Content-Length - the body is taken as exactly that many bytes.
  • Transfer-Encoding: chunked - the chunks are decoded and concatenated.
  • Neither - the body runs to connection close (the last-resort framing).

request and the verb shortcuts return the body as text (UTF-8): a JSON / HTML / XML body round-trips exactly, but a binary body (an image, a gzip stream) is not valid UTF-8 and raises an error. For binary content use the byte verbs below, which return the body as raw bytes - the framing above is byte-exact, so only the text decode differs.

Binary downloads

http.getBytes(url, headers) (and the general requestBytes / requestWithBytes) return an http.BytesResponse whose body is raw bytes - the byte-safe path for a .tar.gz, an image, or any non-text payload the string Response cannot hold:

use fs;
use compress;
use archive;
def r as http.BytesResponse init http.getBytes("https://example.com/app.tar.gz", {});
fs.writeBytes("app.tar.gz", $r.body);                  # exact bytes, no corruption
def entries as list of archive.Entry init archive.unpack(compress.unpack($r.body, "gzip"), "tar");

headers keys are lowercased, so read metadata directly ($r.headers["content-type"]). A large download that would exceed the default 64 MiB cap uses requestWithBytes(..., maxBytes, tls) with a negative maxBytes (unbounded), and the same call carries TlsOptions for a self-signed host.

Timeouts

The initial connect (and, for https, the TLS handshake) is bounded by a connection-establishment timeout, so a slow or unreachable server fails the dial instead of blocking it forever. Every request then carries a per-read idle timeout (default 30 s): the deadline is re-armed before each read, so a server that accepts the connection and then stalls (or a hung endpoint) fails with a catchable read timed out error instead of blocking the caller forever. This is the difference between a slow dependency degrading one request and one exhausting your process on a pool of hung connections. Pass a different value (in milliseconds) with http.requestWith; a 0 disables the timeout for that request (e.g. a long streaming download):

try {
    def r as http.Response init http.requestWith("GET", url, {}, "", 5000);   # 5 s
} catch (e) {
    io.printf("request timed out or failed\n");
}

The timeout bounds each read, not the whole transfer, so a large but steady download is fine while a stalled one is cut off.

TLS (self-signed / private CA)

An https:// request full-verifies the server certificate against the URL host by default. To reach a server with a self-signed or private-CA certificate (a LAN appliance, an internal service), pass http.TlsOptions through http.requestTls (or requestWithTls for explicit timeout / cap). The zero TlsOptions verifies, so request and the verb shortcuts are unchanged; http:// ignores the options.

use fs;
def opts as http.TlsOptions;
$opts.caCert = fs.readBytes("appliance-ca.pem");   # trust this cert (preferred)
def r as http.Response init http.requestTls("GET", "https://192.168.1.10/", {}, "", $opts);
  • caCert (bytes, PEM) trusts a specific certificate in addition to the system roots. Preferred: the server stays authenticated.
  • skipVerify (true) accepts any certificate. It disables authentication and exposes the connection to a man-in-the-middle - use only for a trusted LAN endpoint you cannot give a proper CA. Mirrors net.TLSOptions.

Errors

A malformed response, a body that is not valid UTF-8, or a network failure raises a positioned error (a thrown Error for a malformed response, kind "http"; a read timed out error on an idle-timeout); wrap a request in try / catch to handle a down or slow server. A non-2xx status is not an error - a 404 or 500 comes back as a normal Response with that status, for the caller to branch on.

Out of scope

  • Redirects. The one-shot verbs return a 3xx as-is; http.send follows them (with a hop limit). No auto-follow on the persistent exchange path (it can cross origins).
  • Connection reuse is per-origin and single-request. connect / exchange keep one socket alive to one origin; there is no cross-origin connection pool and no request pipelining.
  • Cookie jar is session-scoped. Session keeps a name -> value jar with no domain / path scoping or expiry (a full RFC 6265 store is a follow-up); the one-shot verbs keep no cookies. No automatic decompression, no multipart builder - set the headers and body you need directly.
  • Binary bodies use the byte verbs (getBytes / requestBytes); the text path decodes UTF-8.

See also

  • net.md - the transport (and its TLS) http builds on.
  • json.md - encode / decode JSON request and response bodies.
  • modules/index.md - the module catalog and import rules.