Architecture¶
This page explains how occamyd is built: its parts, how a key press
travels from a client to the remote computer, and how the remote's
screen travels back. Read it to find your way around the code before
you change or review it.
How occamyd works¶
occamyd is one Go program between your users and the computers they
reach. Its HTTP server signs users in and serves the
HTTP API. Each connection that a user opens starts a
session, whose backend connects to the remote computer in its own
protocol: VNC, RDP, SSH or the Kubernetes API. For each client that
watches the session, a viewer, a stream carries the session over
protocol v1.
flowchart LR
client["Client"]
subgraph occamyd
http["HTTP server"]
stream["Stream<br>per viewer"]
session["Session"]
backend["Backend"]
end
remote["Remote"]
client -- "API" --> http
http -- "upgrade" --> stream
client <-- "protocol v1" --> stream
stream <--> session
session <--> backend
backend <-- "VNC, RDP, SSH<br>or Kubernetes" --> remote
A session owns the remote's screen, which the backend draws into: a framebuffer of pixels for a desktop, or a terminal emulator for a shell. A stream never forwards the remote's messages. Instead, each viewer's writer sends only the difference between the screen and what that viewer was last sent, at the pace of the viewer's own connection, so viewers who share a session don't hold each other up.
This is the path of a key press over VNC:
sequenceDiagram
participant C as Client
participant R as Reader
participant S as Session
participant B as Backend
participant X as Remote
participant F as Framebuffer
participant W as Writer
C->>R: key, seq 42
R->>S: input 42
Note over S: Records 42 and the<br>framebuffer's clock
S->>B: the key
B->>X: a key event, in RFB
X-->>B: a screen update
B->>F: draws: each tile it touches<br>takes a new version
F-->>W: the screen changed
Note over W: The tiles newer than the clock<br>are the key's effect: they go<br>first, in a frame of their own
W->>C: tiles, then frame-end, last_seq 42
C->>R: frame-ack, with its decode time
R-->>W: the ack, for the pacer
The session records the key's sequence number and the framebuffer's clock before the backend sees the key, which is how the writer tells the key's effect from other changes. Those tiles go even when the viewer's frame window is full.
Why it's built this way¶
Apache Guacamole reaches remotes through guacd, a daemon written in C, and talks to the browser in a text protocol of drawing instructions. Occamy differs in four ways:
- A terminal travels as text. guacd sends pictures of the rendered
text.
occamydsends the changes to the terminal's screen, and the client draws the text, so you can select it, and bulk output costs what changed on the screen rather than every line that scrolled past. - A desktop travels as binary tiles. Guacamole carries each image
in base64, a third larger than the image.
occamydsends only the tiles that changed, each in the codec that suits it: content in motion at half size, a scroll as a copy of pixels the client has, and a tile the client has seen by its cache slot. - Input comes first. Small messages, such as the cursor or the clipboard, never queue behind tiles, and the frame that shows an input's effect goes before any other.
- One process. The VNC, RDP, SSH and Kubernetes clients run inside
occamyd, a static Go binary without C for Linux, macOS and Windows.
A session's life¶
A session starts when its owner opens
a connection's stream. occamyd checks the token and
the connection's allow list, starts the session and its backend, and
sends welcome at once, while the backend connects. The first frame
follows once the backend has drawn the remote's screen. When the remote
asks for a credential that the configuration lacks, the backend asks
the owner through the stream and waits for the answer.
A session that POST /api/v1/sessions
opens waits for its owner's stream, as it does when the owner goes
away:
stateDiagram-v2
direction LR
state "Owner present" as present
state "Owner away" as away
[*] --> present: opened by a stream
[*] --> away: opened by the API
present --> away: the owner leaves
away --> present: the owner returns
away --> Ended: 60 s pass
present --> Ended: a reason below
Ended --> [*]
A session also ends when its owner closes it, the remote ends it,
nobody gives input, the backend fails, or occamyd stops.
The life of a session gives the
close code that each viewer then receives,
and Lifetimes the timeouts. Other viewers join with
a share, each through a stream of its own.
The parts of occamyd¶
The HTTP server¶
server serves API v1 under /api/v1/, the health check
at /healthz, and the web client under /static/ when web_client
names its directory. It speaks HTTP/1.1 only, because a stream is a
WebSocket that occamyd upgrades by taking over an HTTP/1.1
connection, and it terminates TLS when the configuration names a
certificate.
The two stream routes authenticate on their own, because a browser
can't set Authorization on a WebSocket, and refuse after the upgrade,
because a browser can't read the status of a refused upgrade
(Streams). The server enforces the configuration's
limits, and on Linux sets the TCP congestion control
that congestion_control names on its listener.
The debug listener, which serves expvar and pprof, is a separate server
on its own address. On SIGINT or SIGTERM, occamyd drains the
requests in flight for up to five seconds, and a second signal ends it
at once.
Tokens and the configuration¶
internal/auth issues and checks tokens: JSON Web
Tokens signed with Ed25519 and no other algorithm, whatever a token's
header says. An access token names a user and the user's groups, and
a session token names a session and its holder's role. occamyd
signs both with its own key, which it creates on its first start, and
accepts tokens from the issuers that the configuration trusts.
internal/config reads the configuration file.
It refuses unknown keys, fills in the defaults, and checks the whole
file before occamyd starts, including that the files and environment
variables it names exist, without reading any secret. It reports every
problem at once. Configuration describes every key.
Sessions¶
internal/session turns a backend's connection
into a session that any number of viewers share. One goroutine owns
each session: it runs the backend, applies the viewers' input in the
order it arrives, drops a read-only viewer's input, tells the backend
whether anyone can see the screen, and ends the session.
The session holds the small state that every viewer sees, such as the cursor's image, the title and the clipboard, and keeps only the latest value of each, so a slow viewer receives the newest title once instead of a queue of old ones. Its registry holds the running sessions by ID, and their shares by the hash of each share's token, so a share dies with its session.
Backends¶
A backend connects to the remote, draws into the session's framebuffer
or feeds its terminal emulator, and applies the viewers' input; it
never encodes anything for a viewer. Every backend implements the
interface in internal/backend:
| Method | What it does |
|---|---|
Run |
Connects and serves the remote. It reports what the remote does through the sink that the session gives it: Resize, Lock and Unlock around the pixels it draws, TermData for a terminal's output, the cursor, the clipboard, the title, the bell, and Prompt for a missing credential |
Input |
Applies a viewer's input |
Demand |
Says whether any viewer can see the screen, so that the backend asks the remote for updates only while someone looks |
Each backend is a client written in Go, without C:
| Backend | What it speaks |
|---|---|
internal/backend/vnc |
RFB 3.3, 3.7 and 3.8, with the encodings Tight, ZRLE, Hextile, Raw and CopyRect. It also speaks VeNCrypt with TLS, Extended Desktop Size for resizing, Continuous Updates, QEMU's extended key events, the extended clipboard in UTF-8, and Apple Remote Desktop's sign-in when the connection allows it |
internal/backend/rdp, on internal/rdp |
RDP over TLS with NLA, bitmap updates in the interleaved RLE and planar codecs, the graphics pipeline with RemoteFX and RemoteFX Progressive, the clipboard's channel for text, and Display Control for resizing. It doesn't decode ClearCodec, AVC or the pipeline's cache. Windows draws most of its screen with ClearCodec and the cache, so a Windows host needs disable_gfx: true. Decoding takes at most the cores that limits.decode_cores gives all RDP sessions together |
internal/backend/ssh |
SSH, with a shell in a pseudo-terminal. It accepts only a pinned host key unless the connection ignores host keys, and asks the owner for the answers its configuration lacks |
internal/backend/kubernetes |
A command in a pod's container, or the container's main process, through the API server's exec and attach over WebSockets. It reaches the API server over TLS only, and checks the server's certificate before it sends anything |
The VNC and RDP backends share internal/keymap,
which turns the USB HID usage of the physical key that a client reports
into the scan code that both protocols carry. Both check every length
that the remote sends against a limit before they read or allocate
anything for it.
The screen¶
internal/framebuffer holds a desktop's
screen as RGBA pixels in tiles of 64 by 64. Every change to a tile
gives it the next version of one counter for the whole screen. A
writer finds the tiles a viewer hasn't seen by comparing their versions
with those it last sent that viewer, and an input's effect by comparing
them with the counter's value when the input arrived. Each row of tiles
has a lock of its own, so a backend that draws one part of the screen
and writers that read another don't wait for each other.
internal/terminal is a headless terminal
emulator that follows xterm. Each writer takes a snapshot of its
screen, which never changes once taken, and sends the viewer the
difference from the snapshot it last sent, as escape sequences. The
emulator keeps no scrollback: lines that leave the top of the screen go
to the clients as scrolled lines, and the clients keep them.
A viewer's stream¶
internal/stream serves one viewer on two
goroutines. The reader turns the client's messages into input, the
viewer's visibility, and frame acknowledgements. The writer sends the
session's screen as frames, through the stages that this figure shows:
Only the dirty tiles in the viewer's viewport count. To find scrolls, the writer keeps a copy of the pixels its client shows, and it mirrors the slots of the client's tile cache.
The viewers of a session share an encode cache: a tile's encoding is made once for every viewer whose client accepts the same codecs, by whichever writer or worker gets there first. A session's two encode workers, fewer on a machine with fewer than four cores, encode the rows of a frame ahead of the writers.
When nothing changes, the writer refines: tiles that went at a lower quality or at half size go again, exactly or at the quality they need, once they have been still for 300 ms. Refinement goes a little at a time, and gives way at once to any input or change.
A viewer that stalls for the viewer timeout is
closed with too-slow, so that it can't hold its last screen forever.
Encoding tiles¶
internal/encoder chooses each tile's codec by
counting its colours, by the rules of
What occamyd sends. It sends a run of
up to 16 lossy tiles side by side as one image, at a quality that
depends on what the tiles show:
| Lossy tiles | Quality |
|---|---|
| In motion | The pacer's lowest, and at half size for a client that accepts the scaled codec |
| An input's effect | The pacer's current, without a search, so that the effect isn't delayed |
| Other still tiles | The lowest, up to the pacer's current, at which each tile reaches a structural similarity (SSIM) of 0.91 against its pixels |
Every encoder is pure Go and runs on the writer's goroutine, so one viewer's encoding takes at most one core. Codecs shows the measurements behind these choices.
internal/codec holds the formats that Occamy
defines, raw pixels and the deflate container, both ways, and the
scaling of the scaled codec. It uses the standard library only, so a
client can decode with it without linking the server's lossy encoders.
Pacing¶
internal/pacer decides when a writer starts the
next frame, when it writes the next bytes, and at what lossy quality.
It models the viewer's network path from the frame acknowledgements, as
TCP's BBR does from its own: the delivery rate is the highest rate
measured recently, and the path's delay the shortest recent round trip.
The pacer then:
- writes at about the delivery rate, so that data doesn't queue in the network;
- keeps the bytes in flight to about twice the product of that rate and delay;
- lets as many frames await an acknowledgement as cover the time an acknowledgement takes, up to the agreed frame window;
- sends the frame that carries an input's effect at once, unpaced, even beyond a full window;
- steps the lossy quality down from 90 toward 30 while the writer spends most of its time waiting for the path, and back up while it doesn't, so motion loses detail before input loses responsiveness.
A pacer does no I/O and reads no clock: every method takes the current time, so its tests drive it on simulated networks. Frames and flow control explains what the client sees of this.
The protocol package¶
protocol implements protocol v1 as a public package:
each message as a Go type, the handshake, and the decoding rules. It is
the reference implementation of the protocol page, whose
test vectors are checked against it.
The clients¶
| Client | Where | How it's built |
|---|---|---|
| The web client | client/occamy-web |
TypeScript and React, built with Vite. It decodes tiles in a Web Worker, applies each frame to a framebuffer in memory, and draws the changed parts on a canvas at the next animation frame. Then it acknowledges the frame. xterm.js draws terminals |
| The Go client | client, with client/decoders |
A public package that opens a session, applies frames, and acknowledges them as the caller takes them. The end-to-end tests, the benchmark and Occamy for macOS use it |
| Occamy for macOS | client/occamy-swift |
SwiftUI, on the Go client built as a C library, liboccamy.a. SwiftTerm draws terminals |
Tests and tools¶
| Path | What it holds |
|---|---|
tests |
The end-to-end, pressure and soak tests, which run against the demo stack (Development) |
client/internal/webinterop |
A server of occamyd's stream code, against which the web client's interop tests run |
bench |
The benchmark harness, with Guacamole as the baseline (Benchmarks) |
bench/codecs |
The measurement of the tile codecs, a Go module of its own (Codecs) |
Design¶
The specs record how each part was designed, the options weighed, and
what was built. The native architecture
covers occamyd as a whole, and the index lists
every spec:
| Part | Spec |
|---|---|
| API v1 and the server | API v1 and server stack |
| Sessions and the backend interface | Session core and backend interface |
| The framebuffer, codecs, scroll detection, the tile cache, the encode cache and refinement | Graphics pipeline |
| Pacing and the frame window | Real-time pacing and transport |
| Protocol v1 | Occamy protocol v1 |
| The Go client | Go client core |
| The backends | VNC, RDP, SSH and Kubernetes |
| The web client | React web client |
| Occamy for macOS | Native Swift client on a Go core |