Joseph has been writing software since 1998. He was an early Java adopter, moved through Ruby, Go, and Elixir, and spent the last decade in security engineering: leading detection and response at Marqeta, where his team's real-time SIEM processed 920 GB of logs a day, and building Instacart's unified security data lake as a Staff Security Engineer. In 2026 he co-founded an AI startup built on Elixir/OTP.
Rust came from a real constraint. His product needed a storage and caching layer for video analysis data at a scale where garbage-collected runtimes start to hurt, and he wanted OTP-style resilience without shipping the BEAM VM. He had already tried Axum on his own and stalled. AI assistants did not help either: in Java he could tell when a model was confidently wrong, in Rust he could not. So he wrote every line of the Rust himself, on a real product with real constraints.
The project is a CMS platform for nonprofit organizations (events, campaigns, donations, monthly pledges), built on Axum, ractor, Askama, and htmx, with Postgres behind it. Small enough to finish, hard enough to force every concept that matters. The goal was a production codebase he could put in his consulting portfolio and defend line by line, not a tutorial repo.
The interesting part was never the syntax. Joseph already thought in processes, supervisors, and mailboxes, and ractor gives Rust the same vocabulary. What it does not give is the BEAM runtime underneath: cheap crashes, restarts from plain data, untyped messages, process monitors. Every one of those had to become an explicit decision in his code, and every decision is written down in the module docs of the repo, with the reasoning and the alternative he rejected.
ractor
Supervision strategy chosen per actor
Postgres RLS
Tenant isolation enforced by the database
htmx + WS
Live admin UI with no NPM toolchain
Inline
Publish pipeline after reverting a Lambda split
The challenge
OTP instincts, no BEAM underneath
In Elixir, a crashed process is restarted by its supervisor from a child spec, and the rest of the system finds it again by name. In ractor, a re-spawned actor gets a new id, so every ActorRef the rest of the system holds now points at a dead actor. “Just restart it” quietly becomes “restart it and rebuild a name registry”.
Worse, the live session actor owns the WebSocket write half, which cannot be cloned or reconstructed. Restarting that actor restores nothing the user can see. For an OTP engineer the hard part was not learning actors, it was noticing which defaults no longer exist and deciding each one explicitly.
•Restarts that invalidate every reference to the restarted actor
•No process monitors, so dead subscribers are discovered on send
•Two architecture pivots in three days, mid-build
The outcome
Every trade-off written into the code
The root supervisor now terminates on any child panic and lets the process watchdog restart the binary in under a second. Same user impact as an in-process restart (live sockets drop either way), no stale references, and the restart machinery deleted instead of maintained. The module doc says why, and what would justify going back.
The rest of the system follows the same rule. Tenant isolation lives in Postgres, not in handler code. The publish pipeline is an in-process crate behind a typed job boundary that can be split out again the day traffic justifies it. The key calls are documented in the code with the cost they avoid and what would justify revisiting them.
✓Tenant isolation enforced by Postgres RLS on every tenant table
✓Supervision strategy chosen per actor, not inherited
✓A distributed design he reverted when it stopped paying off
[ Let it crash ]
When crashing stops being free
In OTP, a crash is the recovery strategy: the process dies, the supervisor restarts it, nobody writes defensive code. Joseph carried that instinct into ractor, where a ? inside an actor handler turns a transient database error into actor death. The catch is what dies with it: the auth actor kept every admin session in its own state, so one flaky query would have logged out every admin at once.
The fix was to split failures by kind. The Settings, Content, and Contact actors now send expected errors back through the reply port and keep crashing for actual bugs. Session expiry moved from lazy on-access checks to a send_interval timer that sweeps the map, and unknown usernames are verified against a dummy hash so login timing does not leak which accounts exist.
[ Supervision ]
A supervisor without child specs
An OTP child spec is plain data, module plus arguments, so restarting a child is trivial. In ractor, re-spawning a crashed actor means storing a boxed async factory with exact Send + Sync + 'static bounds, or an enum that couples the supervisor to every child type. Worse, the re-spawned actor gets a new id, so every ActorRef held elsewhere goes stale.
Joseph made the call a senior engineer should make: the root supervisor terminates on any child panic and lets the process orchestrator restart the service in under a second. Same user impact as an in-process restart, without ChildSpec machinery or a name registry. LiveSession actors get the opposite policy. Each one owns a WebSocket write half that cannot be cloned or rebuilt, so restarting it is meaningless: the supervisor only logs, and htmx reconnects to spawn a fresh session.
[ Typed mailboxes ]
No more sending any term to any process
Elixir lets any process receive any term, and Phoenix LiveView computes DOM diffs for you. In Rust, every ActorRef is typed on its message, and the htmx layer is hand-rolled, so both conveniences had to become explicit design decisions.
For PubSub, type erasure through ActorCell was on the table; Joseph kept it concretely typed to LiveSession messages because those are the only subscribers, and documented the swap path for later. Without process monitors, a failed cast is the only signal a subscriber died, so dead entries are evicted on the next publish. And since each Askama fragment type must match its hx-swap-oob target id by convention, a drift renders fine and silently updates nothing.
[ Multi-tenancy ]
Tenant isolation the database enforces, not the handlers
When the project pivoted to multi-tenant SaaS, Joseph did not thread a tenant_id filter through every query and hope nobody forgot one. He put Postgres row-level security on every tenant table, with a policy that reads the tenant from the transaction itself. A query that forgets the tenant returns nothing instead of another nonprofit's donors.
The detail that matters: the tenant is set per transaction with set_config('app.tenant_id', $1, true), not SET LOCAL, because SET LOCAL cannot take a bind parameter and string-formatting a tenant id into SQL is exactly the bug RLS exists to prevent. Transaction scope also means a pooled connection can never carry one tenant's context into the next request.
[ Architecture ]
Reverting his own Lambda split
With the SaaS pivot, rendering moved into a separate publisher Lambda. Shortly after, Joseph reverted it to an in-process library crate. At nonprofit traffic, the split bought nothing and cost a custom Docker runtime, HMAC-signed transport, a second deploy pipeline, and dev/prod divergence. The render job types stayed as the function-call boundary, so a future re-split does not redo the type design.
Inside that boundary the pipeline got serious: tenants render in parallel on a JoinSet capped by a semaphore, assets go through a content-hashed manifest, uploads to object storage skip unchanged files by comparing the stored MD5 ETag from a HEAD request, Cache-Control is set per call through a CachePolicy, and the CDN cache is purged explicitly after publish.
The journey
One clear path through the build
Step 011 / 5
Week 1
Workspace split into domain, actors, web, publish, and server crates, a written design doc, and the full Postgres schema before any handler code.
Step 022 / 5
Week 2
Admin auth middleware, sliding-window sessions, htmx layouts, and the WebSocket LiveSession actor with heartbeat and clean shutdown.
Step 033 / 5
Week 3
Actor errors split by kind (reply port for expected failures, crash for bugs), root supervisor simplified to terminate on child panic, and the Content actor.
Step 044 / 5
Week 4
Double-submit CSRF middleware, then two pivots back to back: static publish to a CDN, and multi-tenant SaaS with row-level security on every tenant table.
Step 055 / 5
Week 5-6
Parallel render pipeline, content-hashed asset manifest, ETag-skip uploads, explicit CDN purge, and strict URL validation against stored XSS.
Drag to explore · Select a milestone to read the full story