Shamsher Singh

POST /chats/{chat_id}/reply/stream returns Server-Sent Events built with axum::response::sse and async-stream. Upstream, OpenAI's own SSE response is read through bytes_stream() and eventsource-stream, so each token is forwarded as a typed delta, error or done event.

Malformed chunks, role-only chunks and empty deltas are skipped by a small extract_delta parser with its own unit tests. When the client stops generating, the dropped connection ends the stream future on the server, and whatever text already arrived is persisted as the assistant message instead of being discarded.

Sending a message and generating the reply are two steps. If the user message was already persisted when the reply failed, retry re-requests only the reply, so the conversation never shows the same prompt twice.

The model sees a bounded window of recent messages rather than the whole history, which keeps cost and latency predictable on long chats. Chats sort by an updated_at bumped on every message, and deleting a chat cascades its messages while detaching its documents instead of destroying them.

Every primary key is a Unid<Pk>: a UUID tagged with a phantom marker type. Passing a chat id where a user id is expected becomes a compile error instead of a silent wrong-row query.

Unid implements sqlx's Type, Encode and Decode for Postgres plus serde, so the wrapper costs nothing at the storage or API layer. Permissions go through an access-control trait checked against the authenticated user extracted from a JWT.

The browser sends raw 16 kHz f32 PCM. The server validates the byte length, then transcribes through Groq when an API key is set, or through whisper-rs locally when the local-whisper cargo feature is enabled.

Local inference runs in spawn_blocking so the CPU-heavy model never stalls the async runtime, and the real-time factor is logged per request. Because transcription and LLM calls cost money, both endpoints carry their own quotas, keyed per IP for anonymous callers and per user once authenticated.