Building AI Applications with Rust and Rig in 2026

Rig is Rust's LLM orchestration framework: type-safe, async, and OpenAI/Anthropic/Cohere compatible. This guide covers LLM clients, RAG pipelines, agents, and tool use with Rig 0.6.

Rig brings LangChain-level orchestration to Rust: handling LLM clients, RAG pipelines, agents, and tool calling with the performance benefits of Rust (10x lower memory, zero GC pauses, sub-millisecond latency).

By Max Wells, updated June 2026

TL;DR: Rig is Rust's equivalent of LangChain: an LLM orchestration framework that handles model clients, embeddings, vector stores, RAG pipelines, and agents. It's async-first, type-safe, and supports OpenAI, Anthropic, Cohere, and local Ollama models. In 2026, Rust is the preferred language for production AI infrastructure due to its predictable latency, zero GC pauses, and memory efficiency.

• LLM clients: OpenAI, Anthropic, Cohere, Perplexity, Ollama: unified API
• Embeddings: same providers, embed_text() returns Vec<f32>: consistent interface
• RAG: rig::pipeline: chain extraction, embedding, vector search, and synthesis
• Agents: tool use with #[tool] macro: structured function calling
• Why Rust for AI: no GC pauses in inference hot paths, WASM deployment, sub-millisecond latency

Who Should Read This?

This guide is for backend engineers and AI engineers who want to build production AI services in Rust. You are comfortable with async Rust: you have used tokio, written a REST API, and understand futures. You are building something that needs predictable latency, low memory overhead, or WASM deployment, and you are evaluating whether Rust and Rig are the right foundation. You may be a senior engineer targeting AI infrastructure roles ($185K–$230K in the US), or a developer whose Python LangChain prototype works but cannot scale to production traffic. This guide covers Rig's core APIs with working code examples that you can adapt directly into a real project.

Why Use Rust for AI Applications?

Python dominates AI research, but production AI infrastructure increasingly runs in Rust: because AI systems need the same properties as other high-performance services: predictable latency, low memory overhead, and correctness under load.

The case for Rust in AI production:

Python remains best for model training and research. Rust is best for the infrastructure that serves those models: API servers, RAG pipelines, embedding workers, and real-time inference proxies.

The economic argument is straightforward: AI services that handle millions of daily requests benefit enormously from Rust's memory efficiency. A Python LangChain RAG server that requires 512 MB of RAM per instance costs roughly 8x more to operate at scale than an equivalent Rust Rig service running at 60 MB. At meaningful traffic volumes, the infrastructure savings justify the additional development time.

How Do You Set Up Rig?

Rig uses a provider-agnostic client model: configure a provider once and use the same API across OpenAI, Anthropic, and local models.

[dependencies]
rig-core = "0.6"
tokio = { version = "1", features = ["full"] }
serde = { version = "1", features = ["derive"] }

# Set API key (OpenAI)
export OPENAI_API_KEY="sk-..."
 
# For Anthropic
export ANTHROPIC_API_KEY="sk-ant-..."

use rig::{completion::Prompt, providers};
 
#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    // OpenAI client
    let openai = providers::openai::Client::from_env();
    let gpt4 = openai.completion_model(providers::openai::GPT_4O);
 
    // Basic completion
    let response = gpt4
        .prompt("What is Rust's ownership model in one sentence?")
        .await?;
    println!("{}", response);
 
    // Anthropic client (same API)
    let anthropic = providers::anthropic::Client::from_env();
    let claude = anthropic.completion_model(providers::anthropic::CLAUDE_SONNET_4_5);
 
    let response = claude
        .prompt("Explain Rust lifetimes in one sentence.")
        .await?;
    println!("{}", response);
 
    Ok(())
}

The provider-agnostic design is one of Rig's key advantages. You can prototype with one model provider and switch to another by changing a single line without touching the rest of your application code. This also makes A/B testing across providers straightforward: route 10% of traffic to Anthropic and 90% to OpenAI with identical application logic.

How Do You Build a Chat Agent with System Prompt?

Rig's agent() builder creates stateful conversations with system prompts, context injection, and tool use.

use rig::{completion::Prompt, providers};
 
#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    let openai = providers::openai::Client::from_env();
 
    // Create an agent with a system prompt and model configuration
    let rust_tutor = openai
        .agent(providers::openai::GPT_4O)
        .preamble("You are a Rust programming expert. \
                   Answer questions about Rust clearly and concisely. \
                   Always show code examples in Rust. \
                   If asked about other languages, redirect to Rust equivalents.")
        .temperature(0.3)  // Lower temperature = more deterministic, factual responses
        .max_tokens(1024)
        .build();
 
    // Single-turn prompt
    let answer = rust_tutor
        .prompt("What is the difference between String and &str?")
        .await?;
    println!("{}", answer);
 
    Ok(())
}

For multi-turn conversations with history:

use rig::completion::{Chat, Message};
 
let history = vec![
    Message::user("How does Rust handle concurrency?"),
    Message::assistant("Rust uses the Send and Sync traits..."),
];
 
let follow_up = rust_tutor
    .chat("Can you show me an Arc<Mutex<T>> example?", history)
    .await?;
println!("{}", follow_up);

The temperature parameter is worth tuning carefully for production AI applications. Values near 0.0 produce highly deterministic responses suitable for structured data extraction and code generation. Values near 0.7–1.0 produce more varied, creative responses suitable for writing assistance and brainstorming. Most information retrieval tasks perform best at 0.1–0.3.

How Do You Implement Tool Use with Rig?

Tool use (function calling) lets LLMs call Rust functions with type-checked arguments: the #[derive(Tool)] macro generates the JSON schema that the model uses to call your function.

use rig::{completion::Prompt, providers, tool::Tool};
use serde::{Deserialize, Serialize};
use serde_json::json;
 
// Define tool input/output types
#[derive(Deserialize)]
struct GetWeatherArgs {
    city: String,
    unit: Option<String>,
}
 
#[derive(Serialize)]
struct WeatherResult {
    city: String,
    temperature: f32,
    description: String,
}
 
// Implement the Tool trait
struct WeatherTool;
 
impl Tool for WeatherTool {
    const NAME: &'static str = "get_weather";
 
    type Error = String;
    type Args = GetWeatherArgs;
    type Output = WeatherResult;
 
    fn definition(&self, _name: String) -> rig::tool::ToolDefinition {
        rig::tool::ToolDefinition {
            name: Self::NAME.to_string(),
            description: "Get current weather for a city".to_string(),
            parameters: json!({
                "type": "object",
                "properties": {
                    "city": {"type": "string", "description": "City name"},
                    "unit": {"type": "string", "enum": ["celsius", "fahrenheit"]}
                },
                "required": ["city"]
            }),
        }
    }
 
    async fn call(&self, args: Self::Args) -> Result<Self::Output, Self::Error> {
        // Real implementation would call a weather API
        Ok(WeatherResult {
            city: args.city.clone(),
            temperature: 22.5,
            description: format!("Sunny in {}", args.city),
        })
    }
}
 
// Build agent with tools
#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    let openai = providers::openai::Client::from_env();
 
    let weather_agent = openai
        .agent(providers::openai::GPT_4O)
        .preamble("You are a helpful assistant. Use tools when needed.")
        .tool(WeatherTool)
        .build();
 
    let response = weather_agent
        .prompt("What's the weather like in Berlin right now?")
        .await?;
    println!("{}", response);
 
    Ok(())
}

The type safety of Rig's tool system is its key advantage over Python equivalents. In LangChain, a tool argument type mismatch surfaces as a runtime error in production. In Rig, the Args type is deserialized from the model's JSON output using serde: if the model sends an argument with the wrong type or an unexpected structure, the error surfaces before your tool function is called, and the agent can retry or report the failure cleanly.

How Do You Build a RAG Pipeline with Rig?

RAG (Retrieval-Augmented Generation) retrieves relevant documents from a vector store and includes them as context in the LLM prompt. Rig's pipeline chains these steps with a fluent API.

[dependencies]
rig-core = { version = "0.6", features = ["derive"] }
tokio = { version = "1", features = ["full"] }

use rig::{
    embeddings::EmbeddingsBuilder,
    providers::openai,
    vector_store::in_memory_store::InMemoryVectorStore,
    Embed,
};
 
// Documents must implement Embed to be indexed
#[derive(Embed, Clone, serde::Serialize, serde::Deserialize)]
struct Document {
    id: String,
    #[embed]  // This field is embedded for vector search
    content: String,
    title: String,
}
 
#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    let openai = openai::Client::from_env();
    let embed_model = openai.embedding_model(openai::TEXT_EMBEDDING_3_SMALL);
 
    // Build the vector store from documents
    let documents = vec![
        Document {
            id: "1".into(),
            title: "Rust Ownership".into(),
            content: "Rust's ownership system ensures memory safety without GC. \
                      Each value has exactly one owner. When the owner goes out of scope, \
                      the value is dropped.".into(),
        },
        Document {
            id: "2".into(),
            title: "Rust Lifetimes".into(),
            content: "Lifetimes are Rust's way of ensuring references are valid. \
                      The borrow checker verifies that references never outlive the data they point to.".into(),
        },
        Document {
            id: "3".into(),
            title: "Rust Async".into(),
            content: "Rust's async/await syntax enables concurrent programming without threads. \
                      The Tokio runtime executes async tasks efficiently on a thread pool.".into(),
        },
    ];
 
    // Embed all documents and store in in-memory vector index
    let store = InMemoryVectorStore::from_documents_with_id_f(
        EmbeddingsBuilder::new(embed_model.clone())
            .documents(documents)?
            .build()
            .await?,
        |doc| doc.id.clone(),
    );
 
    // Build RAG agent: automatically retrieves relevant docs before answering
    let rag_agent = openai
        .agent(openai::GPT_4O)
        .preamble("You are a Rust expert. Use the provided documentation to answer questions. \
                   Be concise and accurate.")
        .dynamic_context(3, store.index(embed_model))  // Top 3 relevant docs as context
        .build();
 
    // Ask a question: Rig embeds the question, finds top-3 docs, includes them in the prompt
    let answer = rag_agent
        .prompt("How does Rust prevent memory errors?")
        .await?;
    println!("{}", answer);
 
    Ok(())
}

For production use, replace InMemoryVectorStore with a persistent vector database: Rig has integrations with Qdrant, MongoDB Atlas Vector Search, and PostgreSQL with pgvector.

How Do You Deploy a Rig AI Service to Production?

A production Rig service wraps agents and pipelines behind an axum HTTP API, handles authentication and rate limiting, and deploys as a single binary with no runtime dependencies.

[dependencies]
rig-core = "0.6"
axum = "0.8"
tokio = { version = "1", features = ["full"] }
serde = { version = "1", features = ["derive"] }
tracing = "0.1"
tracing-subscriber = "0.3"

The deployment pattern:

                    ┌─────────────────────────────┐
  HTTP request      │    axum API server           │
  ─────────────►    │  /chat (POST)                │
                    │  /embed (POST)               │
                    │  /search (POST)              │
                    └────────────┬────────────────-┘
                                 │
                    ┌────────────▼────────────────-┐
                    │    Rig agent layer           │
                    │  - system prompt             │
                    │  - tool definitions          │
                    │  - context retrieval         │
                    └────────────┬────────────────-┘
                                 │
              ┌──────────────────┴──────────────────┐
              │                                     │
  ┌───────────▼──────────┐             ┌────────────▼──────────┐
  │   LLM Provider API   │             │   Qdrant Vector DB     │
  │  (OpenAI / Anthropic)│             │   (embedding search)   │
  └──────────────────────┘             └───────────────────────-┘

A single Rust binary handles the full stack. Deployment is straightforward: build with cargo build --release, copy the binary to your server, set the API key environment variables, and start the process. No Python virtual environments, no dependency conflicts, no runtime version mismatches.

What Common Mistakes Do Rust AI Developers Make When Using Rig?

Most Rig mistakes come from treating it like Python LangChain: different design philosophy, different failure modes.

• Creating a new provider client on every request instead of sharing one. The client holds the HTTP connection pool and API key. Creating one per request wastes connections and may hit API rate limits faster. Create the client once at startup and store it in shared state (Arc<Client> wrapped in axum State).

• Using InMemoryVectorStore in production. The in-memory store loses all indexed documents when the process restarts. For any production deployment, use Qdrant or pgvector. The InMemoryVectorStore is suitable only for development and testing.

• Not handling API rate limit errors with retry logic. LLM APIs return 429 (Too Many Requests) errors under load. Without retry logic with exponential backoff, your service will surface API errors directly to users. Use the tower middleware layer (which axum uses natively) to add retry and timeout policies.

• Setting temperature too high for structured extraction tasks. Tool calling and structured data extraction work best at temperature 0.0–0.2. Higher temperatures cause the model to deviate from the required JSON schema, producing parse errors. Always test your tool-calling code with temperature 0.0 first before raising it.

• Embedding full documents instead of chunks. Embedding a 50-page PDF as a single vector makes retrieval imprecise: the retrieved "document" may be mostly irrelevant to the query. Chunk documents into 512-token segments with 50-token overlap before embedding. Smaller, focused chunks produce dramatically better retrieval quality.

• Not validating tool output types before using them. Even with Rig's type system, LLMs occasionally generate malformed tool arguments that fail deserialization. Wrap tool call results in explicit error handling and provide the model with a structured error message so it can retry rather than crashing the request.

How Do You Move from Prototype to Production AI Services?

Prototyping with Rig is straightforward: instantiate a client, call .prompt(), get a response. Production is harder: it requires error handling for API rate limits, context window truncation logic, cost tracking, latency monitoring, and graceful degradation when LLM services fail.

Common production additions:

• Rate limiting with exponential backoff: LLM APIs return 429 errors under load
• Token counting: embed token count estimates in decisions about context window size and chunk count
• Cost tracking: log token usage per request and aggregate cost per user
• Fallback models: if GPT-4 quota exhausted, retry with Claude 3.5 Sonnet
• Circuit breaker pattern: if LLM API is down, serve a cached response or degrade gracefully
• Prompt caching (OpenAI) or batch APIs (Anthropic): reduce cost and latency for repeated queries

If you want to accelerate from "API call works in my notebook" to "production AI service serving customers at scale," Rustify's 9-week bootcamp covers the full stack: Rig fundamentals, production patterns, error handling, deployment, and cost optimization: with 1:1 mentorship and real project deliverables reviewed by AI infrastructure engineers.

Keep Reading

• Rust Developer Salary in the USA (2026)
• How Long to Learn Rust by Background: Hours, Timeline & ROI
• Best Way to Learn Rust in 2026
• Is Rust Hard to Learn?

Frequently Asked Questions

Sources

• Rig GitHub: 0xPlaygrounds/rig
• Rig documentation
• Rig examples: official cookbook
• OpenAI function calling guide
• Qdrant Rust client: vector store integration
• Ollama: local LLM server