A transparent, high-performance API proxy that sits between your code/agents and any LLM backend (OpenAI, Anthropic, Gemini, Azure, Ollama). Save up to 90% on metered token costs and drop request latencies to single-digit milliseconds.
Click the simulation buttons below to trigger actual mocked request cycles and see how Relay's caching drop request times and metrics in real time.
Trigger requests to see how Relay behaves on cache misses, exact cache hits, and concurrent deduplications.
A robust feature set built to optimize multi-agent loops and production services.
Optionally encrypt cache entries locally using AES-256 with automatically generated or custom runtime keys. Safeguard prompts and system credentials.
Consolidates concurrent in-flight requests requesting identical prompts. Serves all waiting duplicate request streams simultaneously from a single upstream network call.
Sophisticated word-level edit-distance similarity comparison. Reuses responses for minor spacing or typos differences, governed by an automated rapid-edit kill switch.
Establish strict daily token limits per user or API key. Prevents runaway recursive agent loops from draining upstream provider credits.
Acts as an OpenAI-compatible endpoint. Out-of-the-box support for OpenAI, Anthropic (Claude Code / Claude CLI), Gemini wire format, Azure OpenAI, and local Ollama/vLLM.
Control background daemons directly via CLI commands or mount the proxy container alongside your docker application with automated configuration backups.
Start saving tokens and speeding up your applications today. Follow the simple steps to launch Relay using our npm CLI.
Use Homebrew or global npm registry to install the CLI command.
Verify configuration and system setup using relay doctor.
Launch the server inline or in the background via relay start.
# Option A: Install via Homebrew (macOS & Linux) brew tap vrushankpatel/relay brew trust vrushankpatel/relay # (If prompted by brew) brew install relay # Option B: Install via NPM registry npm install -g relay-llm-proxy # Verify environment & config relay doctor # Launch Relay daemon relay start -d