Show HN: Foreman, a self-hosted LLM gateway for cost aware model routing
Foreman is an open-source, self-hosted LLM gateway that enables cost-aware model routing across multiple providers. It allows users to manage and route requests to different language models based on cost and performance, giving teams more control over their AI infrastructure without relying on third-party services.
Background
- An LLM gateway is middleware that sits between an application and multiple AI model providers (like OpenAI, Anthropic, or local open-source models). It handles API key management, routing, caching, logging, and cost tracking.
- "Self-hosted" means you run the software on your own servers rather than using a paid cloud service. This gives you full control over your data and avoids vendor lock-in.
- "Cost-aware model routing" is the key feature: the gateway can automatically send simpler requests to cheaper models (e.g., a small local model) and harder requests to expensive flagship models, based on the prompt complexity or the user's budget rules.
- Foreman is an open-source alternative to commercial gateways like Portkey, Helicone, or LiteLLM's hosted offering. It's built by Northwood Systems and targets developers who manage multiple LLM integrations and want to optimize spending without manual switching between providers.