Show HN: Sipp – Run small local LLMs in browser 3x faster
Sipp is an open-source AI library that runs small local LLMs in browsers up to 3x faster using WebGPU. It offers a unified API for local and cloud inference, allowing developers to switch between on-device and remote backends with the same code. Built with Rust and C++, it focuses on optimizing browser performance through kernel fusion and bespoke inference pipelines.
Background
Sipp is a new open-source Rust/C++ library for running small AI language models directly in the browser (via WebGPU) or locally on a desktop, achieving up to 3x faster text generation than existing browser-based solutions. Key context: LLM inference ("decoding") normally requires a cloud GPU, but smaller models can now run on-device for privacy, offline use, and low-latency interactions. The challenge is that browser-based AI has been slow; Sipp improves this by optimizing memory management and fusing GPU operations. Its API is designed to let a developer write one code path that works both locally and in the cloud, simply by changing an endpoint. The team also contributed to llama.cpp, the foundational open-source project for running LLMs on consumer hardware.