14× faster embeddings: how we rebuilt the ONNX path in Manticore
Manticore Search rebuilt its ONNX inference path for vector embeddings, achieving a 14× performance improvement. The optimization focused on direct matrix operations and reduced overhead, significantly speeding up embedding generation for search and AI applications.
Background
- Manticore Search is an open-source search engine (a fork of Sphinx Search) that competes with Elasticsearch for full-text and vector search.
- ONNX (Open Neural Network Exchange) is an open format for running ML models across different frameworks efficiently.
- Embeddings are the numerical vector representations that power semantic/"meaning-based" search — the article's "14× faster" refers to a major speedup in how Manticore computes these vectors internally.
- Technical background: Manticore added native ONNX support for generating embeddings, but the initial implementation was slow. This post details a rewrite that dramatically improved throughput for production vector-search workloads.