I got tired of slow DB inserts, so I wrote a zero-syscall IPC in Go (18M TPS)
A Go developer created a new zero-syscall IPC mechanism called hft-ipc, achieving 18 million transactions per second to address slow database insert performance. The implementation uses a wait-free, lockless ring buffer design with zero allocation and no system calls during operation, inspired by LMAX Disruptor and DPDK.
Background
- The author built a custom inter-process communication (IPC) mechanism in Go that avoids system calls (syscalls), achieving 18 million transactions per second (TPS) for database inserts. Normally, IPC between processes requires syscalls, which add overhead. This project uses shared memory and lock-free techniques to bypass that.
- IPC (inter-process communication) is how separate processes exchange data; typical methods include pipes, sockets, and message queues, all of which involve the OS kernel. "Zero-syscall" means the processes talk directly via shared memory without asking the kernel, which is much faster but harder to implement safely.
- 18M TPS is an extremely high throughput — far beyond what traditional databases or message queues typically achieve — suggesting this is designed for specialized, latency-sensitive use cases (e.g., high-frequency trading, real-time analytics).
- The project name "hft-ipc" strongly implies the author works in or is inspired by high-frequency trading, where microseconds matter. The "zero-syscall" approach parallels techniques used in kernel bypass (e.g., DPDK, RDMA) but applied to database ingestion.