Skip to content
TopicTracker
From HackerNewsView original
TranslationTranslation

Scaling PgBouncer across every core with SO_REUSEPORT and peering

ClickHouse's managed Postgres service implemented SO_REUSEPORT and socket peering to scale PgBouncer across all CPU cores, eliminating connection bottlenecks and improving performance under high load.

Background

- **PgBouncer** is a lightweight, open-source connection pooler for PostgreSQL. It sits between applications and the database to reuse connections efficiently, reducing overhead. It is widely used in production. - **SO_REUSEPORT** is a Linux socket option that lets multiple processes or threads bind to the same TCP port. The kernel distributes incoming connections among them, enabling better load balancing without a single bottleneck. - **Peering** here refers to a technique where worker processes share connection state directly (e.g., via shared memory or ring buffers), avoiding the need for a central coordinator. - **ClickHouse** (the blog's author) is a real-time analytics database company. They run a managed PostgreSQL service and needed to scale PgBouncer beyond what a single core can handle. - The blog describes how ClickHouse engineers combined SO_REUSEPORT with a peering approach to let PgBouncer use all CPU cores, solving a classic "one-core bottleneck" problem in high-traffic PostgreSQL setups.