Summary: OpenAI has disclosed core details of Habitat: more than 70 million requests per second, over 500 petabytes and nearly 40 regions. More revealing than the scale are its deliberate limits. A narrow NoSQL interface and separated complex queries are meant to prevent outages.
From Python client to central storage service
Habitat began in 2023 as a Python library in front of Azure Cosmos DB. As OpenAI added services, the model became brittle: routing and resilience changes had to reach dozens of applications. In mid-2025, OpenAI moved the logic into a standalone service. Access controls, encryption, data residency, auditing and rate limits can now be enforced centrally.
OpenAI says the Python service peaked above 20 million requests per second. But asyncio offered I/O concurrency, not CPU parallelism. Compression, encryption and checksumming sometimes left completed responses waiting hundreds of milliseconds—and several seconds at the edge—for another event-loop slot.
A small pool policy amplified overload
OpenAI describes a particularly clear feedback loop in its connection pool. aiohttp reused the most recently returned TCP connection first. Slow servers returned connections later and consequently received even more work. Switching from LIFO to FIFO broke the loop. Istio and Envoy now pool connections, upgrade HTTP/1 to HTTP/2 and apply central rate limits and circuit breakers.
Meta documented a similar metastable state in 2014: connection-pool behaviour concentrated traffic on congested network paths. The comparison supports the mechanism, not OpenAI’s performance figures.
Rust saves resources—according to OpenAI
Two engineers rewrote the service in Rust with Codex and GPT-5.5 during the second quarter of 2026. Rust now handles 95% of production traffic and is claimed to be six times more CPU-efficient and 15 times more memory-efficient. OpenAI provides no absolute latency figures, hardware configuration, workload profile or independent measurements. These ratios are vendor claims, not a general Python-versus-Rust benchmark.
Habitat avoids arbitrary SQL to keep request costs predictable. Applications can access objects and direct relationships, while large joins or graph traversals remain their responsibility. Complex analytics run separately in Rockset instances fed through change data capture. The trade-off reduces convenience but protects the transactional path from one expensive query.
Pandorex assessment: Habitat shows that OpenAI scaled through narrow interfaces, separated workloads and broken feedback loops. Rust probably cuts operating costs, but without methodology or absolute figures, the gains cannot be transferred beyond OpenAI’s environment.