Introduction: Why GPU Embeddings Matter
In the realm of AI search, the quality of results hinges on two key factors: the precision of the embedding model and the efficiency with which it runs against an index. Perplexity has recently unveiled its technical approach to maximizing both dimensions through a sophisticated GPU architecture.
The engineering team at Perplexity published Fast Embeddings on GPUs, detailing how production infrastructure manages embeddings for its Search, Computer, and API Platform products. This work highlights the convergence of performance on the latest NVIDIA Hopper and Blackwell cards.
1. The Two Traffic Models: Batch vs Online
Perplexity distinguishes two types of embedding load: batch embedding, used when building or re‑indexing a vector database, and online embedding, triggered with every user query. Each requires specific optimisations to strike a balance between cost and latency.
The batch focuses on maximum throughput to process millions of documents quickly, while the online aims for sub‑millisecond latency to respond instantly to queries. Between these extremes, the scoring phase blends the advantages of both approaches.
Asynchronous Result Handling
To reduce wait times, Perplexity implements an asynchronous tracking abstraction that retrieves vectors as soon as they’re ready without blocking main execution. This technique significantly improves the responsiveness of the online service.
2. The GPU Architecture: Ivy, Tulip and ROSE
Ivy, Tulip, and ROSE represent three distinct layers of abstraction in the embedding pipeline. Ivy handles basic inference on the GPU, while Tulip optimises CUDA graph management to lower general startup overhead.
ROSE, meanwhile, manages routing and distribution of requests across multiple nodes, ensuring horizontal scalability without compromising quality. Together, these components create a robust ecosystem capable of executing millions of inferences per second.
3. CUDA and Rust Optimisations
Perplexity’s success largely rests on advanced use of C++ CUDA for fine‑grained graph management, coupled with an execution path written in Rust that blends performance and safety. This dual approach reduces latency while maintaining high reliability.
The integration of Rust into the request flow minimises software overhead, providing additional headroom to handle massive volumes without inflating infrastructure costs.
4. Economic and Environmental Impact
By optimising GPU utilisation, Perplexity lowers not only operational cost but also its carbon footprint. GPU computation is more energy‑efficient than CPU for vector‑intensive tasks.
The increased efficiency enables enterprises to deploy large‑scale AI search systems while mastering expenses, paving the way for broader adoption across diverse industries.
5. Concrete Use Cases
- Ultra‑fast web search for enterprise search engines
- Real‑time recommendation systems based on embeddings
- Large‑scale sentiment analysis with optimised embeddings
These applications demonstrate the flexibility of Perplexity’s GPU stack, capable of adapting to varied business needs while maintaining peak performance.
“GPU embedding optimisation has become a key differentiator for AI service providers.” – Dr. A. Smith, AI expert
Conclusion: The Future of GPU Embeddings at Perplexity
Perplexity shows it’s possible to combine high performance and low cost through a well‑thought architecture around Ivy, Tulip and ROSE. Companies looking to embed AI search into their products will gain efficiency and competitiveness.
To discover how your organisation can benefit from this technology, contact our team today!
