Optimizing LLM Inference for the Rest of Us
ABDEL
SGHIOUAR
Senior Cloud Developer Advocare
Google Cloud
Abdel Sghiouar is a senior Developer @Google Cloud. A co-host of the Kubernetes Podcast by Google and a CNCF Ambassador. His focused areas are High Scale distributed systems on Kubernetes, Service Mesh, and Serverless. With a background in datacenter scale architecture and operations and consulting.
He spends most of his time working on optimizing GenAI Apps for large scale operations using Cloud Native Technologies and producing content targeting developers and ops professionals.
ABDEL
SGHIOUAR
Senior Cloud Developer Advocare
Google Cloud
Abstract
Not every organization operates with the hyperscale resources of Anthropic, Google, or OpenAI. For the majority of businesses integrating Large Language Models (LLMs) into their critical paths, the high costs and scarcity of GPU/TPU accelerators present a significant challenge. Striking the balance between performance, availability, scalability, and cost-efficiency is a must.
While Kubernetes is a ubiquitous runtime for modern workloads, deploying LLM inference effectively demands a specialized approach. This session dives deep into practical strategies for optimizing your Kubernetes clusters and LLM Inference workloads to run efficiently and cost effectively. We will explore:
– Container and Model Optimization
– Accelerator Management
– Data & Storage
– Network & Load Balancing
– Observability
Attendees will leave with practical techniques for maximizing cost/performance for LLM inference for their AI-powered applications on Kubernetes.
