Optimizing inference with GKE Inference Gateway

View the content before continuing.

แบบฝึกหัดนี้เป็นส่วนหนึ่งของหลักสูตร

AI Infrastructure: Networking Techniques

ดูคอร์ส

แบบฝึกหัดเชิงโต้ตอบแบบลงมือทำจริง

เปลี่ยนทฤษฎีให้เป็นการลงมือทำด้วยแบบฝึกหัดเชิงโต้ตอบหนึ่งในของเรา

เริ่มแบบฝึกหัด

แบบฝึกหัดนี้เป็นส่วนหนึ่งของหลักสูตร

AI Infrastructure: Networking Techniques

SkillTag.level.intermediateSkillTag.label

4.8+

11 reviews

เริ่มคอร์สฟรี

This module offers an overview of the course and outlines the learning objectives.

Exercise 1: Course introduction

This module details the specialized networking requirements for AI workloads compared to traditional web applications. It covers the specific bandwidth and latency demands of each pipeline stage—from ingestion to inference—and analyzes the "rail-aligned" network architectures of Google Cloud's A3 and A4 GPU machine types designed to maximize "Goodput."

Exercise 1: Role of networking in AI pipeline Exercise 2: AI Hypercomputer networking for ML workloads Exercise 3: Networking for GPU machine types Exercise 4: Quiz Question 1 Exercise 5: Quiz Question 2

This module details strategies for efficiently moving massive datasets into the cloud. It covers the use of the Cross-Cloud Network and Cloud Interconnect to establish high-bandwidth pipelines, and outlines configuration best practices—such as enabling Jumbo Frames (MTU)—to reduce protocol overhead and optimize throughput.

Exercise 1: Overview Exercise 2: Data ingestion over the Cross-Cloud Network Exercise 3: Quiz Question 1

This module details the critical role of low-latency networking in distributed model training. It covers the necessity of Remote Direct Memory Access (RDMA) for gradient synchronization, the benefits of Google's Titanium offload architecture in freeing up CPU resources, and the topology choices required to scale clusters without bottlenecks.

Exercise 1: Networking considerations for training Exercise 2: Networking for distributed training Exercise 3: Titanium's modern offloads architecture Exercise 4: Quiz Question 1 Exercise 5: Quiz Question 2

This module details the networking challenges specific to Generative AI inference, such as bursty traffic and long-lived connections. It covers optimizing Time-to-First-Token using the GKE Inference Gateway and "Queue Depth" routing, while also addressing best practices for network reliability and Identity and Access Management (IAM).

Exercise 1: Networking for generative AI inferencing Exercise 2: Optimizing network reliability for AI inference Exercise 3: Optimizing inference with GKE Inference Gateway

แบบฝึกหัดปัจจุบัน

Exercise 4: AI networking best practices Exercise 5: Congratulations and next steps Exercise 6: Quiz Question 1 Exercise 7: Quiz Question 2 Exercise 8: Quiz Question 3

Student PDF links to all modules

Exercise 1: Course Resources