Benchmarking small LLM inference on SageMaker AI: G7 vs G5 and G6

Choosing the right GPU instance for large language model (LLM) inference is one of the most impactful decisions you make when deploying generative AI at scale. A single generation jump can slash latency, increase throughput, and reduce cost-per-token. However, the real-world magnitude of those gains depends on model architecture, quantization format, and workload shape. In this post, we benchmark two representative 30B Mixture-of-Experts (MoE) models across three GPU instance families on Amazon SageMaker AI Inference using Amazon SageMaker AI Generative AI inference recommendation. This feature provides both recommendations for throughput, cost, and latency, and benchmarking for common metrics such as time to first token and latency. Using this feature, we demonstrate how the new G7 instances powered by NVIDIA Blackwell GPUs deliver measurable gains in throughput, latency, and cost-per-token. Use case 1: AI coding assistant Deploy Qwen3-Coder-30Bfor enterprise coding tasks such as code generation, debugging, refactoring, and developer... Copyright of this story solely belongs to aws.amazon.com. To see the full text click HERE