Inference Chip

A processor or accelerator used to run a trained AI model. The main measures are latency, throughput, memory bandwidth, energy use, software support and cost per output token.

GPU

A graphics processing unit adapted for AI because it handles many parallel mathematical operations well. GPUs dominate AI infrastructure because the software ecosystem is mature and flexible.

ASIC

An application-specific integrated circuit. In AI inference, ASICs trade flexibility for speed, power efficiency or lower cost on a narrower workload.

LPU

Language Processing Unit, a term used by Groq for an architecture built around predictable, low-latency large language model execution.

TPU

Tensor Processing Unit, Google’s custom AI accelerator family. TPUs are used for Google services and Google Cloud customers across training and inference workloads.

RDU

Reconfigurable Dataflow Unit, SambaNova’s term for an architecture that maps neural network computation through a configurable dataflow system.

NPU

Neural Processing Unit, usually an on-device or edge accelerator for phones, PCs, vehicles, cameras and embedded products.

Prefill

The stage where a model processes the input prompt and context. It is more parallel than token generation and often benefits from GPU-like throughput.

Decode

The stage where a model generates output token by token. Decode is often memory-bound and latency-sensitive, which is why many inference specialists target it.

KV Cache

Stored key/value attention state used during generation so the model does not recompute the full context at every token. KV cache size can dominate memory demand in long-context serving.

HBM

High Bandwidth Memory. HBM sits close to the processor package and is critical for moving model weights and activations fast enough to keep accelerators busy.

SRAM

Fast on-chip memory. More SRAM can reduce expensive data movement, but it is limited and costly compared with off-chip memory.

Cost per Token

The effective cost of producing model output, including hardware, power, memory, utilization, cooling, networking, software and operational overhead.

First-Token Latency

The time between a request arriving and the first generated token. This matters for chat, agents, voice, code tools and interactive products.

Tokens per Second

The rate at which a system generates output tokens. It should be interpreted alongside model, batch size, precision, context length and latency target.

Inference Cloud

A hosted service that exposes AI inference through APIs, usually hiding the chip and data-center complexity from the customer.