How it worksHow it works

1. KV cache calculation (prefill phase)

• When you send a prompt (e.g., “how are you”), the model processes all input tokens in parallel.

• For each token, it computes Key and Value vectors and stores them in the KV cache. This cache holds the context for the entire prompt and is ready before any output tokens are generated.

2. Token generation (decoding phase)

• For each new output token, the following steps happen on the fly:

  • Query: The model computes a query vector for the current position.

  • Dot product (attention score): The query is compared (dot product) with all stored keys in the KV cache, producing attention scores.

  • Softmax: The scores are normalized into attention weights (probabilities).

  • Weighted sum: The model computes a weighted sum of all value vectors using these weights, blending information from the prompt and previous outputs.

  • Project to logits: The result is projected onto the vocabulary space via a linear layer, producing a logit (score) for every possible output token.

  • Argmax or sampling: The model selects the next token by either picking the highest logit (argmax) or sampling from the probability distribution for more creative outputs.

  • Detokenization: The selected token ID is converted back to text.

  • The new token’s key and value are added to the KV cache, and the process repeats for the next output token.

Key pointsKey points

• The KV cache is calculated once for the prompt and reused for all output tokens, making generation efficient.

• Query, attention scores, softmax and weighted sum are computed dynamically for each output token.

• Projecting to logits and selecting the next token (argmax or sampling) are the final steps before detokenization.

Example: Prompting with “how are you?” (with KV cache in action)Example: Prompting with “how are you?” (with KV cache in action)

Suppose you prompt the model with “how are you?” and it will reply “I am fine”. Here’s what the transformer does when KV cache is used:

1. Tokenization: The prompt is split into tokens — “how”, “are”, “you”, “?”.

2. Embedding: Each token is mapped to a numeric vector (embedding). The tokenizer converts each token to a token ID (an integer). The embedding layer uses this token ID as an index to look up a row in the embedding matrix (a table of learned vectors). The result is the embedding vector for that token.

3. Layer processing: The embedding vectors are then passed into the next layer (or layers) of the neural network, such as self-attention or a hidden layer. In these layers, the embedding vectors are multiplied by weights and combined with biases and activation functions to produce new representations.

4. Self-attention and KV cache (prefill phase)

   • The transformer looks at all tokens in the prompt at once. For each token, it decides how much to “pay attention” to every other token.

   • As it processes the prompt [“how”, “are”, “you”, “?”], the model computes key and value tensors for each token in the attention layers and stores them in the KV cache. This cache now holds the context for the entire prompt.

   • In a transformer, each attention layer is a neural network layer that contains parameters (weights) and performs the self-attention operation. It is made up of multiple heads, each with its own set of weights and processes the input embeddings using matrix multiplications and softmax.

Attention scores, softmax and the KV cache

• For each token, the model computes an “attention score” for all other tokens by taking the dot product of their query and key vectors.

• The softmax function turns these raw scores into normalized weights (probabilities that sum to 1), letting the model blend information from all tokens in a learnable way.

• During inference, the KV cache stores the key and value vectors for previous tokens. This allows the model to quickly compute new attention scores and softmax weights for each new token, without recalculating everything for the whole sequence.

5. Layer processing: The model passes these representations through many layers, each refining its understanding of the prompt and building up context.

6. Decoding and KV cache (autoregressive decoding)

   • The model predicts the next token (“I”) by considering the entire prompt. It uses the KV cache to efficiently access the context [“how”, “are”, “you”, “?”].

   • The new token (“I”) is appended to the sequence, and its key and value tensors are added to the KV cache.

   • This process repeats at each step, as the model uses the cached context from previous tokens to predict the next one. After the new token is generated, its key and value are added to the KV cache for the next iteration.

   • The token-by-token generation loop using the KV cache is called the autoregressive decoding.

   • Common autoregressive examples include GPT, LLaMA, Qwen, BLOOM and Falcon models. In contrast, non-autoregressive models like BERT and diffusion models predict multiple tokens simultaneously. BERT and other bidirectional masked models do not use a KV cache for decoding. Inspect architectures, is_decoder, is_encoder_decoder and use_cache in the model configuration to confirm autoregressive or non-autoregressive behavior — as we’ll explain in the next section.

Model architecture and artifactsModel architecture and artifacts

When you download a model (e.g., from Hugging Face), you get more than weights:

• Weights: The model parameters, learned during training.

• Configuration (config.json): Architecture hyperparameters and positional strategy.

• Tokenizer assets: Including tokenizer.json, tokenizer_config.json, vocab/merges, specials.

• Generation defaults (optional): generation_config.json. When you load a model with transformers, these defaults are automatically applied unless you override them in your code. This helps ensure consistent, reproducible generation behavior across different environments and makes it easier to use the model as intended by its authors.

• Adapters (optional): Small sets of extra weights (such as LoRA or PEFT) that let you fine-tune a large model for a specific task without retraining all its parameters. LoRA (Low-Rank Adaptation) and PEFT (Parameter-Efficient Fine-Tuning) are popular techniques that add or modify a few layers or parameters, making it much cheaper and faster to adapt a model. Community adapters are pre-made fine-tuning weights shared by others for tasks like sentiment analysis, chat or code generation. You can load these adapters on top of a base model to quickly switch its behavior for different use cases. In simple terms: Adapters are extra weights you can load into your model to change or specialize its behavior for new tasks, without retraining the whole model. For example, you can add an adapter to make a general language model better at answering medical questions or writing poetry.

• Custom code (rare): trust_remote_code. You should only enable trust_remote_code for models from sources you trust, as this code runs with full permissions and could be unsafe.

How model artifacts get loaded into the CPU/GPU memoryHow model artifacts get loaded into the CPU/GPU memory

When you load a model for inference, the Python framework (like Hugging Face Transformers) performs these steps:

• Load model config (architecture and hyperparameters) into CPU/system memory.

• Load tokenizer assets (tokenizer and vocab files) into CPU/system memory.

• Load model parameters (weights and embedding matrix) into GPU memory.

This process ensures that the model structure, vocabulary and embeddings are all consistent with how the model was trained, so inference works as expected.

You can run the following code to inspect a model’s configuration, including its layers and hidden size. This process does not require a GPU or perform any neural network computation. It only loads configuration data and the tokenizer into CPU memory, which the CPU then uses to generate token IDs for the input text.

Inspect a model’s configuration:

#!/usr/bin/env bash
set -euo pipefail

# change this if you prefer another path
BASE_DIR="$HOME/vens"
VENV_DIR="$BASE_DIR/hf"

# ensure parent folder exists, idempotent
mkdir -p "$BASE_DIR"

# create venv if missing
if [ ! -d "$VENV_DIR" ]; then
python3 -m venv "$VENV_DIR"
fi

# activate and install required packages
# this activation only affects this script's shell
. "$VENV_DIR/bin/activate"
python -m pip install --upgrade pip transformers sentencepiece

# run the inspection script
python - <<'PY'
from transformers import AutoTokenizer, AutoConfig
model_id = "bigscience/bloom"
config = AutoConfig.from_pretrained(model_id)
print("Model config:", config)
print("Number of layers:", getattr(config, "num_hidden_layers", "N/A"))
print("Hidden size (neurons per layer):", getattr(config, "hidden_size", "N/A"))
tokenizer = AutoTokenizer.from_pretrained(model_id)
text = "Write a short poem about the moon."
token_ids = tokenizer.encode(text, add_special_tokens=True)
print("Token IDs:", token_ids)
print("Decoded text:", tokenizer.decode(token_ids))
PY

# optional, deactivate the venv
deactivate || true

Notes

• The script is idempotent and it reuses an existing venv.

• It requires Python 3.8 or newer and internet access to download the tokenizer/config.

• Caches go to ~/.cache/huggingface on your device.

• Paste the script into print-config.sh, make it executable, then run it:

vi print-config.sh
chmod +x print-config.sh
./print-config.sh

Operational implications

• Token counts drive latency and cost.

• Tokenizers differ across models.

• Ensure tokenizer and weights are from the same model repo/revision.

Model licensesModel licenses

What to check

• License type: Fully open source, researchonly or restricted/commercial. Examples include BLOOM RAIL (open with use constraints) and the Tongyi Qianwen license (commercial use allowed under specific terms).

• Commercial use: Verify if it’s permitted and under what conditions. Some licenses require registration or approval for commercial deployments.

• Redistribution and derivatives: Check whether you are allowed to redistribute weights, finetuned variants or quantized artifacts.

• Attribution and restrictions: Some licenses include RAILstyle acceptableuse clauses or attribution requirements.

• Practical guidance: Read the model’s license, follow all required registration or attribution steps, add a LICENSE file linking to the original license, record model details in a NOTICE file and consult legal advice if any terms are unclear.

Model profilesModel profiles

A comparison between BLOOM-176B vs Qwen-72B modelsA comparison between BLOOM-176B vs Qwen-72B models

Use official model cards for authoritative specs. Below are practitioner notes:

BLOOM-176B (bigscience/bloom)

• Size and memory: 176B parameters; BF16/FP16 weights alone are ~352 GB. Expect multi-node tensor parallelism or quantization for serving.

• Context and positions: The model was trained with ~2k context and ALiBi positional bias. ALiBi impacts attention backend choice, so FA3 is not recommended. Torch SDPA or Triton are better options — Torch SDPA has been tested and it worked.

• Tokenizer and prompts: HF fast tokenizer. The model has no built-in chat template — you will need to add one for chat-style prompts.

• License: BigScience BLOOM RAIL 1.0.

• Read the Model Card here.

Qwen-72B (Qwen/Qwen-72B)

• Size and memory: 72B parameters. Authors note that BF16/FP16 requires ~144 GB of total GPU memory, while INT4 variants can fit ≈48 GB GPU memory. Consider this when planning the number of GPUs for deployment.

• Context and positions: The model supports 32k context via extended RoPE. Backend kernels like FlashAttention v2 are supported. SDPA is a safe fallback for the backend.

• Tokenizer and prompts: tiktoken-derived large vocab (>150k). Some transformer flows require trust_remote_code, so make sure your runtime supports the model transformer version. Chat variants may provide templates. A chat template is not needed for this variant.

• License: Tongyi Qianwen license.

• Read the Model Card here.

vLLM core concepts and featuresvLLM core concepts and features

vLLM is a fast, open-source library for serving and running LLMs with high efficiency and throughput. It is designed to make LLM inference easy, scalable and cost-effective for both research and production. vLLM achieves state-of-the-art performance by using advanced memory management (PagedAttention), continuous batching and optimized GPU kernels. It supports seamless integration with Hugging Face models, streaming outputs and an OpenAI-compatible API server. vLLM runs on a wide range of hardware and supports distributed inference with tensor, pipeline, data and expert parallelism.

Key features

• PagedAttention for efficient KV cache management

• Continuous batching of incoming requests

• Optimized attention backends, like FlashAttention, FlashInfer, SDPA and Triton

• Fast model execution with CUDA/HIP graphs

• Quantization support (GPTQ, AWQ, INT4, INT8, FP8)

• Speculative decoding and chunked/disaggregated prefill

• Prefix caching for repeated prompts

• Multi-LoRA and multimodal model support

• Streaming outputs

• OpenAI-compatible API server

• Metrics and logging for production

Parallelism in vLLMParallelism in vLLM

vLLM supports several types of parallelism to scale LLM inference across multiple GPUs and nodes:

• Tensor Parallelism (TP): Splits the model’s tensor computations, such as matrix multiplications in each layer, across multiple GPUs. Each GPU handles a slice of the computation for every layer. TP is not strictly one-to-one with GPUs, but for most users and typical LLM deployments, matching TP size to GPU count is the standard and recommended approach. Advanced setups may use more flexible mappings. TP is the most common way to scale very large models that cannot fit on a single GPU. --tensor-parallel-size (set to number of GPUs per node, e.g. --tensor-parallel-size=4).

• Model parallelism: Splits different layers or blocks of the model across GPUs or nodes. Each device holds a part of the model and passes activations between devices. This is useful for extremely large models. vLLM does not natively support pipeline or model parallelism (splitting layers across GPUs/nodes) in the same way as Megatron-LM or DeepSpeed. Most vLLM deployments use tensor parallelism for scaling. The --pipeline-parallel-size <int> enables pipeline parallelism, but it’s not widely used.

• Data parallelism: Each GPU runs a full copy of the model and processes different batches of input data. Gradients or outputs are synchronized as needed. While this approach is more common during training, vLLM is focused on inference. Although data parallelism (multiple copies of the model processing different batches) is not a primary feature, you can run multiple vLLM servers for scaling inferencing throughput.

• Expert parallelism: Used for Mixture-of-Experts (MoE) models, where different “experts” (sub-neural-networks) are distributed across devices. Each expert processes only the relevant part of the input and vLLM coordinates routing and aggregation. vLLM supports Mixture-of-Experts (MoE) models, where expert parallelism is used. This is a more advanced and model-specific setup. A Mixture-of-Experts (MoE) model contains multiple expert sub-models (networks), each specialized for certain types of input or tasks. The model dynamically routes tokens to the most relevant expert (s) for efficient and specialized processing.

How vLLM works — high-level flow

Internally, vLLM manages the parallelism strategies listed above using efficient scheduling, memory management and communication primitives for GPU-to-GPU transfers. You can configure tensor parallel size and other options to match your hardware and workload. For most LLMs, tensor parallelism is set to the number of GPUs per node, but advanced deployments may combine multiple strategies for optimal scaling. Make sure to choose builds and images compatible with your accelerator.