From production data to faster inference: the model optimization loop

Data Lab provides a workspace for inspecting, filtering and saving inference data. In the demonstration, Sujee Maniyam selects a time window and model, imports available logs, and uses SQL to isolate relevant records, including fields within JSON data.

The saved dataset can then be selected for training. The session also discusses bringing existing datasets through S3-compatible storage. Capturing inference logs depends on the account’s retention settings; traffic that was not retained cannot be recovered through the import workflow.

Supervised fine-tuning uses examples to adapt a model’s behavior to a task. Distillation transfers knowledge from a teacher model to a student, often to make a smaller model useful for a specific workload. Speculative decoding accelerates generation by pairing a drafter with a target model that verifies its proposed tokens.

These approaches address different goals and can be combined. The webinar introduces fine-tuning and distillation as options, then focuses on preparing data and training custom speculators for inference optimization.

A small draft model proposes several tokens, and the larger target model verifies them together. Accepting multiple proposed tokens can reduce the number of sequential target-model decoding steps required to generate a response.

The draft model does not replace the target model’s verification. With a correct lossless verification algorithm, speculative decoding preserves the target model’s output distribution rather than relying on the smaller model’s quality alone. The speed benefit depends on how many tokens are accepted and the cost of drafting and verification.

A general-purpose drafter may not match the prompts and responses that dominate a particular application. Training on representative target-model completions can help the drafter propose tokens that the target model is more likely to accept on that workload.

Mashrur Haider describes combining synthetic examples with outputs from production traffic to train a custom speculator. The goal is better draft-target agreement on the application’s requests. Acceptance rate is an important signal, but the improvement must also translate into better serving performance.

Conversational records need to be converted into the format required by the selected training method. This includes tokenizing messages into input IDs and preparing the relevant masks and training labels.

Sujee shows conversational JSONL data alongside prepared training fields and explains that the platform can perform the conversion in the demonstrated workflow. Teams then select the dataset, configure the training job and monitor its progress. The data should represent the target model and workload the speculator will serve.

Measure end-to-end latency, token-generation speed, throughput, output quality and cost on representative requests. For speculative decoding, also track draft-token acceptance. A higher acceptance rate or lower training loss alone does not establish that the application has become faster.

The session emphasizes validation data drawn from the team’s own workload. For agents, measure the complete task because sequential model calls and tool interactions contribute to user-visible latency. Compare results with the baseline under comparable traffic and serving conditions, and include training costs when assessing the benefit.