---
title: "LLM Engineering: From Model Call to Production System"
description: "Aleksandr Filippov practices LLM engineering as a production discipline — structured output against Pydantic schemas, retrieval and agent orchestration, provider fallback and retry ladders, tracing and cost observability, and measured quality before anything goes wide."
date: "2026-08-27T07:00:00Z"
last_updated: "2026-08-28"
build_time: "2026-08-28T03:11:43Z"
skill_name: "LLM Engineering"
skill_description: "Building LLM systems that survive production — structured output, per-stage fallback and retry ladders, model tiering by task, tracing wired into the serving path, and a measured quality bar before anything ships wide."
skill_category: ["ai-engineering","ai-platforms-agents"]
since: "2025-02-01"
last_used: "2026-08-28"
related_skills: ["artificial-intelligence","python","langgraph","rag-enterprise-search","mcp","fastapi","claude-code"]
related_projects: ["air-api","aila","mcp-context-server","claude-code-gateway","cxr-draft-auditor","agent-memory-context-benchmark"]
related_experience: ["ai-product-manager-at-spotware"]
keywords: ["LLM Engineering","Large Language Models","Prompt Engineering","Structured Output","Tool Calling","Retrieval-Augmented Generation","Agentic Systems","LangChain","LangGraph","LangSmith","LLM Observability","Model Fallback","Token Budgeting","Fine-Tuning","Python"]
license: "https://creativecommons.org/licenses/by/4.0/"
canonical: "https://www.alexfeel.info/skills/llm-engineering/"
---

My LLM engineering skill is about **the distance between a working prompt and a working system**. A model call that behaves in a notebook still has everything ahead of it: schema-validated structured output so downstream code never parses free prose, explicit fallback when a generation comes back malformed, retry ladders and provider failover when an API degrades, token and context-window budgeting, and model tiering that matches each stage of a pipeline to the cheapest model that does the job well. I build those layers in Python, and I treat every one of them as product surface rather than plumbing — a bad generation should be a logged, recoverable, visible event, never a silent wrong answer.

In production this shows up as systems, not techniques. A retrieval platform whose every model-calling stage asks for **Pydantic-validated structured output** and carries its own degradation path, so one bad generation costs a stage instead of a request. A localization system that pairs strong models with terminology and translation-memory grounding, and whose output was scored **blind against professional human translation before it went wide**. An LLM gateway that fronts multiple providers with accounting and failover. A durable memory layer for agents, with embeddings, hybrid retrieval, and reranking behind a standard protocol. And in a one-week hackathon build, a **fine-tuned 4B medical model** paired with a stock 4B parser shipped as a working demo under hard constraints. Observability is wired in rather than promised: **LangSmith tracing** on the serving path, so latency, token spend, and cost per request are observed instead of guessed.

Measurement is where I hold this skill to account, and where I am deliberately extending it: the shipped systems carry per-system quality checks — the blind translation evaluation, the medical demo's held-out harness reporting precision, recall, and F1, the retrieval platform's per-stage score telemetry — and building that discipline into a systematic, reusable evaluation practice is the capability I am now developing in the open, starting with a benchmark for agent memory retrieval. The concrete systems are listed in the **Related Projects** and **Related Experience** panels above.


## Site index

- [llms.txt](https://www.alexfeel.info/llms.txt)
- [Complete index](https://www.alexfeel.info/llms-index.txt)