AI Deployment & MLOps interview questions & answers

Picture handing your house keys to a stranger and saying "go live in my house for me, I will not be there, and I cannot see inside." You would not just hand over the keys -- you would first make four promises to yourself about that stranger's life in your house: the lights and water work the same way they did when you lived there, the stranger can change the thermostat without you rebuilding a wall, you can check on the house without standing in it, and if something goes wrong you can get your house back the way it was. Deployment is that handover, except the house is your code and the stranger is a machine you will probably never log into by name.

Ch10 got the app running on your machine, and sketched the architecture it grows into. This topic is everything after the sentence "it runs on my machine" -- the moment the same code starts running somewhere you are not watching it, for people you have never met.

In production, "it works" stops meaning "it ran once when I clicked it" and starts meaning you can answer four separate questions about code you can no longer see, run by a user you will never meet, on a machine that is not the one sitting under your desk:

1. Runs the same everywhere -- the image and its pinned dependencies are identical on your machine, in CI, and in production. No "works on my machine" gap, because there is no separate machine-specific setup step left to skip.
2. Configured without a rebuild -- switching database, provider key, or environment is an environment-variable change, not a new image. If changing a setting means editing code and shipping a new build, that is not configuration, it is a hidden redeploy.
3. Visible while running -- logs, metrics and cost tell you what it is doing right now, because you cannot walk over and look at the screen. A deployed process you cannot observe is not "running", it is just "out there somewhere."
4. Reversible -- you can name the exact previous version and get back to it in minutes. If you cannot say which version is live, you cannot roll it back, and rollback is the one fix that works when everything else is still on fire.

🌍 Real-world example: a FastAPI chat endpoint reads its provider key with os.environ["PROVIDER_API_KEY"] inside the request handler. It works every time on your laptop, because your laptop's shell profile has always exported that variable -- you set it up once, months ago, and forgot about it. The health check passes on the new machine too, because the health check never touches that line. The first real user's chat message is the first time anything on the new machine actually reads that variable, and it is not there. The container looks deployed, healthy, and broken, all at once, and it took a paying user to find out.

An AI app adds a fifth promise no CRUD app has ever had to make: the model itself is a remote dependency you do not own. Its latency, its availability and its exact version are someone else's decisions, made on their schedule, and your deployment has to keep working while all three change without warning -- a slower day at the provider, a maintenance window you were not told about, a model version quietly swapped for a newer one. None of the four promises above protect you from that; it is the reason this chapter keeps coming back to "the provider is down / slow / different" as a first-class failure, not an edge case.

💡 The four promises = runs the same everywhere, configured without a rebuild, visible while running, reversible. Every topic later in this chapter is one of these four -- when a topic says "this is promise 2", this is what it means.

💡 Remote dependency you do not own = a service (here, the model provider) whose latency, uptime and version you cannot control, patch, or guarantee -- only monitor and have a fallback for.

💡 Deploy artefact = the frozen thing that actually ships -- an image, a set of pinned dependencies -- as opposed to "the repo", which is source you edit, not the thing running.

What actually breaks when code leaves your laptop, concretely:

• A machine that is not yours -- different OS, different CPU, a library that compiled against your laptop's exact Python and nothing else's.
• An environment that is not yours -- no shell profile, no .env file you forgot was there, no folder of test files you created by hand and never committed.
• Users you cannot see -- you cannot ask "what did you just click" the way you can lean over a colleague's shoulder; the only version of events you get is whatever you chose, in advance, to record.
• No print() -- the debugging reflex that works everywhere on your laptop has nowhere to print to. If you did not wire up logs before the deploy, that information is gone the moment the request finishes, not "somewhere I can go look for it later."

When to use this framing: run the four-promises checklist mentally before anything you ship reaches a second human -- a demo for a hiring manager, a friend testing your project, a teammate pulling your branch. The moment someone besides you starts the process, all four apply, whether or not you called it "deploying."

When NOT to use it / Trade-off: for a script that only you ever run, on your own machine, going through all four promises is pure overhead -- there is no second observer to configure for, no unseen user to log for, nothing to roll back for anyone but yourself. The frame earns its cost exactly at the point someone other than you becomes a stakeholder in the process staying up.

Standard definition: Deployment is the set of guarantees that let code you can no longer directly observe keep working for users you will never meet: it must run identically across environments (pinned dependencies, a frozen artefact), be reconfigurable without a rebuild (externalised config and secrets), remain observable while running (logs, metrics, cost), and be reversible to a known-good prior version. An AI application adds a fifth: the model it calls is a remote dependency it does not own, whose latency, availability and version can change on the provider's schedule, not the deployer's.

"""
Same tiny "app" written two ways: the naive way reads an env var lazily,
inside the request handler -- the way almost every first FastAPI+AI app
is written. The validated way reads it once, at startup, through
pydantic-settings. Run on a machine where PROVIDER_API_KEY is NOT set
(simulating "the new machine" -- the laptop had it in a shell profile
nobody copied over).
"""
import os
from pydantic import Field
from pydantic_settings import BaseSettings

# make sure the var that only ever existed on "the laptop" is absent here
os.environ.pop("PROVIDER_API_KEY", None)


def naive_handle_request():
    """This is what most first FastAPI+AI apps look like: read the key
    right where it is used, inside the request handler."""
    api_key = os.environ["PROVIDER_API_KEY"]  # ambient laptop state
    return f"called provider with key starting {api_key[:3]}..."


class Settings(BaseSettings):
    provider_api_key: str = Field(min_length=8)


def run_naive():
    print("== naive: env var read lazily, inside the handler ==")
    print("  app import / startup : OK  (nothing touched the var yet)")
    try:
        print("  health check          : 200 OK (health check never reads it either)")
        result = naive_handle_request()
        print(f"  first request         : {result}")
    except KeyError as e:
        print(f"  FIRST REAL REQUEST     : 500 KeyError: {e}   <-- found in production, by a user")


def run_validated():
    print("== validated: typed settings, loaded once at startup ==")
    try:
        Settings()
        print("  startup                : OK")
    except Exception as e:
        first_line = str(e).splitlines()[0]
        print(f"  startup                : CRASH  {first_line}")
        print("  -> the process never becomes healthy, so the deploy never")
        print("     shifts traffic to it; the previous, working version keeps serving")


if __name__ == "__main__":
    run_naive()
    print()
    run_validated()

Picture packing an entire apartment -- every piece of furniture, the wiring already tested, the exact brand of every light bulb -- into one sealed shipping container, then trucking that same container to every destination. Nobody re-decorates the apartment on arrival. What shows up is exactly what was packed, screw for screw, at every single stop.

In production, a container image is that packed apartment: a frozen filesystem plus every pinned dependency your app needs, built once and shipped unchanged to every machine that runs it. Your repo is the furniture catalogue and the instructions for assembling it -- useful for building the container, but nobody runs a catalogue in production; they run the sealed container. This is the mental flip a beginner has to make: the thing that ships is not your code, it is the image your code produced.

🌍 Real-world example: a team builds an image, tags it myapp:latest, and deploys it. Two weeks later a bug appears in production. Someone rebuilds from the same Dockerfile to "reproduce" it, tags that build myapp:latest too, and now nobody can tell which of the two actually is running -- both answer to the same name. Nothing about latest points at a specific, frozen set of bytes; it is just a label anyone's next docker build can silently overwrite. The fix that always works in an outage -- "put back the version that worked" -- requires knowing exactly which image that was, and latest cannot answer that question.

💡 Image = a frozen filesystem snapshot: your app's code plus every pinned dependency, built once, run identically everywhere.

💡 Layer = one instruction's worth of filesystem change in the image (one RUN, one COPY); Docker caches each layer separately and only re-executes an instruction whose inputs changed.

💡 .dockerignore = a file listing what never enters the build context -- .git, .env, __pycache__, local venvs -- the same idea as .gitignore, but for what gets baked into the image.

💡 Non-root user = the process inside the container runs as an unprivileged user, not root, so a compromised process cannot write anywhere it should not.

THE MEASUREMENT -- why layer order is not cosmetic

Docker is not installed on this machine (verified), so no image was built for this topic. What is real and measured, in the same venv this course's FastAPI backend runs in:

import torch     : 3.5 s warm / 10.2 s cold      torch (no __pycache__) : 462.4 MB
import fastapi   : well under 1 s                fastapi                :   0.8-1.4 MB
                                                  whole venv             : over a gigabyte

A typical Python AI backend's requirements.txt pulls in torch (or transformers, which depends on it) alongside fastapi. fastapi itself is under 1.5 MB and imports in well under a second; torch alone is roughly 500x fastapi on disk (~462 MB as shipped, ~514 MB once __pycache__ exists) and costs seconds, not milliseconds, to import -- measured at 3.5 s with a warm filesystem cache and 10.2 s cold, and a freshly started container is always cold. Installing that dependency set is the single most expensive step in building the image -- far more expensive than copying your own source, which is typically a few hundred KB.

This is exactly why the order of two lines in a Dockerfile matters more than almost anything else in it: COPY requirements.txt . and RUN pip install -r requirements.txt before COPY . . (the rest of your source). Docker builds an image layer by layer, and it caches each layer keyed on its inputs. If requirements.txt has not changed, the pip install layer -- the one that did the ~462 MB, multi-second work -- is reused untouched from cache, even though you just edited app.py for the tenth time today. Copy the source before requirements, and every single source edit invalidates the pip-install layer too, so the multi-second torch install re-runs on every rebuild for a one-line code change. Same Dockerfile content, opposite build speed, purely from line order.

The Dockerfile

Pinned base image, .dockerignore kept alongside it, dependencies copied and installed before the source, a non-root user, and the port the process actually listens on declared with EXPOSE. The CMD is the same uvicorn command you would run by hand, just as the container's one persistent process.

Never bake a secret into the image

A copied .env file, or a secret passed as a build ARG, becomes part of a build layer -- and a layer is not deleted when a later instruction deletes the file. docker history (or simply re-extracting an older layer) can recover it, because every layer that was ever built is still stored inside the image, even the ones a subsequent RUN rm tried to erase. Anyone who can docker pull the image -- a teammate, a compromised CI runner, anyone with registry read access -- can recover a secret that was ever baked in, permanently, no matter how many steps later you "deleted" it. Secrets belong in the environment at deploy time (a secrets manager, CI-injected env vars), never in a COPY, never in an ARG, never in the image at all. config-and-secrets (a locked topic in this chapter) owns the full pattern; the rule that matters here is narrower and absolute: nothing that should stay secret may ever appear in a Dockerfile instruction.

When to use it: write your own Dockerfile when you need control over layer order (to protect the expensive install step above), a non-root user, an exact pinned base image, and a specific CMD -- which is every real AI backend with a heavy dependency like torch or transformers.

When NOT to use it: for a small, dependency-light service, a platform's auto-build path (a buildpack such as Cloud Native Buildpacks or Nixpacks) can build a working image straight from source with no Dockerfile at all. The trade-off is control: you cannot reorder its layers, choose its base image, or guarantee a non-root user the way a hand-written Dockerfile lets you. For a service whose build time is dominated by a multi-hundred-megabyte, ten-second-import dependency, that lost control is exactly the control you need -- write the Dockerfile.

Standard definition: A container image is a frozen, immutable filesystem snapshot -- application code plus every pinned dependency -- built once from a Dockerfile and run unchanged on any host with a container runtime; it, not the repository, is the artefact that ships. A Dockerfile builds that image as an ordered sequence of cached layers, so instructions that change rarely (installing requirements.txt) belong before instructions that change often (copying application source), letting an unrelated code edit reuse an expensive dependency-install layer instead of re-running it. latest is a mutable tag, not a version, and cannot identify which bytes are actually running in production -- images must be tagged by an immutable identifier, typically the commit SHA, so that a rollback means redeploying a known-good tag rather than guessing.

One plain sentence on what ran here and what did not: the Dockerfile below is illustrative and was never built (Docker is not installed on this machine); the size and import-time numbers quoted above are real measurements taken from this course's actual Python venv.

# ---- ILLUSTRATIVE (package not installed here; not executed) ----
FROM python:3.13.7-slim

WORKDIR /app

# system packages needed only to build wheels; the apt cache is removed in the
# SAME layer so it never inflates the image
RUN apt-get update && apt-get install -y --no-install-recommends build-essential \
    && rm -rf /var/lib/apt/lists/*

# requirements.txt copied BEFORE the source: this layer only re-runs pip install
# when requirements.txt itself changes, not on every source edit
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt

# source enters the image last -- editing app.py invalidates only this layer
# and everything below it, never the (expensive) install layer above
COPY . .

# never run the process as root inside the container
RUN useradd --create-home --shell /bin/bash appuser
USER appuser

EXPOSE 8000

CMD ["uvicorn", "app:app", "--host", "0.0.0.0", "--port", "8000"]

What we're building: you already built a tiny RAG service back in Ch4 -- chunk some pages, embed them, retrieve the closest ones for a question, hand them to a model as grounded context. This project takes that exact idea and makes it deployable: a stranger's laptop can run it, a bad config crashes it before a user sees it, an orchestrator can tell if it is actually working, and every request leaves a trail you can measure. This project is FREE but topics 3-16 of this chapter are LOCKED -- assume you have seen none of them. Every ## Step N below re-derives its own idea from scratch; nothing here depends on a topic you have not read.

🌍 Real-world example: think of the difference between a script that works when YOU run it on YOUR machine, and an API a hiring panel could point a browser at tomorrow morning without you standing next to it. The gap between those two is this whole project: pinned versions so it installs the same way twice, a config that refuses to start half-broken, a way to ask "is it actually ready to answer," logs that tell you what happened after the fact, a number for how slow it really is (not how slow it feels), and a container description any platform can run.

💡 deployable = a stranger, on a machine that is not yours, can run this and get the same behaviour you got -- without reading your terminal history to find out what you had installed. 💡 compose = one YAML file describing several containers (here: the API and a cache) that come up together with one command, wired to talk to each other.

Standard definition: deploying a RAG service means pinning what it depends on, validating its configuration before it accepts traffic, proving it is ready rather than merely running, emitting logs and a metrics summary that describe what actually happened, and packaging it as a container artefact plus a deploy checklist any platform can follow -- the same four promises (runs the same everywhere, configured without a rebuild, visible while running, reversible) applied to one concrete app.

⚠ One plain sentence on what ran and what didn't: every step below except Step 7 (the Dockerfile and compose file) was run for real against a live uvicorn server on this machine, with a deterministic fake retriever and fake model so nothing needed an API key. Docker itself is not installed here, so the Dockerfile and compose file are illustrative artefacts -- but the compose file was still parsed for real with yaml.safe_load to prove it is valid YAML, and its parsed service list is printed as genuine output.

Step 1 -- Pin the dependencies (a floating version breaks promise 1)

fastapi==0.141.1
uvicorn==0.52.0
pydantic==2.13.4
pydantic-settings==2.14.2
httpx==0.28.1

Why a bare fastapi in requirements.txt is already a bug, before you write a line of app code: promise 1 of deployment is "runs the same everywhere." An unpinned fastapi installs whatever the LATEST release happens to be on the day the image is built -- today, next week, and the day a teammate rebuilds it are three different installs of three different versions, and nothing in the file says which one you tested against. A minor bump can rename a parameter, change a default, or drop a deprecated code path out from under you, and it happens silently -- the build does not fail, it just installs different code and calls it the same requirements file. Pin every direct dependency to the exact version you tested (==, not >=), and regenerate the pin file deliberately (a new PR, reviewed) rather than letting it drift on every rebuild. The versions above are the ones this project was actually run against on this machine.

Step 2 -- Typed settings with a SecretStr provider key, validated at startup

from pydantic import Field, SecretStr
from pydantic_settings import BaseSettings

class Settings(BaseSettings):
    provider_api_key: SecretStr = Field(min_length=8)
    model_name: str = "small-chat-model"
    request_timeout_s: float = 12.5

settings = Settings()   # reads env vars; raises immediately if anything is missing or malformed

Why this must run at IMPORT time, not inside a request handler: if you read os.environ["PROVIDER_API_KEY"] lazily, inside the endpoint that needs it, the container starts fine, its health check passes fine, and the missing variable is discovered by your first real user as a 500. Reading it into a pydantic-settings class at module load time flips that: a missing or too-short key raises before the app can accept a single request, so the rollout of a broken config never becomes "successful" in the first place -- it crashes, the platform keeps the previous working version live, and nobody's request ever sees the gap. A crash at startup is a feature, not a bug.

Verified on this machine (real run, no API key needed to prove the crash):

== missing key entirely ==
CONTAINER START -> CRASH: ('provider_api_key',) - Field required

== key too short (fails min_length=8) ==
CONTAINER START -> CRASH: ('provider_api_key',) - Value should have at least 8 items after validation, not 5

Both are exactly the outcome you want: the process refuses to come up rather than serving traffic on a config nobody checked.

SecretStr also stops the key leaking through the places people forget about -- printing the settings object, dumping it into an error report, logging it by accident:

print(settings)      -> provider_api_key=SecretStr('**********') model_name='small-chat-model' request_timeout_s=12.5
f-string of field     -> **********
model_dump()          -> {'provider_api_key': SecretStr('**********'), ...}
model_dump_json()     -> {"provider_api_key":"**********", ...}
explicit read         -> sk-not...     (only via .get_secret_value())

A plain str field would print the real key in every one of those lines. SecretStr makes the SAFE thing the default and forces you to type get_secret_value() at the one call site that genuinely needs the raw value -- which is also exactly why a real key should never appear as a literal in a deploy command: a command lands in shell history and CI logs, SecretStr cannot redact something that never went through it.

Step 3 -- lifespan startup that loads the index once, and reports NOT READY until it has

STATE = {"ready": False, "chunks": None}

async def load_index_in_background():
    # a real embedding-model / vector-index load can take real seconds
    await asyncio.sleep(0.5)
    STATE["chunks"] = build_chunks()
    STATE["ready"] = True

@asynccontextmanager
async def lifespan(app: FastAPI):
    asyncio.create_task(load_index_in_background())   # do NOT await this -- see below
    yield
    STATE["ready"] = False

app = FastAPI(lifespan=lifespan)

Why lifespan and not the older @app.on_event("startup"): on_event is deprecated -- lifespan is the current, supported way to run code once at process start and once at process stop, and it is what lets you clean up (close clients, cancel background work) in the SAME place you set things up. Why the load is a background task instead of something the startup step waits for: if load_index_in_background() were awaited directly inside lifespan, the whole ASGI server would not start accepting ANY connection -- not even /health -- until the load finished. For a small demo that is invisible; for a real embedding model that load can take real seconds, and during that whole window the container's port would not even answer, which is worse than answering "not ready" -- an orchestrator cannot tell "still starting" from "dead" if nothing is listening at all. Firing the load as a background task lets the server come up immediately and answer honestly while the real work happens behind it.

Step 4 -- /health (liveness) and /ready (readiness) -- two different questions

@app.get("/health")
def health():
    return {"status": "alive"}          # "is this process wedged? restart it if not."

@app.get("/ready")
def ready():
    if not STATE["ready"]:
        return JSONResponse(status_code=503, content={"status": "not ready", "reason": "index still loading"})
    return {"status": "ready"}          # "should traffic be sent to this instance right now?"

Why a dependency check must NEVER go inside liveness: liveness answers "restart me if I fail," and a restart cannot fix a problem that lives on another machine. If /health itself checked the vector DB or the model provider and returned 503 whenever either was down, the orchestrator would kill and restart a perfectly healthy process over and over -- every instance restart-loops while the real fault sits somewhere the restart never touches. Readiness is the right place for that check, because its answer ("don't send traffic here yet") is exactly the correct response to a remote dependency being down, or to the index still loading.

Verified on this machine (real uvicorn, port asserted free before starting):

== liveness vs readiness, right after startup (index still loading) ==
/health -> 200 {'status': 'alive'}
/ready  -> 503 {'status': 'not ready', 'reason': 'index still loading'}

== /ready once the index finished loading ==
/ready  -> 200 {'status': 'ready'}

The process was alive the whole time (/health never dipped); it was traffic-worthy only once the index existed. That gap is exactly what a startup probe is for.

Step 5 -- Structured JSON logs with a request id, model, tokens, latency and cache status

request_id = str(uuid.uuid4())
log_entry = {
    "request_id": request_id, "model": settings.model_name,
    "tokens_in": tokens_in, "tokens_out": tokens_out,
    "latency_ms": round(latency_ms, 2), "cache_status": cache_status, "status": "ok",
}
print("LOG", json.dumps(log_entry))

Why a dict and not a sentence: f"user asked something, took a while" is readable once and unqueryable forever -- you cannot ask "what was my p95 latency for cache MISSes yesterday" of a pile of prose. A structured entry is what a real log aggregator (this project just prints it; a production one ships it somewhere) can filter, group and aggregate on. The request_id is what lets you find every line belonging to ONE request across a service that may be answering hundreds at once -- without it, two concurrent requests' log lines are indistinguishable. Verified real output from this run, one line per request:

LOG {"request_id": "74fa2c73-...", "model": "small-chat-model", "tokens_in": 7, "tokens_out": 17, "latency_ms": 0.18, "cache_status": "MISS", "status": "ok"}
LOG {"request_id": "322f42e8-...", "model": "small-chat-model", "tokens_in": 7, "tokens_out": 17, "latency_ms": 0.03, "cache_status": "HIT",  "status": "ok"}
LOG {"request_id": "9faed588-...", "model": "small-chat-model", "tokens_in": 0, "tokens_out": 0,  "latency_ms": 0.01, "cache_status": "N/A",  "status": "error"}

Never log the raw key -- notice settings.model_name is logged, never settings.provider_api_key; SecretStr from Step 2 makes an accidental logging.info("config=%s", settings) redact itself anyway.

Step 6 -- A summary/metrics endpoint: p50/p95/p99, error rate, cache hit rate

def percentile(values, p):
    s = sorted(values)
    return s[min(int(len(s) * p), len(s) - 1)] if s else 0.0

@app.get("/metrics")
def metrics():
    latencies = [e["latency_ms"] for e in REQUEST_LOG]
    errors = sum(1 for e in REQUEST_LOG if e["status"] == "error")
    hits = sum(1 for e in REQUEST_LOG if e["cache_status"] == "HIT")
    served = sum(1 for e in REQUEST_LOG if e["cache_status"] in ("HIT", "MISS"))
    return {
        "total_requests": len(REQUEST_LOG),
        "p50_ms": round(percentile(latencies, 0.50), 2),
        "p95_ms": round(percentile(latencies, 0.95), 2),
        "p99_ms": round(percentile(latencies, 0.99), 2),
        "error_rate": round(errors / len(REQUEST_LOG), 3),
        "cache_hit_rate": round(hits / served, 3) if served else 0.0,
    }

Why percentiles and not an average: an LLM endpoint's latency is naturally long-tailed -- most requests are fast, a slow generation or a cache miss drags a few far out, and averaging those together describes nobody. p50 is what a typical user felt; p95/p99 is what your unluckiest users felt, and that is the number worth alerting on. Verified real output, after seven real requests through this server (four MISS/HIT pairs and one deliberate error):

{
  "total_requests": 7, "p50_ms": 0.03, "p95_ms": 0.18, "p99_ms": 0.18,
  "error_rate": 0.143, "cache_hit_rate": 0.5
}

error_rate (1 of 7, from the request with no query) and cache_hit_rate (2 of 4 served requests) are exactly the two other numbers a dashboard needs alongside latency -- how often it fails, and how often it avoided doing the expensive work at all.

Step 7 -- A Dockerfile + compose stack (ILLUSTRATIVE -- Docker is not installed on this machine)

# ---- ILLUSTRATIVE (package not installed here; not executed) ----
FROM python:3.13-slim

WORKDIR /app
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt

COPY . .
RUN useradd --create-home appuser && chown -R appuser /app
USER appuser

EXPOSE 8000
CMD ["uvicorn", "app:app", "--host", "0.0.0.0", "--port", "8000"]

requirements.txt is copied and installed BEFORE the rest of the source: Docker caches each layer, so changing your app code (which happens constantly) does not force a reinstall of every dependency (which barely changes) -- only touching requirements.txt invalidates that layer. USER appuser means a container escape does not hand the attacker root inside the box. Nothing secret is copied in; a real key is injected as an environment variable at deploy time, never baked into a layer -- a COPY .env . would leave the key readable in the image forever, even after the file is later deleted from a later layer.

# ---- ILLUSTRATIVE (package not installed here; not executed) ----
services:
  api:
    build: .
    image: rag-api:${GIT_SHA:-dev}
    ports: ["8000:8000"]
    env_file: [".env"]
    depends_on:
      redis:
        condition: service_healthy
    healthcheck:
      test: ["CMD", "python", "-c", "import urllib.request as u; u.urlopen('http://localhost:8000/health')"]
      interval: 10s
      timeout: 3s
      retries: 3
  redis:
    image: redis:7-alpine
    volumes: ["redis_data:/data"]
    healthcheck:
      test: ["CMD", "redis-cli", "ping"]
      interval: 5s
      timeout: 3s
      retries: 5
volumes:
  redis_data:

Notice what is absent: a version: key. It belonged to an older Compose file format; the Compose Specification this file follows does not use it, and a current Compose install warns if you add it back -- omit it. Notice the depends_on shape: a bare depends_on: [redis] only waits for the redis CONTAINER to start, not for redis to be ready to accept connections -- the api container could come up and try to connect to a redis that is still booting. condition: service_healthy makes it wait for redis's own healthcheck to pass first, which is the actual guarantee you wanted. compose here means exactly this: one file, two services, one command to bring the whole local stack up together -- it is a local/CI convenience, not a production orchestrator.

Docker was not run to produce this -- there is no build log, no image size, and no container actually started. What WAS run for real: parsing this exact compose file with yaml.safe_load, which proves it is syntactically valid and lets you print its real structure:

parsed OK, top-level keys: ['services', 'volumes']
services: ['api', 'redis']
api depends_on: {'redis': {'condition': 'service_healthy'}}
has version key: False

Step 8 -- A deploy checklist you can carry to ANY container platform

Whichever platform runs the container (a managed container service, a Kubernetes cluster, a small VM), the same checklist applies -- these are properties of the deployment, not of one vendor's product:

1. Pinned dependencies -- every package in requirements.txt has an exact version (Step 1).
2. Config validated at startup -- a missing or malformed setting crashes the container before it becomes healthy, never discovered by a user (Step 2).
3. No secret is a literal anywhere -- not in the image, not in the repo, not in a deploy command's argument list; it is injected as an environment variable or read from a secret manager at deploy time, and SecretStr (or equivalent) keeps it out of logs (Step 2).
4. Liveness and readiness are separate endpoints, and only liveness restarts the process -- a remote dependency being down must show up as NOT READY, never as NOT ALIVE (Step 4).
5. Every request is traceable -- a request id threads through every log line for that request, and the log is structured, not prose (Step 5).
6. A metrics summary exists and reports percentiles, not just an average -- p95/p99, error rate, cache hit rate, at minimum (Step 6).
7. The image is tagged by something you can point back to (a commit SHA), never only latest -- if you cannot say which image is running, you cannot roll back, and rollback is the outage fix that always works.
8. The stack that supports the container is declared, not remembered -- what it depends on (a cache, a database), and what waits for what to be truly ready, not merely started.

None of these eight items name a platform. That is the point of a checklist: it is the same eight things whether the container ends up on a managed service, a cluster, or a single VM -- what changes between platforms is how you point the deploy command at them, not what the deployed thing must already be true.

What this project genuinely proves, and what it does not

Proves: every piece a deployable AI service needs was built and run for real against a live server on this machine -- pinned dependencies, config that crashes on a bad value instead of serving one, a background-loaded index with an honest not-ready window, two different health questions answered correctly, structured logs with a request id and token counts, and a metrics endpoint reporting real percentiles from real requests.

Does not (say so before an interviewer finds it): the model call is a deterministic fake, not a live provider -- swapping it in changes nothing about the wiring above it. There is no real Redis, database, or container runtime on this machine, so the compose file and Dockerfile are illustrative shapes, verified only as valid YAML, never actually built or run. And everything here is a single process -- Step 6's chapter sibling on shared state (workers-and-shared-state, locked) is the next problem this exact app hits the moment it scales past one worker.

import asyncio
import hashlib
import json
import time
import uuid
from collections import Counter
from contextlib import asynccontextmanager

from fastapi import FastAPI
from fastapi.responses import JSONResponse
from pydantic import Field, SecretStr
from pydantic_settings import BaseSettings


# ---- Step 1: pinned deps live in requirements.txt (fastapi==0.141.1, uvicorn==0.52.0, ...) ----

# ---- Step 2: typed settings, SecretStr key, validated at import time ----
class Settings(BaseSettings):
    provider_api_key: SecretStr = Field(min_length=8)
    model_name: str = "small-chat-model"
    request_timeout_s: float = 12.5

settings = Settings()   # crashes here if PROVIDER_API_KEY is missing or too short


# ---- Step 3: lifespan loads the index in the background; not-ready until loaded ----
STATE = {"ready": False, "chunks": None}
PAGES = {
    "https://hirenix.in/docs/pricing": "Hirenix offers a free plan and a Pro plan...",
    "https://hirenix.in/docs/refunds": "Refunds are available within 7 days...",
}

def fake_embed(text):
    import re
    return Counter(re.findall(r"[a-z]+", text.lower()))

def build_chunks():
    return [{"text": t, "source": u, "vector": fake_embed(t)} for u, t in PAGES.items()]

async def load_index_in_background():
    await asyncio.sleep(0.5)          # stand-in for a real embedding-model / index load
    STATE["chunks"] = build_chunks()
    STATE["ready"] = True

@asynccontextmanager
async def lifespan(app: FastAPI):
    asyncio.create_task(load_index_in_background())   # NOT awaited -- server stays responsive
    yield
    STATE["ready"] = False

app = FastAPI(lifespan=lifespan)


# ---- Step 4: liveness vs readiness ----
@app.get("/health")
def health():
    return {"status": "alive"}

@app.get("/ready")
def ready():
    if not STATE["ready"]:
        return JSONResponse(status_code=503, content={"status": "not ready", "reason": "index still loading"})
    return {"status": "ready"}


REQUEST_LOG = []
CACHE = {}

def count_tokens(text):
    return len(text.split())

def cosine_sim(v1, v2):
    import math
    common = set(v1) & set(v2)
    dot = sum(v1[w] * v2[w] for w in common)
    n1 = math.sqrt(sum(v * v for v in v1.values()))
    n2 = math.sqrt(sum(v * v for v in v2.values()))
    return dot / (n1 * n2) if n1 and n2 else 0.0

def retrieve(query, k=2):
    qvec = fake_embed(query)
    scored = [(cosine_sim(qvec, c["vector"]), c) for c in STATE["chunks"]]
    scored.sort(key=lambda x: x[0], reverse=True)
    return scored[:k]


# ---- Step 5 + 6: structured logs (request id, tokens, latency, cache) + /metrics ----
@app.post("/ask")
def ask(payload: dict):
    request_id = str(uuid.uuid4())
    start = time.perf_counter()
    query = payload.get("query", "")

    key = hashlib.sha256(f"{settings.model_name}:{query}".encode()).hexdigest()
    cached = CACHE.get(key)
    if cached is not None:
        cache_status, answer = "HIT", cached["answer"]
        tokens_in, tokens_out = count_tokens(query), cached["tokens_out"]
    else:
        cache_status = "MISS"
        top = retrieve(query)
        answer = f"Based on {', '.join(c['source'] for _, c in top)}: grounded answer to '{query}'."
        tokens_in, tokens_out = count_tokens(query), count_tokens(answer)
        CACHE[key] = {"answer": answer, "tokens_out": tokens_out}

    latency_ms = (time.perf_counter() - start) * 1000
    entry = {"request_id": request_id, "model": settings.model_name, "tokens_in": tokens_in,
              "tokens_out": tokens_out, "latency_ms": round(latency_ms, 2),
              "cache_status": cache_status, "status": "ok"}
    REQUEST_LOG.append(entry)
    print("LOG", json.dumps(entry))
    return {"request_id": request_id, "answer": answer, "cache_status": cache_status}


def percentile(values, p):
    s = sorted(values)
    return s[min(int(len(s) * p), len(s) - 1)] if s else 0.0

@app.get("/metrics")
def metrics():
    latencies = [e["latency_ms"] for e in REQUEST_LOG]
    errors = sum(1 for e in REQUEST_LOG if e["status"] == "error")
    hits = sum(1 for e in REQUEST_LOG if e["cache_status"] == "HIT")
    served = sum(1 for e in REQUEST_LOG if e["cache_status"] in ("HIT", "MISS"))
    return {"total_requests": len(REQUEST_LOG),
            "p50_ms": round(percentile(latencies, 0.50), 2),
            "p95_ms": round(percentile(latencies, 0.95), 2),
            "p99_ms": round(percentile(latencies, 0.99), 2),
            "error_rate": round(errors / len(REQUEST_LOG), 3) if REQUEST_LOG else 0.0,
            "cache_hit_rate": round(hits / served, 3) if served else 0.0}