Model Serving
Model serving is the production runtime that loads a versioned artifact, applies the expected preprocessing, validates requests, executes inference, and returns a stable response schema. It is narrower than the whole ML system lifecycle but broader than model.predict: serving owns latency, resource limits, rollout compatibility, and telemetry.
Artifact: Serving Contract
The serving contract should be explicit enough that canary deployment, shadow deployment, and rollbacks can move traffic without changing client code.
openapi: 3.1.0
info:
title: Fraud score API
version: "2026-07-11"
paths:
/v1/fraud:score:
post:
x-model-version-header: X-Model-Version
requestBody:
required: true
content:
application/json:
schema:
required: [transaction_id, amount, currency, account_age_days]
responses:
"200":
description: Score and decision metadata
content:
application/json:
schema:
required: [score, threshold, decision, model_version, feature_version]The minimum runtime flow is request validation, feature retrieval, model invocation, thresholding, response serialization, and event emission. A model-versioning record must include thresholds and preprocessing, not only the serialized weights. If the feature store or model process fails, the service should return a documented fallback or an explicit error; silent default scores corrupt monitoring.
Failure Modes
Serving fails when artifacts are mutable, schemas are informal, or clients infer meaning from undocumented fields. Cold starts and large model loads can violate latency SLOs even when accuracy is unchanged. Generative systems add prompt, retrieval, and safety-policy versions, which is why the parallel page on generative AI model serving treats context construction as part of the served behavior.
References
Nav
Section — ML Engineering and MLOps
← Golden Datasets Batch and Online Inference →
Learning path — Production ML