MLOps Engineer

What a MLOps Engineer does daily

• ML model serving and deployment — building the infrastructure that serves trained ML models as prediction APIs; REST and gRPC model serving (FastAPI, Flask, TorchServe, TensorFlow Serving — the standard serving frameworks for PyTorch and TensorFlow models respectively; FastAPI is the most common general-purpose framework for custom ML APIs in 2026); containerisation (every production ML model is packaged as a Docker container; the Dockerfile defines the model's runtime environment, dependencies, and startup command; Docker multi-stage builds for efficient ML image sizes); BentoML (the most developer-friendly ML model serving framework; bentofile.yaml defines the service; bento build creates a deployable artifact; the best tool for teams that want production-grade serving without platform lock-in); Seldon Core (Kubernetes-native model serving; the most widely used enterprise model serving platform; supports PyTorch, TensorFlow, scikit-learn, XGBoost, and custom models; pre-packaged inference servers; explainability integrations); Triton Inference Server (NVIDIA's high-performance inference server; GPU-accelerated batch inference; dynamic batching for throughput optimisation; the standard for production deep learning inference at scale); Ray Serve (distributed model serving on Ray clusters; the most scalable Python-native serving framework; natural integration with Ray Tune for hyperparameter optimisation)
• MLOps pipeline automation — building the automated pipelines that move models from training through validation to production; MLflow (the most widely adopted open-source MLOps platform; MLflow Tracking for experiment logging — logging parameters, metrics, and artifacts for every training run; MLflow Model Registry for model versioning and lifecycle management — Staging and Production stages; MLflow Models for packaging models with their signatures; MLflow Projects for reproducible training pipelines; free, self-hosted or managed; Databricks MLflow for managed scale); Kubeflow (the Kubernetes-native MLOps platform; Kubeflow Pipelines for ML workflow automation — Python-based pipeline definitions compiled to Kubernetes Jobs; Kubeflow Training Operator for distributed training; Katib for hyperparameter optimisation on Kubernetes; the most production-hardened open-source MLOps platform but with significant infrastructure complexity); ZenML (modern MLOps framework; pipeline-as-code; stack-based infrastructure abstraction; the most developer-friendly open-source MLOps framework for teams starting MLOps from scratch; free Community tier)
• Feature store design and operation — building and maintaining the feature store — the shared repository of computed ML features that ensures consistency between training-time and serving-time feature values (the train-serve skew problem — when features computed during training differ from features computed at serving time — is one of the most common causes of ML model production failures); Feast (the most widely used open-source feature store; offline store for training data; online store for low-latency serving; feature registry for discoverability; free open-source); Tecton (commercial feature platform; managed feature pipelines; real-time features from streaming data; the most comprehensive commercial feature store); Hopsworks (open-source feature store with a managed cloud tier; Flink and Spark feature pipelines; the most feature-complete open-source option); AWS SageMaker Feature Store (managed feature store in AWS; offline store on S3 + online store on DynamoDB; native SageMaker integration); Vertex AI Feature Store (Google Cloud managed feature store; Online Serving for low-latency lookups; Batch Serving for training data generation; native BigQuery integration)
• Model training infrastructure and experiment management — building the infrastructure that enables data scientists to train models efficiently and reproducibly; Weights & Biases (W&B — the most popular experiment tracking platform in the research and production ML community; run tracking; sweeps for hyperparameter optimisation; artefact management; model registry; the de facto standard for deep learning experiment management; free for individuals); MLflow Tracking (the most widely deployed experiment tracking for enterprise teams; parameter and metric logging; artifact storage; experiment comparison; the standard for teams already using MLflow for model deployment); distributed training (PyTorch DDP — Distributed Data Parallel; Horovod; DeepSpeed — Microsoft's framework for large model training; FSDP — Fully Sharded Data Parallel for multi-billion parameter models; the ability to configure and operate distributed training is required for MLOps engineers supporting large model work); compute cluster management (Kubernetes for containerised training jobs; AWS SageMaker Training Jobs; Google Cloud Vertex AI Training; Azure Machine Learning Compute Clusters; GPU node management for deep learning workloads)
• CI/CD for machine learning — applying continuous integration and deployment practices to ML model development; model training CI (running model training on every commit to a model training repository; running unit tests for feature engineering code; running data validation checks on the training dataset; GitHub Actions or GitLab CI for CI pipeline execution); model validation gates (automated performance evaluation before deployment — if the model's accuracy, AUC, or F1 score on a holdout test set falls below a threshold, the deployment is blocked; shadow mode deployment — running the new model alongside the current model without affecting live users, comparing predictions to validate before full deployment); CD for models (automated deployment to staging environments; canary deployment — serving the new model to a small percentage of traffic while monitoring for degradation; blue-green deployment for zero-downtime model updates; automated rollback when model performance degrades below SLA); DVC (Data Version Control — the primary tool for versioning training datasets and model artifacts alongside code in Git; free open-source; the Git for ML data)
• Model monitoring and observability — building the systems that detect when a deployed model's performance is degrading; data drift detection (detecting when the statistical distribution of input features has shifted from the training distribution — the most common cause of gradual model degradation; Evidently AI — the most widely used open-source ML monitoring framework; free; drift reports for tabular data; classification and regression performance metrics; data quality reports; the primary tool for monitoring ML models in production in the Sri Lankan market); concept drift detection (detecting when the relationship between features and the target variable has changed — the model's predictions are based on a relationship that no longer holds; harder to detect than data drift because it requires ground truth labels from production); prediction distribution monitoring (alerting when the distribution of model predictions changes significantly — a simpler proxy for model degradation that does not require ground truth); model performance tracking (accuracy, precision, recall, AUC, RMSE — logged to Prometheus + Grafana or to MLflow; comparison of production performance to training performance; performance SLA alerting)
• Kubernetes and cloud infrastructure for ML — operating the cloud infrastructure that runs ML workloads; Kubernetes for ML (Kubernetes is the standard orchestration platform for production ML; understanding Pods, Deployments, Services, ConfigMaps, Secrets, PersistentVolumeClaims, Namespaces, Resource Requests and Limits is essential for every MLOps engineer; Helm for packaging ML infrastructure deployments; Kubectl and Kubernetes RBAC for access control; the ability to deploy and operate an ML serving stack on Kubernetes is the most consistently required MLOps infrastructure skill); GPU infrastructure (NVIDIA GPU operator for Kubernetes; CUDA; cuDNN; GPU resource scheduling; node taints and tolerations for GPU nodes; understanding GPU utilisation monitoring — nvidia-smi, DCGM — for cost-efficient deep learning inference); managed ML platforms (AWS SageMaker — the most comprehensive managed ML platform; Endpoint deployment; Pipelines for training automation; Model Monitor for drift detection; Clarify for bias and explainability; Azure Machine Learning — the most enterprise-integrated managed ML platform; Vertex AI — Google's managed ML platform; Vertex AI Pipelines; Vertex AI Endpoints; Vertex AI Model Monitoring)
• LLM Operations (LLMOps) — the rapidly emerging specialisation within MLOps focused on operationalising Large Language Models; LLM API integration (OpenAI API, Anthropic API, Google Gemini API, AWS Bedrock, Azure OpenAI Service — wrapping third-party LLM APIs in production-grade serving layers with rate limiting, retry logic, fallback models, and response caching); RAG (Retrieval Augmented Generation) pipeline operations (the dominant LLM application architecture — embedding generation pipelines; vector database management — Pinecone, Weaviate, Chroma, pgvector; document chunking and indexing pipelines; retrieval quality monitoring; context window management); LLM fine-tuning pipeline operations (LoRA and QLoRA fine-tuning pipelines; PEFT — Parameter-Efficient Fine-Tuning; Axolotl; Unsloth; fine-tuned model versioning and serving); LLM evaluation and monitoring (LangSmith, LlamaIndex Evaluation, DeepEval, TruLens — the LLM-specific evaluation and monitoring tools; evaluating hallucination rate, faithfulness, and answer relevancy; the most underdeveloped area of LLMOps in 2026); prompt versioning and management (PromptLayer, LangChain Hub; versioning prompts alongside model versions); guardrails (NeMo Guardrails, Guardrails AI — enforcing output safety and format constraints on LLM outputs)
• ML security and governance — ensuring that ML systems are secure, fair, explainable, and compliant; model explainability (SHAP — SHapley Additive exPlanations — the most widely used model explainability library; LIME — Local Interpretable Model-agnostic Explanations; Captum for PyTorch models; generating feature importance explanations for credit scoring, fraud detection, and medical prediction models that require regulatory explainability); model fairness assessment (IBM AI Fairness 360; Fairlearn — Microsoft's fairness toolkit; bias detection across demographic groups; the requirement to assess and document model fairness is increasing in Sri Lankan financial services regulation); ML model access control (authenticating and authorising API requests to model serving endpoints; JWT-based API authentication for internal model APIs; the same security principles as any production API apply to ML serving endpoints); adversarial robustness (detecting and defending against adversarial inputs — inputs deliberately crafted to cause incorrect model predictions; IBM Adversarial Robustness Toolbox)
• DataOps and MLOps integration — connecting data engineering pipelines to ML training and serving pipelines; feature pipeline orchestration (Apache Airflow or Prefect orchestrating the pipelines that compute features from raw data and load them into the feature store; the same orchestration tools used for data engineering pipelines are used for feature pipelines); training data versioning (DVC for versioning training datasets alongside model code; the ability to recreate the exact training data used for any previous model version is essential for model auditing and debugging); ML metadata management (tracking the full lineage from raw data to trained model artifact — which data version, which preprocessing pipeline version, which training code version, which hyperparameters produced this model; MLflow Tracking or Kubeflow Metadata for lineage capture; the ability to answer "why did this model make this prediction?" requires complete metadata lineage)