Thank you for sending your enquiry! One of our team members will contact you shortly.
Thank you for sending your booking! One of our team members will contact you shortly.
Duration 14 hours
Course Outline
Deployment Strategies
- Comparing trade-offs between Hosted APIs and self-hosted solutions
- Patterns for on-premises, edge, and air-gapped environments
- Distinguishing between single models and model portfolios
- Analyzing cost per decision
Hardware Sizing and Capacity Planning
- Assessing memory footprints relative to parameter counts
- Evaluating CPU-only performance versus GPU acceleration
- Planning for concurrency and throughput
- Accounting for headroom in task length expansion
Serving Stacks
- Exploring vLLM and SGLang options
- Implementing llama.cpp and GGUF execution
- Utilizing Ollama for local development
- Selecting stacks based on the shape of decision workloads
Maintaining Calibration During Quantisation
- Reviewing precision formats and associated memory savings
- Quantifying quality and calibration loss
- Applying per-tensor and importance-aware variants
- Conducting acceptance tests post-quantisation
Latency Optimization
- Differentiating between median and tail latency
- Analyzing latency growth relative to context length
- Implementing warm starts and model caching
- Batching multiple decision queries
Edge and Constrained Environments
- Running small decision models on CPU resources
- Deploying on Apple Silicon and embedded systems
- Strategizing for offline operation and updates
- Distributing weights in air-gapped networks
Containerisation and Scaling
- Designing images optimized for model serving
- Managing volumes and caches effectively
- Implementing horizontal scaling and routing
- Configuring resource limits and GPU sharing
Observability
- Tracking request, latency, and error metrics
- Monitoring confidence distributions
- Recording decision outcomes and escalations
- Setting alerts for calibration drift
Versioning and Rollout Processes
- Managing model and adapter versions
- Executing shadow and canary deployments
- Pinning model files for audit compliance
- Defining rollback procedures
Security and Multi-Tenancy
- Enforcing authentication and rate limiting
- Isolating prompts and logs per tenant
- Controlling network isolation and egress
- Maintaining audit logs of decisions
Requirements
- Experience with Linux administration and containerization.
- Knowledge of model inference servers and GPU management.
- Foundational understanding of REST APIs, load balancing, and monitoring.
Target Audience
- MLOps and platform engineers managing inference infrastructure
- SRE teams supporting AI services
- AI engineers responsible for on-premises decision model deployments
- Architects designing AI solutions for air-gapped or regulated environments
