Thank you for sending your enquiry! One of our team members will contact you shortly.
Thank you for sending your booking! One of our team members will contact you shortly.
Duration 14 hours
Course Outline
Deployment Strategies
- Evaluating trade-offs between hosted APIs and self-hosted solutions
- Patterns for on-premises, edge, and air-gapped environments
- Single model versus multi-model portfolio approaches
- Comparing cost per decision
Hardware Sizing and Capacity Planning
- Memory footprint analysis by parameter count
- CPU-only performance versus GPU acceleration
- Planning for concurrency and throughput
- Accounting for potential growth in task length
Serving Infrastructure
- Exploring vLLM and SGLang options
- Utilizing llama.cpp and GGUF execution
- Using Ollama for local development workflows
- Selecting stacks based on the shape of the decision workload
Quantization While Preserving Calibration
- Understanding precision formats and memory efficiency
- Measuring quality degradation and calibration loss
- Applying per-tensor and importance-aware quantization variants
- Conducting acceptance testing post-quantization
Latency Optimization
- Distinctions between median and tail latency
- Latency impacts from increasing context length
- Leveraging warm starts and model caching
- Batching multiple decision queries
Edge and Constrained Environments
- Running small decision models on CPU resources
- Deployment on Apple Silicon and embedded systems
- Strategies for offline operation and updates
- Distributing weights in air-gapped settings
Containerization and Scaling
- Designing images specifically for model serving
- Managing volumes and caches effectively
- Implementing horizontal scaling and routing
- Setting resource limits and sharing GPUs
Observability and Monitoring
- Tracking request, latency, and error metrics
- Monitoring confidence distribution patterns
- Following decision outcomes and escalation paths
- Setting alerts for calibration drift
Versioning and Deployment
- Versioning models and adapters
- Implementing shadow and canary deployments
- Using pinned model files for audit purposes
- Establishing rollback procedures
Security and Multi-Tenancy
- Enforcing authentication and rate limiting
- Isolating tenants for prompts and logs
- Controlling network isolation and egress
- Maintaining audit logs for all decisions
Requirements
- Proficiency in Linux administration and containerization.
- Experience with model inference servers and GPU management.
- Fundamental knowledge of REST APIs, load balancing, and monitoring systems.
Target Audience
- MLOps and platform engineers managing inference infrastructure.
- SRE teams providing support for AI services.
- AI engineers implementing decision models on-premises.
- Architects designing AI deployments for air-gapped or regulated environments.
