Thank you for sending your enquiry! One of our team members will contact you shortly.
Thank you for sending your booking! One of our team members will contact you shortly.
Duration 14 hours
Course Outline
AI Sovereignty and Local LLM Deployment
- Addressing cloud LLM risks: data retention, input usage for training, and jurisdictional concerns.
- Exploring Ollama’s architecture: the model server, registry, and OpenAI-compatible API.
- Evaluating Ollama against alternatives such as vLLM, llama.cpp, and Text Generation Inference.
- Navigating model licensing terms for Llama, Mistral, Qwen, and Gemma.
Installation and Hardware Configuration
- Installing Ollama on Linux with support for CUDA and ROCm.
- Implementing CPU-only fallbacks and optimizing AVX/AVX2 instructions.
- Managing Docker deployments and persistent volume mappings.
- Configuring multi-GPU setups and strategies for VRAM allocation.
Model Management
- Fetching models from the Ollama registry (e.g., ollama pull llama3).
- Importing GGUF models from HuggingFace and TheBloke.
- Analyzing quantization levels: Tradeoffs between Q4_K_M, Q5_K_M, and Q8_0.
- Managing model switching and understanding concurrent loading limits.
Custom Modelfiles
- Mastering Modelfile syntax: FROM, PARAMETER, SYSTEM, and TEMPLATE directives.
- Tuning generation parameters such as temperature, top_p, and repeat_penalty.
- Engineering system prompts to define role-specific behaviors.
- Creating and publishing custom models to a local registry.
API Integration
- Leveraging the OpenAI-compatible /v1/chat/completions endpoint.
- Handling streaming responses and enforcing JSON output modes.
- Integrating with LangChain, LlamaIndex, and bespoke applications.
- Implementing authentication and rate limiting via reverse proxies.
Performance Optimization
- Optimizing context window sizes and managing KV caches.
- Efficiently handling batch inference and parallel requests.
- Allocating CPU threads with NUMA awareness.
- Monitoring GPU utilization and addressing memory pressure issues.
Security and Compliance
- Isolating network traffic for model serving endpoints.
- Implementing input filtering and output moderation pipelines.
- Maintaining comprehensive audit logs of prompts and completions.
- Verifying model provenance and integrity through hashing.
Requirements
- Proficiency in intermediate Linux administration and container management.
- A foundational understanding of machine learning principles and transformer architectures.
- Working knowledge of REST APIs and JSON data formats.
Target Audience
- AI engineers and developers looking to substitute cloud LLM APIs with local solutions.
- Organizations with strict data sensitivity requirements that preclude the use of cloud models.
- Government and defense sectors requiring air-gapped, isolated language model environments.
