Get in Touch
 Duration 35 hours

Course Outline

Introduction and Diagnostic Principles

  • Survey of common failure modes in LLM systems and specific issues associated with Ollama
  • Creating reproducible experiments and controlled testing environments
  • Debugging toolkit: local log analysis, request/response capture, and sandboxing techniques

Reproducing and Isolving Issues

  • Methods for constructing minimal failing examples and seeds
  • Stateful versus stateless interactions: pinpointing context-related bugs
  • Managing determinism, randomness, and controlling non-deterministic behavior

Behavioral Evaluation and Metrics

  • Quantitative indicators: accuracy, ROUGE/BLEU variations, calibration, and perplexity proxies
  • Qualitative assessments: human-in-the-loop scoring and rubric development
  • Task-specific fidelity validation and acceptance criteria definition

Automated Testing and Regression Analysis

  • Unit testing for prompts and components, along with scenario and end-to-end tests
  • Building regression suites and establishing golden example baselines
  • Integrating Ollama model updates into CI/CD with automated validation gates

Observability and Monitoring

  • Structured logging, distributed tracing, and correlation ID management
  • Key operational metrics: latency, token consumption, error rates, and quality indicators
  • Implementing alerting, dashboards, and SLIs/SLOs for model-supported services

Advanced Root Cause Analysis

  • Tracing through graphed prompts, tool invocations, and multi-turn flows
  • Comparative A/B diagnosis and ablation study methodologies
  • Data provenance tracking, dataset debugging, and mitigating dataset-induced failures

Safety, Robustness, and Remediation Strategies

  • Mitigation techniques: filtering, grounding, retrieval augmentation, and prompt scaffolding
  • Rollback, canary, and phased rollout patterns for model updates
  • Conducting post-mortems, extracting lessons learned, and fostering continuous improvement

Summary and Future Directions

Requirements

  • Extensive experience in developing and deploying LLM applications.
  • Working knowledge of Ollama workflows and model hosting practices.
  • Proficiency with Python, Docker, and fundamental observability tools.

Target Audience

  • AI Engineers
  • MLOps Specialists
  • QA Teams overseeing production LLM systems

Number of participants


Price per participant

Upcoming Courses

Related Categories