Safe & Secure Agentic AI: Governance, Identity, and Red-Teaming Training Course

Foundations: Threat Models for Agentic AI

• Categorizing agentic threats: misuse, privilege escalation, data leakage, and supply-chain vulnerabilities.
• Profiling adversaries and assessing attacker capabilities specific to autonomous agent environments.
• Mapping assets, defining trust boundaries, and identifying critical control points for agent interactions.

Governance, Policy, and Risk Management

• Establishing governance frameworks for agentic systems, including roles, responsibilities, and approval gates.
• Crafting policies that address acceptable use, escalation protocols, data handling standards, and auditability.
• Addressing compliance requirements and ensuring evidence collection for regulatory audits.

Non-Human Identity & Authentication for Agents

• Designing agent identities using service accounts, JWTs, and short-lived credentials.
• Implementing least-privilege access patterns and just-in-time credentialing mechanisms.
• Managing identity lifecycles, including rotation, delegation, and revocation strategies.

Access Controls, Secrets, and Data Protection

• Applying fine-grained access control models and capability-based patterns tailored for agents.
• Overseeing secrets management, encryption (in-transit and at-rest), and data minimization practices.
• Safeguarding sensitive knowledge sources and PII from unauthorized access by agents.

Observability, Auditing, and Incident Response

• Building telemetry for agent behavior, covering intent tracing, command logs, and provenance tracking.
• Integrating with SIEM platforms, defining alerting thresholds, and preparing for forensic analysis.
• Developing runbooks and playbooks for managing agent-related incidents and containment.

Red-Teaming Agentic Systems

• Planning red-team engagements, including scope definition, rules of engagement, and safe failover procedures.
• Executing adversarial techniques such as prompt injection, tool misuse, chain-of-thought manipulation, and API abuse.
• Conducting controlled attacks to measure exposure levels and assess impact.

Hardening and Mitigations

• Deploying engineering controls like response throttles, capability gating, and sandboxing.
• Implementing policy and orchestration controls, including approval flows, human-in-the-loop mechanisms, and governance hooks.
• Applying model and prompt-level defenses such as input validation, canonicalization, and output filtering.

Operationalizing Safe Agent Deployments

• Utilizing deployment patterns such as staging, canary releases, and progressive rollouts for agents.
• Enforcing change control, testing pipelines, and pre-deployment safety checks.
• Coordinating cross-functional governance across security, legal, product, and operations teams.

Capstone: Red-Team / Blue-Team Exercise

• Executing a simulated red-team attack against a sandboxed agent environment.
• Defending, detecting, and remediating as the blue team using established controls and telemetry.
• Presenting findings, remediation plans, and proposed policy updates.

Summary and Next Steps