Step-by-step security checklists for the most common AI security tasks. Each playbook distils field experience into the minimum viable set of controls — what to do, in what order, and what to watch for.
Six analytic rules and a dedicated workbook for Copilot telemetry, contributed to the official Microsoft Sentinel GitHub repository by Samik Roy (May 2026). Deploy as a single solution from Sentinel Content Hub → Microsoft Copilot solution. Requires CopilotActivity table ingestion via the Copilot Data Connector.
Step 1: Ensure CopilotActivity logs are ingested
→ Sentinel Content Hub → Copilot Data Connector (Public Preview Feb 2026)
→ Requires Global/Security Administrator role
Step 2: Deploy the Microsoft Copilot solution
→ Microsoft Sentinel → Content Hub → search "Microsoft Copilot" → Install
→ Includes: workbook + 6 analytic rules + hunting queries in one solution
GitHub source:
Analytic rules: Azure-Sentinel/Solutions/Microsoft Copilot/Analytic Rules
Hunting queries: Azure-Sentinel/Solutions/Microsoft Copilot/Hunting Queries
Workbook: Azure-Sentinel/Solutions/Microsoft Copilot/Workbooks/MicrosoftCopilotActivityMonitoring.json
Step 3: Open workbook and validate data against your environment
Step 4: Enable and tune analytic rules to your policy and risk appetiteWhen you enable "Continuously detect managed devices" in Agent 365 Shadow AI, Intune automatically creates a Properties catalog profile called A365 - Monitor OpenClaw. It uses the new Local AI Agent Settings Catalog node, runs via the Intune Management Extension (IME), and refreshes every 24 hours. Read-only — safe to deploy. Source: Derk van der Woude, May 2026.
Agent Name — canonical identifier for the agent typeAgent Version — version string of the installed agentHost Process — parent process executing the agentInstall Location — filesystem path of the installationInstall Scope — per-user vs per-machineInstall Scope Platform User ID — Windows SID of installing userInstall Scope User ID — Entra ID user identifier (UPN)Local AI Agent Execution Context — user / elevated / SYSTEM ⚠️ SYSTEM = high risk// Requires Intune device inventory data in Log Analytics
// IntuneDeviceCompliancePolicies or custom inventory table from IME
// Filter for elevated/SYSTEM execution context
IntuneDevices
| where Properties has "Local AI Agent Execution Context"
| extend AgentName = extract("AgentName:([^,]+)", 1, Properties)
| extend ExecContext = extract("ExecutionContext:([^,]+)", 1, Properties)
| extend InstallUser = extract("InstallScopeUserID:([^,]+)", 1, Properties)
| where ExecContext in ("elevated","SYSTEM")
| project DeviceName, AgentName, ExecContext, InstallUser, LastSync
| order by ExecContext desc, LastSync descThe CloudAppEvents table captures Copilot and agent activity from the M365 Unified Audit Log via Defender for Cloud Apps. Requires: Settings → Cloud Apps → App connectors → M365 activities enabled. Metadata only — no prompt content.
CloudAppEvents
| where Timestamp > ago(7d)
| where ActionType startswith "CopilotAgent" or ActionType startswith "UpdateCopilot"
| project Timestamp, ActionType, AccountDisplayName,
AgentName = tostring(RawEventData.CopilotAgentName),
ChangeDetail = tostring(RawEventData)
| order by Timestamp descCloudAppEvents
| where Timestamp > ago(7d)
| where ActionType startswith "UpdateCopilotAgent"
| project CopilotActionTime = Timestamp,
AdminAccount = AccountDisplayName,
AgentName = tostring(RawEventData.CopilotAgentName),
Action = ActionType
| join kind=inner (
EmailEvents
| where Timestamp > ago(7d)
| where Subject has_any ("confidential","restricted","sensitive")
) on $left.AdminAccount == $right.SenderFromAddress
| project CopilotActionTime, AdminAccount, AgentName, Action,
EmailTime = Timestamp1, EmailSubject = SubjectCloudAppEvents
| where Timestamp > ago(30d)
| where ActionType == "CopilotForSecurityTrigger"
| summarize TriggerCount = count(), UniqueUsers = dcount(AccountObjectId)
by ActionType, Application, bin(Timestamp, 1d)
| order by Timestamp descIdentifies which AI model each Copilot Studio agent is using by extracting modelNameHint from RawAgentInfo. Flags EU Data Boundary (EUDB) compliance status per agent — critical for EU organisations where Anthropic-hosted models process data outside the EUDB. Source: Blue161616/Agent-Identity.
AIAgentsInfo
| summarize arg_max(Timestamp, *) by AIAgentId
| where Platform == "Copilot Studio" and AgentStatus != "Deleted"
| extend ModelNameHint = extract(@"modelNameHint:\s*([A-Za-z0-9_\-\.]+)", 1, RawAgentInfo)
| extend HintLower = tolower(ModelNameHint)
| extend Provider = case(
HintLower startswith "sonnet" or HintLower startswith "haiku" or HintLower startswith "opus", "Anthropic",
HintLower startswith "gpt" or HintLower startswith "o1" or HintLower startswith "o3", "OpenAI (Microsoft-hosted)",
isempty(ModelNameHint), "Environment default",
"Other"
)
| extend EUDB_Status = case(
Provider == "Anthropic", "OUT OF EUDB — cross-geo processing",
Provider == "OpenAI (Microsoft-hosted)", "In EUDB (if environment is in EU)",
Provider == "Environment default", "Depends on tenant default — verify",
"Verify"
)
| project AIAgentName, Provider, ModelNameHint, EUDB_Status,
EnvironmentId, CreatorAccountUpn, OwnerAccountUpns, LastModifiedTime
| sort by Provider asc, AIAgentName ascRegistrySource column distinguishing agent origin: "A365" (Agent 365 registered) vs "PowerPlatform" (Copilot Studio via Power Platform connector). Use these filters to target the right agent population. The four queries below cover the highest-risk patterns for A365-registered agents.RegistrySource == "PowerPlatform" in the same queries. The no-auth query from Step 1 remains the most important for Copilot Studio — Query 8a–8d are specifically designed for A365-registered agents.pip install pyrit # Required env vars (for Azure OpenAI target + scorer) export AZURE_OPENAI_API_KEY="your-key" export AZURE_OPENAI_ENDPOINT="https://your-instance.openai.azure.com/" export AZURE_OPENAI_DEPLOYMENT="gpt-4o"
Config-driven — switch strategies, datasets, or thresholds with no code changes. Point target at your agent entry point.
# scan.yaml
scan:
name: "my-agent-security-scan"
target:
module: "my_agent.main" # Your agent module
function: "handle_prompt" # Your agent entry function
strategies:
- name: "prompt_sending" # Single-turn baseline
- name: "crescendo" # Gradual escalation
max_turns: 5
- name: "tap" # Tree of Attacks with Pruning
converters: # Stack — applied in order
- name: "Base64Converter"
- name: "LeetspeakConverter"
- name: "TranslationConverter"
datasets:
- "airt_illegal" # Microsoft AI Red Team dataset
- "harmbench" # Harm evaluation benchmark
- "advbench" # Adversarial benchmark
- path: "datasets/custom.yaml" # Your domain-specific prompts
gate:
fail_on:
critical_findings: 0 # Zero tolerance for critical
high_findings: 3
max_score: 0.3 # Defense rate must exceed 70%Run in sequence — if plain prompts pass, layer evasion. Each phase builds on the previous.
Phase 1: Plain prompts → Baseline — establishes what passes without evasion → Catches basic prompt injection and policy violations Phase 2: Encoded prompts → Base64, ROT13, Leetspeak, Unicode confusables → Tests whether your agent/guardrails decode before evaluating Phase 3: Semantic attacks → LLM-powered rephrasing, translation, multimodal injection → Converters stack: translate → Base64 → embed in image Phase 4: Multi-turn dialogue attacks → CrescendoAttack: gradual escalation over 5–10 turns → TreeOfAttacksWithPruning (TAP): branching attack trees → Tests whether context accumulation bypasses initial guardrails
Map findings to OWASP LLM Top 10 (2025) for structured risk reporting. Turns PyRIT output into a risk register your security team understands.
LLM01 Prompt Injection → PromptSendingAttack + injection datasets (airt_illegal) LLM02 Sensitive Info → Data exfiltration datasets + PII scorers LLM06 Excessive Agency → Tool-calling attack datasets (advbench) LLM07 System Prompt Leakage → System prompt extraction datasets LLM10 Unbounded Consumption → High-volume automated attack patterns
Integrate into your pipeline. Exit code 0 = pass (deploy), exit code 1 = fail (block). No custom actions needed.
# GitHub Actions example
jobs:
security-scan:
steps:
- name: Run AI security scan
run: |
pip install pyrit
python scanner.py --config scan.yaml --output reports/
- name: Evaluate release gate
run: |
python gate.py --report reports/scan_results.json
# Exit 1 blocks deployment automatically
# When to run:
# Every merge to main: Quick scan only (phases 1–2, ~10 min)
# Pre-release branch: Full scan (all 4 phases, architect approval)
# Weekly scheduled: Full scan across full agent estate