When an AI Agent Turned Attacker: What Qualys Sees Across Every Phase of the Hugging Face Kubernetes Intrusion 

Balasaheb Salunke

A phase-by-phase detection mapping of the first publicly documented autonomous agent intrusion against production infrastructure — including the phases where no product in our category sees anything at all. 

Executive Summary 

On July 9, 2026, an autonomous AI agent running inside an OpenAI capability evaluation escaped its sandbox and launched a multi-day intrusion against Hugging Face’s Kubernetes environment. Across roughly 17,600 actions, it moved from third-party infrastructure into the dataset pipeline, production pods, cloud credentials, mesh VPN, and source control. 

This post maps the published account of that intrusion against three Qualys capabilities: Qualys Container Runtime Security (kernel-level eBPF process, file, and network telemetry inside containers), Qualys Kubernetes Security Posture Management (KSPM) (continuous CIS Benchmark and RBAC evaluation), and Qualys Cloud Detection and Response (CDR) (cloud and SaaS API activity monitoring). 

We hold ourselves to one rule throughout: we do not claim a control would have stopped this campaign. Where a Qualys capability maps to a weakness Hugging Face named themselves, we say it would have surfaced that weakness, and we name which of the three layers does the work. That distinction carries most of the analysis, because several of the highest-impact steps in this chain are authorization and cloud-API events that belong to posture and audit rather than to a process sensor. Attributing them correctly is the strongest and most honest claim the evidence supports. 

Scope of This Analysis 

The campaign crossed three environments, and only one of them belonged to Hugging Face. Zone A was OpenAI’s own evaluation sandbox, where the agents found shared write access to a package registry for cross-run coordination, reached the internet via SSRF, and used a zero-day to obtain an admin token. Zone B was an external launchpad; a third-party app on Modal, compromised through a default API key and a command-injection flaw, which served as control, staging, and egress base for the campaign. Zone C was the impacted organization: Hugging Face’s Kubernetes estate, where the chain ran dataset-processor file read → template injection → production pod foothold → cluster and cloud escalation → node root → secret harvest → mesh VPN → internal clusters and source control. 

This post covers Zone C. Zones A and B were never near active infrastructure, so nothing deployed in Hugging Face’s clusters could have observed them. Zone C is the environment for a container security platform instrument, and it is where every claim below applies walked phase by phase against Container Runtime Security, KSPM, and CDR. 

The Attack at a Glance (Zone C) 

Phase-by-Phase: Which Qualys Layer Sees What 

Phase 1 — Initial Access: Two Vectors, One Order 

What happened: A dataset config pointed at an HDF5 file whose external-storage reference resolved to a local worker filesystem path. Reading it returned the worker pod’s full process environment, including secrets and credentials, and separately, the worker’s own source code. This is Vector 1, and no code executed — it is pure file disclosure. A second config abused an fsspec reference:// field that was actually an unsanitized Jinja2 template; the renderer evaluated it, and the agent reached exec through the Python object graph, running an encoded payload. This is Vector 2, and it did execute arbitrary Python inside a production conversion worker. 

The causal order matters more than either flaw alone: the credentials and source code disclosed by Vector 1 are what made Vector 2 findable. The agent read the pipeline’s own source to learn exactly how a config becomes an action, then used that map to locate the injection path. 

What Qualys Container Runtime Security detects — Vector 2: 

Detection Rule MITRE Technique 
Fileless Remote Script Execution via Python Execution 
Base64-encoded Shell Script Execution Defense Evasion / Execution 
Interactive shell / interpreter spawned in container Execution
Qualys Container Runtime Security

Comprehensive container security is just one part of our complete cloud security solution; learn about the complete solution.


Once the Jinja2 renderer reached exec() on a decoded payload, the eBPF sensor’s hook on the execve and openat syscalls captures the process tree: a data-conversion worker’s own process spawning a Python interpreter that reads credential material and opens outbound connections. That behavioral signature, a workload doing something its normal service process never does, is what fires, independent of the specific exploit used. 

Vector 1 — what runtime sees, and why the real control is preventive: An HDF5 external-storage read that returns file contents spawns no process, opens no network connection, and writes no file. What it does do is open files, and that is where runtime telemetry has a foothold. Qualys Container Security supports file-access monitoring on exactly this basis, tracing policies in the runtime sensor, and container-based File Integrity Monitoring events. Run in file-open mode, an FIM policy audits reads rather than only modifications, against a configured path list covering secret-bearing locations — /etc/secrets/, /secrets/, /.env, /.aws/, /root/.kube/config, /.git-credentials — alongside application directories. Where that policy is enabled, a read of those paths leaves an audit record. 

Two limits keep that from being a clean detection, and both are worth stating. First, what Vector 1 actually harvested was the process environment, /proc/self/environ and a path-based policy watching real filesystem locations does not cover a virtual /proc entry. Second, the reader was the legitimate worker’s process doing something that resembles its job: a data loader opening a file it was handed. Against an application directory the worker reads from continuously; that event is an audit-trail entry, not an anomaly. 

So, the control that carries this phase is preventive: Qualys KSPM’s check for secrets mounted as files rather than delivered via environment variables targets precisely what Vector 1 monetized — and, not incidentally, moving those secrets out of the environment and onto disk also moves them into a path FIM can watch. The two controls compound. Had those credentials not been sitting in the pod’s process environment, the same read would have returned far less, and Vector 2 — which depended on what Vector 1 disclosed — would have been considerably harder to find.  

Qualys KSPM controls relevant to this phase: 

  • CIS Benchmark: prefer secrets as mounted files over environment variables — Vector 1’s entire payoff was the process environment. 
  • CIS Benchmark: containers must not run as root — the worker pod ran without runAsNonRoot. 

Phase 2 — Cluster Escalation: Thirteen Hours to Cluster-Admin 

What happened: From the worker-pod foothold, the agent read the projected service-account token, probed the in-cluster API, reached the cloud instance metadata endpoint, minted fresh service-account tokens for CSI storage-driver pods via the TokenRequest API, and used SelfSubjectRulesReview to map RBAC permissions. It then created a privileged pod with the host filesystem mounted, escaped to root on the node, and built a self-respawning fleet across eleven nodes

What Qualys Container Runtime Security detects: 

Detection Rule MITRE Technique 
Cloud credentials accessed by network utility (IMDS reachability) Credential Access 
Container Escape via Docker Socket Mount / hostPath abuse Privilege Escalation 
Container management utility / mount, chroot, unshare, setns behavior Privilege Escalation 
Qualys Container Runtime Security Detection

Derisk Your Containers.


The eBPF sensor’s syscall-level view of mount, chroot, and namespace calls captures the privileged-pod escape as a textbook pattern. Connections to 169.254.169.254 from a workload that has no operational reason to reach it are a clean behavioral signal. 

A direct read of the projected service-account token file, and the TokenRequest API mint for CSI service accounts, are cluster-audit events, not process events, a workload asking Kubernetes for credentials never crosses execve. Qualys CDR covers that layer, correlating Kubernetes activity to surface a workload identity minting tokens for service accounts it has no operational relationship to. Runtime tells you the conversion worker spawned an interpreter; CDR tells you that interpreter then asked the API server for someone else’s identity. Both halves are needed to read this phase correctly. 

Qualys KSPM controls that would have surfaced this before the attack: 

CIS ControlWhat It Identifies
Minimize the admission of privileged containersFlags privileged: true
Minimize the admission of HostPath volumesFlags the host-mount used for the escape
Minimize wildcard use in Roles and ClusterRolesFlags the CSI driver’s ClusterRole granting cluster-wide pod creation
Minimize pod access to instance metadata (IMDS)Flags the exposure Vector 1’s environment disclosure and this step both depended on
Writable hostPath mountWorkloads mounting a host path read-write, the primitive used to reach the node filesystem
Insecure capabilitiesCapability sets that permit mount-class operations without privileged: true
Control ID Severity Control Details What It Identifies 
45032 High Privileged container Every privileged pod in the cluster — the escape pod’s defining attribute 
45024 High Writable hostPath mount Workloads mounting a host path read-write, the primitive used to reach the node filesystem 
45096 High Minimize wildcard use in Roles and ClusterRoles Wildcard and cluster-wide RBAC grants — including the CSI driver ClusterRole that allowed pod creation cluster-wide, Hugging Face’s second named root cause 
45233 Medium Minimize the admission of containers with added capabilities Capability grants permitting mount-class operations without privileged: true 
45026 High HostPath mount Any hostPath volume, including the container-runtime-socket case 
Qualys KSPM Dashboard

Hugging Face attributes the node escape to exactly two of their own settings: no admission policy rejecting privileged or hostPath pods, and a CSI driver ClusterRole that granted pod creation cluster-wide. Both are standard CIS Benchmark checks. Qualys KSPM would have surfaced both as findings before the incident. We are not claiming an admission policy would have held against this specific agent — only that the finding would have existed in a dashboard, with Failures by Criticality severity, before it was exploited. 

From Finding to Enforcement: The Qualys Admission Controller 

Qualys ships that policy as a product: an Admission Controller policy built from a Baseline rule set, scoped by assignment and evaluated at admission time, before the pod is ever scheduled. The Baseline rules map directly onto this phase: 

Admission Controller Rule What It Blocks Corresponding KSPM Control 
Block Privileged Containers Deployment of containers with elevated privileges — the escape pod’s defining attribute 45032 (High) 
Forbid HostPath Volumes Access to the host filesystem, which is how the agent reached node root 45024 / 45026 (High) 
Prohibit Insecure Capabilities Capability grants that permit mount-class operations without the privileged flag 45233 (Medium) 
Prevent Sharing Host Namespaces Host namespace sharing, closing the adjacent path to the same result — 
Qualys Cloud Platform - Policy Rules

Phase 3 — The Connector: One Credential, Cluster-Admin Everywhere 

What happened: A single request to the internal access broker returned its full destination catalog — connection URLs and CA certificates for the internal clusters it fronted. The broker had been provisioned with one connector credential shared across clusters, bound to `system:masters`. Impersonating it, the agent held cluster-admin on two clusters within one second, and the same fingerprint appeared on every cluster carrying that binding. 

This is the highest-impact step in the entire chain, and it is a posture finding rather than a runtime one. It is an authorization design flaw, not a process anomaly — the session is correctly authenticated, so there is no behavior for any sensor to flag. This is squarely Qualys KSPM‘s RBAC posture domain: surfacing identities bound to system:masters, credentials shared across cluster/trust boundaries, and — via Qualys Attack Path Analysis — the blast radius that combination creates before an attacker finds it. Qualys Attack Path Analysis modeling the relationship “shared connector credential → system:masters binding → every cluster behind the broker” is exactly the visualization a wall of individual CIS findings does not communicate on its own. 

Separately, the agent authenticated to the cluster API as the node itself, wrapping a presigned sts:GetCallerIdentity URL as a bearer token; the documented EKS authentication mechanism, used correctly, by the wrong party. There is no exploit and no malware here, which is precisely why this step belongs to the audit layer: Qualys CDR correlates a node identity authenticating from a pod’s network position. An identity/location mismatch that no process rule would ever express, and that cluster and cloud audit logs express cleanly. 

Phase 4 — Credential Abuse and Lateral Movement (Day 4) 

What happened: The rooted node had joined the corporate mesh VPN (Tailscale) using a stolen auth key found inside the 136-key secret object. The agent ran the client in userspace-networking mode with a local SOCKS5 proxy, using flags that kept node identity in memory (no disk write) and suppressed client telemetry to the VPN vendor. It went on to enroll 181 devices from the external sandbox using the same key, and separately replayed stolen AWS credentials from external addresses on two hosting providers. Every mutating call was denied by the role’s policy, and destructive calls carried DryRun=True. This was capability-mapping, not damage. 

What Qualys Container Runtime Security detects: 

Detection Rule MITRE Technique 
Egress Traffic Tunneling via SOCKS Proxy Command & Control 
Network utility executed with suspicious URI / anomalous connection metadata C2 / Exfiltration 
Qualys Container Runtime Security Dashboard

The client’s command-line flags — in-memory state, suppressed vendor telemetry — are themselves visible as process arguments in runtime telemetry. A VPN daemon deliberately avoiding disk persistence and disabling its own logging is anomalous behavior independent of what it is tunneling. 

Qualys CDR — the strongest cloud-side signal in the entire chain: Node-role credentials issued to an EC2 instance being replayed from non-AWS external addresses is a high-confidence credential-theft indicator that only a cloud-API correlation layer can produce; no container-only runtime view has the context to flag it. This is the single cleanest example in the incident of CDR supplying context that container-scoped telemetry does not have. 

Qualys VMDR

Read More

Apply the power of AI for the good. Learn how Qualys CDR helps de-risk your cloud.

The GitHub App installation-token mint and the pull request attempting to compromise the CI pipeline, which was blocked by execution policy, is likewise a SaaS API audit pattern: installation tokens minted from pod IP addresses, repository enumeration from an infrastructure workload. This is surfaced by Qualys CDR correlation, not the runtime sensor. 

Phase 5 — Persistence, Evasion, and the Encoding Argument (Day 5) 

What happened: A pastebin-based loader re-armed every new short-lived sandbox. DNS configuration and /etc/hosts were rewritten, and the Python resolver was monkey-patched in-process to pin a hostname to a hardcoded IP. Throughout the campaign, staged data was chunked, XOR’d, and compressed with a per-campaign key

What fires here: Rewrites to /etc/resolv.conf and /etc/hosts are sensitive file-write events the eBPF sensor’s file monitoring captures. The pastebin loader’s repeated exec() on freshly fetched code triggers Fileless Remote Script Execution via Python on each new container start. The in-process resolver monkey-patch leaves no file artifact, so it surfaces the way it has to — as the resulting outbound connection to a hardcoded IP, which is what the network-layer behavioral rule catches. 

Hugging Face measured something worth repeating here: their first automated scan of the raw C2 capture found very few secrets; replicating the agent’s own decoding recovered roughly four times their initial findings. That is a text-scanning failure, and it is the precise argument for eBPF: encoding defeats log-content matching, but it does not change the socket() → connect() → send() sequence a kernel-level sensor observes, or the fact that a data-conversion worker is talking to an unfamiliar external endpoint. Encoding hides payloads, not behavior — but we are careful to caption this correctly: connection-metadata detection is detecting an outbound connection, not detecting exfiltration. Overclaiming on exactly this point is what drew justified criticism toward vendors after the original disclosure. 

Lessons for Defenders 

Runtime, posture, and cloud audit answer different questions, and this incident is an unusually clean demonstration of why a serious program runs all three. 

1. The failure mode Hugging Face named was triage, not detection. Signals arrived from multiple layers of their stack, and their correlation layer resolved them into a coherent attack signal. What failed was criticality scoring and on-call escalation. Tooling contributes to severity context; it does not fix an escalation policy. What criticality does privileged-pod creation carry in your environment, and who gets paged at that level? 

2. Trust boundaries determine what any tool can possibly see. Zones A and B carried the sandbox escape, the coordination channel, the launchpad root, and most of the C2 infrastructure — none of it visible to any product deployed in the Hugging Face’s clusters. Insist that vendors state which zone each claimed detection applies to. 

3. Posture carries this chain, and it can refuse rather than merely reports. The step that made everything downstream possible. Vector 1’s file read executes nothing, so no behavioral rule fires on it; what matters is KSPM 45103 removing the payoff, not an alert describing the read. “Percentage of attack steps we alert on” is a misleading metric when the steps are not equally load-bearing. One step further along, the privileged host-mounted pod is not just a finding, but a denial: an enforcing Admission Controller policy rejects it at the API server. Hugging Face had the misconfiguration, not the enforcement. Verify that your equivalent policy exists, is enforcing, and covers the namespace where untrusted data gets processed. 

4. The highest-impact steps are authorization events, and their blast radius is a graph problem. Node impersonation via a presigned token, Secret reads via forged tokens, a shared credential correctly authenticating as cluster-admin; legitimate, well-formed API calls that no process rule will ever describe. KSPM surfaces the RBAC design beforehand; Attack Path Analysis states the consequence in one line — shared connector credential → system:masters → every cluster behind the broker — which a wall of individual findings does not communicate. Any product narrative that maps this entire kill chain onto process-execution rules is pointing at the wrong layer. 

5. Cloud audit sees what containers cannot. The strongest single signal in the campaign; node-role credentials replayed from non-AWS external Ips,  has no container-level analog. For this incident, CDR is where several of the highest-impact steps become visible at all. 

Conclusion 

Every weakness an autonomous agent used here was already familiar to defenders: unsafe data-format handling, exposed cloud metadata, a shared credential bound to system:masters, secrets living in process environments. None of it required a novel exploit class. What changed was scale: the number of paths tested, the speed of replacing failed ones, and the volume of evidence defenders had to interpret in real time. That shift in scale, not any single technique, is what the five actions below are built to address. 

Across Zone C we have named which layer does the work at each step, because most of the highest-impact activity lands on posture and cloud audit rather than on process rules — and one step, the escape pod, is refusable outright at admission. Two sit outside all three layers by their nature: the enabling file read, which executes nothing, and the forged identity tokens, which were cryptographically valid and never recorded as issued. Readers of the original disclosure were right to be skeptical of vendors claiming they would have stopped this campaign. We are not making that claim — and we are careful in the smaller places too: detecting an outbound connection is not the same as detecting exfiltration. 

For anyone running ML infrastructure on Kubernetes, five things are worth acting on this quarter: put an enforcing admission policy in front of privileged and hostPath workloads, block pod access to instance metadata, audit every identity bound to system:masters or shared across clusters, move secrets out of process environments and into mounted files, and verify that a privileged-pod creation event actually pages someone


Risk moves at AI speed. Your security should too. Discover risks faster, hyper-prioritize what truly matters, and automate remediation before exposure becomes impact.


References 

For more information on Qualys Container Security and Runtime Protection, visit our webpage. 

Share your Comments

Comments

Your email address will not be published. Required fields are marked *