FORGE-09
Insecure Operational Infrastructure Services
Compromise of schedulers, control planes, provisioning workflows, automation, or admin interfaces can give attackers broad control over AI workloads and nodes.
FORGE-09 Insecure Operational Infrastructure Services
Definition
Insecure Operational Infrastructure Services refers to compromise of the privileged platforms and automation used to operate AI infrastructure, including job schedulers, cluster control planes, server-preparation workflows, configuration automation, and internal administrative interfaces. These services decide what runs, where it runs, and how GPU systems are prepared and maintained. A single compromise can give attackers broad control over training and inference environments.
Description
AI infrastructure depends on an operational layer that turns GPU servers into usable training and inference capacity. Schedulers and cluster control planes such as Slurm, Kubernetes, and Ray decide where jobs run and what resources they receive. Server-preparation workflows install or configure the operating environment, GPU drivers, container runtime, monitoring agents, and baseline security settings before workloads are placed on nodes. Configuration automation, such as Ansible or internal fleet-management tooling, applies system settings, runtime configuration, and security controls across large fleets. Internal administrative tools expose these actions to operators through APIs, dashboards, and support workflows.
This risk arises when schedulers, provisioning systems, admin tools, automation pipelines, or cluster-management services are exposed, weakly authenticated, over-permissioned, or insufficiently segmented from tenant and enterprise environments. A compromised administrator account, SaaS integration, automation token, CI/CD system, or service credential can turn these trusted operational paths into attacker-controlled paths inside the AI infrastructure environment.
A compromise of these privileged services is severe because they hold broad authority over high-value GPU nodes and workloads. By controlling schedulers or configuration systems, an attacker can manipulate workloads, push malicious changes, weaken isolation, disable monitoring, alter node state, or compromise the integrity of rebuilt systems across the infrastructure.
Impact and Failure Modes
Compromised scheduler or orchestration controller
An attacker who controls Slurm, Kubernetes, Ray, or similar orchestration services may be able to submit jobs, modify workloads, change placement decisions, stop training runs, or access operational metadata across the cluster.
Exposed or weakly controlled job-submission services
Job-submission APIs such as Ray Jobs can become direct execution paths into the cluster. If exposed without appropriate controls, attackers can use them to run unauthorized workloads on GPU infrastructure.
Configuration automation blast radius
Configuration automation such as Ansible or internal fleet-management tooling often has broad administrative reach. Compromise of one automation path can push malicious settings, weaken isolation, disable monitoring, alter runtime configuration, or deploy persistence across many GPU nodes.
Privileged internal admin interface abuse
Internal dashboards and platform APIs that wrap scheduler, server-preparation, or lifecycle functions can become direct control surfaces. Weak authentication, broad operator roles, or insufficient approval for high-impact actions can turn a single admin interface compromise into cluster-wide operational control.
Node-preparation pipeline compromise
Server images, bootstrap scripts, driver installers, startup templates, and node preparation workflows define the trusted baseline for GPU hosts. If compromised, they can introduce backdoors, insecure configuration, vulnerable drivers, or disabled security controls into every node built or rebuilt from that pipeline.
Service credential and automation token exposure
Tokens used by CI/CD systems, SaaS integrations, deployment tools, schedulers, or automation services may carry broad infrastructure permissions. If leaked or over-scoped, they can allow attackers to submit jobs, modify configuration, pull artifacts, or access operational systems without compromising an interactive user account.
Prevention and Mitigation Strategies
For providers and datacenter operators
- Block tenant access to operational services: Tenant workloads, notebooks, containers, and ordinary job networks should not be able to reach schedulers, cluster control-plane APIs, configuration automation, node-preparation services, or privileged admin interfaces.
- Do not expose operational services broadly: Operational services should not be reachable from the public internet or broad corporate networks. Restrict access to dedicated management networks, approved VPNs, bastion hosts, or controlled administrative workstations.
- Enforce strong authorization for operational actions: Apply least privilege, RBAC, MFA where supported, and separation of duties for actions that submit jobs, alter cluster state, promote images, push configuration, or return nodes to service.
- Require approval and tamper-resistant audit records for fleet-wide actions: Configuration pushes, server image changes, scheduler policy changes, and broad administrative actions should require explicit approval and generate audit records suitable for incident investigation.
- Protect node build and bootstrap files: Restrict who can modify the operating-system images, startup scripts, and templates used to prepare GPU nodes, and require review before changes are promoted.
- Monitor operational service behavior: Alert on unusual job submissions, node build changes, configuration pushes, scheduler policy changes, and administrative access.
For customers
- Do not expose your own job-submission and orchestration endpoints: If you run Ray, a Kubernetes dashboard, a Jupyter or notebook server, or similar interfaces on rented GPUs, keep them off the public internet and behind authentication. Exposed orchestration endpoints (the ShadowRay pattern) are one of the most actively exploited paths into AI workloads, and on rented infrastructure they are usually yours to secure, not the provider's.
- Authenticate and scope your own automation: Require authentication on any job-submission API you operate, and scope the credentials and tokens used by your CI/CD, automation, and SaaS integrations so a single leaked token cannot reach your whole environment.
- Check that nothing operational is exposed to the internet: From outside, confirm that your rented machines' external IPs do not expose schedulers, dashboards, job-submission APIs, admin interfaces, or metrics endpoints without authentication. The ShadowRay pattern is exactly this kind of exposure, found from the internet.
Attack Scenarios
Ray is an open-source framework for running distributed AI and Python workloads, and its dashboard and Jobs API are used to manage and submit work to Ray clusters. An attacker discovers an exposed Ray dashboard or job-submission interface used for AI workloads. Because the interface is reachable without appropriate access controls, the attacker submits unauthorized jobs through the same operational path used to run distributed workloads. Those jobs can consume GPU capacity, run attacker-controlled code, access environment variables or operational metadata available to the workload, and create a foothold for follow-on activity. Oligo documented this pattern in its ShadowRay research on exposed Ray environments used for AI workloads
An attacker compromises Ansible or an internal fleet automation tool used to manage GPU nodes. They push a change that disables security controls, adds an unauthorized key, or alters container runtime configuration. Because these tools are designed to apply changes across many systems, a single compromise can affect many nodes before defenders notice.
An attacker compromises a low-privilege container used for preprocessing or job setup. Because the environment reuses service-account credentials and does not strongly separate workload, scheduler, and automation paths, the attacker pivots into job-control systems or supporting operational services. From there, they can influence workload placement, access model storage, alter jobs, or reach administrative paths that affect the broader AI environment. A limited workload compromise becomes a wider operational compromise because trusted infrastructure services are over-permissioned or weakly segmented.
References
NIST SP 800-223, High-Performance Computing Security: Architecture, Threat Analysis, and Security Posture. NIST. HPC security architecture, threat model, and posture guidance relevant to AI infrastructure operations.
https://nvlpubs.nist.gov/nistpubs/SpecialPublications/NIST.SP.800-223.pdf
NIST SP 800-234, High-Performance Computing (HPC) Security Overlay. NIST. Security control overlay for HPC environments, including systems used for AI and machine learning workloads.
https://nvlpubs.nist.gov/nistpubs/SpecialPublications/NIST.SP.800-234.pdf
CIS Kubernetes Benchmark. Center for Internet Security. Kubernetes hardening benchmark.
Ray Security Documentation. Ray. Security guidance for Ray clusters and interfaces.
Ray Jobs API. Ray. API for submitting and managing jobs on remote Ray clusters.
https://docs.ray.io/en/latest/cluster/running-applications/job-submission/api.html
ShadowRay: AI Workloads Actively Exploited in the Wild. Oligo Security, 2024. Reporting on exposed Ray environments abused against AI workloads.
https://www.oligo.security/blog/shadowray-attack-ai-workloads-actively-exploited-in-the-wild
Slurm Authentication. SchedMD. Authentication guidance for Slurm workload manager deployments.