Meta | CNCF

Enroll your company as a CNCF End User and save more than $10K in training and conference costs
Meta’s Cloud Foundation team of more than 15 engineers supports approximately 100 Kubernetes clusters across multiple cloud providers and accelerator types for AI research and inference. Previously, routine updates to the cluster stack, including drivers and schedulers, could take more than four hours through a monolithic Terraform workflow.
To address this challenge, the team is building a GitOps platform using Argo CD and Kargo. Application teams can declare their deployment requirements through a short application definition, enabling routine updates to reach a verified, healthy release within minutes. Terraform continues to manage the underlying infrastructure.
Terraform was responsible for provisioning Meta’s cloud infrastructure and preparing Kubernetes clusters. It was already well integrated with Meta’s identity, secrets, and compliance controls, as well as its internal review, build, and rollout systems. Reusing these existing integrations made sense for a small team, so Terraform took on a second responsibility: deploying Kubernetes applications.
However, applications required more frequent updates than the underlying infrastructure. Even a one-line change to Helm values had to pass through the entire infrastructure workflow, including configuration generation, internal builds and validation, a Terraform plan, shared test rollouts, and promotion. This process could take more than four hours, even when no cloud resources had changed.
Helm releases sharing a Terraform workspace also created dependencies between otherwise unrelated deployments. Like trains sharing a single track, one stalled rollout could delay dozens of other releases.
The Terraform application workflow also lacked application health validation. A successful Terraform application confirmed that the infrastructure operation had completed, but it did not necessarily mean the application was healthy. Each application needed its own independent promotion process, rollout policy, and verification checks.
“A green Terraform apply told us Terraform succeeded. Did it contain our changes? It wasn’t always easy to tell. Deploying infra was first class, while deploying apps was second class and rode its coattails”
Shaun Hopper, Production Engineer, Meta
GitOps offered a more natural approach to managing Kubernetes applications: defining the desired state in Git and using controllers to keep running systems aligned with that state. This gives application updates a versioned, traceable release process.
The team chose Argo CD, a mature and widely adopted platform for Kubernetes GitOps. Kargo, an open source promotion engine designed to work with Argo CD, manages the movement of releases across environments.
Installing the controllers was relatively straightforward. The greater challenge was integrating GitOps with Meta’s existing systems while making application deployment as simple as possible for developers.
“GitOps and Argo CD give our small team a great foundation, shaped by years of community experience in Kubernetes delivery. We focus on connecting it to Meta’s systems and aim to share our own lessons along the way.”
Phong Nguyen, Production Engineer, Meta
Simplifying deployments with built-in platform standards
Application teams define their deployment requirements in a short application definition. This includes ownership, artifacts, versioning rules, deployment targets, rollout policies, and verification checks.
The platform team manages the schema and automation that convert these definitions into Argo CD and Kargo manifests, along with the supporting configuration. This process validates inputs and enforces deployment standards automatically, reducing the complexity application developers need to manage.
“Developers describe what they want to deploy, then get back to building. Our platform team takes the Argo CD and Kargo exam so they don’t have to”
Phong Nguyen, Production Engineer, Meta
During early testing, Kargo updated release settings directly in the live Argo CD configuration. As a result, Git did not contain enough information to reconstruct a complete release, and recovery depended on the live configuration and Kargo’s retained history.
To address this limitation, the team changed the design so that Kargo writes a complete record of the desired release for each application and target cluster back into Git. Argo CD then reconciles the cluster against that versioned record.
This approach provides a clearer, more reliable record of what should be running in each environment.
Moving from per-cluster Argo CD to a centrally managed service
Initially, the team deployed a separate Argo CD instance inside each workload cluster. Every additional cluster introduced another instance that needed to be upgraded, configured, and maintained.
The team is now moving toward a centrally managed Argo CD service that operates outside the workload clusters. Under this model, new clusters will connect to the shared service rather than requiring their own Argo CD instances, reducing the operational overhead of managing the platform.
The new GitOps approach has significantly improved the application deployment process. Routine updates that previously took more than four hours can now reach a verified, healthy release within minutes.
Each application has its own promotion process, health checks, and versioned release records in Git. This gives application teams greater visibility into deployments while reducing dependencies between unrelated releases.
Developers only need to provide a short application definition, while the platform handles manifest generation, configuration, and integration with Meta’s existing systems.
The result is a simpler deployment experience for application teams, supported by a platform that manages much of the underlying complexity.
The GitOps platform remains under active development. The team is building a cent