Platform engineering maintenance pitfalls and smart strategies to stay ahead | CNCF

Enroll your company as a CNCF End User and save more than $10K in training and conference costs
Posted on January 21, 2026 by Jehoszafat Zimnowoda, Senior Software Engineer, Akamai and Matthias Erll, Senior Software Engineer, Akamai
CNCF projects highlighted in this post
Platform engineering is a discipline that aims to increase the productivity of software engineering teams by designing, building, and maintaining internal platforms that abstract underlying infrastructure complexity and provide self-service capabilities. Kubernetes-based platforms are often complex multi-Open Source Software (OSS) integrations; thus, platform engineering is not a “declare once and forget it” process. It requires continuous dependency maintenance and strategies for inevitable breaking changes.
In this blog post, we’ll go over the integration of fourteen OSS projects to maintain the platform that is built on top of a Kubernetes cluster. We explore the following challenges:
For this article, we have analyzed the last three years of releases for the following OSS projects: argo-cd, knative-serving, istio, harbor, keycloak, cloudnative-pg, gitea, ingress-nginx, grafana, sealed-secrets, kyverno, prometheus, external-dns, and cert-manager.
Based on our experience and analysis, you can expect between 2-5 major upgrades, 43-52 minor upgrades, and 276-327 software patches every year. That’s nearly an update a day!
A significant challenge arises from the responsibility of maintaining a robust security posture. Every day, a number of security vulnerabilities are discovered. They are addressed in upstream projects that your platform needs to apply as security patches. This significantly contributes to the number of patches from the diagram above. Such changes must be monitored and quickly adopted into your platform.
In line with this, a project or its chart maintainers may opt to stop maintaining previous versions and only provide an upgrade path towards the latest major release, which effectively marks an older major version as end-of-life. Even if you only intend to adopt a security upgrade, you are practically forced to follow the maintained version.
A significant amount of maintenance effort can be automated, particularly when it comes to identifying new application versions. Helm chart releases are organized in repositories and registries, whereas binary software releases are found on GitHub using their API. This makes essential download and upgrading tasks a good candidate for being delegated to scripts that open a pull request accordingly. Automation addresses the short-term end-of-life problem.
Recent years taught us that critical parts of the supply chain, such as container registries or Helm chart repositories, can be subject to rate limiting or disruptive changes like deprecation on short notice by either OSS communities or software vendors. Adopting those changes and ensuring platform instances are using the new platform version is often impossible in such a short time frame.
Maintaining a platform-dedicated container registry cache and a mono repository with Helm charts makes code-breaking changes transparent to users and buys platform engineers time to adapt. Similarly, this provides platform administrators the time to plan for an upgrade.
While each Kubernetes minor version is maintained for 12 months, cloud platform providers often push for faster adoption, and expectations are set that users will keep up with the latest version. A new Kubernetes version may come with deprecated APIs that your Helm chart relies on, making upgrading your Helm chart an inevitability.
A Kubernetes conformance smoke test enables you to discover potential breaking changes. Render all manifests that the platform would deploy and statically test the entire output against the Kubernetes OpenAPI schema, including all added custom resource definitions (CRDs).
With a platform as a product mindset, you own the full impact of what you deploy. Community Helm charts may reduce the number of manifests you need to define, but they do not remove the responsibility of understanding exactly what is being introduced into your Kubernetes cluster.
In practice, simply reviewing the Go template code is rarely sufficient to assess the effects of a new chart version on your platform, so other strategies need to be applied. Some of the Helm charts upgrades are delivered with modifications to immutable Kubernetes properties, such as Deployment’s label selector, complicating maintenance and upgrades further.
By implementing a custom platform operator capable of managing changes to immutable Kubernetes properties, you can ensure a smooth upgrade. Detect potentially disruptive changes before they affect production environments by analyzing rendered Helm chart manifests: current version vs new version. The diff can also be performed against a running cluster, performing a dry run before applying changes.
Many Helm charts are delivered with dependencies that simplify deployment by bundling essential components such as SQL databases and key-value stores. These maintain the state on their dependent apps, and frequently require persistent volumes to do so.
For example, Helm dependencies declare manifests to deploy a PostgreSQL database service, but do not come with any operator that is aware of database lifecycle management (upgrades, migrations, replication). This “batteries-included” approach simplifies the initial deployment, but is a technical debt for platform operations.
To address these challenges, you must review the included dependency charts to check if they maintain application-critical data. Avoid using SQL databases provided as Helm chart dependencies. Instead, deploy database services via database operators, such as CloudnativePG, that are aware of both Kubernetes storage limitations (e.g., volume attachments) and database lifecycle requirements, such as migration workflows on major upgrades.
A Helm chart upgrade that deploys without errors