Post Mortem: Patching the PHP and Node.js Vulnerabilities | Laravel - The clean stack for Artisans and agents

How we patched every Laravel Cloud app in place within about 27 hours of the PHP and Node.js disclosures, what broke on Forge, Envoyer, and Vapor, and what we are changing.
On the evening of 20 October, we got an early warning that PHP and Node.js were about to ship security fixes for critical vulnerabilities. Less than 24 hours later, the fixes were public, and every Laravel Cloud application needed PHP patched. We found no evidence that any customer was exploited as a result of these vulnerabilities.
This post covers what we did, what went well, and where we fell short. Laravel Cloud, Forge, Envoyer, and Vapor all needed different work, and the products that took the longest are where most of our lessons are.
PHP and Node.js each announced security releases on 21 October. The PHP releases fixed two critical remote code execution vulnerabilities, and the Node.js releases fixed ten vulnerabilities, including a Brotli decoder flaw that could allow remote code execution.
The PHP project gave us a pre-announcement at 19:58 UTC on 20 October, and the public releases followed at 13:38 UTC the next day. Details were scarce on purpose, which is how responsible disclosure is supposed to work. But knowing a vulnerability exists is enough for people to start hunting, and word was already spreading online. We had under 24 hours to patch the fleet.
Most of the time, we handle a CVE by updating our base images and notify customers they need to deploy to pull in the latest image and pick up the fix. We generally avoid forcing redeployments, for two reasons. A fleet-wide rebuild is slow, and and could take many hours during a critical period. Applications also run their own deploy commands, which can include database migrations, notifications, and file changes, and we can't know which of those are safe to trigger on someone's behalf.
This vulnerability was severe enough that we wanted every workload we manage patched, so we needed a second playbook.
While we built the real fix, we added a layer of protection that did not depend on any patch. At 08:55 UTC on 21 October, hours before the PHP release was public, we turned on Cloudflare's managed firewall ruleset for our Cloud domains, including newer rules that flag remote code execution and anomalous code execution in incoming requests. By 10:24 UTC we had added a smaller set of those newer rules for Forge and our dedicated zones. A firewall can't cover every route to a vulnerability like this, so the rules reduced exposure and did not replace the patch. We are keeping them because they inspect traffic before it reaches a customer's application, which gives every application, patched or not, an extra layer against this kind of request.
We normally get PHP through the official Docker Hub images. With so many people about to pull PHP and Node.js at once, we weren't sure how quickly those would be rebuilt.
So we planned two routes and decided we would commit to whichever worked first. Docker Hub open sources the way it builds its images, so we could copy the compilation process and build the patched PHP ourselves the moment the release landed. In the end, we shipped our own image at 14:24 UTC, and we used it. Docker Hub didn't have patched images until hours after the release, and some of the third-party download hosts our image builds rely on were down for part of the day. Building our own image allowed us to move faster.
Cloud runs on Kubernetes, which works by reconciliation. Think of it as a loop that compares the desired state of each workload with its current state, then makes changes until they match. We write Kubernetes operators to add our own logic to that loop, and one of them, the app operator, manages customer applications in each cluster.
That gave us a place to put the patch. We updated the app operator to drop the patched PHP binaries into each workload when it starts, instead of rebuilding and redeploying customer images. Kubernetes workloads are immutable, so the operator restarts each one to apply the change, and the platform absorbs the restart so customer applications see no downtime.
A built-in pacer stops the operator from restarting everything at once, which protects downstream systems like our source of truth and our billing pipeline. We sped it up by 5x for this rollout, which our estimates put at about 35 minutes of pacing instead of about three hours. We stopped short of removing the pacer entirely, because a rush of restarts could overload those systems.
We first proved the mechanism in our dev and staging environments, then canaried it at 11:17 UTC in two of our lowest-density regions before the PHP release was even public. The first wave began at 14:31 UTC, starting in Canada and Tokyo and verified on a small group first. From about 15:30 UTC the patch rolled out in waves across the remaining regions, so we could watch for failures before it reached our larger regions and so the restarts did not all land at once. That staging caught problems before they reached most customers, and the mechanism is now a standard tool for us. By 18:05 UTC, every region was at 100%. We paused new deployments on Cloud for about three and a half hours while this happened, from 13:54 to 17:33 UTC.
Planning took longer than the code. We already had the building blocks.
All times are UTC. Times marked ~ are reconstructed from call recordings and are accurate to about 10 minutes.
PHP pre-announcement of the security release shared internally.
20 Oct, 19:58 to 20:56
Engineers plan a quiet fix before disclosure, and the incident channel opens at 20:56.
Plan agreed: patch PHP inside running pods instead of rebuilding customer images.
Cloudflare managed firewall rules enabled, extended to Forge and dedicated zones by 10:24.
Canary of the patch mechanism begins in Canada and Tokyo.
Node.js security releases published.
PHP security releases published.
Forge: degraded as upgrade jobs surge. Envoyer: taken offline for server upgrades.
Cloud deployments paused