Planet Python

Last update: November 13, 2026 01:48 PM UTC
The Python Software Foundation (PSF) is the charitable organization behind Python, dedicated to advancing, supporting, and protecting the Python programming language and the community that sustains it. That mission is more than words to us; it’s the community we show up for every day. Our tiny but mighty team works hard to deliver the projects, services, and programs that allow Python to be the thriving, independent, community-driven language it is today. Some of what the PSF does includes producing PyCon US, hosting the Python Package Index (PyPI), supporting Developers in Residence, maintaining critical community infrastructure, and more.Python is for gathering, teaching, learning, playing, researching, exploring, creating, building, working, sustaining. The list goes on and on and on! Support this year’s fundraiser with your donations and memberships to help the PSF, the Python community, and the language stay strong and sustainable. Because Python is for everyone, thanks to you.
November 13, 2026 08:41 AM UTC
In releases, not restarts I showed the Muru daemon deploying a new release beside the one serving, and only moving traffic across once the new release was ready. That protects you from a release which can't start. It does nothing for a release which starts perfectly well and then fails some of the requests it is given, which in my experience is the more common sort of bad deploy. A health check passes, the deploy reports success, and the first you hear about it is from the people using the site.
The usual answer is a canary. The new release is given a small share of the traffic first, is compared with the release already serving, and only gets more if it holds up. Automating that is normally something added around the server rather than something the server does. On Kubernetes it means something like Argo Rollouts or Flagger, typically with a service mesh or an ingress controller doing the traffic split and Prometheus queries deciding whether the new version is healthy. Off Kubernetes it tends to mean load balancer weights and someone watching a dashboard. Gunicorn, uvicorn and the other Python servers have nothing to say about it, since none of them has any notion of two versions of an application being compared.
Under the Muru daemon, the edge already splits traffic between releases and already counts what each release does with the requests it is given, so the daemon can run the canary itself.
Since the last post the shop has gained a page for each order, deployed as release r5 the way that post showed. The next change adds prices to those pages:
Someone forgot about mugs. The order pages for teapots and kettles work, the home page works, and the page for the one order of a mug raises a KeyError. The release starts without a problem, so a plain deploy would put it in front of everyone.
Instead it is deployed with --hold, which starts the release but leaves it out of traffic:
In the last post I used the Muru-Preview header to reach a held release, and doing that here would find the bug for anyone who thought to check the right page:
The point of a canary is not having to know which page to check.
A canary is judged on real requests, so to stand in for people using the site I ran a small script with four clients, each requesting one of the home page and the three order pages at random, as fast as the shop would answer them, for 25 seconds. The script doesn't keep cookies, which matters below. With it running:
The plan was to give the new release 10% of the requests for 30 seconds, then 50% for another 30, and then all of them. It never got past the first step. The events show how quickly it was decided:
Five seconds after the canary started, the candidate was out of traffic again. Over the run the script got this back:
so 27 requests failed out of 6,645. A plain deploy would have failed a quarter of the order page requests for everyone until somebody noticed and rolled back.
What decided it is what Muru calls a gate. The edge keeps counters for every release, of the requests it answered, those answered with a 5xx status, and histograms of how long requests took overall and how long they waited for a worker. During a step the daemon reads those for the candidate and for the baseline, the release serving everything else, over the same window since the step began. Once the candidate has served 100 requests in the step, the gates are checked every second, and a failing gate rolls back straight away rather than waiting for the hold to end.
Importantly, a gate only fails if the candidate is both over its limit and worse than the baseline. If the database goes away and both releases start failing, that isn't the candidate's fault, and it isn't rolled back for it. The candidate is also kept running out of traffic, so its log can still be looked at, which muructl logs shop@r6 shows has the traceback:
The next release adds the price of a mug. This time I deployed it held as before, but started the canary with shorter steps, and asked for it to wait for my approval before each step after the first:
While the candidate had its 10%, I wanted to see who was getting it. The edge doesn't pick a release at random for each request. It gives each new client an anonymous id in a muru_id cookie, and hashes that to decide which release the client gets. Because my load script keeps no cookies, each of its requests counts as a new client, which is why it saw both releases. A browser keeps the cookie, so to see what that means I used curl with a cookie jar as 20 separate clients, each making five requests to the home page:
Four of the 20 clients landed on the new release, and each of them stayed on it for every request, as did everyone else on the old one. Someone clicking around the site doesn't flip between versions from one page to the next. The buckets are also laid out so that a client in the canary at 10% is still in it at 50%.
After 20 seconds with the