Worker Versioning Garbage Collection Bug?

Sophie · May 28, 2026, 6:04pm

Hello

Sadly we face an issue right now with the worker versioning garbage collection.

From the docs:

when you deploy your 101st Worker Version in a Worker Deployment, the server looks at the oldest drained version in the Worker deployment. If it has had no pollers in the last 5 minutes, the server deletes it

( Worker Versioning | Temporal Platform Documentation )

We are exactly in this situation. But instead of the expected behavior we now get bombarded with this error on the matching pod:

cannot add version #### since maximum number of versions (100) have been registered in the deployment

There are plenty of drained versions. These worker pods were also already deleted by the controller like expected.

Besides this issue we have no other so far with versioning or the worker controller.

Version deployed in K8n is: 1.29.1

Is this a bug? One that’s maybe solved in newer versions already? I’m already reading through patch notes and even commits to find something, but failed so far.

Have we just configured something wrong? A hint would be nice.

Best,
Sophie

Jay_Pipes · August 4, 2026, 9:26am

Sophie, apologies for the long delay in responding to you. Would you mind letting us know the version of Temporal Worker Controller that you are using? And, to confirm, you are using Temporal Server v1.29.1 deployed in a self-hosted Kubernetes cluster, yes?

FYI, I was alerted to this community post via this Github bug/issue that linked here:

github.com/temporalio/temporal

Worker Deployment version GC does not reclaim eligible drained versions at maxVersionsInDeployment, wedging rollouts

opened 07:45PM - 16 Jun 26 UTC

noamyehudai

### What are you trying to do Run Worker Versioning (Worker Deployments) with a… controller that deploys frequently. Each deploy registers a new Worker Deployment Version. We expect old, drained versions to be reclaimed so we never hit the `matching.maxVersionsInDeployment` cap. ### Describe the bug Drained Worker Deployment Version records are not reclaimed in practice, so a deployment's version count grows by one per deploy until it reaches the cap, after which a new build can no longer register as a poller and the rollout is stuck (the new version stays unregistered and cannot be promoted to current). Observations on a busy namespace (server 1.30.x; also reproduced by another user on 1.29.1, see the forum link below): * A single Worker Deployment had accumulated well over 100 versions. The large majority were `drained` and had no pollers (their workers were scaled to zero long ago). The oldest drained version dated back to the deployment's creation months earlier and had never been reclaimed. * When the deployment reached the configured cap, registering the next version failed with `cannot add version ... since maximum number of versions (N) have been registered in the deployment`, even though many of the existing versions were drained with no pollers and should have been eligible for the server's at-cap reclaim. * Manually deleting drained versions via `DeleteWorkerDeploymentVersion` immediately unblocked registration. So two things appear to be happening: 1. Below the cap there is no background reclamation of drained, poller-less versions. They accumulate up to the cap. 2. At the cap, the lazy reclaim that is supposed to delete the oldest eligible version does not free room, despite many drained, poller-less versions being present. ### Expected behavior Either the oldest drained, poller-less version is reliably reclaimed when a new version is registered at the cap (the documented behavior), or drained, poller-less versions are reclaimed in the background before the cap is reached. A deployment doing frequent deploys should not wedge. ### Questions 1. At the cap, does the reclaim path iterate from oldest to newest to find any eligible version (drained, no pollers, not current or ramping), or does it only try the single oldest and give up if that one is ineligible? Our repro suggests eligible versions were present but not reclaimed. 2. What is the exact eligibility predicate for at-cap deletion, and how long after a version drains does it become eligible? 3. Is there any background reclamation below the cap, or is reclamation only ever triggered by registering a new version at the cap? ### Versions and Tags Server 1.30.x (reproduced on 1.29.1 per the forum thread). ### Additional context Community forum thread describing the same symptom, currently unanswered: https://community.temporal.io/t/worker-versioning-garbage-collection-bug/19600

Please feel free to add any details to that issue.

ShivamSaraf · August 5, 2026, 10:10pm

Hello @Sophie , hoping things are well with you.

I work closely with @Jay_Pipes and would love to quickly chime in by saying that auto-deletion of drained, poller-less worker-versions when used specifically with the worker-controller was not working as expected in Temporal Versions v1.29.1 and below.

Having said that, we have patched and fixed this problem in this commit: ( Versioning bug: don't let the server set the last modifier identity to itself during garbage collection by Shivs11 · Pull Request #8614 · temporalio/temporal · GitHub ), which has made it’s first occurrence in Temporal Server Version v1.30.0. We are truly sorry that you are hitting this issue, but would it be possible for you to upgrade your server version to v1.30.0?

If not, we also do have a PR (in progress as we speak) which is aimed at deleting the worker-versions (Temporal Server Side) with calls that are initiating from the Temporal Worker Controller itself. This shall make the experience smoother for folks that are using the worker-controller but choose not to update the server version! I expect this PR to come out in a new worker-controller release quite soon.

viktoriyanavrotskaya · August 11, 2026, 9:17am

The GC only kicks in on a drained version with zero pollers in the last 5 min, so the usual trap is that the pods are deleted but the versions never actually reached “drained” status, or something’s still polling them, so they never become eligible and you hit the hard 100 cap. Worth checking each stuck version’s drainage status and last-poll time directly. Given it’s 1.29.1, could well be a fixed bug, so worth pasting the version states when you file it with the team.

Topic		Replies	Views
Production / Workflow or Worker Versioning Community Support java-sdk , versioning	2	315	August 7, 2024
Announcing Public Preview of Temporal Worker Versioning Announcements versioning , worker	1	297	July 24, 2025
Worker version already exists error when deploying old worker again Community Support	3	384	September 22, 2023
Legacy versioning: How to delete old version sets? Community Support	0	74	June 3, 2025
Worker versioning: Best practices when reverting the default version to an old Build ID? Community Support versioning , worker	0	1005	October 17, 2023

Worker Versioning Garbage Collection Bug?

Related topics