Worker version already exists error when deploying old worker again

akashsax14 · September 19, 2023, 9:03pm

Hey,

We are using worker versioning to deploy newer workers to prevent non-deterministic error when things change in our workflow. For deploying a new version, we keep the old version running and we deploy the newer versions. While doing this we are running into the following error consistently:

{"error":{"cause":{"code":3,"details":"**version 1.8.0 already exists**","metadata":{"content-type":["application/grpc"]}}},"level":"error","message":"Issue while setting up temporal worker","timestamp":"2023-09-19T20:52:03.765Z"}

The version mentioned in the above error is the older version. While deploying the new version, for some reason our deploy tool re-deploys the old version as well. Could this be the reason for the above error?

Any ideas what we might be doing wrong while deploying the new version.

bergundy · September 20, 2023, 2:42am

This error indicates that you’re trying to set build ID 1.8.0 as default for the queue but it’s already set as non default.

github.com

temporalio/temporal/blob/69cc69763af17a456b2ce4efc3f4f520325bedca/service/matching/version_sets.go#L225


      
          
          	if req.GetAddNewBuildIdInNewDefaultSet() != "" {
          		targetIsInDefaultSet := targetSetIdx == numExistingSets-1
          		targetIsOnlyBuildIdInSet := versionInSetIdx == 0 && len(data.VersionSets[numExistingSets-1].BuildIds) == 1
          		// Make the request idempotent
          		if numExistingSets > 0 && targetIsInDefaultSet && targetIsOnlyBuildIdInSet {
          			return data, nil
          		}
          		// If it's not already in the sets, add it as the new default set
          		if targetSetIdx != -1 {
          			return nil, serviceerror.NewInvalidArgument(fmt.Sprintf("version %s already exists", targetedVersion))
          		}
          
          		data.VersionSets = append(data.GetVersionSets(), &persistencespb.CompatibleVersionSet{
          			SetIds:   []string{hashBuildId(targetedVersion)},
          			BuildIds: []*persistencespb.BuildId{{Id: targetedVersion, State: persistencespb.STATE_ACTIVE, StateUpdateTimestamp: &timestamp}},
          		})
          		makeVersionInSetDefault(data, len(data.VersionSets)-1, 0, &timestamp)
          		makeDefaultSet(data, len(data.VersionSets)-1, &timestamp)
          	} else if addNew := req.GetAddNewCompatibleBuildId(); addNew != nil {
          		compatVer := addNew.GetExistingCompatibleBuildId()

Ivan · September 21, 2023, 5:16pm

Thank you, we had a bug in compatibility update logic.

It looks like today it’s not possible to remove sets/buildIds. I can do it by removing records per task queue in task_queue_user_data and build_id_to_task_queue if I want to reset sets. But is there a better way?

bergundy · September 22, 2023, 6:11am

We don’t allow manually removing build IDs because workflows may be linked to them and would get stuck.
There’s a scavenger process that automatically removes build IDs that have no references.