I’ve read through the various docs on versioning, and it looks to me like a robust versioning framework has been implemented for Workflows. I am trying to find a similar pattern for Activities, but failing.
When running a Workflow, I see that it is possible to specify the version of the workflow to run in the SDK, have multiple workers running on the same task queue with different versions, and the Temporal framework will route the request to the worker on the task queue satisfying the requested version (if available).
For Activities, I only see that the SDK exposes the deprecated flag for a VersioningIntent, but even then, no way to specify the desired version. Similarly, I see no way for the worker to specify its version when registering an Activity. Absent anything in the SDK, I’m thinking the only way to implement versioning for my Activities is to add the semantic version of the Activity to the task queue name. This would result in an explosion of task queues, as well as a somewhat clunky mechanism for Workflows to discover the task queue satisfying the version of the Activity they require. This approach is mentioned in some of the versioning docs, but discouraged in favor of the preferred strategy – which, as far as I can tell, only exists for Workflows.
Is there a recommended approach to versioning Activities? Or are there plans to roll out a similar pattern for Workflow versioning to Activities in a future release? I’m interested to hear how others have solved this problem. And in anticipation of possible replies, I know one approach is to make all Activity code backwards compatible, but this really doesn’t address the problem… if one team builds a Workflow and tests it against a known set of versioned Activity dependencies, they don’t want their Workflow breaking when the behavior of one of those Activity implementations changes later, even if the Activity maintains a backwards-compatible API.
After reading through the source code for the Temporal Service, I have confirmed that it is not possible to request a specific version of an Activity when calling it, except possibly by inheriting the build ID from the Workflow.
Based on my testing, I am able to create Deployments with Build IDs (and set as current or ramping) for both Workflow and Activity workers. There is no way to specify the versioningOverride when executing an Activity, however it is possible to have Activity execution inherit the build ID of the Workflow… but that would not meet my needs as I have Workflows and Activities developed by different teams on different release cycles.
So, I’ve concluded that adding support for Activity Worker versioning would require:
adding the versioningOverride attribute to ActivityOptions
generalizing the Workflow-specific code in worker_versioning.go to apply to Activities as well (mostly renaming variables)
requests to scheduleActivity to reuse the code in worker_versioning.go to find the Worker corresponding to the correct Deployment – possibly by changing wh.validateVersioningInfo in PollActivityTaskQueue in workflow_handler.go to use VersioningOverride rather than WorkerVersionCapabilities?
I’m curious if anyone is working on this, or if it is not planned until Worker versioning for Workflows has gone through some iterations that might change the API.
Hi @jbrody
I’m one of the developers of worker versioning. If I understand correctly, it sounds like your goal here is to version your Activities and Workflows separately because they have separate release schedules.
This is currently supported by Worker Deployments, in that Activity Task Queues / Activity-only workers and Workflow Task Queues / Workflow-only workers can be part of separate Worker Deployments.
For example, when your Activity code is ready to be released, you could deploy an Activity-only worker with DeploymentName=”my-activity-deployment” and BuildID=”xyz” polling on TaskQueue=”my-task-queue". This will create an Activity Task Queue named “my-task-queue” that is a part of the Worker Deployment “my-activity-deployment”. If you then ramp traffic to and eventually set current BuildID “xyz” on Worker Deployment “my-activity-deployment”, all tasks on that Activity Task Queue will go to Activity workers with BuildID “xyz”.
At some other time, when your Workflow code is ready to be released, you can deploy a Workflow-only worker with DeploymentName=”my-workflow-deployment” and BuildID=”abc” (technically you could give it the same build id, but that might be confusing). You could let that Workflow-worker poll on TaskQueue=”my-task-queue” (or a different name), which will create a Workflow Task Queue that is part of the Worker Deployment “my-workflow-deployment”. Same as above, if you then set BuildID “abc” as the current version of Worker Deployment “my-workflow-deployment”, all tasks on your Workflow Task Queue will go to Workflow workers with BuildID “abc”.
If you do things this way, you’ll need to be mindful of keeping Activity-interface compatibility between the workflow and activity definitions that are being used together at the same time.
Please let me know if this setup does not address your use-case!
Hi Carly, thanks for the reply! What you describe is exactly what I’m doing – my Activity-only workers add themselves to a Deployment using the Activity version as the Build ID. And I can see those versions in the Deployment in the Temporal UI.
However, when a Workflow calls an Activity and knows which version (Build ID) of the Activity it should use, there is no way to specify in the ActivityOptions that the execution should be pinned to a specific Deployment/Build ID. So, maybe your solution works in case the desired behavior is AUTO-UPGRADE, but it does not support the use case for executing an Activity with a PINNED version. I have not tested to confirm that your solution implements AUTO-UPGRADE correctly, but will try it out of curiosity.
Can you advise on potential workarounds or a timeline for supporting PINNED versioning for Activities? I did open a feature request in the Temporal GitHub as well.
Ah ok, I think I’m understanding your desired behavior here.
Is it that basically you know that Workflows with build 1w have interface compatibility with Activities with build 1a
So even if you deploy a newer Activity worker with build 2a, you want to make it so that the Workflows that are pinned to build 1w have the Actvities that they invoke run on build 1a instead of 2a, which is the “current version” of the Activity Worker Deployment?
Yes, that’s exactly right. That way the owner of the Activity can push version 2a and know that it can’t break any Workflows using pinned behavior. Workflows owners can either test and deploy a new version pinned to 2a, or they can accept auto-upgrade behavior if they want to receive updated functionality at the risk of regression. However they need to be able to specify this in the ActivityOptions.
Re: the activity versioning issue, would offering a VersioningOverride in ActivityOptions solve this problem for you? Like, are you able to make it so that the workflow that is invoking the Activity always knows what Activity-worker-version it wants to use? This was not on our radar, but I agree your use case should be solved, so really appreciate you bringing it up!
Re: the issue you linked, see my response on the issue itself! Let’s continue that conversation there.
Another note since it sounds like you’re using the pre-release worker versioning (v1.27.x), please be aware that we have made breaking changes to the implementation of Worker Deployments between pre-release and public preview, and you’ll need to delete all of your v1.27.x worker deployments before upgrading and then create new ones after upgrading to v1.28.0. You can see the v1.28.0 release notes or the #safe-deploys channel in the community slack for more details about that.
Thanks for the tip, I will try to get our servers back on 1.28+ ASAP, and will make sure to redo the Deployments when I do. I’ve also joined the Slack group.