The problem: our product runs a set a workflows using a schedule with BUFFER_ONE (it actually runs one workflow, but many get created as children) and we have a need to be able to upgrade the product reliably without having workflow issues. We are OK with not completing workflows in flight on upgrade, the schedule is regular enough.
Worker versioning with workflow pinning seems promising, but I wonder how that will interact with existing in flight workflows. I am assuming that without further action the schedule will not start another workflow until in flight workflows are done. Of course, they won’t be done because we’ve removed the workers (and since this is a deployed product in customer environments we don’t have the flexibility one might have otherwise so upgrade is upgrade everything - no draining.) Would resetting the scheduled workflow with the new build_id fix this? What happens to any child workflows that have been started, would they need the same treatment? Is that even feasible under heavy dev changes? I can imagine scenarios where workflow/activity inputs change deep in the hierarchy.
Terminating the running workflows is also an option - there is nothing they do that next run can’t catch up. What would be the best way to do that given there are no workers for the old build id, can we terminate purely temporal server side? This would be the simpler option if possible.