Hi everyone!
Our team encountered strange behavior. We had many identical workflows running in parallel (about 100 workflows). All workflows were spawned as child of the same parent
At some point, they began to create a heavy load on the infrastructure and we needed to stop them. We stopped the worker processing the tasks queue and then terminated the workflows through the user interface (the workflows stopped because the parent workflow was stopped). After we restarted the worker, the load began to increase again, but none of the workflows were running. Someone suggested that perhaps the task queue still stores tasks for executing the activities. So we removed all the logic from the activities (just “return nil”), deployed the changes and restarted the worker. The load no longer increased. After some time, we returned the activity to a working state and the load no longer increased.
I know it sounds strange
But we had a question to which we did not find the answer either in the official documentation or on the forum
Could it be that the task queue stores tasks for executing the activity even after the workflow in which the activity was supposed to be executed has stopped? Especially in cases where it is necessary to stop many workflows. In our case, the workflows stopped because the parent workflow was stopped
We use golang sdk.