Temporal CLI (tctl) commands throwing "operation GetCurrentExecution encountered not found" error

Community Support
, , ,
1

We are running several performance tests on our temporal-cassandra instances to determine the right configuration. As part of that, thousands of workflows are created for every run. Before a new test is triggered we try to clean up the running workflows but most of them are failing with exceptions.

Below are the two commands tried and each of them are failing with respective errors.

tctl wf terminate --wid $id

Error: Terminate workflow failed.
Error Details: operation GetCurrentExecution encountered not found
(‘export TEMPORAL_CLI_SHOW_STACKS=1’ to see stack traces)

tctl admin workflow delete --wid $id

Error: Get workflow mutableState failed
Error Details: rpc error: code = NotFound desc = operation GetCurrentExecution encountered not found
(‘export TEMPORAL_CLI_SHOW_STACKS=1’ to see stack traces)

Using tctl version 1.14.0

Any help is appreciated.

3

Hi @svaddempudi
Are you sure the workflow ids are valid? These error typically indicate that the workflow with the provided id does not exist.

4

Hi @tihomir

Thanks for your response.

Yes. They are valid. Actually they are in thousands so I am listing them first using tctl and looping the list to terminate. Some of them are actually getting terminated but some of them are failing with the errors I mentioned earlier.

Here is the for loop.

for id in $(tctl wf listall --open --ns default | awk ‘{print $3}’ | grep -v |); do
echo “The worklfow id is” $id
tctl wf desc --wid $id
tctl wf terminate --wid $id
RESULT=$?
if [ ${RESULT} -eq 1 ]; then
echo “Termination of workload id ${id} is failed. Please check /tmp/terminate.txt” | tee -a /tmp/terminate.txt
else
echo “Termination of workload id ${id} is successful”
fi
done

To summarize, some workflows I could list using tctl are failing to terminate or describe with a not found GetCurrentExecution operation

5

The NotFound error with message "operation GetCurrentExecution encountered not found" comes from Cassandra. Can you check if it reports any errors? I am not sure what’s going on if those workflow Ids are correct but looks as possible Cassandra issues.