Namespace deletion workflow is failing due to request timeout

Community Support
1

We recently deleted 3 namespaces from our cluster, but those have been failing (due to timeouts). It has been more than 16 hours now. Deletion workflows are failing and starting again since then. Our persistence is served by astra.
This is the error

wrapError: unable to execute activity: GetNextPageTokenActivity: operation ListOpenWorkflowExecutions encountered Operation timed out - received only 0 responses. (type: Unavailable, retryable: true)
Caused By: unable to execute activity

Now due to this, our critical workflows are also getting impacted.
I have also attached the screenshot of the queries that are timing out.

Screenshot 2023-07-15 at 8.42.36 AM
Screenshot 2023-07-15 at 8.42.36 AM1920×813 133 KB

Anything that we do here to fix this.

2

@tihomir @maxim can you please help here

3

Operation timed out - received only 0 responses

This means Cassandra is not responding. There is some info on this here if that helps.
Do you configure Cassandra for visibility? If so I don’t think this is recommended (should use ES).

Would try to contact your persistence provider and see if they know how to resolve this. Do you have access to Cassandra logs?