Problem with AsyncCompletionClient (can't find a valid activityId)

Community Support
1

Greetings. Alas, need to ask a question.

Imagine two workflows, no parent/child relationship, A and B. B acts as a queue. A calls B with a signal to enqueue a task (and setting doNotCompleteOnReturn). When B has processed the task it does a callback to to A using the ActivityCompletionClient, passing the token issued from A’s activity.

However, when B’s activity tries to complete the activity on A: it’s throwing an ActivityNotExistsException. But, A is still running (i.e. it hasn’t timed out yet). The time elapsed between A enqueuing the task, and B attempting to callback/complete the activity is ~19m.

We cracked the token and the ActivityId, RunId, etc., look correct.

I’ve tried with simple examples (though using Postgres on docker, same versions of Temporal & SDK) and it works as expected. Which makes this very strange.

Note - prior to upgrading the server to Temporal 1.20.2, this worked great. We were on 1.15.2.

Using:

  • Temporal Java SDK 1.19.1
  • Temporal Server 1.20.2
  • AWS RDS MySQL 5.7.37

Any ideas? Thank you!

Sean

PS. I am aware the “Workflow Update (alpha)” exists in V1.20.0+. My understanding is that eventually we can switch to that pattern to manage queues.

2

Hi Sean,

Have you tried to verify the pending activity via tctl? e.g.

tctl workflow desc -w <replace-with-your-workflow-id> | jq .pendingActivities