Strange Activity Type not registered errors

java.lang.IllegalArgumentException: Activity Type "SendApproverTaskStatusEventEnd" is not registered with a worker. Known types are: CancelApproverTask, SubmitApproverTask, CreateApproverTask 
io.temporal.internal.activity.POJOActivityTaskHandler.handle(POJOActivityTaskHandler.java:110)
io.temporal.internal.worker.ActivityWorker$TaskHandlerImpl.handle(ActivityWorker.java:208)
io.temporal.internal.worker.ActivityWorker$TaskHandlerImpl.handle(ActivityWorker.java:166)
io.temporal.internal.worker.PollTaskExecutor.lambda$process$0(PollTaskExecutor.java:93)
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
java.base/java.lang.Thread.run(Thread.java:829)

But actually the activities are registered.

        worker.registerActivitiesImplementations(approverActivitiesImpl)
        worker.registerActivitiesImplementations(executionActivitiesImpl)
        worker.registerActivitiesImplementations(statusActivitiesImpl) // this one
        worker.registerActivitiesImplementations(localActivitiesImpl)

And the logs say the retry help the activity succeed.

mock ApprovalStatusActivitiesImpl.sendApproverTaskStatusEventEnd(), taskID : 1e7d7a91-56a4-340f-98b1-8a69870a55bf
mock ApprovalStatusActivitiesImpl.sendApproverTaskStatusEventStart(), taskID: cf88f866-67cd-345b-a445-cc069ff59a5e
2022-06-13T21:07:57,389 [Activity Executor taskQueue="tp_queue_approval", namespace="default": 10] DEBUG temporal.ApprovalExecutionActivitiesImpl - [CORRELATION_ID=] - [Normal Activity]: createApprovalExecutionData
mock ApprovalStatusActivitiesImpl.sendApproverTaskStatusEventStart(), taskID: e2f87539-bb80-367e-b2f9-dae89d87386d
mock ApprovalStatusActivitiesImpl.sendApproverTaskStatusEventEnd(), taskID : e2f87539-bb80-367e-b2f9-dae89d87386d

Looks as you might have some worker that does not have the activity registered that is listening on the same activity task queue. Check to make sure all workers have this activity registered.

Hey. I have pasted the evidence that all are registered. Could you take a second look at the second and third paragraph?

I read your whole message.

java.lang.IllegalArgumentException: Activity Type “SendApproverTaskStatusEventEnd” is not registered with a worker. Known types are: CancelApproverTask, SubmitApproverTask, CreateApproverTask

This is when a worker picks up the activity task and says it does not have it registered. If the activity still completes it most likely means that it was retried (server puts the task back on the tq) and another worker that polls on same tq and does have it registered picked it up and completed it.

You can see the number of pollers on the tq that this activity is supposed to execute on via tctl, for example:

tctl tq desc --tq <tq_name>

and see if there is some old worker polling on it that might not have the activity registered.

If you are getting this inside a test, can you show your test? If not, you can also look at your workflow history (please share) and see what could be going on.