ActivityNotExistsException while sending activity heartbeat

Community Support
1

What could cause an activity to fail with an ActivityNotExistsException when calling Activity.heartbeat?

Another observation, I see the below error when the activity is moved from the ‘STARTED’ state to the ‘SCHDULED’ state. Not sure why this happened. Any inputs on this issue is appreciated.

com.uber.cadence.client.ActivityNotExistsException: Execution=WorkflowExecution(workflowId:1dacf648-037c-4457-8e0a-89fb51292d90, runId:3f79d82d-be7d-41e5-a750-5ebad18902ef), ActivityType=SegmentExtractionActivity::runJob, ActivityID=3
	at com.uber.cadence.internal.sync.ActivityExecutionContextImpl.sendHeartbeatRequest(ActivityExecutionContextImpl.java:178)
	at com.uber.cadence.internal.sync.ActivityExecutionContextImpl.doHeartBeat(ActivityExecutionContextImpl.java:131)
	at com.uber.cadence.internal.sync.ActivityExecutionContextImpl.recordActivityHeartbeat(ActivityExecutionContextImpl.java:99)
	at com.uber.cadence.internal.sync.ActivityInternal.recordActivityHeartbeat(ActivityInternal.java:34)
	at com.uber.cadence.activity.Activity.heartbeat(Activity.java:240)
	at com.liveramp.segment.core.operations.lib.SparkActivity.runWithHeartbeat(SparkActivity.scala:28)
	at com.liveramp.segment_extraction_spark.SegmentExtractionActivityImpl.runJob(SegmentExtractionActivityImpl.java:30)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.uber.cadence.internal.sync.POJOActivityTaskHandler$POJOActivityImplementation.execute(POJOActivityTaskHandler.java:214)
	at com.uber.cadence.internal.sync.POJOActivityTaskHandler.handle(POJOActivityTaskHandler.java:190)
	at com.uber.cadence.internal.worker.ActivityWorker$TaskHandlerImpl.handle(ActivityWorker.java:180)
	at com.uber.cadence.internal.worker.ActivityWorker$TaskHandlerImpl.handle(ActivityWorker.java:143)
	at com.uber.cadence.internal.worker.PollTaskExecutor.lambda$process$0(PollTaskExecutor.java:71)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
Caused by: EntityNotExistsError(message:activity task not found)
	at com.uber.cadence.WorkflowService$RecordActivityTaskHeartbeat_result$RecordActivityTaskHeartbeat_resultStandardScheme.read(WorkflowService.java:22476)
	at com.uber.cadence.WorkflowService$RecordActivityTaskHeartbeat_result$RecordActivityTaskHeartbeat_resultStandardScheme.read(WorkflowService.java:22444)
	at com.uber.cadence.WorkflowService$RecordActivityTaskHeartbeat_result.read(WorkflowService.java:22343)
	at org.apache.thrift.TDeserializer.deserialize(TDeserializer.java:81)
	at org.apache.thrift.TDeserializer.deserialize(TDeserializer.java:67)
	at com.uber.tchannel.messages.ThriftSerializer.decodeBody(ThriftSerializer.java:72)
	at com.uber.tchannel.messages.Serializer.decodeBody(Serializer.java:49)
	at com.uber.tchannel.messages.EncodedResponse.getBody(EncodedResponse.java:85)
	at com.uber.cadence.serviceclient.WorkflowServiceTChannel.recordActivityTaskHeartbeat(WorkflowServiceTChannel.java:1109)
	at com.uber.cadence.serviceclient.WorkflowServiceTChannel.lambda$RecordActivityTaskHeartbeat$14(WorkflowServiceTChannel.java:1096)
	at com.uber.cadence.serviceclient.WorkflowServiceTChannel.measureRemoteCall(WorkflowServiceTChannel.java:569)
	at com.uber.cadence.serviceclient.WorkflowServiceTChannel.RecordActivityTaskHeartbeat(WorkflowServiceTChannel.java:1095)
	at com.uber.cadence.internal.sync.ActivityExecutionContextImpl.sendHeartbeatRequest(ActivityExecutionContextImpl.java:171)
	... 18 more
2

Usually, this happens due to activity task timeout. After an activity task times out it the workflow forgets about it. So if a worker tries to heartbeat or complete this activity the EntityNotExistsError(message:activity task not found) is reported.