We are using activities/workflow with session to have stickiness with worker. However we are seeing some of our long running activities timing out sporadically (even though we are heartbeating regularly at ~5 mins) which is resulting in “Context deadline exceeded” errors. We also notice some session failure message as below:
debug internal/session.go:363 Session failed {“Namespace”: “hammer”, “TaskQueue”: “haas-tq-aws-prod4-apsoutheast2-coretip1-hmr121”, “WorkerID”: “7@hmr121-hammer-hmrapp-79758f4f5f-8qq9l@”, “WorkflowType”: “HammerExecutionUnitWorkflow”, “WorkflowID”: “94e6a487-ea21-4e24-8d7d-667cef832c73_apex_1”, “RunID”: “c68a9200-4871-47b2-9ac1-e773ddcc0810”, “Attempt”: 1, “sessionID”: “7a800394-cb7d-4c7c-a5df-53e803d520a5”, “Error”: “activity error (type: internalSessionCreationActivity, scheduledEventID: 6, startedEventID: 344, identity: ): activity Heartbeat timeout (type: Heartbeat)”, “logging-call-at”: “internal_workflow.go:1070”}
Just wondering what could have caused it…Typically for how long these sessions are kept valid and any guidance around when these session should be recreated?