Activity HeartBeat Issue

Hello All,

 I am trying to play around activity Heatbeat feature 

I am facing an issue with

 activity.RecordHeartbeat()

In this go-example

I did some additional changes

changed activity option to:

  	ao := workflow.ActivityOptions{
	ScheduleToStartTimeout: time.Minute,
	StartToCloseTimeout:    time.Minute * 30,
	HeartbeatTimeout:       30 * time.Second,
	WaitForCancellation:    true,
	RetryPolicy:            &temporal.RetryPolicy{InitialInterval: 1 * time.Second},
} 

and I Replaced the “ActivityToBeCanceled” (added some logging related to heartBeat)

   func (a *Activities) ActivityToBeCanceled(ctx context.Context) (string, error) {
logger := activity.GetLogger(ctx)
logger.Info("activity started, to cancel workflow, use 'go run cancelactivity/cancel/main.go " +
	"-w <WorkflowID>' or CLI: 'tctl wf cancel -w <WorkflowID>' to cancel")
a.test++
heartBeatTime := time.Time{}
for {
	select {
	case <-time.After(1 * time.Second):
		seq := fmt.Sprintf("heat beat det %v", time.Now())
		activity.RecordHeartbeat(ctx, seq)

		if time.Now().After(heartBeatTime.Add(10 * time.Second)) {
			fmt.Printf("HeartBeating....%v: \n", time.Now())
			heartBeatTime = time.Now()

			if activity.HasHeartbeatDetails(ctx) {
				var prev string
				if err := activity.GetHeartbeatDetails(ctx, &prev); err != nil {
					clog.Error(ctx, "GetHeartbeatDetails fail", err)
				}
				fmt.Printf("value of heart beat det : %v \n", prev)

			} else {
				fmt.Printf("has heartbeat false %v:\n ", time.Now())
			}
		}
	case <-ctx.Done():
		logger.Info("context is cancelled")
		return "I am canceled by Done", nil
	}
}

}

when I am running this I am getting logs

1   2020-08-29T13:30:20.574+0530	INFO	log/with_logger.go:60	activity started, to cancel workflow, 
2   HeartBeating....:         2020-08-29 13:30:21.593334 +0530 IST m=+5.168517102 
3   has heartbeat false       2020-08-29 13:30:21.593361 +0530 IST m=+5.168544344:
4   HeartBeating....:         2020-08-29 13:30:31.620144 +0530 IST m=+15.195262496 
5   has heartbeat false       2020-08-29 13:30:31.620178 +0530 IST m=+15.195296394:
6   HeartBeating....:         2020-08-29 13:30:41.638978 +0530 IST m=+25.214031810 
7   has heartbeat false       2020-08-29 13:30:41.639046 +0530 IST m=+25.214100403:
8   HeartBeating....:         2020-08-29 13:30:51.664958 +0530 IST m=+35.239948039 
9   has heartbeat false       2020-08-29 13:30:51.664988 +0530 IST m=+35.239977791:
10  2020-08-29T13:30:52.605+0530	INFO	log/with_logger.go:60	activity started, to cancel workflow, 
11  HeartBeating....:         2020-08-29 13:30:53.617806 +0530 IST m=+37.192782776 
12  value of heart beat det:  2020-08-29 13:30:21.579813 +0530 IST m=+5.154996146 
    HeartBeating....:         2020-08-29 13:31:01.692941 +0530 IST m=+45.267865867 
    has heartbeat false       2020-08-29 13:31:01.692974 +0530 IST m=+45.267899475:
    HeartBeating....:         2020-08-29 13:31:03.64545 +0530 IST m=+47.220362588 
    value of heart beat det:  2020-08-29 13:30:21.579813 +0530 IST m=+5.154996146 
    HeartBeating....:         2020-08-29 13:31:11.717037 +0530 IST m=+55.291897364 
    has heartbeat false       2020-08-29 13:31:11.717057 +0530 IST m=+55.291917021:
    HeartBeating....:         2020-08-29 13:31:13.679608 +0530 IST m=+57.254456094 
    value of heart beat det:  2020-08-29 13:30:21.579813 +0530 IST m=+5.154996146 
    HeartBeating....:         2020-08-29 13:31:21.746301 +0530 IST m=+65.321096978 
    has heartbeat false       2020-08-29 13:31:21.746335 +0530 IST m=+65.321130773:
    HeartBeating....:         2020-08-29 13:31:23.703096 +0530 IST m=+67.277879083 
    value of heart beat det:  2020-08-29 13:30:21.579813 +0530 IST m=+5.154996146 
    2020-08-29T13:31:25.628+0530	INFO	log/with_logger.go:60	activity started, to cancel workflow, 
    HeartBeating....:         2020-08-29 13:31:26.642248 +0530 IST m=+70.217012505 
    value of heart beat det:  2020-08-29 13:30:53.606229 +0530 IST m=+37.181206030 
    HeartBeating....:         2020-08-29 13:31:31.774526 +0530 IST m=+75.349257482 
    has heartbeat false       2020-08-29 13:31:31.774553 +0530 IST m=+75.349283756:
    HeartBeating....:         2020-08-29 13:31:33.723904 +0530 IST m=+77.298622057 
    value of heart beat det:  2020-08-29 13:30:21.579813 +0530 IST m=+5.154996146 

I suspect only the first heartbeat is taking into consideration ( the one that fires when activity start , means it behaves like the heartbeat is not working ) we can see activity retrying because of heartbeat timeout, even in the next retry we could see the heartbeat (the argument (the passed value as details) is the last activity start time, not the last heartbeat time)

line number “12” in the logs prints time

 "020-08-29 13:30:21.579813"   ( the first try start time)

but it should be near to

 "2020-08-29 13:30:51.66495".  (the first try end time )

?

what is going on here, is it expected behavior?

I needed to run a long activity that should retry immediately when server worker restarts ( so I need to add retry option, but this behavior blocking me

any help would be really helpful

thanks in advance,
Junaid

1 Like

I’m not able to reproduce it on my notebook. I copied your code and this is what I’m getting. It is still going.

So my guess is that you indeed have some issue with connectivity from the activity to the service.

 workflow started Namespace default TaskQueue hello-world WorkerID 21573@maxpro.local@ WorkflowType Workflow WorkflowID hello_world_workflowID RunID 2135589c-6038-450f-b91a-942394618658 name Temporal
2020/08/31 09:03:49 DEBUG ExecuteActivity Namespace default TaskQueue hello-world WorkerID 21573@maxpro.local@ WorkflowType Workflow WorkflowID hello_world_workflowID RunID 2135589c-6038-450f-b91a-942394618658 ActivityID 5 ActivityType ActivityToBeCanceled
2020/08/31 09:03:49 INFO  activity started, to cancel workflow, use 'go run cancelactivity/cancel/main.go -w <WorkflowID>' or CLI: 'tctl wf cancel -w <WorkflowID>' to cancel Namespace default TaskQueue hello-world WorkerID 21573@maxpro.local@ ActivityID 5 ActivityType ActivityToBeCanceled WorkflowType Workflow WorkflowID hello_world_workflowID RunID 2135589c-6038-450f-b91a-942394618658
HeartBeating....2020-08-31 09:03:50.659767 -0700 PDT m=+4.950487523: 
has heartbeat false 2020-08-31 09:03:50.659785 -0700 PDT m=+4.950505353:
 HeartBeating....2020-08-31 09:04:00.695896 -0700 PDT m=+14.986499008: 
has heartbeat false 2020-08-31 09:04:00.695914 -0700 PDT m=+14.986517145:
 HeartBeating....2020-08-31 09:04:10.720099 -0700 PDT m=+25.010585199: 
has heartbeat false 2020-08-31 09:04:10.720111 -0700 PDT m=+25.010597656:
 HeartBeating....2020-08-31 09:04:20.751286 -0700 PDT m=+35.041655530: 
has heartbeat false 2020-08-31 09:04:20.751322 -0700 PDT m=+35.041691512:
 HeartBeating....2020-08-31 09:04:30.783135 -0700 PDT m=+45.073388271: 
has heartbeat false 2020-08-31 09:04:30.783166 -0700 PDT m=+45.073418470:
 HeartBeating....2020-08-31 09:04:40.813368 -0700 PDT m=+55.103504334: 
has heartbeat false 2020-08-31 09:04:40.813402 -0700 PDT m=+55.103538272:
 HeartBeating....2020-08-31 09:04:50.847494 -0700 PDT m=+65.137513126: 
has heartbeat false 2020-08-31 09:04:50.847529 -0700 PDT m=+65.137547766:
 HeartBeating....2020-08-31 09:05:00.876499 -0700 PDT m=+75.166401596: 
has heartbeat false 2020-08-31 09:05:00.876519 -0700 PDT m=+75.166421526:
 HeartBeating....2020-08-31 09:05:10.899475 -0700 PDT m=+85.189260363: 
has heartbeat false 2020-08-31 09:05:10.899492 -0700 PDT m=+85.189277320:
 HeartBeating....2020-08-31 09:05:20.931878 -0700 PDT m=+95.221546827: 
has heartbeat false 2020-08-31 09:05:20.931929 -0700 PDT m=+95.221597296:
 HeartBeating....2020-08-31 09:05:30.961863 -0700 PDT m=+105.251414630: 
has heartbeat false 2020-08-31 09:05:30.961888 -0700 PDT m=+105.251439340:
 HeartBeating....2020-08-31 09:05:40.992841 -0700 PDT m=+115.282276182: 
has heartbeat false 2020-08-31 09:05:40.992876 -0700 PDT m=+115.282310570:
 HeartBeating....2020-08-31 09:05:51.029184 -0700 PDT m=+125.318502055: 
has heartbeat false 2020-08-31 09:05:51.029208 -0700 PDT m=+125.318525495:
 HeartBeating....2020-08-31 09:06:01.062211 -0700 PDT m=+135.351411594: 
has heartbeat false 2020-08-31 09:06:01.062227 -0700 PDT m=+135.351427750:
 HeartBeating....2020-08-31 09:06:11.094616 -0700 PDT m=+145.383699797: 
has heartbeat false 2020-08-31 09:06:11.094649 -0700 PDT m=+145.383733301:
 HeartBeating....2020-08-31 09:06:21.126464 -0700 PDT m=+155.415430820: 
has heartbeat false 2020-08-31 09:06:21.126481 -0700 PDT m=+155.415448722:
 HeartBeating....2020-08-31 09:06:31.154584 -0700 PDT m=+165.443433980: 
has heartbeat false 2020-08-31 09:06:31.154602 -0700 PDT m=+165.443452927:
 HeartBeating....2020-08-31 09:06:41.185037 -0700 PDT m=+175.473770893: 
has heartbeat false 2020-08-31 09:06:41.185066 -0700 PDT m=+175.473799636:
 HeartBeating....2020-08-31 09:06:51.222007 -0700 PDT m=+185.510623800: 
has heartbeat false 2020-08-31 09:06:51.222022 -0700 PDT m=+185.510638790:
 HeartBeating....2020-08-31 09:07:01.249169 -0700 PDT m=+195.537669173: 
has heartbeat false 2020-08-31 09:07:01.249199 -0700 PDT m=+195.537698632:
 HeartBeating....2020-08-31 09:07:11.27228 -0700 PDT m=+205.560663120: 
has heartbeat false 2020-08-31 09:07:11.2723 -0700 PDT m=+205.560682908:
 HeartBeating....2020-08-31 09:07:21.307676 -0700 PDT m=+215.595941733: 
has heartbeat false 2020-08-31 09:07:21.307698 -0700 PDT m=+215.595964347:
 HeartBeating....2020-08-31 09:07:31.325498 -0700 PDT m=+225.613647883: 
has heartbeat false 2020-08-31 09:07:31.325518 -0700 PDT m=+225.613666941:
 HeartBeating....2020-08-31 09:07:41.349389 -0700 PDT m=+235.637422053: 
has heartbeat false 2020-08-31 09:07:41.349426 -0700 PDT m=+235.637458247:
 HeartBeating....2020-08-31 09:07:51.368741 -0700 PDT m=+245.656656980: 
has heartbeat false 2020-08-31 09:07:51.368767 -0700 PDT m=+245.656682975:
 HeartBeating....2020-08-31 09:08:01.400565 -0700 PDT m=+255.688364029: 
has heartbeat false 2020-08-31 09:08:01.40058 -0700 PDT m=+255.688379360:
 HeartBeating....2020-08-31 09:08:11.427202 -0700 PDT m=+265.714883908: 
has heartbeat false 2020-08-31 09:08:11.427224 -0700 PDT m=+265.714906321:
 HeartBeating....2020-08-31 09:08:21.458756 -0700 PDT m=+275.746321772: 
has heartbeat false 2020-08-31 09:08:21.458793 -0700 PDT m=+275.746358539:
 HeartBeating....2020-08-31 09:08:31.480857 -0700 PDT m=+285.768305524: 
has heartbeat false 2020-08-31 09:08:31.480879 -0700 PDT m=+285.768327910:
 HeartBeating....2020-08-31 09:08:41.514181 -0700 PDT m=+295.801513047: 
has heartbeat false 2020-08-31 09:08:41.514234 -0700 PDT m=+295.801565739:
 HeartBeating....2020-08-31 09:08:51.543881 -0700 PDT m=+305.831096132: 
has heartbeat false 2020-08-31 09:08:51.543921 -0700 PDT m=+305.831135596:
 HeartBeating....2020-08-31 09:09:01.576618 -0700 PDT m=+315.863715549: 
has heartbeat false 2020-08-31 09:09:01.576636 -0700 PDT m=+315.863734224:
 HeartBeating....2020-08-31 09:09:11.605066 -0700 PDT m=+325.892047424: 
has heartbeat false 2020-08-31 09:09:11.605104 -0700 PDT m=+325.892084805:
 HeartBeating....2020-08-31 09:09:21.632644 -0700 PDT m=+335.919508559: 
has heartbeat false 2020-08-31 09:09:21.632668 -0700 PDT m=+335.919531946:
 HeartBeating....2020-08-31 09:09:31.655607 -0700 PDT m=+345.942355058: 
has heartbeat false 2020-08-31 09:09:31.655621 -0700 PDT m=+345.942368797:
 HeartBeating....2020-08-31 09:09:41.684531 -0700 PDT m=+355.971162255: 
has heartbeat false 2020-08-31 09:09:41.684563 -0700 PDT m=+355.971193280:
 HeartBeating....2020-08-31 09:09:51.714787 -0700 PDT m=+366.001300698: 
has heartbeat false 2020-08-31 09:09:51.714807 -0700 PDT m=+366.001320499:
 HeartBeating....2020-08-31 09:10:01.746824 -0700 PDT m=+376.033221129: 
has heartbeat false 2020-08-31 09:10:01.746854 -0700 PDT m=+376.033250577:
 HeartBeating....2020-08-31 09:10:11.785161 -0700 PDT m=+386.071440775: 
has heartbeat false 2020-08-31 09:10:11.785174 -0700 PDT m=+386.071453920:
 HeartBeating....2020-08-31 09:10:21.813451 -0700 PDT m=+396.099613766: 
has heartbeat false 2020-08-31 09:10:21.813472 -0700 PDT m=+396.099634901:
 HeartBeating....2020-08-31 09:10:31.843164 -0700 PDT m=+406.129210675: 
has heartbeat false 2020-08-31 09:10:31.843177 -0700 PDT m=+406.129223043:
 HeartBeating....2020-08-31 09:10:41.879308 -0700 PDT m=+416.165237492: 
has heartbeat false 2020-08-31 09:10:41.879325 -0700 PDT m=+416.165254485:
 HeartBeating....2020-08-31 09:10:51.915684 -0700 PDT m=+426.201496577: 
has heartbeat false 2020-08-31 09:10:51.915735 -0700 PDT m=+426.201547894:
 HeartBeating....2020-08-31 09:11:01.949263 -0700 PDT m=+436.234958710: 
has heartbeat false 2020-08-31 09:11:01.949298 -0700 PDT m=+436.234994091:
 HeartBeating....2020-08-31 09:11:11.975104 -0700 PDT m=+446.260682723: 
has heartbeat false 2020-08-31 09:11:11.975158 -0700 PDT m=+446.260736696:
 HeartBeating....2020-08-31 09:11:21.99589 -0700 PDT m=+456.281351890: 
has heartbeat false 2020-08-31 09:11:21.995929 -0700 PDT m=+456.281390942:
 HeartBeating....2020-08-31 09:11:32.029294 -0700 PDT m=+466.314639024: 
has heartbeat false 2020-08-31 09:11:32.029309 -0700 PDT m=+466.314653845:
 HeartBeating....2020-08-31 09:11:42.058842 -0700 PDT m=+476.344069963: 
has heartbeat false 2020-08-31 09:11:42.05887 -0700 PDT m=+476.344097890:
 HeartBeating....2020-08-31 09:11:52.090673 -0700 PDT m=+486.375784553: 
has heartbeat false 2020-08-31 09:11:52.090702 -0700 PDT m=+486.375813193:
 HeartBeating....2020-08-31 09:12:02.109862 -0700 PDT m=+496.394856322: 
has heartbeat false 2020-08-31 09:12:02.109882 -0700 PDT m=+496.394876760:
 HeartBeating....2020-08-31 09:12:12.141778 -0700 PDT m=+506.426656335: 
has heartbeat false 2020-08-31 09:12:12.141794 -0700 PDT m=+506.426671403:
 HeartBeating....2020-08-31 09:12:22.174754 -0700 PDT m=+516.459515046: 
has heartbeat false 2020-08-31 09:12:22.174783 -0700 PDT m=+516.459543907:
 HeartBeating....2020-08-31 09:12:32.205348 -0700 PDT m=+526.489991951: 
has heartbeat false 2020-08-31 09:12:32.205362 -0700 PDT m=+526.490005890:
 HeartBeating....2020-08-31 09:12:42.237783 -0700 PDT m=+536.522310548: 
has heartbeat false 2020-08-31 09:12:42.237806 -0700 PDT m=+536.522333196:
 HeartBeating....2020-08-31 09:12:52.275171 -0700 PDT m=+546.559581163: 
has heartbeat false 2020-08-31 09:12:52.27519 -0700 PDT m=+546.559600503:
 HeartBeating....2020-08-31 09:13:02.311272 -0700 PDT m=+556.595565202: 
has heartbeat false 2020-08-31 09:13:02.311309 -0700 PDT m=+556.595601985:
 HeartBeating....2020-08-31 09:13:12.342152 -0700 PDT m=+566.626328898: 
has heartbeat false 2020-08-31 09:13:12.342169 -0700 PDT m=+566.626345623:
 HeartBeating....2020-08-31 09:13:22.371558 -0700 PDT m=+576.655618218: 
has heartbeat false 2020-08-31 09:13:22.371584 -0700 PDT m=+576.655643291:
 HeartBeating....2020-08-31 09:13:32.398077 -0700 PDT m=+586.682019818: 
has heartbeat false 2020-08-31 09:13:32.398107 -0700 PDT m=+586.682049969:
 HeartBeating....2020-08-31 09:13:42.422725 -0700 PDT m=+596.706550731: 
has heartbeat false 2020-08-31 09:13:42.422758 -0700 PDT m=+596.706584361:
 HeartBeating....2020-08-31 09:13:52.462665 -0700 PDT m=+606.746373900: 
has heartbeat false 2020-08-31 09:13:52.462748 -0700 PDT m=+606.746457437:
 HeartBeating....2020-08-31 09:14:02.494135 -0700 PDT m=+616.777727501: 
has heartbeat false 2020-08-31 09:14:02.494173 -0700 PDT m=+616.777765239:
 HeartBeating....2020-08-31 09:14:12.521546 -0700 PDT m=+626.805021844: 
has heartbeat false 2020-08-31 09:14:12.521578 -0700 PDT m=+626.805053949:
 HeartBeating....2020-08-31 09:14:22.546212 -0700 PDT m=+636.829571006: 
has heartbeat false 2020-08-31 09:14:22.546249 -0700 PDT m=+636.829607580:
 HeartBeating....2020-08-31 09:14:32.574292 -0700 PDT m=+646.857533472: 
has heartbeat false 2020-08-31 09:14:32.574328 -0700 PDT m=+646.857570280:
 HeartBeating....2020-08-31 09:14:42.6083 -0700 PDT m=+656.891424693: 
has heartbeat false 2020-08-31 09:14:42.608321 -0700 PDT m=+656.891446406:
 HeartBeating....2020-08-31 09:14:52.637305 -0700 PDT m=+666.920313307: 
has heartbeat false 2020-08-31 09:14:52.637334 -0700 PDT m=+666.920342507:
 HeartBeating....2020-08-31 09:15:02.669385 -0700 PDT m=+676.952276542: 
has heartbeat false 2020-08-31 09:15:02.66942 -0700 PDT m=+676.952311274:
 HeartBeating....2020-08-31 09:15:12.704768 -0700 PDT m=+686.987542772: 
has heartbeat false 2020-08-31 09:15:12.704784 -0700 PDT m=+686.987558589:
 HeartBeating....2020-08-31 09:15:22.738569 -0700 PDT m=+697.021226226: 
has heartbeat false 2020-08-31 09:15:22.738587 -0700 PDT m=+697.021244342:
 HeartBeating....2020-08-31 09:15:32.772872 -0700 PDT m=+707.055412626: 
has heartbeat false 2020-08-31 09:15:32.772895 -0700 PDT m=+707.055435308:
 HeartBeating....2020-08-31 09:15:42.805413 -0700 PDT m=+717.087837068: 
has heartbeat false 2020-08-31 09:15:42.805427 -0700 PDT m=+717.087850689:
 HeartBeating....2020-08-31 09:15:52.8252 -0700 PDT m=+727.107507222: 
has heartbeat false 2020-08-31 09:15:52.825235 -0700 PDT m=+727.107541890:
 HeartBeating....2020-08-31 09:16:02.856859 -0700 PDT m=+737.139049054: 
has heartbeat false 2020-08-31 09:16:02.856885 -0700 PDT m=+737.139074934:
 HeartBeating....2020-08-31 09:16:12.880388 -0700 PDT m=+747.162460785: 
has heartbeat false 2020-08-31 09:16:12.880401 -0700 PDT m=+747.162474763:
 HeartBeating....2020-08-31 09:16:22.908114 -0700 PDT m=+757.190070887: 
has heartbeat false 2020-08-31 09:16:22.908127 -0700 PDT m=+757.190083496:
 HeartBeating....2020-08-31 09:16:32.939354 -0700 PDT m=+767.221193992: 
has heartbeat false 2020-08-31 09:16:32.939374 -0700 PDT m=+767.221213632:
 HeartBeating....2020-08-31 09:16:42.971222 -0700 PDT m=+777.252944667: 
has heartbeat false 2020-08-31 09:16:42.971233 -0700 PDT m=+777.252956073:
 HeartBeating....2020-08-31 09:16:52.999785 -0700 PDT m=+787.281391197: 
has heartbeat false 2020-08-31 09:16:52.999801 -0700 PDT m=+787.281406633:
 HeartBeating....2020-08-31 09:17:03.036386 -0700 PDT m=+797.317874725: 
has heartbeat false 2020-08-31 09:17:03.036398 -0700 PDT m=+797.317887284:
 HeartBeating....2020-08-31 09:17:13.065 -0700 PDT m=+807.346371868: 
has heartbeat false 2020-08-31 09:17:13.065027 -0700 PDT m=+807.346399425:
 HeartBeating....2020-08-31 09:17:23.096263 -0700 PDT m=+817.377518355: 
has heartbeat false 2020-08-31 09:17:23.096323 -0700 PDT m=+817.377577913:
 HeartBeating....2020-08-31 09:17:33.121913 -0700 PDT m=+827.403051011: 
has heartbeat false 2020-08-31 09:17:33.121923 -0700 PDT m=+827.403061645:
 HeartBeating....2020-08-31 09:17:43.152571 -0700 PDT m=+837.433592338: 
has heartbeat false 2020-08-31 09:17:43.152591 -0700 PDT m=+837.433612134:
 HeartBeating....2020-08-31 09:17:53.194353 -0700 PDT m=+847.475257166: 
has heartbeat false 2020-08-31 09:17:53.194442 -0700 PDT m=+847.475346429:
 HeartBeating....2020-08-31 09:18:03.220605 -0700 PDT m=+857.501393207: 
has heartbeat false 2020-08-31 09:18:03.220632 -0700 PDT m=+857.501419823:
 HeartBeating....2020-08-31 09:18:13.25471 -0700 PDT m=+867.535380609: 
has heartbeat false 2020-08-31 09:18:13.254729 -0700 PDT m=+867.535399783:
 HeartBeating....2020-08-31 09:18:23.268893 -0700 PDT m=+877.549447691: 
has heartbeat false 2020-08-31 09:18:23.268913 -0700 PDT m=+877.549467299:
 HeartBeating....2020-08-31 09:18:33.300597 -0700 PDT m=+887.581034844: 
has heartbeat false 2020-08-31 09:18:33.300617 -0700 PDT m=+887.581054246:
 HeartBeating....2020-08-31 09:18:43.326131 -0700 PDT m=+897.606451542: 
has heartbeat false 2020-08-31 09:18:43.326146 -0700 PDT m=+897.606466686:
 HeartBeating....2020-08-31 09:18:53.362318 -0700 PDT m=+907.642522054: 
has heartbeat false 2020-08-31 09:18:53.362339 -0700 PDT m=+907.642542601:
 HeartBeating....2020-08-31 09:19:03.397374 -0700 PDT m=+917.677460754: 
has heartbeat false 2020-08-31 09:19:03.397392 -0700 PDT m=+917.677478617:
 HeartBeating....2020-08-31 09:19:13.416977 -0700 PDT m=+927.696946566: 
has heartbeat false 2020-08-31 09:19:13.417 -0700 PDT m=+927.696970197:
 HeartBeating....2020-08-31 09:19:23.455347 -0700 PDT m=+937.735199989: 
has heartbeat false 2020-08-31 09:19:23.455368 -0700 PDT m=+937.735220863:
 HeartBeating....2020-08-31 09:19:33.486803 -0700 PDT m=+947.766539340: 
has heartbeat false 2020-08-31 09:19:33.486841 -0700 PDT m=+947.766576958:
 HeartBeating....2020-08-31 09:19:43.52142 -0700 PDT m=+957.801039571: 
has heartbeat false 2020-08-31 09:19:43.521432 -0700 PDT m=+957.801051119:
 HeartBeating....2020-08-31 09:19:53.559561 -0700 PDT m=+967.839062874: 
has heartbeat false 2020-08-31 09:19:53.559577 -0700 PDT m=+967.839079108:
 HeartBeating....2020-08-31 09:20:03.595024 -0700 PDT m=+977.874409871: 
has heartbeat false 2020-08-31 09:20:03.595048 -0700 PDT m=+977.874433848:
 HeartBeating....2020-08-31 09:20:13.621531 -0700 PDT m=+987.900800092: 
has heartbeat false 2020-08-31 09:20:13.621559 -0700 PDT m=+987.900828070:
 HeartBeating....2020-08-31 09:20:23.654869 -0700 PDT m=+997.934020661: 
has heartbeat false 2020-08-31 09:20:23.654892 -0700 PDT m=+997.934043853:
 HeartBeating....2020-08-31 09:20:33.674823 -0700 PDT m=+1007.953858126: 
has heartbeat false 2020-08-31 09:20:33.674839 -0700 PDT m=+1007.953874424:
 HeartBeating....2020-08-31 09:20:43.707499 -0700 PDT m=+1017.986416660: 
has heartbeat false 2020-08-31 09:20:43.707522 -0700 PDT m=+1017.986439655:
 HeartBeating....2020-08-31 09:20:53.734552 -0700 PDT m=+1028.013352992: 
has heartbeat false 2020-08-31 09:20:53.734591 -0700 PDT m=+1028.013392152:
 HeartBeating....2020-08-31 09:21:03.766769 -0700 PDT m=+1038.045453381: 
has heartbeat false 2020-08-31 09:21:03.766789 -0700 PDT m=+1038.045473299:
 HeartBeating....2020-08-31 09:21:13.799051 -0700 PDT m=+1048.077618532: 
has heartbeat false 2020-08-31 09:21:13.799071 -0700 PDT m=+1048.077638561:
 HeartBeating....2020-08-31 09:21:23.822737 -0700 PDT m=+1058.101188048: 
has heartbeat false 2020-08-31 09:21:23.822775 -0700 PDT m=+1058.101226055:
 HeartBeating....2020-08-31 09:21:33.855441 -0700 PDT m=+1068.133774958: 
has heartbeat false 2020-08-31 09:21:33.855465 -0700 PDT m=+1068.133799105:
 HeartBeating....2020-08-31 09:21:43.875237 -0700 PDT m=+1078.153454580: 
has heartbeat false 2020-08-31 09:21:43.875256 -0700 PDT m=+1078.153473070:
 HeartBeating....2020-08-31 09:21:53.899645 -0700 PDT m=+1088.177745185: 
has heartbeat false 2020-08-31 09:21:53.899684 -0700 PDT m=+1088.177784289:
 HeartBeating....2020-08-31 09:22:03.923676 -0700 PDT m=+1098.201659725: 
has heartbeat false 2020-08-31 09:22:03.923704 -0700 PDT m=+1098.201687670:
 HeartBeating....2020-08-31 09:22:13.951129 -0700 PDT m=+1108.228995983: 
has heartbeat false 2020-08-31 09:22:13.95116 -0700 PDT m=+1108.229026483:
 HeartBeating....2020-08-31 09:22:23.977084 -0700 PDT m=+1118.254834145: 
has heartbeat false 2020-08-31 09:22:23.977103 -0700 PDT m=+1118.254853317:
 HeartBeating....2020-08-31 09:22:34.014035 -0700 PDT m=+1128.289369578: 
has heartbeat false 2020-08-31 09:22:34.014057 -0700 PDT m=+1128.289391623:
 HeartBeating....2020-08-31 09:22:44.045453 -0700 PDT m=+1138.316961677: 
has heartbeat false 2020-08-31 09:22:44.045464 -0700 PDT m=+1138.316973148:
 HeartBeating....2020-08-31 09:22:54.080239 -0700 PDT m=+1148.349662938: 
has heartbeat false 2020-08-31 09:22:54.080292 -0700 PDT m=+1148.349716746:
 HeartBeating....2020-08-31 09:23:04.118149 -0700 PDT m=+1158.386400841: 
has heartbeat false 2020-08-31 09:23:04.118168 -0700 PDT m=+1158.386420515:
 HeartBeating....2020-08-31 09:23:14.14984 -0700 PDT m=+1168.417398599: 
has heartbeat false 2020-08-31 09:23:14.149858 -0700 PDT m=+1168.417416631:
 HeartBeating....2020-08-31 09:23:24.183322 -0700 PDT m=+1178.450436804: 
has heartbeat false 2020-08-31 09:23:24.183345 -0700 PDT m=+1178.450459561:
 HeartBeating....2020-08-31 09:23:34.214368 -0700 PDT m=+1188.481169524: 
has heartbeat false 2020-08-31 09:23:34.21442 -0700 PDT m=+1188.481221448:
 HeartBeating....2020-08-31 09:23:44.244986 -0700 PDT m=+1198.511543663: 
has heartbeat false 2020-08-31 09:23:44.245021 -0700 PDT m=+1198.511578421:
 HeartBeating....2020-08-31 09:23:54.267365 -0700 PDT m=+1208.533713675: 
has heartbeat false 2020-08-31 09:23:54.267426 -0700 PDT m=+1208.533774520:
 HeartBeating....2020-08-31 09:24:04.294486 -0700 PDT m=+1218.560645372: 
has heartbeat false 2020-08-31 09:24:04.294523 -0700 PDT m=+1218.560682694:
 HeartBeating....2020-08-31 09:24:14.316295 -0700 PDT m=+1228.582274857: 
has heartbeat false 2020-08-31 09:24:14.316331 -0700 PDT m=+1228.582310457:
 HeartBeating....2020-08-31 09:24:24.334192 -0700 PDT m=+1238.599996254: 
has heartbeat false 2020-08-31 09:24:24.334217 -0700 PDT m=+1238.600021296:
 HeartBeating....2020-08-31 09:24:34.351965 -0700 PDT m=+1248.617598079: 
has heartbeat false 2020-08-31 09:24:34.352004 -0700 PDT m=+1248.617637220:
 HeartBeating....2020-08-31 09:24:44.373699 -0700 PDT m=+1258.639160764: 
has heartbeat false 2020-08-31 09:24:44.373734 -0700 PDT m=+1258.639195922:
 HeartBeating....2020-08-31 09:24:54.403985 -0700 PDT m=+1268.669276909: 
has heartbeat false 2020-08-31 09:24:54.403998 -0700 PDT m=+1268.669290546:
 HeartBeating....2020-08-31 09:25:04.434566 -0700 PDT m=+1278.699688425: 
has heartbeat false 2020-08-31 09:25:04.434602 -0700 PDT m=+1278.699724091:
 HeartBeating....2020-08-31 09:25:14.452512 -0700 PDT m=+1288.717465239: 
has heartbeat false 2020-08-31 09:25:14.452547 -0700 PDT m=+1288.717499888:
 HeartBeating....2020-08-31 09:25:24.484488 -0700 PDT m=+1298.749272209: 
has heartbeat false 2020-08-31 09:25:24.484501 -0700 PDT m=+1298.749284697:
 HeartBeating....2020-08-31 09:25:34.51307 -0700 PDT m=+1308.777684295: 
has heartbeat false 2020-08-31 09:25:34.51309 -0700 PDT m=+1308.777704211:
 HeartBeating....2020-08-31 09:25:44.541851 -0700 PDT m=+1318.806296543: 
has heartbeat false 2020-08-31 09:25:44.541864 -0700 PDT m=+1318.806309927:
 HeartBeating....2020-08-31 09:25:54.573546 -0700 PDT m=+1328.837822464: 
has heartbeat false 2020-08-31 09:25:54.573575 -0700 PDT m=+1328.837851358:
 HeartBeating....2020-08-31 09:26:04.596713 -0700 PDT m=+1338.860820374: 
has heartbeat false 2020-08-31 09:26:04.596734 -0700 PDT m=+1338.860840816:
 HeartBeating....2020-08-31 09:26:14.629142 -0700 PDT m=+1348.893079490: 
has heartbeat false 2020-08-31 09:26:14.629164 -0700 PDT m=+1348.893102245:
 HeartBeating....2020-08-31 09:26:24.656974 -0700 PDT m=+1358.920742793: 
has heartbeat false 2020-08-31 09:26:24.65699 -0700 PDT m=+1358.920758658:
 HeartBeating....2020-08-31 09:26:34.678023 -0700 PDT m=+1368.941622293: 
has heartbeat false 2020-08-31 09:26:34.678068 -0700 PDT m=+1368.941668043:
 HeartBeating....2020-08-31 09:26:44.710886 -0700 PDT m=+1378.974316774: 
has heartbeat false 2020-08-31 09:26:44.710907 -0700 PDT m=+1378.974337182:
 HeartBeating....2020-08-31 09:26:54.7397 -0700 PDT m=+1389.002960858: 
has heartbeat false 2020-08-31 09:26:54.739726 -0700 PDT m=+1389.002987410:
 HeartBeating....2020-08-31 09:27:04.768235 -0700 PDT m=+1399.031326763: 
has heartbeat false 2020-08-31 09:27:04.768247 -0700 PDT m=+1399.031338807:
 HeartBeating....2020-08-31 09:27:14.797738 -0700 PDT m=+1409.060660989: 
has heartbeat false 2020-08-31 09:27:14.797758 -0700 PDT m=+1409.060681336:
 HeartBeating....2020-08-31 09:27:24.821039 -0700 PDT m=+1419.083792381: 
has heartbeat false 2020-08-31 09:27:24.821068 -0700 PDT m=+1419.083821714:
 HeartBeating....2020-08-31 09:27:34.852987 -0700 PDT m=+1429.115571387: 
has heartbeat false 2020-08-31 09:27:34.853023 -0700 PDT m=+1429.115607836:
 HeartBeating....2020-08-31 09:27:44.881344 -0700 PDT m=+1439.143759753: 
has heartbeat false 2020-08-31 09:27:44.881372 -0700 PDT m=+1439.143787118:
 HeartBeating....2020-08-31 09:27:54.916233 -0700 PDT m=+1449.178478784: 
has heartbeat false 2020-08-31 09:27:54.916261 -0700 PDT m=+1449.178507098:
 HeartBeating....2020-08-31 09:28:04.939586 -0700 PDT m=+1459.201663371: 
has heartbeat false 2020-08-31 09:28:04.93962 -0700 PDT m=+1459.201696857:
 HeartBeating....2020-08-31 09:28:14.973662 -0700 PDT m=+1469.235569549: 
has heartbeat false 2020-08-31 09:28:14.973676 -0700 PDT m=+1469.235583841:
 HeartBeating....2020-08-31 09:28:25.006559 -0700 PDT m=+1479.268297144: 
has heartbeat false 2020-08-31 09:28:25.00658 -0700 PDT m=+1479.268318087:

thank you for the reply @maxim

I am using

this YAML file to deploy the temporal services,if these versions can be the issue. I hope this is stable, unlike Postgres DB ?.

my entire program is a simple test go file: exactly look like

func TestWorkFlow(t *testing.T) {

// The client is a heavyweight object that should be created once per process.
wfClient := ctemp.Client(config.Temporal{HostPort: "localhost:7233"}, clog.NewZapLogger(true, "test", "debug"))

w := worker.New(wfClient, "cancel-activity", worker.Options{})

w.RegisterWorkflow(Workflow1)
w.RegisterActivity(&Activities{test: 8})

err := w.Start()
if err != nil {
	log.Fatalln("Unable to start worker", err)
}

workflowOptions1 := client.StartWorkflowOptions{
	ID:                    "workflowID12313d",
	TaskQueue:             "cancel-activity",
	RetryPolicy:           &temporal.RetryPolicy{InitialInterval: time.Second},
	WorkflowIDReusePolicy: enums.WORKFLOW_ID_REUSE_POLICY_ALLOW_DUPLICATE_FAILED_ONLY,
}

we, err := wfClient.ExecuteWorkflow(context.Background(), workflowOptions1, Workflow1)
if err != nil {
	log.Fatalln("Unable to execute workflow", err)
}
log.Println("Started workflow", "WorkflowID", we.GetID(), "RunID", we.GetRunID())
ctx := context.Background()
var valuePtr error
err = we.Get(ctx, &valuePtr)
if err != nil {
	log.Fatalln("Unable to execute workflow", err)
}

}



func Workflow1(ctx workflow.Context) error {
ao := workflow.ActivityOptions{
	ScheduleToStartTimeout: time.Minute,
	StartToCloseTimeout:    time.Minute * 30,
	HeartbeatTimeout:       30 * time.Second,
	WaitForCancellation:    true,
	RetryPolicy:            &temporal.RetryPolicy{InitialInterval: 1 * time.Second},
}
ctx = workflow.WithActivityOptions(ctx, ao)
logger := workflow.GetLogger(ctx)
logger.Info("cancel workflow started")
defer func() {
	// When workflow is canceled, it has to get a new disconnected context to execute any activities
	newCtx, _ := workflow.NewDisconnectedContext(ctx)
	err := workflow.ExecuteActivity(newCtx, new(Activities).CleanupActivity).Get(ctx, nil)
	if err != nil {
		logger.Error("Cleanup activity failed", "Error", err)
	}
}()

var result string
activi := workflow.ExecuteActivity(ctx, new(Activities).ActivityToBeCanceled)

err := activi.Get(ctx, &result)
if err != nil {
	logger.Info("Workflow error.")
}

logger.Info("Workflow completed.")

return nil
}

type Activities struct {
test int
}

func (a *Activities) ActivityToBeCanceled(ctx context.Context) (string, error) {
logger := activity.GetLogger(ctx)
logger.Info("activity started, to cancel workflow, use 'go run cancelactivity/cancel/main.go " +
	"-w <WorkflowID>' or CLI: 'tctl wf cancel -w <WorkflowID>' to cancel")
a.test++
heartBeatTime := time.Time{}
for {
	select {
	case <-time.After(1 * time.Second):
		seq := fmt.Sprintf("%v", time.Now())
		activity.RecordHeartbeat(ctx, seq)
		if time.Now().After(heartBeatTime.Add(10 * time.Second)) {
			fmt.Printf("HeartBeating....:         %v \n", time.Now())
			heartBeatTime = time.Now()
			if activity.HasHeartbeatDetails(ctx) {
				var prev string
				if err := activity.GetHeartbeatDetails(ctx, &prev); err != nil {
					clog.Error(ctx, "GetHeartbeatDetails fail", err)
				}
				fmt.Printf("value of heart beat det:  %v \n", prev)

			} else {
				fmt.Printf("has heartbeat false       %v:\n", time.Now())
			}
		}
	case <-ctx.Done():
		logger.Info("context is cancelled")
		return "I am canceled by Done", nil
	}
}
}

func (a *Activities) CleanupActivity(ctx context.Context) error {
logger := activity.GetLogger(ctx)
logger.Info("cleanupActivity started")
return nil
}

and what type of connection issue it can have, it is communicating because activities are getting scheduled

Regards

and please let me know which setup you are using for temporal service deployment ?. so that i can mimic.

Thank you,

I ran initially with docker-compose.yml, but now I repeated with docker-compose-mysql.yml and it worked fine again. I deploy locally on my mac laptop running docker-compose up.

@maxim thanks for trying

i will try to find out the issue , and will update here