Long-Running workflow, memory exhaustion, and parameter passing

Community Support
1

I’m quite new with Go and really new to Temporal. I tried to search on Google and here to find answers, but I did not find them. I already saw this video youtube(.)com/watch?v=qce_AqCkFys (sorry, cannot post more than 2 links) and also check the https://stackoverflow.com/questions/61157400/temporal-workflow-vs-cadence-workflow answer from Maxim. But I’m still not sure about a few things.

Long running workflow stays in memory

I was playing a bit with Go Examples, and I used this one https://github.com/temporalio/samples-go/blob/master/timer/workflow.go with little modification, the code below is added at the end of SampleTimerWorkflow.

But when I executed it, the workflow was just sleeping. Wasn’t terminated. In the mentioned video Maxim said that you can run 1M workflows and it will not exhaust memory. But if workflow is just sleeping, it is consuming memory.
So either I understand it badly, or when there will be too many running workflows, it will kill some of them, and restore later. Or is there something hidden I’m missing?

workflow.Sleep(ctx, time.Hour*24*30)
_ = workflow.ExecuteActivity(ctx, SendEmailActivity2).Get(ctx, nil) 
workflow.GetLogger(ctx).Info("Workflow completed.")
return nil

Which arguments are safe to pass into the workflow or activity?

When I watched the video mentioned above, I know parameters are not passed directly to the workflow or activity. They are always stored in DB, because a worker can take the entity later. So they must be first serialized into some storage and then deserialized.

So it is safe to pass any kind of argument? Like gRPC connection (more about that below), or DB connection, etc. What about passing structures with not exported attributes. Are those OK as well?
And what about Value Context with some values? Are those also OK?

Passing connections into Activity

We have an Activity called SendVerificationEmailActivity which is triggered by microservice A. It needs 2 gRPC connections into microservices B and C. What is the best way how to pass those connections into activity? I see 2 possible problems here. The first is mainly answered by the previous question.

But if we pass connections from main into the workflow, and later to activity. Configuration can be changed later, and even after restart, the configuration is stored in DB. So it can be outdated. It really does not matter what kind of connection we are passing, but in this case it is 2x gRPC connection and configuration for mailer system. Email configuration is now passed via the Context.

What is the best solution to this problem?

Thank you

2

So either I understand it badly, or when there will be too many running workflows, it will kill some of them, and restore later. Or is there something hidden I’m missing?

Workflows are not immediately removed from the worker’s memory. For efficiency, they are stored in an LRU cache. So as soon as the cache is full they will be removed from memory. The size of the pool is controlled by the worker.SetStickyWorkflowCacheSize global setting. You can restart the worker at any time to see that workflows are not loaded on restart. Then go to UI and open the stack trace tab (while the worker is running) to see the stack trace of the blocked workflow.

So it is safe to pass any kind of argument? Like gRPC connection (more about that below), or DB connection, etc. What about passing structures with not exported attributes. Are those OK as well?
And what about Value Context with some values? Are those also OK?

No, it is not safe to pass such objects. The objects that are passed should be value types that can be converted to a binary payload. The conversion is pluggable with JSON/Protobuf supported out of the box. Here is the default DataConverter.

Passing connections into Activity

They are passed as arguments to the activity struct during its initialization. See the greetings sample.

3

Thank you for the tips! Now the whole system makes more sense to me.

1 Like