We are using the Temporal SDK in one of our microservices deployed on an ECS container with a Linux OS. During load testing, we observed that the memory utilization (heap memory) gradually increased up to 50%, and it did not decrease until the server was restarted or stopped. Further investigation revealed that running the same codebase on a local system with Windows OS exhibited no issues. The heap memory was released promptly after 10 to 15 minutes. However, this behavior was not replicated when running the application on AWS environment.
However, after passing the maxCachedWorkflows option as zero while creating the worker, the workflows are no longer being cached by the worker, and this has resolved the memory utilization issue (but at the cost of high CPU utilization). As a result, the heap memory is now getting released properly, and the problem is resolved temporarily as of now.
Please let us know if we need to do any specific configuration for container application to resolve above issue without passing maxCachedWorkflows option to worker.
await Worker.create({
workflowsPath: require.resolve(‘./workflow’),
connection,
activities: createActivities(),
taskQueue: ‘test-’+process.env.stage,
namespace: ‘test’,
sinks: defaultSinks(workflowLogger),
maxCachedWorkflows: 0
})
Below is the configuration we are using for our own application.
- OS and processor: [Linux]
- SDK version:
“@temporalio/activity”: “^1.7.4”,
“@temporalio/client”: “^1.7.4”,
“@temporalio/worker”: “^1.7.4”,
“@temporalio/workflow”: “^1.7.4” - We have deployed our containerized application on AWS ECS with Node image 16.4.1
FYI, we tried with latest versions (1.8.2) of above libraries but no luck.