High CPU Usage on long running workflows

Community Support
,
1

I have thousands of infinite running golang workflows, which only execute one activity. A subscription to a NATS topic and a followed by a

select {}

However when running those thousands of workflows, the cpu usage of the temporal and database container are drastically increasing, like about 30-40% on 12000 concurrently running events. The worker itself almost takes up no cpu usage, only about 5%

I have set custom policies with heartbeat timeout of 20 seconds and sending a heartbeat every 10 seconds, which could cause some of the CPU usage. However when removing that part, CPU usage is still pretty high for the temporal container (not even the worker) and increases with increasing running workflow count.

What could cause the high CPU usage on that container although the workflows are all long running which are kind of stopped via a select at the end of the activity? Is this normal behavior and is there a way to decrease it?

2

12000 heartbeating every 10 seconds is 1200 heartbeats per second. If you remove heartbeats you are getting activity timeout and its retry every 20 seconds. For a small deployment this can generate reasonable load as every heartbeat results in a DB update. And if the history service cache is small might cause its churn.