No_poller_tasks despite num_pollers showing that pollers exist on a task queue

Community Support
1

I’ve come across a weird scenario where after doing a rolling restart of my Temporal cluster where it seems like there are no_poller_tasks on a task queue partition despite there being associated pollers which can be seen in the num_pollers metric. This number doesn’t seem to go up or down.

The interesting part is that no_poller_tasks shows that all tasks are in a single partition of task_type==Workflow. On the other hand we can see that num_pollers has poller_task==workflow_task which I feel like should handle this partition. Is this a misconfiguration issue or some issue with how Temporal is handling the task queues?

The no_poller looks like:

temporal_no_poller_tasks{
  namespace="hibernate",
  operation="TaskQueuePartitionManager",
  partition="0",
  service_name="matching",
  task_type="Workflow",
  taskqueue="HIBERNATE_TIMER_aws_us_west_2_4"
} == 1561

and the num_pollers looks like:

temporal_num_pollers{
  namespace="hibernate",
  activity_type="none",
  client_name="temporal_go",
  poller_type="activity_task",
  task_queue="HIBERNATE_TIMER_aws_us_west_2_4",
  worker_type="none",
  workflow_type="none"
} == 2

temporal_num_pollers{
  namespace="hibernate",
  activity_type="none",
  client_name="temporal_go",
  poller_type="workflow_sticky_task",
  task_queue="HIBERNATE_TIMER_aws_us_west_2_4",
  worker_type="none",
  workflow_type="none"
} == 1

temporal_num_pollers{
  namespace="hibernate",
  activity_type="none",
  client_name="temporal_go",
  poller_type="workflow_task",
  task_queue="HIBERNATE_TIMER_aws_us_west_2_4",
  worker_type="none",
  workflow_type="none"
} == 1

temporal_num_pollers{
  namespace="hibernate",
  activity_type="none",
  client_name="temporal_go",
  poller_type="activity_task",
  task_queue="HIBERNATE_TIMER_aws_us_west_2_4",
  worker_type="none",
  workflow_type="none"
} == 2

temporal_num_pollers{
  namespace="hibernate",
  activity_type="none",
  client_name="temporal_go",
  poller_type="workflow_sticky_task",
  task_queue="HIBERNATE_TIMER_aws_us_west_2_4",
  worker_type="none",
  workflow_type="none"
} == 1

temporal_num_pollers{
  namespace="hibernate",
  activity_type="none",
  client_name="temporal_go",
  poller_type="workflow_task",
  task_queue="HIBERNATE_TIMER_aws_us_west_2_4",
  worker_type="none",
  workflow_type="none"
} == 1
2

Sorry not sure I fully understand question. Server metric no_poller_tasks checks pollers only on the root task queue partition (so regular task queue which has default 4 partitions, and not the worker-specific or “sticky” task queue for workflow tasks). no_poller_tasks is also like a “sliding window” type of metric, where its incremented within a 2 minute default window. Could be that during restarts there was no poller on root partition for that long.