Workflow Signals not working with GKE

Community Support
,
1

Dears,

Thanks for this amazing piece of technology,

I’ve a workflow which at some point in future gets a signal tostart processing another activity, and its tested on Local using the docker-compose and working fine, however when I deployed Temporal on google GKE everything works good except for the signals.

when I send a signal from the client side, it never have any effect on the workflow, and the workflow is still being on Running and stuck at the WorkflowTaskCompleted step.

however when I change the connect address from the production cluster address into the localhost, everything works like a charm

Client Code

async function run() {
  const workflowId = "id-3095";
    const client = new WorkflowClient({
      address: "FRONTEND-CLUSTER-ADDRESS:7233",
      // address: "localhost:7233",
    });
    const handle =  client.getHandle(workflowId);
    console.log("Sending Signal: ", workflowId);
    await handle.signal(cnnSignal);
    console.log("Signal Sent: ");
}

Thanks

2

however when I change the connect address from the production cluster address into the localhost, everything works like a charm

Do you have any more info, any client errors, timeouts?
Can you health check your frontend service using tctl / cli, for example:

tctl --address FRONTEND-CLUSTER-ADDRESS:7233 cl h

Can you check if the execution you are signaling to exists on this cluster / was started?

tctl --address FRONTEND-CLUSTER-ADDRESS:7233 wf desc -w <workflowid>

On this cluster do you set up “default” namespace or are you running executions on a different ns?

3

Hi @tihomir thanks for your reply

I’ll try to add some more info here

My workflow:

export async function onxWorkflow(caseInfo) {
console.log("Pickle Activity Started");
  await qusActivity(caseInfo);
  await sleep(5000);
  console.log("Pickle Activity Completed");

  const caseId = caseInfo.number;
  let isCnnTriggered = false;
  setHandler(cnnSignal, () => {
    console.log(`caseId: ${caseId} 1 ====> CNN Trigger Received`);
    isCnnTriggered = true;
  });

  try {
    await wf.condition(() => isCnnTriggered);
    await cnnActivity(caseInfo);
    console.log(`caseId: ${caseId} 4 ====> CNN Activity Completed`);
  } catch (err) {
    if (err instanceof wf.CancelledFailure) {
      console.log("Cancelled");
    }
    throw err;
  }
}

My Client Code:

async function run() {
  const workflowId = "id-3095";
    const client = new WorkflowClient({
      address: "FRONTEND-CLUSTER-ADDRESS:7233",
      // address: "localhost:7233",
    });
    const handle =  client.getHandle(workflowId);
    console.log("Sending Signal: ", workflowId);
    await handle.signal(cnnSignal);
    console.log("Signal Sent: ");
}

When I ran it on local cluster, everything works as expected, but when I change into the dev cluster on GKE using the helm charts I get the workflow to be working but without the signal being triggered at all, the first activity works, the sleep works as well, but it never rans the signal part.

I believe maybe the issue has to do with smth with the Client Signal code the const client = new WorkflowClient() as it automatically connects to the local cluster, but I don’t know the way to make it connected to the dev cluster
even when doing this

 const client = new WorkflowClient({
      address: "FRONTEND--DEV-CLUSTER-ADDRESS:7233",
      // address: "localhost:7233",
    });

I believe I maybe have smth wrong on how the Client Signal, that its not connected to the dev cluster correctly as theres no response from the signal client, even if I wrote a wrong address I still get these line

to be logged out
console.log("Sending Signal: ", workflowId);
but never get that line to be logged out
console.log("Signal Sent: ");

Wether running that on Dev Cluster or the Localhost Cluster

this outputs

temporal.api.workflowservice.v1.WorkflowService: SERVING

Output:

{
  "executionConfig": {
    "taskQueue": {
      "name": "onx",
      "kind": "Normal"
    },
    "defaultWorkflowTaskTimeout": "10s"
  },
  "workflowExecutionInfo": {
    "execution": {
      "workflowId": "Onx-8020",
      "runId": "57e16d35-ea9c-41cb-bb13-7860e0be5972"
    },
    "type": {
      "name": "onxWorkflow"
    },
    "startTime": "2023-04-10T09:08:16.246795187Z",
    "status": "Running",
    "historyLength": "9",
    "memo": {

    },
    "searchAttributes": {
      "indexedFields": {
        "BinaryChecksums": "[\"@temporalio/worker@1.5.2+7b9fc73820688f4bf0481a087861f461237c9561a0fa4ca8a5cc72adacadc6b8\"]"
      }
    },
    "autoResetPoints": {
      "points": [
        {
          "binaryChecksum": "@temporalio/worker@1.5.2+7b9fc73820688f4bf0481a087861f461237c9561a0fa4ca8a5cc72adacadc6b8",
          "runId": "57e16d35-ea9c-41cb-bb13-7860e0be5972",
          "firstWorkflowTaskCompletedId": "4",
          "createTime": "2023-04-10T09:08:16.467499138Z",
          "resettable": true
        }
      ]
    },
    "stateTransitionCount": "6"
  }
}

I’m running it on the default namespace

Please let me know if you need any more info and thanks a lot for your help

Also please find below a screenshot from the web-ui

Workflow
Workflow1920×1688 189 KB

Thanks

4

Can you try to send a dummy signal to this execution via tctl and see if it ends up in the execution event history? For example:

tctl --address <address> wf signal -w <wfid> -n <some random signal name> --reason <test reason>

After check event history via tctl as well:

tctl --address <address> wf show -w <wfid> -r <runid> --of myhistory.json

5

Output: Signal workflow succeeded.

Output:

  1  WorkflowExecutionStarted   {WorkflowType:{Name:onxWorkflow}, ParentInitiatedEventId:0,
                                 TaskQueue:{Name:onx, Kind:Normal},
                                 Input:[{"number":3095,"caseId":"temporal_dev_1","module_name":"module_1.tar"}],
                                 WorkflowTaskTimeout:10s, Initiator:Unspecified,
                                 OriginalExecutionRunId:ab5f4536-8019-449b-8dea-5179edbabde7,
                                 Identity:59199@Abds-Laptop.local,
                                 FirstExecutionRunId:ab5f4536-8019-449b-8dea-5179edbabde7, Attempt:1,
                                 FirstWorkflowTaskBackoff:0s, ParentInitiatedEventVersion:0}
   2  WorkflowTaskScheduled      {TaskQueue:{Name:onx,
                                 Kind:Normal},
                                 StartToCloseTimeout:10s,
                                 Attempt:1}
   3  WorkflowTaskStarted        {ScheduledEventId:2,
                                 Identity:59185@Abds-Laptop.local,
                                 RequestId:bd40cacf-6db7-4765-8723-e3ea43f559a8,
                                 SuggestContinueAsNew:false,
                                 HistorySizeBytes:308}
   4  WorkflowTaskCompleted      {ScheduledEventId:2, StartedEventId:3, Identity:59185@Abds-Laptop.local,
                                 BinaryChecksum:@temporalio/worker@1.5.2+49870a8a5d2bca031a0046d96df09df850bd18b98793694588560c5604294c41}
   5  ActivityTaskScheduled      {ActivityId:1, ActivityType:{Name:qusActivity}, TaskQueue:{Name:onx-qus, Kind:Normal},
                                 Input:[{"number":3095,"caseId":"temporal_dev_1","module_name":"module_1.tar"}],
                                 ScheduleToCloseTimeout:0s, ScheduleToStartTimeout:0s, StartToCloseTimeout:24h0m0s,
                                 HeartbeatTimeout:0s, WorkflowTaskCompletedEventId:4, RetryPolicy:{InitialInterval:1s,
                                 BackoffCoefficient:2, MaximumInterval:1m40s, MaximumAttempts:0,
                                 NonRetryableErrorTypes:[]}}
   6  ActivityTaskStarted        {ScheduledEventId:5,
                                 Identity:48499@Abds-Laptop.local,
                                 RequestId:f4fe6b1d-c7a1-4254-9017-15b753e693d3,
                                 Attempt:1}
   7  ActivityTaskCompleted      {Result:[true],
                                 ScheduledEventId:5,
                                 StartedEventId:6,
                                 Identity:48499@Abds-Laptop.local}
   8  WorkflowTaskScheduled      {TaskQueue:{Name:59185@Abds-Laptop.local-onx-bba908f6f2bf43c5a57fcb3a414571df,
                                 Kind:Sticky}, StartToCloseTimeout:10s, Attempt:1}
   9  WorkflowTaskStarted        {ScheduledEventId:8,
                                 Identity:59185@Abds-Laptop.local,
                                 RequestId:3c69fb82-aa0c-47eb-8153-0026157b231c,
                                 SuggestContinueAsNew:false,
                                 HistorySizeBytes:1025}
  10  WorkflowTaskCompleted      {ScheduledEventId:8, StartedEventId:9, Identity:59185@Abds-Laptop.local,
                                 BinaryChecksum:@temporalio/worker@1.5.2+49870a8a5d2bca031a0046d96df09df850bd18b98793694588560c5604294c41}
  11  TimerStarted               {TimerId:1,
                                 StartToFireTimeout:5s,
                                 WorkflowTaskCompletedEventId:10}
  12  TimerFired                 {TimerId:1, StartedEventId:11}
  13  WorkflowTaskScheduled      {TaskQueue:{Name:59185@Abds-Laptop.local-onx-bba908f6f2bf43c5a57fcb3a414571df,
                                 Kind:Sticky}, StartToCloseTimeout:10s, Attempt:1}
  14  WorkflowTaskStarted        {ScheduledEventId:13,
                                 Identity:59185@Abds-Laptop.local,
                                 RequestId:46e1c5cf-154d-4f4c-9fc7-a54c363f45dd,
                                 SuggestContinueAsNew:false,
                                 HistorySizeBytes:1428}
  15  WorkflowTaskCompleted      {ScheduledEventId:13, StartedEventId:14, Identity:59185@Abds-Laptop.local,
                                 BinaryChecksum:@temporalio/worker@1.5.2+49870a8a5d2bca031a0046d96df09df850bd18b98793694588560c5604294c41}
  16  WorkflowExecutionSignaled  {SignalName:cnnSignal,
                                 Identity:tctl@Abds-Laptop.local}
  17  WorkflowTaskScheduled      {TaskQueue:{Name:59185@Abds-Laptop.local-onx-bba908f6f2bf43c5a57fcb3a414571df,
                                 Kind:Sticky}, StartToCloseTimeout:10s, Attempt:1}
  18  WorkflowTaskStarted        {ScheduledEventId:17,
                                 Identity:61740@Abds-Laptop.local,
                                 RequestId:2b660d45-50a0-4a41-9682-37ef3e9be585,
                                 SuggestContinueAsNew:false,
                                 HistorySizeBytes:1828}
  19  WorkflowTaskCompleted      {ScheduledEventId:17, StartedEventId:18, Identity:61740@Abds-Laptop.local,
                                 BinaryChecksum:@temporalio/worker@1.5.2+49870a8a5d2bca031a0046d96df09df850bd18b98793694588560c5604294c41}
    ~/S/o/onx-helm    master !1 ?2  ls                                            ✔  base   07:09:48 PM 
ingress.yml    myhistory.json onx-cnn        onx-combiner   onx-pickle     onx-qus        onx-temporal
    ~/S/o/onx-helm    master !1 ?2  cat myhistory.json                            ✔  base   07:09:51 PM 
    ~/S/o/onx-helm    master !1 ?2  cat myhistory.json                            ✔  base   07:09:51 PM 
{
 "events": [
  {
   "eventId": "1",
   "eventTime": "2023-04-10T09:56:07.511162172Z",
   "eventType": "WorkflowExecutionStarted",
   "taskId": "1048576",
   "workflowExecutionStartedEventAttributes": {
    "workflowType": {
     "name": "onxWorkflow"
    },
    "taskQueue": {
     "name": "onx",
     "kind": "Normal"
    },
    "input": {
     "payloads": [
      {
       "metadata": {
        "encoding": "anNvbi9wbGFpbg=="
       },
       "data": "eyJudW1iZXIiOjMwOTUsImNhc2VJZCI6InRlbXBvcmFsX2Rldl8xIiwibW9kdWxlX25hbWUiOiJtb2R1bGVfMS50YXIifQ=="
      }
     ]
    },
    "workflowTaskTimeout": "10s",
    "originalExecutionRunId": "ab5f4536-8019-449b-8dea-5179edbabde7",
    "identity": "59199@Abds-Laptop.local",
    "firstExecutionRunId": "ab5f4536-8019-449b-8dea-5179edbabde7",
    "attempt": 1,
    "firstWorkflowTaskBackoff": "0s",
    "header": {

    }
   }
  },
  {
   "eventId": "2",
   "eventTime": "2023-04-10T09:56:07.511228227Z",
   "eventType": "WorkflowTaskScheduled",
   "taskId": "1048577",
   "workflowTaskScheduledEventAttributes": {
    "taskQueue": {
     "name": "onx",
     "kind": "Normal"
    },
    "startToCloseTimeout": "10s",
    "attempt": 1
   }
  },
  {
   "eventId": "3",
   "eventTime": "2023-04-10T09:56:07.521930644Z",
   "eventType": "WorkflowTaskStarted",
   "taskId": "1048582",
   "workflowTaskStartedEventAttributes": {
    "scheduledEventId": "2",
    "identity": "59185@Abds-Laptop.local",
    "requestId": "bd40cacf-6db7-4765-8723-e3ea43f559a8",
    "historySizeBytes": "308"
   }
  },
  {
   "eventId": "4",
   "eventTime": "2023-04-10T09:56:07.751087215Z",
   "eventType": "WorkflowTaskCompleted",
   "taskId": "1048586",
   "workflowTaskCompletedEventAttributes": {
    "scheduledEventId": "2",
    "startedEventId": "3",
    "identity": "59185@Abds-Laptop.local",
    "binaryChecksum": "@temporalio/worker@1.5.2+49870a8a5d2bca031a0046d96df09df850bd18b98793694588560c5604294c41"
   }
  },
  {
   "eventId": "5",
   "eventTime": "2023-04-10T09:56:07.751139913Z",
   "eventType": "ActivityTaskScheduled",
   "taskId": "1048587",
   "activityTaskScheduledEventAttributes": {
    "activityId": "1",
    "activityType": {
     "name": "qusActivity"
    },
    "taskQueue": {
     "name": "onx-qus",
     "kind": "Normal"
    },
    "header": {

    },
    "input": {
     "payloads": [
      {
       "metadata": {
        "encoding": "anNvbi9wbGFpbg=="
       },
       "data": "eyJudW1iZXIiOjMwOTUsImNhc2VJZCI6InRlbXBvcmFsX2Rldl8xIiwibW9kdWxlX25hbWUiOiJtb2R1bGVfMS50YXIifQ=="
      }
     ]
    },
    "scheduleToCloseTimeout": "0s",
    "scheduleToStartTimeout": "0s",
    "startToCloseTimeout": "86400s",
    "heartbeatTimeout": "0s",
    "workflowTaskCompletedEventId": "4",
    "retryPolicy": {
     "initialInterval": "1s",
     "backoffCoefficient": 2,
     "maximumInterval": "100s"
    }
   }
  },
  {
   "eventId": "6",
   "eventTime": "2023-04-10T09:56:07.761057645Z",
   "eventType": "ActivityTaskStarted",
   "taskId": "1048592",
   "activityTaskStartedEventAttributes": {
    "scheduledEventId": "5",
    "identity": "48499@Abds-Laptop.local",
    "requestId": "f4fe6b1d-c7a1-4254-9017-15b753e693d3",
    "attempt": 1
   }
  },
  {
   "eventId": "7",
   "eventTime": "2023-04-10T09:56:07.940046416Z",
   "eventType": "ActivityTaskCompleted",
   "taskId": "1048593",
   "activityTaskCompletedEventAttributes": {
    "result": {
     "payloads": [
      {
       "metadata": {
        "encoding": "anNvbi9wbGFpbg=="
       },
       "data": "dHJ1ZQ=="
      }
     ]
    },
    "scheduledEventId": "5",
    "startedEventId": "6",
    "identity": "48499@Abds-Laptop.local"
   }
  },
  {
   "eventId": "8",
   "eventTime": "2023-04-10T09:56:07.940054920Z",
   "eventType": "WorkflowTaskScheduled",
   "taskId": "1048594",
   "workflowTaskScheduledEventAttributes": {
    "taskQueue": {
     "name": "59185@Abds-Laptop.local-onx-bba908f6f2bf43c5a57fcb3a414571df",
     "kind": "Sticky"
    },
    "startToCloseTimeout": "10s",
    "attempt": 1
   }
  },
  {
   "eventId": "9",
   "eventTime": "2023-04-10T09:56:07.947443574Z",
   "eventType": "WorkflowTaskStarted",
   "taskId": "1048598",
   "workflowTaskStartedEventAttributes": {
    "scheduledEventId": "8",
    "identity": "59185@Abds-Laptop.local",
    "requestId": "3c69fb82-aa0c-47eb-8153-0026157b231c",
    "historySizeBytes": "1025"
   }
  },
  {
   "eventId": "10",
   "eventTime": "2023-04-10T09:56:08.130059343Z",
   "eventType": "WorkflowTaskCompleted",
   "taskId": "1048602",
   "workflowTaskCompletedEventAttributes": {
    "scheduledEventId": "8",
    "startedEventId": "9",
    "identity": "59185@Abds-Laptop.local",
    "binaryChecksum": "@temporalio/worker@1.5.2+49870a8a5d2bca031a0046d96df09df850bd18b98793694588560c5604294c41"
   }
  },
  {
   "eventId": "11",
   "eventTime": "2023-04-10T09:56:08.130075669Z",
   "eventType": "TimerStarted",
   "taskId": "1048603",
   "timerStartedEventAttributes": {
    "timerId": "1",
    "startToFireTimeout": "5s",
    "workflowTaskCompletedEventId": "10"
   }
  },
  {
   "eventId": "12",
   "eventTime": "2023-04-10T09:56:13.133156891Z",
   "eventType": "TimerFired",
   "taskId": "1048606",
   "timerFiredEventAttributes": {
    "timerId": "1",
    "startedEventId": "11"
   }
  },
  {
   "eventId": "13",
   "eventTime": "2023-04-10T09:56:13.133165180Z",
   "eventType": "WorkflowTaskScheduled",
   "taskId": "1048607",
   "workflowTaskScheduledEventAttributes": {
    "taskQueue": {
     "name": "59185@Abds-Laptop.local-onx-bba908f6f2bf43c5a57fcb3a414571df",
     "kind": "Sticky"
    },
    "startToCloseTimeout": "10s",
    "attempt": 1
   }
  },
  {
   "eventId": "14",
   "eventTime": "2023-04-10T09:56:13.141519074Z",
   "eventType": "WorkflowTaskStarted",
   "taskId": "1048611",
   "workflowTaskStartedEventAttributes": {
    "scheduledEventId": "13",
    "identity": "59185@Abds-Laptop.local",
    "requestId": "46e1c5cf-154d-4f4c-9fc7-a54c363f45dd",
    "historySizeBytes": "1428"
   }
  },
  {
   "eventId": "15",
   "eventTime": "2023-04-10T09:56:13.340451828Z",
   "eventType": "WorkflowTaskCompleted",
   "taskId": "1048615",
   "workflowTaskCompletedEventAttributes": {
    "scheduledEventId": "13",
    "startedEventId": "14",
    "identity": "59185@Abds-Laptop.local",
    "binaryChecksum": "@temporalio/worker@1.5.2+49870a8a5d2bca031a0046d96df09df850bd18b98793694588560c5604294c41"
   }
  },
  {
   "eventId": "16",
   "eventTime": "2023-04-10T17:08:16.720193025Z",
   "eventType": "WorkflowExecutionSignaled",
   "taskId": "1048617",
   "workflowExecutionSignaledEventAttributes": {
    "signalName": "cnnSignal",
    "identity": "tctl@Abds-Laptop.local"
   }
  },
  {
   "eventId": "17",
   "eventTime": "2023-04-10T17:08:16.720197873Z",
   "eventType": "WorkflowTaskScheduled",
   "taskId": "1048618",
   "workflowTaskScheduledEventAttributes": {
    "taskQueue": {
     "name": "59185@Abds-Laptop.local-onx-bba908f6f2bf43c5a57fcb3a414571df",
     "kind": "Sticky"
    },
    "startToCloseTimeout": "10s",
    "attempt": 1
   }
  },
  {
   "eventId": "18",
   "eventTime": "2023-04-10T17:08:16.727997898Z",
   "eventType": "WorkflowTaskStarted",
   "taskId": "1048622",
   "workflowTaskStartedEventAttributes": {
    "scheduledEventId": "17",
    "identity": "61740@Abds-Laptop.local",
    "requestId": "2b660d45-50a0-4a41-9682-37ef3e9be585",
    "historySizeBytes": "1828"
   }
  },
  {
   "eventId": "19",
   "eventTime": "2023-04-10T17:08:17.166025716Z",
   "eventType": "WorkflowTaskCompleted",
   "taskId": "1048626",
   "workflowTaskCompletedEventAttributes": {
    "scheduledEventId": "17",
    "startedEventId": "18",
    "identity": "61740@Abds-Laptop.local",
    "binaryChecksum": "@temporalio/worker@1.5.2+49870a8a5d2bca031a0046d96df09df850bd18b98793694588560c5604294c41"
   }
  }
 ]
}
6

@tihomir I’ve run the Signal from tctl with the actual signal names and and it actually works good and the signal is being triggered and processed with no problem and workflow is completed, so I believe the issue is in the JS SDK for client to send a Signal

7

For Future reference, I got it to work , the problem was with the client signal, the code on the documentation in this page only works for local setup Send Signal from Client

the correct code for production should be

const workflowId = "workflow-Id";
  const connection = await Connection.connect({
    address: "Production-address",
  });
  const client = new Client({
    connection,
  });
  const handle = client.workflow.getHandle(workflowId);
  console.log("Sending Signal: ", workflowId);
  await handle.signal(cnnSignal);
  console.log("Signal Sent: ");
1 Like