Temporal activity timeout -- where are the logs?

Community Support
1

in the temporal UI, I can see a WorkflowExecutionFailed event, but I can’t seem to find the same in the teamporal-server logs – what am I missing?

In the UI:

{
  "message": "Activity task timed out",
  "source": "",
  "stackTrace": "",
  "encodedAttributes": null,
  "cause": {
    "message": "activity StartToClose timeout",
    "source": "Server",
    "stackTrace": "",
    "encodedAttributes": null,
    "cause": {
      "message": "activity StartToClose timeout",
      "source": "Server",
      "stackTrace": "",
      "encodedAttributes": null,
      "cause": null,
      "timeoutFailureInfo": {
        "timeoutType": "StartToClose",
        "lastHeartbeatDetails": null
      }
    },
    "timeoutFailureInfo": {
      "timeoutType": "StartToClose",
      "lastHeartbeatDetails": null
    }
  },
  "activityFailureInfo": {
    "scheduledEventId": "5",
    "startedEventId": "6",
    "identity": "",
    "activityType": {
      "name": "myActivityName"
    },
    "activityId": "1",
    "retryState": "MaximumAttemptsReached"
  }
}
2

The server doesn’t log per activity errors as it would be too noisy. I believe they are logged by the SDKs.

3

i’m confused… how would the typescript SDK be able to log when a workflow times out and is killed by the server? especially if it’s hung or a worker crashed, etc

4

What is the problem you are trying to solve? When a workflow fails or times out, this is reflected in the workflow execution history. Why do you need this logged?

5

I’d like to write a datadog alert to know when specific workflows fail or timeout.

For context, we have workflows that run as importers for any number of tenants. Once a tenant defines them, they run on cron schedules in temporal. So one they start running, if they time out or fail, it’s always an error we should know about.

We recently ran into an issue where one tenant’s import workflow failed to finish within the startToClose timeout, and it would just hang when temporal subsequently retried, including when the cron started a fresh workflow.

Either way, we’re trying to make sure that we can alert our oncall engineer if an import workflow fails or times out.

6

Workflow timeouts should not be used for business logic. This is a protection mechanism to avoid runaway resources. So in your case I would recommend changing workflow timeout to a timer that fails workflow. Then you can change your workflow to either emit metrics or notify you any other way on failure. If you want a more generic solution put this logic into an interceptor.

7

I’m a bit confused… I just want to know when a workflow fails or times out, and what the workflow name was. I don’t want to trigger any business logic when that occurs. :slight_smile:

I’m simply following this: TypeScript SDK developer's guide - Features | Temporal Documentation

8

Another question, it looks like interceptors run on the worker, is that correct? I was hoping for a way to detect timeout/failure even if the worker crashes or workers become unavailable, but if that’s not possible I’ll just use interceptors.

Do you have any pointers towards which call I would intercept to determine when a workflow has timed out?

9

In this case metrics for timed out and terminated workflows are already reported by the Temporal service (or cloud), so you can import them to Datadog and set any alerts you want.

10

I’m able to see those metrics in datadog, but they only specify taskqueue, not workflow name :frowning:

11

@maxim thank you for all of the help so far – is there any other way to get failure by workflow name from the logs/metrics? if not, is there any chance your team would be willing to add workflow name as a dimension of the metrics?

12

The reason some metrics don’t contain some dimension is due to metric backends not being able to support dimensions with a large number of values.

13

So, if you really need to act when a specific workflow fails, then using try-catch or an interceptor is the way to go. Don’t use workflow timeout, but use a timer within the workflow code to fail it after the specified interval.