Error, message length too large

My typescript workflow/activity worker runs into this error:

Error: Unexpected server response: 502
    at ClientRequest.<anonymous> (/home/ubuntu/stakexyz/stakexyz-staking/node_modules/ws/lib/websocket.js:912:7)
    at ClientRequest.emit (node:events:517:28)
    at ClientRequest.emit (node:domain:489:12)
    at HTTPParser.parserOnIncomingClient (node:_http_client:700:27)
    at HTTPParser.parserOnHeadersComplete (node:_http_common:119:17)
    at TLSSocket.socketOnData (node:_http_client:541:22)
    at TLSSocket.emit (node:events:517:28)
    at TLSSocket.emit (node:domain:489:12)
    at addChunk (node:internal/streams/readable:368:12)
    at readableAddChunk (node:internal/streams/readable:341:9)
    at TLSSocket.Readable.push (node:internal/streams/readable:278:10)
    at TLSWrap.onStreamRead (node:internal/stream_base_commons:190:23)
    at TLSWrap.callbackTrampoline (node:internal/async_hooks:128:17)

Trying to restart the worker produces this error:

2024-02-23T09:48:43.613394Z WARN temporal_client::retry: gRPC call get_workflow_execution_history retried 8 times error=Status { code: OutOfRange, message: "Error, message length too large: found 7015665 bytes, the limit is: 4194304 bytes", metadata: MetadataMap { headers: {"content-type": "application/grpc", "date": "Fri, 23 Feb 2024 09:48:43 GMT", "server": "temporal"} }, source: None }

This is from the Typscript SDK. I am aware of the 4MB grpc limit but there is no activity payload that surpasses that threshold. Perhaps the get_workflow_execution_history method needs to split the requests to more packets?

I’m looking into it.

The current suspicion is that it’s server side misconfiguration, but I’d need to confirm.

Are you self hosting the server? What version and how is it configured?

Thanks! No self-hosting, I am using Temporal cloud.

Note that the first error might be unrelated. Stopping a worker and trying to restart it again results to the same “message length too large” error for get_workflow_execution_history.

Do you have a proxy in front of Temporal Cloud?

No proxy in front of the cloud.

We’ll look into it.

Could you please open a support ticket on ZenDesk with your account and namespace details so we can inspect and verify if it’s coming from our side?

1 Like

Support request sent.

The workflow seems healthy on the dashboard, it’s just that, if the worker stops it cannot replay/resume. This sounds like a problem on the client side but…

I read up that grpc has a 4MB receive limit but no send limit, so perhaps your server is sending a grpc response that is larger than that and it rejected by the typescript sdk.

Looks like this 4MB limit is enforced by the Rust Tonic gRPC library.

This limit was introduced in version 0.9.0.

We’ll have to bump the limit in sdk-core and cut a new TypeScript SDK release ASAP.

In the meantime, downgrading the TS SDK to 1.8.6 will allow you to process this large workflow history.

1 Like

Here’s the fix:

I’ll try to push for a TypeScript release to include it today.

I’ll just published this in 1.9.3.

1 Like