Error, message length too large

Community Support
1

My typescript workflow/activity worker runs into this error:

/home/ubuntu/stakexyz/stakexyz-staking/node_modules/ws/lib/websocket.js:912
      abortHandshake(
      ^
Error: Unexpected server response: 502
    at ClientRequest.<anonymous> (/home/ubuntu/stakexyz/stakexyz-staking/node_modules/ws/lib/websocket.js:912:7)
    at ClientRequest.emit (node:events:517:28)
    at ClientRequest.emit (node:domain:489:12)
    at HTTPParser.parserOnIncomingClient (node:_http_client:700:27)
    at HTTPParser.parserOnHeadersComplete (node:_http_common:119:17)
    at TLSSocket.socketOnData (node:_http_client:541:22)
    at TLSSocket.emit (node:events:517:28)
    at TLSSocket.emit (node:domain:489:12)
    at addChunk (node:internal/streams/readable:368:12)
    at readableAddChunk (node:internal/streams/readable:341:9)
    at TLSSocket.Readable.push (node:internal/streams/readable:278:10)
    at TLSWrap.onStreamRead (node:internal/stream_base_commons:190:23)
    at TLSWrap.callbackTrampoline (node:internal/async_hooks:128:17)

Trying to restart the worker produces this error:

2024-02-23T09:48:43.613394Z WARN temporal_client::retry: gRPC call get_workflow_execution_history retried 8 times error=Status { code: OutOfRange, message: "Error, message length too large: found 7015665 bytes, the limit is: 4194304 bytes", metadata: MetadataMap { headers: {"content-type": "application/grpc", "date": "Fri, 23 Feb 2024 09:48:43 GMT", "server": "temporal"} }, source: None }

This is from the Typscript SDK. I am aware of the 4MB grpc limit but there is no activity payload that surpasses that threshold. Perhaps the get_workflow_execution_history method needs to split the requests to more packets?

2

Iā€™m looking into it.

The current suspicion is that itā€™s server side misconfiguration, but Iā€™d need to confirm.

Are you self hosting the server? What version and how is it configured?

3

Thanks! No self-hosting, I am using Temporal cloud.

Note that the first error might be unrelated. Stopping a worker and trying to restart it again results to the same ā€œmessage length too largeā€ error for get_workflow_execution_history.

4

Do you have a proxy in front of Temporal Cloud?

5

No proxy in front of the cloud.

6

Weā€™ll look into it.

Could you please open a support ticket on ZenDesk with your account and namespace details so we can inspect and verify if itā€™s coming from our side?

1 Like
7

Support request sent.

The workflow seems healthy on the dashboard, itā€™s just that, if the worker stops it cannot replay/resume. This sounds like a problem on the client side butā€¦

I read up that grpc has a 4MB receive limit but no send limit, so perhaps your server is sending a grpc response that is larger than that and it rejected by the typescript sdk.

8

Looks like this 4MB limit is enforced by the Rust Tonic gRPC library.

This limit was introduced in version 0.9.0.

Weā€™ll have to bump the limit in sdk-core and cut a new TypeScript SDK release ASAP.

In the meantime, downgrading the TS SDK to 1.8.6 will allow you to process this large workflow history.

1 Like
9

Hereā€™s the fix:

Iā€™ll try to push for a TypeScript release to include it today.

10

Iā€™ll just published this in 1.9.3.

1 Like