Streaming error Logs in Multi-Cluster Temporal Server Setup

Community Support Server Deployment
1

We are currently running a self-hosted multi-cluster Temporal server setup with replication enabled between the clusters. While workflows are being replicated successfully, we are encountering numerous error logs in the following categories:

  1. service/history/replication/stream_receiver.go:232 - ReplicationStreamError: StreamReceiver exit recv loop
  2. service/history/replication/stream_receiver.go:202 - ReplicationStreamError: StreamReceiver exit send loop
  3. service/history/replication/stream_sender.go:198 - ReplicationStreamError: StreamSender failed to receive
  4. service/history/replication/bi_direction_stream.go:197 - BiDirectionStream encountered unexpected error, closing: *serviceerror.Unavailable closing transport due to: connection error: desc = “error reading from server: EOF”, received prior goaway: code: NO_ERROR, debug data: “max_age”

We are using Temporal server version 1.25.0.

Could anyone provide insights into the possible reasons behind these errors? What impact might they have on our system, and how can we resolve them?

Thank you for your support and guidance.

2

@maxim @tihomir - Can you please help us on understanding the reasons for these error logs being generated and impact of them!