Workflow Signal throwing Connection Drop exception

Community Support
1

image
image1876×572 33.1 KB

In the attached image we can see that worklow is running, but still when the below code is executed to pass the signal getting { cause: { code: 14, details: ‘Connection dropped’, metadata: {} } }
const handle = client.getHandle(payload.data.body.workflowId);
console.log(handle.describe);
console.log(handle.workflowId);
await handle.signal(signal, payload.data.body.action);

Could you help us here

2

Are you able to use the client for anything else?
Is this an intermittent error?

You may have some success getting more info with these environment variables:

GRPC_TRACE=all GRPC_VERBOSITY=DEBUG
3

It is consitently failing with connection dropped, also could you help me how to set GRPC_TRACE=all GRPC_VERBOSITY=DEBUG

4 
const handle = client.workflow.getHandle(workflowId);
console.log(handle.workflowId);
await handle.signal(siganlReq, action);

Above is the code snippet and i see workflow id is getting printed in the log which matches the workflow id , but i still get connection dropped error

5 
export GRPC_TRACE=all GRPC_VERBOSITY=DEBUG

<command to run your client here>

It’s likely that you’re having connection issues, it’s not related to the signal.

6

Basically we have two different mono repo, signal are sent from one mono repo to another mono repo workflow. During that scenario we are seeing connection dropped error. below is the error
ServiceError: Failed to signal Workflow
at WorkflowClient.rethrowGrpcError (/Users/mitshett/Documents/PIUCD/piucd-backend/node_modules/@temporalio/client/src/workflow-client.ts:707:13)
at WorkflowClient._signalWorkflowHandler (/Users/mitshett/Documents/PIUCD/piucd-backend/node_modules/@temporalio/client/src/workflow-client.ts:881:12)

temporal service log

 {"level":"error","ts":"2024-02-14T17:51:20.490Z","msg":"Unable to call matching.PollWorkflowTaskQueue.","service":"frontend","wf-task-queue-name":"/_sys/core-workflow-interaction/1","timeout":"1m9.999385166s","error":"Not enough hosts to serve the request","logging-call-at":"workflow_handler.go:920","stacktrace":"go.temporal.io/server/common/log.(*zapLogger).Error\n\t/home/builder/temporal/common/log/zap_logger.go:156\ngo.temporal.io/server/service/frontend.(*WorkflowHandler).PollWorkflowTaskQueue\n\t/home/builder/temporal/service/frontend/workflow_handler.go:920\ngo.temporal.io/api/workflowservice/v1._WorkflowService_PollWorkflowTaskQueue_Handler.func1\n\t/go/pkg/mod/go.temporal.io/api@v1.24.1-0.20231003165936-bb03061759c8/workflowservice/v1/service.pb.go:1679\ngo.temporal.io/server/common/rpc/interceptor.(*RetryableInterceptor).Intercept.func1\n\t/home/builder/temporal/common/rpc/interceptor/retry.go:63\ngo.temporal.io/server/common/backoff.ThrottleRetryContext\n\t/home/builder/temporal/common/backoff/retry.go:145\ngo.temporal.io/server/common/rpc/interceptor.(*RetryableInterceptor).Intercept\n\t/home/builder/temporal/common/rpc/interceptor/retry.go:67\ngoogle.golang.org/grpc.getChainUnaryHandler.func1\n\t/go/pkg/mod/google.golang.org/grpc@v1.58.2/server.go:1195\ngo.temporal.io/server/common/rpc/interceptor.(*CallerInfoInterceptor).Intercept\n\t/home/builder/temporal/common/rpc/interceptor/caller_info.go:80\ngoogle.golang.org/grpc.getChainUnaryHandler.func1\n\t/go/pkg/mod/google.golang.org/grpc@v1.58.2/server.go:1195\ngo.temporal.io/server/common/rpc/interceptor.(*SDKVersionInterceptor).Intercept\n\t/home/builder/temporal/common/rpc/interceptor/sdk_version.go:69\ngoogle.golang.org/grpc.getChainUnaryHandler.func1\n\t/go/pkg/mod/google.golang.org/grpc@v1.58.2/server.go:1195\ngo.temporal.io/server/common/rpc/interceptor.(*RateLimitInterceptor).Intercept\n\t/home/builder/temporal/common/rpc/interceptor/rate_limit.go:88\ngoogle.golang.org/grpc.getChainUnaryHandler.func1\n\t/go/pkg/mod/google.golang.org/grpc@v1.58.2/server.go:1195\ngo.temporal.io/server/common/rpc/interceptor.(*NamespaceRateLimitInterceptor).Intercept\n\t/home/builder/temporal/common/rpc/interceptor/namespace_rate_limit.go:93\ngoogle.golang.org/grpc.getChainUnaryHandler.func1\n\t/go/pkg/mod/google.golang.org/grpc@v1.58.2/server.go:1195\ngo.temporal.io/server/common/rpc/interceptor.(*ConcurrentRequestLimitInterceptor).Intercept\n\t/home/builder/temporal/common/rpc/interceptor/concurrent_request_limit.go:121\ngoogle.golang.org/grpc.getChainUnaryHandler.func1\n\t/go/pkg/mod/google.golang.org/grpc@v1.58.2/server.go:1195\ngo.temporal.io/server/common/rpc/interceptor.(*NamespaceValidatorInterceptor).StateValidationIntercept\n\t/home/builder/temporal/common/rpc/interceptor/namespace_validator.go:194\ngoogle.golang.org/grpc.getChainUnaryHandler.func1\n\t/go/pkg/mod/google.golang.org/grpc@v1.58.2/server.go:1195\ngo.temporal.io/server/common/rpc/interceptor.(*TelemetryInterceptor).UnaryIntercept\n\t/home/builder/temporal/common/rpc/interceptor/telemetry.go:165\ngoogle.golang.org/grpc.getChainUnaryHandler.func1\n\t/go/pkg/mod/google.golang.org/grpc@v1.58.2/server.go:1195\ngo.temporal.io/server/service/frontend.(*RedirectionInterceptor).handleRedirectAPIInvocation.func2\n\t/home/builder/temporal/service/frontend/redirection_interceptor.go:238\ngo.temporal.io/server/service/frontend.(*NoopRedirectionPolicy).WithNamespaceRedirect\n\t/home/builder/temporal/service/frontend/dc_redirection_policy.go:125\ngo.temporal.io/server/service/frontend.(*RedirectionInterceptor).handleRedirectAPIInvocation\n\t/home/builder/temporal/service/frontend/redirection_interceptor.go:235\ngo.temporal.io/server/service/frontend.(*RedirectionInterceptor).Intercept\n\t/home/builder/temporal/service/frontend/redirection_interceptor.go:195\ngoogle.golang.org/grpc.getChainUnaryHandler.func1\n\t/go/pkg/mod/google.golang.org/grpc@v1.58.2/server.go:1195\ngo.temporal.io/server/common/authorization.(*interceptor).Interceptor\n\t/home/builder/temporal/common/authorization/interceptor.go:158\ngoogle.golang.org/grpc.getChainUnaryHandler.func1\n\t/go/pkg/mod/google.golang.org/grpc@v1.58.2/server.go:1195\ngo.temporal.io/server/common/metrics.NewServerMetricsContextInjectorInterceptor.func1\n\t/home/builder/temporal/common/metrics/grpc.go:66\ngoogle.golang.org/grpc.getChainUnaryHandler.func1\n\t/go/pkg/mod/google.golang.org/grpc@v1.58.2/server.go:1195\ngo.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc.UnaryServerInterceptor.func1\n\t/go/pkg/mod/go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc@v0.42.0/interceptor.go:344\ngoogle.golang.org/grpc.getChainUnaryHandler.func1\n\t/go/pkg/mod/google.golang.org/grpc@v1.58.2/server.go:1195\ngo.temporal.io/server/common/rpc/interceptor.(*NamespaceLogInterceptor).Intercept\n\t/home/builder/temporal/common/rpc/interceptor/namespace_logger.go:84\ngoogle.golang.org/grpc.getChainUnaryHandler.func1\n\t/go/pkg/mod/google.golang.org/grpc@v1.58.2/server.go:1195\ngo.temporal.io/server/common/rpc/interceptor.(*NamespaceValidatorInterceptor).NamespaceValidateIntercept\n\t/home/builder/temporal/common/rpc/interceptor/namespace_validator.go:111\ngoogle.golang.org/grpc.getChainUnaryHandler.func1\n\t/go/pkg/mod/google.golang.org/grpc@v1.58.2/server.go:1195\ngo.temporal.io/server/common/rpc.ServiceErrorInterceptor\n\t/home/builder/temporal/common/rpc/grpc.go:145\ngoogle.golang.org/grpc.chainUnaryInterceptors.func1\n\t/go/pkg/mod/google.golang.org/grpc@v1.58.2/server.go:1186\ngo.temporal.io/api/workflowservice/v1._WorkflowService_PollWorkflowTaskQueue_Handler\n\t/go/pkg/mod/go.temporal.io/api@v1.24.1-0.20231003165936-bb03061759c8/workflowservice/v1/service.pb.go:1681\ngoogle.golang.org/grpc.(*Server).processUnaryRPC\n\t/go/pkg/mod/google.golang.org/grpc@v1.58.2/server.go:1376\ngoogle.golang.org/grpc.(*Server).handleStream\n\t/go/pkg/mod/google.golang.org/grpc@v1.58.2/server.go:1753\ngoogle.golang.org/grpc.(*Server).serveStreams.func1.1\n\t/go/pkg/mod/google.golang.org/grpc@v1.58.2/server.go:998"}
7

There seems to be a server side configuration issue that’s not related to your app sending a signal.
The error you’re seeing seems to indicate an issue with the matching service being unavailable. Hard to tell exactly what’s happening with the limited info.