Always getting context canceled while querying workflow instance

Community Support
1

Whenever I trying to query a workflow instance using workflow ID and RunId, I can see that the workflow instance is processing the request fine but when it;'s about to return the response, the following exceptions is throw:

  {
      "@timestamp": "2020-10-18T03:35:21.845Z",
      "@version": "1",
      "message": "uncaught exception",
      "logger_name": "io.temporal.internal.worker.PollerOptions",
      "thread_name": "Workflow Executor taskQueue=\"ctct:dp:onboarding\", namespace=\"ctct:dp:onboarding\": 24",
      "level": "ERROR",
      "level_value": 40000,
      "stack_trace": "java.lang.RuntimeException: Failure processing workflow task. WorkflowId=5417632e-1c79-4d02-b314-ba6668fba0c7, RunId=ee02bb0c-c460-49d5-9474-da175d649db2\n\tat io.temporal.internal.worker.WorkflowWorker$TaskHandlerImpl.wrapFailure(WorkflowWorker.java:337)\n\tat io.temporal.internal.worker.WorkflowWorker$TaskHandlerImpl.wrapFailure(WorkflowWorker.java:275)\n\tat io.temporal.internal.worker.PollTaskExecutor.lambda$process$0(PollTaskExecutor.java:79)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)\n\tat java.base/java.lang.Thread.run(Thread.java:834)\nCaused by: io.grpc.StatusRuntimeException: DEADLINE_EXCEEDED: deadline exceeded after 9.999975601s. [remote_addr=woe.eigi-dl-rmf-woe-dev.local/10.51.12.115:7233]\n\tat io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:262)\n\tat io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:243)\n\tat io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:156)\n\tat io.temporal.api.workflowservice.v1.WorkflowServiceGrpc$WorkflowServiceBlockingStub.respondQueryTaskCompleted(WorkflowServiceGrpc.java:2966)\n\tat io.temporal.internal.worker.WorkflowWorker$TaskHandlerImpl.sendReply(WorkflowWorker.java:402)\n\tat io.temporal.internal.worker.WorkflowWorker$TaskHandlerImpl.handle(WorkflowWorker.java:313)\n\tat io.temporal.internal.worker.WorkflowWorker$TaskHandlerImpl.handle(WorkflowWorker.java:275)\n\tat io.temporal.internal.worker.PollTaskExecutor.lambda$process$0(PollTaskExecutor.java:73)\n\t... 3 common frames omitted\n",
      "TaskQueue": "ctct:dp:onboarding",
      "Namespace": "ctct:dp:onboarding"
  }

and exception from the client side,

  {
      "@timestamp": "2020-10-18T03:32:42.076Z",
      "@version": "1",
      "message": "Retrying after failure",
      "logger_name": "io.temporal.internal.common.GrpcRetryer",
      "thread_name": "default-nioEventLoopGroup-1-2",
      "level": "WARN",
      "level_value": 30000,
      "stack_trace": "io.grpc.StatusRuntimeException: DEADLINE_EXCEEDED: deadline exceeded after 4.887217319s. [buffered_nanos=1008362861, remote_addr=woe.eigi-dl-rmf-woe-dev.local/10.51.12.115:7233]\n\tat io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:262)\n\tat io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:243)\n\tat io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:156)\n\tat io.temporal.api.workflowservice.v1.WorkflowServiceGrpc$WorkflowServiceBlockingStub.queryWorkflow(WorkflowServiceGrpc.java:2992)\n\tat io.temporal.internal.external.GenericWorkflowClientExternalImpl.lambda$query$5(GenericWorkflowClientExternalImpl.java:209)\n\tat io.temporal.internal.common.GrpcRetryer.retryWithResult(GrpcRetryer.java:127)\n\tat io.temporal.internal.external.GenericWorkflowClientExternalImpl.query(GenericWorkflowClientExternalImpl.java:203)\n\tat io.temporal.internal.sync.WorkflowStubImpl.query(WorkflowStubImpl.java:470)\n\tat io.temporal.internal.sync.WorkflowInvocationHandler$SyncWorkflowInvocationHandler.queryWorkflow(WorkflowInvocationHandler.java:309)\n\tat io.temporal.internal.sync.WorkflowInvocationHandler$SyncWorkflowInvocationHandler.invoke(WorkflowInvocationHandler.java:272)\n\tat io.temporal.internal.sync.WorkflowInvocationHandler.invoke(WorkflowInvocationHandler.java:178)\n\tat com.sun.proxy.$Proxy60.getState(Unknown Source)\n\tat com.eigi.rmf.workflow.management.controllers.instance.QueryActionImpl.lambda$execute$0(QueryActionImpl.java:59)\n\tat java.base/java.util.Optional.map(Optional.java:265)\n\tat com.eigi.rmf.workflow.management.controllers.instance.QueryActionImpl.lambda$execute$2(QueryActionImpl.java:49)\n\tat java.base/java.util.Optional.map(Optional.java:265)\n\tat com.eigi.rmf.workflow.management.controllers.instance.QueryActionImpl.execute(QueryActionImpl.java:48)\n\tat com.eigi.rmf.workflow.management.controllers.WorkflowInstanceController.doQuery(WorkflowInstanceController.java:72)\n\tat com.eigi.rmf.workflow.management.controllers.$WorkflowInstanceControllerDefinition$Intercepted.$$access2(Unknown Source)\n\tat com.eigi.rmf.workflow.management.controllers.$WorkflowInstanceControllerDefinition$Intercepted$$proxy2.invokeInternal(Unknown Source)\n\tat io.micronaut.context.AbstractExecutableMethod.invoke(AbstractExecutableMethod.java:146)\n\tat io.micronaut.aop.chain.MethodInterceptorChain.proceed(MethodInterceptorChain.java:73)\n\tat io.micronaut.validation.ValidatingInterceptor.intercept(ValidatingInterceptor.java:139)\n\tat io.micronaut.aop.chain.MethodInterceptorChain.proceed(MethodInterceptorChain.java:82)\n\tat com.eigi.rmf.workflow.management.controllers.$WorkflowInstanceControllerDefinition$Intercepted.doQuery(Unknown Source)\n\tat com.eigi.rmf.workflow.management.controllers.$$WorkflowInstanceControllerDefinition$InterceptedDefinition$$exec3.invokeInternal(Unknown Source)\n\tat io.micronaut.context.AbstractExecutableMethod.invoke(AbstractExecutableMethod.java:146)\n\tat io.micronaut.context.DefaultBeanContext$4.invoke(DefaultBeanContext.java:474)\n\tat io.micronaut.web.router.AbstractRouteMatch.execute(AbstractRouteMatch.java:312)\n\tat io.micronaut.web.router.RouteMatch.execute(RouteMatch.java:118)\n\tat io.micronaut.http.server.netty.RoutingInBoundHandler.lambda$buildResultEmitter$10(RoutingInBoundHandler.java:1363)\n\tat io.reactivex.internal.operators.flowable.FlowableDefer.subscribeActual(FlowableDefer.java:35)\n\tat io.reactivex.Flowable.subscribe(Flowable.java:14918)\n\tat io.reactivex.Flowable.subscribe(Flowable.java:14865)\n\tat io.reactivex.internal.operators.flowable.FlowableFlatMap$MergeSubscriber.onNext(FlowableFlatMap.java:163)\n\tat io.micronaut.reactive.rxjava2.RxInstrumentedSubscriber.onNext(RxInstrumentedSubscriber.java:59)\n\tat io.reactivex.internal.subscriptions.DeferredScalarSubscription.complete(DeferredScalarSubscription.java:132)\n\tat io.reactivex.internal.operators.maybe.MaybeToFlowable$MaybeToFlowableSubscriber.onSuccess(MaybeToFlowable.java:70)\n\tat io.micronaut.reactive.rxjava2.RxInstrumentedMaybeObserver.onSuccess(RxInstrumentedMaybeObserver.java:65)\n\tat io.reactivex.internal.operators.maybe.MaybeDoOnEvent$DoOnEventMaybeObserver.onSuccess(MaybeDoOnEvent.java:86)\n\tat io.micronaut.reactive.rxjava2.RxInstrumentedMaybeObserver.onSuccess(RxInstrumentedMaybeObserver.java:65)\n\tat io.reactivex.internal.operators.flowable.FlowableElementAtMaybe$ElementAtSubscriber.onNext(FlowableElementAtMaybe.java:80)\n\tat io.micronaut.reactive.rxjava2.RxInstrumentedSubscriber.onNext(RxInstrumentedSubscriber.java:59)\n\tat io.reactivex.internal.operators.flowable.FlowableFlatMap$MergeSubscriber.drainLoop(FlowableFlatMap.java:501)\n\tat io.reactivex.internal.operators.flowable.FlowableFlatMap$MergeSubscriber.drain(FlowableFlatMap.java:366)\n\tat io.reactivex.internal.operators.flowable.FlowableFlatMap$InnerSubscriber.onNext(FlowableFlatMap.java:665)\n\tat io.micronaut.reactive.rxjava2.RxInstrumentedSubscriber.onNext(RxInstrumentedSubscriber.java:59)\n\tat io.reactivex.internal.subscriptions.DeferredScalarSubscription.complete(DeferredScalarSubscription.java:117)\n\tat io.reactivex.internal.operators.maybe.MaybeToFlowable$MaybeToFlowableSubscriber.onSuccess(MaybeToFlowable.java:70)\n\tat io.micronaut.reactive.rxjava2.RxInstrumentedMaybeObserver.onSuccess(RxInstrumentedMaybeObserver.java:65)\n\tat io.reactivex.internal.operators.maybe.MaybeMap$MapMaybeObserver.onSuccess(MaybeMap.java:89)\n\tat io.micronaut.reactive.rxjava2.RxInstrumentedMaybeObserver.onSuccess(RxInstrumentedMaybeObserver.java:65)\n\tat io.reactivex.internal.operators.flowable.FlowableElementAtMaybe$ElementAtSubscriber.onNext(FlowableElementAtMaybe.java:80)\n\tat io.micronaut.reactive.rxjava2.RxInstrumentedSubscriber.onNext(RxInstrumentedSubscriber.java:59)\n\tat io.reactivex.internal.operators.flowable.FlowableFlatMap$MergeSubscriber.drainLoop(FlowableFlatMap.java:501)\n\tat io.reactivex.internal.operators.flowable.FlowableFlatMap$MergeSubscriber.drain(FlowableFlatMap.java:366)\n\tat io.reactivex.internal.operators.flowable.FlowableFlatMap$InnerSubscriber.onSubscribe(FlowableFlatMap.java:646)\n\tat io.micronaut.reactive.rxjava2.RxInstrumentedSubscriber.onSubscribe(RxInstrumentedSubscriber.java:52)\n\tat io.reactivex.internal.operators.flowable.FlowableJust.subscribeActual(FlowableJust.java:34)\n\tat io.reactivex.Flowable.subscribe(Flowable.java:14918)\n\tat io.reactivex.Flowable.subscribe(Flowable.java:14865)\n\tat io.micronaut.reactive.rxjava2.RxInstrumentedFlowable.subscribeActual(RxInstrumentedFlowable.java:57)\n\tat io.reactivex.Flowable.subscribe(Flowable.java:14918)\n\tat io.reactivex.Flowable.subscribe(Flowable.java:14865)\n\tat io.reactivex.internal.operators.flowable.FlowableFlatMap$MergeSubscriber.onNext(FlowableFlatMap.java:163)\n\tat io.micronaut.reactive.rxjava2.RxInstrumentedSubscriber.onNext(RxInstrumentedSubscriber.java:59)\n\tat io.reactivex.internal.operators.flowable.FlowableFromIterable$IteratorSubscription.slowPath(FlowableFromIterable.java:236)\n\tat io.reactivex.internal.operators.flowable.FlowableFromIterable$BaseRangeSubscription.request(FlowableFromIterable.java:124)\n\tat io.reactivex.internal.operators.flowable.FlowableFlatMap$MergeSubscriber.onSubscribe(FlowableFlatMap.java:117)\n\tat io.micronaut.reactive.rxjava2.RxInstrumentedSubscriber.onSubscribe(RxInstrumentedSubscriber.java:52)\n\tat io.reactivex.internal.operators.flowable.FlowableFromIterable.subscribe(FlowableFromIterable.java:69)\n\tat io.reactivex.internal.operators.flowable.FlowableFromIterable.subscribeActual(FlowableFromIterable.java:47)\n\tat io.reactivex.Flowable.subscribe(Flowable.java:14918)\n\tat io.reactivex.Flowable.subscribe(Flowable.java:14865)\n\tat io.micronaut.reactive.rxjava2.RxInstrumentedFlowable.subscribeActual(RxInstrumentedFlowable.java:57)\n\tat io.reactivex.Flowable.subscribe(Flowable.java:14918)\n\tat io.reactivex.internal.operators.flowable.FlowableFlatMap.subscribeActual(FlowableFlatMap.java:53)\n\tat io.reactivex.Flowable.subscribe(Flowable.java:14918)\n\tat io.reactivex.internal.operators.flowable.FlowableElementAtMaybe.subscribeActual(FlowableElementAtMaybe.java:36)\n\tat io.reactivex.Maybe.subscribe(Maybe.java:4290)\n\tat io.micronaut.reactive.rxjava2.RxInstrumentedMaybe.subscribeActual(RxInstrumentedMaybe.java:53)\n\tat io.reactivex.Maybe.subscribe(Maybe.java:4290)\n\tat io.reactivex.internal.operators.maybe.MaybeMap.subscribeActual(MaybeMap.java:40)\n\tat io.reactivex.Maybe.subscribe(Maybe.java:4290)\n\tat io.reactivex.internal.operators.maybe.MaybeToFlowable.subscribeActual(MaybeToFlowable.java:45)\n\tat io.reactivex.Flowable.subscribe(Flowable.java:14918)\n\tat io.reactivex.Flowable.subscribe(Flowable.java:14865)\n\tat io.reactivex.internal.operators.flowable.FlowableFlatMap$MergeSubscriber.onNext(FlowableFlatMap.java:163)\n\tat io.micronaut.reactive.rxjava2.RxInstrumentedSubscriber.onNext(RxInstrumentedSubscriber.java:59)\n\tat io.reactivex.internal.operators.flowable.FlowableFromIterable$IteratorSubscription.slowPath(FlowableFromIterable.java:236)\n\tat io.reactivex.internal.operators.flowable.FlowableFromIterable$BaseRangeSubscription.request(FlowableFromIterable.java:124)\n\tat io.reactivex.internal.operators.flowable.FlowableFlatMap$MergeSubscriber.onSubscribe(FlowableFlatMap.java:117)\n\tat io.micronaut.reactive.rxjava2.RxInstrumentedSubscriber.onSubscribe(RxInstrumentedSubscriber.java:52)\n\tat io.reactivex.internal.operators.flowable.FlowableFromIterable.subscribe(FlowableFromIterable.java:69)\n\tat io.reactivex.internal.operators.flowable.FlowableFromIterable.subscribeActual(FlowableFromIterable.java:47)\n\tat io.reactivex.Flowable.subscribe(Flowable.java:14918)\n\tat io.reactivex.Flowable.subscribe(Flowable.java:14865)\n\tat io.micronaut.reactive.rxjava2.RxInstrumentedFlowable.subscribeActual(RxInstrumentedFlowable.java:57)\n\tat io.reactivex.Flowable.subscribe(Flowable.java:14918)\n\tat io.reactivex.internal.operators.flowable.FlowableFlatMap.subscribeActual(FlowableFlatMap.java:53)\n\tat io.reactivex.Flowable.subscribe(Flowable.java:14918)\n\tat io.reactivex.internal.operators.flowable.FlowableElementAtMaybe.subscribeActual(FlowableElementAtMaybe.java:36)\n\tat io.reactivex.Maybe.subscribe(Maybe.java:4290)\n\tat io.micronaut.reactive.rxjava2.RxInstrumentedMaybe.subscribeActual(RxInstrumentedMaybe.java:53)\n\tat io.reactivex.Maybe.subscribe(Maybe.java:4290)\n\tat io.reactivex.internal.operators.maybe.MaybeDoOnEvent.subscribeActual(MaybeDoOnEvent.java:39)\n\tat io.reactivex.Maybe.subscribe(Maybe.java:4290)\n\tat io.reactivex.internal.operators.maybe.MaybeToFlowable.subscribeActual(MaybeToFlowable.java:45)\n\tat io.reactivex.Flowable.subscribe(Flowable.java:14918)\n\tat io.reactivex.internal.operators.flowable.FlowableFlatMap.subscribeActual(FlowableFlatMap.java:53)\n\tat io.reactivex.Flowable.subscribe(Flowable.java:14918)\n\tat io.reactivex.internal.operators.flowable.FlowableSwitchIfEmpty.subscribeActual(FlowableSwitchIfEmpty.java:32)\n\tat io.reactivex.Flowable.subscribe(Flowable.java:14918)\n\tat io.reactivex.Flowable.subscribe(Flowable.java:14865)\n\tat io.micronaut.http.server.context.ServerRequestContextFilter.lambda$doFilter$0(ServerRequestContextFilter.java:62)\n\tat io.reactivex.internal.operators.flowable.FlowableFromPublisher.subscribeActual(FlowableFromPublisher.java:29)\n\tat io.reactivex.Flowable.subscribe(Flowable.java:14918)\n\tat io.reactivex.Flowable.subscribe(Flowable.java:14868)\n\tat io.micronaut.http.server.netty.RoutingInBoundHandler.lambda$buildExecutableRoute$6(RoutingInBoundHandler.java:1068)\n\tat io.micronaut.web.router.DefaultUriRouteMatch$1.execute(DefaultUriRouteMatch.java:80)\n\tat io.micronaut.web.router.RouteMatch.execute(RouteMatch.java:118)\n\tat io.micronaut.http.server.netty.RoutingInBoundHandler$2.doOnComplete(RoutingInBoundHandler.java:994)\n\tat io.micronaut.core.async.subscriber.CompletionAwareSubscriber.onComplete(CompletionAwareSubscriber.java:71)\n\tat io.micronaut.http.server.netty.jackson.JsonContentProcessor$1.doOnComplete(JsonContentProcessor.java:140)\n\tat io.micronaut.core.async.subscriber.CompletionAwareSubscriber.onComplete(CompletionAwareSubscriber.java:71)\n\tat java.base/java.util.Optional.ifPresent(Optional.java:183)\n\tat io.micronaut.core.async.processor.SingleThreadedBufferingProcessor.doOnComplete(SingleThreadedBufferingProcessor.java:48)\n\tat io.micronaut.jackson.parser.JacksonProcessor.doOnComplete(JacksonProcessor.java:137)\n\tat io.micronaut.core.async.subscriber.SingleThreadedBufferingSubscriber.onComplete(SingleThreadedBufferingSubscriber.java:70)\n\tat io.micronaut.http.server.netty.jackson.JsonContentProcessor.doOnComplete(JsonContentProcessor.java:165)\n\tat io.micronaut.core.async.subscriber.CompletionAwareSubscriber.onComplete(CompletionAwareSubscriber.java:71)\n\tat io.micronaut.http.netty.reactive.HandlerPublisher.complete(HandlerPublisher.java:416)\n\tat io.micronaut.http.netty.reactive.HandlerPublisher.handlerRemoved(HandlerPublisher.java:403)\n\tat io.netty.channel.AbstractChannelHandlerContext.callHandlerRemoved(AbstractChannelHandlerContext.java:946)\n\tat io.netty.channel.DefaultChannelPipeline.callHandlerRemoved0(DefaultChannelPipeline.java:637)\n\tat io.netty.channel.DefaultChannelPipeline.remove(DefaultChannelPipeline.java:477)\n\tat io.netty.channel.DefaultChannelPipeline.remove(DefaultChannelPipeline.java:423)\n\tat io.micronaut.http.netty.stream.HttpStreamsHandler.removeHandlerIfActive(HttpStreamsHandler.java:388)\n\tat io.micronaut.http.netty.stream.HttpStreamsHandler.handleReadHttpContent(HttpStreamsHandler.java:253)\n\tat io.micronaut.http.netty.stream.HttpStreamsHandler.channelRead(HttpStreamsHandler.java:228)\n\tat io.micronaut.http.netty.stream.HttpStreamsServerHandler.channelRead(HttpStreamsServerHandler.java:121)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)\n\tat io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)\n\tat io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)\n\tat io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)\n\tat io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103)\n\tat io.netty.handler.codec.MessageToMessageCodec.channelRead(MessageToMessageCodec.java:111)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)\n\tat io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)\n\tat io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)\n\tat io.netty.handler.codec.http.HttpServerKeepAliveHandler.channelRead(HttpServerKeepAliveHandler.java:64)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)\n\tat io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)\n\tat io.netty.handler.flow.FlowControlHandler.dequeue(FlowControlHandler.java:200)\n\tat io.netty.handler.flow.FlowControlHandler.read(FlowControlHandler.java:139)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeRead(AbstractChannelHandlerContext.java:686)\n\tat io.netty.channel.AbstractChannelHandlerContext.read(AbstractChannelHandlerContext.java:671)\n\tat io.micronaut.http.netty.reactive.HandlerPublisher.requestDemand(HandlerPublisher.java:163)\n\tat io.micronaut.http.netty.stream.HttpStreamsHandler$1.requestDemand(HttpStreamsHandler.java:220)\n\tat io.micronaut.http.netty.reactive.HandlerPublisher$ChannelSubscription.receivedDemand(HandlerPublisher.java:547)\n\tat io.micronaut.http.netty.reactive.HandlerPublisher$ChannelSubscription.lambda$request$0(HandlerPublisher.java:474)\n\tat io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164)\n\tat io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472)\n\tat io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:497)\n\tat io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)\n\tat io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)\n\tat io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)\n\tat java.base/java.lang.Thread.run(Thread.java:834)\n"
  }
2

Temporal excepton.

{
    "level": "error",
    "ts": "2020-10-18T03:35:36.867Z",
    "msg": "Unknown error",
    "service": "frontend",
    "error": "context canceled",
    "logging-call-at": "workflowHandler.go:3417",
    "stacktrace": "go.temporal.io/server/common/log/loggerimpl.(*loggerImpl).Error\n\t/temporal/common/log/loggerimpl/logger.go:138\ngo.temporal.io/server/service/frontend.(*WorkflowHandler).error\n\t/temporal/service/frontend/workflowHandler.go:3417\ngo.temporal.io/server/service/frontend.(*WorkflowHandler).RespondQueryTaskCompleted\n\t/temporal/service/frontend/workflowHandler.go:2911\ngo.temporal.io/server/service/frontend.(*DCRedirectionHandlerImpl).RespondQueryTaskCompleted.func2\n\t/temporal/service/frontend/dcRedirectionHandler.go:1017\ngo.temporal.io/server/service/frontend.(*NoopRedirectionPolicy).WithNamespaceIDRedirect\n\t/temporal/service/frontend/dcRedirectionPolicy.go:111\ngo.temporal.io/server/service/frontend.(*DCRedirectionHandlerImpl).RespondQueryTaskCompleted\n\t/temporal/service/frontend/dcRedirectionHandler.go:1013\ngo.temporal.io/server/service/frontend.(*AccessControlledWorkflowHandler).RespondQueryTaskCompleted\n\t/temporal/service/frontend/accessControlledHandler.go:610\ngo.temporal.io/server/service/frontend.(*WorkflowNilCheckHandler).RespondQueryTaskCompleted\n\t/temporal/service/frontend/workflowNilCheckHandler.go:409\ngo.temporal.io/api/workflowservice/v1._WorkflowService_RespondQueryTaskCompleted_Handler.func1\n\t/go/pkg/mod/go.temporal.io/api@v0.29.0/workflowservice/v1/service.pb.go:1521\ngo.temporal.io/server/service/frontend.interceptor\n\t/temporal/service/frontend/service.go:325\ngo.temporal.io/api/workflowservice/v1._WorkflowService_RespondQueryTaskCompleted_Handler\n\t/go/pkg/mod/go.temporal.io/api@v0.29.0/workflowservice/v1/service.pb.go:1523\ngoogle.golang.org/grpc.(*Server).processUnaryRPC\n\t/go/pkg/mod/google.golang.org/grpc@v1.31.0/server.go:1180\ngoogle.golang.org/grpc.(*Server).handleStream\n\t/go/pkg/mod/google.golang.org/grpc@v1.31.0/server.go:1503\ngoogle.golang.org/grpc.(*Server).serveStreams.func1.2\n\t/go/pkg/mod/google.golang.org/grpc@v1.31.0/server.go:843"
}
3

What is the query code doing? Does it do anything besides returning data? From the traces, it appears like the query method took a long time to execute.

Also, what version of temporal SDK and the service are you running? Are you able to reproduce the problem with the latest ones?

4

Using version 0.29.0, and It takes on avg 120ms to run the code inside the query, it’s only converting a map to a json string:

try {
    val json = ObjectMapperUtils.instance().convertValue(result, JsonNode.class);
    return json.toString();
} catch (Exception e) {
    log.error("Failed to create workflow response", e);
}
5

Would you try reproducing the problem against the latest service release? It could be caused by using the newer Java SDK against the old service version.
We have added check for service version in the production service release to avoid such issues.

6

Just did, same exceptions. I’m using v1.0.0. Everything is running using the same version.