Often seeing "consistent query buffer is full, cannot accept new consistent queries"

hi i am seeing may “consistent query buffer is full, cannot accept new consistent queries” in my wofklow.

The only thing which i did recently is to move to 1.6.3, can this be related?

{“level”:“error”,“ts”:“2021-02-05T12:40:25.910Z”,“msg”:“Internal service error”,“service”:“history”,“error”:“consistent query buffer is full, cannot accept new consistent queries”,“wf-id”:“feb20210504e2-job”,“wf-namespace-id”:“493f72d9-4de5-4e0f-b113-3e37449990fa”,“logging-call-at”:“handler.go:1788”,“stacktrace”:“go.temporal.io/server/common/log/loggerimpl.(*loggerImpl).Error\n\t/temporal/common/log/loggerimpl/logger.go:138\ngo.temporal.io/server/service/history.(*Handler).updateErrorMetric\n\t/temporal/service/history/handler.go:1788\ngo.temporal.io/server/service/history.(*Handler).error\n\t/temporal/service/history/handler.go:1806\ngo.temporal.io/server/service/history.(*Handler).QueryWorkflow\n\t/temporal/service/history/handler.go:1167\ngo.temporal.io/server/api/historyservice/v1._HistoryService_QueryWorkflow_Handler.func1\n\t/temporal/api/historyservice/v1/service.pb.go:1401\ngo.temporal.io/server/common/rpc.ServiceErrorInterceptor\n\t/temporal/common/rpc/grpc.go:100\ngo.temporal.io/server/api/historyservice/v1._HistoryService_QueryWorkflow_Handler\n\t/temporal/api/historyservice/v1/service.pb.go:1403\ngoogle.golang.org/grpc.(*Server).processUnaryRPC\n\t/go/pkg/mod/google.golang.org/grpc@v1.34.0/server.go:1210\ngoogle.golang.org/grpc.(*Server).handleStream\n\t/go/pkg/mod/google.golang.org/grpc@v1.34.0/server.go:1533\ngoogle.golang.org/grpc.(*Server).serveStreams.func1.2\n\t/go/pkg/mod/google.golang.org/grpc@v1.34.0/server.go:871”}

the above issue can be related to: Timeout issues after upgrading to 1.6.3 temporal and 1.0.5 java client

basically the java SDK cannot make forward progress (and unable to handle workflow query)

@agreed, this is side effect of race condition issue and is addressed in this PR