Any namespace manipulations -- context deadline exceeded

Community Support
1

Hi there.
I created temporal instance on my k8s, with helm chart:

helm install \
  --namespace temporal \
  --create-namespace \
  -f values/values.mysql.yaml \
  -f values/values.prometheus.external.yaml \
  --set server.replicaCount=1 \
  --set grafana.enabled=false \
  --set prometheus.enabled=false \
  --set elasticsearch.enabled=false \
  --set web.image.tag=2.27.1 \
temporal . --timeout 15m

as you can see, i use external mysql, no ES and external prometheus.
i created db and fill it with temporal-sql-tool.
first start have no namespaces, ok i create namespace with
temporal operator namespace create default
all ok, but now when i reload ui, i has 503 on search-attributes request, like

https://example/api/v1/namespaces/default/search-attributes

{
    "code": 14,
    "message": "Unable to get namespace info with error: default"
}

in log temporal-frontend pod:

{"level":"error","ts":"2024-05-24T07:04:18.862Z","msg":"service failures","operation":"OperatorListSearchAttributes","wf-namespace":"main","error":"Unable to get namespace info with error: main","logging-call-at":"telemetry.go:341","stacktrace":"go.temporal.io/server/common/log.(*zapLogger).Error\n\t/home/runner/work/docker-builds/docker-builds/temporal/common/log/zap_logger.go:156\ngo.temporal.io/server/common/rpc/interceptor.(*TelemetryInterceptor).handleError\n\t/home/runner/work/docker-builds/docker-builds/temporal/common/rpc/interceptor/telemetry.go:341\ngo.temporal.io/server/common/rpc/interceptor.(*TelemetryInterceptor).UnaryIntercept\n\t/home/runner/work/docker-builds/docker-builds/temporal/common/rpc/interceptor/telemetry.go:174\ngoogle.golang.org/grpc.getChainUnaryHandler.func1\n\t/home/runner/go/pkg/mod/google.golang.org/grpc@v1.63.2/server.go:1186\ngo.temporal.io/server/service/frontend.(*RedirectionInterceptor).Intercept\n\t/home/runner/work/docker-builds/docker-builds/temporal/service/frontend/redirection_interceptor.go:184\ngoogle.golang.org/grpc.getChainUnaryHandler.func1\n\t/home/runner/go/pkg/mod/google.golang.org/grpc@v1.63.2/server.go:1186\ngo.temporal.io/server/common/authorization.(*interceptor).Interceptor\n\t/home/runner/work/docker-builds/docker-builds/temporal/common/authorization/interceptor.go:162\ngoogle.golang.org/grpc.getChainUnaryHandler.func1\n\t/home/runner/go/pkg/mod/google.golang.org/grpc@v1.63.2/server.go:1186\ngo.temporal.io/server/service/frontend.GrpcServerOptionsProvider.NewServerMetricsContextInjectorInterceptor.func1\n\t/home/runner/work/docker-builds/docker-builds/temporal/common/metrics/grpc.go:66\ngoogle.golang.org/grpc.getChainUnaryHandler.func1\n\t/home/runner/go/pkg/mod/google.golang.org/grpc@v1.63.2/server.go:1186\ngo.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc.UnaryServerInterceptor.func1\n\t/home/runner/go/pkg/mod/go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc@v0.51.0/interceptor.go:315\ngoogle.golang.org/grpc.getChainUnaryHandler.func1\n\t/home/runner/go/pkg/mod/google.golang.org/grpc@v1.63.2/server.go:1186\ngo.temporal.io/server/common/rpc/interceptor.(*NamespaceLogInterceptor).Intercept\n\t/home/runner/work/docker-builds/docker-builds/temporal/common/rpc/interceptor/namespace_logger.go:84\ngoogle.golang.org/grpc.getChainUnaryHandler.func1\n\t/home/runner/go/pkg/mod/google.golang.org/grpc@v1.63.2/server.go:1186\ngo.temporal.io/server/common/rpc/interceptor.(*NamespaceValidatorInterceptor).NamespaceValidateIntercept\n\t/home/runner/work/docker-builds/docker-builds/temporal/common/rpc/interceptor/namespace_validator.go:113\ngoogle.golang.org/grpc.getChainUnaryHandler.func1\n\t/home/runner/go/pkg/mod/google.golang.org/grpc@v1.63.2/server.go:1186\ngo.temporal.io/server/common/rpc.ServiceErrorInterceptor\n\t/home/runner/work/docker-builds/docker-builds/temporal/common/rpc/grpc.go:145\ngoogle.golang.org/grpc.NewServer.chainUnaryServerInterceptors.chainUnaryInterceptors.func1\n\t/home/runner/go/pkg/mod/google.golang.org/grpc@v1.63.2/server.go:1177\ngo.temporal.io/api/operatorservice/v1._OperatorService_ListSearchAttributes_Handler\n\t/home/runner/go/pkg/mod/go.temporal.io/api@v1.29.2/operatorservice/v1/service_grpc.pb.go:355\ngoogle.golang.org/grpc.(*Server).processUnaryRPC\n\t/home/runner/go/pkg/mod/google.golang.org/grpc@v1.63.2/server.go:1369\ngoogle.golang.org/grpc.(*Server).handleStream\n\t/home/runner/go/pkg/mod/google.golang.org/grpc@v1.63.2/server.go:1780\ngoogle.golang.org/grpc.(*Server).serveStreams.func2.1\n\t/home/runner/go/pkg/mod/google.golang.org/grpc@v1.63.2/server.go:1019"}

and if i want manipulate with namespace with cli, i has error too

$ TEMPORAL_CLI_SHOW_STACKS=1 temporal operator search-attribute create --name="CustomSA" --type="Keyword" --namespace="default"
Error: unable to get existing search attributes: context deadline exceeded
Stack trace:
goroutine 1 [running]:
runtime/debug.Stack()
	/opt/hostedtoolcache/go/1.21.9/x64/src/runtime/debug/stack.go:24 +0x5e
runtime/debug.PrintStack()
	/opt/hostedtoolcache/go/1.21.9/x64/src/runtime/debug/stack.go:16 +0x13
github.com/temporalio/cli/app.HandleError(0xc00099f408?, {0x318d760, 0xc000610100})
	/home/runner/work/docker-builds/docker-builds/cli/app/app.go:73 +0x153
github.com/urfave/cli/v2.(*App).handleExitCoder(0xc000118840?, 0xc000768600?, {0x318d760?, 0xc000610100?})
	/home/runner/go/pkg/mod/github.com/urfave/cli/v2@v2.25.7/app.go:452 +0x33
github.com/urfave/cli/v2.(*Command).Run(0xc000118840, 0xc0005e01c0, {0xc0005e0200, 0x4, 0x4})
	/home/runner/go/pkg/mod/github.com/urfave/cli/v2@v2.25.7/command.go:276 +0x9c5
github.com/urfave/cli/v2.(*Command).Run(0x46e0ae0, 0xc0005e00c0, {0xc0005aae10, 0x5, 0x5})
	/home/runner/go/pkg/mod/github.com/urfave/cli/v2@v2.25.7/command.go:267 +0xbe5
github.com/urfave/cli/v2.(*Command).Run(0x46e0820, 0xc00081bfc0, {0xc000778600, 0x6, 0x6})
	/home/runner/go/pkg/mod/github.com/urfave/cli/v2@v2.25.7/command.go:267 +0xbe5
github.com/urfave/cli/v2.(*Command).Run(0xc00058e2c0, 0xc00076b800, {0xc000050070, 0x7, 0x7})
	/home/runner/go/pkg/mod/github.com/urfave/cli/v2@v2.25.7/command.go:267 +0xbe5
github.com/urfave/cli/v2.(*App).RunContext(0xc0000361e0, {0x31ada70?, 0x4735a60}, {0xc000050070, 0x7, 0x7})
	/home/runner/go/pkg/mod/github.com/urfave/cli/v2@v2.25.7/app.go:332 +0x5b7
github.com/urfave/cli/v2.(*App).Run(0x60?, {0xc000050070?, 0xc0000aa058?, 0x45b3338?})
	/home/runner/go/pkg/mod/github.com/urfave/cli/v2@v2.25.7/app.go:309 +0x2f
main.main()
	/home/runner/work/docker-builds/docker-builds/cli/cmd/temporal/main.go:17 +0x2d

health check:

$ temporal operator cluster health --address temporal-frontend.temporal.svc:7233
temporal.api.workflowservice.v1.WorkflowService: SERVING

what i do wrong?

2 
$tctl ---ad temporal-frontend.temporal.svc --namespace default wf start --tq defaultr --wt CreateWorkflow
Started Workflow Id: f78e9f93-c029-4b43-af21-83661bf81234, run Id: 98f3c37c-2409-47f6-82ab-c49eb5cf1343

image
image1695×807 69.4 KB

3

up. local docker works good

4

I’m really the only person who installed temporal in cuber via helm, only on mysql driver and have a similar error?

5

I haven’t figured out the reason for this behavior, but I found an alternative to helm chart – temporal operator
https://temporal-operator.pages.dev/
interesting experience