Temporal was running just fine with 2 schedules and around 19k workflow executions.
During the weekend suddenly retrieving information about a schedule fails.
We can reproduce it running the admin-tools using:
temporal --namespace schedule describe --schedule-id <ID_OF-SCHEDULE>
which results in an error: context deadline exceeded.
or by going to temporal-web, go to schedules (which works) and select any of the 2 existing schedules.
Also, in the web interface we can navigate to workflows and see all details there.
no clue on
Extra information:
After execution, these metrics increase:
service_error_with_type{container=“temporal-frontend”, endpoint=“metrics”, error_type=“serviceerror_DeadlineExceeded”, exported_namespace=“twintos”, instance=“10.2.1.191:9090”, job=“temporal-frontend-headless”, namespace=“cluster-temporal”, operation=“DescribeSchedule”, pod=“temporal-frontend-7487cb4889-z4jqd”, service=“temporal-frontend-headless”, service_name=“frontend”, type=“frontend”}
service_error_with_type{container=“temporal-history”, endpoint=“metrics”, error_type=“context_DeadlineExceeded”, exported_namespace=“twintos”, instance=“10.2.2.149:9090”, job=“temporal-history-headless”, namespace=“cluster-temporal”, operation=“QueryWorkflow”, pod=“temporal-history-5979f45bd5-ngxh2”, service=“temporal-history-headless”, service_name=“history”, type=“history”}
GRPC_HEALTH_PROBE_VERSION=v0.4.37
wget -qO grpc_health_probe "https://github.com/grpc-ecosystem/grpc-health-probe/releases/download/${GRPC_HEALTH_PROBE_VERSION}/grpc_health_probe-linux-amd64"
chmod +x ./grpc_health_probe
temporal operator cluster health --address temporal-frontend:7233
./grpc_health_probe -addr=temporal-frontend:7233 -service=temporal.api.workflowservice.v1.WorkflowService
./grpc_health_probe -addr=temporal-matching-headless:7235 -service=temporal.api.workflowservice.v1.MatchingService
./grpc_health_probe -addr=temporal-history-headless:7234 -service=temporal.api.workflowservice.v1.HistoryService
Inside the admin tools pod gives:
SERVING
status: SERVING
status: SERVING
status: SERVING
persistence psql schema version: 1.16
visibility psql schema version: 1.9
temporalio/server:1.27.2
temporalio/ui:2.37.1
deploying temporal using helm chart
version: “0.62.0”
repository: https://go.temporal.io/helm-charts
Any help much appreciated