Web-UI and tctl won't list workflows

Community Support
, ,
1

Hi everyone,
We are having an issue with not getting any workflows to show up when running tctl wf la
We have activated archival and visibility records for completed status are getting into the archival bucket but there are no records in ElasticSearch for running workflows.
tctl wf show [workflowId] works fine though.
It seems like nothing is being written to ElasticSearch either.
A query to /_cat/indices?v on Elastic shows that docs.count and docs.deleted are both 0 for the temporal_visibility_v1_dev index.
We are currently running version 1.17.1 of temporal-server.
Does anyone know what could be wrong or have any good pointer on how to troubleshoot this?

ElasticSearch config looks like this:

    numHistoryShards: 4
    defaultStore: default
    visibilityStore: visibility
    advancedVisibilityStore: es-visibility
    datastores:
        default:
            sql:
                pluginName: "postgres"
                databaseName: "temporal"
                connectAddr: "***:5432"
                connectProtocol: "tcp"
                user: "***"
                password: "***"
                maxConns: 20
                maxIdleConns: 20
                maxConnLifetime: 1h
                tls:
                    enabled: true
                    caFile:
                    certFile:
                    keyFile:
                    enableHostVerification: false
                    serverName:
        visibility:
            sql:
                pluginName: "postgres"
                databaseName: "temporal_visibility"
                connectAddr: "***:5432"
                connectProtocol: "tcp"
                user: "***"
                password: "***"
                maxConns: 10
                maxIdleConns: 10
                maxConnLifetime: 1h
                tls:
                    enabled: true
                    caFile:
                    certFile:
                    keyFile:
                    enableHostVerification: false
                    serverName:
        es-visibility:
            elasticsearch:
                version: v7
                url:
                    scheme: http
                    host: "***:9200"
                indices:
                    visibility: "temporal_visibility_v1_dev"

… and the output of http://redacted:9200/temporal_visibility_v1_dev/_stats/docs:

$ curl http://redacted:9200/temporal_visibility_v1_dev/_stats/docs
{"_shards":{"total":1,"successful":1,"failed":0},"_all":{"primaries":{"docs":{"count":0,"deleted":0}},"total":{"docs":{"count":0,"deleted":0}}},"indices":{"temporal_visibility_v1_dev":{"uuid":"OxToD8MnS-qRDRvzeFGv6w","primaries":{"docs":{"count":0,"deleted":0}},"total":{"docs":{"count":0,"deleted":0}}}}}
3

How are you deploying the server? Did you install both the default and visibility schemas? Any ES error logs during server startup?

If you have server metrics enabled what do you get for visibility latencies:

histogram_quantile(0.95, sum(rate(task_latency_bucket{operation=~"VisibilityTask.*", service_name="history"}[1m])) by (operation, le))

1 Like
4

Hi @tihomir!

Many thanks for getting back to us on this.
We actually just found out that we had a version mismatch after upgrading to version 1.17.1.
By accident version 1.7.1 was pushed so choosing the proper version (1.17.1) solved the issue.
The good thing though is that we learned a lot about visibility that will come in handy in the future. :grin: