Cannot See Workflows while Running Temporal on AWS ECS with AWS Elasticsearch

Community Support
, ,
1

Hi,

I deployed Temporal along with Temporal Web on AWS ECS. I replicated the configuration using this Docker Compose file: docker-compose/docker-compose.yml at main · temporalio/docker-compose · GitHub

I used this configuration file: docker-compose/development_es.yaml at main · temporalio/docker-compose · GitHub

For Elasticsearch, I used a single node password enabled AWS Elasticsearch domain having ES v7.10.

For Postgres, I used a single instance password enabled AWS RDS database having Postgres v13.3.

My task definition for Temporal is as follows:

{
    "ipcMode": null,
    "executionRoleArn": "arn:aws:iam::999999999999:role/dev-ecs-task-execution-role",
    "containerDefinitions": [
      {
        "dnsSearchDomains": null,
        "environmentFiles": null,
        "logConfiguration": {
          "logDriver": "awslogs",
          "secretOptions": null,
          "options": {
            "awslogs-group": "awslogs-dev",
            "awslogs-region": "eu-central-1",
            "awslogs-create-group": "true",
            "awslogs-stream-prefix": "awslogs-temporal"
          }
        },
        "entryPoint": [],
        "portMappings": [
          {
            "hostPort": 7233,
            "protocol": "tcp",
            "containerPort": 7233
          }
        ],
        "command": [],
        "linuxParameters": null,
        "cpu": 32,
        "environment": [
          {
            "name": "DB",
            "value": "postgresql"
          },
          {
            "name": "DB_PORT",
            "value": "5432"
          },
          {
            "name": "DYNAMIC_CONFIG_FILE_PATH",
            "value": "/temporal/development_es.yaml"
          },
          {
            "name": "ENABLE_ES",
            "value": "true"
          },
          {
            "name": "ES_PORT",
            "value": "443"
          },
          {
            "name": "ES_SCHEME",
            "value": "https"
          },
          {
            "name": "ES_VERSION",
            "value": "v7"
          },
          {
            "name": "LOG_LEVEL",
            "value": "info"
          },
          {
            "name": "SERVICES",
            "value": "history,matching,frontend,worker"
          }
        ],
        "resourceRequirements": null,
        "ulimits": [],
        "dnsServers": null,
        "mountPoints": [
          {
            "readOnly": true,
            "containerPath": "/temporal",
            "sourceVolume": "configVolume"
          }
        ],
        "workingDirectory": null,
        "secrets": [
          {
            "valueFrom": "arn:aws:ssm:eu-central-1:999999999999:parameter/docker/dev/temporal_POSTGRES_USERNAME",
            "name": "POSTGRES_USER"
          },
          {
            "valueFrom": "arn:aws:ssm:eu-central-1:999999999999:parameter/docker/dev/temporal_POSTGRES_PASSWORD",
            "name": "POSTGRES_PWD"
          },
          {
            "valueFrom": "arn:aws:ssm:eu-central-1:999999999999:parameter/docker/dev/temporal_POSTGRES_HOST",
            "name": "POSTGRES_SEEDS"
          },
          {
            "valueFrom": "arn:aws:ssm:eu-central-1:999999999999:parameter/docker/dev/temporal_ES_HOST",
            "name": "ES_SEEDS"
          },
          {
            "valueFrom": "arn:aws:ssm:eu-central-1:999999999999:parameter/docker/dev/temporal_ES_USERNAME",
            "name": "ES_USER"
          },
          {
            "valueFrom": "arn:aws:ssm:eu-central-1:999999999999:parameter/docker/dev/temporal_ES_PASSWORD",
            "name": "ES_PWD"
          }
        ],
        "dockerSecurityOptions": null,
        "memory": 256,
        "memoryReservation": 128,
        "volumesFrom": [],
        "stopTimeout": 60,
        "image": "temporalio/auto-setup:1.15.0",
        "startTimeout": null,
        "firelensConfiguration": null,
        "dependsOn": null,
        "disableNetworking": null,
        "interactive": null,
        "healthCheck": null,
        "essential": true,
        "links": null,
        "hostname": null,
        "extraHosts": null,
        "pseudoTerminal": null,
        "user": null,
        "readonlyRootFilesystem": null,
        "dockerLabels": null,
        "systemControls": null,
        "privileged": null,
        "name": "temporal"
      }
    ],
    "placementConstraints": [],
    "memory": null,
    "taskRoleArn": null,
    "compatibilities": [
      "EC2"
    ],
    "taskDefinitionArn": "arn:aws:ecs:eu-central-1:999999999999:task-definition/temporal-dev:36",
    "family": "temporal-dev",
    "requiresAttributes": [
      {
        "targetId": null,
        "targetType": null,
        "value": null,
        "name": "com.amazonaws.ecs.capability.logging-driver.awslogs"
      },
      {
        "targetId": null,
        "targetType": null,
        "value": null,
        "name": "ecs.capability.execution-role-awslogs"
      },
      {
        "targetId": null,
        "targetType": null,
        "value": null,
        "name": "ecs.capability.efsAuth"
      },
      {
        "targetId": null,
        "targetType": null,
        "value": null,
        "name": "com.amazonaws.ecs.capability.docker-remote-api.1.19"
      },
      {
        "targetId": null,
        "targetType": null,
        "value": null,
        "name": "ecs.capability.efs"
      },
      {
        "targetId": null,
        "targetType": null,
        "value": null,
        "name": "com.amazonaws.ecs.capability.docker-remote-api.1.21"
      },
      {
        "targetId": null,
        "targetType": null,
        "value": null,
        "name": "ecs.capability.container-ordering"
      },
      {
        "targetId": null,
        "targetType": null,
        "value": null,
        "name": "com.amazonaws.ecs.capability.docker-remote-api.1.25"
      },
      {
        "targetId": null,
        "targetType": null,
        "value": null,
        "name": "ecs.capability.secrets.ssm.environment-variables"
      },
      {
        "targetId": null,
        "targetType": null,
        "value": null,
        "name": "com.amazonaws.ecs.capability.docker-remote-api.1.18"
      },
      {
        "targetId": null,
        "targetType": null,
        "value": null,
        "name": "ecs.capability.task-eni"
      },
      {
        "targetId": null,
        "targetType": null,
        "value": null,
        "name": "com.amazonaws.ecs.capability.docker-remote-api.1.29"
      }
    ],
    "pidMode": null,
    "requiresCompatibilities": [
      "EC2"
    ],
    "networkMode": "awsvpc",
    "runtimePlatform": null,
    "cpu": null,
    "revision": 36,
    "status": "ACTIVE",
    "inferenceAccelerators": null,
    "proxyConfiguration": null,
    "volumes": [
      {
        "fsxWindowsFileServerVolumeConfiguration": null,
        "efsVolumeConfiguration": {
          "transitEncryptionPort": null,
          "fileSystemId": "fs-99999999",
          "authorizationConfig": null,
          "transitEncryption": null,
          "rootDirectory": "temporal"
        },
        "name": "configVolume",
        "host": null,
        "dockerVolumeConfiguration": null
      }
    ]
}

My task definition for Temporal Web is as follows:

{
  "ipcMode": null,
  "executionRoleArn": "arn:aws:iam::999999999999:role/dev-ecs-task-execution-role",
  "containerDefinitions": [
    {
      "dnsSearchDomains": null,
      "environmentFiles": null,
      "logConfiguration": {
        "logDriver": "awslogs",
        "secretOptions": null,
        "options": {
          "awslogs-group": "awslogs-dev",
          "awslogs-region": "eu-central-1",
          "awslogs-create-group": "true",
          "awslogs-stream-prefix": "awslogs-temporal-web"
        }
      },
      "entryPoint": [],
      "portMappings": [
        {
          "hostPort": 8088,
          "protocol": "tcp",
          "containerPort": 8088
        }
      ],
      "command": [],
      "linuxParameters": null,
      "cpu": 32,
      "environment": [
        {
          "name": "TEMPORAL_GRPC_ENDPOINT",
          "value": "temporal.dev-local:7233"
        },
        {
          "name": "TEMPORAL_PERMIT_WRITE_API",
          "value": "true"
        }
      ],
      "resourceRequirements": null,
      "ulimits": [],
      "dnsServers": null,
      "mountPoints": [],
      "workingDirectory": null,
      "secrets": [],
      "dockerSecurityOptions": null,
      "memory": 256,
      "memoryReservation": 128,
      "volumesFrom": [],
      "stopTimeout": 60,
      "image": "temporalio/web:1.13.0",
      "startTimeout": null,
      "firelensConfiguration": null,
      "dependsOn": null,
      "disableNetworking": null,
      "interactive": null,
      "healthCheck": null,
      "essential": true,
      "links": null,
      "hostname": null,
      "extraHosts": null,
      "pseudoTerminal": null,
      "user": null,
      "readonlyRootFilesystem": null,
      "dockerLabels": null,
      "systemControls": null,
      "privileged": null,
      "name": "temporal-web"
    }
  ],
  "placementConstraints": [],
  "memory": null,
  "taskRoleArn": null,
  "compatibilities": [
    "EC2"
  ],
  "taskDefinitionArn": "arn:aws:ecs:eu-central-1:999999999999:task-definition/temporal-web-dev:7",
  "family": "temporal-web-dev",
  "requiresAttributes": [
    {
      "targetId": null,
      "targetType": null,
      "value": null,
      "name": "com.amazonaws.ecs.capability.logging-driver.awslogs"
    },
    {
      "targetId": null,
      "targetType": null,
      "value": null,
      "name": "ecs.capability.execution-role-awslogs"
    },
    {
      "targetId": null,
      "targetType": null,
      "value": null,
      "name": "com.amazonaws.ecs.capability.docker-remote-api.1.19"
    },
    {
      "targetId": null,
      "targetType": null,
      "value": null,
      "name": "com.amazonaws.ecs.capability.docker-remote-api.1.21"
    },
    {
      "targetId": null,
      "targetType": null,
      "value": null,
      "name": "ecs.capability.container-ordering"
    },
    {
      "targetId": null,
      "targetType": null,
      "value": null,
      "name": "com.amazonaws.ecs.capability.docker-remote-api.1.18"
    },
    {
      "targetId": null,
      "targetType": null,
      "value": null,
      "name": "ecs.capability.task-eni"
    },
    {
      "targetId": null,
      "targetType": null,
      "value": null,
      "name": "com.amazonaws.ecs.capability.docker-remote-api.1.29"
    }
  ],
  "pidMode": null,
  "requiresCompatibilities": [
    "EC2"
  ],
  "networkMode": "awsvpc",
  "runtimePlatform": null,
  "cpu": null,
  "revision": 7,
  "status": "ACTIVE",
  "inferenceAccelerators": null,
  "proxyConfiguration": null,
  "volumes": []
}

I’m able to run those 2 containers successfully and temporal connects to and writes data on Postgres and Elasticsearch successfully.

The issue is, I’m not able to see any workflows on the Temporal Web UI neither in my created namespace nor under the default namespace but I can see data on Elasticsearch.

image
image1920×693 123 KB

I’m not able to find any error logs, I only see info logs under AWS CloudWatch for both Temporal and Temporal Web UI.

What did I miss? What’s the actual issue? Is there any tutorial or guide available to make it work on AWS ECS with AWS RDS Postgres and AWS Elasticsearch using dynamic configuration?

Note: When I run it locally using docker-compose, it works.

2

image
image3562×694 96.8 KB

No data on Temporal Web UI.

3

The data you see in Elasticsearch are the Temporal system workflows, they are accessible under the
temporal-system namespace.
You could for example see that with:
http://localhost:8088/namespaces/temporal-system/workflows?range=last-30-days&status=ALL

I don’t see any user specific workflows there from the image provided.

Could you show your code where you start your workflows? Maybe we can look at that first. Are you getting any client side exceptions when creating workers/starting workflows in client?

1 Like
4

Hi @tihomir
There is no client-side error and workflow will be done successfully. We can run temporal with postgresql and es on our local without any issue and we can see the records on temporal-web. But when we install it on AWS, temporal-web doesn’t show any record but workflows completed successfully.

1 Like
5

Hi @tihomir and @hamed_Yousefi1
I checked the temporal-system namespace here and I was able to see the workflows: http://localhost:8088/namespaces/temporal-system/workflows?range=last-30-days&status=ALL

image
image3002×912 236 KB

I can also see workflows in our namespaces.

I did the following things:

  • I enabled authentication (username and password) on Elasticsearch for successful connectivity.
  • I deleted the temporal index from Elasticsearch.
  • I upgraded the temporal version to v1.15.0 by using the following image: temporalio/auto-setup:1.15.0
    • It was 1.13.1 before.
  • I deleted the temporal databases from Postgres.
  • I restarted the temporal ECS services and the services that create namespaces.

It worked.

1 Like