Archival and docker-compose

Pedro_Almeida · December 3, 2020, 2:40pm

Hello,

We currently use temporal server with docker-compose. Looking into the Archival Docs, seems it is not supported when running with docker-compose?

Just wondering if there is a workaround before thinking in changing our setup.

Also, i believe the retention period is set when creating the namespace correct? Is there a way to change the retention period afterwards? If yes, how?

Thanks!

ryland · December 3, 2020, 5:16pm

Hey! First off I want to make sure that it’s clear that archival and retention period are separate things. While it might make sense to run with retention on your local docker setup, it’s very unlikely that running locally with archival makes sense. Can I ask what your use case is for archival when running with docker-compose?

In general it isn’t straightforward to change config such as archival with our docker-compose setup. That being said, depending on your use case it might make sense.

Pedro_Almeida · December 4, 2020, 9:56am

Hello @ryland,

So far, we only used temporal with docker-compose in all envs. I understand this is not the recommended way for production, but for now we have to stick with docker-compose. We will be looking into the helm-charts when the solution is ready.

If not easy use Archive with docker-compose, we’ll have to deal with it when we switch to a non docker-compose setup. Goal here was to see how it works, test it and check the features we can use within our reality.

I’ve already managed to change the Retention Period in the namespace with the tctl tool.

Thanks for the feedback!

ryland · December 4, 2020, 5:15pm

Hey, completely understand. Let me add @derek to this thread who probably knows best how to change config with our existing docker-compose setup.

Ry

derek · December 4, 2020, 6:42pm

Hi @Pedro_Almeida

We use dockerize to generate a config from that template which the container uses at startup:

github.com

temporalio/temporal/blob/master/docker/start-temporal.sh

#!/bin/bash

set -ex

dockerize -template /etc/temporal/config/config_template.yaml:/etc/temporal/config/docker.yaml

# Convert semicolon (or comma for backward compatibility) separated string (i.e. "history:matching")
# to valid flags list (i.e. "--service=history --service=matching").
IFS=':,' read -ra SERVICE_LIST <<< "$SERVICES"
SERVICE_ARGS=$(printf -- "--service=%s " "${SERVICE_LIST[@]}")

exec temporal-server --root $TEMPORAL_HOME --env docker start $SERVICE_ARGS

It’s possible to do things we haven’t exposed via docker-compose if you edit the config-template.yaml before you start your containers:

github.com

temporalio/temporal/blob/master/docker/config_template.yaml#L243


        fileMode: "0666"
        dirMode: "0766"
  visibility:
    state: "enabled"
    enableRead: true
    provider:
      filestore:
        fileMode: "0666"
        dirMode: "0766"
namespaceDefaults:
  archival:
    history:
      state: "disabled"
      URI: "file:///tmp/temporal_archival/development"
    visibility:
      state: "disabled"
      URI: "file:///tmp/temporal_vis_archival/development"
kafka:
    tls:

The provider in the archival section and the namespaceDefaults should be adjusted as needed.

You will also potentially need to add access to credentials depending on the provider used or mount a path on your host for the container to write to if using the filesystem.

Good luck! If you test this out, we’d love to hear how it goes!