DB schema migration steps when upgrading Temporal via Helm chart with standalone Postgres

Community Support
,
1

Hi Temporal Support,

New to temporal, I’m running a dev self-hosted Temporal cluster and need clarification on the correct upgrade procedure for our setup. Here are our specifics:

Environment:

  • Helm chart: temporalio/temporal 0.73.2 (from ArtifactHub)
  • Database: AWS RDS PostgreSQL (external — not a bundled sub-chart)
  • Current values.yaml has: schema.setup.enabled: true and schema.update.enabled: true

What we understand so far:

  1. The Helm chart’s schema.update job runs as a pre-upgrade Helm hook
  2. Temporal requires sequential minor version upgrades — skipping versions is not supported
  3. The DB schema must be migrated BEFORE the new server version is deployed

Our questions:

  1. For our setup, is it safe to rely on schema.update: true in values.yaml to run migrations automatically on each sync? Or is this known to be unreliable?

  2. If we should run migrations manually, the approach we are planning is:
    a. Before syncing, run a one-off job using the new version’s admin-tools image
    b. Let the Job run temporal-sql-tool update-schema against our RDS for both the default and visibility stores
    c. Once the Job completes successfully, deploy the new server version
    Is this the correct and recommended approach?

  3. If we are multiple minor versions behind (e.g. currently on 1.29.1 and targeting 1.30.0), do we need to run the migration Job and sync separately for each intermediate minor version? Or can we run all schema migrations first and then deploy the final target version in one sync?

  4. Should we set schema.update: false in values.yaml if we are managing migrations manually, to prevent the chart from attempting to run its own migration job on sync? Is this recommended setup for production using helm chart?

Thank you for your help, we want to make sure we have the right process before deploying to staging environment

2

The Helm chart’s schema.update job runs as a pre-upgrade Helm hook

Are you sure? To my understanding it became a pre-upgrade hook after helm chart 1.x version. With your version i think there is no guarantee that the schema update job finishes before upgrade starts.
So what you might run into is, while schema job finishes,
your service hosts might start up, but they will keep crash-looping because server checks schema version and will see schema version being lower, and wont start until schema job finishes.
Just fyi if helps.

For our setup, is it safe to rely on schema.update: true in values.yaml to run migrations automatically on each sync? Or is this known to be unreliable?

It should be ok as not destructive (does not run with --overwrite so wont drop any tables). The only think that would potentially worry about is the job name is keyed to “.Release.Revision”
and once it ran for specific combination its immutable meaning you might need to, idk, have to manage this yourself if needed

If we should run migrations manually, the approach we are planning is:

Yeah, see maybe if also playbook here helps for the schema update process.

If we are multiple minor versions behind (e.g. currently on 1.29.1 and targeting 1.30.0), do we need to run the migration

The schema tools themselves can do it in one pass, but the binary itself has the requirement of one minor at a time. Thus its best to do one minor at time also for schema updates as well

Should we set schema.update: false in values.yaml if we are managing migrations manually,

I believe yes. You should probably also set schema.setup.enabled, and schema.update.enabled and once you updated your schema, chema.createDatabase.enabled all to false