How to automatically trim history files?

Community Support
, ,
1

There is a ‘development’ folder that is full of history files and they grow very fast (i.e. 10s of GBs per day).
Is there a way to limit the growth (since we only have 50GB of hard drive), or do not store these file (I’m not sure what they are used for).

bash-5.0# pwd
/tmp/cadence_archival/development

bash-5.0# ls | head
11093309881475487913100002220303248752326812576525522501232_-24.history
11093309881475487913100015175505633736545199696002987527410_-24.history
11093309881475487913100020188383221999794947647897645539603_-24.history
11093309881475487913100033180645436958768900643448851036185_-24.history
110933098814754879131000353956197481610516632589115003061383_-24.history
11093309881475487913100074114266917810093847313950235056999_-24.history
11093309881475487913100074755737666488483194014166826314495_-24.history
11093309881475487913100111717864956357815612909120989550088_-24.history
110933098814754879131001208989330711234415072881271952789432_-24.history
1109330988147548791310015817981932280241139625968815260707_-24.history

bash-5.0# ls | wc
719615 719615 51659495

1 Like
3

Looks like you are running Temporal with archival turned on. There is a file based implementation for archival provided for open source as an example. For real production environment we expect to use some blob store like s3 for archival data. If archival is turned on for the cluster, workflow execution history is never deleted but instead archived to your blob store after retention interval.
If you do not need this feature you can turn it off through config.

1 Like
4

Can you link me to a doc on how to turn off archival? We don’t need archival since our workflows are short lived and we don’t need history past 3 days (our retention)

1 Like
5

Here is the current Archival documentation: https://docs.temporal.io/docs/learn-archival

The “Configuring” section should help you make the changes you want, but feel free to provide feedback, I am in the process of updating this information.

6

Here is a config snippet to disable archival:

archival:
  history:
    state: "disabled"
  visibility:
    state: "disabled"

namespaceDefaults:
  archival:
    history:
      state: "disabled"
    visibility:
      state: "disabled"
1 Like
7

Awesome, thank you for that!

8

Hi @samar, Is there a way to control the archival flag on the standard docker image? I mean can we pass some parameter to docker run command to enable/disable archival?

We use the standard docker image and directly run it for the server. Can we override archival settings similar to mysql and other settings?

9

@Anand_Sarda,
We don’t have a mechanism for overriding archival config section. Here is the task tracking this request. You have to override the entire config_template.yaml to achieve this currently.