We are looking for a replacement for Airflow. The way we use Airflow is to manage containerized applications, specifically Docker images that we host on JFrog. So my question is, before I spend way too much time PoCing Temporal, does Temporal support workflows where you pull a Docker image from an artifactory, run it (with given arguments), and so on? I would need to be able to spin up pods for the containers to achieve concurrency, etc.
If this is possible, can someone kindly point me to some documentation about how I might go and create a workflow that does it?
Temporal workflow is as dynamic as necessary as opposed to fixed Airflow DAGs. So having an activity that pulls a docker image and then the next activity to be implemented by that docker image is absolutely fine.
I don’t think we have specific samples. If you have any concrete questions, we can help.
The basic idea is that some activities are always running in the cluster, and these activities use K8s APIs to start new jobs. These new jobs can host more activities or even workflows. Temporal supports routing requests to specific types of jobs or even to individual pods.