I am writing a monitoring program in Go and have been looking into numerous orchestration design patterns and stumbled upon Temporal. I am looking to determine if Temporal is a good approach for my program and would love some high-level guidance on roughly how to implement the Temporal portion of it before deciding to dive in full-force.
For the sake of simplicity, I will only refer to HTTP status monitoring for this post, but there are numerous types of monitoring I have in place, all defined within the same table in postgres using an HSTORE for the specific parameters.
My application is intended to be as follows:
- Define the parameters for a Monitor (e.g. url, expected status code, timeout, etc) in a PSQL database table
- Run each monitor exactly once regardless of how many instances of my program are running
- Each monitor is extremely lightweight and should run in a single goroutine,
- Monitors are infinite loops. They should run continuously until the go program terminates OR it is requested for it to be cancelled (which can happen at any time). Each monitoring action (e.g. HTTP request) only occurs once per minute.
- Upon a status change, the database is updated and a NOTIFY trigger is called in the database
- Many hundreds or even thousands of monitors should be able to run on any given instance of my program running
If I do a standalone application, all of the pieces work absolutely fine, but the issue arises with scalability. I need to be able to run multiple instances of my program for high availability so that if anything goes down, other instances will take over the monitoring responsibilities.
How would I best approach running thousands of concurrent, independent, unitary, lightweight, infinitely-looping, and cancel-able goroutines using Temporal?
Thank you for any information or recommendations!