Hi everyone.
I have recently discovered Temporal, and while learning Go, its a wonderful tool.
I’d like to open a discussion regarding using Temporal to manage data import process. to be honest, I still do not know the correct approach to this, I just know, that I’d like to use current functionality of Temporal to help me with the task.
Main goal is to import product data, from file/rest api source, to MongoDB instance. We are talking about thousands to tens of thousands records in single import run. (domain of data is e-commerce, with highly customizable products )
Of course, getting just the plain json to Mongo poses no issues, trouble comes with data processing, e.g. mapping. There are certain attributes in the data, which need to be mapped before shown to the end-user. one of such example could be color, or product category. source data set may contain thousands of source values, I need to map them to, let’s say max 50 values. I can have several, (up to 20) such fields, where mapping has to be resolved.
one option ( imho worst one) is to cancel whole import process, unless I have all mapping values present in my mapping table.
one of better options would be to have import workflow (constantly?) running, and if I have still some unmapped values, I will notify the user. once the user changes the mapping table, makes some improvements, he can restart the task, which should result in successful mapping of more and more records, he should also have an option to discard some mapping values, and not import records containing those values.
trouble I have here with combining this with Temporal way of life is… that usually I’d make changes in the database for all these steps. I can make a select, see how many records I have in import queue, how many of them are resolved for particular mapping, how many have to be yet resolved, but all data related to this is stored in the MongoDB, which is the primary DB for my solution.
I do know, that activity can well access the database, and that there are even options to control, which activities shall be executed next.
Where I see strong side of Temporal is the option of really managing the whole import process, and after all mappings ( and checks, and whatever) have been finished/resolved, I can continue the import process.
What I want to avoid is some database inconsistency in Mongo, caused by incorrect setup of workflow / activities.
thanks in advance for all the suggestions
Kind regards
Marek