In my evaluation of Temporal I’m on the lookout for challenges we might face when scaling it to a large-ish development team. One concern is that less-experienced or rushed developers might introduce non-deterministic code into a workflow. Proper testing would discover such code but at scale there are often overlooked test cases. I’m interested in tips for catching such code beyond discipline.
I imagine static analysis combined with a workflow naming convention would only catch the obvious. We’ll be using Go. In Java there’s a facility (i.e. SecurityContext and class loaders) for running code in a context that restricts access to specified classes or namespaces (e.g. rand, io) - even indirect access. Does Go have a similar facility? Or if something completely different might help, all ideas welcomed.
At this point, there is no bulletproof solution besides using an airtight container like V8. After NodeJs we plan to invest in WebAssembly which allows deterministic execution. So the set of languages that are supported in such a model will be much larger.
Our experience is that non deterministic errors do happen, but they usually are discovered during development and testing.