We had a surprise issue yesterday where someone imported a package to use a few string constants from it in a workflow. The usage looked fine as they were just strings, and surely those are deterministic. However, the package they were imported from, also contained @paralleldrive/cuid2, and caused workflows to suddenly fail.
I was surprised that the bundle built and that everything deployed just fine, but then we started having workflow failures.
What is the best means of protecting ourselves from this kind of issue? I’m looking for some kind of automated tool like eslint, or something in our build step, where we can check to avoid this kind of mishap in the future.
You’ve hit a known limitation of the current bundler.
@paralleldrive/cuid2 depends on @noble/hashes , which provides two implementations: one using a require on node:crypto, obviously for use in NodeJS, and another implementation that access the globalThis.crypto variable, which would be available in browsers. Directives in @noble/hashes’s package.json is used to automatically determine which of those two implementations to load.
Our bundler will use the “browser” implementation. I know that’s surprising, but really, neither of those environment is appropriate for our use case, as the Workflow sandbox doesn’t provide all the APIs that one would expect in either of these environments. The “browser” environment is simply closer to what the Workflow sandbox is than the “node” environment.
So @noble/hashes access the globalThis.crypto global variable. There’s no require/import involved, hence our bundler is failing to identify the problematic use of that non-deterministic API. And Webpack is not complaining either, because it believes that globalThis.crypto is supported by the target environment.
The solution? Add @paralleldrive/cuid2 to the bundler’s ignoredModules. Also make sure that nothing actually tries to use that module at runtime.
Regarding ESLint, it would technically be possible to have some custom lint rules to catch this, but that would require deep analysis of the dependency tree, which would make your linter very slow to execute. ESLint is more appropriate to detect shallow errors in your code, not issues in dependencies.
Thank you so much for the detailed explanation. I think adding that particular module to the ignoredModules is a great first step. It does still leave me wondering if there’s some other library that might be accidentally added that could cause the same issue, and how this can be defended against more broadly.
I think we’ve come to a solution that will work for us. Here’s how it works:
We create a test (we use vitest, but others should work fine) that runs a workflow of a type that does not exist
We wait for the workflow to run in the test workflow environment and check its history
We convert the history to a string and we look for the error indicating the workflow wasn’t found
If that error is not present the most likely reason is that something went wrong trying to load up the workflows/activities
Let me know if you think this misses anything, but it seems like a fairly complete solution that should fail our tests and prevent this from ever going out again. Wanted to share it in the event that it’s of use to someone else.
import { TestWorkflowEnvironment } from "@temporalio/testing";
import { type Worker, Runtime, DefaultLogger } from "@temporalio/worker";
import { teardown } from "temporal-test-helpers";
import { createWorker } from "./worker-builder.js";
let testEnv: TestWorkflowEnvironment;
let worker: Worker;
describe("worker-builder", () => {
beforeAll(async () => {
Runtime.install({ logger: new DefaultLogger("WARN") });
testEnv = await TestWorkflowEnvironment.createTimeSkipping();
worker = await createWorker({
taskQueue: "my-queue",
connectionOverride: testEnv.nativeConnection,
});
worker.run();
return async () => {
await teardown(worker, testEnv);
};
});
it("should fail for the right reason", async () => {
const handle = await testEnv.client.workflow.start("fake-workflow", {
workflowId: "workflow-does-not-exist",
taskQueue: "my-queue",
workflowExecutionTimeout: "3s",
});
try {
await handle.result();
} catch (error) {
console.log("ERROR");
}
const history = await handle.fetchHistory();
const historyString = JSON.stringify(history, null, 2);
const correctFailureMessageIsPresent = historyString.includes(
"TypeError: Failed to initialize workflow of type 'fake-workflow': no such function is exported by the workflow bundle",
);
if (!correctFailureMessageIsPresent) {
console.error(
"If you're getting a failure on this test, it probably means that " +
"something was imported into a workflow that will fail when the worker " +
"is deployed. Please inspect the actual error message. In this test we " +
"expect an error because we're passing in workflow type that doesn't " +
"exist, and that should always be the case.",
);
}
expect(correctFailureMessageIsPresent).toBe(true);
}, 15000);
});