Worker error with .mul is not a function

[rifruf]

Have a workflow that executes properly with local workers. When trying to do this in my production cluster with deployed workers, I’m getting the following error, does anyone have any ideas on what could be happening

2022-08-04T03:32:56.766Z [ERROR] Failed to activate workflow {
  runId: 'b2fd073b-42a6-40ce-b719-1468474d7f2c',
  error: TypeError: (seconds || long_1.default.UZERO).mul is not a function
      at tsToMs (/opt/app-root/src/.yarn/cache/@temporalio-internal-workflow-common-npm-1.0.0-rc.1-b66f54d7d6-4272b669aa.zip/node_modules/@temporalio/internal-workflow-common/lib/time.js:30:10)
      at /opt/app-root/src/.yarn/cache/@temporalio-worker-npm-1.0.0-rc.1-2998f0d7a7-147819cf17.zip/node_modules/@temporalio/worker/lib/worker.js:664:90
      at async /opt/app-root/src/.yarn/cache/@temporalio-worker-npm-1.0.0-rc.1-2998f0d7a7-147819cf17.zip/node_modules/@temporalio/worker/lib/tracing.js:65:20
      at async /opt/app-root/src/.yarn/cache/@temporalio-worker-npm-1.0.0-rc.1-2998f0d7a7-147819cf17.zip/node_modules/@temporalio/worker/lib/worker.js:593:28,
  workflowExists: false
}

Seems like some type of dependency that should exist doesn’t exist here?

[loren]

I recommend getting all @temporalio/* packages to the latest version:

yarn add @temporalio/client@latest @temporalio/worker@latest @temporalio/workflow@latest @temporalio/activity@latest

Here are the only breaking changes between your current version and latest:

temporalio/sdk-typescript/blob/main/CHANGELOG.md

[rifruf]

Upgraded to the latest version, still getting this in my worker. I’m also running on yarn 3.2.0, and have pnp enabled if that’s of any note.

[loren]

I think it’s due to the absence long, or having multiple versions of it. What’s the output of yarn list long?

[jwang97]

yarn list doesn’t work
but running yarn why gives me:

(production-admin 🔐)➜  temporal-workflows git:(d3084b2372) ✗ yarn why long
├─ @grpc/proto-loader@npm:0.6.9
│  └─ long@npm:4.0.0 (via npm:^4.0.0)
│
├─ @scale/root@workspace:.
│  └─ long@npm:5.2.0 (via npm:^5.2.0)
│
├─ @scale/task-pipeline@workspace:packages/task-pipeline
│  └─ long@npm:5.2.0 (via npm:5.2.0)
│
├─ @temporalio/internal-workflow-common@npm:1.0.1
│  └─ long@npm:5.2.0 (via npm:^5.2.0)
│
├─ @temporalio/proto@npm:1.0.1
│  └─ long@npm:5.2.0 (via npm:^5.2.0)
│
├─ mysql2@npm:2.3.3
│  └─ long@npm:4.0.0 (via npm:^4.0.0)
│
├─ protobufjs@npm:6.11.2
│  └─ long@npm:4.0.0 (via npm:^4.0.0)
│
├─ protobufjs@npm:6.11.3
│  └─ long@npm:4.0.0 (via npm:^4.0.0)
│
├─ protobufjs@npm:7.0.0
│  └─ long@npm:5.2.0 (via npm:^5.0.0)
│
└─ python-struct@npm:1.1.3
   └─ long@npm:4.0.0 (via npm:^4.0.0)

[jwatkins]

You mention your workflows work correctly in development… What are the differences in how you prepare your code for production vs development? i.e. ts-node or not? pnp? bundleWorkflows() being called explicitly?

[rifruf]

Ah yes I should’ve mentioned that. In both development and production, we use bundleWorkflows(). The way it’s currently run is via this code

import { bundleWorkflowCode } from '@temporalio/worker';
import { writeFile } from 'fs/promises';

async function main() {
  const { code, sourceMap } = await bundleWorkflowCode({
    workflowsPath: require.resolve('./all-workflows'),
    workflowInterceptorModules: [require.resolve('./lib/interceptors')],
  });
  await writeFile('dist/workflowBundle.js', code);
  await writeFile('dist/workflowBundle.js.map', sourceMap);
}

if (require.main === module) {
  main().catch(err => {
    // eslint-disable-next-line no-console
    console.error('Error bunding workflow code', err);
    process.exit(1);
  });
}

which runs via a yarn build script/command.

[rifruf]

Actually upon further inspection, it seems we just use pnp and do a require.resolve instead of bundling it in production. Going to try bundling it in production.

[rifruf]

Have also tried bundling, still no dice
What I’ve gone through:

• Upgrade to latest Temporal Typescript SDK
• Use pre bundled workflow instead of bundling at runtime
• Specify dependencies in package.json
  • Tried specifying long as a dependency

[loren]

Sorry about that. Still not sure what’s going on, but one idea is making sure there’s no long v4 dep, by upgrading all the packages that depend on v4.

If you can make a small repro, we can take a closer look

[rifruf]

Does long v4 not have the mul function?

[rifruf]

Also wanted to confirm that this mul function is something that should be provided by the long package? I had tried downloading the Temporal Typescript SDK locally and looking at the types, but for some reason couldn’t get the package to build and install properly, so wasn’t able to determine the types.

[rifruf]

Hmm, seems kinda impossible to have no long v4 dep actually. I ran a worker container locally, and running yarn why long gives

├─ @scale/temporal-workflows@workspace:packages/temporal-workflows
│ └─ long@npm:5.2.0 (via npm:^5.2.0)
│
├─ @temporalio/internal-workflow-common@npm:1.0.1
│ └─ long@npm:5.2.0 (via npm:^5.2.0)
│
├─ @temporalio/proto@npm:1.0.1
│ └─ long@npm:5.2.0 (via npm:^5.2.0)
│
├─ protobufjs@npm:6.11.3
│ └─ long@npm:4.0.0 (via npm:^4.0.0)
│
└─ protobufjs@npm:7.0.0
  └─ long@npm:5.2.0 (via npm:^5.0.0)

and yarn info long gives:

bash-4.4$ yarn info long
└─ long@npm:5.2.0
   └─ Version: 5.2.0

which implies we’re using the correct version.

However, if I drill down on what is using the one long package with version 4.0.0, which seems to be protobufjs@npm:6.11.3, I see that the only package I have that uses that as a dependency is proto3-json-serializer@npm:^1.0.2, and in turn, the only package that has proto3-json-serializer@npm:^1.0.2 as a dependency is "@temporalio/common@npm:^1.0.1, @temporalio/common@npm:~1.0.1":. As I’m already on the latest version of temporal, I don’t see how to ensure that long 4.0.0 isn’t present in the yarn cache whatsoever.

@loren do you have any other suggestions or information I could provide? I’ve still been unable to repro this, as it works fine locally, and only runs into this in my deployments. Thank you!

[loren]

I’ll update temporalio/common, but for now can you try pinning proto3-json-serializer to 1.0.3? That uses protobufjs@7

v4 has mul and UZERO, but @jwatkins has a theory:

it could be that tsc get somehow confused about the actual structure of the Long object… long@5.2.0 has been converted to ESM. tsc needs to injects some adaptation code to correctly work with ESM modules when running in non-ESM context. What if tsc considers it doesn’t need to do that work on long because it believes it is the 4.0.0, but at runtime, it actually load the 5.2.0 which needs those adaptations…

[rifruf]

Gotcha thanks for the quick reply, I can give that a try. What’s the proper way to pin proto3-json-serializer to 1.0.3, is it just by specifying proto3-json-serializer: "^1.0.3", in my package.json? Just curious if you had smth specific in mind.

[loren]

No, I’d think that should work

[jwatkins]

Try adding this to your root package.json file:

  "resolutions": {
    "proto3-json-serializer": "1.0.3",
    "long": "5.2.0"
  }

[jwatkins]

If that doesn’t help, would it a possibility for you test your deployment with the following addition to your .yarnrc.yml file:

pnpMode: loose

I strongly believe that PnP is somehow involved in this issue, though I can’t figure out why… This flag still enables PnP, but changes its resolution algorithm to be more compatible with that of NPM. If it works, it would at least confirm that we are indeed looking in the right direction…

[rifruf]

Thanks for all the suggestions, trying these out now!

[rifruf]

So those suggestions didn’t seem to fix the problem, but after some help, the problem seems to be the way that we build our Docker containers that run these workers in our deployments. Specifically, we have a line where we do RUN yarn workspaces focus --production @temporal-workflows, causing the container to only install dependencies, but not devDependencies. We tried running RUN yarn workspaces focus --all @scale/temporal-workflows instead, which seems to work.

What happens here is that the number passed in to this line (sdk-typescript/time.ts at main · temporalio/sdk-typescript · GitHub) is a Long when running with --all, but is just a Javascript number when running with --production.

We tried moving all the devDependencies to dependencies in our repositories package.json files, but this still didn’t fix the issue, so it led us to believe that it’s a possibility the error stems from within the temporalio typescript SDK package not having a dependency listed (possibly long listed here sdk-typescript/package.json at main · temporalio/sdk-typescript · GitHub). We also had a similar problem earlier where for some reason long had to be specified as a dependency in our root package.json, even though we were only using temporalio in a package located within our packages/ folder (and specifying long in that package’s package.json didn’t work either).

Curious if you guys have suggestions moving forward about how to best remedy this/test if this is the case? We were running into issues building/installing your typescript SDK package locally (so that we could use this) to verify.