If you have multiple workers polling the same task queue, it is random which one is going to pick up the activity task.
If a worker receives a task for an activity type that it doesn’t know about it rejects it by failing with “Activity type is not registered with the worker” exception. Server then reschedules this task and this will happen until a worker that has the activity registered picks it up. So yes, you might experience this until all your workers have all activities registered.
Are the multiple workers that poll on this task queue in different processes?
One possible solution is to use a separate task queue for each worker type. With this only activities that the worker supports will be dispatched to it. You would set task queue in your Activity options for each activity type, for example:
.setTaskQueue("<taskQueueForMyActivity>")
or set per-activity options via WorkflowImplementationOptions.