TLDR: Don’t use new Date() in Workflow code. Use Workflow.currentTimeMillis() instead. To get a Date object, use new Date(Workflow.currentTimeMillis()).
In Java SDK, new Date() returns the true, current date at the moment it is called in this Workflow Execution on this Worker. This is almost never what you want in a Workflow, since it is fundamentally non-deterministic, as it will return a different value if the Workflow is replayed at a later time.
Now, its true that your Workflow’s logic does not depend on the value of that date; it might therefore appears to be safe to obtain real time in this case. However, as you experienced, you may still get tricked if a Workflow Execution gets evicted from the local Workflow cache, then gets replayed at a later time. In that case, the activity call will replay almost instantly (ie. the activity method will immediately return with the result stored in the Workflow History), even though the activity itself originally took 2 minutes.
Workflow.currentTimeMillis() returns the start time of the current Workflow Task, which is part of the Workflow History itself, which guarantees that it is deterministic.