Local activities aren't cancelled when I cancel the CancellationScope they are being called in

Cancellation of local activities is currently not possible via Java SDK. This is something that will be added in the future.

This is because cancellation is delivered through heartbeats (see sample here) and local activities do not support heartbeating (more info in this post).

For your use case you would need to use regular activities and make sure they heartbeat to receive cancellation request.