Trying to downgrade from 1.22.0 to 1.21.5

As a part of testing temporal upgrade , I had managed to upgrade from 1.21.3 to temporal latest. I tried to downgrade it to 1.21.3 and the least i can downgrade is till 1.22.0. I had taken a db backup and still the upgrade fails only for temporal-history giving an error :

panic: unable to find queue category by queye category ID: 7

goroutine 118 [running]:

go.temporal.io/server/service/history/shard.storeShardInfoCompatibilityCheckWithoutReplication(0xc000612af0)

/home/builder/temporal/service/history/shard/compatibility.go:168 +0x45c

Kindly help with this