Do we have any way to restrict specific group of ppl in temporal UI

Hi Team,
We have enabled sso in temporal but we want to restrict few ppl. How to achieve it?