Hi,
I think in the moment is a selection as you described not possible.
Currently are always all categories (where the user has the permission to see it) visible.
But maybe could it be a new option in a coming release. So I will move your request to the Suggestions.
By the way, you can disable the All Categories view in the options (TAB Frontend).