you can use the joomla groups: all, registered and specials,
Alternate, you can define your own groups in jD groups manager.
When you edit a category, you can select this for category access (see pics).
So i understand not your problem.
I am a bit confused (and new to jDownloads). I have installed jDownloads, and I can restrict access to either registered or special users. If I select special users, I get the same access as I get for registered users. What is the difference supposed to be?
Members from specially groups can allways also seen categories with registered access (Special members are also registered but have a higher level).
Manager and administrators are special members (as sample).