I was in the process of replying along the lines you have found. You might also find it useful to have a jD menu item My Downloads set with a Registered view Access. That will list all those Dowloads which have the relevant user set as the 'creator' of the Download irrespective of which Category the Download is in. You can set this in the Additional Info tab when a Download is created/edited.Also assuming your public info has a common top level Category then using the jD menu Item List a Single Category, which automatically includes the sub cats, is helpful.
Yes the present arrangement is 'interesting' with the Joomla! permissions system.
We are currently working on a new major release that amongst many other enhancements specifically allows setting a Single User Access. It is one of the things I am about to start testing! And the documenting. Please see attached pics.
Your next question is of course When. Not certain so all I can say is it is our highest priority - basically it is like the length of a pieece of string: namely twice as long as half of it!
More seriously we will be looking for beta testers so if you are interested please let us know.