There is a way based upon using the jD menu item List My Downloads. Obviously you should make the view Access of it as Registered. Then for each Download ensure that the 'Created By' parameter is set to the person who is to have sole access to the Download.
jD also needs a 'List All Categories' menu item but obviously this should not be seen. So for that one set it as hidden by setting the parameter 'Display in Menu' in the Link Type tab to No.
For maximum flexibility I would suggest you create a new Joomla! User Group called say 'Own-uploadersUG' with Registered as its parent. Then make sure that all those users who are to have their own Downloads belong to that user group. At the same time create a Joomla! view Access level called say 'Own-uploaders-view'. Then in the Users Groups settings - Group Creatin/Editing tab you can preset the view access to 'Own-uploaders-view' and aso in the tick box part ensure that the question "Created by' will be asked. The ranking value, second item in the User Groups setting, should be set to a high level, say 123. Please see documentation on Which User Group is used.
You could use the Existing Registered User Group but this might cause a challenge if at a later date you had 'regular' registered users or you wanted to have some public Downloads.