There is no standard way to to have a direct link to the downloadable file itself. There are issues about Joomla! permissions and view Access to be considered so modifying the code would, I suspect, be a "challenge".
You basically want to generate someting like below
<a target="_blank" href="https://www.your-site.com/topLevelDir/nextLevelDir/.../filemame.ext
where the ... represents the rest of the directories.
This would mean modifying file <yoursite>\components\com_jdownloads\views\downloads\tmpl\default.php to expand the category chain and getting the filename part by using the CatID and FileID to make relevant equiries from the database.
This is not something for the faint hearted. Also you would need to preserve the code over updates as well. Getting it wrong could prove 'interesting'.