No you are not missing the way to change them in the configuration. There is no config command to change the size of those particular icons. jD always uses those icons from a specic location so you could replace them with your own versions.
The symbols are located in components/com_jdownloads/assets/images
The file names are home_fe.png , search.png , upper.png and upload.png .
So you could use one of numerous picture editting programs to change their size and replace the existing images with resized ones with the same names. Make sure you keep a backup copy of the new files as technically they could be changed in an update.