I think that the Add button should be treated the same as the Create Menu item, after all a button is just the same as a menu item in its effect. That is it does not depend on upload permissions but on View Access. That would be totally consistent. This is the same as different categories being shown dependant upon View Access.
It is simpler of course in the menu item, and in categories, as the View Access Level is set explicitly. Maybe use could be made of the User Group Rankings to determine the applicable User-Group and see if that user group has View Access. Perhaps another way is that the Configuration (front end tab?) specifies the View Access Level for the Add button. This I think would be a practical scheme as one has to specify View Access everywhere else.
It is then entirely reasonable to expect that if the admin person has given View Access to the button for some user groups then they should also have given create rights to those user groups in the view access. This is exactly how one controls the Create menu item. As ever documentation can be supplied.