Special characters

[DylanStr]

Hello all

I have a little problem... I sync a folder with 26.000 pdf files to my jDownloads directory. Those  files sometimes contain characters like # and ( for example. Now i noticed that everyday the sync software copies the files containing these special characters again, because jDownloads deletes those characters from the filenames in the directory, so my sync software thinks the files arnt there anymore. Is it possible to keep those characters in the filename? So i dont end up copying these files everyday again because the filenames have changed. I have to keep syncing the folders because new pdf's are added to the source folder and i want them to be automaticly published on my website.

These are the settings I tried but it still deletes the special characters from the files:

- Use UTF-8 = Yes
- Replace spaces with underscores = No
- Change name to lowercase = No
- Remove/Change special characters in name = No

Any solution for this?

Thanks, Dylan

btw: I use Centos 7 for my hosting.

[DylanStr]

Anyone? Or is it that # ( ) are all illegal characters? I really need a way to get jDownloads to work with these characters in filenames.

[ColinM]

Hi
I think perhaps jDownloads is being overzealous in the characters it allows.   It might be that the Download title will not allow those characters but it maybe possible in the file name itself.
I assume that the files are uploaded directly to the relevant directory by  ftp or similar.  Is that correct?   Do you then either run a CRON or manually start a scan or use the automatic scan?

Colin
NB it may take a couple of  days to see into the code about what is happening.

[ColinM]

Hi
I have looked more into the detail.    Now the Download title may include characters such as  # ( and )

Basically jD uses the Joomla! clean function in the filter package to 'sanitise' the file  name.   This includes removing characters such as # ( and ) as you already know.
Now having looked on the web the general consensus is that as one does not know what operating system is being used by the people downloading, and more importantly what restrictions apply to file naming conventions on their system then the use of special characters such as those you want should be avoided.  This is what happens.

However there is perhaps a way of  re-looking at the  problem.  That is to 'clean' the file name string before  doing the check on existence. This would avoid your main problem whilst still producing  universally acceptable file names.   You could still use  # ( and )  characters in the uploaded files provided their exclussion still gave a  unique file name.

Would such a solution be acceptable?
Colin
NB This is just exploring possibilities, it is not promising a solution as investigations continue

[DylanStr]

Thanks for your reply Colin.

I use Syncbackpro instead of an ftp program. I redirected my jDownloads file dir to the share where my files are uploaded to. Everything is fine except for the filenames that change after I do the monitor scan from the control panel to check and publish the files. I dont do auto monitoring, but instead have a working cron job. I think a solution like that would be fine if i understand what you said. How would I manage to do this? Btw the downloads on the frontend do have the characters in the title's, it's only the filename itself that gets changed, and thats where the problem is. My Syncbackpro thinks the files are missing, so  all the files with those characters get copied again to the jDownloads dir. And than  I end up with duplicate files in my folders, with different names.

Please see the attachment for my folders. As you see that weird characters get replaced by spaces.

[gelöscht durch Administrator]

[ColinM]

Hi
Yes it is just the file names that are changed, not the Download titles.  This is because some operating systems will not accept characters like ( ) or #  and jD does not know which operating system a user has on their device.  So jD must play safe.

In the code jD first 'cleans' the filename of the potential new Download for the 'unacceptable' characters. It then effectively compares that 'clean name' with the names of all the existing filenames - it actually asks the database for files of that name.

Now in your case you already have existing files with what I call here 'uncleaned' file names from an earlier series (probably a 1.9.x series?).  So the comparison fails and jD thinks it has a new Download.  The process is a little more complex than above as one can have the same file name in different categories (folders).

Arno has been looking at the code.  Could you please see if it has been fixed with the latest version of jD (v44).

Colin

[DylanStr]

Hi Colin,
I updated to the new version and I just tested it by copying a new file with #  ) and  ( characters in the filename into my jDownloads directory, clicked the ''Check downloads area now'' in the control panel, but the filename got cleaned again after that, so still no succes I understand it does that to make filenames save for every OS. But both windows and linux can get along with #  ) and  ( in filenames as I tested that already. Is there a way to modify the code  to get it working with #  ) and  ( ?

Thanks a lot for all your help Colin!

Btw: I forgot to mention that in the previous post in my attachment, the left side is my source folder where i upload my downloads from, so thats not on my webserver, the right side is the jDownloads directory on my webserver after a manual monitor.

[Arno]

I will check this later today again. I am not sure what was really the reason to handle it so. This was written long time ago. 

[ColinM]

Hi
I have now reproduced your problem! It is illustrated in the two composite pics attached.
Firstly the  'clean' used by jD is the standard Joomla! one.  Code changes to that would not be a good idea.  However that is not the problem as jD correctly replaces the file with the new version.  The problem is that jD then goes on to create a new Download, or rather does not delete the old one.   This is obviously not correct so I have raised it to a 'bug'. 

The first pic shows a normal sequence. The file name is correctly cleaned and the Download is created.
The second upload sequence of the same original file is also good in updating the file but creates two Downloads.  Both of these Downloads point to the same file. If you delete one of the Downloads then it deletes OK. On deleting the second Download then it gives an error message saying file is missing, but goes on to complete the deletion.

The test files were uploaded using  File Zilla ftp.
Colin

[gelöscht durch Administrator]

[DylanStr]

Thanks again both for your time.
Yes you reproduced my problem exactly right! So we can say this is a bug that needs to be fixed? It sounds like a good fix to me, that when you upload the same file again with the characters still in it, it cleans the characters from the filename, but wont create a new seperate download for it. Right? I will be fine with a fix like this. That way I can keep synchronizing my folders without jDownloads making a new download for the same file again and again.

Dylan

[ColinM]

Hi
Yes it is now listed in bugs!
Colin
Your next question is of course how long? Normally Arno is very quick and posts a  modified file for you to upload so you do not have to await the next release.
A valid but unhelpful answer to the 'how long' question is "twice as long as half of it!" - and also remember we do this as a community effort.
C

[DylanStr]

Don't worry about the time it takes to fix it. I am very happy with the support you guys are giving me. I assume that I can expect an update on a fix  in this topic?

Dylan

[Arno]

Yes. 

[DylanStr]

Any update on a fix yet guys? 

Best regards,

Dylan

[ColinM]

Dylan
Arno is working on four items.  One of these is the special characters; another is about forcing permissions propagation, the third is on the Download Button filtering and the fourth relates to the Recaptcha (again).
We have been 'hit' by some cosequences of unannounced changes in Joomla 3.5.1
Colin