Bug with encoding filenames to UTF-8 in jD3.2.16 [Fixed in 3.2.18]

[Makulia]

Hello!

First of all, thank you for the great component. I have moved from Phoca Download and can say jDownloads is a way better!
But I have some annoying problem with file names.

I have carefully read docs and have enabled UTF-8 support in the component settings. After that, folders named in russian are creating without any problem only when utf-8 option is enabled. But when I uploaded file with the russian name, for example, "тест.txt", I was unable to download it. System just didn't found it. I have looked into DB jdownloads_files table and turns out, that file name is not saved correctly. Instead of "тест.txt" we have ".txt". File itself is uploaded correctly and shows in folder like "тест.txt".

So main problem is that file names are not converted from windows-1251 to utf-8 and, probably, because of this, a not saving correctly to DB. I asume we sould do something like this $file = iconv("windows-1251", "UTF-8", $filename), but can't figure out, where to do this.

Please, can you help me with this issue?

Info
JDownloads: 3.2.16 Beta
Joomla version: Joomla! 3.3.6 Stable [ Ember ]
MySQL: 5.6.19
PHP: 5.5.9
Cache: all Joomla cache options are deactivated
Files are uploading from Windows 8.1
P.S. My web server runs on production server with Ubuntu 14.04.1 LTC and support utf-8.

[Arno]

Hi,
the utf-8 support is not easy and i think this is the reason that nearly no software will support this.

I asume we sould do something like this $file = iconv("windows-1251", "UTF-8", $filename), but can't figure out, where to do this.

All upload forms should be defined as utf-8 but maybe must we convert here the filename.

Thanks for this hint. I will check it.

[Makulia]

Hi! That would be great, because it is IMHO most simple and working solution! Uploading files in utf-8 and folder creation are working great, problem is only with saving correct value into the DB.

BTW, may be you can point at the place in the jDownloads code, where it will be most appropriate to do?

Thank you!

[Arno]

Sorry but i can not view now in the source code.
So you must wait a few hours. 

[Makulia]

Ok, it's not a problem! Thank you very much!

[Makulia]

I also have found the bug with transliteration of russian directory names. When it is enabled, folders rename into this format: "2014-11-14 10:11:54" instead of transliteration. Standart Joomla transliteration is working correctly! Are this problems connected?

[Arno]

So this was not correct in your first post?

After that, folders named in russian are creating without any problem.

Edit:
please post here all your settings from the 'folder and files' TAB in your jD configuration.

[Makulia]

My mistake! Working well, if only utf-8 option is enabled.

Info:
1) When UTF-8 is set to "No"
You see, alias is formed correctly "proverka", but folder not "2014-11-14 15:53:48", problem with DB exist.
2) When UTF-8 is set to Yes
Current settings. Folders are created correctly with russian names, problem with DB exist.
P.S. I have tried to disabled "change names to lowercase" and "Replace spaces with underscores in names" problems with transliteration and DB population stay.
I have attached file with russian name for tesing (inside zip).

[gelöscht durch Administrator]

[Makulia]

One strange behavior - when uploading file with russian name and "_" betwin words, like Тестиг_тестинг.doc, JDocs trims it to _тестинг.doc, and when uploading тестинг.doc it simply trims it to .doc.
I am speaking about value in DB. File, itself, uploaded fine and remain untouched.

[Arno]

So the problem is only with the(uploaded) filename self, or also how is stored the filename in the DB? 

[Makulia]

1) Problem is not with filename itself (it is uploaded correctly), but with how it is stored in the DB (because of it we a getting broken download link)!
2) There is the problem with transliting russian Folder names into transliteration. Here we are talking only about a folders.

[ColinM]

Hi
I think that the jDownloads transliteration tables need extending.  The present tables work fine for 'Western European' languages but have a problem with Cyrillic languages such as Russian.  Also I do not think it is possible to user a single table for Cyrillic languages or include the in the existing table as there are potential clashes of usage.  Rather it would seem to me to make use of the native locale to decide what language to use.  An alternative and perhaps more extensible scheme is to treat the transliteration tables as 'transliteration packages'  along the lines of the 'language packages' that are loaded by the admin person.


Attached as an example is a sort of transliteration package for Russian. With a few minor extensions I think this would also cover most of the Cyrillic countries.
I can readily get Czech, Hungarian, Lithuanian, Polish, Slovak, Serbian, Yugoslavian and Ukrainian txt files is the same form as the Russian one.
Also have a Greek one that could be added into the existing table as a short term measure.

As a very short term measure the selection of which transliteration table should be used could be specified in the Configuration, with Western European as the default for existing users.

Colin

[gelöscht durch Administrator]

[Makulia]

to ColinM

I agree about the transliteration table, but have a question - could we use Joomla core transliteration functional, as we do in alias field, for example?
What do you think about the problem with storing cyrillic filenames in a DB?

[ColinM]

Hi
Following your question I delved a bit deeper got useful article https://www.corejoomla.com/forum/community-polls/13873-alias-transliteration.html
and followed up to see stringurlsafe  at http://www.reference.joomlademo.de/nav.html?_functions/index.html#stringurlsafe

It processes a string and replaces all accented UTF-8 characters by unaccented ASCII-7 "equivalents", white spaces are replaced by hyphens and the string is lowercase.
Best reference is http://www.reference.joomlademo.de/nav.html?libraries/joomla/filter/output.php.source.html#l77
Code is very short

Code Select Expand

public static function stringURLSafe($string)
  88      {
  89          // Remove any '-' from the string since they will be used as concatenaters
  90          $str = str_replace('-', ' ', $string);
  91 
  92          $lang = JFactory::getLanguage();
  93          $str = $lang->transliterate($str);
  94 
  95          // Trim white spaces at beginning and end of alias and make lowercase
  96          $str = trim(JString::strtolower($str));
  97 
  98          // Remove any duplicate whitespace, and ensure all characters are alphanumeric
  99          $str = preg_replace('/(\s|[^A-Za-z0-9\-])+/', '-', $str);
100 
101          // Trim dashes at beginning and end of alias
102          $str = trim($str, '-');
103 
104          return $str;
105      }

Maybe it could be included by Arno in a similar way but without the strtolower

Colin

[Makulia]

I think, the last one is the best solution!

I have tested it with another Joomla extention - Fabrik, upload element. And it works great!