Cron Job to upload 200 000 + documents

[rene@ntoni.com]

The scan.php gave me problems regarding time when I tried to add about 240 000 large documents to jdownloads. I had to modify the scan job in order to improve performance. Hope this could help someone, or let me know if you find bugs.
The original ran for over 2 hours and sometimes never stopped. Now its running in 15 minutes.

The code contains custom requirements :
1. log table must be added: vzr01_vzrupload_scanlog fields: id,start,end,duration,dir,script,countfiles
2. sortCategories(); contains custom (hardcoded) sorting requirements - can be omitted or modified.
3. These lines can also be removed
        $VZRlistConfig = VzruploadHelper::buildVZRlistConfig();
        $status= $VZRlistConfig['ImportProcessStatus'];
         if($status=='2'){ //files fixed




[gelöscht durch Administrator]

[Arno]

Hi Rene,
many thanks that you share here your work with us.
You have right, it exists in the old file many points for optimization.
So i will test also your file here by me. When it works good, i can add your corrections also in the coming jD 2.0.
I hope this is okay.

[rene@ntoni.com]

no problem,please excuse any bugs. My script assumes the files are located in the "last" child folder, and might be a dangerous assumption - I customised this for my needs and might not be the same as for you. The problem I had with the original script was that the array of file (buildArray) was just to be for 200000 documents. One could probably use another method to subdivide this array process, but this method works for me. One could probably build an array of directories containing files and loop through that?

[rene@ntoni.com]

Hi Arno
I also created a jsTree(ajax tree) (www.jstree.com/) module for jDownloads, and modified the jDownloads frontend component to use ajax(jquery), which seems to work well. Once again catering for my own needs, and not tested with different backend setups.
website :www.vanzylrudd.co.za to have a look.
I'm working on the search module at the moment, since I actually use JiFile to index and search the documents.