Print Page - Initial scan of large amount of files

Title: Initial scan of large amount of files
Post by: nord_tramper on 30.09.2017 22:29:56

Hello!
I have about 30 000 files in jdownloads folder. When I try to scan it manually (click link Check Download area now) it works some time. Scan 300-400 new files and hangs.
How can I scan all files?

Title: Re: Initial scan of large amount of files
Post by: nord_tramper on 01.10.2017 10:37:22

Whe I try to use curl to run scan I get the error like this

Quote
<html xmlns="http://www.w3.org/1999/xhtml" lang="en" xml:lang="en">
<head>
<title>Server error!</title>
<link rev="made" href="mailto:%5bno%20address%20given%5d" />
<style type="text/css"><![CDATA[/*></style>
</head>

<body>
<h1>Server error!</h1>
<p>

The server encountered an internal error and was
unable to complete your request.

</p>
<p>

Error message:

mod_fcgid: error reading data from FastCGI server

</p>
<p>
If you think this is a server error, please contact
the webmaster.

</p>

<h2>Error 500</h2>
<address>
<a href="/">mapsforyou.org</a>

<span>Apache</span>
</address>
</body>
</html>

Title: Re: Initial scan of large amount of files
Post by: ColinM on 02.10.2017 18:44:18

Hi
Sorry for delay but I missed this one!!
Basically I think you are running out of time on your web server as defined in the php.ini file

You could ask your Web service people to temporarily increase the allowed time. As an indicator assuming you have the usual 30 second max time then it looks like 10/second. So you might need to do in 'blocks' of files. The other way is to do it on a local PC web site using for example XAMPP.
Colin

Title: Re: Initial scan of large amount of files
Post by: nord_tramper on 04.10.2017 19:46:55

Hello!

I have 600 sec time limit. In hosting logs I can find only error about FastCGI.

Error log:
mod_fcgid: ap_pass_brigade failed in handle_request_ipc function

Title: Re: Initial scan of large amount of files
Post by: ColinM on 05.10.2017 14:40:13

Hi
600 seconds is a long time! I have not seen the error you showed. Will ask Arno to see if he is aware.
Colin

Title: Re: Initial scan of large amount of files
Post by: Arno on 05.10.2017 15:09:49

Ten minutes is not really a long time for this amount of files. Please note that the script must also create for every file two data sets in the database (one in _jdownloads_files and one in _assets). I know from other users that the import process for this amount of files can need hours. ::)

QuoteError log:
mod_fcgid: ap_pass_brigade failed in handle_request_ipc function

I have never seen an error like this.

jDownloads Support Forum

Older Versions => jDownloads 3.2 (Support ended) => Topic started by: nord_tramper on 30.09.2017 22:29:56