Php async http download

I want to download 100 websites sources. And check for some content. It really takes long time with file_get_contents. How can i do this async? Trying to solve this whole day. Help. Is it that hard?

$sql = “SELECT * FROM Address”;
$result = mysql_query($sql,$con);

while ( $row = mysql_fetch_assoc($result) )
$text = file_get_contents($row[‘Address’]);
//other code

with a cron job…

ye i use cron job, and its ok for now. But later there is going to be more addresses. Isnt there 30s timeout on php script or something.


now there isnt ^^

oh ok, will do for me. Thanks. Anyway, is it hard with php to make async dls?

to do something like that requires threading, which php really does not have a great way to handle.

you can give this a try, it might work.

Works ok if i set time limit on my pc, but when i host, there still is 30s limit, and set_time_limit doesn’t work. So now again i have a problem. Need to optimize my script. If its hard with php, maybe i should use some other language?

I am getting this error:

[Mon Feb 04 22:21:26 2013] [error] [client] FastCGI: comm with (dynamic) server "/usr/local/fastcgi/cgi-bin/php-cgi" aborted: (first read) idle timeout (30 sec), referer: [Mon Feb 04 22:21:26 2013] [error] [client] FastCGI: incomplete headers (0 bytes) received from server "/usr/local/fastcgi/cgi-bin/php-cgi", referer:

Sponsor our Newsletter | Privacy Policy | Terms of Service