Hey there,
I am not new tot PHP. I know the basic stuff etc.
Got the following script from the Piratebay to scrape and update torrents.
I have the SQL database set up and running. Now I use the script but everytime an error occurs:
Fatal error: Out of memory (allocated 180092928) (tried to allocate 49 bytes) in ***/test.php on line 40
What did I try already?
- I already increased the memory limit (in the script, with php.ini and with htaccess), but it keeps saying the same thing.
- Put the bzopen-file on my own server, does not help.
It seems the bottleneck moved from another line to line 40 when i increased the memory.
How could i adjust the script to overcome the problem? Maybe a change in the script to save memory over time or something similar?
Here is the code:
[php]<?
$mysql_host = “127.0.0.1”;
$mysql_user = “”;
$mysql_pass = “”;
$mysql_db = “”;
$connection = @mysql_connect("$mysql_host","$mysql_user","$mysql_pass") or dberr();
$db = @mysql_select_db("$mysql_db", $connection) or die(“Couldn’t select database.”);
// Get info_hash for all our torrents by trackers included in the bulk-scrape.
$r=mysql_query(“SELECT info_hash,seeders,leechers FROM torrents WHERE tracker LIKE ‘%thepiratebay.org/announce%’ OR tracker LIKE ‘%tracker.prq.to/announce%’”);
// Count total number of torrents we have that are tracked by trackers included in the bulk-scrape.
$totaltorrents=mysql_num_rows($r);
while ($rs=mysql_fetch_assoc($r)) {
/*
* Here is the place to recode the info_hash to however you stored it in your real database.
* For example if you store it base64-encoded you would:
*
* $rs[info_hash]=base64_encode(urldecode($rs[info_hash]))
*/
$torrents[$rs[info_hash]][seeders]=$rs[seeders];
$torrents[$rs[info_hash]][leechers]=$rs[leechers];
}
// Download the scrape data
$fp=bzopen(“http://scrape.thepiratebay.org/all.txt.bz2”, “r”);
// Read data from file.
$scrape_data="";
while (!feof($fp)) {
$scrape_data .= bzread($fp, 65536);
}
bzclose($fp);
$scrape_data_rows=explode("\n", $scrape_data);
// Save some memory.
unset($scrape_data);
$updated=0; // count how many torrents we updated;
foreach ($scrape_data_rows as $rows) {
$row=explode(":", $rows); // $row[0] is info_hash in base64 encoding
// $row[1] is number of seeders
// $row[2] is number of leechers
if (is_array($torrents[$row[0]])) { // We have the torrent in our database.
// if we already have the correct number of peers, no need to update the database.
if (($torrents[$row[0]][seeders] != $row[1]) && ($torrents[$row[0]][leechers] != $row[2])) {
$r=mysql_query("UPDATE torrents SET seeders=$row[1], leechers=$row[2], last_scraped=unix_timestamp() WHERE info_hash='$row[0]'");
$updated++;
}
}
}
// Set seeders/leechers to 0 for torrents we have’nt seen for 2 days.
$r=mysql_query(“UPDATE torrents SET seeders=0, leechers=0 WHERE last_scraped < unix_timestamp() - (86400*2) AND (tracker LIKE ‘%thepiratebay.org/announce%’ OR tracker LIKE ‘%tracker.prq.to/announce%’)”);
echo “We updated $updated of $totaltorrents in our database\n”;
?>[/php]