Get and Post request in PHP with RollingCurl library for multiple HTTP requests

General PHP Help
#1

For the given php script, how can I use RollingCurl library to make POST request for each of www.site_01.com, www.site_02.com and www.site_03.com using parsed variables grabbed with “my_request” function from the GET request ? Thank you.

[php]<?
require(“RollingCurl.php”);

function my_request($response) {

…
…

(code used to parse some variables to use later in POST request)

…
…

}

$urls = array(“http://www.site_01.com”,
“http://www.site_02.com”,
“http://www.site_03.com”);

$rc = new RollingCurl(“my_request”);

$rc->window_size = 3;

foreach ($urls as $url) {
$request = new RollingCurlRequest($url, “GET”);
$rc->add($request);
}

$rc->execute();
?> [/php]

#2

Well, either you use POST or you use GET! POST would be much better for this as a huge GET would be a line that would overwhelm the browser… So, to create a huge POST to another PHP page, you just echo whatever you are posting as a field…

Well, after rereading your post, you are NOT talking about PHP-Get’s and PHP-Post’s, so…

In the code you posted, do you mean you want the output that is pulled by the RollingCurl???
You execute it with $rc=>ececute… So, what is the results? What do you want to post?
You can loop thru output and post it, but, you are not clear on what or to what you want to post.

Help us help you… A little more info, thanks!

#3

Thank you for your reply!

I’m a beginner in php and I apologise for not being good into explaining better what I want to do.
Basically I don’t know how to implement POST requests for those 3 websites, with RollingCurl.
“my_request” function is parsing the content of those websites, with GET requests, and then I store the parsed data into some variables as follows:

[php]$post_var = array();
$post_var[‘01’] = $some_var[1];
$post_var[‘02’] = $some_var[2];
$post_var[‘03’] = $some_var[3];
$post_var[‘04’] = $some_var[4];[/php]

Now with the content parsed and variables set how should I implement into that script the POST requests for that websites?

#4

Well, very sorry for not understanding your project. You mentioned some URL’s (websites) and then talk about capturing data. Then, you want to send data that is in variables to multiple sites?

Okay, are they your sites or some public sites? This does not make a lot of sense because you can not open 3 URL (websites) all at once unless you open 3 new browsers. So, really not sure what you mean by sending the data to all of the sites.

Normally, data used in PHP is placed into fields on a webpage. Usually they are just HTML tags and the values of these fields are just echo’d in the PHP. Then, this page has a SUBMIT button that “POSTS” the data to another PHP page which handles all of the work. You have not mentioned any of this. So, I do think that your version of POST-TO-SEVERAL-SITES means something different than what I think it means.
Sorry for not understanding your needs…

#5

RollingCurl library is capable of processing multiple HTTP requests in parallel. My fault for not mentioning in my previous posts the link to the project http://code.google.com/p/rolling-curl/.

#6

I understand what rollingcurl is. It is a system for a server to handle mulitple HTTP request from users in a manor that keeps the responses to the browsers in a timely manor… So, what are you talking about POST to sites using arrays? Do you mean you want to create some special type of server?

Guess that we do not understand what you are attempting to do in your project.

Here is the full source for the rollingcurl:
http://code.google.com/p/rolling-curl/source/browse/trunk/RollingCurl.php
It is basically just an experiment to speed up multiple curl requests on a server and did not go very far and has not been used since 2010… So, not sure what you are attempting to do with this code.

I found this quote:

if you’re using Rolling Curl’s group functionality it seems to break when handling a large number of requests (50k+)
So, large curl calls can be an issue with this. Another quote suggests to use “parallel-curl” instead as it performs better. Not sure if that is true!

So, here is a link with a full listing of a sample of how to use RollingCurl. It seems straight forward. Look down to the read-me for the code. Maybe you can get what you want from there…
https://github.com/LionsAd/rolling-curl

#7

I have to modify content on different websites (6000 to be more specific). I want to do that in multithreading with RollingCurl. The pages that I have to modify has the same structure. The content replacing process is done through POST requests. Before doing POST needs to be the GET requests, so to speak, to parse the page and grab the needed info that I have to post in the POST requests in order to have a successful submit.
I searched over the internet for some examples and I also came across the link you posted but they don’t give an example with GET and POST requests combined, but only GET requests like in the script I mentioned in my first post. Large calls are not a problem in my situation, I plan to open only 20 threads max. I have the necessary resources so this is not a problem.

#8

Okay, I understand your wants better now. Sorry I did not understand before!
I found this code to post using CURL. I is generic, so you will have to alter it, but, it shows the basics…
Hopefully, it is what you need.

[php]
//where are we posting to?
$url = ‘http://foo.com/script.php’;

//what post fields?
$fields = array(
‘field1’=>$field1,
‘field2’=>$field2
);

//build the urlencoded data
$postvars=’’;
$sep=’’;
foreach($fields as $key=>$value)
{
$postvars.= $sep.urlencode($key).’=’.urlencode($value);
$sep=’&’;
}

//open connection
$ch = curl_init();

//set the url, number of POST vars, POST data
curl_setopt($ch,CURLOPT_URL,$url);
curl_setopt($ch,CURLOPT_POST,count($fields));
curl_setopt($ch,CURLOPT_POSTFIELDS,$postvars);

//execute post
$result = curl_exec($ch);

//close connection
curl_close($ch);
[/php]

Good luck!

#9

Thanks for getting back to me. I tried the method you are mentioning a few days ago and didn’t had luck.
A GET request with RollingCurl is like this:

[php]$request = new RollingCurlRequest($url, “GET”);[/php]

a POST request is as follows:

[php]$request = new RollingCurlRequest($url, “POST”, $post_vars);[/php]

There’s no need actually for some other basic curl scripting. My problem here is that I don’t know where in the mentioned script, from my first post in this thread, I have to place the POST call, so to speak. I don’t know how to integrate (combine) GET and POST request to fit my needs. Thank you.

#10

Well, you didn’t show us any code, you showed us an array. ( $urls ) So, I can’t tell you where to stick your post… ( Sorry, LOL! )

But, basically I guess you would create a loop to go thru the $urls variable array using a foreach
and pull each URL out. Then, post it and then loop back and get another. Is that what you are asking?
I do not think you can post an array of URL’s.

Also, you can’t just “post” the way you just showed here. You have to build the “$post_vars” as you called them. They have be in the correct urlencoded format. If they are and you have some code you did not show, then, you can do it that way.

So, not sure if that all helps, but, you did NOT show us any code, so we can not help you where to place additional code. good luck…

Sponsor our Newsletter | Privacy Policy | Terms of Service