Okay, internal site was NOT mentioned…
Now, you are talking about capturing XML versions of images.
So, all the data you need are in the XML, correct? Usually you store images by saving them to a folder while using their names and XML data storing in a database. So, expand on what the data actually is. I really don’t want to download and install that program just for testing. Let’s see some data, or at least the format of it and we can help you save it…
Thanks so much for your speedy reply! It means the world
I didnt realize I could be super descriptive. So here is what I’m trying to do.
My goal is for corporate users to insert their work profile url into a form.
Once submitted. I want to grab their profile pic’s absolute path. (this is where that custom class comes in)
once ive gotten the pic’s location…I would like to drop that in an xml file and
using ajax dynamically create .
There are 60,000 + employess and I dont want to store all those photos.
Trying to be fancy.
Please advise to how you think I can accomplish this. Maybe I’m over thinking. 
First, thanks for the details… Now I understand a little better…
If the corporate users already have a URL pointing to their profile, it is somewhere. Is it 'inhouse" on your server or database? If so, just link to that. If it is wherever they put it, such as Google/Yahoo/AOL/Hotmail/etc… Then, it would be against company policies to save that data… Not secure… So, I will assume it is “inhouse”!
It is VERY easy to store just the URL in a simple database or XML file. Then, to “view” it, you can simply use an iFrame. Just have a frame with a web browser in it and post the URL to that frame. Then, their profile will show on your page, but, really be their page. Did that make sense?
Now on another note, is their profiles actual webpages? You mentioned URL’s. So, if it is a webpage, you can view their page inside your page, or “scrap” the data off of the page and capture parts of it as easy as any other string data. There are lots of ways to capture this data and display it. Is it possible to post a link to see a picture of the data you are talking about or perhaps just post an image of it here? I think we still need a little more info… But, maybe this helps you… Hope so… Good luck!
Thanks…
If the corporate users already have a URL pointing to their profile, it is somewhere.
I wish it were that easy! There are multiple servers and hosts.It would be impossible to grab thousands of links! This is why I thought the image grabber class would come it to save the day.
In regards to the iFrames, Looking to build something non-deprecated but they may be a last resort since we don’t need SEO
Now on another note, is their profiles actual webpages? Yes, these are actually webpages.
So, if it is a webpage, you can view their page inside your page, or “scrap” the data off of the page and capture parts of it as easy as any other string data. There are lots of ways to capture this data and display it. Now this sounds cool with Less guess work involved. OMG, this would make page optimization beautiful without all that processing! Would you be able to list a function/variable/class/etc. Thanks so much for your know how.
Is it possible to post a link to see a picture of the data you are talking about or perhaps just post an image of it here? I would love to but it’s quite sensitve
LOL, sounds like we have a plan… Here is a link that tells you how to “scrape” the data off of a webpage.
It is quite simple and I do not want to repeat it all. This tutorial should get you started.
A couple of notes: First, if these “webpages” that have your “profiles” on them are all the same, this will become a very simple process. You would have to study want is inside of one page and then duplicate it for all of them. If they are different layouts, you would have to create a parser that pulls out the garbage first and then parse the results. As an example, I use a small parsing routine that removes all text between as it is not content, but, programming and is not needed for capturing content. Did that make sense?
I did this for Dictionary.com so my robot A.I. can understand new words that are not in her database. She scrapes their site and pulls the meaning (up to 20) for any new word, stores it in her database of knowledge for future use. If you use a definition a lot, she moves it up as a better memory of the word… Very cool stuff!
So, here is the link. Hope that helps you…
http://www.devblog.co/php-web-page-scraping-tutorial/ Ask any further questions about it…
This is SUPER!!! It certainly put me on the right track.
[tt][tt]THANK YOU SO MUCH!!![/tt][/tt]
I now get to have my weekend back!!! ;D
Many many many thanks!
Glad I could help. Hope it works out for you… Good luck!
Dang it…
It worked for about 20 mins until I got stuck on the same scenario.
Trying to grab my URL from the database, grabbing the image and placing it into
I thought I was so close. Whatev’s, I’m just going to use the iFrames. Sighs!
Still thanks for the insight thought. You rock!
So, what holds you up at that point? I mean, you have the URL, what is going wrong with the display?
Give a little more info on what is happening. First, you aren’t saving pictures inside the database are you?
That is not good. Also, if you have permission to grab the page, in what manner are you saving the page?
What data are you displaying… It shouldn’t be that hard. At the very least, you can just use a simple PHP
include after stripping out all of the header info…
All programming is possible, just have to figure out the puzzle…
Morning!
Okay here is what’s happening. I followed the tutorial last night but it was based on text files and strings. I have no idea about expressions, I just learned how to fetch data from the database. I literally still have the bottle in my mouth. 
I did a Google search for a similar web scraping/harvesting tut with images, the only thing that came up was another custom class (Simple HTML DOM). That worked handsomely, I was excited until I tried to extend the functionally.
The below works like a charm, very fast by the way.
[php]
I need to be able to make $html = file_get_html(‘url’); dynamic.
This needs to talk to the data that was submitted by the user in the database…
The users information is stored in a table called ‘user_info’ with ‘user_name’ & ‘user_url’
My goal is to fetch the url, scrape the image and dump into a preloader so that it can be called in by ajax when needed. My failed attempt below…
[php]
<?php include 'connect.php'; $sql="SELECT * FROM user_info"; $result = mysql_query($sql); while($row = mysql_fetch_array($result)) { echo "" . $row['user_name'] . "
"; // I would like for the below information to dynamically populate file_get_html('url'); and scrape images echo "" . $row['user_url'] . "
"; } ?>[/php]
It all look so beautiful In my head, but I guess in theory not so much 
I am sure there is a way easier way to accomplish this I just don’t have the knowledge to do so.
The iFrames are so slow!!! I’m sad
Yes, any project is very pretty in the owner’s head! LOL…
So, this: $html = file_get_html(‘http://www.cnn.com’);
is simple to change… You have your lists of URL for each profile…
Just parse thru them assigning $url=“yourdomain.com/someonesprofile.php”, whatever URL for their profile.
Then, SCRAPE the page and find the image tag for their picture. Remember in that page, there will be just a tag for a location of a picture, NOT the actual picture. So, you would be looking for THAT URL’s which you would then save in your database. (Just the SRC part). At this point, you have stolen the link to their picture and have it stored in your database. Then, to use it, you display it on whatever page you are creating by inserting it into an image tag: <img src=“http://”.$pixurl> or whatever you call it
when retrieved from the database…
Did that make sense? The hardest part is the SCRAPE routine. It would need to know which image tag is the person’s profile picture. Most likely it is clearly marked with some other tag or displayed text. You would have to pull a few pages and look at the actual code of each to see how to tell your SCRAPE routine to get the correct image URL.
Hope that helps…
Just parse thru them assigning $url=“yourdomain.com/someonesprofile.php”, whatever URL for their profile.Then, SCRAPE the page and find the image tag for their picture.
This is where I’m having troubles. Would that look something like…
[php]
<?php include 'connect.php'; include('simple_html_dom.php'); // Grab Data $sql="SELECT * FROM user_info"; $result = mysql_query($sql); // Iterate through tables while($row = mysql_fetch_array($result)) { // Please don't judge the syntax to bad, I'm struggling a bit $url = $row['user_url']; $urlList=array(); array_push($url); scrapeThis(); } function scrapeThis($urlList){ $html = file_get_html($url); foreach($html->find('div#pic') as $e); echo "[/php]
Well, almost… But, try it this way… (No need to build an array, as the query already is one!)
[php]
// Please don’t judge the syntax to bad, I’m struggling a bit
$url = $row[‘user_url’]; //Already got an user url, now scrape it…
$html = file_get_html($url);
foreach($html->find(‘div#pic’) as $e)
{
echo “” ;
}
}
[/php]
Not sure about the scraping function with the “find(‘div#pic’)”, but, the rest should work.
Basically, you pull all of the users info and parse thru each row of users and scrape each user’s url page.
Should work, but, for testing I would add " LIMIT 10" to the query, space-limit-10 so that you only test
with 10 users. If you have 50,000 users, it would run for hours… LOL Good luck!
Always almost there!!!
Warning: file_get_html() [function.file-get-contents]: Filename cannot be empty
I keep running into this error on every aspect, even with the web grabber, it was the same scenario.
This why I think I was trying to iterate through each row and trying to output $row[‘user_url’] into a string to dynamically.
[php]<?php
include ‘connect.php’;
include(‘simple_html_dom.php’);
$result = mysql_query(“SELECT * FROM user_info”);
while($row = mysql_fetch_array($result))
{
$url = $row[‘user_url’];
$html = file_get_html($url);
foreach($html->find(‘div#pic’) as $e)
{
echo “” ;
}
};
?> [/php]
That code looks good.
But, are you sure you have a valid URL in the database?
Try echo’ing it so you can see. I bet the first part is not HTTP or quotes or something…
Like this:
[php]
$result = mysql_query(“SELECT * FROM user_info”);
while($row = mysql_fetch_array($result))
{
$url = $row[‘user_url’];
echo “URL = ***” . $url . “***
”; //debugging…delete this line later…
$html = file_get_html($url);
echo "Length of page: " . strlen($html) . “
”; //debugging…delete this line later…
if(strlen($html)>0){
foreach($html->find(‘div#pic’) as $e)
{
echo “” ;
}
}else{
echo “Zero sized URL!!!”;
}
};
[/php]
Look between the stars for odd code. Remember it should be in the format:
“http://www.xyz.com/somefolder/somepage.ext” Any difference could cause no output…
And, I added check for zero size page…
Yea, all the links are absolute paths. no crazy looking ones.Same error comes up. Maybe I should look into a client side scrapper if they have one. This is way over my front-end head
So, after trying my last code, you got a displayed URL.
Did it have HTTP:// in front of it? Was it formed correctly?
If you copy it into a browser, where does it take you?
???
OH, PS: a CLIENT-SIDE scrapper, everyone has that built-in. Just RIGHT-CLICK on any blank area of the page, select VIEW-SOURCE. Then, you will see exactly what is being displayed!
LOL… Never heard it called that! I like it!
The URL showed up but I still got the error.
LOL, not the view source!
A Client-Side (meaning Javascript) Scraper in addition to a web service that helps across domains.
http://www.kelvinluck.com/2009/02/data-scraping-with-yql-and-jquery/
I’m losing gas!
I’m just going to ask the corporate folks to upload their own damn images with an upload tool!
I need a drink now!!! Thanks for all your help on this. I never imagined I would get so much help.
Most of the time people will be douche bags when questions are asked but you were not!
No problem! I enjoy helping… (That’s why my Karma is so high from helping and being patient to see it thru to the end… I think you should be able to get it though. I know you are close!
Hey, are the pages you are trying to get the pix off of available online? Can you private-message me a link to one so I can see what you are scraping? Then, maybe I could run up a sample to work with it… ???
Leaving for a bit… later…
Okay, let’s back up a few posts…
You echo’d the URL and it displayed it. And, it looked okay.
You typed it into a browser and you saw the page with the picture you wanted.
You viewed the source of that page and made sure the picture’s URL is in the <DIV#pic> tag.
You typed in that URL and can view the picture.
If all this is true, then your “scraping” routine is bad!