Searching large database

AnotherWannabe · May 24, 2012, 10:05pm

I’m looking for a way to create a “search engine” for my site. Suppose you have a very large database (millions of rows) and each row has a field called “Title.” Title can be 10 words or so. I want to be able to search using search terms and bring back relevant entries. I’m not looking for code here. I just need an idea of how to go about this, or where to buy software that helps with it. I don’t want to require every word to match, but I want to return results that are most relevant first…

Could I check to see how many words match, and then order them based on that count? This seems like it would be really slow on a large database. If the rows were sorted could I make this easier? Any help is appreciated.

Ojoshiro · May 25, 2012, 9:11am

Hi,

How about you add ‘tags’ to each data-record. Your search then runs through the tags and returns a list of data-records attached to the tags. The more tags you ‘hit’, the higher the ranking of the result.
If you want the system to ‘learn’ you keep a score in the link-record between the tag and the data-record.
Every time a resulting data-record gets chosen you increase the Link-score.

.--------.     .--------.     .--------.
| Data   |_____| Link   |_____| Tag    |
|        |1   N|        |N   1|        |
'--------'     '--------'     '--------'

‘Tags’ would be single words or maybe brief (atomic) descriptions.

Just brainstorming, hope this helps, ;D
O.

AnotherWannabe · May 25, 2012, 6:41pm

That’s about how I’ve envisioned it, assuming each keyword is treated as a tag. So I could take each keyword entered, and for each one run a query returning the results that have that tag. Do some manipulation to count how frequently a record appears in all the queries, and score them accordingly. But with a large database, is something like this feasible? It seems like it requires many passes through. Suppose I need to look through 2 million rows. I use 4 words in my search. Has anyone ever worked on a large project like this? Would this take a long time (more than 2-3 seconds)?