Search Engines of the Future

[Ian]

Hey all,

As far as I understand, Google spiders crawl the net in search of meta-tags, keywords, web addresses, links, god knows what else... an index of the web is made... your search pulls up results based on their index and search criteria.

So if you type "Garry Barlow wallpapers", you'll get what you want. But inevitably, if you type something like "I'm tired" or "why do my eyes hurt?" you'll get answers that have nothing to do with what you were thinking about.

With Wolfram Alpha, the last two queries pull up very different results, but it's still a far cry from having a conversation with the Internet, like:

You: give me apple
Internet: no apple for you
You: can not has?

I'm curious, what is the future of search?

What would you like to see (i.e. what would you like to be able to do or see on the web, apart from pron)?

[bailewen]

Actually, the biggest improvement that could be made in search engines would be, IMO, a step backwards. Google, Yahoo and the rest are now massively influenced by advertising dollars. When I search "Thailand", for example, 10 years ago I would get a bunch of varied and interesting information about Thailand. History, culture, current events. Now the first page of results is mostly travel agents. When I am googling for computer stuff its much worse. Damn near impossible to get info on what a particular driver is for or what a certain registry entry does. All I get is sales pitches for sites that sell drivers or software that checks my registry. Try searching for software and I get blasted with sites selling membership to download software and they don't even have the software I was looking for.

[yusuf]

interesting thoughts Omar ..

[bailewen]

heh.

It particularly saddens me because the main reason I pretty much went with google back in the day when Yahoo and others were still serious competitors was exactly because it gave me less marketing oriented results. Yahoo was always a total whore that way. With time, google is getting as bad as Yahoo once was.

I suppose a compromise improvement would be if search results could be have a new category on the top along with pics, news etc that was something like, "not sales related". Not sure how that filtering could be accomplished but it would be nice. Sometimes I really am looking for restaurant reservations or a price on an airline ticket but sometimes I am trying to actually research a topic. I'd like to be able to easily filter out books, movies and other media with the search terms for say..."spider" and not get spiderman and virtual spiders and whatnot.

[Darthwing Teorist]

I would like to see a personal search engine: software that crawls the net and works from your PC, so you get to tweak it to get the best results.
Search power to the people!

[Darth Rock&Roll]

what happened to semantic web TBL's project?
Is that not the purpose of it? Or did the orig tech get relegated to suggestions in the search only?

[qiphlow]

don't worry. google will soon deliver the information you seek before you knew you were seeking it.

[Darth Rock&Roll]

google stole my man butter while i was sleeping!!!

[Bao]

I don't really agree with neither of Ian or Omar. It's not all about business, but about SEO and content. Companys has obviously more money to keep SEO up-to-date, buy inlinks etc. than an informative site, or cultural one.

But I liked that Wolfram Alpha, I must say. Excellent, I think I will use that instead.

"don't worry. google will soon deliver the information you seek before you knew you were seeking it."

That's what Google claimed they could do for april fools more than five years ago.

Edit: Damn! That Wolfram Alpha was really, really good!!!
http://www.wolframalpha.com/input/?i=gazpacho

[qiphlow]

google stole my man butter while i was sleeping!!!

no, that was father time, you old geezer.

[Ian]

I don't really agree with neither of Ian or Omar. It's not all about business, but about SEO and content.

When did I say anything about business

[marqs]

I agree with Bao in that search engine optimization (SEO) is the main reason why commercial sites pop up in search results. The paid content (where google gets the money, not the SEO company) is actually separated quite well.

Some more realistic scenarios would be
• create a different version of this http://www.google.fi/advanced_search?hl=en
• improve the use of semantic elements like microformats (telling explicitly in a standardized format what the data inside the html tag is)
• improve the semantic understanding of the spiders (possible, but would be costly because of increased processor needs, growing amount of data etc)
• give the search engines more personal data, so they could customize results based on your interest-profile. A bit scary, but would be efficient.
• create more efficient systems of social search and recommendation (similar to delicious, which relies on community tagging) - this would reduce the "computer-aspect" of search

All of these directions are currently pursued, but the visibility varies. My guess is that the huge growth seen in Facebook, Twitter etc has provides data from individuals that would enable quite good profiling. FB strategy seems to be to use the Facebook Connect to integrate with other sites, and when you start to login in to the search engine with your FB login, that engine probably uses your profile, data from your social networks etc to refine your search results.

Taken further, every (Connect-enabled) site you visit could be tagged invisibly with data about how many pages you read there, which pages etc and when someone with a similar profiles enters the site, they see directly content that most likely is relevant to them. I'm not sure how legal this is though, but if it's an opt-in system, it should be legally ok. Still, makes me get some 1984-vibes.

[Steve James]

No one has to use Google or Facebook, and the fact that they make money from your usage should be expected. How to make sites commercially viable has always been an issue. It's natural to put ads on sites that are popular with people. The data-collection aspect, however, was something that emerged without being planned (a la 1984). The expansion of internet usage has been concurrent with the increase in computer processing power. Unlike Orwell's world, there's no Big Brother behind it (not Gates, the guys at Google, or the owner of Facebook). People self-limit their sources of information and voluntarily opt-in to giving others access to information on their browsing and buying habits.

[bailewen]

You guys have got a logical fallacy.

Just because sites are ranked according to SEO protocols it does not follow that commercial sites are not pushed to the top. In fact, you openly admitted that they are pushed to the top:

I agree with Bao in that search engine optimization (SEO) is the main reason why commercial sites pop up in search results. The paid content (where google gets the money, not the SEO company) is actually separated quite well.

I made no claims as to the cause. I simply answered Ian's original question of how search engines could be made better. The first thing that I would like to see to make them better is to NOT have all the commercial sites pushed to the top. I don't really give a fuck how that is accomplished or why it's not the case now.

To whit:

What would you like to see (i.e. what would you like to be able to do or see on the web, apart from pron)?

I would like to see a search engine that does not push commercial sites to the top.

edited for grammar

[Dmitri]

That WolframAlpha thing is clearly biased. I mean, a search for Stephen Hawking returns results with a picture, and a search for Paris Hilton doesn't?! I mean, WTF?!

It's not bad though, nice/interesting layout. But looks like it's made "by the scientifically-minded for the scientifically-minded"

However that's not really a "Web search engine" at all. Google, Yahoo, etc. go out and crawl every page all over the world, whereas it seems like this one doesn't/has its own proprietary data bank.

In any case, once (if) they ever get good and interesting enough, Google will probably just buy them off... (or is it ?)

Damn near impossible to get info on what a particular driver is for or what a certain registry entry does. All I get is sales pitches for sites that sell drivers or software that checks my registry.

I hear ya, but it's not the fault of the search engines as much as it is of the internet adapting to them to get more exposure.
In fact, they (at least Google) have done incredibly good job IMO filtering out BS pages whose whole purpose of existence is to get indexed near the top. I remember a few years ago a LOT of results were like that, until Google started filtering those out. Try searching for some random phone number to see what I mean. It was like that when you searched for just about anything at all, several top results were porn-related links or other useless crap. Now there's a lot more relevance and, really, hats off to them for being able to balance between useful information and junk when filtering the ENTIRE WEB. Just as an example, I just searched for the word "test", and got "about 587,000,000 results (0.25 seconds)", with the top results being reasonably relevant and "sane" (i.e. I'm sure there are a lot of "test pages" out there that have that word in the title, content, keywords, etc. but they didn't come near the top.) That's VERY impressive work IMO.