As the Internet continues to grow, popular search engines like Google may begin having trouble searching through the more than 900 billion existing Web pages. To make things worse, many of these pages are not linked by other more modern Web sites, giving Google no reference for any of them.
For one Binghamton University professor, the solution could be around the corner: a “meta-search” engine.
“A meta-search engine has a similar interface to other search engines,” said Weiyi Meng, a computer science professor at BU. “But the way it processes queries is very different. It queries other search engines and then organizes the results, utilizing the other search engines to put [the results] together.”
Meta-search engines have been in development since 1994, and several have already been released to the public. Meng helped develop one such engine, AllinOneNews.com, which was released in August 2007.
“If you go there [AllinOneNews.com.com],” Meng said, “you will find a meta-search engine for news articles.”
AllinOneNews.com has a catalog of news articles from 1,800 search engines spanning 200 different countries. It has news stories from as far back as 1989.
Meng was the co-founder of the meta-search engine project, which began in 1997. About 25 people have assisted the professor in its development, some of whom were students who have since graduated. Students from the University of Illinois at Chicago and the University of Louisiana at Lafayette have also researched the project.
“So far we have received funding from investors and rich capitalists, and we have received a grant from the professional science foundation for $600,000 total,” Meng said.
Vijay Raghavan, a faculty member of the University of Louisiana who is working on the project, said there may be troubles for older engines, should the Web continue to grow.
“A company like Google may run out of resources to crawl all of those pages,” Raghavan said. “We won’t have that problem. We will scale much better.”
Everyone from students to businessmen to the government, will be able to use meta-search engine technology to their advantage. Meta-search engines can be used for research, fact checking or just simple questions.
“People can use it to find collaborators,” Meng said. “It could also help prospective students find the programs they’re interested in.”
However, meta-search engines still face challenges. Since any given search would query a vast number of other, smaller search engines, it will take time to program them to determine which smaller search engine is best for a certain query. On top of this, the meta-search engines must automate this interaction and then organize the search results.
“There are already some search engines that compete with Google,” Meng said. “But they are not as popular as Google. The direction we are going is very different; we are trying to connect smaller search engines together. That’s why we think this approach will take much more effort and will get more publicity.”