Making the Most of Web Data


New York – As research becomes more and more primary-data intensive, researchers are attempting to employ advanced technologies aimed at getting value out of internet data. Internet data is becoming commoditized over time at an accelerating rate. Tools are being developed to assess textual data and attempt to determine whether products or services are being talked about and whether the opinions are positive or negative, and gathering steam or dissipating. Firms collect information from blogs and forums and other social media sites and run it through quantitative and/or qualitative filters to glean information about what is hot and what is not.

While the data scrappers, feed-takers and analysis firms have typically been lesser know names, some of the big boys are getting into the space. One particularly well thought out approach is called Yahoo! Research. Yahoo! Research is a group of scientists that are focused on data-driven analysis, improved search, algorithms and economic models.

The functional areas of the research include: Search & Web Scraping – utilizing algorithms, data mining, natural language techniques and data processing to gain information; Machine Learning – utilizing computer science techniques, statistics, mathematical optimization to help computers learn abstraction, patterns, probability distributions, etc. from web data; Microeconomics and Social Systems – algorithmic games theory, economics, econometrics, psychology and ethnography; Computational Advertising – a discipline aimed at combining the above analysis to suggest the best advertisement to present to a particular web demographic and; Web Information Management – development of infrastructure to maintain and Analyze next-generation web information and software services.

The team at Yahoo! Research, by virtue of its multi-disciplinary approach, is likely to produce some interesting results. Having been trained as an economist, I learn towards the Yahoo! approach. This approach is not just an approach that finds the strongest relationships within a dataset, but rather imposes a theoretical construct prior to testing that theory. All important finance theory results have been developed in this way. As such, the team at Yahoo! presents and exciting approach to gaining greater insight from the ocean of information sloshing around the internet.


About Author

Leave A Reply