What has information overload done to us?
The search engine Bing would have us believe that we are all just a moment away from starting a food fight in the supermarket produce section. Even if the fruit doesn't start flying, experts agree that society is almost at critical mass when it comes to the amount of available data. But what are we to do with all of this information? Enter data scientists - the professionals responsible for filtering out the noise and analyzing essential information.
The emerging world of competitive intelligence
Data scientists are an integral part of competitive intelligence, a newly emerging field that encompasses a number of activities, such as data mining and analysis, that can help businesses gain a competitive edge. Ken Garrison, CEO of the industry group Strategic and Competitive Intelligence Professionals (SCIP), explains, "The field involves collecting data, analyzing it and delivering the data as intelligence that is actionable."
Data collection can be done either internally or externally, but competitive intelligence professionals stress that their collection techniques are legal and ethical.
Competitive intelligence is poised to offer data scientists increasing job opportunities in coming years. SCIP reports that the market for business intelligence is worth approximately $2 billion annually, and Garrison says that many corporations now operate their own competitive intelligence divisions.
The use of competitive intelligence by data scientists can pay big dividends to businesses who invest in these services. A May 2011 study by McKinsey Global Institute suggests that retailers analyzing large data sets to their fullest could increase operating margins by 60 percent and the health care industry could lower annual costs by 8 percent or $200 billion.
However, the study also warns there is a significant shortage of qualified workers to analyze these data sets adequately. According to the report, a shortfall of about 140,000 to 190,000 individuals with analytical expertise is projected by 2018. The study also predicts a need for an additional 1.5 million managers and analysts by that same date to fully engage the true potential of the currently available data.
Data scientists in the information age
While it may be conventional wisdom that data is growing exponentially, the actual amount of that growth can be staggering to consider. A 2003 study conducted by the University of California Berkley found that worldwide information production increased 30 percent each year from 1999 until 2002. In 2010, then-Google CEO Eric Schmidt turned heads at the 2010 Techonomy Conference when he said people currently create as much data every two days as was previously created in all of history up to 2003.
Some observers question whether Schmidt's analysis was correct, but no one doubts that the world is awash in new data. Everything from the latest scientific research to Twitter feeds highlighting last night's dinner are contributing to the data deluge. To sort through the massive quantity of information, businesses are increasingly turning to data scientists who have the skills to pinpoint what is valuable and leave the rest behind.
"The question facing every company today, every startup, every non-profit, every project site that wants to attract a community, is how to use data effectively," writes Mike Loukides, Vice President of Content Strategy for O'Reilly Media, on the O'Reilly radar website. He adds, "not just their own data, but all the data that's available and relevant."
With so much data needing analysis, it is no wonder careers in this field are expected see significant growth. The University of California San Diego Extension lists data mining and analytics as the second hottest career for college graduates in 2011. Even the Cheezburger Network, home of the web's infamous LOLCats, recently brought a data scientist on board.
Finding work as a data scientist
As part of a relatively new field, data scientists may come from many different backgrounds. Garrison says that employers are often looking for two things when considering a job applicant. "The first part is the technical background," he says. Companies may want professionals with an industry background who are familiar with its specific jargon and trends. "If you want to work for a pharmaceutical company, you might need a degree in biochemistry," he explains. Other jobs may require only a general degree in business.
In addition to the technical expertise, data scientists and competitive intelligence professionals also need to know where to find data and how to analyze it. Some colleges and universities offer graduate degrees or certificate programs in specialties such as data mining and data analysis. Professional groups such as SCIP also provide training opportunities for members.
Since data scientists spend a significant amount of time using computer programs and algorithms, it may seem logical that a computer science degree would be preferable for these professionals. However, many argue that a degree in physics makes more sense. Loukides writes that physicists not only have mathematical and computing skills but also an ability to see the "big picture."
Daniel I. Shostak, President of Strategic Affairs Forecasting, has been tracking changes in the field of analytics for several years and says that those interested in working as a data scientist need more than just computer skills. "[They] need to demonstrate very good communication skills because many folks are very skeptical about the value of data driven analysis," he said. In addition, Shostak suggests that potential job candidates become proficient in the statistical language R and have experience working with computer networks since they are often an integral part of working with large data sets.
As a hot new career, the government has yet to begin tracking data scientist occupational information. However, the Bureau of Labor Statistics (BLS) reports that demand for operations research analysts, who provide some similar services, is expected to jump 22 percent from 2008-2018.
Data scientists with the skills necessary to fill these positions can expect to earn healthy salaries. Operations research analysts, who increasingly hold a master's degree and have advanced math skills, earned a mean annual wage of $76,980 in 2010, according to the BLS.
Glassdoor.com, a site which allows workers to self-report salaries, shows average data mining scientist salaries ranging from $60,000 to $115,000.
While new graduates with the skills to manipulate big data sets may need years of training, the McKinsey report suggests that current professionals can get training in data analysis as well to help meet the growing demand for this hot job.