The movement of large portions of life to a more digital space has spawned a huge influx in data collection. Government agencies and private businesses alike are gathering massive amounts of information in a trend that has been dubbed big data. And yet, many of these organizations are struggling under the weight of all this raw data.
Let's take a look at the biggest challenges facing big data that are preventing government agencies from fully utilizing it:
Big data is just too big
The main problem with these collections of information is the fact that there is entirely too much data to reasonably comb through. FedTech contributor Calvin Hennick stated that the EPA expects many sectors within the agency to begin seeing more than 100 million data points collected annually.
The first problem this wave of information poses is in deciding what to do with it all. The 2010 federal data center consolidation initiative is pushing government agencies to centralize their IT needs in order to cull the massive data sprawl that had been created in the previous few years. While this effort has proven worthwhile due to decreases in costs and confusion, it's also cut the number of facilities government agencies can rely on in terms of data storage.
This is a big issue because data collection is only going to increase in the years to come. Cisco's visual networking index predicted that by year's end 2016, global IP traffic will surpass a zettabyte annually. In fact, the report also found that by 2019, traffic is expected to hit an astounding 2 zettabytes every year. That's an incredible amount of information, and collecting and storing even a small portion will continue to prove troublesome.
Big data's size affects analysis, too
The other major challenge agencies must overcome if they wish to utilize big data has to do with analysis. One area that is expected to benefit from big data analytics is the financial sector. Fraud is a problem with every industry, but Wall Street is a particular problem area for the government. The Securities and Exchange Commission could very well use big data to help catch some of these criminals, but CIO Pamela Dyson said in FedTech that the agency is a long way off from being able to comb through the massive amount of information.
"If you can analyze more data faster, you can do deeper examinations or you can examine more registrants," said Dyson. "But there's a lot of data to collect, and right now, the ability and capability of our systems restrict us.