IT organizations will need to transform their existing relationships with business units
A white paper released today from IDC revised the research firm's earlier estimates to show that by 2011, the amount of electronic data created and stored will grow to 10 times the 180 exabytes that existed in 2006, reflecting a compound annual growth rate of almost 60%.
By 2011, there will be 1,800 exabytes of electronic data in existence, or 1.8 zettabytes (an exabyte is equal to 1 billion gigabytes). In fact, the number of bits stored already exceeds the estimated number of stars in the universe, IDC stated. And because data is growing by a factor of 10 every five years, by 2023 the number of stored bits will surpass Avogadro's number, which is the number of carbon atoms in 12 grams, or 602,200,000,000,000,000,000,000 (6.022 x 10^23).
See also: "From bits to yottabytes. How much data is that?"
The study, "The Diverse and Exploding Digital Universe," also found that the rate at which electronic containers for that data — files, images, packets and tag contents — is growing 50% faster than the data itself. And in the year 2011, data will be contained in more than 20 quadrillion — 20 million billion — of those containers, creating a tremendous management challenge for both businesses and consumers.
The study is an updated forecast to a white paper released last March.
Along with the new white paper, which was sponsored by EMC Corp., IDC created a "Personal Digital Footprint Calculator," which resides on the same Web page as EMC's worldwide digital growth-tracking ticker, which counts the amount of data created second by second.
"Obviously, none of us will relate to a number with 21 zeros at the end. The way that the ticker is practically blurred at the far end we hope will give people an idea of the pace — of how fast this is happening. In a way, this is the message of the study," John F. Gantz, IDC's chief research officer, said in a recorded interview.
IDC also acknowledged that it underestimated earlier digital data figures for 2007, saying the actual amount of data — 281 exabytes — is 10% greater than it had previously forecasted in the first "Digital Universe" study (see "A zettabyte by 2010: Corporate data grows fiftyfold in three years"). IDC said the bigger numbers were the result of faster growth in digital cameras and televisions, as well as a better understanding of data replication.
The diversity of the digital data that's producing massive growth runs the gamut, from 6GB movies on DVD to 128-bit signals from RFID tags.
Less than half of the digital data being created by individuals can be accounted for by user activities, such as photos, phone calls and e-mails. The majority of the data is made up by what IDC calls digital "shadows" — including surveillance photos, Web search histories, financial transaction journals and mailing lists.
Gantz said the impact of the tremendous data growth — even if the data is not being created within the company's four walls — will directly affect CIOs. A CIO in a data center that has been crunching numbers in IBM format for 40 years "may be surprised what will happen" when all of a sudden all the voice calls in the enterprise are stored on disk and all the video cameras that used to be closed-circuit TVs are now digital surveillance cameras forcing you to deal with video files, Gantz said.
Also, much of the data being created by consumers outside of an enterprise's four walls will still force that company to protect the data. "At some point in the life of every file, or bit or packet, 85% of that information somewhere goes through a corporate computer, Web site, network or asset," Gantz said. "The corporation at that point in time has responsibility for that information. This is something that can be lost on the CIOs of the enterprises."
"It's not until Viacom sues Google for a billion dollars for the information that its customers are putting on these basically rented electronic storage space[s], and all of a sudden Google has got to fight a lawsuit," Gantz said. "Whether they win or lose, it's hundreds of millions of dollars in lawyer fees. So the enterprise is picking up responsibility for information that's not actually created at the hub of the enterprise."
To deal with this explosion of the digital universe in size and complexity, IT organizations will face three main imperatives:
1. They will need to transform their existing relationships with the business units. It will take all competent hands in an organization to deal with information creation, storage, management, security, retention and disposal in an enterprise. Dealing with the digital universe is not a technical problem alone.
2. They will need to spearhead the development of organizationwide policies for information governance: information security, information retention, data access and compliance.
3. They will need to rush new tools and standards into the organization, from storage optimization, unstructured data search and database analytics to resource pooling (virtualization) and management and security tools. All will be required to make the information infrastructure as flexible, adaptable and scalable as possible.
Also adding to the deluge of digital data being created is the worldwide changeover in the next several years from analog TV systems to digital. According to IDC, the number of digital TVs in the world doubled last year and should surpass 500 million by the end of 2011.
Digital growth's impact on the environment
IDC estimates that the amount of electronic waste being created alongside data is already accumulating at more than 1 billion units a year. Most of it comes from mobile phones, but personal digital electronics and PCs are also greatly impacting the environment.
As the world changes over to digital televisions, analog sets and obsolete set-top boxes and DVDs "will be heaped on the waste piles, which will double by 2011," IDC said. How much power will be consumed by the added data glut is more difficult to determine, but manufacturers developing power-saving chips and users installing power-saving systems, including new cooling and management systems, should help. According to a 2006 study by IDC, server power and cooling costs are escalating rapidly as more powerful servers are being introduced. "Power consumption that was 1 kW per server rack in 2000 is now closer to 10 kW," the study found. "Customers building new data centers are planning for 20 kW per rack."
The main findings of the "Digital Universe" study:
* The digital universe is bigger than we thought: 281exabytes in 2007 — 10% bigger than we said it would be last year. Digital cameras and security cameras out-shipped our expectations.
* The digital universe will grow tenfold in five years.
* In 2007, the digital universe contained more bits (computer 1s and 0s) than there are stars in the physical universe.
* While 70% of the information in the digital universe is created by individuals, enterprises have some responsibility or liability for 85% of it (think of Google getting sued by Viacom for $1 billion because of the videos uploaded by consumers).
* There is now more information created in a year than there is capacity to store it.
* Responsibility for the information in the digital universe by industry does NOT match IT spend or GDP distribution by industry.
* A single e-mail with a 1MB attachment sent to four colleagues can generate 50MB of information in the digital universe (based on an IDC-like architecture).
A white paper released today from IDC revised the research firm's earlier estimates to show that by 2011, the amount of electronic data created and stored will grow to 10 times the 180 exabytes that existed in 2006, reflecting a compound annual growth rate of almost 60%.
By 2011, there will be 1,800 exabytes of electronic data in existence, or 1.8 zettabytes (an exabyte is equal to 1 billion gigabytes). In fact, the number of bits stored already exceeds the estimated number of stars in the universe, IDC stated. And because data is growing by a factor of 10 every five years, by 2023 the number of stored bits will surpass Avogadro's number, which is the number of carbon atoms in 12 grams, or 602,200,000,000,000,000,000,000 (6.022 x 10^23).
See also: "From bits to yottabytes. How much data is that?"
The study, "The Diverse and Exploding Digital Universe," also found that the rate at which electronic containers for that data — files, images, packets and tag contents — is growing 50% faster than the data itself. And in the year 2011, data will be contained in more than 20 quadrillion — 20 million billion — of those containers, creating a tremendous management challenge for both businesses and consumers.
The study is an updated forecast to a white paper released last March.
Along with the new white paper, which was sponsored by EMC Corp., IDC created a "Personal Digital Footprint Calculator," which resides on the same Web page as EMC's worldwide digital growth-tracking ticker, which counts the amount of data created second by second.
"Obviously, none of us will relate to a number with 21 zeros at the end. The way that the ticker is practically blurred at the far end we hope will give people an idea of the pace — of how fast this is happening. In a way, this is the message of the study," John F. Gantz, IDC's chief research officer, said in a recorded interview.
IDC also acknowledged that it underestimated earlier digital data figures for 2007, saying the actual amount of data — 281 exabytes — is 10% greater than it had previously forecasted in the first "Digital Universe" study (see "A zettabyte by 2010: Corporate data grows fiftyfold in three years"). IDC said the bigger numbers were the result of faster growth in digital cameras and televisions, as well as a better understanding of data replication.
The diversity of the digital data that's producing massive growth runs the gamut, from 6GB movies on DVD to 128-bit signals from RFID tags.
Less than half of the digital data being created by individuals can be accounted for by user activities, such as photos, phone calls and e-mails. The majority of the data is made up by what IDC calls digital "shadows" — including surveillance photos, Web search histories, financial transaction journals and mailing lists.
Gantz said the impact of the tremendous data growth — even if the data is not being created within the company's four walls — will directly affect CIOs. A CIO in a data center that has been crunching numbers in IBM format for 40 years "may be surprised what will happen" when all of a sudden all the voice calls in the enterprise are stored on disk and all the video cameras that used to be closed-circuit TVs are now digital surveillance cameras forcing you to deal with video files, Gantz said.
Also, much of the data being created by consumers outside of an enterprise's four walls will still force that company to protect the data. "At some point in the life of every file, or bit or packet, 85% of that information somewhere goes through a corporate computer, Web site, network or asset," Gantz said. "The corporation at that point in time has responsibility for that information. This is something that can be lost on the CIOs of the enterprises."
"It's not until Viacom sues Google for a billion dollars for the information that its customers are putting on these basically rented electronic storage space[s], and all of a sudden Google has got to fight a lawsuit," Gantz said. "Whether they win or lose, it's hundreds of millions of dollars in lawyer fees. So the enterprise is picking up responsibility for information that's not actually created at the hub of the enterprise."
To deal with this explosion of the digital universe in size and complexity, IT organizations will face three main imperatives:
1. They will need to transform their existing relationships with the business units. It will take all competent hands in an organization to deal with information creation, storage, management, security, retention and disposal in an enterprise. Dealing with the digital universe is not a technical problem alone.
2. They will need to spearhead the development of organizationwide policies for information governance: information security, information retention, data access and compliance.
3. They will need to rush new tools and standards into the organization, from storage optimization, unstructured data search and database analytics to resource pooling (virtualization) and management and security tools. All will be required to make the information infrastructure as flexible, adaptable and scalable as possible.
Also adding to the deluge of digital data being created is the worldwide changeover in the next several years from analog TV systems to digital. According to IDC, the number of digital TVs in the world doubled last year and should surpass 500 million by the end of 2011.
Digital growth's impact on the environment
IDC estimates that the amount of electronic waste being created alongside data is already accumulating at more than 1 billion units a year. Most of it comes from mobile phones, but personal digital electronics and PCs are also greatly impacting the environment.
As the world changes over to digital televisions, analog sets and obsolete set-top boxes and DVDs "will be heaped on the waste piles, which will double by 2011," IDC said. How much power will be consumed by the added data glut is more difficult to determine, but manufacturers developing power-saving chips and users installing power-saving systems, including new cooling and management systems, should help. According to a 2006 study by IDC, server power and cooling costs are escalating rapidly as more powerful servers are being introduced. "Power consumption that was 1 kW per server rack in 2000 is now closer to 10 kW," the study found. "Customers building new data centers are planning for 20 kW per rack."
The main findings of the "Digital Universe" study:
* The digital universe is bigger than we thought: 281exabytes in 2007 — 10% bigger than we said it would be last year. Digital cameras and security cameras out-shipped our expectations.
* The digital universe will grow tenfold in five years.
* In 2007, the digital universe contained more bits (computer 1s and 0s) than there are stars in the physical universe.
* While 70% of the information in the digital universe is created by individuals, enterprises have some responsibility or liability for 85% of it (think of Google getting sued by Viacom for $1 billion because of the videos uploaded by consumers).
* There is now more information created in a year than there is capacity to store it.
* Responsibility for the information in the digital universe by industry does NOT match IT spend or GDP distribution by industry.
* A single e-mail with a 1MB attachment sent to four colleagues can generate 50MB of information in the digital universe (based on an IDC-like architecture).