Data management problems widespread: organizations should regard data as their greatest asset--and invest in data management accordingly.

A recently published Virginia Commonwealth University (VCU) study--the first comprehensive look at how data management is practiced worldwide--reveals that most organizations do not manage information well.

And there's more: despite organizations recognizing that data is the lifeblood of their businesses, "Measuring Data Management Practice Maturity: A Community's Self-Assessment" also concludes that businesses face significant data management challenges.

The study, published in the April issue of Computer, the publication of the Institute of Electrical and Electronic Engineers Computer Society, assessed the data management practices of 175 public- and private-sector organizations between 2000 and 2006. A vast majority scored far too low for their own good.

According to lead study author Peter Aiken, associate professor in VCU's Department of Information Systems and the founding director and owner of data management consulting firm Data Blueprint, fewer than 10 percent of the organizations studied are using documented processes to manage data, which means that more than 90 percent are ineffective in this area.

Data Management Often Neglected

All organizations depend on data, and good data management practices are critical to many technology-based organizational initiatives, including business intelligence, customer relationship management, and data warehousing. Bad, incomplete, or inaccurate information has been the downfall of projects, departments, and even entire organizations.

To wit, the 2006 InformationWeek article "Hamstrung by Defective Data" cited a 2005 Gartner estimate that more than 25 percent of critical data within large businesses is somehow inaccurate or incomplete. In late 2005, the Data Warehousing Institute surveyed 750 IT professionals and business executives, and 53 percent of them said their companies had experienced problems and suffered losses or increased costs because of poor-quality data, up from 44 percent in a similar 2001 survey. (Alarmingly, 36 percent admitted they had not even studied the issue.)

Although data is a critical asset for organizations, the VCU study is an indication that organizations don't fully appreciate its value.

Aiken and other VCU researchers found that many businesses do not invest adequately in data management, treating data as a maintenance cost rather than as an asset. According to the study, that is a mistake that "is costly in terms of market share, profit, strategic opportunity, stock price, and so on."

The study's results are based on self-reporting by the organizations involved; approximately 15 percent of the organizations also participated in an in-person investigation by researchers to validate the self-assessments. Researchers tried to measure not only whether a data process was performed in an organization, but also the maturity with which the process was performed.

This is important because, as increasing amounts of data flow within and between organizations, the problems that can result from poor data management practices are becoming more common--and more serious. Studies cited by VCU research have shown that such poor practices are widespread. For example:

* PricewaterhouseCoopers reported that in 2004, only one in three organizations was highly confident in its own data, and only 18 percent were confident in data received from other organizations. Further, just two in five companies have a documented board-approved data strategy.

* According to Aiken, approximately two-thirds of organizational data managers have had formal data management training; slightly more than two-thirds of organizations use or plan to use formal metadata management techniques; and slightly less than one-half manage their metadata with computer-aided software engineering tools and repository technologies.

This is not good news, especially considering that information is only getting more difficult to manage. For example, according to the 2006 InformationWeek article, the amount of data created and maintained by organizations doubles every 12 to 18 months. Managing all that information is certainly not something that can be done half-heartedly or on the fly.

Data Management Guidelines

The VCU study aimed to investigate why so many organizational data management practices fall below expectations. Its findings suggest that organizations need a more formalized feedback loop they can use to improve their data management practices. According to the survey, organizations can use this data as a baseline from which to look for, describe, and measure data management improvements.

The study's authors provide a type of "checklist" for good data management practices. According to them, data management

* Must be viewed as a means to an end, not the end itself. Organizations must stop treating and practicing data management as an abstract discipline and start seeing it as a process that supports specific organizational objectives--specifically, one that provides a shared-resource basis on which to build additional services.

* Involves both process and policy. Data management tasks range from strategic data planning to the creation of data dement standards to database design, implementation, and maintenance.

* Has a technical component: interfacing with and facilitating interaction between software and hardware

* Has a specific focus: creating and maintaining data to provide useful information

* Includes management of metadata artifacts that addresses the data's form as well as its content

In addition, everyone in an organization--especially executives, but including every last employee--should understand the importance of effective data management to their organization. Without that understanding, there is little chance data management strategies can be successfully implemented.

Managing Data Better

According to the Virginia Commonwealth University study lead author, Peter Aiken, there are several things organizations can do to better manage their vast amounts of data or improve their current data management practices, including:

* Understand that data is an asset that someone in the organization must be a steward of.

* Realize the significant difference between data duplication (involuntary) and data replication (controlled).

* Realize data-related problems are huge, hidden sources of various forms of systems and IT failures.

* Manage metadata--if an organization doesn't understand and control its metadata (semantics as well as syntax), it has no hope of enterprise-wide data management. That requires an enterprise-wide metadata vetting data management process committee (enterprise-wide data stewardship). As a corollary, establish and maintain an ISO 11179 standard-based data registry (it's really about metadata registration).

* Establish an enterprise-wide data management program ... although Aiken said this is "politically and financially speaking, extremely difficult to accomplish."

* Enlist the support of at least the CIO, or better, the CEO--or best, both the CIO and CEO--one of whom is going to be around for at LEAST five years; without this support, Aiken said, "Forget any successful accomplishment of any enterprise-wide data management success in the long term."

Aiken stressed that the prerequisite to any serious data management steps is a "very serious and very long-term corporate commitment."

References

Aiken, Peter, M. David Allen, Burt Parker, and Angela Mattia. "Measuring Data Management Practice Maturity: A Community's Self-Assessment." Computer, April 2007. Available at /www.computer.org/portal/site/computer/menuitem.5d61cld 591162e4b0ef1bd108bcd45f3/index.jsp?&pName=computer_level1_article&TheCat= 1005&path=computer/homepage/April07&file=cover.xml&xsl=article.xsl& (accessed 31 July 2007).

VCU News Services. "Study Reveals Widespread Data Management Problems." Press Release, 5 April 2007. Available at www.news.vcu.edu/news.aspx?v=detail&nid=2015 (accessed 31 My 2007).

Whiting, Rick. "Hamstrung by Defective Data"' InformationWeek, 8 May 2006. Available at www.informationweek.com/showArticle.jhtml;jsessionid=DVG2W21TD SW5OQSNDBOCKICCJUMEKJVN?articleID=187200771&queryText=Hamstrung +By+Defective+Data (accessed 31 July 2007).

Nikki Swartz is a freelance writer based in Kansas City, Missouri, and former Associate Editor of The Information Management Journal. She may be contacted at nikkiswartz@hotmail.com.