Today, IBM announced that it is acquiring Initiate Systems.
This was widely rumored last week, but the announcement of Informatica’s acquisition of Siperian took my mind off this temporarily.
On the face of it, it makes all the sense in the world. IBM knows a good product when it sees it, and Initiate has been doing well in the MDM world lately, particularly in the healthcare vertical, where it grew up, and in the public sector vertical. IBM’s press release explicitly mentions Initiate as a leader in “data integrity software for information sharing” among healthcare and government organizations. I thought it was interesting that the IBM release didn’t mention the terms “master data management” or “MDM” even once.
I was a little surprised that IBM’s release made no mention of the financial terms, since IBM is a public company, but I’m sure it will only be a matter of time before those details become available to those who know where to look or whom to ask.
It wasn’t a surprise to see the IBM release mention the stimulus funding being invested around the globe. When I first saw the rumors last week, I immediately thought – IBM is buying Initiate to be better prepared for the various e-Healthcare initiatives that are coming down the pike.
Where things may get a bit tricky is explaining the multiple MDM platforms from IBM to potential customers, and managing several different development roadmaps and product portfolios. There’s the IBM InfoSphere MDM Server (the former DWL product) and there’s also IBM InfoSphere MDM Server for Product Information Management (the former Trigo product). And now there’s the Initiate product too.
While the acquisition does make sense, there is an “embarrassment of riches” factor. IBM will, of course, develop a sales playbook explaining what situations at what type of customer are a good fit for each product.
I think the lingering feeling I have with Initiate Systems is that it may be headed for a “golden ghetto” at IBM – never to reach its full potential as a solution across many different industries, and eventually to handle many different domains of master data. IBM may (and rightly so, in its mind) pigeonhole it into the healthcare and government verticals.
But Initiate’s had some good success outside those two industries. In the Financial Services vertical, they’ve got customers like Capital One Financial, Countrywide Financial (now Bank of America), eSure Insurance, and Wells Fargo. In the Hospitality industry, they’ve got Choice Hotels. In manufacturing, they’ve got Mitsubishi Motors Australia. In the Logistics vertical, they’ve got Federal Express. In the retail sector, Barnes & Noble, CVS, Longs Drug Stores and SuperValu are all customers. And in the high tech space, they’ve got Dell, Ingenix, Intuit, LocatePLUS, Microsoft and National Instruments.
Unfortunately, they didn’t achieve enough critical mass in any of these other verticals to compete with the strong momentum they’d developed in healthcare and government.
As I said last week, these are interesting times in the MDM world. The recent M&A activity, the healthy demand from large and medium sized corporations, the large number of consultants from other areas claiming to now have experience in MDM – these are all signals to me of a large and fast-growing market. So the New Year, for those of us in the MDM space, is off to a good start.
Siperian, one of the last best-of-breed providers of master data management (MDM) technology, is being acquired by Informatica.
The two firms were already working together closely, having an alliance and OEM relationship through Informatica’s acquisitions in 2008 of Identity Systems (for entity resolution and matching) and in 2009 of Address Doctor (for customer address cleansing).
This will strengthen the Siperian product further by bringing Informatica’s technology even more tightly into the Siperian MDM Hub.
At the same time, it eliminates the “company viability” question mark that sometimes gets raised in large IT shops’ minds when evaluating enterprise software vendors. When a Fortune 500 company is evaluating a smaller company, they sometimes wonder how long a company like Siperian can last against companies like IBM, Oracle and SAP. I’ve never been a big fan of that argument, since some of the best software gets created at small and medium-sized companies, but there’s no doubt it’s a obstacle to be overcome with the larger enterprises. Now, it shouldn’t be an issue.
As a Siperian partner, Hub Designs is excited about this acquisition. Based on the information we’ve got at this point, it seems like a good thing for Siperian’s customers, products, shareholders, partners and people. In today’s economic climate, dreams of a big IPO (for any venture-backed technology company) are unlikely, so an acquisition by a well-run larger company is a good outcome.
I know a lot of the people at Siperian personally, and have worked closely with them over the last few years. I hope the people at Informatica realize what a strong team they are getting in this acquisition, and do everything they can to hang onto them all.
I do suggest they stop using the term “MDM Infrastructure” though (which appeared 5 times in Informatica’s press release announcing the acquisition). First, it’s not accurate – MDM projects typically need to be drive by the business to be successful, so they can’t and shouldn’t be thought of as “IT Infrastructure” projects. Secondly, from a marketing perspective, “infrastructure” is about as exciting as mud – it’s hard to get senior management excited about spending money on something with the word “infrastructure” in the name.
As for the acquisition’s impact on the rest of the MDM market, it’s still growing pretty quickly, but the number of players is shrinking. So I think we’ll see it become even more competitive, and with Informatica now becoming a strong player in the MDM hub market, that’s got to cool its relationship with Oracle, who selected Informatica as an OEM component of its Oracle Fusion MDM hub.
IBM is rumored to be acquiring Initiate Systems, which is an interesting play in its own right, especially given the expected growth in spending in the e-healthcare space over the next few years.
And SAP continues to improve its products slowly but steadily, while D&B/Purisma is doing some interesting things with web services access to the D&B central database of information on businesses.
As for the remaining independent MDM vendors, like Orchestra Networks and Kalido, or Product Information Management (PIM) solutions like Stibo and Riversand, they should see this as further validation of the strength of the MDM market. Kalido feels that it’s the only independent MDM provider who can manage every master data domain. That may be true. I plan on learning more about Kalido over the next few months.
So like the Chinese curse, “may you live in interesting times”, the beginning of 2010 promises to be interesting for all of us in the MDM business!
If you’d like to continue the discussion on the “Impact of Informatica’s Acquisition of Siperian”, click http://ning.it/aJ1Xj5.
The Hub Designs Blog welcomes Part 4 of this series by Rob DuMoulin, an information architect with more than 26 years of IT experience, specializing in master data management, database administration and design, and business intelligence.
Part 4: Profiling Relationships and Patterns
This is part four of a five-part series describing how data profiling assists in all aspects of system development, from design through deployment.
Part One introduced different perspectives on data profiling. Part Two identified valuable system and entity metrics to track. Part Three discussed attributes. In this segment, we dive deeper into attribute relationships and pattern recognition. Also, we expand on primary key identification discussion and discuss hidden relationships.
Pattern grouping provides a mask of distinct format patterns within an attribute data set and a count of the number of occurrences. Patterns give insight into the type of values found in an attribute. For example, a numeric pattern analysis may show values such as 999.99999, 99, or -.9999.
Observing distinct patterns gives insight into the maximum digits and precision, and also domains such as integer or real. Pattern of a database date or date-time type provides unremarkably similar patterns for all dates. Because the database management system typically enforces the domain, date analysis provides no value and can be ignored. If dates are stored in character format, however, patterns quickly show variations in date formatting. Character patterns only have significance to a limited number of positions. It makes no sense to pattern a description field of 200 or 2000 characters. Smaller code attributes of less than 10 characters though do provide value. Ignore pattern profiling for character strings over 20 characters at first, then refine to shorter character strings if the results do not add value.
In pure database theory, referential integrity (RI) is your friend. In practice, designers and software vendors often forgo RI to improve system performance on data inserts. These designers place the data quality burden on the application and do not endorse external data manipulation outside the application interfaces. In the real world, though, data corruption occurs and without RI or routine data quality checks, corruptions may not be found for a long time or not at all. Personally, I have identified over $50,000 of recent orphaned sales in a retail client resulting from deliberately disabled RI. These unreported sales were not added to the ledger and were allowed to occur for performance reasons until I found them through simple profiling. Enforcement of RI is a topic for another discussion but is mentioned here because it does identify a valid reason for data profiling.
In even presumably good relational designs, some parent-child relationships are not enforced for different reasons. First, interrogate the RI listed in the system catalogs to identify all enforced relationships. Reverse-engineering a system with a good modeling tool is probably the best way to do this. A harder and more valuable analysis is to identify unenforced relationships and determining the probability of the relationship if not all values are an exact match. Do this by counting all the candidate child attribute values that exist within a known parent attribute table. If all match and there are a non-trivial number of matches, there is a good probability of a non-identified relationship. A small number of mismatches could identify data quality issues.
In Part 5, we tie all the techniques discussed in the first four parts together to show the value of a repeatable data profiling process.
Continue with Part 5 or go back to Part 3.
The Hub Designs Blog welcomes Part 3 of this series by Rob DuMoulin, an information architect with more than 26 years of IT experience, specializing in master data management, database administration and design, and business intelligence.
Part 3: Attribute-Level Analyses
This is part three of a five-part series on data profiling.
In Part One, we took a light-hearted view of where profiling benefits an organization and in Part Two, we discussed the fundamentals of a profiling strategy. The remaining three parts discuss attributes, relationships, patterns, and how to use the combined data profiling information you collect. In this section, we introduce attributes, the lowest-level components of a profiling effort.
An attribute is simply a individual data element. Alone, an attribute has no context. Given the simple descriptor of “Cost” for an attribute tells us very little about the attribute’s true purpose and immediately drives a need for additional information, such as units (hours, Dollars, Euros…), type (weighted, unit, gross…), and use (invoice, sum, average…). Attributes therefore must be analyzed within the context of their business purpose to have meaning.
Some characteristics require business knowledge to define and others can be determined through interrogation of existing values and underlying rules of the storage medium. It takes both analyses to get a complete picture of information within a system. While assembling this puzzle, though, keep in mind that until you validate the enforcement of business rules, only assumptions can result from physical profiling or business context.
Analyses of values, domains, and constraints allows insight into use (or abuse) of an attribute. The larger the sample size, the better confidence you gain in the results. Without explicit proof of business rule enforcement, though, you must assume that just because a value does not presently exist does not mean it cannot exist. Business rules are defined by business experts and enforced through database constraints, data type/precision, and application code. Knowing the methods of enforcement allow you to narrow a domain but not totally understand it. Profiling of actual values provides additional refinement in terms of percentage of NULL values, percentage of distinct values, minimum, maximum, and average values, top x and bottom x recurring values along with their counts, and minimum, maximum, and average data lengths.
Some attributes within a data set serve valuable purposes that are important to identify. Attributes that individually or in conjunction with others define uniqueness of the data set also may support relationships between entities. Uniqueness can be further classified as being either members of a system-enforced primary key or of a business key (outside of the defined primary key). System-enforced primary keys are relatively easy to define within a database system through interrogation of the system catalog. Business keys that exist in tables in addition to a primary key may be more difficult to identify, especially if more than one attribute is needed to define uniqueness.
Attribute-level information of interest includes: data type (size and precision), the number and percent of NULL values, column descriptions, number and percent of distinct values, and the minimum-maximum-average values and lengths. Uses of the system catalog provides some of this information, but others must be collected through sampling the data.
Other types of attributes that may help in identifying relevancy are those that provide system-level auditing or change control. Knowing which attributes fill these roles may either allow you to (a) ignore them for profiling purposes or (b) use them to help explain versions or data anomalies.
Part 4 expands on attribute profiling with the introduction of relationships and patterns.
Continue with Part 4 or go back to Part 2.
The Hub Designs Blog welcomes Part 2 of this series by Rob DuMoulin, an information architect with more than 26 years of IT experience, specializing in master data management, database administration and design, and business intelligence.
This discussion is the second of a five-part series on data profiling. In Part 1, we discussed the project roles that benefit from data profiling and how better understanding information results in more reliable information systems. Important goals of any profiling strategy include automation of metric collection and socializing results to support the differing objectives of a data-centric project.
Early in a system development life cycle, profiling helps define sources, data storage requirements, and data transformations. As a system goes into production (or if profiling is added to an existing system for quality control purposes), routine profiling is useful to audit system quality and business rule enforcement. The frequency of collection and amount of effort you expend to automate your profiling methods should be based on the ability of the organization to benefit from the profile results.
This section discusses the beginnings of a profiling effort. Information assembled here forms the foundation of other profiling activities. For this discussion, consider a Profile Group as a set of information sharing a common purpose and data management methods. Examples of profile groups include tables within a single database schema or a group of spreadsheets with the same format but each spreadsheet representing a different time slice of data.
The underlying System managing a set of information within the profile group may be a named relational database, a file system directory, or even a web site being accessed through web services. The reason we abstract information into Systems is to group the information into distinct governance methods common to the underlying information. Relevant metadata and governance methods we track at the system-level include: technical contacts, backup schedules, system descriptors, connection strings, business unit owners, and host operating systems. System-level metadata common to a profile group helps us understand and troubleshoot future analyses. This level of information also provides developers with an understanding of inherent restrictions (or freedoms) they may encounter when trying to use or integrate the information.
Entities within a profile group belong to the same system, may have a common unique identifier, and, for database entities, have the same schema owner. Typically, entities are database tables, but may also be similar files or spreadsheet tabs containing like attribute lists. For entities, we track characteristics common to all the attributes they contain. These include: row counts, entity-level descriptors, growth characteristics (size and frequency), last analyzed date, and various customized indicators such as active/inactive, existence of change data management attributes such as insert/update timestamps, and existence of audit traceability indicators such as insert/update username.
The combination of system and entity level profiling supply the foundation for the attribute-level profiling, which is where physical information in a system resides. It also provides valuable metadata to classify information and allows for future correlation of like information across systems. Assembly and publication of entity and system level information benefits the various consumers of the information by providing a centralized “master” source of contact and context information.
In Part 3, we will dive into the attribute level analyses around data profiling.
The Hub Designs Blog welcomes another guest post by Rob DuMoulin, an information architect with more than 26 years of IT experience, specializing in master data management, database administration and design, and business intelligence.
Part 1: The Psychology of Data Profiling
Swiss psychologist Carl Gustav Jung founded the Analytical School of Psychology. His word association theories form the basis of the Myers-Briggs Type Indicator Assessment test to identify career aptitude in today’s high school students. Dr. Jung’s approach assigned personality profiles based on how an individual’s thoughts associated to various phrases. By analyzing responses, he could understand how an individual viewed the world around them and perceived themselves. Typically, subjects are asked to speak the first thought entering their minds after hearing a trigger phrase. For the following example, remember, there are no wrong answers. If I say the words “Data Profiling”, what is the first thing you think of?
If you thought of food, cats, country music, CSI NY, or residential plumbing, you are either not in IT or are an IT Manager.
If your first thought was “Quality Assurance”, you align yourself with data quality professionals having anti-social thoughts of failing test cases and sadistically reporting lazy developers for buggy code. You gleefully scour test cases looking for any evidence of truncation, missing values, non-matching codes, numeric precision errors, and inconsistent abbreviation, text, and date formatting.
If “Integration” comes first in your mind, past legacy integration projects have scarred you with a disdain for source system data quality levels. You view production apps with contempt and loathe the time it takes to track down data issues caused by system integrations. You investigate upstream sources to create detailed mappings and transformation rules. Typical debugging sessions consist of validating relationships to identify orphaned data, identifying attributes that contain overloaded columns (attributes containing more than one distinct data element), or fixing format errors from implied decimals.
Some of you responded with “Value Domains” or “Data Types”, indicating you are obsessive compulsive data architects compelled to organize the world into strict and orderly fashion with some degree of normalization, though you are not considered “normal” by your peers. Your concerns lie in understanding and regulating naming conventions, relationships, existence of NULL or default values, and understanding the meaning of each data element to accurately identify business rules and when two or more objects are related or redundant.
Lastly, if “Debugging” is the first item in your thought queue, you are a coder justifying why presumably good code is not working. Extreme paranoia has taught you to assume nothing about data quality, so you add tests to identify duplicates, validate relationships, enforce business rules, track change data capture, provide substitute values. Your phobia of early morning phone calls cause you to add auditing to your code to inform a DBA of data issues rather than waking you up in the middle of the night.
It is truly amazing how much we can conclude from the response to one simple phrase.
As stated before, there are no wrong answers. Aside from the innocent jab at Managers and non-IT resources, we all realize the benefits of information quality and absolutely need business involvement to understand context and domains of business information. The meaning and actions of Data Profiling change both by role and by project phase. Through profiling, we are able to identify best sources of information, learn proper ways to categorize and store it, reactively identify quality issues, and proactively define business rules to prevent future issues.
Identifying what is important to profile, when and how profiling is done, and how to share our findings across business and project resources is key. Done properly, profile results integrate to a master metadata repository and are periodically refreshed through an automated process.
This five-part series provides a tool-agnostic approach to comprehensive data profiling, focusing on information meaning and use. The next part of the series discusses system and table-level profiling. In particular, what information is important to collect at the system and table level and how can that information be leveraged by the Enterprise to help assure quality. The third part dives into attribute-level profiling and the fourth discusses attribute patterns and relationships. The final part discusses the benefits and utility of gathering profiled information into a single repository.
Continue with Part 2.
It had to happen eventually: Oracle is acquiring Silver Creek Systems, a leading provider of product data quality solutions.
I first became familiar with Silver Creek through a chance meeting with Martin Boyd, Silver Creek’s VP of Marketing, at the Fall 2007 MDM Summit in New York. We both ran into someone from Weyerhaeuser, and all of us ended up going out to dinner at a great New York steak house.
I stayed in touch with Martin after that, and gradually learned more about Silver Creek’s product data quality solution, DataLens. I’ve said for a long time that data quality plays a critical role in master data management, so as I learned more about product information management (PIM) and product MDM, I naturally wanted to learn more about Silver Creek.
I profiled Silver Creek in April 2009, and my first hunch that they might end up getting acquired by Oracle came with the announcement later that April about the OEM relationship between Oracle and Silver Creek, where Oracle would pre-integrate Silver Creek’s DataLens solution with Oracle’s Product Data Hub.
This blog covered Silver Creek again in October 2009, where Martin Boyd did a great presentation at Oracle OpenWorld, saying that “10% of the total effort will be on the MDM software implementation, 40% on establishing governance and documenting the master data architecture, and 50% on data remediation” (according to AMR Research).
So I’m pleased but not surprised to see the news of Oracle’s acquisition today. For more information, you can read Oracle’s press release here.
As we’re about to enter 2010, it’s a good time to reflect on what happened in 2009 and what it all means.
“It was the best of times; it was the worst of times…” So Dickens begins “A Tale of Two Cities”, but it’s also a good description of the past year.
The first half of the year was one of the most challenging I’ve faced in my twenty-three year career in business and technology. The second half of 2009 was better – not without its speed bumps but every month was a little better than the one before it.
The macro-economic climate has been tumultuous at best. But the second half of the year showed enough improvement that Hub Designs’ revenue for the year was up 33%. Not bad for a two and a half year old company during the worst economic conditions in 80 years …
Marketing and Thought Leadership
We launched a new web site in January, and it’s been well received. Total visits to www.hubdesigns.com were up 14% over 2008.
A little later in the year, we updated the “look and feel” of the Hub Designs Blog, branding it as the “world’s fastest growing blog covering master data management and data governance”. We’ve gotten more than 43,000 hits since we started writing in July 2007, and our readership more than doubled in 2009, to about 27,000 hits per year.
We published six issues of our “Best Practices in Master Data Management” newsletter this year. We publish the newsletter about six times a year to roughly 3,300 subscribers.
I wrote six articles for Information Management magazine, including some popular ones on “Product Information Management Challenges”, how to build a business case for master data management, and how to select the right MDM vendor for your organization. I also wrote for Identity Resolution Daily, on “The Growing Role of Identity Resolution in MDM” and “Matching – MDM’s Secret Sauce”.
With our partner Siperian, we wrote a white paper in August called “When Data Governance Turns Bureaucratic: How Data Governance Police Can Constrain the Value of Your MDM Initiative” that has generated quite a bit of discussion. You can download a copy of it here.
A second white paper, called “Best Practices for Leveraging D&B in Oracle E-Business Suite”, was written in partnership with Dun & Bradstreet. It describes using D&B information to drive better supply chain performance for companies using Oracle E-Business Suite. You can download it here.
I volunteer for the Education Committee of the Oracle Applications Users Group (OAUG). A big part of that effort lies in programming the MDM track for the annual conference. This year, it was in Orlando in May, and I really enjoyed speaking there and seeing people from the Oracle community that I don’t see very often. Here’s a link to my OAUG presentation.
We participated in conference calls with Oracle Development during the year, and ultimately attended the Oracle Fusion “Hands-On Validation & Testing” session for Customer MDM at Oracle headquarters in August. It was a great chance to get some early insights into Oracle’s next major product release and to see the progress Oracle has made in building out its Fusion MDM vision, which is striking in its powerful hub technology and its elegant & productive user interface.
In 2008, we attended the Gartner MDM Summit to decide whether to exhibit there in 2009. We were impressed enough with the conference that we did exhibit in 2009, in October in Los Angeles. We had a positive experience, so we’ll be a Silver level sponsor in April 2010 in Las Vegas. Since we specialize in MDM and data governance, we find the association with Gartner’s MDM event a powerful one.
I didn’t attend Oracle OpenWorld for the past couple of years, but this year I was glad I did. It was like “old home week”, seeing people from Oracle itself and from the broader Oracle community that I’ve met over the past 15 years. David Butler, Senior Director of MDM Marketing at Oracle, posted my presentation on Oracle’s web site, and said “you were our cleanup hitter and you hit a home run with the bases loaded”.
We also did webinars with our partners Siperian and Initiate Systems. The Siperian webinar covered the differences between MDM platforms like Siperian and ERP platforms like SAP from a master data perspective. The Initiate webinar, with Initiate’s CTO Marty Moseley, discussed developing strong MDM business case, deploying core MDM technologies and lessons learned on the “build vs. buy” question.
Social Networking
After experimenting with social networking in 2008, this year we had a coordinated strategy to use the Hub Designs Blog, Facebook, LinkedIn and Twitter to communicate & collaborate with our clients, potential clients, team members, partners, suppliers, etc.
It’s a pretty simple strategy. Short updates (140 characters or less) go out on Twitter, and are re-published on both LinkedIn and Facebook. Longer updates (i.e. blog articles) are published on the Hub Designs Blog. We encourage responses and feedback using @replies on Twitter and comments on LinkedIn and Facebook, as well as longer-form comments on the blog. And we get them – almost every blog article gets at least one comment, sometimes as many as a dozen.
When a new blog article comes out, we notify everyone via a single update on Twitter. What’s amazing is that during 2009, social networking now drives about 15% of the Hub Designs Blog’s total traffic. And one of our clients gave us some good feedback that our social networking activities help her stay current on what we’re up to, and help her feel connected to us as a company.
Another social networking experiment that developed further in 2009 was the MDM Community. We started this using Ning (a “social network in a box”) in November 2008, out of frustration with LinkedIn’s “Group” functionality. It now has more than 210 members, from 23 different countries. It’s still a work in progress, but if you’re interested in master data management or data governance, you should check it out at http://mdmcommunity.ning.com. It’s becoming an international “who’s who” of the MDM world.
Summary of Client Projects
In case you think the Hub Designs team has been doing nothing but marketing, writing white papers and magazine articles, speaking at conferences, and volunteering for user groups, here’s a summary of our 2009 client projects:
- Technology provider for vehicle dealers: integration of Oracle E-Business Suite with D&B data
- Payroll services company: integration of Oracle E-Business Suite with external credit information
- Information services company: technical support for customers using Oracle E-Business Suite
- Legal information services company: readiness assessment and product MDM strategy & design
- Simulation and engineering software company: advisor to data governance board
- Manufacturer of oil and gas equipment: integration of Oracle E-Business Suite R12 with D&B
- Software company: built connector between Oracle AR and D&B’s DNBi risk management solution
- Technology company: customer MDM strategy workshop
Out With The Old, In With The New
This past year has been a lot of fun, but it has also been somewhat exhausting. So I’m looking forward to a bit more deliberate pace in 2010.
We’re very excited about the coming year at Hub Designs. We’ve got some great projects underway and in the pipeline, and we’ll be continuing to grow and expand to meet our clients’ needs and market demands.
In closing, I’d like to say how grateful I am to my family, for their patience with my traveling so much and for their unconditional love.
A client asked me last week about what rate of duplicate data was “normal” in customer master data.
My initial answer was that, among companies that don’t have any formal master data management, data governance or data quality initiatives in place, duplication rates of 10%-30% (or more) are not uncommon.
When I was at D&B, we used to routinely see that level of duplication in client’s customer files.
In a study in the healthcare field, Children’s Medical Center Dallas engaged an outside firm to help clean up their duplicate data:
“Solving both the current and future problems around duplicate records helped Children’s improve the quality of patient care and increase physician acceptance of the new EHR. The duplicate record rate was initially reduced from 22.0% to 0.2% and five years later it remains an exceptionally low 0.14%. The 5 FTEs initially tasked with resolving duplicate records have been reduced to less than 1 FTE.”
“For the Children’s Medical Center, the results were heartening, not only from a care delivery standpoint but also because of the significant cost-savings that can be realized. A study conducted on Children’s data showed that on average, a duplicate medical record costs the organization more than $96.”
So it is possible to get the duplication rate down to really low levels through careful analysis and the application of the right tools, as part of an ongoing data governance program. Even the hospital above (and hospitals are usually not mentioned as practitioners of best practices) was able to maintain a duplication rate of only 0.14% after 5 years.
And there are very real costs to not de-duplicating your customer data. Depending on the functional area (marketing, sales, finance, customer service, etc.) and the business activities you undertake, high levels of duplicate customer data can:
- annoy customers or undermine their confidence in your company,
- increase mailing costs,
- cause hundreds of hours of manual reconciliation of data,
- increase resistance to implementation of new systems,
- result in multiple sales people, sales teams or collectors calling on the same customer,
- etc.
The best studies I’ve seen of the cost of duplicate data have been in the healthcare industry. One study I saw said:
“According to Just Associates, the direct cost of leaving duplicates in an Master Patient Index database is anywhere from $20 per duplicate to several hundred dollars. The lower cost reflects the organization’s labor and supply costs to identify and fix the record while the higher expense reflects the costs of repeated diagnostic tests done on a patient whose previous medical records could not be located.
The American Health Information Management Association (AHIMA) estimates that it costs between $10 and $20 per pair of duplicates to reconcile the records. If the records aren’t reconciled, however, the costs are even higher.”
Here are three more case studies backing up the range I quoted of 10%-30%:
- Once the analysis was complete, Sentara discovered they had a significant duplication rate, over 18%. They had attempted to address the duplication rate in the past through a remediation process, but due to either technology issues or because the cost of merging and cleaning up the duplicates across their many different systems was too high, they had not yet successfully reduced their duplication rate. Source: Initiate Systems success story
- Emerson Process Management faced a tremendous challenge four years ago in getting its CRM data in order: There were potentially 400 different master records for each customer, based on different locations or different functions associated with the client. “You have to begin to think about a customer as an organization you do business with that has a set of addresses tied to it,” says Nancy Rybeck, the data warehouse architect at Emerson who took charge of the cleanup. Working with Group 1, Rybeck analyzed the customer records for similarities and connections using everything from postal standards to D&B data, and managed to eliminate the 75 percent site-duplication rate the company suffered in its data. “That’s going to ripple through everything,” she says. Source: DestinationCRM.com
- Problem: Number of duplicate records: 20.9% of Utah Statewide Immunization Information System records. Impact of Problem: Difficult to find patients in system—key barrier to provider participation, risk of over-immunization—unable to find reliable patient record, cost of unnecessary immunizations, risk of adverse effects on patients. Source: health.utah.gov.
And here’s a good quote from a white paper titled “Data Quality and the Bottom Line” by The Data Warehousing Institute:
“Peter Harvey, CEO of Intellidyn, a marketing analytics firm, says that when his firm audits recently ‘cleaned’ customer files from clients, it finds that 5 percent of the file contains duplicate records. The duplication rate for untouched customer files can be 20 percent or more.”
Every organization will need its own metrics, but left unchecked, the duplication problem is a hidden cost that drags at your company, slowing down your processes and making your analyses less reliable.
If your sales analysis reports can’t be sure that there’s one and only one record for each of your largest customers, then the sales figures for those customers are probably not right. So the entire report becomes suspect at that point.
I’d like to end with a great quote on data quality by Ken Orr from the Cutter Consortium in “The Good, The Bad, and The Data Quality”:
“Ultimately, poor data quality is like dirt on the windshield. You may be able to drive for a long time with slowly degrading vision, but at some point, you either have to stop and clear the windshield or risk everything.”
Please let us know what you think by commenting here. We’re interested in hearing your thoughts on data quality and the issue of customer data duplication.
I’ve been getting a number of “Happy Thanksgiving” e-mails today, and they’re very nice. But they’ve prompted me to think about all of the things I have that I need to grateful for, as this year starts to winds to a close.
First, my family. It’s hard having an entrepreneur as a husband and father. I have spent more time this year in hotel rooms and clients’ offices than at home, and for that I’m sorry. Hopefully the new year will see me spending more time in the Boston area than on the road, and being more present in all of your lives. And I’m hoping the sacrifices we’ve all made in building this business will continue to pay dividends in 2010 and beyond.
Second, our clients. They’ve been great this year. As usual, I won’t name names here. But you know who you are, and you know how grateful we are that you are working with us. We try to work hard for you and to always make your projects a success, but we recognize that you’re right there with us, working hard and investing yourselves in our success.
Third, our people and team members. It is a privilege to work with you. Every day, I learn something from you. Some of the best times in my professional career have been this year (some of the hardest too!). But I always learned something, and I thank you for taking the time to teach me. It is always interesting.
Lastly, to everyone who reads our web site, this blog, our newsletter, our magazine articles, who caught one of our speaking engagements this year, or who joined the MDM Community – thank you! The extended community that Hub Designs is part of is very special to me. People that I run into at conferences, or that send me e-mails offering to connect me with people they know, or that reach out to me through LinkedIn, or that read my postings on Twitter, are all very important to me.
A few years ago, before I really knew the value of social networking, I didn’t understand it and thought it was a little frivolous. Now, I understand its power – to connect us to one another, to effect change, to weave people and companies together in a new way, to make the 21st century more intimate, to allow me to sit in my office and say thank you to thousands of people at once, without blasting a newsletter into people’s inboxes, without sending out a mailing, without placing an ad, without doing any of the traditional things companies would have done at one time to get their message across (and are still doing).
Today, to me, it’s all about authenticity, and helping people, and being in the right place at the right time. And for that, I’m grateful.
The Hub Designs Blog welcomes a guest post by Rob DuMoulin, an information architect with more than 26 years of IT experience, specializing in master data management, database administration and design, and business intelligence.
Most business intelligence architects are well versed in the value of the time dimension.
With query performance and the need to support complex analyses being the two most important considerations in BI, a flattened set of time dimensions provides a multitude of options to represent and standardize time with limited overhead.
It’s easy to see the value of having a flexible, consistent, and integrated representation of time when thinking of business activities. Aspects such as when a transaction or activity occurs in relationship to other transactions, activities, or even pre-defined thresholds form the basis of Business Process Management activities. And accounting departments group transactions into time periods every financial reporting period.
So, how valuable can this same time dimensions be to a Master Data Management solution? If you are well versed in MDM at this point, you’re probably saying “What you’ve talked about so far is useful for relating transactions but it doesn’t tie back to mastering business objects like customers, products, or locations”.
But remember that mastering those objects does require standardization during information acquisition and publishing and that the various inputs and outputs to an MDM system are often diverse. Also, don’t underestimate the value of mastering “Time Tables” themselves as a component in your MDM universe.
First, let’s define just what we mean by a set of time tables before we apply them to MDM. A typical implementation would have two distinct groups of tables to represent time: day, and time-of-day. At the lowest level of the day group is a day-level table with every imaginable way the business can identify a day, such as: by its day of year, week, month, quarter, advertising week (for retail), same day last year (in some special context), or special tags like holiday, weekend, season, positional sunrise/sunset times, or even astrological sign and full moon cycles. And that just covers the calendar view of the business. There is an equally important and extensive set of calendar hierarchies and attributes associated with the business fiscal reporting needs. Add to that every way you want to represent attributes like day of the week or month of the year (number, 3-letter abbreviation, full name) and ending up with over 100 attributes in the day-level table is not uncommon.
Related to the day-level table are hierarchy tables at levels such as: month, quarter, year (and their fiscal counterparts). Each of the hierarchy tables contains all the attributes that define that level and higher levels. For example, the calendar month table would contain attributes defining month of year, month of quarter, and month overall, in addition to quarter and year and all the ways to call the month. Primary keys for the higher level hierarchy tables, like month, would have child entries in the lower level tables, like day, for every entry that rolls into the higher level.
The same holds true for time of day, with hierarchies like hour, minute of hour, shift, peak time, off-peak time, and others.
Because all the higher-level attributes are repeated in the lower-levels, there is typically not a compelling need to join the two tables. The relationships are there for flexibility. Having the various hierarchy tables as stand-alone entities allows you to attach them to business tables at all of the levels you collect or report time values. These tables and hierarchy relationships allow you to easily merge data of different time grains.
The best thing about time is that time is constant. There are always sixty seconds to the minute, sixty minutes to the hour, twenty-four hours to the day (excluding Daylight Savings Time adjustments), seven days to the week, the number of days to the month is fixed, the number of days in a year is predictable. Except for adjustments to fiscal calendars and special events, most of the information related to time hierarchies is static.
BI uses these techniques to conform information allowing it to readily apply to many views of the business… which sounds a lot like the same business issues we try to solve when integrating data within an MDM solution.
Introducing a robust set of Master Time dimensions into an MDM architecture opens up flexibility in how you consolidate information and also how you can apply it to many business purposes. It’s a natural expansion of MDM to include a master version of the corporate calendar (particularly the fiscal calendar) using a common set of time-related identifiers complete with any time references relevant to business operations.
Please let us know what you think of mastering the Time dimension or other types of corporate reference data in the MDM hub by leaving a comment here.
“All NDAs are lifted” were the magic words uttered by Steve Miranda from Oracle at the Fusion Inner Circle Event at Oracle OpenWorld on October 15th.
Just to make sure, I asked Steve explicitly during the Q&A section of the program if it was okay under the non-disclosure agreement we had all signed to write about Fusion on my blog, and he said “Yes.”
Hub Designs was invited back in February to help Oracle’s Fusion MDM team with some design review, validation, and testing activities. In return for our assistance, we’ve gotten to see Fusion MDM inside and out, and we can proudly say that we are one of the very few trusted partners who helped Oracle to design and develop the application.
We participated in a lot of conference calls with Haidong Song, Oracle’s Product Strategy Director for Customer MDM, and other members of his team. And we attended a week-long “hands-on validation” event at Oracle headquarters in August, looking specifically at the customer data management aspects of the Fusion MDM hub.
My first impressions of Fusion MDM during that hands-on session were very favorable. I remember thinking to myself, “Oracle could almost start selling this into the MDM hub market right now!”
Of course, Fusion isn’t scheduled to ship until sometime in 2010, and there’s still plenty of work to be done between now and then. But the core functionality needed for master data management was there, and the Oracle Fusion MDM team had a room full of customers and partners banging on it for a week without any significant crashes or issues.
There was plenty to like in Fusion that didn’t relate specifically to master data management – the new and improved user interface, the embedded analytics, the modern, standards-based architecture, the usability research that Oracle has done, the improved business processes, the built-in collaboration capabilities …
But the fundamentals of MDM were strong as well. Haidong and his team demonstrated how to import and consolidate customer data from outside sources, and we did our first hands-on lab session bringing in a small customer data load from a desktop file, such as a list of trade show leads.
We also tested a larger volume of customer data being brought into Fusion MDM through the Bulk Import process.
We did another exercise simulating how a typical customer data steward would identify potential duplicate customers, and then resolve those duplicates by merging the duplicate parties.
We also got a good look at the Informatica components that Oracle is bundling into Fusion on an OEM basis: the former Identity Systems matching engine and the former Address Doctor address cleansing tool. Previous Oracle MDM products like Customer Data Hub have had loose integration with Trillium and Firstlogic for address cleansing, but it’s refreshing to see Oracle investing in deep integration with industry leading solutions.
I think there are going to be a lot of Oracle customers who will move to Fusion MDM as the first wave of their overall migration to Fusion, who will see Fusion MDM as a good way to get some early experience with the Fusion applications family, before committing their mission critical Enterprise Resource Planning (ERP) applications to the Fusion platform.
And in 2010 and beyond, I think will be a lot of potential customers who evaluate Fusion MDM positively on its own merits against competitive MDM hubs. Oracle brings a robust data model, open architecture, and a next-generation approach to master data management, with state-of-the-art matching, data quality, middleware, and business process management.
Please let me know by commenting here what your thoughts and expectations are for Oracle’s Fusion MDM hub.
I had a great time at the Oracle OpenWorld conference this year.
Oracle did a great job organizing the MDM track. There were a lot of great presentations, and a good balance of speakers between Oracle people, outside consultants and experts, and end users with success stories to share.
David Butler, Senior Director of MDM Marketing at Oracle, was kind enough to convert my presentation titled “Best Practices in Master Data Management and Data Governance” to PDF format and to post it on the Oracle.com MDM web page.
You can find it in the ‘Partners’ portlet on the right hand side of the page, or just click here.
I had a laptop “near death experience” over the past few days. It actually started on Saturday (which was Halloween). So I guess that makes this a “Halloween Hard Drive Horror Show”.
First, my Sony Vaio, which I’ve had for two years, got a little wobbly. Windows Vista wanted to run the dreaded CHKDSK utility. Things went down hill from there very quickly.
Monday night, I went back to my hotel room after working at my client’s offices all day, and the laptop refused to boot up at all. I gave it my best “I am not a techie” try, and realized this was not something I was going to be able to resolve on my own. No problem, I thought. I bought this laptop at Best Buy and was smart enough (I thought) to purchase a three-year extended warranty at the time (for an additional $600).
So yesterday morning, I showed up when the local Best Buy opened their doors, with my service plan number in hand. After a brief wait, I spoke with a member of the Geek Squad. He regretted to inform me that neither hard drive failure or reinstalling Windows Vista were covered by my extended warranty. But they were kind enough to let me borrow their Yellow Pages.
I got really lucky finding D&D Computers in Huber Heights, Ohio.
Brian Dean, the Chief Tech, told me to come right over. I got there a little after 11:00 am, and was there until just after 4:00 pm. Brian took extremely good care of me and my laptop. At my request, he replaced my failing 150 GB hard drive with a brand new 500 GB drive, bumped my RAM up from 2 GB to 4 GB, and installed Windows 7 on the new drive.
I had to reinstall all of my applications, which took a few hours last night. But to be back up and running in less than 24 hours, and to have gotten a major laptop upgrade out of all this, was a great outcome. I even got my old hard drive installed in a little enclosure so I could hook it up to my laptop using a USB cable, to access all of my data.
The total cost was $885 ($321 at Staples for the full version of Windows 7 Professional, $500 at D&D Computers for the new hard drive, new RAM and their labor, and $64 at Best Buy for the USB drive enclosure).
The moral of the story: read the fine print of your extended warranty, let your fingers do the walking and make sure you’re current on backing up your hard drive!
At the Gartner MDM Summit conference three weeks ago in Los Angeles, I sat down with Anurag Wadehra and Ravi Shankar from Siperian. I usually go to Siperian’s user conference, which was held last week in Princeton, NJ. I couldn’t make it this time but had a great time at their Spring 2009 event.
So instead, I thought I’d do a blog article on Siperian’s momentum in the last year or so, based on the briefing that Anurag and Ravi were kind enough to give me in Los Angeles.
Siperian’s ambition is to be a leader in multi-domain master data management and since their product is not tied to a specific data model, that’s a realistic goal. Many of their customers find the business problem they’re initially trying to solve does in fact involve multiple domains (or areas) of master data.
Siperian’s most recent fiscal year ended May 31st, and they wrapped up the new year’s first quarter on August 31st. Impressively, their license sales more than doubled over the last 4 quarters, and overall revenue almost doubled.
The reduction in dependence on services revenue and the corresponding increase in license revenue, indicates a positive trend that Siperian continues to shift its implementations to its alliance partners.
One of the reasons Siperian wanted to sit down with myself and others in the MDM space was to dispel some rumors that have been floating around about the company. The economic downturn that began in the fall of 2008 has been widely felt, to be sure, and Siperian had significant exposure at that time to the financial services industry, which was one of the hardest hit industry sectors.
But Siperian has done a good job diversifying its customer base into other verticals, more than a dozen total to date, and is continuing to close deals with new customers, extending its footprint at existing customers, and building significant relationships with global systems integrators.
With customers like Johnson & Johnson, Merrill Lynch, and Cephalon speaking on behalf of Siperian at events like the Gartner MDM Summit and Siperian’s own user conference, there definitely seems to be a pattern emerging of large organizations with challenging MDM requirements turning to Siperian.
Another trend worth mentioning is that a large portion of Siperian’s revenue is repeat business – customers who have done a successful project with the company and are expanding their MDM footprint into another domain, geography, etc. This speaks volumes about the success of Siperian customers’ current implementations.
Siperian’s “Business Data Director” (BDD) product, launched at the spring user conference, has already signed up more than a dozen customers, with 2-3 already “live” and more going live in the next few months. I was there for the launch of BDD and remain impressed with it.
To a large degree, Siperian’s strategy of scaling through alliances is paying off. Ninety percent of its revenue in the last 4 quarters was partner influenced, with its top four partners accounting for 60% of that business.
I’ve followed the company closely for the past couple of years, and I think their company strategy and product roadmap is solid. Siperian helps keep the “Big Three” of MDM (Oracle, IBM and SAP) on their toes, and has generated a lot of innovation in this space.
I’m sorry to have missed their user conference last week, and I continue to expect great things from Siperian. Please share your thoughts on the company and their products here using the Comments feature.
