An MDM taxonomy is simply a structured hierarchy applied to the topic of the MDM project (for example: products, people, or customers) that defines that topic’s attribution.  At each level, this hierarchy enforces the inheritance of characteristics to all of its children and their children. For example, the taxonomy of biology that has remained in my memory since 6th grade contains the levels Kingdom, Phylum, Class, Order, Family, Genus, and Species. Any animal or plant can belong to only one member of the lowest level, species, and each level of the taxonomy defines the inherited characteristics of its children. The same concept is core to an MDM design and each widget in an MDM topic can only reside in only one of the lowest taxonomy levels.

The number of levels in the MDM taxonomy varies based on the business need, topic, and a count of the widgets in the topic. There are standards available to guide you in level counts and names if you want to follow them, but the assignment of attributes, definitions, and placement of your widgets in the structure is business-specific.  Plan for a significant investment  of effort to get the taxonomy and item assignments correct.  This effort should result in the business agreeing on a taxonomy containing the fewest levels necessary to accurately represent the MDM topic’s widgets, along with a few other guidelines.

The topic of an MDM project may have many business purposes and be categorized by business users in a variety of different ways. This is expected and encouraged. We are not trying to restrict how the business analyzes the topic’s widgets.  The taxonomy we are concerned with is a single hierarchy defining widgets through attribution characteristics as described in the prior biology example. We do this to create a single unambiguous definition that can be applied to every existing and new widget so that each widget falls under one and only one of the taxonomy’s lowest levels. The business must validate the one widget per lowest taxonomy level rule, what attributes are common to each level, and that the attributes of any level apply to all levels below it. The taxonomy not only results in a standardized method of defining widgets, but also allows for automatic inheritance of widget properties during definition which reduces the workload and chances of errors during the widget information entry.

Expect to encounter puzzled looks when introducing the concept of attribution-driven taxonomy. Business subject matter experts do not think of their widgets in those terms.  Instead, they will be thinking in terms of how the business reports on the widgets.  The distinction is clear only when you remember the purpose the taxonomy serves. Conducting workshops with business users across the board promotes the required consensus. After a few episodes of realigning discussions from a reporting mindset into an attribution mindset, the users will start to change their thinking and the results will be a valid taxonomy that the MDM initiative can grow on.  Without this foundation, your success will be limited.

MDM Application Sales Rep, Western Territory — located in Southern California (LA) or Denver

Primary job duty is to sell business applications software/solutions and related services to prospective and existing customers. Manage sales through forecasting, account resource allocation, account strategy, and planning. Develop solution proposals encompassing all aspects of the application. Participate in the development, presentation and sales of a value proposition. Negotiate pricing and contractual agreement to close the sale. Identify and develop strategic alignment with key third party influencers.

Acknowledged authority within the Corporation. Provides leadership and expertise in the development of new products/services/processes, frequently operating at the leading edge of technology. 12 years applicable experience including 9 years of sales experience. Successful sales track record.

Ability to penetrate accounts, meet with stakeholders within accounts. Interaction with C level players. Team player with strong interpersonal /communication skills. Excellent communication/negotiating/closing skills with prospects/customers. Travel may be needed. Bachelor degree or equivalent.

If you’re interested in this position, please reply to info@hubdesigns.com and we’ll pass your response on to the hiring manager.

What does this mean for the average Fortune 1000 company buying MDM technology? Not as much as you might think.

On the mega-vendor side, they’ve still got Oracle, IBM and SAP to choose from.  IBM, obviously, now has three MDM platforms to offer (InfoSphere MDM Server, InfoSphere MDM Server for PIM, and Initiate Systems) where they used to have two. But Oracle has three as well, and will soon have four: Customer Data Hub and Universal Customer Master for customer MDM, PIM Data Hub for product MDM, and Fusion MDM Hub, Release 1 of which is supposed to ship later in 2010.  And SAP continues to forge ahead with improved versions of their NetWeaver MDM product. So the recent consolidation doesn’t seem to have affected the mega-vendors that much – “the big get bigger”, you might say.

Outside of the “Big Three”, I continue to think Siperian being acquired by Informatica is a good thing, for Siperian’s customers, for the product roadmap, and for the market as a whole. Informatica brings a lot of expertise in integration and data quality to the table, and its Identity Systems matching engine and Address Doctor data cleansing tools are very good at what they do. It will be interesting to see how Informatica integrates Siperian as a company and as a product into itself, but I have a lot of confidence that they’ll do it well.

All this does pose an interesting issue for Oracle, however. Oracle made a big commitment to Informatica in its Fusion MDM Hub by including Informatica components for matching and cleansing on an OEM basis. But by buying Siperian, Informatica has declared itself a direct competitor in the MDM market. So there’s a lot of speculation as to what Oracle will do about this. In the short term, it may be too late to pull the Informatica components out of Fusion MDM Release 1.0, but longer term, I wouldn’t be surprised to see the Informatica components either replaced or deemphasized, perhaps with an open architecture approach allowing other third party identity resolution / matching and address cleansing products to be plugged in, in place of Informatica’s. Although there’s also been a lot of speculation about Oracle buying Informatica.

D&B/Purisma remains an interesting player. Disclosure: prior to starting Hub Designs, I worked for D&B. I saw D&B’s launch of a hosted version of Purisma last fall and was impressed by it. For a lot of situations, Purisma’s product can be a good solution. So even though I wouldn’t call Purisma a full-fledged master data management solution, it’s worth keeping an eye on because it does a great job of integrating internal customer data with D&B’s external reference data. And having it available on a hosted basis can be very helpful.

So the bottom line is, where there used to be six players, now there are five.  Of course, the MDM market being as hot as it is, everyone and their brother claims to be an MDM solution, but these are the five products that I pay the most attention to, and that we see the most in the marketplace. What do you think?  Please let us know by commenting here.

DISCLAIMER: While the facts that I’ve included here are true, I’m speculating on the reasons why they’re taking place. I have no affiliation with any company mentioned here, nor should my opinions be construed as knowledge of their actions.

If you, like me, have followed MDM for the past year or two, you knew that what has been happening recently was going to happen sooner or later. Whether it was due to choice or necessity, MDM has been in the IT press a lot lately. Oracle acquired Silver Creek to enrich its product information management offering. Talend has announced and started to promote its open source MDM application. Data integration provider Informatica acquired Siperian recently in order to enter the MDM space and IBM recently acquired Initiate Systems as well.

Each of these events leads to one key question – how will this impact MDM in the short term and in the future? Given my understanding of the space, I think three scenarios are likely:

Scenario 1

It is hard to ignore the movements that IBM and Oracle have been making in the past year or so. In a market economy, the goal is to have as much market share as possible. In order to do this, you either build new products or acquire existing companies that have the technologies that you want.

While each company has done a combination of both building and buying solutions, their strategic plans are hardly a secret. IBM has proposed a vision of an end-to-end data management platform, which includes their MDM offering as well as reporting tools like Cognos and analytics/statistics from SPSS. Now that IBM has acquired Initiate Systems to complement their MDM stack, the question is why. Do they want to be known as a serious player in the health care industry? There could be other reasons too – they may consider MDM just a small piece of their data management toolkit and this could solidify that position, moving MDM from one of the hottest ‘technologies’ out there to just a “means to an end” to increase market share for their software business unit. Regardless of the reason, it means one less major MDM player in the market.

Then we have Oracle. For as long as I can remember, Oracle has been promoting its Fusion strategy. For those of you who are not familiar with it, Fusion is Oracle’s attempt to provide one code base that would pull together the applications it has built and purchased. This momentous undertaking was finally demonstrated at last year’s Oracle Open World (while Oracle continued to acquire other companies such as Silver Creek Systems).

However, like IBM, one can speculate on where MDM fits in this Fusion strategy. Oracle has always promoted its database first and sold its applications second. Even with the numerous special purpose hubs they’ve been developing lately, could this finally be the technology that enables Oracle to transcend from being a database vendor to a true platform player. Only time will tell with this one.

Scenario 2

There’s always the possibility that MDM has been considered the “secret sauce” – the so-called missing link – that rounds out the product lines for data integration/migration vendors.

Talend’s acquisition of French software company Amalto provided them a way to enter the MDM space. The open source vendor has been a darling of the analysts for a number of years and even won an award by Gartner, one of the first (if not the first) they offered such a company. However, since I don’t have contacts within Talend, it’s not clear what their next step will be, since they seem to be focusing their energies mostly in MDM after hiring two people to drive that effort within the past 6 months or so.

As the de facto leader in data integration, Informatica needed to extend its reach beyond that space. If you look at their job listings, they are looking for someone to market their CEP (Complex Event Processing) efforts. Relatively recently, they were looking to hire someone who had experience with ERP or MDM, but it is unclear which path they decided to take with that. Regardless, there were always looming rumors of them wanting to add MDM to their portfolio with the press suggesting that they would acquire Initiate Systems. However, instead of buying them, they purchased Siperian – a company half its size in terms of customer base and revenue.

In either of these cases, MDM may not be their flagship product, but at least it shows that it is a viable technology and shows that it is something that won’t be going away any time soon.

Scenario 3

People like me who have been in the data management space are always interested in improving something. I believe in the statement, “even if something isn’t broken, there’s always a reason to make it better.” This was clear when Customer Data Integration (CDI) first came about and many companies hopped on that bandwagon, knowing that they wanted a way to track their customers more efficiently.

At the same time, other companies explored Product Information Management (PIM), a way to have a single source of product information which was sourced from PLM, inventory and supply chain systems. Following that was the concept of MDM – a beautiful vision – having a single source of truth that can be used by an entire company.

Now we have a new concept that has been promoted – Multi-domain MDM. Siperian and other companies have began to promote this to show the world that they are truly the most advanced players out there. While this has been going on, there have been rumblings about Enterprise Information Management (EIM). What I’m still not clear on is – what’s the difference between multi-domain MDM and EIM? Are they the same? If not, what are the differences between the two concepts?

In any case, there’s a lot to think about. I don’t know where you stand, but one thing is certain – MDM is not going away, at least for the foreseeable future.

Part 5: The Profiling Payoff

This is the final part of a five-part series, describing how data profiling benefits both IT projects and business operations.  In Part One, we discussed profiling perspectives.  In Parts Two, Three and Four, we introduced the value of system, entity, and attribute-level metrics.  This part discusses the archival and beneficial uses of profile results.

If you have defined your corporate data profiling strategy similar to the methods discussed in the preceding parts of this series, you’ll have amassed a robust collection of metadata spanning relevant systems across your business.  Although systems may be of different types and locations, the structured approach and common metrics you collected create a centralized repository of information that can be examined holistically. Ideally, this information will exist in an open-source database repository with reports made available across the enterprise. System and Entity information help planners and developers organize information strategies. Attribute-level domains, constraints, and business rules help data architects understand existing systems. Relationships and value patterns are readily available to support validation of information-related hypotheses as needed.

If you plan to design your own repository, consider adding timestamps and indicators to help you manage and present the information.  To keep your repository relevant to business needs, design collection rules to be configurable. This allows you to easily ignore superfluous information or enable tests only at certain critical times. Allow initial system profiling efforts to gather a large set of metrics and store them as your baseline.  As you learn about the information, you will see which tests or which data objects add no value.  Us geeky DBA-types who understand system-level catalogs have our own scripts to do much of what was described inParts Two,Three and Four. Those less-inclined may prefer to use a third-party tool for profiling. Either way works as long as the business needs are satisfied and the entire enterprise standardizes on one approach (and thus one integrated repository).

You will find that collecting and maintaining this level of detail has a definite cost.  Even if the collection is automated, interrogations of large data sets places an overhead on production systems that may not be practical. Record and monitor profile execution metrics to identify bottlenecks or tuning opportunities. Realize that the extent of data profiling is contingent on the project phase, specific data elements, and most of all, business value. Review profiling goals on a regular basis and eliminate unnecessary and redundant checks.

How much profile history to maintain is another consideration.  Even though disk is “relatively” cheap, maintaining all historical entries in a live repository may not be necessary. Consider business needs and value for historical profile information. Even consider archiving at a summarized (or less frequent) level and keep only a limited time window of statistics online.

This discussion on data profiling was intended to broaden perceptions of what it means to a business and the value it can bring if done in a sustainable way. The blog format is not conducive to in-depth discussions, but hopefully the topics covered here spur some thoughts into how you can add value to your business by implementing some of these concepts.  Use your imagination, but remember that no matter how cool it might be to collect and store some profile output, if it does not add business value to somebody, it might not be worth the overhead to continue recording it.

Go back to Part 4.

This was widely rumored last week, but the announcement of Informatica’s acquisition of Siperian took my mind off this temporarily.

On the face of it, it makes all the sense in the world. IBM knows a good product when it sees it, and Initiate has been doing well in the MDM world lately, particularly in the healthcare vertical, where it grew up, and in the public sector vertical. IBM’s press release explicitly mentions Initiate as a leader in “data integrity software for information sharing” among healthcare and government organizations. I thought it was interesting that the IBM release didn’t mention the terms “master data management” or “MDM” even once.

I was a little surprised that IBM’s release made no mention of the financial terms, since IBM is a public company, but I’m sure it will only be a matter of time before those details become available to those who know where to look or whom to ask.

It wasn’t a surprise to see the IBM release mention the stimulus funding being invested around the globe. When I first saw the rumors last week, I immediately thought – IBM is buying Initiate to be better prepared for the various e-Healthcare initiatives that are coming down the pike.

Where things may get a bit tricky is explaining the multiple MDM platforms from IBM to potential customers, and managing several different development roadmaps and product portfolios. There’s the IBM InfoSphere MDM Server (the former DWL product) and there’s also IBM InfoSphere MDM Server for Product Information Management (the former Trigo product). And now there’s the Initiate product too.

While the acquisition does make sense, there is an “embarrassment of riches” factor. IBM will, of course, develop a sales playbook explaining what situations at what type of customer are a good fit for each product.

I think the lingering feeling I have with Initiate Systems is that it may be headed for a “golden ghetto” at IBM – never to reach its full potential as a solution across many different industries, and eventually to handle many different domains of master data. IBM may (and rightly so, in its mind) pigeonhole it into the healthcare and government verticals.

But Initiate’s had some good success outside those two industries. In the Financial Services vertical, they’ve got customers like Capital One Financial, Countrywide Financial (now Bank of America), eSure Insurance, and Wells Fargo. In the Hospitality industry, they’ve got Choice Hotels. In manufacturing, they’ve got Mitsubishi Motors Australia. In the Logistics vertical, they’ve got Federal Express. In the retail sector, Barnes & Noble, CVS, Longs Drug Stores and SuperValu are all customers. And in the high tech space, they’ve got Dell, Ingenix, Intuit, LocatePLUS, Microsoft and National Instruments.

Unfortunately, they didn’t achieve enough critical mass in any of these other verticals to compete with the strong momentum they’d developed in healthcare and government.

As I said last week, these are interesting times in the MDM world. The recent M&A activity, the healthy demand from large and medium sized corporations, the large number of consultants from other areas claiming to now have experience in MDM – these are all signals to me of a large and fast-growing market. So the New Year, for those of us in the MDM space, is off to a good start.

The two firms were already working together closely, having an alliance and OEM relationship through Informatica’s acquisitions in 2008 of Identity Systems (for entity resolution and matching) and in 2009 of Address Doctor (for customer address cleansing).

This will strengthen the Siperian product further by bringing Informatica’s technology even more tightly into the Siperian MDM Hub.

At the same time, it eliminates the “company viability” question mark that sometimes gets raised in large IT shops’ minds when evaluating enterprise software vendors. When a Fortune 500 company is evaluating a smaller company, they sometimes wonder how long a company like Siperian can last against companies like IBM, Oracle and SAP. I’ve never been a big fan of that argument, since some of the best software gets created at small and medium-sized companies, but there’s no doubt it’s a obstacle to be overcome with the larger enterprises. Now, it shouldn’t be an issue.

As a Siperian partner, Hub Designs is excited about this acquisition. Based on the information we’ve got at this point, it seems like a good thing for Siperian’s customers, products, shareholders, partners and people. In today’s economic climate, dreams of a big IPO (for any venture-backed technology company) are unlikely, so an acquisition by a well-run larger company is a good outcome.

I know a lot of the people at Siperian personally, and have worked closely with them over the last few years. I hope the people at Informatica realize what a strong team they are getting in this acquisition, and do everything they can to hang onto them all.

I do suggest they stop using the term “MDM Infrastructure” though (which appeared 5 times in Informatica’s press release announcing the acquisition). First, it’s not accurate – MDM projects typically need to be drive by the business to be successful, so they can’t and shouldn’t be thought of as “IT Infrastructure” projects. Secondly, from a marketing perspective, “infrastructure” is about as exciting as mud – it’s hard to get senior management excited about spending money on something with the word “infrastructure” in the name.

As for the acquisition’s impact on the rest of the MDM market, it’s still growing pretty quickly, but the number of players is shrinking. So I think we’ll see it become even more competitive, and with Informatica now becoming a strong player in the MDM hub market, that’s got to cool its relationship with Oracle, who selected Informatica as an OEM component of its Oracle Fusion MDM hub.

IBM is rumored to be acquiring Initiate Systems, which is an interesting play in its own right, especially given the expected growth in spending in the e-healthcare space over the next few years.

And SAP continues to improve its products slowly but steadily, while D&B/Purisma is doing some interesting things with web services access to the D&B central database of information on businesses.

As for the remaining independent MDM vendors, like Orchestra Networks and Kalido, or Product Information Management (PIM) solutions like Stibo and Riversand, they should see this as further validation of the strength of the MDM market. Kalido feels that it’s the only independent MDM provider who can manage every master data domain. That may be true.  I plan on learning more about Kalido over the next few months.

So like the Chinese curse, “may you live in interesting times”, the beginning of 2010 promises to be interesting for all of us in the MDM business!

If you’d like to continue the discussion on the “Impact of Informatica’s Acquisition of Siperian”, click http://ning.it/aJ1Xj5.

Part 4: Profiling Relationships and Patterns

This is part four of a five-part series describing how data profiling assists in all aspects of system development, from design through deployment.

Part One introduced different perspectives on data profiling. Part Two identified valuable system and entity metrics to track. Part Three discussed attributes. In this segment, we dive deeper into attribute relationships and pattern recognition. Also, we expand on primary key identification discussion and discuss hidden relationships.

Pattern grouping provides a mask of distinct format patterns within an attribute data set and a count of the number of occurrences. Patterns give insight into the type of values found in an attribute. For example, a numeric pattern analysis may show values such as 999.99999, 99, or -.9999.

Observing distinct patterns gives insight into the maximum digits and precision, and also domains such as integer or real. Pattern of a database date or date-time type provides unremarkably similar patterns for all dates. Because the database management system typically enforces the domain, date analysis provides no value and can be ignored. If dates are stored in character format, however, patterns quickly show variations in date formatting. Character patterns only have significance to a limited number of positions. It makes no sense to pattern a description field of 200 or 2000 characters. Smaller code attributes of less than 10 characters though do provide value. Ignore pattern profiling for character strings over 20 characters at first, then refine to shorter character strings if the results do not add value.

In pure database theory, referential integrity (RI) is your friend. In practice, designers and software vendors often forgo RI to improve system performance on data inserts. These designers place the data quality burden on the application and do not endorse external data manipulation outside the application interfaces. In the real world, though, data corruption occurs and without RI or routine data quality checks, corruptions may not be found for a long time or not at all. Personally, I have identified over $50,000 of recent orphaned sales in a retail client resulting from deliberately disabled RI. These unreported sales were not added to the ledger and were allowed to occur for performance reasons until I found them through simple profiling. Enforcement of RI is a topic for another discussion but is mentioned here because it does identify a valid reason for data profiling.

In even presumably good relational designs, some parent-child relationships are not enforced for different reasons. First, interrogate the RI listed in the system catalogs to identify all enforced relationships. Reverse-engineering a system with a good modeling tool is probably the best way to do this. A harder and more valuable analysis is to identify unenforced relationships and determining the probability of the relationship if not all values are an exact match. Do this by counting all the candidate child attribute values that exist within a known parent attribute table. If all match and there are a non-trivial number of matches, there is a good probability of a non-identified relationship. A small number of mismatches could identify data quality issues.

In Part 5, we tie all the techniques discussed in the first four parts together to show the value of a repeatable data profiling process.

Continue with Part 5 or go back to Part 3.

Part 3: Attribute-Level Analyses

This is part three of a five-part series on data profiling.

In Part One, we took a light-hearted view of where profiling benefits an organization and in Part Two, we discussed the fundamentals of a profiling strategy.  The remaining three parts discuss attributes, relationships, patterns, and how to use the combined data profiling information you collect.  In this section, we introduce attributes, the lowest-level components of a profiling effort.

An attribute is simply a individual data element.  Alone, an attribute has no context.  Given the simple descriptor of “Cost” for an attribute tells us very little about the attribute’s true purpose and immediately drives a need for additional information, such as units (hours, Dollars, Euros…), type (weighted, unit, gross…), and use (invoice, sum, average…).  Attributes therefore must be analyzed within the context of their business purpose to have meaning.

Some characteristics require business knowledge to define and others can be determined through interrogation of existing values and underlying rules of the storage medium. It takes both analyses to get a complete picture of information within a system. While assembling this puzzle, though, keep in mind that until you validate the enforcement of business rules, only assumptions can result from physical profiling or business context.

Analyses of values, domains, and constraints allows insight into use (or abuse) of an attribute. The larger the sample size, the better confidence you gain in the results. Without explicit proof of business rule enforcement, though, you must assume that just because a value does not presently exist does not mean it cannot exist. Business rules are defined by business experts and enforced through database constraints, data type/precision, and application code. Knowing the methods of enforcement allow you to narrow a domain but not totally understand it. Profiling of actual values provides additional refinement in terms of percentage of NULL values, percentage of distinct values, minimum, maximum, and average values, top x and bottom x recurring values along with their counts, and minimum, maximum, and average data lengths.

Some attributes within a data set serve valuable purposes that are important to identify. Attributes that individually or in conjunction with others define uniqueness of the data set also may support relationships between entities.  Uniqueness can be further classified as being either members of a system-enforced primary key or of a business key (outside of the defined primary key).  System-enforced primary keys are relatively easy to define within a database system through interrogation of the system catalog.  Business keys that exist in tables in addition to a primary key may be more difficult to identify, especially if more than one attribute is needed to define uniqueness.

Attribute-level information of interest includes: data type (size and precision), the number and percent of NULL values, column descriptions, number and percent of distinct values, and the minimum-maximum-average values and lengths.  Uses of the system catalog provides some of this information, but others must be collected through sampling the data.

Other types of attributes that may help in identifying relevancy are those that provide system-level auditing or change control. Knowing which attributes fill these roles may either allow you to (a) ignore them for profiling purposes or (b) use them to help explain versions or data anomalies.

Part 4 expands on attribute profiling with the introduction of relationships and patterns.

Continue with Part 4 or go back to Part 2.

Part 1: The Psychology of Data Profiling

Swiss psychologist Carl Gustav Jung founded the Analytical School of Psychology. His word association theories form the basis of the Myers-Briggs Type Indicator Assessment test to identify career aptitude in today’s high school students. Dr. Jung’s approach assigned personality profiles based on how an individual’s thoughts associated to various phrases. By analyzing responses, he could understand how an individual viewed the world around them and perceived themselves. Typically, subjects are asked to speak the first thought entering their minds after hearing a trigger phrase. For the following example, remember, there are no wrong answers. If I say the words “Data Profiling”, what is the first thing you think of?

If you thought of food, cats, country music, CSI NY, or residential plumbing, you are either not in IT or are an IT Manager.

If your first thought was “Quality Assurance”, you align yourself with data quality professionals having anti-social thoughts of failing test cases and sadistically reporting lazy developers for buggy code. You gleefully scour test cases looking for any evidence of truncation, missing values, non-matching codes, numeric precision errors, and inconsistent abbreviation, text, and date formatting.

If “Integration” comes first in your mind, past legacy integration projects have scarred you with a disdain for source system data quality levels. You view production apps with contempt and loathe the time it takes to track down data issues caused by system integrations. You investigate upstream sources to create detailed mappings and transformation rules. Typical debugging sessions consist of validating relationships to identify orphaned data, identifying attributes that contain overloaded columns (attributes containing more than one distinct data element), or fixing format errors from implied decimals.

Some of you responded with “Value Domains” or “Data Types”, indicating you are obsessive compulsive data architects compelled to organize the world into strict and orderly fashion with some degree of normalization, though you are not considered “normal” by your peers. Your concerns lie in understanding and regulating naming conventions, relationships, existence of NULL or default values, and understanding the meaning of each data element to accurately identify business rules and when two or more objects are related or redundant.

Lastly, if “Debugging” is the first item in your thought queue, you are a coder justifying why presumably good code is not working. Extreme paranoia has taught you to assume nothing about data quality, so you add tests to identify duplicates, validate relationships, enforce business rules, track change data capture, provide substitute values. Your phobia of early morning phone calls cause you to add auditing to your code to inform a DBA of data issues rather than waking you up in the middle of the night.

It is truly amazing how much we can conclude from the response to one simple phrase.

As stated before, there are no wrong answers. Aside from the innocent jab at Managers and non-IT resources, we all realize the benefits of information quality and absolutely need business involvement to understand context and domains of business information. The meaning and actions of Data Profiling change both by role and by project phase. Through profiling, we are able to identify best sources of information, learn proper ways to categorize and store it, reactively identify quality issues, and proactively define business rules to prevent future issues.

Identifying what is important to profile, when and how profiling is done, and how to share our findings across business and project resources is key. Done properly, profile results integrate to a master metadata repository and are periodically refreshed through an automated process.

This five-part series provides a tool-agnostic approach to comprehensive data profiling, focusing on information meaning and use. The next part of the series discusses system and table-level profiling. In particular, what information is important to collect at the system and table level and how can that information be leveraged by the Enterprise to help assure quality. The third part dives into attribute-level profiling and the fourth discusses attribute patterns and relationships. The final part discusses the benefits and utility of gathering profiled information into a single repository.

Continue with Part 2.

I first became familiar with Silver Creek through a chance meeting with Martin Boyd, Silver Creek’s VP of Marketing, at the Fall 2007 MDM Summit in New York. We both ran into someone from Weyerhaeuser, and all of us ended up going out to dinner at a great New York steak house.

I stayed in touch with Martin after that, and gradually learned more about Silver Creek’s product data quality solution, DataLens. I’ve said for a long time that data quality plays a critical role in master data management, so as I learned more about product information management (PIM) and product MDM, I naturally wanted to learn more about Silver Creek.

I profiled Silver Creek in April 2009, and my first hunch that they might end up getting acquired by Oracle came with the announcement later that April about the OEM relationship between Oracle and Silver Creek, where Oracle would pre-integrate Silver Creek’s DataLens solution with Oracle’s Product Data Hub.

This blog covered Silver Creek again in October 2009, where Martin Boyd did a great presentation at Oracle OpenWorld, saying that “10% of the total effort will be on the MDM software implementation, 40% on establishing governance and documenting the master data architecture, and 50% on data remediation” (according to AMR Research).

So I’m pleased but not surprised to see the news of Oracle’s acquisition today. For more information, you can read Oracle’s press release here.

“It was the best of times; it was the worst of times…” So Dickens begins “A Tale of Two Cities”, but it’s also a good description of the past year.

The first half of the year was one of the most challenging I’ve faced in my twenty-three year career in business and technology. The second half of 2009 was better – not without its speed bumps but every month was a little better than the one before it.

The macro-economic climate has been tumultuous at best. But the second half of the year showed enough improvement that Hub Designs’ revenue for the year was up 33%. Not bad for a two and a half year old company during the worst economic conditions in 80 years …

Marketing and Thought Leadership

We launched a new web site in January, and it’s been well received. Total visits to www.hubdesigns.com were up 14% over 2008.

A little later in the year, we updated the “look and feel” of the Hub Designs Blog, branding it as the “world’s fastest growing blog covering master data management and data governance”. We’ve gotten more than 43,000 hits since we started writing in July 2007, and our readership more than doubled in 2009, to about 27,000 hits per year.

We published six issues of our “Best Practices in Master Data Management” newsletter this year. We publish the newsletter about six times a year to roughly 3,300 subscribers.

I wrote six articles for Information Management magazine, including some popular ones on “Product Information Management Challenges”, how to build a business case for master data management, and how to select the right MDM vendor for your organization. I also wrote for Identity Resolution Daily, on “The Growing Role of Identity Resolution in MDM” and “Matching – MDM’s Secret Sauce”.

With our partner Siperian, we wrote a white paper in August called “When Data Governance Turns Bureaucratic: How Data Governance Police Can Constrain the Value of Your MDM Initiative” that has generated quite a bit of discussion. You can download a copy of it here.

A second white paper, called “Best Practices for Leveraging D&B in Oracle E-Business Suite”, was written in partnership with Dun & Bradstreet. It describes using D&B information to drive better supply chain performance for companies using Oracle E-Business Suite. You can download it here.

I volunteer for the Education Committee of the Oracle Applications Users Group (OAUG). A big part of that effort lies in programming the MDM track for the annual conference. This year, it was in Orlando in May, and I really enjoyed speaking there and seeing people from the Oracle community that I don’t see very often. Here’s a link to my OAUG presentation.

We participated in conference calls with Oracle Development during the year, and ultimately attended the Oracle Fusion “Hands-On Validation & Testing” session for Customer MDM at Oracle headquarters in August. It was a great chance to get some early insights into Oracle’s next major product release and to see the progress Oracle has made in building out its Fusion MDM vision, which is striking in its powerful hub technology and its elegant & productive user interface.

In 2008, we attended the Gartner MDM Summit to decide whether to exhibit there in 2009.  We were impressed enough with the conference that we did exhibit in 2009, in October in Los Angeles. We had a positive experience, so we’ll be a Silver level sponsor in April 2010 in Las Vegas. Since we specialize in MDM and data governance, we find the association with Gartner’s MDM event a powerful one.

I didn’t attend Oracle OpenWorld for the past couple of years, but this year I was glad I did. It was like “old home week”, seeing people from Oracle itself and from the broader Oracle community that I’ve met over the past 15 years. David Butler, Senior Director of MDM Marketing at Oracle, posted my presentation on Oracle’s web site, and said “you were our cleanup hitter and you hit a home run with the bases loaded”.

We also did webinars with our partners Siperian and Initiate Systems. The Siperian webinar covered the differences between MDM platforms like Siperian and ERP platforms like SAP from a master data perspective. The Initiate webinar, with Initiate’s CTO Marty Moseley, discussed developing strong MDM business case, deploying core MDM technologies and lessons learned on the “build vs. buy” question.

Social Networking

After experimenting with social networking in 2008, this year we had a coordinated strategy to use the Hub Designs Blog, Facebook, LinkedIn and Twitter to communicate & collaborate with our clients, potential clients, team members, partners, suppliers, etc.

It’s a pretty simple strategy. Short updates (140 characters or less) go out on Twitter, and are re-published on both LinkedIn and Facebook. Longer updates (i.e. blog articles) are published on the Hub Designs Blog.  We encourage responses and feedback using @replies on Twitter and comments on LinkedIn and Facebook, as well as longer-form comments on the blog. And we get them – almost every blog article gets at least one comment, sometimes as many as a dozen.

When a new blog article comes out, we notify everyone via a single update on Twitter. What’s amazing is that during 2009, social networking now drives about 15% of the Hub Designs Blog’s total traffic. And one of our clients gave us some good feedback that our social networking activities help her stay current on what we’re up to, and help her feel connected to us as a company.

Another social networking experiment that developed further in 2009 was the MDM Community.  We started this using Ning (a “social network in a box”) in November 2008, out of frustration with LinkedIn’s “Group” functionality.  It now has more than 210 members, from 23 different countries. It’s still a work in progress, but if you’re interested in master data management or data governance, you should check it out at http://mdmcommunity.ning.com. It’s becoming an international “who’s who” of the MDM world.

Summary of Client Projects

In case you think the Hub Designs team has been doing nothing but marketing, writing white papers and magazine articles, speaking at conferences, and volunteering for user groups, here’s a summary of our 2009 client projects:

1. Technology provider for vehicle dealers: integration of Oracle E-Business Suite with D&B data
2. Payroll services company: integration of Oracle E-Business Suite with external credit information
3. Information services company: technical support for customers using Oracle E-Business Suite
4. Legal information services company: readiness assessment and product MDM strategy & design
5. Simulation and engineering software company: advisor to data governance board
6. Manufacturer of oil and gas equipment: integration of Oracle E-Business Suite R12 with D&B
7. Software company: built connector between Oracle AR and D&B’s DNBi risk management solution
8. Technology company: customer MDM strategy workshop

Out With The Old, In With The New

This past year has been a lot of fun, but it has also been somewhat exhausting. So I’m looking forward to a bit more deliberate pace in 2010.

We’re very excited about the coming year at Hub Designs. We’ve got some great projects underway and in the pipeline, and we’ll be continuing to grow and expand to meet our clients’ needs and market demands.

In closing, I’d like to say how grateful I am to my family, for their patience with my traveling so much and for their unconditional love.

My initial answer was that, among companies that don’t have any formal master data management, data governance or data quality initiatives in place, duplication rates of 10%-30% (or more) are not uncommon.

When I was at D&B, we used to routinely see that level of duplication in client’s customer files.

In a study in the healthcare field, Children’s Medical Center Dallas engaged an outside firm to help clean up their duplicate data:

“Solving both the current and future problems around duplicate records helped Children’s improve the quality of patient care and increase physician acceptance of the new EHR. The duplicate record rate was initially reduced from 22.0% to 0.2% and five years later it remains an exceptionally low 0.14%. The 5 FTEs initially tasked with resolving duplicate records have been reduced to less than 1 FTE.”

“For the Children’s Medical Center, the results were heartening, not only from a care delivery standpoint but also because of the significant cost-savings that can be realized. A study conducted on Children’s data showed that on average, a duplicate medical record costs the organization more than $96.”

So it is possible to get the duplication rate down to really low levels through careful analysis and the application of the right tools, as part of an ongoing data governance program. Even the hospital above (and hospitals are usually not mentioned as practitioners of best practices) was able to maintain a duplication rate of only 0.14% after 5 years.

And there are very real costs to not de-duplicating your customer data.  Depending on the functional area (marketing, sales, finance, customer service, etc.) and the business activities you undertake, high levels of duplicate customer data can:

• annoy customers or undermine their confidence in your company,
• increase mailing costs,
• cause hundreds of hours of manual reconciliation of data,
• increase resistance to implementation of new systems,
• result in multiple sales people, sales teams or collectors calling on the same customer,
• etc.

The best studies I’ve seen of the cost of duplicate data have been in the healthcare industry. One study I saw said:

“According to Just Associates, the direct cost of leaving duplicates in an Master Patient Index database is anywhere from $20 per duplicate to several hundred dollars. The lower cost reflects the organization’s labor and supply costs to identify and fix the record while the higher expense reflects the costs of repeated diagnostic tests done on a patient whose previous medical records could not be located.

The American Health Information Management Association (AHIMA) estimates that it costs between $10 and $20 per pair of duplicates to reconcile the records. If the records aren’t reconciled, however, the costs are even higher.”

Here are three more case studies backing up the range I quoted of 10%-30%:

• Once the analysis was complete, Sentara discovered they had a significant duplication rate, over 18%. They had attempted to address the duplication rate in the past through a remediation process, but due to either technology issues or because the cost of merging and cleaning up the duplicates across their many different systems was too high, they had not yet successfully reduced their duplication rate. Source: Initiate Systems success story

• Emerson Process Management faced a tremendous challenge four years ago in getting its CRM data in order: There were potentially 400 different master records for each customer, based on different locations or different functions associated with the client. “You have to begin to think about a customer as an organization you do business with that has a set of addresses tied to it,” says Nancy Rybeck, the data warehouse architect at Emerson who took charge of the cleanup. Working with Group 1, Rybeck analyzed the customer records for similarities and connections using everything from postal standards to D&B data, and managed to eliminate the 75 percent site-duplication rate the company suffered in its data. “That’s going to ripple through everything,” she says. Source: DestinationCRM.com

• Problem: Number of duplicate records: 20.9% of Utah Statewide Immunization Information System records. Impact of Problem: Difficult to find patients in system—key barrier to provider participation, risk of over-immunization—unable to find reliable patient record, cost of unnecessary immunizations, risk of adverse effects on patients. Source: health.utah.gov.

And here’s a good quote from a white paper titled “Data Quality and the Bottom Line” by The Data Warehousing Institute:

“Peter Harvey, CEO of Intellidyn, a marketing analytics firm, says that when his firm audits recently ‘cleaned’ customer files from clients, it finds that 5 percent of the file contains duplicate records. The duplication rate for untouched customer files can be 20 percent or more.”

Every organization will need its own metrics, but left unchecked, the duplication problem is a hidden cost that drags at your company, slowing down your processes and making your analyses less reliable.

If your sales analysis reports can’t be sure that there’s one and only one record for each of your largest customers, then the sales figures for those customers are probably not right. So the entire report becomes suspect at that point.

I’d like to end with a great quote on data quality by Ken Orr from the Cutter Consortium in “The Good, The Bad, and The Data Quality”:

“Ultimately, poor data quality is like dirt on the windshield. You may be able to drive for a long time with slowly degrading vision, but at some point, you either have to stop and clear the windshield or risk everything.”

Please let us know what you think by commenting here.  We’re interested in hearing your thoughts on data quality and the issue of customer data duplication.

First, my family. It’s hard having an entrepreneur as a husband and father. I have spent more time this year in hotel rooms and clients’ offices than at home, and for that I’m sorry. Hopefully the new year will see me spending more time in the Boston area than on the road, and being more present in all of your lives. And I’m hoping the sacrifices we’ve all made in building this business will continue to pay dividends in 2010 and beyond.

Second, our clients. They’ve been great this year. As usual, I won’t name names here. But you know who you are, and you know how grateful we are that you are working with us. We try to work hard for you and to always make your projects a success, but we recognize that you’re right there with us, working hard and investing yourselves in our success.

Third, our people and team members. It is a privilege to work with you. Every day, I learn something from you. Some of the best times in my professional career have been this year (some of the hardest too!). But I always learned something, and I thank you for taking the time to teach me. It is always interesting.

Lastly, to everyone who reads our web site, this blog, our newsletter, our magazine articles, who caught one of our speaking engagements this year, or who joined the MDM Community – thank you! The extended community that Hub Designs is part of is very special to me.  People that I run into at conferences, or that send me e-mails offering to connect me with people they know, or that reach out to me through LinkedIn, or that read my postings on Twitter, are all very important to me.

A few years ago, before I really knew the value of social networking, I didn’t understand it and thought it was a little frivolous. Now, I understand its power – to connect us to one another, to effect change, to weave people and companies together in a new way, to make the 21st century more intimate, to allow me to sit in my office and say thank you to thousands of people at once, without blasting a newsletter into people’s inboxes, without sending out a mailing, without placing an ad, without doing any of the traditional things companies would have done at one time to get their message across (and are still doing).

Today, to me, it’s all about authenticity, and helping people, and being in the right place at the right time. And for that, I’m grateful.