Tag: data quality

Test data management

Posted on by Ferenc Györ in Data quality, Data strategy, Test data management

There is a significant data management aspect to testing – the test data management. Let’s have a quick intro and have a glimpse on project management reality on testing – then,  look at the test data management a bit more in detail.

Software testing

How long you have to test a Dacia to turn it to a Mercedes?

I’ve heard the above quote during university from one of our professors. Although the answer is obvious a relatively big amount of software projects still somehow believe that poor design can be corrected during the testing phase.

In reality the amount of code going into production without proper testing is surprisingly big (source of the image).

 

testing-is-for-wimps-real-men-test-in-production

 

 

Excuses

Sometimes we hear excuses why not to do testing properly. Here some of them:

  • I’ve heard already statements from a vendor that code-level unit tests were enough and no further testing should be done. That is just simply not true.
  • If you are in the custom software development business OR you customise a piece of existing software you must keep track of your business requirements and maintain the changes of those during the project until you hand over the software. How you do this depends a whole lot on your methodology, but there is no way avoiding this. The granularity of the requirements should reach a level so that they are testable: also you can objectively decide if a requirement was fulfilled or not. Please do not think this is an expensive “gold plating”. If you do not have a control over your requirements you will go into a trial-and-error loop, just like the one we described in this blog entry.
  • One of the other excuses in the top 5 is that software supporting the organisation in the testing is expensive. If you hear this excuse just check out some open-source testing software like TestLink & co. (likewise for bug tracking you could use Mantis).
  • If you hear someone saying the amount of labor put into setting up these tools are huge and you should use Excel to manage the test cases and the execution then please ask this person to measure the time needed to manage the efforts in lack of a centralised tool.
  • If you hear that there are no resources to do regression testing (testing if new function does not destroy properly working functionality) you might think about using robotics to automate at least a part of the testing.

Test data management

Now imagine for a moment that in a software project proper amount of resources went into the requirement assessment and there are proper test cases to execute.

What often times is still missing is the proper test data and an agreed way how to manage it e.g.:

  • Ensure how to put the system into an initial state “ready for testing” before a test run
  • Ensure your test data supports retesting in case of bugfixes
  • Ensure you have proper test data for multiple test runs (sequentially/parallel)
  • Know which data can not be reused (e.g. certain identifiers) and how to generate new data in a systematic way
  • Ensure that the above is conducted as a routine task – in an optimal situation in an automated manner

Please note if you make end-user trainings with a training system you have the same challenges to solve.

Here are some recommendation who to put together a proper test data set:

  • Tests should be reproducible. Optimally you should be able to restore the initial state of the system (before the testing take place) easily by e.g.:
    • Having a virtual image of the systems in place that you restore+patch+upgrade  and store before each test round. You can think about putting such a solution to AWS or to Azure (or the like) – if your architecture is modern enough.
    • Backing up the database and restoring it (usually not easily possible with multiple systems integrated)
    • Generating test data with robots or even manually before the test run and assigning the test data to the proper test cases
  • Systems are integrated. This means that during testing you have to consider that you have some limitations.
    • Sometimes it is possible to have a fully separated test environment with all the integrated systems. If you have this lucky situation you can usually think you have a “single system” to deal with.
    • If this is not possible (usual case) you should think about system-level data consistency rules. (Well-well, if you have a proper model of your data, that helps.)
  • Automate-automate-automate
    • If you have 30% more initial efforts to automate test data generation, just invest. The more often you test the bigger the gain will be on your original efforts.
  • Privacy
    • There are cases where test data must be very close to production data. Should this be the case consider privacy rules.
    • Whenever possible just please make your lives easier and use non-productive data for testing. Some database provider offer cloning features with data masking/scrambling.
    • Mostly with data & analytics projects you have algorithms (e.g. grouping, classification etc.) that must be trained with productive data. Please note that this is not testing, and a separate, restricted environment can be needed. As a result of such training runs a model is constructed, that is usually small in size. This model usually does not contain any sensitive information by itself and hence can be transported into any other systems – including the test system. Please note however that a model working well with productive data can be useless when meets training data.

Quality

Posted on by Ferenc Györ in Data quality

A typical (suboptimal) timeline in many project dealing with data integration (warning, provocative):

  1. What is the data we need? – Do workshops
  2. Where is the data? – Look at the documentation/Glossary — if there is one
  3. Challenges like: Oh, wow, we meant some other data… Oh, wow, why are these fields empty?
  4. All set, we are writing ETL/WS/… code to integrate the data
  5. No data could be loaded/integrated, lots of errors
  6. Repeat 3-4-5 in a trial-and-error loop as data quality is not good enough
  7. NO ERRORS FROM THE LOAD PROCESS 🙂 We are ready! (Fanfares)
  8. Oh, no, business says data does not make sense – data quality is not good enough
  9. Work additional 2 months (the timeline can be anything from 5 days to 12 months) repeating steps 1-2-3-4-5-6-7 and 8 until data quality will be good enough
  10. OK, now most of the data makes sense (Fanfares again)

Of course the above is exaggerated but everyone who were involved in data integration knows that this is not that far away from reality.

There is a lot of things you can do to make this loop much shorter (see in a later post), and you can also have a historic view how similar situations arise.

In this post I will point out just two aspects:

  • Do you see how many times bad data quality is mentioned? But what is exactly bad data quality? Can we measure data quality in an objective manner?
  • Sadly the point #7 is where some projects way to often think that the integration work is finished. In fact by that time you are not even half way through. Why is this?

Before answering I need to cover a bit of theory. The theory of relational databases together with the common implementation teaches us how to put together a database structure that guarantee certain aspects of data quality. Possibilities to ensure good data are (without mathematical precision):

  1. You can define data types (text, number etc.) – but of course you just can store all numbers as texts…
  2. You can set up keys (also unique, non-empty values that identify e.g. a car, a person etc.) – but you are allowed in most database systems not to
  3. You can set up referential integrity constraints (e.g. there is no credit card without a cardholder) – but you are allowed not to
  4. You can define domains like size for humans is something between 0-270 cm – but you are allowed not to
  5. You can define patterns your data must have like if some text is a valid phone number – but you are allowed not to
  6. You can even define more complex rules/programs that allows you to check every aspects of the data you can possibly think of  – but you are allowed not to

The reality is that these are possibilities to enhance the data quality of your data that a project may or might not implement. The more of these you are willing to implement the more you have to know your business rules. The more of these you implement in fact the slower will your system be (last time this was a valid excuse in the early 2000’s).

 

So back to the above questions:

  • Data quality can be measured ultimately with validation through business rules. The more data complies with the business rules the better the quality is. Sometimes not all business rules are known in advance. Sometimes they change and the data management is not updated. Sometimes the business rules are so complicated that coding them does not worth the effort. Sometimes there is not enough time or money to implement all rules. In short: data quality is usually not fully known before the data integration begins.
  • When the data integration stream reaches point #7 only that subset of business rules is validated that is somehow implemented. Usually this is only the fraction of the existing business rules. Pont #8 and #9 is nothing else but figuring out the business rules that are not stored in any systematical way and trying to clean up using the new knowledge.

Are there ways to do this better? Definitely: with data strategy you can do a lot to get out of such troubles. I’ll share you some best practices in a future post.

Did you make a different experience? Do you have a different view?

Client data management

Posted on by Ferenc Györ in Data strategy

Reading the news on the train today I found an interesting article with data aspects – and I decided to share it with you in a short post.

A recurring topic in many discussions around data management is the current quality of the data and where peer companies stand.

Maybe it is easier to give an answer if you read this article on engadget. Summary: a couple living in Indiana ordered ca. 2’700 items from Amazon, of a value USD 1.2 Mio. They reported Amazon that they want to return the products and got their money back. In fact they never returned anything and re-sold the items, making USD 750k over 2 years.

How did they do this? According to engadget:

The Finans created hundreds of false identities and fake accounts in order to pull off their scheme.

We know how great job Amazon is doing in terms of data quality and data management. And even then this fraud was possible by temporarily tricking out Amazon’s customer deduplication algorithms (a somewhat older article but still a good summary).

Reaching good data quality is a continuous work and not a one-time effort, not just for your company but also for key players like Amazon.

Multimaster

Posted on by Ferenc Györ in Data strategy

If you are working with data, from time to time you will be confronted with “multimaster” situations where the same kind of information (typically client, product, installed base etc. data) is managed in multiple systems in parallel.

Let me give you an example: one of my projects some years ago. Our goal was to cleanse and sustain high quality of master client data. The data was managed in a CRM system and other systems in parallel.

But why did our client have all these systems? Why didn’t they build just a single client management system? Didn’t they know that everything else was not good ab ovo?

Well, it is easy to make this statement now. But historically the story is a different one.

Some years back all business lines were happy with their own systems and processes. (Hands up how many of you have seen this…) Some of these business lines were even completely different companies. There was no valid business case justifying the integration of the client data.

Also different systems stored different information about the same client. For one business unit a client meant a corporate, for an other unit a private person.
The business requirements for the systems managing client data were captured in separate projects. One of the projects finished one year ago and one in the last century. Of course one of the systems was designed to serve the needs primarily e.g. of the sales team, the other system design concentrated on the marketing needs, and so on.
How many times, do you think the sales team spoke about the client during their project? I guess the client was the single central topic of most discussions. I would bet the same applies to the marketing project. Even if the two projects would have had run in parallel, the chances that the projects share the same understanding of the client in terms of processes and data is (at least in the real life) surprisingly small. Why? The respective project teams were responsible for their own scope and budget and had to decide what aspects of the client are in scope and which are not.
Imagine the second project were started 2-3 years after finishing the first: how many people would participate in both projects? (Attrition rate.) Is the first project documented in a way that the knowledge gained can be reused in the second project? Well, usually not.

After the systems were finally created some teams were interested in more data their systems could manage. An easy and cost-efficient solution was to create some spreadsheet-tables to store some specific attributes in some special cases. The more so as this could work without involving any other business lines or IT. Anyhow the business team was under pressure and needed a solution ASAP – no time for lengthy discussions and no energy to convince the whole corporate.

Later times changed. Marketing and sales both wanted to have all client information stored and managed in a single, central place. The world has changed and there were dozens of viable business cases requiring to do so (up-selling, cross-selling, uniform communications, GDPR – just to name a few).
Of course designing the CRM the goal was to make it the single system managing client data. Sadly the task of collecting, cleansing and syncing already existing data was so huge that the integration was never fully finished in fact. Why? Because before the CRM implementation nobody integrated the client data in this depth in a permanent read-write manner and hence nobody had a realistic estimate on the complexity of this task.
When I joined the project the same client data was partially synchronized automatically, partially manually and partially… well, not at all.
The business processes were modified to work around this situation – somehow. There was a responsible person for the CRM system – but there was no single person responsible for the client data.

If you realize that this could be your organization as well, you might be right. This situation is more common then participants of an analytics conference would admit.

It is important to emphasise that a multimaster situation is usually a consequence of a series of valid business decisions.

This is especially interesting now when most companies want to exploit data in a more advanced way then before. Sadly with questionable data quality no really valuable analytics can be done.

In our project IT experts were working on finding a solution. They were trying to figure out e.g. why client types for marketing are entirely different from client types for sales. They involved business experts but somehow there was no agreement in sight.

In my view the root cause of data challenges lies within the business processes. In this project it was also not different: IT tried to solve a business problem – and it did not work out.

In the next blog entry we I will give you some insights what we did to get rid of this situation and how we managed to give business quality data.

Stay tuned!

(Please note: the above client situation is based on actual client projects. Some details were changed to comply with contractual obligations.)