Over the last couple of days I have presented at a couple of conferences on the subject of Asset Information Management. When discussing with a delegate on Tuesday the nature of data quality problems, they asked:
Are data quality problems often user problems?
My immediate response was that all data quality problems are user/people problems. If we recognise this fact, then solving data quality problems involves solving people problems.
This may be a slightly controversial view, but if we recognise that activities such as data profiling, data matching, data cleansing etc. are only correcting the symptoms of the problem, then these activities will only tend to provide short term, unsustainable improvements in quality if used in isolation.
So what are the root causes of data quality problems? Some examples from my work in transport and utilities include:
- Users not providing data updates because they don’t understand the need
- Staff finding field data problems, but assuming that someone else will correct them
- Productivity targets not allowing staff the time to provide data updates as they will lose pay
- Staff keeping ‘local’ copies of data sets in spreadsheets and data bases
- Admin staff bulk corrupting data through poor process controls
- Data being migrated between systems without assessing and correcting data quality issues
- Not assessing the accuracy of data (those DQ ostriches again!)
- User input forms with the order of entry fields incorrect
- Drop down categorisation lists left at the default setting by users
- Lists of fault codes that are so long, staff have problems finding the correct code, so ask for a new one to be created…
- Staff storing data in a password protected spreadsheets which are in turn stored in a Notes database
I could go on, but you probably get the idea….
So if the above list represents some of the root causes of problems, these are the areas that will need to be corrected in order to provide sustainable improvements in the quality of outputs, i.e. improvements in data quality.
Am I being radical? What do you think?