13.3. DATA QUALITY CHALLENGES

There is an interesting but strange aspect of the whole data cleansing initiative for the data warehouse. We are striving toward having clean data in the data warehouse. We want to ascertain the extent of the pollution. Based on the condition of the data, we plan data cleansing activities. What is strange about this whole set of circumstances is that the pollution of data occurs outside the data warehouse. As part of the data warehouse project team, you are taking measures to eliminate the corruption that arises in a place outside your control.

All data warehouses need historical data. A substantial part of the historical data comes from antiquated legacy systems. Frequently, the end-users use the historical data in the data warehouse for strategic decision making without knowing exactly what the data really means. In most cases, detailed metadata hardly exists for the old legacy systems. You are expected to fix the data pollution problems that emanate from the old operational systems without the assistance of adequate information about the data there.