Cleaning and Mapping Data for Collections Management Systems

Once an organization chooses a collections management system (CMS), making a smooth transition necessitates planning. While archives should do data cleanup regularly, most only realize that they have a data integrity issue, and its extent, when they are planning to migrate data into a new system. 

Before cleaning the data, see how your current data will work in the CMS. Whenever possible, use a copy of real data when evaluating the system. Evaluating a system with real data gives a higher chance of uncovering potential issues as well as identifying areas for data improvement. Review a list of routine tasks performed with current collection information, including generating reports or tracking the information staff members access. Conduct exercises addressing the range of processing situations encountered. Report any issues to the vendor; if a modification resolves the problem, the system may still prove a good fit. 

Cleaning Data

When planning and transitioning to a CMS, clean up the data in advance. The better the data is, the easier it will transfer. If the team disregards data clean up, any problems with the records will exist in the new CMS too. Ask the team:

• How much cleanup and standardization of existing collection records will be necessary? 
• Which records will the team transfer to the new system, and which are obsolete? 
• What paper records do archivists need to enter, if any?

Data cleaning is an ongoing, laborious process. In a cleanup project, consistency is critical. If multiple people clean the data, ensure they apply the same standards. Use a task management system to track collaboration, minimizing oversight and confusion. Archivists can use software for global “find and replace” features. They can search data fields or a keyword or phrase and then replace the selected word or phrase with a case sensitive change, correct spelling, or a different word. If the organization lacks the in-house expertise to clean, map, and migrate data, contact experienced consultants or find out if your CMS vendor offers such services. 

Beginning Anew

Implementing a CMS—either new to the organization or a migration—is an opportunity to start afresh. Delete outdated or unnecessary records. The fewer the records, the easier migration will be. Vendors often charge by the record, so reduce the records to only what is necessary. You will want to perform deduplication by identifying all duplicates in the system. Finding and deleting these records is usually relatively easy. Archivists will also want to verify and validate that information is correct throughout their legacy system.

Mapping Data

If the archives imports legacy data from one or more existing systems, it is unlikely that the fields will correspond exactly to those in the new CMS. During data mapping, old fields match with fields in the new software. Without careful data mapping between the systems, archivists risk importing data incorrectly. 

Normalize the data so that the fields are uniform, names are in the correct order, and data is displayed correctly. Work with the vendor to develop procedures for mapping data between systems and request a copy of the mapping process to ensure that the software maps old fields. Check that field lengths on the mapping document ensure that data is untruncated. Review audit trails of the importing procedures to make sure data is importing correctly. During the process, sample records within the new software to compare old fields to new fields and ensure data quality. 

If your organization will enter paper-based documentation into the new CMS, the cost of entering the collection data will be high. Maximize the investment by setting up standard operating procedures to ensure consistent data entry. Errors and omissions can compile over time, resulting in lost institutional knowledge. Look for other opportunities to complete records with missing or erroneous data. 

Confirm Data Governance in the New System

Ensure that your cleanup and migration efforts last. Establish data governance rules for your CMS to minimize the need for data cleaning in the future. Moving legacy data into new systems is always complicated, revealing history hidden in data from past years. However, careful planning and execution allows for the migration of legacy data while avoiding pitfalls. As with any significant work changes, use the implementation of a collections management system to foster better information practices that make cultural heritage collections easier to preserve and access.