Incremental Migration
It is often necessary or desirable to migrate documents from one system to a new one in several steps. This adds to the complexity!
If you are aware of the worst risks and pitfalls in advance, they can be dealt with in due time.
Breakdown of Migration
One of the most important things to consider when planning a migration is whether to migrate everything at one in a ‘big-bang’, or split the content into parts and do incremental migrations. The decision may be made for either technical, project and business reasons.
In this article we will shed some light on some of the arguments for and against incremental migration based on our experience.
The Business Drives the Segmentation
If you decide on an incremental migration, the business’ needs and usage of the documents will, in most cases, provide a natural segmentation of the documents. There is no one right way to segment, and it varies from project to project and from company to company. The segmentation may be derived from the structure of the company, geographics or business functions. The most common we have seen are:
- Business divisions
- Country
- Language
- Departments
- Business processes
In any segmented migration it is highly advisable to begin the migration with one of the simpler segments of documents. If you have a small department, a simple business process or another way of dividing that will provide you with a simple start, it may be worth considering beginning the process here.
However, don’t use this parameter blindly. You should weigh the pros and cons against each of the issues and benefits discussed below.
Issues in Incremental Migration
When splitting your migration into parts performed at different times, there are several fundamental issues to deal with.
Two Systems
In any segmented migration you will have a period with two parallel systems in operation. Some users will have their documents in the new system, others in the old. Some users might even be using both systems at the same time. This should be avoided if possible, and is something that should be factored in, when planning how to segment the migration.
Duplicate Copies
When migrating digital documents, as opposed to physical ones, they are actually copied rather than moved. This provides a path for fallback and for verification. The drawback is, of course, that it creates duplicate documents which must be handled.
Once a document has been migrated, the copy in the old system should either be locked for editing, tagged as migrated or perhaps just deleted. Eventually it will be deleted when the old system is decommissioned, but if not all users have access to the new system yet, the documents need to be available (for viewing) in the old system for a period. In some cases, it has not been technically possible to lock or tag the document, we have solved this by clearly marking inside the document, that this is a view only legacy copy.
If having both copies available for updates a necessity, there are solutions for this. However, it would require something close to real-time synchronization. In most cases, technical solutions do exist, but due to the complexity we would only suggest this if it is absolutely necessary.
Don’t Duplicate Data
Once a segment of the data has been migrated, new data of this kind in the old system should be avoided. Ideally all work on this segment of documents should be done in the new system, including creation of new content. Although, depending on your choice of segmentation, it may not be possible. Some users who need to create content, may not have access to the new system. If this is the case, it is important to have a migration strategy and/or system to keep track of migrated vs. new documents.
Workload and Business Continuity
The more often a process must be executed, the greater the need will be for a simple and smooth process. If your migration, for example, requires you to perform a full back-up, shut down the system and import via hand-written scripts, then the process will neither be simple nor smooth. The process that you need is one that can be executed with minimal disruption of normal operations. Think automations and hot imports.
Benefits of Incremental Migration
With all these issues, why would anyone ever choose to do a segmented migration? We here at Strator would, when asked, recommend it in most cases simply because there are many advantages to segmentation.
Keep Up with System Implementation
When implementing a new EDMS system, the implementation, or at least roll-out, is typically done step by step. A segmented migration can align with the roll-out and ensure a smooth transition to the new system for the users.
You Learn as You Go
The concept of a migration may appear as a very simple and straight forward process, or it may seem like a very abstract thing. Most users and, indeed even most project participants, are not used to doing migrations. Seeking advice and starting with a pilot migration will get you off to a good start. Still, there will inevitably be lessons learned along the way, and an incremental migration can help you manage these issues and adapt your strategy.
Quality
Data quality is imperative for a successful migration. Lack of quality leads (at the bare minimum) to the inability to find the document you are looking for. This in turn causes user frustrations, bad experience with the new system, loss of productivity and in regulated industries perhaps even a remark from the authorities. Summed up, the cost of poor quality in the migration process can be very high.
Usually, the old and the new systems will have a different metadata structure and requirements. Furthermore, implementing a new system is a good opportunity to clean up not just your documents, but also the quality of the metadata. To map, clean up, index and enrich the metadata, requires a lot of processing and work.
Trying to improve data quality on all types of documents at the same time is overwhelming and will exhaust the project. However, by dividing the migration into segments the project can focus on quality bit by bit as the effort is spread out and thereby making it possible for a higher quality to be reached.
Downtime and Processing
Migration takes time. There is simply a physical limit to how much data you can squeeze through the technical connections, and how fast it can be processed. If the migration requires downtime for one or both systems, the time needed may not be acceptable for the business. In this case, the incremental migration can distribute the downtime over several weekends instead.
How To Go About It
As discussed above, an incremental migration may be a good idea in many situations. The most challenging part is finding a meaningful segmentation of the documents. If a good segmentation can be found and aligned with the plans for the system implementation, then the pros of the segmented migration quickly outweighs the cons.
Many of the challenges mentioned above can be addressed by involving business stakeholders and agreeing on the best and most practical solution.
As always, make sure what’s agreed upon is communicated and aligned throughout the company.
Technically, the top-most important factor for success in an incremental migration is control. You need to know exactly which documents and which versions are migrated. You need to know who has access to what and with which privileges. Given the amount of data that needs to be tracked, this cannot be done manually or by keeping score in an excel sheet. You need a tool that can track the migration and keep you updated with reports.
Many migration tools on the market can scan your documents, process the metadata, and perhaps even import the documents into the new system. Some of these tools can keep track of document versions, track and trace the data transformations and perform baseline- and delta-scans to report on any changes that might have happened. In an incremental migration with multiple segments, executed over a period, we will always recommend a tool with full traceability.
Conclusion
Regardless of whether your incremental migration is the result of a deliberate choice or an imposed requirements the challenges are the same. These challenges are not trivial but can be overcome by careful planning and coordination with the business and by using the right tools.
Important things to consider when choosing the right tool is whether:
- It is in fact a tool for migrating documents (and not just for structured data)
- The tool has sufficient functionality for tracking and reporting
Migration can be complicated and overwhelming, but if done right, it can make a huge difference to the new system and the end-users.
Read more about migration on our knowledge page.
