What Is Data Migration?
Data migration is the process of moving data from one storage device to another. The premise is simple, but the process can be complex. When migrating data, database or application logic may need to be re-executed, such as reformatting or transforming data, changing the database schema, or refactoring database stored procedures.
Data migration is often required when an organization moves data to a modern database, transfers it from an older storage solution that is no longer supported, or migrates it from an on-premises solution to a cloud-hosted solution. Another use case is big data migration—migrating large volumes of data to improve availability for other applications that need to access it.
It is important to ensure the security and integrity of data during the data migration process. Therefore, developing a robust data migration plan requires careful analysis and selection of an appropriate data migration plan. Choosing the right approach and migration tool can be the difference between a smooth migration and a migration with bugs, potential security issues, and data integrity issues.
Main Types of Data Migration
Storage Migration
Storage migration is the process of moving data from one storage location to another. The process involves validating, duplicating, and cleaning data. The more data the organization transfers, the longer the migration time.
Organizations undergo storage migration to gain a technical advantage rather than to increase storage capacity. For example, an organization can shift data to a certain cloud to increase scalability and gain access to more innovative features.
Database Migration
Shifting data from one database engine to another or upgrading an existing database is known as database migration. Opting for the first option requires careful planning as there might be a difference in data structures between the two database engines. For example, the source database might be relational while the target is non-relational, or vice versa.
Application Migration
This refers to moving software programs from one operating environment to another. An example of this migration strategy would be when an organization moves a critical business application from an on-premise data center to public cloud servers.
When transferring a software application, the difference in each computing environment’s data models, specifications, and configurations may become a problem. To overcome this challenge, middleware products can bridge the gap between different technologies, allowing for an easier migration process.
Data Center Migration
A data center is an organization’s control hub, where all of the critical data, applications, and softwares are stored or hosted. Shifting from a data center to a completely new operating environment (another data center or a cloud) is called data center migration.
Migrating between data centers becomes important when a company’s needs have outgrown the current data center. Since this is an extensive process, it must be well-planned and assessed before execution to ensure data safety.
Business Process Migration
This type of migration pertains to the movement of business applications, business metrics, and processes data to new environments. For example, an organization’s products, services, customer data, and operational information are typically migrated.
Business process migrations most commonly occur when multiple businesses are merged or acquired (M&A), or when a need exists to reconstruct business models and enter new markets.
Cloud Data Migration
Cloud data migration refers to transferring data from local storage, including all operations and processes, to the cloud. It can also mean transferring data from one cloud provider to another.
Organizations use cloud computing services like AWS, Google Cloud, or Microsoft Azure to store and manage their data in the cloud. These services provide unlimited scalability, improved performance, no maintenance costs, and reduced costs for some data storage scenarios.
Data Migration Process
Below are the main phases of data migration:
Planning
A data migration process should always start with a planning phase. It requires evaluating existing data assets and creating a suitable migration plan. It typically involves the following steps:
- Refine the scope—this step involves filtering out excess data and defining the smallest amount of data required to run the migrated system effectively. It requires performing a high-level analysis of the source and target systems and consulting with the data users directly impacted by the migration.
- Assess source and target systems—a migration plan must be informed by a comprehensive assessment of the source system’s operational requirements, followed by an analysis that determines how to adapt these requirements to the new environment.
- Set data standards—this information enables teams to identify problem areas across each migration phase and avoid unexpected issues during the post-migration phase.
- Estimate the budget and timeline—after refining the scope and evaluating the systems, you can move on to selecting the suitable migration approach (trickle or big bang), estimating the required resources, and setting realistic schedules and deadlines. According to Oracle, an enterprise-scale data migration project lasts 6-24 months on average.
Migration Design
A migration design defines the following aspects:
- Migration and testing rules
- Acceptance criteria
- Migration roles and responsibilities
- Data migration technologies
Extract, transform, and load (ETL)
A data migration process typically involves an ETL process. You can use various technologies for this process. Projects involving complex data flows and large data volumes typically require employing an ETL developer or a software engineer with expertise in ETL. ETL developers and data engineers can either customize third-party ETL tools or create scripts for data transition.
Data mapping
Data mapping is an integral part of ETL. It is typically performed by a team consisting of an ETL developer, a system analyst familiar with the source and target system, and a business analyst with a deep understanding of the migrated data’s business value.
Duration
The migration design phase depends on the time required to write scripts for ETL processes or acquire the relevant automation tools. The migration design phase can take a few weeks if you already have all the required software in place and only need to customize it. Otherwise, this phase can span over several months.
Execution and Testing
This phase involves implementing the ETL processes previously designed. A big bang migration typically lasts a couple of days, while transferring data in trickles can take longer. However, a trickle strategy has the lowest risk of critical failures and zero downtime.
Here are execution and testing best practices to consider:
- When executing a phased migration, ensure these processes do not hinder normal system operations. The migration team should constantly communicate with business units to determine which group of users to roll out each sub-migration.
- Test continuously rather than as a separate phase. Carry out testing across all phases, including design, execution, and post-migration. For a trickle approach, test each portion of the migrated data to remediate issues timely.
Frequently test to ensure data elements are safely transferred to the target infrastructure at a high quality and per all predefined requirements.
Data Migration Challenges
Because the data migration process involves many moving parts and large amounts of data, there are a number of issues that can interfere with the process. Below are some of them and their impact on the migration process.
Risk of Business Disruption
Organizations typically don’t want to stop production work during the migration process, so users don’t face downtime and can ensure that all systems are up and running. However, this can be challenging to achieve. Another risk is that changes to data during the migration process will lead to system inconsistencies and inaccurate data.
Risk of Data Loss or Corruption
During the migration process, organizations should aim to minimize the risk of data loss or corruption. Data may be lost due to various reasons such as incomplete or inaccurate transfer, system incompatibility, human error, etc. This can cause financial losses, hurt an organization’s reputation, and result in compliance violations.
Risk of Exposure
The risk of a data breach is a serious risk in the migration process. In the process of data migration, the system and the data itself become more vulnerable. In order to move data, By exploiting vulnerabilities in the data transfer method or in target storage system, an attacker could corrupt, steal, or tamper with data in transit, resulting in a failed, incomplete, or corrupt migration.
Data Migration Strategies
“Big Bang” Migration
Big Bang data migration completes the transfer within a limited time. When data is processed by ETL and migrated to a new database, there is downtime on the production system.
The advantage of this approach is that everything happens in one time-boxed event. However, the pressure can be high because the business is running with a critical one resource offline.
If the big bang approach is best for your business, it’s a good idea to run a realistic test of the migration process before the actual event.
“Trickle” Migration
Trickle migration completes the migration process in stages. During implementation, old and new systems run in parallel, eliminating downtime. Processes running in real time keep data synchronized between the two environments.
The design of these implementations can be quite complex compared to the big bang approach. However, when done correctly, the added complexity often reduces risk rather than increases it.
Lift and Shift
The lift and shift migration moves an application and its data to the cloud with minimal or no changes to the application architecture, authentication mechanisms, or data flow. When no change is required, the application can be “lifted” as-is from the source environment and “shifted” to a new location.
A lift and shift migration should be planned, considering the application’s network, computing, and storage requirements. It involves mapping from the available resources in the source infrastructure to the cloud provider’s resources. Most cloud vendors offer on-the-fly upgrades to ensure customers can start with a smaller product and scale as needed.
Best Practices to Follow for Successful Data Migrations
Explore and Assess the Source
Before migrating data, you need to know what you are migrating and how it will fit your target system. Find out how much data is being collected and what that data looks like.
There can be many data fields, some of which do not need to be mapped to the target system. Your source may have missing data fields that need to be pulled from somewhere else to fill in the blanks. Ask yourself what needs to be migrated, what needs to be left behind, and what is missing.
If you omit this source review step and assume you know the data, migration can be time-consuming and expensive. To make matters worse, organizations can experience serious flaws due to data mapping issues, halting a migration completely.
Have a Solid Data Backup and Protection Plan
Consider a scenario in which the migration does not complete correctly. Back up your data regularly, and use tools and techniques to protect your data from various error scenarios. This is also useful if a file gets corrupted for unknown reasons during the migration or if some data is missing or incomplete. Carefully map your data to destinations to allow team members to see exactly where the data came from, where, how, and when.
Test and Validate Migrated Data
After a successful migration, validate that all data is where it should be. Clean out old data and check that permissions are applied correctly. It is a good idea to back up the old legacy system so that if the new one goes offline, you can access it from another secure location.
Audit and Document Processes
Complete documentation of a data migration process is critical for compliance in regulated industries. In some industries, regulators may require evidence that appropriate or reasonable controls are in place over sensitive data, such as financial or medical information. The documentation should not only provide evidence everything went right, but will also help you identify areas for improvement for your next migration.
Protecting Data During a Migration with Imperva
Imperva Data Security Fabric protects all data workloads in hybrid multicloud environments with a modern and simplified approach to security and compliance automation. Imperva DSF flexible architecture supports a wide range of data repositories and clouds, ensuring security controls and policies are applied consistently everywhere.
Imperva helps organizations protect data during migration by augmenting traditional enterprise security approaches with controls for the data itself, driving policy-compliant data handling behavior, and helping security staff pinpoint and mitigate data threats before they become damaging events.
Imperva Data Security Fabric combines automated security and compliance platforms into a single technology solution to remove the complexity of managing a diverse set of data security tools for data oversight.
