Data migration and data quality
Moving the data is the easy part.
Knowing it is right is harder.
A migration succeeds when the new system opens with customers, suppliers, products, stock, balances and transactions people can actually trust. We treat migration as a controlled business process, not a final technical task before go-live.
We would rather find a data problem before go-live than explain it afterwards.
Sound familiar?
A new system cannot fix data nobody trusts.
Migration problems rarely start with an error message. They appear when somebody asks a simple question and nobody can answer it confidently.
Nobody agrees which record is right
The same customer, supplier or product exists in several systems with different information.
Years of history have become part of the scope
Nobody uses most of it, but nobody wants to be responsible for leaving it behind.
The old system contains workarounds nobody understands
Codes, fields and spreadsheets acquired meaning over years without anyone writing it down.
Reconciliation happens at the end
The data loads, then finance or operations discovers the totals do not agree.
How we approach it
Do not start by moving everything.
Migration decides whether customers are correct, suppliers can be paid, stock can be trusted and finance can explain the opening position. That makes it a business process with a technical step in the middle, rather than the other way round.
- Understand: What exists, who owns it and whether it is still needed.
- Clean: Resolve duplicates and invalid values before they become new-system problems.
- Map: Agree what the old data means in the new model.
- Move: Load it through a repeatable and controlled process.
- Reconcile: Prove that source and destination agree.
- Prove: Let the business confirm the data supports the process it has to run.
What should actually migrate
Not everything deserves to come with you.
Nobody wants to be the person who decides ten years of history can be left behind. It is still a decision, and it sets the size, cost and risk of the migration more than anything else does.
Migrate
Needed to operate in the new platform.
Archive
Needs to stay accessible, but not as live transactional data.
Rebuild
Can be recreated more reliably from another source than carried across.
Leave behind
No continuing business, legal or operational value.
Retention, regulatory and contractual obligations still need to be considered before information is archived or removed. That is a decision for your own advisers, not for a migration plan.
Not all data carries the same risk.
Master data and the opening position fail in different ways, so they should not be planned as if they were the same job.
CDM
Migration capability built from doing this repeatedly.
After working through migrations across different generations of ERP, it stopped making sense to rebuild the mechanics from scratch every time. CDM is our reusable migration capability, built around Microsoft data and automation technologies, to make extraction, transformation, validation and repeatable loading more controlled.
Extract
Business users select the data and legal entities in scope, and it lands in SQL Server staging away from the live source.
Transform
Legacy structures are converted into the shape the target platform needs, with mapping held in one controlled place.
Validate
Records that fail the agreed rules are identified in a test environment rather than in production.
Repeat
Azure Data Factory pipelines run the cycle again as cleansing progresses, so go-live is not the first real attempt.
CDM makes the technical migration controlled and repeatable. It does not decide what your data means, and it does not read an undocumented legacy field for you.
The source system is rarely tidy.
Some migrations start from a recent cloud application. Others begin with an ERP that has been modified for more than a decade, most often Dynamics AX 2009 or AX 2012 on the way to Finance & Supply Chain. The approach has to cope with both.
- 01
Legacy ERP, finance and operational data
- 02
Profile, clean and map
- 03
CDM and Microsoft data services
- 04
Dynamics 365, Business Central or CRM
Prove it before cutover
A successful import is not proof of a successful migration.
A hundred thousand records loading does not prove the right records loaded, the values are correct, balances agree, relationships survived, users can operate or reports reconcile. Go-live is a poor time to discover the stock balance does not add up.
Count it
Did the right number of the right records arrive?
Value it
Do the balances and quantities carry the correct values?
Reconcile it
Can the difference between source and target be explained?
Test it
Can users run their own scenarios on the migrated data?
Sign it off
Do the people who own the data accept the position?
Data quality is whether the information is good enough for the process that depends on it.
Customer data
Can we invoice the customer and reach the right people?
Supplier data
Can purchasing order, and can finance pay correctly?
Product data
Can sales, planning and the warehouse actually use it?
Inventory
Does the quantity and the value reconcile?
Financial data
Can finance explain the opening position?
Go-live should not be the first time the migration works.
Rehearsing the migration before cutover is how a team learns how long it takes, what fails, which cleansing is still outstanding, what depends on what and whether reconciliation holds. By the real run, it should be the most familiar thing on the plan rather than the most exciting.
A file loading successfully does not mean finance will agree with it. That is why business validation, not technical completion, closes the migration.
Who decides what is right
IT can move the data. The business has to say whether it is right.
We profile, extract, transform, map, validate, load and reconcile. What the data means, and whether it is correct, stays with the people who work with it every day.
Finance
Balances, receivables, payables and the opening position.
Sales
Customer records, contacts and the commercial terms attached.
Purchasing
Suppliers, payment details and purchasing terms.
Operations
Products, stock, locations and the detail people work to.
HR
Employee information, where it forms part of the scope.
The same applies after go-live. If nobody owns data quality once the project team has gone, the new system eventually inherits the old problem. Explore Optimise.
Migration and data work in practice
What we can currently show, and what is still being written up.
Selected migration stories are being prepared for publication. Until they are, this is the verified capability behind the migration work rather than a case-study library.
Legacy Dynamics estates
Migrations from Dynamics AX 2009 and AX 2012 into Dynamics 365 Finance & Supply Chain, including the legal-entity model, layered customisation and the reporting dependencies that came with them.
Controlled, repeatable loading
CDM stages selected data and legal entities in SQL Server, applies mapping and transformation, then loads through Azure Data Factory into a Dynamics test environment for validation, as many times as the cleansing needs.
Data work alongside delivery
Documented customer challenges on this site cover master-data control, integration and reporting work. They are related, but we would rather point you at them for what they actually show.
See customer storiesSelected customer examples. More customer stories and delivery evidence are being added.
Timing
Migration is not the last phase of the project.
Data exposes process differences, missing ownership, legacy customisation, duplicate structures and reporting dependencies. All of those are cheaper to find during design than during UAT.
Discover
Data profiling starts
Design
Mapping agreed
Build
Migration developed
Test
Migration rehearsals repeated
UAT
Business validates
Cutover
Final controlled migration
CDM: move and prepare data for change
Migration moves the required business position into the new platform, once, and then the job is finished.
IDM: keep systems connected afterwards
Integration keeps information moving between systems that both remain in use, every day, with someone responsible when it fails.
Explore Integration & Connected SystemsIf a migration is already under way and nobody can say which extract built the current environment, the problem is control rather than data. Explore Recovery.
Check your migration position
Six questions before migration scope is agreed.
A routing aid rather than a quote. Your answers come with you into the conversation, so nothing has to be repeated.
Question 1 of 6
0 of 6 answered
Common questions
Questions leaders ask about data migration.
Start with the data
What would you be worried about migrating tomorrow?
It might be customer records, stock, opening balances, years of history or a legacy field nobody can explain any more. Start there and we can work out what needs to move, what needs cleaning and how it will be proven.
