The problem
Moving AI training data between clouds is a data engineering job first and a network job second. The dataset is rarely one file. It is shards, tokenized records, label files, manifests, checkpoints, evaluation sets, and logs. Some objects are huge; many are small. Training jobs read them in random order, so a copy that works for a backup can still fail for training.
The hard part is keeping bytes, names, metadata, and access rules together. Object stores treat key names and metadata differently. A copy tool may drop custom metadata, change permissions, or turn multipart uploads into a different ETag. If your manifest points at the old keys, the new copy is not usable. If it points at new keys, the old jobs break.
Transfers also happen while data changes. New shards land, labels get fixed, and checkpoints update. You need a freeze window or a versioned snapshot. You need to know which copy is authoritative. Without that, you get a merge problem disguised as a transfer. Egress charges, request quotas, and per-object overhead make the naive copy slow and costly.
What a good solution looks like
A good solution starts with an inventory and a manifest you can query. Every object gets a source URI, size, checksum, content type, metadata hash, target URI, and status. The manifest is the contract. It lets you restart, verify, and explain gaps.
The transfer layer should be boring. Use a managed transfer service or a proven CLI such as rclone, azcopy, or gsutil. Run it in waves. Keep the source read-only. Use server-side encryption and customer-managed keys only if you have a key custody plan for both clouds. Otherwise use provider-managed keys and document it.
The cutover should be a switch in one place. That can be a catalog table, a config value, or a symlink in your training launcher. Do not edit job code across dozens of repos. Run a small training job against the new copy before you move the full workload. Keep the source in place until the new path passes checks and a full epoch.
How to build it
Start by exporting an inventory. Use S3 Inventory, Azure Blob inventory, or a storage inventory export from Google Cloud to list objects. Pull metadata and tags. Classify data by access pattern: hot shards for training, cold archives, checkpoints, and derived data you can rebuild. Move only what you need. Rebuildable data is often cheaper to regenerate than to transfer.
Pick the path. For steady streams up to a few terabytes, a managed online transfer service works. For very large cold sets, an appliance can be faster than a saturated link, but it adds shipping and handling time. Whatever you pick, test with a small sample. Measure throughput per stream, tune concurrency, and raise provider quotas before the full run. Set multipart size and retry limits. For many small files, batch them into tar or parquet shards first if your training loader supports it.
Set identity and keys before the first byte moves. Create a dedicated service account per direction. Grant list, get, put, and kms decrypt or encrypt only on the named buckets. Store keys in Key Management Service, Key Vault, or Cloud KMS. Do not put keys in scripts. Enable access logs and object versioning on the target during the migration.
Run in waves, verify every object, and record results. Compare size and SHA-256 where the provider exposes it. For multipart objects, compare the provider checksum, not the ETag. Use rclone check or the service reconciliation report. After the copy, run a read test from a compute node in the target cloud. Then switch the catalog pointer and keep the source for a rollback window.
Risks and how to manage them
The top risk is a copy that looks complete but is not usable. Counts match, but keys differ, metadata is gone, or a shard is truncated. Full checksum reconciliation catches most of it. A training smoke test catches the rest. Do not skip either.
Key loss is quiet and permanent. If the source uses customer-managed keys and you copy ciphertext or re-encrypt without the right grants, the target data can be unreadable. Test key access with a small object first. Keep the source keys alive until the target data is verified and the rollback window closes.
Source drift is another risk. A copy taken over days can mix old and new shards. Use object versioning, a snapshot, or a read-only freeze. Log the manifest version used by each training run. If labels change during the move, you need a data version, not just a file copy.
Watch network and quota limits. A full link can starve other services. Set bandwidth caps, run during low traffic, and use multiple streams with backoff. Watch request rates on small objects. Plan for egress charges with budgets and transfer reports. Delete source data only after a final reconciliation and owner sign-off.
How to measure success
Define acceptance before you start. Good measures are bytes reconciled, object count matched, checksum pass rate, transfer time against your window, retry rate, and failed training reads after cutover. Add cost variance against the budget you set, without pretending it is exact.
Track at the manifest level, not just the job level. A dashboard that shows almost complete can hide the one shard a training job needs. Report by prefix and by dataset version. Keep a list of exceptions with owner and next action.
After cutover, run a full epoch or a representative training run on the new cloud. Compare loss curves and evaluation results. If they diverge, stop and compare the data, not just the code. Keep the old path readable until the new path passes.
A migration is done when the source is decommissioned and the target is the only authoritative copy. That last step needs its own review. You can run it internally, hire a cloud architect, or use a service. Cloudverg offers an Expert architecture session at $450/month for a second opinion before a big cutover. Pick the option that fits the size of the move. Then close the project with a short record of what changed.
Frequently asked questions
How do I move a large AI dataset between clouds without losing metadata?
Use an inventory and manifest that record metadata hashes for every object. Copy with a tool that preserves custom metadata, then reconcile the target against the manifest. Test a sample with your real training loader before the full run.
Should I use a transfer appliance or the network for AI training data?
Use the network when the dataset changes often or you need a repeatable pipeline. Use an appliance for large cold sets when the link is the bottleneck and you can accept shipping time. Either way, verify checksums and keep a rollback copy.
How do I verify AI training data after a cloud migration?
Compare object counts, bytes, and checksums against your manifest. Handle multipart objects with the provider checksum, not the ETag. Then run a small training job and confirm it reads the expected shards and labels.