How to move terabytes of data to the cloud

Why this matters

At terabyte scale, a copy is a project with a clock. A dropped connection at hour nine is annoying. A manifest you cannot reconcile after a week is expensive. How to move terabytes of data to the cloud is about controlling that job: know the set, pick a route, copy in stages, prove the result.

The move competes with production traffic. A bulk upload can fill the office link and make every SaaS tool feel slow. Plan the window, cap the rate and schedule it outside business hours. On a long link, packet loss and distance matter more than raw bandwidth. Test a real sample before you promise a date.

Small files hurt. A million 10 KB files will crawl even on a fast link because each object carries request overhead. Pack them into archives if you can, but keep a manifest so you can restore individual files later. Decide early whether the cloud copy is a backup, an archive or the new primary. That choice changes encryption, versioning and how long you keep the source.

Before you start

Write down what you are moving: paths, object counts, total bytes, owners and the systems that still write to it. Capture file metadata you need, such as permissions, timestamps and extended attributes. Object storage does not keep all of that the same way a file server does, so agree on what can be dropped.

Measure the link in both directions and run a pilot with a representative folder. Include many small files and a few large ones. Check the transfer tool's concurrency, multipart part size and retry settings. For object storage, multipart upload is how large files move. Too small a part adds requests. Too large a part wastes memory and time on retry.

Pick the route before you buy hardware. A network transfer fits when the dataset is modest, the link is dedicated or the data changes daily. A physical appliance fits when the link would take weeks or the site has poor upload. For cloud to cloud, use the provider's transfer service or a tool that reads the source and writes the target directly. Do not route cloud to cloud traffic through your office.

Set the keys and access rules first. Use a dedicated transfer identity with write access to one target prefix or bucket, not an admin account. Turn on encryption in transit and at rest. Store the keys where the transfer job can reach them and where you can revoke them. Record the target names, regions and storage classes in the runbook.

How to do it

Treat the move as seed, sync, cutover. The seed copy carries the bulk. The syncs pick up changes. The cutover freezes writes, runs the last pass and flips readers. If the source is a live database, use its replication or dump and restore path, not a file copy. That is a different job with its own rules.

Start with a manifest. A manifest is a list of files or objects with size and checksum. You will use it to compare source and target, to restart after a failure and to prove the copy is complete. Generate it at the same time as the seed so any file changed during the seed is caught by the first sync.

Run the seed with logs and a retry queue. Do not babysit it. Let the tool retry failed parts, but cap retries so a broken file does not loop forever. Watch queue depth, throughput and error rate. If errors cluster around one folder, stop and fix permissions or naming before you copy more.

Then run syncs on a schedule. Each pass should be faster because it moves only changed data. Keep a change log or use the source's audit trail to see writes that land during a long sync. Before cutover, stop writes, run the final sync, compare the manifest and then allow writes to the target. Keep the source read only until the check passes.

1. Freeze the scope and build a manifest. List every path or bucket, the owner and the total bytes. Generate a checksum manifest before the first copy so you have a fixed reference.
2. Run a pilot on a real sample. Pick a folder with small files and a few large ones. Measure throughput, error rate and CPU use on the transfer host, then tune concurrency and part size.
3. Choose network or appliance. Use the network when the link can finish the job inside your window. Use a physical appliance when upload would take too long, and plan the chain of custody and encryption keys.
4. Create a least privilege transfer identity. Grant write access to the target prefix only. Use short lived credentials where the tool supports them and log every write.
5. Seed the bulk copy. Run the first full pass with logging and retries. Throttle it so business traffic still works, and leave the source untouched.
6. Sync changes in passes. Run repeated incremental copies. Track files that change during each pass and keep the sync window short enough to finish.
7. Freeze writes and do the final sync. Pause writers, run one last pass and compare counts, bytes and checksums. Do not open the target for writes until the comparison is clean.
8. Verify with a restore test. Read back a random sample and a full small folder from the target. Check file contents, not just object counts.
9. Cut over and keep the source read only. Point readers at the cloud copy, watch errors and keep the old system available for rollback until the agreed hold period ends.

How to check it worked

Checksums are the real test. Compare the manifest you built at the source with one built from the target. Use the same algorithm. If the tool stores checksums in object metadata, check that the values match after upload. For multipart uploads, the target checksum may be over parts, not the whole file, so confirm how the tool computes it.

Count files and bytes, but do not stop there. Counts catch missing folders, not silent corruption. Sample read back files of different sizes and types. Restore a full folder through the application that will use it, not just through the CLI. Open a few archives and check the manifest inside.

Check metadata and access. Look at permissions, ownership, timestamps and content type. Confirm versioning, lifecycle rules and retention settings will not expire the new copy. Check that logging is on and that the transfer identity is disabled after the job.

Run a delta check after the first day of real use. Compare a sample of files written to the target with what the application expects. If the app writes through a cache or a sync client, confirm those writes land in the cloud copy and not only on a local disk.

Mistakes to avoid

The common failure is treating the transfer as one big copy. It is a sequence with checks between steps. Another is copying small files without packing or parallelism, then wondering why the queue never drains. A third is leaving the old system writable after cutover, which creates two sources of truth.

Do not skip the restore test. A clean object listing proves very little. Do not let the transfer identity stay active after the job. Do not use sync with delete on the target until you are sure the source is complete; a missing source folder can wipe the target. For physical appliances, log every handoff and keep the keys separate from the device.

Watch the network and the bill. Throttle the job, cap concurrency and schedule it. Check request charges, early deletion fees and storage class minimums before you move archives. If you need an architecture and a phased plan before the copy starts, the Cloudverg Discovery and roadmap sprint is $9,500/month at a fixed scope, and the fee is credited to phase one if you start within 60 days. It is one option among others.

Frequently asked questions