Why this matters
A GPU cloud is someone else's computer. When you send a training set to it, the data leaves your network and your normal access rules. Privacy covers the network path, the identity that reads the data, the keys that protect it at rest, and the copies left after the job. How to connect your data to a GPU cloud privately is about that control, not about picking a GPU model.
The quick path is a public bucket and a public GPU node. An engineer uploads a zip file, pastes a long lived access key into a startup script, and starts training. That works until the key leaks, a bucket policy is widened for a test, or a dataset sits on a host disk for days. A private endpoint and short lived credentials remove most of that exposure, but only if the job cannot fall back to the public path.
Expect trade-offs. A private path needs DNS and route changes, and it can slow the first load. Streaming data adds latency. Copying a snapshot to local NVMe is faster but creates another copy to protect and erase. For customer records or model weights, the private path is the baseline. For a small public dataset, it may be overkill.
Before you start
Map the data first. List each dataset, its owner, its size, where it lives, and what the job needs. Decide whether the job needs the full set or a sample, and whether fields can be dropped, hashed, or tokenized before upload. Write a retention rule: how long the provider may keep data, where copies may live, and who can approve an exception.
Inventory the network. Record VPC or VNet IDs, CIDR ranges, private DNS zones, egress rules, and any on premises links. Check that the GPU provider offers a private endpoint in the GPU region. The names differ: AWS PrivateLink, Azure Private Link, and Google Private Service Connect. Some GPU providers run their own private network and peer with your cloud. Ask for the endpoint service name, route changes, and DNS names.
Set up identity and keys. Use workload identity or OIDC federation so the job gets a short lived token instead of a static key. Confirm the provider can use your customer managed key for object storage, and ask whether local disks use provider managed keys. If that is not enough, encrypt files before upload and keep the key outside the provider.
Agree on logging and a pass test. Send object access logs, key use logs, and flow logs to an account you control. Before the first real run, write down four checks: no public IP on the node, no public bucket, one readable prefix, and no dataset left on local disk after teardown.
How to do it
Run the work in order. Reduce and stage the data, open a private path, then give the job a narrow read only identity. Keep outputs in a separate bucket or prefix with its own policy. The source bucket stays read only.
Treat the private endpoint as the only route. Disable public access after the endpoint works. If both paths stay open, a DNS failure will quietly send traffic over the internet. Test name resolution from inside the GPU subnet, not from your laptop. Plan cleanup for caches, checkpoints, container layers, and snapshots before the job starts.
- Reduce the dataset before it leaves your network. Take a sample, drop unused columns, and tokenize identifiers the model does not need. Package immutable shards with a manifest and checksums. Record the source and transform version so a reviewer can trace what moved.
- Create a private, encrypted bucket or container. Block public access at the account and bucket level, then turn on default encryption with your key. Scope the bucket policy to one job role and one prefix. Set a lifecycle rule to expire old copies.
- Open a private endpoint in the GPU region. Create the endpoint in the GPU region and attach it to the GPU VPC or VNet. Turn off public access on the storage service. Point the private DNS zone at the endpoint and allow only the GPU subnet.
- Use workload identity, not stored keys. Federate the GPU cluster with your identity provider and create a role that trusts one cluster and namespace. Grant read only access to the exact prefix. Remove static keys from images and scripts.
- Mount or stream the data read only. Use a CSI driver, FUSE mount, or object client with read only credentials. Stream small shards during training. Copy large epochs to encrypted local NVMe and delete the volume at job end.
- Keep secrets out of images and logs. Store tokens in a secret manager and inject them at runtime. Avoid build arguments and command line flags that end up in logs. Scan the image registry for secrets.
- Restrict egress and metadata access. Allow outbound traffic only to the private endpoint, package mirror, and log sink. Block the cloud metadata service from containers or require IMDSv2 with a hop limit of one. Review denied flows.
- Erase copies and revoke access after the run. Delete local copies, caches, and temporary checkpoints. Confirm the snapshot and volume deletion policy. Revoke the job token and export access and key use logs before teardown.
How to check it worked
Start with name resolution. From the GPU node, run nslookup or dig on the storage hostname. It must return a private address in the endpoint subnet, not a public IP. Fetch one object with the job role. Remove the private route in a test and confirm the request times out. A public success means you still have a fallback.
Check identity and policy. The object access log should show the workload role, the source endpoint, and the exact prefix, with no anonymous reads. The key use log should show only the expected key and job window. A read outside the prefix and a write inside it should both fail.
Hunt for leftovers. List local disks, container layers, cache directories, temp folders, and the image registry. Check the provider console for snapshots, volumes, and job artifacts. Revoke the token and repeat the read. It should fail.
Review usage. Storage and egress should show the private endpoint, not the internet gateway. A public route means DNS or policy changed.
Mistakes to avoid
Leaving the public path open as a backup is the most common failure. It wins during an outage. Remove public access on the storage service, then fix DNS or routing when the private path breaks. A long lived key in a startup script is the next leak. Use workload identity and short sessions.
A private endpoint with public DNS is another trap. The endpoint exists, but the hostname still resolves to a public address, so traffic leaves through the internet gateway. Fix the private DNS zone and test from the GPU subnet. A bucket policy that trusts the whole VPC instead of one role and prefix is too broad.
Do not treat a VPN as the whole answer. It protects traffic between networks, but the provider can still read data at rest, and a stolen token can still fetch objects. Pair it with object level authorization, encryption, and audit logs. Ask how provider support access is approved and logged, because that is another path to the data.
If you are choosing between a private endpoint, a provider private network, and a direct interconnect, the Expert architecture session at $450 one-time is one option among others. It gives you a call with a senior architect and a written summary, but a careful internal review can also get you there. Test the failure case before the first real dataset moves.
Frequently asked questions
Can I use a public GPU cloud and still keep training data private?
Yes, if you control the data path and the identity. Put the data in private object storage, reach it through a private endpoint, grant a short lived read only role, and keep local copies temporary. The GPU provider still runs the job, so encrypt sensitive data before upload if you do not trust provider disk encryption.
Do I need a VPN to connect data to a GPU cloud?
Not always. A private endpoint or private interconnect can carry storage traffic without a site to site VPN. A VPN is still useful for admin access or for networks that have no private endpoint support. Use it with object level controls, not as a replacement for them.
Where should training data live for GPU jobs?
Keep the source of truth in your own storage account or bucket, in a region close to the GPUs. Stage a reduced copy into a private bucket that the job can read, or stream shards during training. Delete local copies and checkpoints when the job ends.