Moving terabytes of data between S3 buckets sucks. It’s slow, it breaks halfway through and then you’re stuck wondering which files were actually migrated. I’ve been there — running a single-threaded script for days, only to have it fail at 87 percent completion. Then starting over because who has time to figure out which files transferred successfully? There’s got to be a better way.

Turns out, there is! It involves SkyPilot and a powerful tool called s5cmd. But before we dive in, let’s be clear: this is a cool hack for certain use cases more than it is a recommended production solution for data migration scenarios. In particular, there are limitations if there are extremely large files or datasets with millions of small files.

The S3 migration problem we don’t talk aboutThe S3 migration problem we don’t talk about

Everyone’s moving data between clouds these days, but we don’t often talk about how painful it is. The standard approaches all have issues:

1. AWS CLI: Kind of slow (try moving 50 TB with aws s3 sync and watch your hair turn gray) and, crucially, it doesn’t support cross-account transfers by using different profiles.

2. Custom scripts: You’ll spend more time debugging edge cases than transferring data (especially for cross-account scenarios).

3. Managed services (e.g., DataSync): Often expensive and/or vendor-locked.

What if we could distribute the work across multiple machines? That’s where SkyPilot comes in.

SkyPilot + Nebius: New BFFsSkyPilot + Nebius: New BFFs

SkyPilot recently integrated with Nebius AI Cloud, giving us an easy way to spin up distributed compute resources. While primarily designed for AI workloads, it’s perfect for our data transfer needs.

The best part? While we’re focusing on Nebius as the target, this approach works with literally any S3-compatible storage. No vendor lock-in here!

A small note: there’s an awesome open-source project called Skyplane that was designed specifically for object storage transfers, but it’s been gathering digital dust for the last couple of years. Our approach is built on tools that are actively maintained.

The secret sauce: s5cmd and RAM diskThe secret sauce: s5cmd and RAM disk

One of the biggest advantages of our approach is the ability to handle cross-account transfers easily.

Here’s the hack that makes this all work — a combination of:

• SkyPilot for orchestration: Manages cluster provisioning and distribution.

• s5cmd for transfers: A blazing fast S3 client.

• RAM disk for temporary storage: Eliminates disk I/O bottlenecks.

• GNU parallel: Efficiently manages concurrent transfers.

This approach gives us:

• Distributed workload across multiple nodes

• Parallel processing within each node

• Memory-speed temporary storage

• Error handling and post-transfer verification

How it actually works (the nerdy details)How it actually works (the nerdy details)

Here’s the play-by-play of what happens:

1. SkyPilot provisions your migration cluster on Nebius.

2. Each node mounts a RAM disk for temporary storage.

3. The head node lists all objects in the source bucket.

4. That list gets evenly split (by count, not by size) into chunks among all worker nodes.

5. Each node processes its chunk in parallel.

6. Everything gets verified, with detailed reporting at the end.

Let’s look at some of the key code pieces that make this work:

# Mount RAM disk for temporary storage
sudo mount -t tmpfs -o size=8g tmpfs /mnt/ramdisk
mkdir -p /mnt/ramdisk/s3migration/temp

This is crucial — by using RAM instead of disk for temporary storage, we eliminate the I/O bottleneck that often plagues transfer operations.

The head node distributes work by splitting object lists:

# Split files by node using modulo on line number
for i in \
$(seq 0 \
$((NUM_NODES-1))); do
  awk -v node= \
  $i -v nodes= \
  $NUM_NODES 'NR % nodes == node' 
  $TEMP_DIR/filtered_objects.txt > 
  $TEMP_DIR/node_ \
  ${i}_objects.txt
  # ...
done

Then, each worker processes its assignments with high concurrency:

cat $TEMP_DIR/my_objects.txt | parallel -j \
$NUM_CONCURRENT process_object

The verification step ensures nothing was lost:

Launch your transfer jobLaunch your transfer job

Now, you’re ready to launch:

export SOURCE_AWS_PROFILE=default
export SOURCE_ENDPOINT_URL=https://s3.us-east-1.amazonaws.com # change to your region
export SOURCE_BUCKET=s3://source-bucket

export TARGET_AWS_PROFILE=nebius
export TARGET_ENDPOINT_URL=https://storage.eu-north1.nebius.cloud:443 # change to your region
export TARGET_BUCKET=s3://target-bucket

sky launch -c migration s3_migration.yaml \
  --env SOURCE_AWS_PROFILE \
  --env SOURCE_ENDPOINT_URL \
  --env SOURCE_BUCKET \
  --env TARGET_AWS_PROFILE \
  --env TARGET_ENDPOINT_URL \
  --env TARGET_BUCKET

This will:

1. Provision the specified resources on Nebius (by default, eight nodes with 16 CPUs each).

2. Set up the environment with all dependencies.

3. Mount your AWS credentials.

4. Start the distributed transfer process with 16 concurrent transfers per node.

Performance tuningPerformance tuning

You can adjust several parameters to optimize performance:

1. Number of nodes: Change num_nodes: 8 in the YAML file, to scale out more.

2. Concurrency: Modify NUM_CONCURRENT: 16, to control parallel transfers per node.

3. RAM disk size: Adjust the size=8g in the mount command for larger files.

4. CPUs per node: Change cpus: 16 in the Resource section.

For extremely large transfers (millions of files), consider changing the post-transfer verification to use sampling instead of checking every file, by modifying the script.

Benchmark: ImageNet-scale transfer performanceBenchmark: ImageNet-scale transfer performance

To evaluate data transfer performance across clouds and regions, we recreated the 70 GB ImageNet-style dataset that was used in the original Skyplane benchmark. This synthetic dataset was generated by using the following command:

for i in $(seq 1 1152); do 
  dd if=/dev/urandom of="dummy_file_$i.dat" bs=1M \
  count= $((70*1024/1152)) 
done

We then ran several transfer scenarios using S3-compatible endpoints and compared the timings with Skyplane’s published benchmarks.

Test scenarios and resultsTest scenarios and results

ObservationsObservations

• Nebius intra-region performance is comparable to Skyplane’s fastest results, clocking in at just 26 seconds.

• Cross-cloud transfer from AWS to Nebius (across continents) completed in 2 minutes 26 seconds, significantly outperforming AWS DataSync, which took up to 7 minutes for similar long-haul transfers.

• Skyplane remains the top performer for optimized, cross-cloud transfers, but unfortunately it only supports AWS, GCP, Azure and Cloudflare. Since the project is stale, I doubt there’ll be support for other S3-compatible storage solutions anytime soon.

Comparison to other optionsComparison to other options