rsync

Today I needed to move a ton of static media files over to a new server (we’re switching hosts at work.) In this case we previously had some of the large media files on a CDN that we’re moving away from and these files I now need to host on the same server as the rest of the stuff. Only problem is there is over 3,000 files and many of them large videos and images.

To make matters worse I had no way of connecting to the CDN at all. All I had was a list of file URL’s. I simply put that list into a text document and used the amazing wget with the -f flag to point it to the files. It took over a day but it downloaded all of the files I needed!

So, I could just FTP them up to the new server, right? Yes, I suppose I could do that but it’s going to take many days. Plus, did I mention that some of these files are already on the server, so it’s a waste of time and resources to transfer them again.

Enter rsync! Which does exactly what you think. It syncs files. In this case I am able to do a single command and copy every file up to the server, ignoring the ones that are already there with a simple command like this:

rsync -rzvt --ignore-existing ./ [email protected]:/var/www/vhosts/adomain.com/httpdocs/media/

In this case I’m transferring from the directory I’m in (./), to the server directory where I want to place these files.

Love, love rsync. It’s about a thousand times faster than FTP. You should use it.