2 replies to this topic

[Roger]

For the last 2 weeks, I have been trying to find a good tool to move a massive amount of PDFs (180K files, 40GB+) from one server (GoDaddy shared hosting w/ SSH) to another (Linode). For some reason, the GoDaddy account doesn't allow me to use RSYNC for this. Well, actually, it would allow me to start the RSYNC, but it would then dismount the drive in the middle of the run. I tried multiple ways but could not get it to work.

I then tried to upload the files from my home machine (via DSL) to the Linode. As you can see from http://forum.codecal...html#post271070, that didn't work out. I was only able to move 400 files (out of 180k) before giving up. I really miss my Verizon FiOS speed!!

Then, this past weekend, I decided to try again. After doing some more research, I found this link, which has a well written script on how to do a recursive FTP. It's not prefect, but so far, it's moved 22GB without any problems. I am seeing some errors and restarting is a bit cumbersome, but this is still a pretty neat tool.

Recursive FTP -- here at last. - The UNIX and Linux Forums

[DEViANT]

Great post. I've been using scp -r to do most of my copying work but wanted to look into RSYNC for some projects. Think I'll give this recursive FTP script a look before doing so then.

[Roger]

Just a note of caution, the script did break ~30GB into the transfer. I haven't figured out why. It might be that the transfer has too many files (~180K files).
Good luck!