Clone a website?

[soja]

I have critical data on a semi-unreliable dedi host in USA. They just went down after 100d of uptime and the downtime was near all day. Turned out the Hotswap HDD needed to be reseated. This is the second time it happened(100d between). This server pushes 500+GB/day, needless to say I had a lot of damage control to do.

The data heavy part does not need to be mirrored, the fastdl part of our servers needs to be cloned. That only pushes 70-100GB/day.

I have full ftp/http access, and I want to be able to mirror it to another web host(maybe paid nfo web hosting) or a small vds. It wont push much data, I just need a backup in case this happens again.

Does anyone know how I would go about mirroring this without overwriting files(only transferring what changed)?

Thanks

[soja]

I found this article about using wget for my purpose, but I don't have a linux terminal to check the man page for it right now, I assume it has an option to not overwrite files.

http://fosswire.com/post/2008/04/create ... with-wget/

John, does the 60 sec execution time limit apply to wget on webhosting? Thanks

[Edge100x]

The execution limit would apply, yes.

If your other host is running Linux, then "rsync" would be the ideal option for you, as it only copies changed files and can be configured to use compression, potentially speeding up the transfer further. For example, you could run a command like this from the destination (your shared webhosting account with us):

Code: Select all

rsync -avHz other-server:/path/to/folder/ local-server:/path/to/folder/

To allow you to automate this with a cronjob and retain a good level of security, it would be a good idea to create a local user on the source machine with the same username as your webhosting account here, and to give that user only permissions to the files that you want copied. Then, use SSH keys to allow a login from the the hosted website machine without a password: http://www.linuxproblem.org/art_9.html

Note that our TOS does not allow using our webhosting just for backups, so you'll need to make sure these files are publicly accessible.

[soja]

The remote machine is windows

The account here would be a publicly accessible fastdl set identical to the other host.

Gamebanana uses rsync with ftp to maintain their public mirror system(anyone can add a mirror with FTP details), however it just overwrites files if there are duplicates.

[Edge100x]

Syncing from a Windows host will not be as clean. You could install an rsync server on it, or you could use "wget" through FTP, as you suggested. It can be configured to use a recursive archive mode that attempts to only download changed files.

[soja]

Do the web servers have a rsync server? Is there a way I can set one up within my hosting limits?

Thanks

[Edge100x]

On our end, we don't have an rsync server, and you wouldn't be able to set one up.

[IcEWoLF]

I'd suggest checking out the backup solutions section @ webhostingtalk.com

Your best bet.

http://www.webhostingtalk.com/forumdisplay.php?f=169

[soja]

Thank you icewolf, I will check it out.