[sdiy] archiving info

[Andre Majorel]

On 2007-06-29 15:44 -0500, Scott wrote:

> I'm sort of embarrassed for not knowing the answer to this
> already, but is there some sort of easy web archiving software
> out there?  Something where you can tell it to archive someone's
> entire website?  There are a lot of good sites people have put
> up w/ schematics, instructions, etc, and it would be nice if
> there were a tool that could get the whole thing for you.  

Some variation on

  wget -l 10 --limit-rate 10k -N -np -r -w 1 $url

If you mirror the same site more than once, always do it from the
same directory on your disk or you will download everything again
(as opposed to just the bits that have changed since last time).
First read the man page. Then test it on a small subtree of the
site. When in doubt, always ask for advice.

Sorry for the sermon but mass downloading consumes bandwidth (and
CPU if the web site is dynamic) which can be scarce resources.

-- 
André Majorel <URL:http://www.teaser.fr/~amajorel/>
Do not use this account for regular correspondence.
See the URL above for contact information.