Archiveteam/grab-site
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
840 Commits 1 Branch 0 Tags
Commit Graph

154 Commits

Author SHA1 Message Date
Ivan Kozik
0363f825d5 README: Document how to archive subreddits 2015-12-11 08:48:18 +00:00
Ivan Kozik
52895892d5 README: Tweak 2015-12-11 08:44:23 +00:00
Ivan Kozik
e82937a9f6 README: Tweak 2015-12-11 08:43:30 +00:00
Ivan Kozik
6d3f0f901d README: Fix headings 2015-12-11 08:40:30 +00:00
Ivan Kozik
140efe082a README: Write about how to archive specific websites 2015-12-11 08:39:36 +00:00
Ivan Kozik
b000c1a759 README: Tweak 2015-12-06 06:00:04 +00:00
Ivan Kozik
d80bde8c8c Add warning about warcprox 2015-11-30 06:31:21 +00:00
Ivan Kozik
f00ab84949 README: Tweak 2015-10-25 02:23:23 +00:00
Ivan Kozik
8dc43fe73c Clarify upgrade instructions 2015-10-25 02:21:40 +00:00
Ivan Kozik
6ac6bf69d4 Add Ubuntu 15.10 install instructions 2015-10-21 22:44:14 +00:00
Ivan Kozik
6da093a817 README: Update TOC 2015-10-20 10:50:41 +00:00
Ivan Kozik
3dfb9dc757 README: Remove instructions for grabbing a site that requires a cookie because the --header solution results in the cookie being sent to all domains instead of just the intended domain 2015-10-20 10:50:18 +00:00
Ivan Kozik
77248eb47c README: Be more specific about which Python 3 must be installed 2015-10-18 04:38:38 +00:00
Ivan Kozik
8101d4b04c README: OS X install also works on 10.11 2015-10-18 04:36:21 +00:00
Ivan Kozik
9f3fe3f4cb Add a table of contents 2015-10-12 08:24:26 +00:00
Ivan Kozik
74c385332a Document zless and how to grab a site that requires a cookie 2015-10-11 06:35:39 +00:00
Ivan Kozik
1957154321 Mention --no-skip-getaddrinfo 2015-10-07 08:13:19 +00:00
Ivan Kozik
ea483bc74b README: Tweak 2015-09-30 23:46:46 +00:00
Ivan Kozik
8260a0d7cf Add note about Python 3.5 2015-09-30 23:44:25 +00:00
Ivan Kozik
bfe92c9605 Document how to upgrade grab-site 2015-09-30 22:31:46 +00:00
Ivan Kozik
7a63a3dcd1 Add --no-dupespotter for turning off dupespotter which sometimes has false positives 2015-09-30 22:16:56 +00:00
Ivan Kozik
f1548521ec Write URLs skipped by --max-content-length= to DIR/skipped_max_content_length 2015-09-02 19:15:00 +00:00
Ivan Kozik
b782c23389 Add --no-video option to skip the download of videos 2015-08-21 08:28:27 +00:00
Arkiver2
6f6754f81e Add --warc-max-size=BYTES option for controlling WARC size 2015-08-21 07:47:12 +00:00
Ivan Kozik
49aa6d0dcf Remove mentions of blogs ignore set 2015-08-21 05:20:26 +00:00
Ivan Kozik
13d73ad59f README: tweak 2015-08-12 07:49:20 +00:00
Ivan Kozik
26c7ea84d8 Implement --wpull-args for passing additional arguments to wpull 2015-08-12 06:39:49 +00:00
Ivan Kozik
db212f6716 Reorder options in README 2015-08-12 05:39:16 +00:00
Ivan Kozik
668c03d5d2 Implement -i / --input-file, supporting both local input files and URLs 2015-08-12 05:24:09 +00:00
Ivan Kozik
37a6cb655c README: tweak 2015-08-10 13:40:52 +00:00
Ivan Kozik
b7743e780a Implement --ua= for setting the User-Agent 2015-08-10 13:38:00 +00:00
Ivan Kozik
ee4dbe162e Implement --igon / --igoff 2015-08-10 13:23:43 +00:00
Ivan Kozik
76ba117d34 Document DIR/max_content_length 2015-08-10 13:15:20 +00:00
Ivan Kozik
bf080c7cb4 Implement --max-content-length=N for skipping large responses 2015-08-10 13:12:34 +00:00
Ivan Kozik
e1bb1ec749 README: tweak 2015-07-31 22:47:58 +00:00
Ivan Kozik
ed869864d4 README: link to ArchiveBot 2015-07-31 03:52:42 +00:00
Ivan Kozik
6cd50f9688 README: tweak 2015-07-31 03:50:39 +00:00
Ivan Kozik
412ea7791f README: changes to ignores may take up to 3 seconds to apply 2015-07-30 23:36:12 +00:00
Ivan Kozik
a418beaff8 README: tweak for the non-ArchiveBot audience 2015-07-29 08:55:06 +00:00
Ivan Kozik
deb05d981d README: link to correct ignore sets 2015-07-29 08:45:25 +00:00
Ivan Kozik
e6f830764e Allow changing concurrency using DIR/concurrency file 2015-07-28 14:21:28 +00:00
Ivan Kozik
1198c88f2a Document --delay in README 2015-07-28 14:01:28 +00:00
Ivan Kozik
7cf8db39d3 Mention pipe to sort | less -S 2015-07-28 11:40:14 +00:00
Ivan Kozik
0dc440ffd8 Tweak README 2015-07-28 11:34:58 +00:00
Ivan Kozik
975f328c95 Document gs-dump-urls 2015-07-28 11:33:51 +00:00
Ivan Kozik
dbe1deb9f0 Clarify ignore sets 2015-07-27 13:28:49 +00:00
Ivan Kozik
41f7683d98 +1 is OK 2015-07-27 08:52:48 +00:00
Ivan Kozik
015df2a0df Link yipdw 2015-07-27 08:06:33 +00:00
Ivan Kozik
a89ef4b22b README: add Thanks and P.S. 2015-07-27 07:59:57 +00:00
Ivan Kozik
3b5f8b4be3 Clarify --concurrency 2015-07-27 07:38:06 +00:00