Ivan Kozik
|
0dc440ffd8
|
Tweak README
|
2015-07-28 11:34:58 +00:00 |
|
Ivan Kozik
|
975f328c95
|
Document gs-dump-urls
|
2015-07-28 11:33:51 +00:00 |
|
Ivan Kozik
|
6bbe9fb3bb
|
Fix formatting
|
2015-07-28 11:27:50 +00:00 |
|
Ivan Kozik
|
e506d6a103
|
Add gs-dump-urls, a utility to dump URLs from a wpull.db file
|
2015-07-28 11:26:10 +00:00 |
|
Ivan Kozik
|
991718b2e2
|
hooks: better ws:// connect messages, slow down reconnects exponentially
|
2015-07-27 14:01:22 +00:00 |
|
Ivan Kozik
|
36f24b03b3
|
hooks: print which ws:// server it can't connect to
|
2015-07-27 13:45:04 +00:00 |
|
Ivan Kozik
|
dbe1deb9f0
|
Clarify ignore sets
|
2015-07-27 13:28:49 +00:00 |
|
Ivan Kozik
|
41f7683d98
|
+1 is OK
|
2015-07-27 08:52:48 +00:00 |
|
Ivan Kozik
|
015df2a0df
|
Link yipdw
|
2015-07-27 08:06:33 +00:00 |
|
Ivan Kozik
|
a89ef4b22b
|
README: add Thanks and P.S.
|
2015-07-27 07:59:57 +00:00 |
|
Ivan Kozik
|
3b5f8b4be3
|
Clarify --concurrency
|
2015-07-27 07:38:06 +00:00 |
|
Ivan Kozik
|
d34c1c5f34
|
Fix formatting
|
2015-07-27 07:31:43 +00:00 |
|
Ivan Kozik
|
493971e2a9
|
Merge branch 'grab-site-py3-launcher'
|
2015-07-27 07:27:28 +00:00 |
|
Ivan Kozik
|
472edf5ebc
|
Put all temporary files in DIR/temp; don't let ctrl-c exit grab-site before wpull
|
2015-07-27 07:26:54 +00:00 |
|
Ivan Kozik
|
99dfbe2756
|
Add --version
|
2015-07-27 07:07:18 +00:00 |
|
Ivan Kozik
|
4c9a935bec
|
Make --igsets actually work
|
2015-07-27 06:58:17 +00:00 |
|
Ivan Kozik
|
b7c2f1d1bd
|
Add --sitemaps/--no-sitemaps
|
2015-07-27 06:55:20 +00:00 |
|
Ivan Kozik
|
2e7d928614
|
Update README
|
2015-07-27 06:50:48 +00:00 |
|
Ivan Kozik
|
84b183ec84
|
Write proper --help text and use aliased inputs too
|
2015-07-27 06:44:55 +00:00 |
|
Ivan Kozik
|
637929ab76
|
First take on converting grab-site to a Python program
|
2015-07-27 06:32:08 +00:00 |
|
Ivan Kozik
|
915ed0eeae
|
Use cchardet for faster encoding detection (imported by wpull/thirdparty/dammit.py)
|
2015-07-21 03:25:56 +00:00 |
|
Ivan Kozik
|
8d2acd669a
|
README: minor tweaks
|
2015-07-20 09:53:13 +00:00 |
|
Ivan Kozik
|
a7f2ee7684
|
Document webarchiveplayer for viewing your WARCs
|
2015-07-20 09:47:22 +00:00 |
|
Ivan Kozik
|
5e85e00201
|
README: document --concurrency=
|
2015-07-20 09:30:51 +00:00 |
|
Ivan Kozik
|
c35b388677
|
Allow archiving archive.org content despite it being in the global ignore set
|
2015-07-20 09:02:19 +00:00 |
|
Ivan Kozik
|
08933f60e2
|
Clarify ?host= dashboard option
|
2015-07-20 08:50:47 +00:00 |
|
Ivan Kozik
|
35d6d780bd
|
Bump version
|
2015-07-20 08:37:46 +00:00 |
|
Ivan Kozik
|
3f78e5f4bf
|
README: use an <h3>
|
2015-07-20 08:30:57 +00:00 |
|
Ivan Kozik
|
58b560257a
|
README: improve docs for options
|
2015-07-20 08:29:37 +00:00 |
|
Ivan Kozik
|
9af02f122b
|
Unbreak README
|
2015-07-20 08:25:33 +00:00 |
|
Ivan Kozik
|
1fce3af4a0
|
Add --1 option for turning off recursion; document options
|
2015-07-20 08:23:35 +00:00 |
|
Ivan Kozik
|
e83375382d
|
README: there are control files in DIR too
|
2015-07-20 08:04:14 +00:00 |
|
Ivan Kozik
|
210c3d03b5
|
README: include suggestions from @ethus3h (thanks!) and wrap long lines
|
2015-07-20 07:50:49 +00:00 |
|
Ivan Kozik
|
c7a272d7ba
|
Document how to fix your PATH for grab-site
|
2015-07-20 07:25:06 +00:00 |
|
Ivan Kozik
|
0e38441234
|
Add OS X support
|
2015-07-20 06:35:32 +00:00 |
|
Ivan Kozik
|
cd893cb1e3
|
Keeping your crawling problems in perspective
Spanish scriptorium? (Madrid, Biblioteca de San Lorenzo de El Escorial, 14th century).
Credit: https://medievalfragments.wordpress.com/2013/11/05/where-are-the-scriptoria/
|
2015-07-20 03:43:33 +00:00 |
|
Ivan Kozik
|
6c6c3197e7
|
Accept more exit codes from wpull as clean exit
|
2015-07-19 22:11:01 +00:00 |
|
Ivan Kozik
|
a95ee28c8d
|
Tell user where the output files are
|
2015-07-19 20:47:39 +00:00 |
|
Ivan Kozik
|
a5cc1d84c6
|
Bump version
|
2015-07-19 20:44:11 +00:00 |
|
Ivan Kozik
|
7566de05e3
|
Mark finished jobs as finished on dashboard
|
2015-07-19 20:42:29 +00:00 |
|
Ivan Kozik
|
8e2e1c5f58
|
Camelcase
|
2015-07-19 20:20:56 +00:00 |
|
Ivan Kozik
|
3ffed7dfbb
|
Tell people to use GitHub issues
|
2015-07-19 20:15:23 +00:00 |
|
Ivan Kozik
|
a66b970bfb
|
Enable faulthandler
|
2015-07-18 21:32:27 +00:00 |
|
Ivan Kozik
|
227052371e
|
Allow only grabbers to announce download/stdout/stderr/ignore
|
2015-07-18 13:35:11 +00:00 |
|
Ivan Kozik
|
fe659e21a6
|
Don't allow setting mode more than once
|
2015-07-18 13:32:26 +00:00 |
|
Ivan Kozik
|
6a866ad530
|
Don't assume WebSocket clients are dashboards by default; announce user agents
|
2015-07-18 13:29:19 +00:00 |
|
Ivan Kozik
|
55e3507122
|
Tweak README
|
2015-07-18 12:09:51 +00:00 |
|
Ivan Kozik
|
9f872f4fae
|
Recommend starting gs-server first
|
2015-07-18 12:06:00 +00:00 |
|
Ivan Kozik
|
877b170fde
|
Move some code around
|
2015-07-18 11:57:34 +00:00 |
|
Ivan Kozik
|
318fb3c03d
|
Set --max-redirect 8 like ArchiveBot
|
2015-07-18 11:52:31 +00:00 |
|