Ivan Kozik
|
1d52a28fac
|
Increase size of compiled regexp cache; remove unused code
|
2015-08-12 07:52:24 +00:00 |
|
Ivan Kozik
|
26c7ea84d8
|
Implement --wpull-args for passing additional arguments to wpull
|
2015-08-12 06:39:49 +00:00 |
|
Ivan Kozik
|
1674751b1c
|
Don't crash if DIR/concurrency is set to 0
|
2015-08-12 05:57:56 +00:00 |
|
Ivan Kozik
|
28f5652404
|
Bump version
|
2015-08-12 05:29:44 +00:00 |
|
Ivan Kozik
|
bf080c7cb4
|
Implement --max-content-length=N for skipping large responses
|
2015-08-10 13:12:34 +00:00 |
|
Ivan Kozik
|
e304c60586
|
Describe why various ignores are in the 'global' ignore set; add support for comments in ignore sets
|
2015-08-10 11:41:16 +00:00 |
|
Ivan Kozik
|
aa9b877843
|
Don't crash with "error: unrecognized arguments" if cwd contains space
Closes #32.
|
2015-08-02 03:51:37 +00:00 |
|
Ivan Kozik
|
9f071a706d
|
setup.py: specify minimum version for all dependencies
Specifically, this solves a problem where trollius is too old to have
ensure_future.
|
2015-08-02 01:47:03 +00:00 |
|
Ivan Kozik
|
e55fa13004
|
Make wpull write .cdx file (its impl does one .cdx covering all WARC files)
|
2015-07-31 23:55:27 +00:00 |
|
Ivan Kozik
|
19f6971261
|
dashboard: don't handle ctrl-f, alt-f, and other ctrl/alt- key combinations
|
2015-07-29 23:04:20 +00:00 |
|
Ivan Kozik
|
d72e4094d1
|
Bump version
|
2015-07-29 18:38:31 +00:00 |
|
Ivan Kozik
|
b806316cb1
|
Use built-in ignore sets; don't crash if invalid ignore set is specified
|
2015-07-29 08:36:36 +00:00 |
|
Ivan Kozik
|
85f7be1936
|
meta referrer: use content="no-referrer" instead of the obsolete content="never"
|
2015-07-29 05:36:46 +00:00 |
|
Ivan Kozik
|
e6f830764e
|
Allow changing concurrency using DIR/concurrency file
|
2015-07-28 14:21:28 +00:00 |
|
Ivan Kozik
|
47c9a20ba7
|
Bump version
|
2015-07-28 14:01:42 +00:00 |
|
Ivan Kozik
|
3c28b53620
|
Allow changing delay (in milliseconds) using DIR/delay file
|
2015-07-28 13:44:51 +00:00 |
|
Ivan Kozik
|
cae516eb5d
|
Cache these control files for 3 seconds to reduce stat calls: ignores, igsets, igoff, stop
|
2015-07-28 13:23:00 +00:00 |
|
Ivan Kozik
|
4eb2805df0
|
Format DUPE/OF messages more nicely in terminal
|
2015-07-28 12:33:59 +00:00 |
|
Ivan Kozik
|
37d1f2e473
|
directory name gen: don't try and fail to create directory with > 255 chars when given a long URL
|
2015-07-28 12:16:32 +00:00 |
|
Ivan Kozik
|
0f1bdfd738
|
Don't spawn wpull in a subprocess, just import it and call its main()
|
2015-07-28 11:53:47 +00:00 |
|
Ivan Kozik
|
e506d6a103
|
Add gs-dump-urls, a utility to dump URLs from a wpull.db file
|
2015-07-28 11:26:10 +00:00 |
|
Ivan Kozik
|
991718b2e2
|
hooks: better ws:// connect messages, slow down reconnects exponentially
|
2015-07-27 14:01:22 +00:00 |
|
Ivan Kozik
|
36f24b03b3
|
hooks: print which ws:// server it can't connect to
|
2015-07-27 13:45:04 +00:00 |
|
Ivan Kozik
|
84b183ec84
|
Write proper --help text and use aliased inputs too
|
2015-07-27 06:44:55 +00:00 |
|
Ivan Kozik
|
915ed0eeae
|
Use cchardet for faster encoding detection (imported by wpull/thirdparty/dammit.py)
|
2015-07-21 03:25:56 +00:00 |
|
Ivan Kozik
|
c35b388677
|
Allow archiving archive.org content despite it being in the global ignore set
|
2015-07-20 09:02:19 +00:00 |
|
Ivan Kozik
|
35d6d780bd
|
Bump version
|
2015-07-20 08:37:46 +00:00 |
|
Ivan Kozik
|
1fce3af4a0
|
Add --1 option for turning off recursion; document options
|
2015-07-20 08:23:35 +00:00 |
|
Ivan Kozik
|
0e38441234
|
Add OS X support
|
2015-07-20 06:35:32 +00:00 |
|
Ivan Kozik
|
a5cc1d84c6
|
Bump version
|
2015-07-19 20:44:11 +00:00 |
|
Ivan Kozik
|
43d8a9594f
|
Move everything and make grab-site installable with pip3
|
2015-07-18 10:39:04 +00:00 |
|