Ivan Kozik
|
e6f830764e
|
Allow changing concurrency using DIR/concurrency file
|
2015-07-28 14:21:28 +00:00 |
|
Ivan Kozik
|
47c9a20ba7
|
Bump version
|
2015-07-28 14:01:42 +00:00 |
|
Ivan Kozik
|
7ac5b07a99
|
Add --delay option
|
2015-07-28 13:57:42 +00:00 |
|
Ivan Kozik
|
3c28b53620
|
Allow changing delay (in milliseconds) using DIR/delay file
|
2015-07-28 13:44:51 +00:00 |
|
Ivan Kozik
|
4f5fb8f108
|
Print IGNOR messages more nicely in the console
|
2015-07-28 13:26:09 +00:00 |
|
Ivan Kozik
|
cae516eb5d
|
Cache these control files for 3 seconds to reduce stat calls: ignores, igsets, igoff, stop
|
2015-07-28 13:23:00 +00:00 |
|
Ivan Kozik
|
4b174ee94f
|
Remove unused imports
|
2015-07-28 12:52:42 +00:00 |
|
Ivan Kozik
|
4c84312462
|
Undo my camelCase mistake
|
2015-07-28 12:51:59 +00:00 |
|
Ivan Kozik
|
4eb2805df0
|
Format DUPE/OF messages more nicely in terminal
|
2015-07-28 12:33:59 +00:00 |
|
Ivan Kozik
|
37d1f2e473
|
directory name gen: don't try and fail to create directory with > 255 chars when given a long URL
|
2015-07-28 12:16:32 +00:00 |
|
Ivan Kozik
|
a82e4017fe
|
directory name gen: whitelist instead of blacklist characters
|
2015-07-28 12:12:35 +00:00 |
|
Ivan Kozik
|
2418ea04e8
|
dashboard: don't include '!ig ID' in the context menu regexp helper, since these are designed to be pasted into a DIR/ignores file
|
2015-07-28 12:01:20 +00:00 |
|
Ivan Kozik
|
0f1bdfd738
|
Don't spawn wpull in a subprocess, just import it and call its main()
|
2015-07-28 11:53:47 +00:00 |
|
Ivan Kozik
|
6bbe9fb3bb
|
Fix formatting
|
2015-07-28 11:27:50 +00:00 |
|
Ivan Kozik
|
e506d6a103
|
Add gs-dump-urls, a utility to dump URLs from a wpull.db file
|
2015-07-28 11:26:10 +00:00 |
|
Ivan Kozik
|
991718b2e2
|
hooks: better ws:// connect messages, slow down reconnects exponentially
|
2015-07-27 14:01:22 +00:00 |
|
Ivan Kozik
|
36f24b03b3
|
hooks: print which ws:// server it can't connect to
|
2015-07-27 13:45:04 +00:00 |
|
Ivan Kozik
|
d34c1c5f34
|
Fix formatting
|
2015-07-27 07:31:43 +00:00 |
|
Ivan Kozik
|
472edf5ebc
|
Put all temporary files in DIR/temp; don't let ctrl-c exit grab-site before wpull
|
2015-07-27 07:26:54 +00:00 |
|
Ivan Kozik
|
99dfbe2756
|
Add --version
|
2015-07-27 07:07:18 +00:00 |
|
Ivan Kozik
|
4c9a935bec
|
Make --igsets actually work
|
2015-07-27 06:58:17 +00:00 |
|
Ivan Kozik
|
b7c2f1d1bd
|
Add --sitemaps/--no-sitemaps
|
2015-07-27 06:55:20 +00:00 |
|
Ivan Kozik
|
84b183ec84
|
Write proper --help text and use aliased inputs too
|
2015-07-27 06:44:55 +00:00 |
|
Ivan Kozik
|
637929ab76
|
First take on converting grab-site to a Python program
|
2015-07-27 06:32:08 +00:00 |
|
Ivan Kozik
|
915ed0eeae
|
Use cchardet for faster encoding detection (imported by wpull/thirdparty/dammit.py)
|
2015-07-21 03:25:56 +00:00 |
|
Ivan Kozik
|
c35b388677
|
Allow archiving archive.org content despite it being in the global ignore set
|
2015-07-20 09:02:19 +00:00 |
|
Ivan Kozik
|
35d6d780bd
|
Bump version
|
2015-07-20 08:37:46 +00:00 |
|
Ivan Kozik
|
1fce3af4a0
|
Add --1 option for turning off recursion; document options
|
2015-07-20 08:23:35 +00:00 |
|
Ivan Kozik
|
0e38441234
|
Add OS X support
|
2015-07-20 06:35:32 +00:00 |
|
Ivan Kozik
|
6c6c3197e7
|
Accept more exit codes from wpull as clean exit
|
2015-07-19 22:11:01 +00:00 |
|
Ivan Kozik
|
a95ee28c8d
|
Tell user where the output files are
|
2015-07-19 20:47:39 +00:00 |
|
Ivan Kozik
|
a5cc1d84c6
|
Bump version
|
2015-07-19 20:44:11 +00:00 |
|
Ivan Kozik
|
7566de05e3
|
Mark finished jobs as finished on dashboard
|
2015-07-19 20:42:29 +00:00 |
|
Ivan Kozik
|
8e2e1c5f58
|
Camelcase
|
2015-07-19 20:20:56 +00:00 |
|
Ivan Kozik
|
227052371e
|
Allow only grabbers to announce download/stdout/stderr/ignore
|
2015-07-18 13:35:11 +00:00 |
|
Ivan Kozik
|
fe659e21a6
|
Don't allow setting mode more than once
|
2015-07-18 13:32:26 +00:00 |
|
Ivan Kozik
|
6a866ad530
|
Don't assume WebSocket clients are dashboards by default; announce user agents
|
2015-07-18 13:29:19 +00:00 |
|
Ivan Kozik
|
63e8f1813b
|
Add --page-requisites-level= and --concurrency= options; use default concurrency of 2
|
2015-07-18 11:45:11 +00:00 |
|
Ivan Kozik
|
5da054a837
|
Report concurrency level
|
2015-07-18 11:18:50 +00:00 |
|
Ivan Kozik
|
d3715fe888
|
Dup -> Dupe
|
2015-07-18 11:00:33 +00:00 |
|
Ivan Kozik
|
2222aafa74
|
Spaces -> tabs
|
2015-07-18 11:00:08 +00:00 |
|
Ivan Kozik
|
e42e33d82f
|
My* -> Grabber*
|
2015-07-18 10:57:57 +00:00 |
|
Ivan Kozik
|
43d8a9594f
|
Move everything and make grab-site installable with pip3
|
2015-07-18 10:39:04 +00:00 |
|