Ivan Kozik
|
3cd416f477
|
Fix formatting
|
2015-07-18 05:51:25 +00:00 |
|
Ivan Kozik
|
19107cbe28
|
Fix formatting
|
2015-07-18 05:51:15 +00:00 |
|
Ivan Kozik
|
1f789d204c
|
Fix just in case wpull stops titlecasing headers
|
2015-07-18 05:50:07 +00:00 |
|
Ivan Kozik
|
0526f8c96e
|
Log URLs being fetched to real stdout
|
2015-07-18 05:48:33 +00:00 |
|
Ivan Kozik
|
758ec1301a
|
Camelcase
|
2015-07-18 05:40:35 +00:00 |
|
Ivan Kozik
|
952eb4c33f
|
print some messages only to real stdout
|
2015-07-18 05:38:59 +00:00 |
|
Ivan Kozik
|
787db7da55
|
Make stdout/stderr capture actually work
|
2015-07-18 05:35:57 +00:00 |
|
Ivan Kozik
|
f1100e7223
|
Try to send stdout/stderr to dashboard and fail at it
|
2015-07-18 05:24:54 +00:00 |
|
Ivan Kozik
|
93adc1ad48
|
Refactor job_data broadcasting
|
2015-07-18 04:34:24 +00:00 |
|
Ivan Kozik
|
937908ef52
|
Report bytes downloaded to dashboard
|
2015-07-18 04:31:54 +00:00 |
|
Ivan Kozik
|
dcbcb28852
|
Reported started_at to dashboard
|
2015-07-18 04:21:35 +00:00 |
|
Ivan Kozik
|
e804f7171e
|
Show job URLs on dashboard
|
2015-07-18 04:14:50 +00:00 |
|
Ivan Kozik
|
f155cbc4ed
|
Make the dashboard sort-of work
|
2015-07-18 03:49:24 +00:00 |
|
Ivan Kozik
|
53fd04a29e
|
Make reconnecting work
|
2015-07-18 03:17:27 +00:00 |
|
Ivan Kozik
|
18a192739b
|
Make WebSocket client/server sort of work; rename ignore_sets to igsets
|
2015-07-18 02:11:18 +00:00 |
|
Ivan Kozik
|
db21e530e2
|
Generate a grab id and put in the dir name; add some temporary print debugging
|
2015-07-18 01:06:56 +00:00 |
|
Ivan Kozik
|
5621863b09
|
Refactor the WebSocket client in hooks
|
2015-07-17 23:57:46 +00:00 |
|
Ivan Kozik
|
732aeb0d5f
|
Start up both an HTTP server and WebSocket server
|
2015-07-17 23:37:55 +00:00 |
|
Ivan Kozik
|
5229ddf5dc
|
Start work on websocket server for future dashboard integration
|
2015-07-17 22:42:25 +00:00 |
|
Ivan Kozik
|
0699689a14
|
Add igoff feature
|
2015-02-05 05:19:34 +00:00 |
|
Ivan Kozik
|
979b843458
|
Load changes from DIR/ignores and DIR/ignore_sets while the crawl is running
|
2015-02-05 04:59:28 +00:00 |
|
Ivan Kozik
|
5f7593fda2
|
Refactor
|
2015-02-05 04:39:52 +00:00 |
|
Ivan Kozik
|
eea440422d
|
Allow specifying --ignore-sets NAME1,NAME2,...
|
2015-02-05 04:24:05 +00:00 |
|
Ivan Kozik
|
a61ed949ca
|
Use global ignore set and also ignore Icecast sites like ArchiveBot
|
2015-02-05 04:03:19 +00:00 |
|