Archiveteam/grab-site
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
1,112 Commits 1 Branch 0 Tags
Commit Graph

38 Commits

Author SHA1 Message Date
Ivan Kozik
cb3068c0ca Use ludios_wpull 3.0.8 to fix https://github.com/ArchiveTeam/grab-site/issues/181 2021-05-10 00:55:45 +00:00
Ivan Kozik
d33a42171d setup.py: https://github.com/ludios/wpull -> https://github.com/ArchiveTeam/ludios_wpull 2019-03-01 06:18:11 +00:00
Katie Holly
bb020e741d
Reference git repo in install_requires 
Fixes #145
 2019-02-12 12:00:46 +01:00
Ivan Kozik
700ea18693 Use ludios/wpull 3.0.7 to fix https://github.com/ludios/grab-site/issues/138 2018-10-16 21:24:44 +00:00
Ivan Kozik
50a3db6e59 Use ludios/wpull 3.0.6 2018-10-16 16:05:06 +00:00
Ivan Kozik
e6b1709ef3 Use facebook/pyre2 because ludios/pyre2 has a buggy PR applied: 
https://github.com/facebook/pyre2/pull/11#issuecomment-428256331
 2018-10-09 16:20:09 +00:00
Ivan Kozik
e790ff8329 Use ludios/wpull 3.0.5 2018-10-09 14:40:43 +00:00
Ivan Kozik
caf0497b93 Use wpull 3.0.4 2018-10-09 14:03:07 +00:00
Ivan Kozik
7373140709 Use ludios/wpull 3.0.3 2018-10-07 22:26:59 +00:00
Ivan Kozik
21a1e44196 wpull_hooks: reimplement the dashboard websocket client 2018-10-07 21:31:22 +00:00
Ivan Kozik
e6d81e81c5 Require ludios/wpull 3.0.2 and remove the install_requires that are in wpull 2018-10-05 08:15:16 +00:00
Ivan Kozik
82c94683d4 Remove unused html5lib dependency 2018-10-05 07:54:13 +00:00
Ivan Kozik
4d8218db7f Use wpull 1.3.1 2018-10-05 07:54:13 +00:00
Ivan Kozik
b7dfb14dd8 Upgrade to and require Python 3.7.0 2018-10-05 07:53:58 +00:00
Ivan Kozik
7ec5accb22 Remove trollius 2018-10-04 14:58:03 +00:00
Ivan Kozik
b32da83a0f Install ludios/pyre2, to be used soon for processing ignores 2018-10-04 14:13:05 +00:00
Ivan Kozik
b6127f2077 Use ludios/wpull 1.2.5 2018-10-04 12:32:47 +00:00
Ivan Kozik
eaaf0ec06e Use ludios/wpull for html5-parser support 2018-10-04 11:13:35 +00:00
Ivan Kozik
5a05fa9761 Lock tornado version to 4.5.3 to avoid 5.0, which breaks with: 
File "[...]/lib/python3.4/site-packages/wpull/abstract/client.py", line 9, in <module>
    from wpull.connection import ConnectionPool
  File "[...]/lib/python3.4/site-packages/wpull/connection.py", line 11, in <module>
    from tornado.netutil import SSLCertificateError
ImportError: cannot import name 'SSLCertificateError'
 2018-03-06 06:29:07 +00:00
Ivan Kozik
63fdb9d5c6 Lock html5lib version to work around https://github.com/chfoo/wpull/issues/332 2016-07-15 05:38:38 +00:00
Ivan Kozik
76f8b2cf48 Lock wpull dependency to 1.2.3 for now 2016-06-12 08:47:36 +00:00
Ivan Kozik
aa366bbb27 Update grab-site URL in setup.py and dashboard 2016-05-27 13:53:35 +00:00
Ivan Kozik
f0bb696dc8 Actually install the favicon.ico 2016-05-22 11:01:53 +00:00
Ivan Kozik
3ee97f9fcd Bump dependencies since we only test with the latest versions 2016-02-25 01:08:01 +00:00
12As
990f24684a Remove aiohttp dependency from setup.py 2016-02-23 14:10:15 -06:00
Daniel Oaks
95012d1e0c gs-server: Use env instead of py3 directly, makes virtualenvs nicer 2016-02-16 17:24:20 +00:00
Ivan Kozik
7c1afbefe0 Use wpull 1.2.3 2016-02-05 19:15:25 +00:00
Ivan Kozik
86e92d684c Send an over18=1 cookie to reddit.com to avoid the age gate on many subreddits 2015-12-12 16:49:42 +00:00
Ivan Kozik
6a647f637e If using Python 3.4.0, depend on an older version of aiohttp that works on Python 3.4.0 2015-12-12 10:08:02 +00:00
Ivan Kozik
ec9f7bdb43 setup.py: if GRAB_SITE_NO_CCHARDET env var set, don't require cchardet; wpull will fall back on chardet 2015-10-28 16:11:23 +00:00
Ivan Kozik
4487c43c83 Use wpull>=1.2.2 2015-10-21 22:38:47 +00:00
Ivan Kozik
9f071a706d setup.py: specify minimum version for all dependencies 
Specifically, this solves a problem where trollius is too old to have
ensure_future.
 2015-08-02 01:47:03 +00:00
Ivan Kozik
b806316cb1 Use built-in ignore sets; don't crash if invalid ignore set is specified 2015-07-29 08:36:36 +00:00
Ivan Kozik
4c0f60cf06 Don't try to install patched-wpull as it doesn't exist 2015-07-29 07:37:52 +00:00
Ivan Kozik
e506d6a103 Add gs-dump-urls, a utility to dump URLs from a wpull.db file 2015-07-28 11:26:10 +00:00
Ivan Kozik
637929ab76 First take on converting grab-site to a Python program 2015-07-27 06:32:08 +00:00
Ivan Kozik
915ed0eeae Use cchardet for faster encoding detection (imported by wpull/thirdparty/dammit.py) 2015-07-21 03:25:56 +00:00
Ivan Kozik
43d8a9594f Move everything and make grab-site installable with pip3 2015-07-18 10:39:04 +00:00