Archiveteam/grab-site
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
974 Commits 1 Branch 0 Tags
Commit Graph

23 Commits

Author SHA1 Message Date
Ivan Kozik
b32da83a0f Install ludios/pyre2, to be used soon for processing ignores 2018-10-04 14:13:05 +00:00
Ivan Kozik
b6127f2077 Use ludios/wpull 1.2.5 2018-10-04 12:32:47 +00:00
Ivan Kozik
eaaf0ec06e Use ludios/wpull for html5-parser support 2018-10-04 11:13:35 +00:00
Ivan Kozik
5a05fa9761 Lock tornado version to 4.5.3 to avoid 5.0, which breaks with: 
File "[...]/lib/python3.4/site-packages/wpull/abstract/client.py", line 9, in <module>
    from wpull.connection import ConnectionPool
  File "[...]/lib/python3.4/site-packages/wpull/connection.py", line 11, in <module>
    from tornado.netutil import SSLCertificateError
ImportError: cannot import name 'SSLCertificateError'
 2018-03-06 06:29:07 +00:00
Ivan Kozik
63fdb9d5c6 Lock html5lib version to work around https://github.com/chfoo/wpull/issues/332 2016-07-15 05:38:38 +00:00
Ivan Kozik
76f8b2cf48 Lock wpull dependency to 1.2.3 for now 2016-06-12 08:47:36 +00:00
Ivan Kozik
aa366bbb27 Update grab-site URL in setup.py and dashboard 2016-05-27 13:53:35 +00:00
Ivan Kozik
f0bb696dc8 Actually install the favicon.ico 2016-05-22 11:01:53 +00:00
Ivan Kozik
3ee97f9fcd Bump dependencies since we only test with the latest versions 2016-02-25 01:08:01 +00:00
12As
990f24684a Remove aiohttp dependency from setup.py 2016-02-23 14:10:15 -06:00
Daniel Oaks
95012d1e0c gs-server: Use env instead of py3 directly, makes virtualenvs nicer 2016-02-16 17:24:20 +00:00
Ivan Kozik
7c1afbefe0 Use wpull 1.2.3 2016-02-05 19:15:25 +00:00
Ivan Kozik
86e92d684c Send an over18=1 cookie to reddit.com to avoid the age gate on many subreddits 2015-12-12 16:49:42 +00:00
Ivan Kozik
6a647f637e If using Python 3.4.0, depend on an older version of aiohttp that works on Python 3.4.0 2015-12-12 10:08:02 +00:00
Ivan Kozik
ec9f7bdb43 setup.py: if GRAB_SITE_NO_CCHARDET env var set, don't require cchardet; wpull will fall back on chardet 2015-10-28 16:11:23 +00:00
Ivan Kozik
4487c43c83 Use wpull>=1.2.2 2015-10-21 22:38:47 +00:00
Ivan Kozik
9f071a706d setup.py: specify minimum version for all dependencies 
Specifically, this solves a problem where trollius is too old to have
ensure_future.
 2015-08-02 01:47:03 +00:00
Ivan Kozik
b806316cb1 Use built-in ignore sets; don't crash if invalid ignore set is specified 2015-07-29 08:36:36 +00:00
Ivan Kozik
4c0f60cf06 Don't try to install patched-wpull as it doesn't exist 2015-07-29 07:37:52 +00:00
Ivan Kozik
e506d6a103 Add gs-dump-urls, a utility to dump URLs from a wpull.db file 2015-07-28 11:26:10 +00:00
Ivan Kozik
637929ab76 First take on converting grab-site to a Python program 2015-07-27 06:32:08 +00:00
Ivan Kozik
915ed0eeae Use cchardet for faster encoding detection (imported by wpull/thirdparty/dammit.py) 2015-07-21 03:25:56 +00:00
Ivan Kozik
43d8a9594f Move everything and make grab-site installable with pip3 2015-07-18 10:39:04 +00:00