Archiveteam/grab-site
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
1,158 Commits 1 Branch 0 Tags
Commit Graph

1158 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Ivan Kozik
f1e6ae4cd1 README: macOS: mention .zshrc 2021-01-03 02:29:04 +00:00
Ivan Kozik
ce93f62a9a README: update both the Homebrew and Nix install steps on macOS 2021-01-03 02:26:01 +00:00
Ivan Kozik
be38f488af README: link to better WARC playback tools 2020-08-07 21:25:56 +00:00
Ivan Kozik
12e798b075 2.2.0 2020-07-10 23:57:08 +00:00
Ivan Kozik
087e145175 wpull_hooks: compile combined ignore with re if re2 fails 2020-07-10 23:55:15 +00:00
Ivan Kozik
e095e1c5b3 README: Python 3.7.7 -> 3.7.8 2020-07-10 23:47:13 +00:00
Ivan Kozik
8343916e3c README: Python 3.7.5 -> 3.7.7 2020-06-20 02:22:26 +00:00
Ivan Kozik
8365a9d5f5 2.1.19 2020-06-20 02:21:40 +00:00
Ivan Kozik
4032234758 Update Firefox UA 2020-06-20 02:21:26 +00:00
Ivan Kozik
e98006f8d0 2.1.18 2019-11-08 06:30:48 +00:00
Ivan Kozik
35f6b6bdae Update Firefox UA 2019-11-08 06:30:08 +00:00
Ivan Kozik
7dfd0faa37 Mention that --finished-warc-dir takes an absolute path 2019-11-08 06:30:06 +00:00
Ivan Kozik
e22dadb759 README: Python 3.7.3 -> 3.7.5 2019-11-08 01:00:44 +00:00
Ivan Kozik
d418ba4d9b README: add note about having to run apt-get again 
Experienced this odd problem on Ubuntu 19.10 on digitalocean:

root@ctest:~# sudo apt-get update
Get:1 http://security.ubuntu.com/ubuntu eoan-security InRelease [97.5 kB]
Get:2 http://archive.ubuntu.com/ubuntu eoan InRelease [255 kB]
Get:3 http://security.ubuntu.com/ubuntu eoan-security/main amd64 Packages [45.5 kB]
Get:4 http://security.ubuntu.com/ubuntu eoan-security/main Translation-en [18.5 kB]
Get:5 http://archive.ubuntu.com/ubuntu eoan-updates InRelease [97.5 kB]
Get:6 http://security.ubuntu.com/ubuntu eoan-security/main amd64 c-n-f Metadata [876 B]
Get:7 http://security.ubuntu.com/ubuntu eoan-security/restricted amd64 Packages [3456 B]
Get:8 http://security.ubuntu.com/ubuntu eoan-security/restricted Translation-en [996 B]
Get:9 http://security.ubuntu.com/ubuntu eoan-security/universe amd64 Packages [12.2 kB]
Get:10 http://archive.ubuntu.com/ubuntu eoan-backports InRelease [88.8 kB]
Get:11 http://security.ubuntu.com/ubuntu eoan-security/universe Translation-en [8672 B]
Get:12 http://security.ubuntu.com/ubuntu eoan-security/universe amd64 c-n-f Metadata [588 B]
Get:13 http://security.ubuntu.com/ubuntu eoan-security/multiverse amd64 Packages [1164 B]
Get:14 http://security.ubuntu.com/ubuntu eoan-security/multiverse Translation-en [632 B]
Get:15 http://archive.ubuntu.com/ubuntu eoan/main amd64 Packages [969 kB]
Get:16 http://security.ubuntu.com/ubuntu eoan-security/multiverse amd64 c-n-f Metadata [116 B]
Get:17 http://archive.ubuntu.com/ubuntu eoan/main amd64 c-n-f Metadata [29.7 kB]
Get:18 http://archive.ubuntu.com/ubuntu eoan/universe amd64 Packages [8798 kB]
Get:19 http://archive.ubuntu.com/ubuntu eoan/universe Translation-en [5198 kB]
Get:20 http://archive.ubuntu.com/ubuntu eoan/universe amd64 c-n-f Metadata [271 kB]
Get:21 http://archive.ubuntu.com/ubuntu eoan/multiverse amd64 Packages [153 kB]
Get:22 http://archive.ubuntu.com/ubuntu eoan/multiverse Translation-en [111 kB]
Get:23 http://archive.ubuntu.com/ubuntu eoan/multiverse amd64 c-n-f Metadata [9424 B]
Get:24 http://archive.ubuntu.com/ubuntu eoan-updates/main amd64 Packages [48.7 kB]
Get:25 http://archive.ubuntu.com/ubuntu eoan-updates/main Translation-en [19.9 kB]
Get:26 http://archive.ubuntu.com/ubuntu eoan-updates/main amd64 c-n-f Metadata [892 B]
Get:27 http://archive.ubuntu.com/ubuntu eoan-updates/restricted amd64 Packages [3456 B]
Get:28 http://archive.ubuntu.com/ubuntu eoan-updates/restricted Translation-en [996 B]
Get:29 http://archive.ubuntu.com/ubuntu eoan-updates/universe amd64 Packages [17.4 kB]
Get:30 http://archive.ubuntu.com/ubuntu eoan-updates/universe Translation-en [10.7 kB]
Get:31 http://archive.ubuntu.com/ubuntu eoan-updates/universe amd64 c-n-f Metadata [588 B]
Get:32 http://archive.ubuntu.com/ubuntu eoan-updates/multiverse amd64 Packages [1164 B]
Get:33 http://archive.ubuntu.com/ubuntu eoan-updates/multiverse Translation-en [632 B]
Get:34 http://archive.ubuntu.com/ubuntu eoan-updates/multiverse amd64 c-n-f Metadata [112 B]
Get:35 http://archive.ubuntu.com/ubuntu eoan-backports/main amd64 c-n-f Metadata [108 B]
Get:36 http://archive.ubuntu.com/ubuntu eoan-backports/restricted amd64 c-n-f Metadata [116 B]
Get:37 http://archive.ubuntu.com/ubuntu eoan-backports/universe amd64 Packages [2936 B]
Get:38 http://archive.ubuntu.com/ubuntu eoan-backports/universe Translation-en [1268 B]
Get:39 http://archive.ubuntu.com/ubuntu eoan-backports/universe amd64 c-n-f Metadata [112 B]
Get:40 http://archive.ubuntu.com/ubuntu eoan-backports/multiverse amd64 c-n-f Metadata [116 B]
Fetched 16.3 MB in 15s (1122 kB/s)
Reading package lists... Done
root@ctest:~# sudo apt-get install --no-install-recommends \
>     git build-essential libssl-dev zlib1g-dev libbz2-dev libreadline-dev \
>     libsqlite3-dev libffi-dev libxml2-dev libxslt1-dev libre2-dev pkg-config
Reading package lists... Done
Building dependency tree
Reading state information... Done
Package libxml2-dev is not available, but is referred to by another package.
This may mean that the package is missing, has been obsoleted, or
is only available from another source

Package libssl-dev is not available, but is referred to by another package.
This may mean that the package is missing, has been obsoleted, or
is only available from another source

E: Unable to locate package build-essential
E: Package 'libssl-dev' has no installation candidate
E: Unable to locate package zlib1g-dev
E: Unable to locate package libbz2-dev
E: Unable to locate package libreadline-dev
E: Unable to locate package libsqlite3-dev
E: Unable to locate package libffi-dev
E: Package 'libxml2-dev' has no installation candidate
E: Unable to locate package libre2-dev
E: Unable to locate package pkg-config
 2019-11-08 00:56:45 +00:00
Ivan Kozik
02343335b0 2.1.17 2019-10-20 19:07:22 +00:00
Ivan Kozik
e5637ca3b1 dashboard: use wss:// if we're behind a reverse proxy serving with https:// 2019-10-20 19:04:52 +00:00
Ivan Kozik
ecea5ec002 Use --no-binary lxml now that newer pip broke --no-binary 
File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/pip/_internal/cli/cmdoptions.py", line 435, in _handle_no_binary
    value, existing.no_binary, existing.only_binary,
  File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/pip/_internal/models/format_control.py", line 46, in handle_mutual_excludes
    "--no-binary / --only-binary option requires 1 argument."
pip._internal.exceptions.CommandError: --no-binary / --only-binary option requires 1 argument.

https://travis-ci.org/ArchiveTeam/grab-site/builds/600264421
 2019-10-20 08:38:52 +00:00
Ivan Kozik
5024832257 README: link to macOS bug 2019-07-30 01:18:58 +00:00
Ivan Kozik
7fb9eecbe2 2.1.16 2019-06-08 01:15:53 +00:00
Ivan Kozik
6f48d929de Don't retry initial URLs forever 
Thanks to JAA for the diagnosis and fix.

Fixes #154
Fixes #129
 2019-06-08 01:14:03 +00:00
Ivan Kozik
5e75c56a7d README: Python 3.7.2 -> 3.7.3 2019-04-28 23:21:34 +00:00
Ivan Kozik
bfe7cae5f8 2.1.15 2019-04-28 23:20:30 +00:00
Ivan Kozik
e9ebab6f93 default_cookies.txt: skip the age gate on saidit.net 2019-04-28 23:19:59 +00:00
Ivan Kozik
985e84abf5 README: fix macOS Homebrew install steps 
Fixes https://github.com/ArchiveTeam/grab-site/issues/150
 2019-04-19 19:54:29 +00:00
Ivan Kozik
33b5a98c7d 2.1.14 2019-03-01 06:19:42 +00:00
Ivan Kozik
d33a42171d setup.py: https://github.com/ludios/wpull -> https://github.com/ArchiveTeam/ludios_wpull 2019-03-01 06:18:11 +00:00
Ivan Kozik
86280b6dcd README: ludios/grab-site -> ArchiveTeam/grab-site 2019-03-01 06:07:18 +00:00
Ivan Kozik
e67f24c800 README: Python 3.7.1 -> 3.7.2 2019-03-01 06:06:04 +00:00
Ivan Kozik
bebeb4c579 2.1.13 2019-02-12 11:43:37 +00:00
Ivan Kozik
502eccb3eb
Merge pull request #147 from Fusl/fix-145 
Remove use of now-unavailable --process-dependency-links
 2019-02-12 11:42:39 +00:00
Katie Holly
fe814bf294
removed references for --process-dependency-links 2019-02-12 12:36:09 +01:00
Katie Holly
bb020e741d
Reference git repo in install_requires 
Fixes #145
 2019-02-12 12:00:46 +01:00
Ivan Kozik
5cbce99e64 README: tweak indentation 2018-12-18 10:22:47 +00:00
Ivan Kozik
ab8009e076 README: update the upgrade steps 2018-12-18 10:21:16 +00:00
Ivan Kozik
0dbfdec3ad README: add alternate Nix-based install steps for macOS 2018-12-18 10:19:41 +00:00
Ivan Kozik
434ee19be0 README: update NixOS & other distribution install instructions 2018-12-18 10:13:34 +00:00
Ivan Kozik
a9727902cd Add back the tumblr Googlebot UA instruction now that it works again; bump Firefox UA 2018-12-16 00:19:25 +00:00
Ivan Kozik
d5b7aa6717 2.1.11 2018-12-14 04:17:08 +00:00
Ivan Kozik
e7b99e761e Fix tests/online-tests 2018-12-14 03:44:53 +00:00
Ivan Kozik
2524e9c5e7 Make online tests use localhost only, and factor out tests that require no networking at all 2018-12-14 03:41:23 +00:00
Ivan Kozik
8d1402e410 2.1.10 2018-12-14 02:17:16 +00:00
Ivan Kozik
9d121a80af Tweak .editorconfig 2018-12-12 01:20:40 +00:00
Ivan Kozik
ac2c5d97ee README: Python 3.7.0 -> 3.7.1 2018-12-08 23:29:36 +00:00
Ivan Kozik
4471660656 README: use spaces for indentation to avoid tab-completion on Tab paste 2018-10-22 15:25:57 +00:00
Ivan Kozik
e650278421 README: try to get people to stop installing grab-site as root and explain the pip options 2018-10-22 15:24:50 +00:00
Ivan Kozik
d22ea9ad78 README: add install steps for NixOS 18.09 2018-10-18 15:04:50 +00:00
Ivan Kozik
700ea18693 Use ludios/wpull 3.0.7 to fix https://github.com/ludios/grab-site/issues/138 2018-10-16 21:24:44 +00:00
Ivan Kozik
50a3db6e59 Use ludios/wpull 3.0.6 2018-10-16 16:05:06 +00:00
Ivan Kozik
0c18c9c0ea Alignment 2018-10-14 10:11:33 +00:00
Ivan Kozik
4f898d2b06 README: mention pkg-config 2018-10-09 18:32:52 +00:00