Archiveteam/grab-site
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
1,097 Commits 1 Branch 0 Tags
Commit Graph

252 Commits

Author SHA1 Message Date
Ivan Kozik
7dfd0faa37 Mention that --finished-warc-dir takes an absolute path 2019-11-08 06:30:06 +00:00
Ivan Kozik
e22dadb759 README: Python 3.7.3 -> 3.7.5 2019-11-08 01:00:44 +00:00
Ivan Kozik
d418ba4d9b README: add note about having to run apt-get again 
Experienced this odd problem on Ubuntu 19.10 on digitalocean:

root@ctest:~# sudo apt-get update
Get:1 http://security.ubuntu.com/ubuntu eoan-security InRelease [97.5 kB]
Get:2 http://archive.ubuntu.com/ubuntu eoan InRelease [255 kB]
Get:3 http://security.ubuntu.com/ubuntu eoan-security/main amd64 Packages [45.5 kB]
Get:4 http://security.ubuntu.com/ubuntu eoan-security/main Translation-en [18.5 kB]
Get:5 http://archive.ubuntu.com/ubuntu eoan-updates InRelease [97.5 kB]
Get:6 http://security.ubuntu.com/ubuntu eoan-security/main amd64 c-n-f Metadata [876 B]
Get:7 http://security.ubuntu.com/ubuntu eoan-security/restricted amd64 Packages [3456 B]
Get:8 http://security.ubuntu.com/ubuntu eoan-security/restricted Translation-en [996 B]
Get:9 http://security.ubuntu.com/ubuntu eoan-security/universe amd64 Packages [12.2 kB]
Get:10 http://archive.ubuntu.com/ubuntu eoan-backports InRelease [88.8 kB]
Get:11 http://security.ubuntu.com/ubuntu eoan-security/universe Translation-en [8672 B]
Get:12 http://security.ubuntu.com/ubuntu eoan-security/universe amd64 c-n-f Metadata [588 B]
Get:13 http://security.ubuntu.com/ubuntu eoan-security/multiverse amd64 Packages [1164 B]
Get:14 http://security.ubuntu.com/ubuntu eoan-security/multiverse Translation-en [632 B]
Get:15 http://archive.ubuntu.com/ubuntu eoan/main amd64 Packages [969 kB]
Get:16 http://security.ubuntu.com/ubuntu eoan-security/multiverse amd64 c-n-f Metadata [116 B]
Get:17 http://archive.ubuntu.com/ubuntu eoan/main amd64 c-n-f Metadata [29.7 kB]
Get:18 http://archive.ubuntu.com/ubuntu eoan/universe amd64 Packages [8798 kB]
Get:19 http://archive.ubuntu.com/ubuntu eoan/universe Translation-en [5198 kB]
Get:20 http://archive.ubuntu.com/ubuntu eoan/universe amd64 c-n-f Metadata [271 kB]
Get:21 http://archive.ubuntu.com/ubuntu eoan/multiverse amd64 Packages [153 kB]
Get:22 http://archive.ubuntu.com/ubuntu eoan/multiverse Translation-en [111 kB]
Get:23 http://archive.ubuntu.com/ubuntu eoan/multiverse amd64 c-n-f Metadata [9424 B]
Get:24 http://archive.ubuntu.com/ubuntu eoan-updates/main amd64 Packages [48.7 kB]
Get:25 http://archive.ubuntu.com/ubuntu eoan-updates/main Translation-en [19.9 kB]
Get:26 http://archive.ubuntu.com/ubuntu eoan-updates/main amd64 c-n-f Metadata [892 B]
Get:27 http://archive.ubuntu.com/ubuntu eoan-updates/restricted amd64 Packages [3456 B]
Get:28 http://archive.ubuntu.com/ubuntu eoan-updates/restricted Translation-en [996 B]
Get:29 http://archive.ubuntu.com/ubuntu eoan-updates/universe amd64 Packages [17.4 kB]
Get:30 http://archive.ubuntu.com/ubuntu eoan-updates/universe Translation-en [10.7 kB]
Get:31 http://archive.ubuntu.com/ubuntu eoan-updates/universe amd64 c-n-f Metadata [588 B]
Get:32 http://archive.ubuntu.com/ubuntu eoan-updates/multiverse amd64 Packages [1164 B]
Get:33 http://archive.ubuntu.com/ubuntu eoan-updates/multiverse Translation-en [632 B]
Get:34 http://archive.ubuntu.com/ubuntu eoan-updates/multiverse amd64 c-n-f Metadata [112 B]
Get:35 http://archive.ubuntu.com/ubuntu eoan-backports/main amd64 c-n-f Metadata [108 B]
Get:36 http://archive.ubuntu.com/ubuntu eoan-backports/restricted amd64 c-n-f Metadata [116 B]
Get:37 http://archive.ubuntu.com/ubuntu eoan-backports/universe amd64 Packages [2936 B]
Get:38 http://archive.ubuntu.com/ubuntu eoan-backports/universe Translation-en [1268 B]
Get:39 http://archive.ubuntu.com/ubuntu eoan-backports/universe amd64 c-n-f Metadata [112 B]
Get:40 http://archive.ubuntu.com/ubuntu eoan-backports/multiverse amd64 c-n-f Metadata [116 B]
Fetched 16.3 MB in 15s (1122 kB/s)
Reading package lists... Done
root@ctest:~# sudo apt-get install --no-install-recommends \
>     git build-essential libssl-dev zlib1g-dev libbz2-dev libreadline-dev \
>     libsqlite3-dev libffi-dev libxml2-dev libxslt1-dev libre2-dev pkg-config
Reading package lists... Done
Building dependency tree
Reading state information... Done
Package libxml2-dev is not available, but is referred to by another package.
This may mean that the package is missing, has been obsoleted, or
is only available from another source

Package libssl-dev is not available, but is referred to by another package.
This may mean that the package is missing, has been obsoleted, or
is only available from another source

E: Unable to locate package build-essential
E: Package 'libssl-dev' has no installation candidate
E: Unable to locate package zlib1g-dev
E: Unable to locate package libbz2-dev
E: Unable to locate package libreadline-dev
E: Unable to locate package libsqlite3-dev
E: Unable to locate package libffi-dev
E: Package 'libxml2-dev' has no installation candidate
E: Unable to locate package libre2-dev
E: Unable to locate package pkg-config
 2019-11-08 00:56:45 +00:00
Ivan Kozik
ecea5ec002 Use --no-binary lxml now that newer pip broke --no-binary 
File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/pip/_internal/cli/cmdoptions.py", line 435, in _handle_no_binary
    value, existing.no_binary, existing.only_binary,
  File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/pip/_internal/models/format_control.py", line 46, in handle_mutual_excludes
    "--no-binary / --only-binary option requires 1 argument."
pip._internal.exceptions.CommandError: --no-binary / --only-binary option requires 1 argument.

https://travis-ci.org/ArchiveTeam/grab-site/builds/600264421
 2019-10-20 08:38:52 +00:00
Ivan Kozik
5024832257 README: link to macOS bug 2019-07-30 01:18:58 +00:00
Ivan Kozik
5e75c56a7d README: Python 3.7.2 -> 3.7.3 2019-04-28 23:21:34 +00:00
Ivan Kozik
985e84abf5 README: fix macOS Homebrew install steps 
Fixes https://github.com/ArchiveTeam/grab-site/issues/150
 2019-04-19 19:54:29 +00:00
Ivan Kozik
86280b6dcd README: ludios/grab-site -> ArchiveTeam/grab-site 2019-03-01 06:07:18 +00:00
Ivan Kozik
e67f24c800 README: Python 3.7.1 -> 3.7.2 2019-03-01 06:06:04 +00:00
Katie Holly
fe814bf294
removed references for --process-dependency-links 2019-02-12 12:36:09 +01:00
Ivan Kozik
5cbce99e64 README: tweak indentation 2018-12-18 10:22:47 +00:00
Ivan Kozik
ab8009e076 README: update the upgrade steps 2018-12-18 10:21:16 +00:00
Ivan Kozik
0dbfdec3ad README: add alternate Nix-based install steps for macOS 2018-12-18 10:19:41 +00:00
Ivan Kozik
434ee19be0 README: update NixOS & other distribution install instructions 2018-12-18 10:13:34 +00:00
Ivan Kozik
a9727902cd Add back the tumblr Googlebot UA instruction now that it works again; bump Firefox UA 2018-12-16 00:19:25 +00:00
Ivan Kozik
ac2c5d97ee README: Python 3.7.0 -> 3.7.1 2018-12-08 23:29:36 +00:00
Ivan Kozik
4471660656 README: use spaces for indentation to avoid tab-completion on Tab paste 2018-10-22 15:25:57 +00:00
Ivan Kozik
e650278421 README: try to get people to stop installing grab-site as root and explain the pip options 2018-10-22 15:24:50 +00:00
Ivan Kozik
d22ea9ad78 README: add install steps for NixOS 18.09 2018-10-18 15:04:50 +00:00
Ivan Kozik
4f898d2b06 README: mention pkg-config 2018-10-09 18:32:52 +00:00
Ivan Kozik
9022a57f6f README: move pkg-config to the end because it's for grab-site's dependencies and not for Python 2018-10-09 18:09:02 +00:00
Ivan Kozik
db251482cf README: abbreviate webrecorder.io instructions 2018-10-09 17:49:15 +00:00
Ivan Kozik
73587696f2 README: http:// -> https:// links 2018-10-09 17:42:35 +00:00
Ivan Kozik
ab7e20eb4d README: update webrecorder.io instructions 2018-10-09 17:40:01 +00:00
Ivan Kozik
20156b1f76 README: write more about how to archive Twitter users 2018-10-09 17:26:22 +00:00
Ivan Kozik
ea94a3d925 README: fix macOS install instructions 2018-10-09 16:53:03 +00:00
Ivan Kozik
d737e09b3b README: fix link 2018-10-09 16:44:19 +00:00
Ivan Kozik
5b67d90cfb README: install instruction fixes 2018-10-09 16:00:57 +00:00
Ivan Kozik
ce80bcc8c1 wpull_hooks: when igon is enabled, print which ignore was responsible 2018-10-09 13:22:39 +00:00
Ivan Kozik
53370c78db README: Ubuntu 14.04 and Debian 8 (jessie) are no longer supported 2018-10-09 06:10:03 +00:00
Ivan Kozik
cc839e3468 README: fix TOC link 2018-10-09 04:10:31 +00:00
Ivan Kozik
8e6649d144 README: install libssl-dev instead of libssl1.0-dev now that we no longer use Python 3.4 2018-10-09 04:03:44 +00:00
Ivan Kozik
2a1f0b9548 README: update macOS instructions for lxml and pyre2 (untested) 
Brew has a Python 3.7.0 right now, so there is no need to compile a Python with pyenv.
 2018-10-08 07:09:50 +00:00
Ivan Kozik
7589d57e75 Thank falconkirtaran 2018-10-07 22:57:42 +00:00
Ivan Kozik
08acad3e1c README: move security note to the end 2018-10-05 16:22:19 +00:00
Ivan Kozik
fca0eaf432 README: wrap lines 2018-10-05 16:19:55 +00:00
Ivan Kozik
c186df0617 README: wrap lines 2018-10-05 16:19:18 +00:00
Ivan Kozik
d3da0899ad README: wrap lines 2018-10-05 16:19:18 +00:00
Ivan Kozik
ea4e4eff74 README: wrap lines 2018-10-05 16:19:18 +00:00
Ivan Kozik
65b4f5caca README: remove phantomjs mention because I removed support from ludios/wpull 2018-10-05 16:19:18 +00:00
Ivan Kozik
e8a2163dd3 README: wrap lines 2018-10-05 16:19:15 +00:00
Ivan Kozik
4473c7040e README: wrap lines 2018-10-05 16:16:04 +00:00
Ivan Kozik
469dfdd0b9 README: wrap lines 2018-10-05 16:13:37 +00:00
Ivan Kozik
e45f6f5b97 README: remove BrowserStack mention 2018-10-05 16:12:41 +00:00
Ivan Kozik
8bf22e410f README: thank JAA 2018-10-05 16:12:15 +00:00
Ivan Kozik
fc54bed43a README: don't tell users to file issues on chfoo/wpull 2018-10-05 16:10:25 +00:00
Ivan Kozik
ca10585c45 README: fix list of options 2018-10-05 16:09:11 +00:00
Ivan Kozik
3f14886435 README: pip3 -> pip 2018-10-05 08:15:23 +00:00
Ivan Kozik
b7dfb14dd8 Upgrade to and require Python 3.7.0 2018-10-05 07:53:58 +00:00
Ivan Kozik
b32da83a0f Install ludios/pyre2, to be used soon for processing ignores 2018-10-04 14:13:05 +00:00