Archiveteam/grab-site
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
1,130 Commits 1 Branch 0 Tags
Commit Graph

273 Commits

Author SHA1 Message Date
Ivan Kozik
14c3bbdf71 README: nixpkgs 21.05 -> 21.11 2022-01-02 07:21:18 +00:00
Ivan Kozik
5a306b6917 README: Python 3.7.11 -> 3.8.12; mention Debian 11 2022-01-02 07:20:55 +00:00
Ivan Kozik
41a77f0812 README: have consistency among the 'and then restart your shell' text 2022-01-02 07:00:21 +00:00
Ivan Kozik
d696604509 README: tabs before spaces 2022-01-02 06:58:29 +00:00
Preservation-Quest
865aae75c9 Update README.md 
Updated as requested.
 2022-01-02 06:57:57 +00:00
Preservation-Quest
9a3e577295 Update README.md 2022-01-02 06:57:57 +00:00
Preservation-Quest
9f6e012c83 Update README.md 2022-01-02 06:57:57 +00:00
Ivan Kozik
6269289a2c README: document --wpull-args=--no-warc-compression 2021-10-25 19:58:55 +00:00
Ivan Kozik
4fbf6469a1 README: fix the nix-based install steps to use release-21.05 because master has the incompatible sqlalachemy 1.4 2021-08-29 04:55:59 +00:00
Ivan Kozik
4fec250a32 README: mention Ubuntu 20.04 2021-08-29 04:33:48 +00:00
Ivan Kozik
0f9585db6f README: Python 3.7.10 -> 3.7.11 2021-08-29 04:25:01 +00:00
Ivan Kozik
fe3cc6ab14 README: Python 3.7.9 -> 3.7.10 2021-05-10 01:00:13 +00:00
Ivan Kozik
132064a24e README: Debian-based install: Python 3.7.8 -> 3.7.9 2021-01-03 02:37:11 +00:00
Ivan Kozik
8c5681c726 README: macOS: update maximum version 2021-01-03 02:35:33 +00:00
Ivan Kozik
d25e419fb6 README: remove the note about the macOS-specific lxml bug because I can't repro it 2021-01-03 02:31:44 +00:00
Ivan Kozik
f1e6ae4cd1 README: macOS: mention .zshrc 2021-01-03 02:29:04 +00:00
Ivan Kozik
ce93f62a9a README: update both the Homebrew and Nix install steps on macOS 2021-01-03 02:26:01 +00:00
Ivan Kozik
be38f488af README: link to better WARC playback tools 2020-08-07 21:25:56 +00:00
Ivan Kozik
e095e1c5b3 README: Python 3.7.7 -> 3.7.8 2020-07-10 23:47:13 +00:00
Ivan Kozik
8343916e3c README: Python 3.7.5 -> 3.7.7 2020-06-20 02:22:26 +00:00
Ivan Kozik
35f6b6bdae Update Firefox UA 2019-11-08 06:30:08 +00:00
Ivan Kozik
7dfd0faa37 Mention that --finished-warc-dir takes an absolute path 2019-11-08 06:30:06 +00:00
Ivan Kozik
e22dadb759 README: Python 3.7.3 -> 3.7.5 2019-11-08 01:00:44 +00:00
Ivan Kozik
d418ba4d9b README: add note about having to run apt-get again 
Experienced this odd problem on Ubuntu 19.10 on digitalocean:

root@ctest:~# sudo apt-get update
Get:1 http://security.ubuntu.com/ubuntu eoan-security InRelease [97.5 kB]
Get:2 http://archive.ubuntu.com/ubuntu eoan InRelease [255 kB]
Get:3 http://security.ubuntu.com/ubuntu eoan-security/main amd64 Packages [45.5 kB]
Get:4 http://security.ubuntu.com/ubuntu eoan-security/main Translation-en [18.5 kB]
Get:5 http://archive.ubuntu.com/ubuntu eoan-updates InRelease [97.5 kB]
Get:6 http://security.ubuntu.com/ubuntu eoan-security/main amd64 c-n-f Metadata [876 B]
Get:7 http://security.ubuntu.com/ubuntu eoan-security/restricted amd64 Packages [3456 B]
Get:8 http://security.ubuntu.com/ubuntu eoan-security/restricted Translation-en [996 B]
Get:9 http://security.ubuntu.com/ubuntu eoan-security/universe amd64 Packages [12.2 kB]
Get:10 http://archive.ubuntu.com/ubuntu eoan-backports InRelease [88.8 kB]
Get:11 http://security.ubuntu.com/ubuntu eoan-security/universe Translation-en [8672 B]
Get:12 http://security.ubuntu.com/ubuntu eoan-security/universe amd64 c-n-f Metadata [588 B]
Get:13 http://security.ubuntu.com/ubuntu eoan-security/multiverse amd64 Packages [1164 B]
Get:14 http://security.ubuntu.com/ubuntu eoan-security/multiverse Translation-en [632 B]
Get:15 http://archive.ubuntu.com/ubuntu eoan/main amd64 Packages [969 kB]
Get:16 http://security.ubuntu.com/ubuntu eoan-security/multiverse amd64 c-n-f Metadata [116 B]
Get:17 http://archive.ubuntu.com/ubuntu eoan/main amd64 c-n-f Metadata [29.7 kB]
Get:18 http://archive.ubuntu.com/ubuntu eoan/universe amd64 Packages [8798 kB]
Get:19 http://archive.ubuntu.com/ubuntu eoan/universe Translation-en [5198 kB]
Get:20 http://archive.ubuntu.com/ubuntu eoan/universe amd64 c-n-f Metadata [271 kB]
Get:21 http://archive.ubuntu.com/ubuntu eoan/multiverse amd64 Packages [153 kB]
Get:22 http://archive.ubuntu.com/ubuntu eoan/multiverse Translation-en [111 kB]
Get:23 http://archive.ubuntu.com/ubuntu eoan/multiverse amd64 c-n-f Metadata [9424 B]
Get:24 http://archive.ubuntu.com/ubuntu eoan-updates/main amd64 Packages [48.7 kB]
Get:25 http://archive.ubuntu.com/ubuntu eoan-updates/main Translation-en [19.9 kB]
Get:26 http://archive.ubuntu.com/ubuntu eoan-updates/main amd64 c-n-f Metadata [892 B]
Get:27 http://archive.ubuntu.com/ubuntu eoan-updates/restricted amd64 Packages [3456 B]
Get:28 http://archive.ubuntu.com/ubuntu eoan-updates/restricted Translation-en [996 B]
Get:29 http://archive.ubuntu.com/ubuntu eoan-updates/universe amd64 Packages [17.4 kB]
Get:30 http://archive.ubuntu.com/ubuntu eoan-updates/universe Translation-en [10.7 kB]
Get:31 http://archive.ubuntu.com/ubuntu eoan-updates/universe amd64 c-n-f Metadata [588 B]
Get:32 http://archive.ubuntu.com/ubuntu eoan-updates/multiverse amd64 Packages [1164 B]
Get:33 http://archive.ubuntu.com/ubuntu eoan-updates/multiverse Translation-en [632 B]
Get:34 http://archive.ubuntu.com/ubuntu eoan-updates/multiverse amd64 c-n-f Metadata [112 B]
Get:35 http://archive.ubuntu.com/ubuntu eoan-backports/main amd64 c-n-f Metadata [108 B]
Get:36 http://archive.ubuntu.com/ubuntu eoan-backports/restricted amd64 c-n-f Metadata [116 B]
Get:37 http://archive.ubuntu.com/ubuntu eoan-backports/universe amd64 Packages [2936 B]
Get:38 http://archive.ubuntu.com/ubuntu eoan-backports/universe Translation-en [1268 B]
Get:39 http://archive.ubuntu.com/ubuntu eoan-backports/universe amd64 c-n-f Metadata [112 B]
Get:40 http://archive.ubuntu.com/ubuntu eoan-backports/multiverse amd64 c-n-f Metadata [116 B]
Fetched 16.3 MB in 15s (1122 kB/s)
Reading package lists... Done
root@ctest:~# sudo apt-get install --no-install-recommends \
>     git build-essential libssl-dev zlib1g-dev libbz2-dev libreadline-dev \
>     libsqlite3-dev libffi-dev libxml2-dev libxslt1-dev libre2-dev pkg-config
Reading package lists... Done
Building dependency tree
Reading state information... Done
Package libxml2-dev is not available, but is referred to by another package.
This may mean that the package is missing, has been obsoleted, or
is only available from another source

Package libssl-dev is not available, but is referred to by another package.
This may mean that the package is missing, has been obsoleted, or
is only available from another source

E: Unable to locate package build-essential
E: Package 'libssl-dev' has no installation candidate
E: Unable to locate package zlib1g-dev
E: Unable to locate package libbz2-dev
E: Unable to locate package libreadline-dev
E: Unable to locate package libsqlite3-dev
E: Unable to locate package libffi-dev
E: Package 'libxml2-dev' has no installation candidate
E: Unable to locate package libre2-dev
E: Unable to locate package pkg-config
 2019-11-08 00:56:45 +00:00
Ivan Kozik
ecea5ec002 Use --no-binary lxml now that newer pip broke --no-binary 
File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/pip/_internal/cli/cmdoptions.py", line 435, in _handle_no_binary
    value, existing.no_binary, existing.only_binary,
  File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/pip/_internal/models/format_control.py", line 46, in handle_mutual_excludes
    "--no-binary / --only-binary option requires 1 argument."
pip._internal.exceptions.CommandError: --no-binary / --only-binary option requires 1 argument.

https://travis-ci.org/ArchiveTeam/grab-site/builds/600264421
 2019-10-20 08:38:52 +00:00
Ivan Kozik
5024832257 README: link to macOS bug 2019-07-30 01:18:58 +00:00
Ivan Kozik
5e75c56a7d README: Python 3.7.2 -> 3.7.3 2019-04-28 23:21:34 +00:00
Ivan Kozik
985e84abf5 README: fix macOS Homebrew install steps 
Fixes https://github.com/ArchiveTeam/grab-site/issues/150
 2019-04-19 19:54:29 +00:00
Ivan Kozik
86280b6dcd README: ludios/grab-site -> ArchiveTeam/grab-site 2019-03-01 06:07:18 +00:00
Ivan Kozik
e67f24c800 README: Python 3.7.1 -> 3.7.2 2019-03-01 06:06:04 +00:00
Katie Holly
fe814bf294
removed references for --process-dependency-links 2019-02-12 12:36:09 +01:00
Ivan Kozik
5cbce99e64 README: tweak indentation 2018-12-18 10:22:47 +00:00
Ivan Kozik
ab8009e076 README: update the upgrade steps 2018-12-18 10:21:16 +00:00
Ivan Kozik
0dbfdec3ad README: add alternate Nix-based install steps for macOS 2018-12-18 10:19:41 +00:00
Ivan Kozik
434ee19be0 README: update NixOS & other distribution install instructions 2018-12-18 10:13:34 +00:00
Ivan Kozik
a9727902cd Add back the tumblr Googlebot UA instruction now that it works again; bump Firefox UA 2018-12-16 00:19:25 +00:00
Ivan Kozik
ac2c5d97ee README: Python 3.7.0 -> 3.7.1 2018-12-08 23:29:36 +00:00
Ivan Kozik
4471660656 README: use spaces for indentation to avoid tab-completion on Tab paste 2018-10-22 15:25:57 +00:00
Ivan Kozik
e650278421 README: try to get people to stop installing grab-site as root and explain the pip options 2018-10-22 15:24:50 +00:00
Ivan Kozik
d22ea9ad78 README: add install steps for NixOS 18.09 2018-10-18 15:04:50 +00:00
Ivan Kozik
4f898d2b06 README: mention pkg-config 2018-10-09 18:32:52 +00:00
Ivan Kozik
9022a57f6f README: move pkg-config to the end because it's for grab-site's dependencies and not for Python 2018-10-09 18:09:02 +00:00
Ivan Kozik
db251482cf README: abbreviate webrecorder.io instructions 2018-10-09 17:49:15 +00:00
Ivan Kozik
73587696f2 README: http:// -> https:// links 2018-10-09 17:42:35 +00:00
Ivan Kozik
ab7e20eb4d README: update webrecorder.io instructions 2018-10-09 17:40:01 +00:00
Ivan Kozik
20156b1f76 README: write more about how to archive Twitter users 2018-10-09 17:26:22 +00:00
Ivan Kozik
ea94a3d925 README: fix macOS install instructions 2018-10-09 16:53:03 +00:00
Ivan Kozik
d737e09b3b README: fix link 2018-10-09 16:44:19 +00:00
Ivan Kozik
5b67d90cfb README: install instruction fixes 2018-10-09 16:00:57 +00:00
Ivan Kozik
ce80bcc8c1 wpull_hooks: when igon is enabled, print which ignore was responsible 2018-10-09 13:22:39 +00:00