Archiveteam
/
grab-site
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
1,158 Commits
1 Branch
0 Tags
1.4 MiB
Commit Graph

1158 Commits (master)

Author SHA1 Message Date
Ivan Kozik f1e6ae4cd1 README: macOS: mention .zshrc 2021-01-03 02:29:04 +00:00
Ivan Kozik ce93f62a9a README: update both the Homebrew and Nix install steps on macOS 2021-01-03 02:26:01 +00:00
Ivan Kozik be38f488af README: link to better WARC playback tools 2020-08-07 21:25:56 +00:00
Ivan Kozik 12e798b075 2.2.0 2020-07-10 23:57:08 +00:00
Ivan Kozik 087e145175 wpull_hooks: compile combined ignore with re if re2 fails 2020-07-10 23:55:15 +00:00
Ivan Kozik e095e1c5b3 README: Python 3.7.7 -> 3.7.8 2020-07-10 23:47:13 +00:00
Ivan Kozik 8343916e3c README: Python 3.7.5 -> 3.7.7 2020-06-20 02:22:26 +00:00
Ivan Kozik 8365a9d5f5 2.1.19 2020-06-20 02:21:40 +00:00
Ivan Kozik 4032234758 Update Firefox UA 2020-06-20 02:21:26 +00:00
Ivan Kozik e98006f8d0 2.1.18 2019-11-08 06:30:48 +00:00
Ivan Kozik 35f6b6bdae Update Firefox UA 2019-11-08 06:30:08 +00:00
Ivan Kozik 7dfd0faa37 Mention that --finished-warc-dir takes an absolute path 2019-11-08 06:30:06 +00:00
Ivan Kozik e22dadb759 README: Python 3.7.3 -> 3.7.5 2019-11-08 01:00:44 +00:00
Ivan Kozik d418ba4d9b README: add note about having to run apt-get again 
Experienced this odd problem on Ubuntu 19.10 on digitalocean:

root@ctest:~# sudo apt-get update
Get:1 http://security.ubuntu.com/ubuntu eoan-security InRelease [97.5 kB]
Get:2 http://archive.ubuntu.com/ubuntu eoan InRelease [255 kB]
Get:3 http://security.ubuntu.com/ubuntu eoan-security/main amd64 Packages [45.5 kB]
Get:4 http://security.ubuntu.com/ubuntu eoan-security/main Translation-en [18.5 kB]
Get:5 http://archive.ubuntu.com/ubuntu eoan-updates InRelease [97.5 kB]
Get:6 http://security.ubuntu.com/ubuntu eoan-security/main amd64 c-n-f Metadata [876 B]
Get:7 http://security.ubuntu.com/ubuntu eoan-security/restricted amd64 Packages [3456 B]
Get:8 http://security.ubuntu.com/ubuntu eoan-security/restricted Translation-en [996 B]
Get:9 http://security.ubuntu.com/ubuntu eoan-security/universe amd64 Packages [12.2 kB]
Get:10 http://archive.ubuntu.com/ubuntu eoan-backports InRelease [88.8 kB]
Get:11 http://security.ubuntu.com/ubuntu eoan-security/universe Translation-en [8672 B]
Get:12 http://security.ubuntu.com/ubuntu eoan-security/universe amd64 c-n-f Metadata [588 B]
Get:13 http://security.ubuntu.com/ubuntu eoan-security/multiverse amd64 Packages [1164 B]
Get:14 http://security.ubuntu.com/ubuntu eoan-security/multiverse Translation-en [632 B]
Get:15 http://archive.ubuntu.com/ubuntu eoan/main amd64 Packages [969 kB]
Get:16 http://security.ubuntu.com/ubuntu eoan-security/multiverse amd64 c-n-f Metadata [116 B]
Get:17 http://archive.ubuntu.com/ubuntu eoan/main amd64 c-n-f Metadata [29.7 kB]
Get:18 http://archive.ubuntu.com/ubuntu eoan/universe amd64 Packages [8798 kB]
Get:19 http://archive.ubuntu.com/ubuntu eoan/universe Translation-en [5198 kB]
Get:20 http://archive.ubuntu.com/ubuntu eoan/universe amd64 c-n-f Metadata [271 kB]
Get:21 http://archive.ubuntu.com/ubuntu eoan/multiverse amd64 Packages [153 kB]
Get:22 http://archive.ubuntu.com/ubuntu eoan/multiverse Translation-en [111 kB]
Get:23 http://archive.ubuntu.com/ubuntu eoan/multiverse amd64 c-n-f Metadata [9424 B]
Get:24 http://archive.ubuntu.com/ubuntu eoan-updates/main amd64 Packages [48.7 kB]
Get:25 http://archive.ubuntu.com/ubuntu eoan-updates/main Translation-en [19.9 kB]
Get:26 http://archive.ubuntu.com/ubuntu eoan-updates/main amd64 c-n-f Metadata [892 B]
Get:27 http://archive.ubuntu.com/ubuntu eoan-updates/restricted amd64 Packages [3456 B]
Get:28 http://archive.ubuntu.com/ubuntu eoan-updates/restricted Translation-en [996 B]
Get:29 http://archive.ubuntu.com/ubuntu eoan-updates/universe amd64 Packages [17.4 kB]
Get:30 http://archive.ubuntu.com/ubuntu eoan-updates/universe Translation-en [10.7 kB]
Get:31 http://archive.ubuntu.com/ubuntu eoan-updates/universe amd64 c-n-f Metadata [588 B]
Get:32 http://archive.ubuntu.com/ubuntu eoan-updates/multiverse amd64 Packages [1164 B]
Get:33 http://archive.ubuntu.com/ubuntu eoan-updates/multiverse Translation-en [632 B]
Get:34 http://archive.ubuntu.com/ubuntu eoan-updates/multiverse amd64 c-n-f Metadata [112 B]
Get:35 http://archive.ubuntu.com/ubuntu eoan-backports/main amd64 c-n-f Metadata [108 B]
Get:36 http://archive.ubuntu.com/ubuntu eoan-backports/restricted amd64 c-n-f Metadata [116 B]
Get:37 http://archive.ubuntu.com/ubuntu eoan-backports/universe amd64 Packages [2936 B]
Get:38 http://archive.ubuntu.com/ubuntu eoan-backports/universe Translation-en [1268 B]
Get:39 http://archive.ubuntu.com/ubuntu eoan-backports/universe amd64 c-n-f Metadata [112 B]
Get:40 http://archive.ubuntu.com/ubuntu eoan-backports/multiverse amd64 c-n-f Metadata [116 B]
Fetched 16.3 MB in 15s (1122 kB/s)
Reading package lists... Done
root@ctest:~# sudo apt-get install --no-install-recommends \
>     git build-essential libssl-dev zlib1g-dev libbz2-dev libreadline-dev \
>     libsqlite3-dev libffi-dev libxml2-dev libxslt1-dev libre2-dev pkg-config
Reading package lists... Done
Building dependency tree
Reading state information... Done
Package libxml2-dev is not available, but is referred to by another package.
This may mean that the package is missing, has been obsoleted, or
is only available from another source

Package libssl-dev is not available, but is referred to by another package.
This may mean that the package is missing, has been obsoleted, or
is only available from another source

E: Unable to locate package build-essential
E: Package 'libssl-dev' has no installation candidate
E: Unable to locate package zlib1g-dev
E: Unable to locate package libbz2-dev
E: Unable to locate package libreadline-dev
E: Unable to locate package libsqlite3-dev
E: Unable to locate package libffi-dev
E: Package 'libxml2-dev' has no installation candidate
E: Unable to locate package libre2-dev
E: Unable to locate package pkg-config
 2019-11-08 00:56:45 +00:00
Ivan Kozik 02343335b0 2.1.17 2019-10-20 19:07:22 +00:00
Ivan Kozik e5637ca3b1 dashboard: use wss:// if we're behind a reverse proxy serving with https:// 2019-10-20 19:04:52 +00:00
Ivan Kozik ecea5ec002 Use `--no-binary lxml` now that newer pip broke `--no-binary` 
File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/pip/_internal/cli/cmdoptions.py", line 435, in _handle_no_binary
    value, existing.no_binary, existing.only_binary,
  File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/pip/_internal/models/format_control.py", line 46, in handle_mutual_excludes
    "--no-binary / --only-binary option requires 1 argument."
pip._internal.exceptions.CommandError: --no-binary / --only-binary option requires 1 argument.

https://travis-ci.org/ArchiveTeam/grab-site/builds/600264421
 2019-10-20 08:38:52 +00:00
Ivan Kozik 5024832257 README: link to macOS bug 2019-07-30 01:18:58 +00:00
Ivan Kozik 7fb9eecbe2 2.1.16 2019-06-08 01:15:53 +00:00
Ivan Kozik 6f48d929de Don't retry initial URLs forever 
Thanks to JAA for the diagnosis and fix.

Fixes #154
Fixes #129
 2019-06-08 01:14:03 +00:00
Ivan Kozik 5e75c56a7d README: Python 3.7.2 -> 3.7.3 2019-04-28 23:21:34 +00:00
Ivan Kozik bfe7cae5f8 2.1.15 2019-04-28 23:20:30 +00:00
Ivan Kozik e9ebab6f93 default_cookies.txt: skip the age gate on saidit.net 2019-04-28 23:19:59 +00:00
Ivan Kozik 985e84abf5 README: fix macOS Homebrew install steps 
Fixes https://github.com/ArchiveTeam/grab-site/issues/150
 2019-04-19 19:54:29 +00:00
Ivan Kozik 33b5a98c7d 2.1.14 2019-03-01 06:19:42 +00:00
Ivan Kozik d33a42171d setup.py: https://github.com/ludios/wpull -> https://github.com/ArchiveTeam/ludios_wpull 2019-03-01 06:18:11 +00:00
Ivan Kozik 86280b6dcd README: ludios/grab-site -> ArchiveTeam/grab-site 2019-03-01 06:07:18 +00:00
Ivan Kozik e67f24c800 README: Python 3.7.1 -> 3.7.2 2019-03-01 06:06:04 +00:00
Ivan Kozik bebeb4c579 2.1.13 2019-02-12 11:43:37 +00:00
Ivan Kozik 502eccb3eb
Merge pull request #147 from Fusl/fix-145 
Remove use of now-unavailable --process-dependency-links
 2019-02-12 11:42:39 +00:00
Katie Holly fe814bf294
removed references for --process-dependency-links 2019-02-12 12:36:09 +01:00
Katie Holly bb020e741d
Reference git repo in install_requires 
Fixes #145
 2019-02-12 12:00:46 +01:00
Ivan Kozik 5cbce99e64 README: tweak indentation 2018-12-18 10:22:47 +00:00
Ivan Kozik ab8009e076 README: update the upgrade steps 2018-12-18 10:21:16 +00:00
Ivan Kozik 0dbfdec3ad README: add alternate Nix-based install steps for macOS 2018-12-18 10:19:41 +00:00
Ivan Kozik 434ee19be0 README: update NixOS & other distribution install instructions 2018-12-18 10:13:34 +00:00
Ivan Kozik a9727902cd Add back the tumblr Googlebot UA instruction now that it works again; bump Firefox UA 2018-12-16 00:19:25 +00:00
Ivan Kozik d5b7aa6717 2.1.11 2018-12-14 04:17:08 +00:00
Ivan Kozik e7b99e761e Fix tests/online-tests 2018-12-14 03:44:53 +00:00
Ivan Kozik 2524e9c5e7 Make online tests use localhost only, and factor out tests that require no networking at all 2018-12-14 03:41:23 +00:00
Ivan Kozik 8d1402e410 2.1.10 2018-12-14 02:17:16 +00:00
Ivan Kozik 9d121a80af Tweak .editorconfig 2018-12-12 01:20:40 +00:00
Ivan Kozik ac2c5d97ee README: Python 3.7.0 -> 3.7.1 2018-12-08 23:29:36 +00:00
Ivan Kozik 4471660656 README: use spaces for indentation to avoid tab-completion on Tab paste 2018-10-22 15:25:57 +00:00
Ivan Kozik e650278421 README: try to get people to stop installing grab-site as root and explain the pip options 2018-10-22 15:24:50 +00:00
Ivan Kozik d22ea9ad78 README: add install steps for NixOS 18.09 2018-10-18 15:04:50 +00:00
Ivan Kozik 700ea18693 Use ludios/wpull 3.0.7 to fix https://github.com/ludios/grab-site/issues/138 2018-10-16 21:24:44 +00:00
Ivan Kozik 50a3db6e59 Use ludios/wpull 3.0.6 2018-10-16 16:05:06 +00:00
Ivan Kozik 0c18c9c0ea Alignment 2018-10-14 10:11:33 +00:00
Ivan Kozik 4f898d2b06 README: mention pkg-config 2018-10-09 18:32:52 +00:00