Archiveteam/grab-site
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
1,118 Commits 1 Branch 0 Tags
Commit Graph

1118 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Ivan Kozik
4315cf1831 Document how to grab GitHub issues and PRs 2016-01-09 22:00:42 +00:00
Ivan Kozik
3b9f8c1a4c global igset: ignore /CaptchaImage.axd 2016-01-09 21:46:35 +00:00
Ivan Kozik
dff87eba2f global igset: also ignore www.digg.com/submit 2016-01-09 02:19:18 +00:00
Ivan Kozik
3f295d7ecb README: Tweak 2016-01-07 07:29:02 +00:00
Ivan Kozik
1dc6cac6d8 Document how to use webrecorder.io instead of grab-site for archiving twitter.com/user pages 2016-01-07 07:28:23 +00:00
Ivan Kozik
01bf6d527b Bump version 2016-01-05 01:08:14 +00:00
Ivan Kozik
64d928b82e Merge pull request #67 from DanielOaks/fix-ws-exception 
server: Fix exception when clients visit ws port via a browser
 2016-01-05 01:06:49 +00:00
Daniel Oaks
04179a2a99 server: Fix exception when clients visit ws port via a browser 2016-01-05 10:53:37 +10:00
Ivan Kozik
111ffca643 global igset: ignore two more loops 2016-01-03 02:00:26 +00:00
Ivan Kozik
5f14263070 global igset: ignore livejournal.com/identity/login.bml 2015-12-30 05:35:57 +00:00
Ivan Kozik
7be4017347 README: Upgrade step is not quite right for pyenv users 2015-12-20 00:06:21 +00:00
Ivan Kozik
6e9fd55463 README: Add ~/.local/bin/ instructions here too 2015-12-20 00:04:53 +00:00
Ivan Kozik
c78ec516dc README: Tweak to fix markdown rendering? 2015-12-20 00:03:20 +00:00
Ivan Kozik
5edd79c547 README: Tweak to fix markdown rendering? 2015-12-20 00:02:50 +00:00
Ivan Kozik
7a928ae7fc README: Tweak to fix markdown rendering? 2015-12-20 00:02:08 +00:00
Ivan Kozik
ccdad68099 README: Update TOC 2015-12-20 00:01:07 +00:00
Ivan Kozik
a51ae254df README: Add pyenv instructions 2015-12-20 00:00:13 +00:00
Ivan Kozik
2acc826d56 lstrip '-' to avoid creating filenames that must be --'ed or quoted 2015-12-17 16:54:55 +00:00
Ivan Kozik
495beca32a Revert "README: Tweak" 
This reverts commit dd0c005331dc182e6251823f5c90030c120680b3.

```bash coloring is unhelpfully broken
 2015-12-17 15:19:20 +00:00
Ivan Kozik
dd0c005331 README: Tweak 2015-12-17 15:18:47 +00:00
Ivan Kozik
cb0fa6591b Update Ubuntu 15.10 instructions now that https://bugs.launchpad.net/ubuntu/+source/python-pip/+bug/1518023 is fixed 2015-12-17 15:17:40 +00:00
Ivan Kozik
4ea80eec80 global igset: Ignore a loop on archive.org 2015-12-16 13:11:16 +00:00
Ivan Kozik
38f733f9d2 global igset: ignore /wp-admin/ 2015-12-16 11:00:54 +00:00
Ivan Kozik
adb35ee4e3 Add --id=, --dir=, and --finished-warc-dir= options 2015-12-12 19:52:32 +00:00
Ivan Kozik
17464e7b74 README: Tweak 2015-12-12 18:35:20 +00:00
Ivan Kozik
f8fece9ebb Set NCR=1 cookie for .blogspot.com to avoid getting redirected 2015-12-12 18:29:53 +00:00
Ivan Kozik
7975566add README: Add travis link and shields.io button 2015-12-12 17:58:52 +00:00
Ivan Kozik
0e5f8c8b83 .travis.yml: Test a few more things 2015-12-12 17:50:09 +00:00
Ivan Kozik
9c3c3c3afd .travis.yml: Remove osx until https://github.com/travis-ci/travis-ci/issues/2312 is fixed 2015-12-12 17:46:31 +00:00
Ivan Kozik
e023105c5e .travis.yml: pip3 install verbosely 2015-12-12 17:41:23 +00:00
Ivan Kozik
bdf1684834 .travis.yml: Fix script 2015-12-12 17:40:48 +00:00
Ivan Kozik
37ddc8c6b8 .travis.yml: remove --user because we're in a virtualenv 2015-12-12 17:39:18 +00:00
Ivan Kozik
2f121a864b Add a .travis.yml 2015-12-12 17:36:35 +00:00
Ivan Kozik
86e92d684c Send an over18=1 cookie to reddit.com to avoid the age gate on many subreddits 2015-12-12 16:49:42 +00:00
Ivan Kozik
6a647f637e If using Python 3.4.0, depend on an older version of aiohttp that works on Python 3.4.0 2015-12-12 10:08:02 +00:00
Ivan Kozik
d96809ea78 README: Tweak 2015-12-11 09:10:07 +00:00
Ivan Kozik
a9a06127ea README: More subreddit crawling 2015-12-11 09:03:02 +00:00
Ivan Kozik
01a72b910a README: Tweak 2015-12-11 08:56:27 +00:00
Ivan Kozik
a2f1192b92 README: Document how to archive websites whose domains have just expired 2015-12-11 08:54:09 +00:00
Ivan Kozik
0363f825d5 README: Document how to archive subreddits 2015-12-11 08:48:18 +00:00
Ivan Kozik
52895892d5 README: Tweak 2015-12-11 08:44:23 +00:00
Ivan Kozik
e82937a9f6 README: Tweak 2015-12-11 08:43:30 +00:00
Ivan Kozik
6d3f0f901d README: Fix headings 2015-12-11 08:40:30 +00:00
Ivan Kozik
140efe082a README: Write about how to archive specific websites 2015-12-11 08:39:36 +00:00
Ivan Kozik
b000c1a759 README: Tweak 2015-12-06 06:00:04 +00:00
Ivan Kozik
8a6eaea16c Bump Firefox version in UA string 2015-12-04 11:45:28 +00:00
Ivan Kozik
d80bde8c8c Add warning about warcprox 2015-11-30 06:31:21 +00:00
Ivan Kozik
58a2711058 Use Roboto font if installed 2015-11-30 05:16:29 +00:00
Ivan Kozik
e72c5fc3a7 Don't crash if psutil is not available on non-Windows OS (it is no longer installed by wpull 1.2.2) 2015-11-21 19:54:13 +00:00
Ivan Kozik
8c306aed50 Don't use --debug-manhole on Windows to avoid a crash 
File "C:\Python34\lib\site-packages\click\core.py", line 680, in main
    rv = self.invoke(ctx)
  File "C:\Python34\lib\site-packages\click\core.py", line 873, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "C:\Python34\lib\site-packages\click\core.py", line 508, in invoke
    return callback(*args, **kwargs)
  File "C:\Python34\lib\site-packages\libgrabsite\main.py", line 276, in main
    wpull.__main__.main()
  File "C:\Python34\lib\site-packages\wpull\__main__.py", line 38, in main
    manhole.install()
  File "C:\Python34\lib\site-packages\manhole.py", line 565, in install
    _MANHOLE.configure(**kwargs)  # Threads might be started here
  File "C:\Python34\lib\site-packages\manhole.py", line 412, in configure
    self.patch_os_fork_functions()
  File "C:\Python34\lib\site-packages\manhole.py", line 505, in patch_os_fork_functions
    self.original_os_fork, os.fork = os.fork, self.patched_fork
AttributeError: 'module' object has no attribute 'fork'
Exception in thread Manhole:
Traceback (most recent call last):
  File "C:\Python34\lib\threading.py", line 920, in _bootstrap_inner
    self.run()
  File "C:\Python34\lib\site-packages\manhole.py", line 192, in run
    sock = self.get_socket()
  File "C:\Python34\lib\site-packages\manhole.py", line 445, in get_socket
    sock = _ORIGINAL_SOCKET(socket.AF_UNIX, socket.SOCK_STREAM)
AttributeError: 'module' object has no attribute 'AF_UNIX'
 2015-10-28 16:31:39 +00:00