mikf/gallery-dl
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
2,018 Commits 1 Branch 0 Tags
Commit Graph

1438 Commits

Author SHA1 Message Date
Mike Fährmann
4465a3ea68
[kissmanga][readcomiconline] add 'captcha' option (#279) 
to configure how to handle CAPTCHA page redirects:
- either interactively wait for the user to solve the CAPTCHA
- or raise StopExtraction like before
 2019-05-27 22:24:48 +02:00
Mike Fährmann
1e3e15c4f3
[sankaku] add article extractor (#258) 2019-05-26 17:42:36 +02:00
Mike Fährmann
48233f00c0
[readcomiconline] detect 'AreYouHuman' redirects (#279) 2019-05-26 15:58:37 +02:00
Mike Fährmann
1cde38110d
[livedoor] return 'date' as datetime object 2019-05-25 23:45:56 +02:00
Mike Fährmann
e88824e1a7
[livedoor] fix adjustments for https:// URLs 2019-05-25 23:45:22 +02:00
Mike Fährmann
b3e4664715
[hentainexus] fix extraction 2019-05-25 22:35:04 +02:00
Mike Fährmann
399e8e965a
also update urllib3's cipher list for versions >= 1.25 2019-05-21 23:02:20 +02:00
Mike Fährmann
f837ea98cb
[deviantart] don't call 'extend()' on folders (fixes #271) 2019-05-20 16:24:13 +02:00
Mike Fährmann
bb32a2d490
[patreon] use file extensions from original filenames (#268) 2019-05-20 15:46:59 +02:00
Mike Fährmann
efa805c5d7
[sankaku] update pagination end condition (fixes #265) 
Pagination over popular listings (`date:...+order:popular") never
terminates, not even on the site itself, and at some point returns the
same results over and over again.
 2019-05-20 15:46:06 +02:00
Mike Fährmann
a4ba34c835
[booru] prevent crash when no tags are present (#259) 2019-05-17 19:32:53 +02:00
Mike Fährmann
ca3bad1779
[patreon] small fixes and adjustments (#226) 
- fix datetime parsing
- rename 'user' to 'creator'
- convert 'id' to integer
- improve tests
 2019-05-17 19:32:41 +02:00
Leonardo Taccari
fb09dd962a [instagram] Fix extraction after `rhx_gis' field removal 2019-05-17 18:17:42 +02:00
Mike Fährmann
7a14aaed7d
[luscious] fix extraction 2019-05-17 10:48:47 +02:00
Mike Fährmann
e82cadac61
[patreon] add extractors (#226) 2019-05-17 10:47:22 +02:00
Mike Fährmann
4891f4a328
[hentainexus] add search extractor (#256) 2019-05-16 23:55:30 +02:00
Mike Fährmann
c02f12ce2f
avoid Cloudflare CAPTCHAs for OpenSSL < 1.1.1 
see https://github.com/Anorov/cloudflare-scrape/pull/242
 2019-05-15 12:25:20 +02:00
Mike Fährmann
0b4be57a10
[sankaku] fix error when no tags available (closes #259) 
[ci skip]
 2019-05-14 23:40:07 +02:00
Mike Fährmann
9890bfdf23
[flickr] improve code and metadata 
- simplify pagination
- add more metadata and slightly change its structure
  - convert suitable values to int or list
  - move keys from ["photo"] to the base level
- proper video support (#246)
- rename method and variable names to better fit with other extractors
 2019-05-14 22:10:50 +02:00
Mike Fährmann
aa8e366b90
[luscious] fix tag extraction 2019-05-14 17:35:52 +02:00
Mike Fährmann
ba8eb1ffec
[hentainexus] add gallery extractor (#256) 2019-05-12 23:59:41 +02:00
Mike Fährmann
b1db194c14
[reactor] update and improve 
- split 'tags' into a list
- parse 'date' into a datetime object
- fix webm/mp4 URLs
 2019-05-09 23:24:49 +02:00
Mike Fährmann
b0e85a42e3
apply workaround from 4736912 in parse_datetime() itself 2019-05-09 21:53:17 +02:00
Mike Fährmann
8de5866fd2
[twitter] replace unit test URLs 
https://twitter.com/PicturesEarth was deleted
 2019-05-09 10:17:55 +02:00
Mike Fährmann
74c7304c6b
[newgrounds] extract 'date', 'favorites', and 'score' 2019-05-08 18:09:17 +02:00
Mike Fährmann
4736912d4e
[pixiv] work around strptime limitations in Python < 3.7 
"%z" doesn't allow a colon separator in older Python versions:
    - "+0900" is OK
    - "+09:00" raises an exception
 2019-05-08 18:08:03 +02:00
Mike Fährmann
1f7fa9dc8e
[exhentai] update data extraction code 
- parse 'date' to datetime object
- use 'text.extract_from()'
 2019-05-08 15:44:29 +02:00
Mike Fährmann
80fdb11508
[pixiv] add 'date' metadata field (closes #248) 2019-05-08 15:43:59 +02:00
Mike Fährmann
049e9fd6ce
[twitter] fix pagination end condition 
Some timelines would cause an endless loop because 'has_more_items' is
always True, even if it would return the same list of tweets over and
over again.
 2019-05-08 15:43:59 +02:00
Mike Fährmann
51e0e92429
[deviantart] fix GIF downloads (#242) 
The "original" download URL for GIF animations is only a preview version
of the original file.
 2019-05-08 15:43:43 +02:00
Leonardo Taccari
f347d2d152 [instagram] Fix for missing edge_media_to_comment' field and add date' metadata (#250) 
* [instagram] Remove no longer always present `comments' field

`edge_media_to_comment' is no longer always present in the response
(also for the same media sometimes is present and sometimes is not present).

* [instagram] Add `date' metadata
 2019-05-08 15:42:58 +02:00
Mike Fährmann
5fd94c6b83
import urllib3 from requests.packages 2019-05-04 22:28:07 +02:00
Mike Fährmann
35f343206c
update default SSL cipher list in urllib3 < 1.25 
Cloudflare now also checks the client's SSL/TLS cipher capabilities and
produces a 403: Forbidden response with CAPTCHA if they are insufficient.

This commit replaces the default cipher list in urllib3 < 1.25 with the
one from 1.25 (1), which doesn't cause problems as long as the client
platform actually supports these ciphers. On some platforms (tested with
Python 3.4 on Linux and Python 3.7 on an outdated Windows 7 VM) it is
necessary to install pyOpenSSL to get everything to work.

Explicitly setting a minimum/maximum version for urllib3 is also no
longer necessary and installing gallery-dl will therefore not pull a
incompatible urllib3 version (#229)

Fixes the "403: Forbidden" error on Artstation (#227)

(1) 0cedb3b0f1
 2019-05-03 22:40:04 +02:00
Mike Fährmann
fc5e4f2b21
[hitomi] simplify data extraction code 2019-05-01 11:14:21 +02:00
Mike Fährmann
2756cc8dde
[hitomi] set Referer header (fixes #239) 2019-05-01 10:56:00 +02:00
Mike Fährmann
dcc1592dbf
[twitter] add fallback URLs (#237) 2019-04-30 15:57:21 +02:00
Mike Fährmann
1c665fd4bd
[mangoxo] fix login 2019-04-30 15:57:06 +02:00
Mike Fährmann
add7e693d0
[tumblr] provide parsed 'date' metadata (#232) 2019-04-29 17:30:42 +02:00
Mike Fährmann
9544683d56
[deviantart] provide 'date' metadata (#232) 2019-04-29 17:30:24 +02:00
Mike Fährmann
0d7e8be987
[dynastyscans] simplify image extractor 2019-04-27 13:24:30 +02:00
Mike Fährmann
9aa0bb5afe
[dynastyscans] encode "[]" in search queries 
urllib3 1.25 classifies URLs with unencoded "[" or "]" as invalid
and raises an exception
 2019-04-27 13:22:40 +02:00
Mike Fährmann
fe849382d8
[komikcast] improve extraction 2019-04-26 15:14:10 +02:00
Mike Fährmann
0318c610dc
[sexcom] add extractor for search results (#147) 2019-04-24 22:10:01 +02:00
Mike Fährmann
a247c94c34
[sexcom] add pin and board extractors (#147) 2019-04-24 22:09:19 +02:00
Mike Fährmann
6264a46212
use 'utcfromtimestamp()' 
'fromtimestamp()' converts its results to the local timezone and causes
problems when running tests on a different machine.
 2019-04-21 16:22:53 +02:00
Mike Fährmann
d84e7c6861
[twitter] extract 'date' metadata (#224) 2019-04-21 15:41:22 +02:00
Mike Fährmann
f2cf1c1d73
use 'text.extract_from()' in a few places 2019-04-21 15:19:20 +02:00
Mike Fährmann
e25ebc4bff
don't disable certificate checks anymore 
Executables generated with PyInstaller auto-include the root certificate
file and certificate checks now work out-of-the-box.
 2019-04-17 13:27:19 +02:00
Mike Fährmann
70be494161
[plurk] add a 'comments' options (#212) 2019-04-14 22:12:46 +02:00
Mike Fährmann
0b2ff406f6
[plurk] add timeline- and post-extractors (#212) 2019-04-14 21:48:38 +02:00