mikf/gallery-dl
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
1,721 Commits 1 Branch 0 Tags
Commit Graph

1232 Commits

Author SHA1 Message Date
Mike Fährmann
399e8e965a
also update urllib3's cipher list for versions >= 1.25 2019-05-21 23:02:20 +02:00
Mike Fährmann
f837ea98cb
[deviantart] don't call 'extend()' on folders (fixes #271) 2019-05-20 16:24:13 +02:00
Mike Fährmann
bb32a2d490
[patreon] use file extensions from original filenames (#268) 2019-05-20 15:46:59 +02:00
Mike Fährmann
efa805c5d7
[sankaku] update pagination end condition (fixes #265) 
Pagination over popular listings (`date:...+order:popular") never
terminates, not even on the site itself, and at some point returns the
same results over and over again.
 2019-05-20 15:46:06 +02:00
Mike Fährmann
a4ba34c835
[booru] prevent crash when no tags are present (#259) 2019-05-17 19:32:53 +02:00
Mike Fährmann
ca3bad1779
[patreon] small fixes and adjustments (#226) 
- fix datetime parsing
- rename 'user' to 'creator'
- convert 'id' to integer
- improve tests
 2019-05-17 19:32:41 +02:00
Leonardo Taccari
fb09dd962a [instagram] Fix extraction after `rhx_gis' field removal 2019-05-17 18:17:42 +02:00
Mike Fährmann
7a14aaed7d
[luscious] fix extraction 2019-05-17 10:48:47 +02:00
Mike Fährmann
e82cadac61
[patreon] add extractors (#226) 2019-05-17 10:47:22 +02:00
Mike Fährmann
4891f4a328
[hentainexus] add search extractor (#256) 2019-05-16 23:55:30 +02:00
Mike Fährmann
c02f12ce2f
avoid Cloudflare CAPTCHAs for OpenSSL < 1.1.1 
see https://github.com/Anorov/cloudflare-scrape/pull/242
 2019-05-15 12:25:20 +02:00
Mike Fährmann
0b4be57a10
[sankaku] fix error when no tags available (closes #259) 
[ci skip]
 2019-05-14 23:40:07 +02:00
Mike Fährmann
9890bfdf23
[flickr] improve code and metadata 
- simplify pagination
- add more metadata and slightly change its structure
  - convert suitable values to int or list
  - move keys from ["photo"] to the base level
- proper video support (#246)
- rename method and variable names to better fit with other extractors
 2019-05-14 22:10:50 +02:00
Mike Fährmann
aa8e366b90
[luscious] fix tag extraction 2019-05-14 17:35:52 +02:00
Mike Fährmann
ba8eb1ffec
[hentainexus] add gallery extractor (#256) 2019-05-12 23:59:41 +02:00
Mike Fährmann
b1db194c14
[reactor] update and improve 
- split 'tags' into a list
- parse 'date' into a datetime object
- fix webm/mp4 URLs
 2019-05-09 23:24:49 +02:00
Mike Fährmann
b0e85a42e3
apply workaround from 4736912 in parse_datetime() itself 2019-05-09 21:53:17 +02:00
Mike Fährmann
8de5866fd2
[twitter] replace unit test URLs 
https://twitter.com/PicturesEarth was deleted
 2019-05-09 10:17:55 +02:00
Mike Fährmann
74c7304c6b
[newgrounds] extract 'date', 'favorites', and 'score' 2019-05-08 18:09:17 +02:00
Mike Fährmann
4736912d4e
[pixiv] work around strptime limitations in Python < 3.7 
"%z" doesn't allow a colon separator in older Python versions:
    - "+0900" is OK
    - "+09:00" raises an exception
 2019-05-08 18:08:03 +02:00
Mike Fährmann
1f7fa9dc8e
[exhentai] update data extraction code 
- parse 'date' to datetime object
- use 'text.extract_from()'
 2019-05-08 15:44:29 +02:00
Mike Fährmann
80fdb11508
[pixiv] add 'date' metadata field (closes #248) 2019-05-08 15:43:59 +02:00
Mike Fährmann
049e9fd6ce
[twitter] fix pagination end condition 
Some timelines would cause an endless loop because 'has_more_items' is
always True, even if it would return the same list of tweets over and
over again.
 2019-05-08 15:43:59 +02:00
Mike Fährmann
51e0e92429
[deviantart] fix GIF downloads (#242) 
The "original" download URL for GIF animations is only a preview version
of the original file.
 2019-05-08 15:43:43 +02:00
Leonardo Taccari
f347d2d152 [instagram] Fix for missing edge_media_to_comment' field and add date' metadata (#250) 
* [instagram] Remove no longer always present `comments' field

`edge_media_to_comment' is no longer always present in the response
(also for the same media sometimes is present and sometimes is not present).

* [instagram] Add `date' metadata
 2019-05-08 15:42:58 +02:00
Mike Fährmann
5fd94c6b83
import urllib3 from requests.packages 2019-05-04 22:28:07 +02:00
Mike Fährmann
35f343206c
update default SSL cipher list in urllib3 < 1.25 
Cloudflare now also checks the client's SSL/TLS cipher capabilities and
produces a 403: Forbidden response with CAPTCHA if they are insufficient.

This commit replaces the default cipher list in urllib3 < 1.25 with the
one from 1.25 (1), which doesn't cause problems as long as the client
platform actually supports these ciphers. On some platforms (tested with
Python 3.4 on Linux and Python 3.7 on an outdated Windows 7 VM) it is
necessary to install pyOpenSSL to get everything to work.

Explicitly setting a minimum/maximum version for urllib3 is also no
longer necessary and installing gallery-dl will therefore not pull a
incompatible urllib3 version (#229)

Fixes the "403: Forbidden" error on Artstation (#227)

(1) 0cedb3b0f1
 2019-05-03 22:40:04 +02:00
Mike Fährmann
fc5e4f2b21
[hitomi] simplify data extraction code 2019-05-01 11:14:21 +02:00
Mike Fährmann
2756cc8dde
[hitomi] set Referer header (fixes #239) 2019-05-01 10:56:00 +02:00
Mike Fährmann
dcc1592dbf
[twitter] add fallback URLs (#237) 2019-04-30 15:57:21 +02:00
Mike Fährmann
1c665fd4bd
[mangoxo] fix login 2019-04-30 15:57:06 +02:00
Mike Fährmann
add7e693d0
[tumblr] provide parsed 'date' metadata (#232) 2019-04-29 17:30:42 +02:00
Mike Fährmann
9544683d56
[deviantart] provide 'date' metadata (#232) 2019-04-29 17:30:24 +02:00
Mike Fährmann
0d7e8be987
[dynastyscans] simplify image extractor 2019-04-27 13:24:30 +02:00
Mike Fährmann
9aa0bb5afe
[dynastyscans] encode "[]" in search queries 
urllib3 1.25 classifies URLs with unencoded "[" or "]" as invalid
and raises an exception
 2019-04-27 13:22:40 +02:00
Mike Fährmann
fe849382d8
[komikcast] improve extraction 2019-04-26 15:14:10 +02:00
Mike Fährmann
0318c610dc
[sexcom] add extractor for search results (#147) 2019-04-24 22:10:01 +02:00
Mike Fährmann
a247c94c34
[sexcom] add pin and board extractors (#147) 2019-04-24 22:09:19 +02:00
Mike Fährmann
6264a46212
use 'utcfromtimestamp()' 
'fromtimestamp()' converts its results to the local timezone and causes
problems when running tests on a different machine.
 2019-04-21 16:22:53 +02:00
Mike Fährmann
d84e7c6861
[twitter] extract 'date' metadata (#224) 2019-04-21 15:41:22 +02:00
Mike Fährmann
f2cf1c1d73
use 'text.extract_from()' in a few places 2019-04-21 15:19:20 +02:00
Mike Fährmann
e25ebc4bff
don't disable certificate checks anymore 
Executables generated with PyInstaller auto-include the root certificate
file and certificate checks now work out-of-the-box.
 2019-04-17 13:27:19 +02:00
Mike Fährmann
70be494161
[plurk] add a 'comments' options (#212) 2019-04-14 22:12:46 +02:00
Mike Fährmann
0b2ff406f6
[plurk] add timeline- and post-extractors (#212) 2019-04-14 21:48:38 +02:00
Mike Fährmann
d6ddb74cde
update test results 
- deviantart: 'index' is now an integer
- flickr: image file with lower quality
- paheal: image server name changed
- rule34: post got deleted
 2019-04-12 09:59:48 +02:00
Mike Fährmann
87b0929bec
Revert "[flickr] restore image quality" 
This reverts commit 3f513f10564a10ece8650e64d2233d8482fc14c7.

Both live.staticflickr and farmN.staticflickr servers now produce the
same image file with a lower overall quality than before this change in
Flickr's end.
 2019-04-11 20:31:05 +02:00
Mike Fährmann
e7cd5510d5
[pixnet] add extractors (closes #177) 
for:
- users/blogs: http://albertayu773.pixnet.net/
- folders: https://albertayu773.pixnet.net/album/folder/1405768
- sets   : https://albertayu773.pixnet.net/album/set/15078995
- photos : https://albertayu773.pixnet.net/album/photo/159443828
 2019-04-11 19:27:02 +02:00
Mike Fährmann
155e1faeaf
[imagebam] support galleries with >100 images (fixes #219) 2019-04-11 19:12:27 +02:00
Mike Fährmann
9587aea98f
[deviantart] don't rewrite URLs for newer deviations 
The '/intermediary/' trick stopped working for recently posted
deviations, but it still appears to be functional for older ones.
 2019-04-11 10:37:01 +02:00
Mike Fährmann
f2220938cb
[mangoxo] improve channel extraction (#184) 2019-04-10 18:56:21 +02:00