mikf/gallery-dl
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
2,287 Commits 1 Branch 0 Tags
Commit Graph

2287 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Mike Fährmann
c606d0c854
[instagram] update pattern for user profile URLs 
Allow for query parameters and fragments,
for example https://www.instagram.com/instagram/?hl=en
 2020-03-18 22:24:20 +01:00
Mike Fährmann
2530db3f4d
[mangadex] transform 'date' timestamps to datetime objects 2020-03-18 02:19:19 +01:00
Mike Fährmann
ae2a33243b
[newgrounds] catch general Exceptions 2020-03-18 02:17:43 +01:00
Mike Fährmann
32e36d8f02
[sexcom] replace tests 2020-03-17 22:47:45 +01:00
Mike Fährmann
33b42dc847
[nozomi] sort search results (fixes #646) 2020-03-17 22:28:23 +01:00
Mike Fährmann
eaa60a438b
[piczel] fix extraction 
- manually filter by folder_id
- extract data for single posts from embedded JSON, since the
  '/api/gallery/image/<id>' endpoint is no longer available
 2020-03-17 17:12:28 +01:00
Mike Fährmann
5bcc7184c9
[danbooru][e621] increase page limits 2020-03-17 15:53:28 +01:00
Mike Fährmann
90d15e3682
[instagram] use 'itertools.chain()' 2020-03-17 15:52:44 +01:00
Leonardo Taccari
160328d21c
[instagram] Add support for user's saved medias (#644) 
* [instagram] Gracefully handle possible 'HttpErrorPage' in _extract_page()

`HttpErrorPage' is returned in shared_data at least  when not authenticated or
when trying to fetch other users saved medias
(i.e. `instagram.com/<user>/saved/').

Gracefully handle it by returning nothing.

* [instagram] Add support for user's saved medias

(Please note that this need the user to be authenticated and they can
only see their saved media (not other users ones).)

Close #643.

* [instagram] Bump copyright year
 2020-03-16 21:09:14 +01:00
Mike Fährmann
e0b0e8d62a
release version 1.13.2 2020-03-14 20:19:36 +01:00
Mike Fährmann
5b676ea59d
[e621] document username & password support (#640) 2020-03-14 01:15:22 +01:00
Mike Fährmann
1b3ba86110
improve lists in man pages 2020-03-14 01:08:31 +01:00
Mike Fährmann
d3482ace7f
[furaffinity] extract more metadata 
- views
- favorites
- comments
- rating
- fa_category (since 'category' is already in use)
- theme
- species
- gender
- width
- height
 2020-03-13 23:56:55 +01:00
Mike Fährmann
f6c5edb76b
pre-compile regex pattern for remove_html() and split_html() 2020-03-13 23:31:54 +01:00
Mike Fährmann
fdd2dd5136
[kabeuchi] add 'user' extractor (closes #561) 2020-03-13 16:45:42 +01:00
Mike Fährmann
59edcdc822
[hitomi] restore metadata fields from before f33b13a 
... and add a 'metadata' option to disable
visiting the gallery page and extracting data from it
if this is not needed.
 2020-03-12 23:43:41 +01:00
Mike Fährmann
2d5703c493
[twitter] use a simpler data structure to store cookies in cache 
Use a dict with name-value pairs instead of an entire
RequestsCookieJar object.
 2020-03-12 22:02:12 +01:00
Mike Fährmann
87d4f83597
[newgrounds] make post extraction nonfatal 2020-03-10 01:49:59 +01:00
Mike Fährmann
823fbeaae6
[newgrounds] add 'favorite' extractor (#394) 2020-03-10 01:07:09 +01:00
Mike Fährmann
a45fbc38ea
[pixiv] implement 'avatar' option (#595, #623) 2020-03-09 21:18:16 +01:00
Mike Fährmann
a63a376ad2
[mangoxo] fix login 2020-03-08 23:01:51 +01:00
Mike Fährmann
ebc70e87ce
[e621] update to new interface / API endpoints (closes #635) 2020-03-06 21:12:58 +01:00
Mike Fährmann
d1cf7ccdb3
[instagram] add 'post_shortcode' metadata field (#525) 2020-03-06 15:20:32 +01:00
Mike Fährmann
402025c3c3
fix some build issues 
- use 'os.name' to decide between Windows/Linux build
- don't check Windows executable version number,
  since Wine fails to run the executable and causes release.sh to stop
 2020-03-06 15:15:59 +01:00
Mike Fährmann
32df8d06fe
[twitter] add 'bookmark' extractor (closes #625) 2020-03-06 01:20:04 +01:00
Mike Fährmann
3fb41c34c8
[bcy] reduce requests to '/item/detail/<id>' (#613) 
The former implementation would try to use the embedded data from
'/item/detail/' pages for every post, even if that wasn't really
necessary.

This commit also fixes some issues with posts only visible to
logged in users.
 2020-03-04 01:37:51 +01:00
Mike Fährmann
f33b13aacf
[hitomi] simplify metadata extraction 
Use the data from https://ltn.hitomi.la/galleries/<id>.js for both
image URLs and metadata and ignore any gallery or reader pages.

This removes 'artist', 'characters', 'group', and 'parody' metadata
fields since this information is, as for now, only available in
gallery pages.
 2020-03-04 01:22:32 +01:00
Mike Fährmann
115fd2c6f2
"fix" incomplete MIME types (#632) 
e-/exhentai's original image downloads currently send
incomplete/invalid Content-Type headers, "jpg" instead
of "image/jpg" etc, since the last update.
(https://forums.e-hentai.org/index.php?showtopic=236113)

This change prepends any Content-Type value missing a
media type specification with "image/", transforming it
into a valid MIME type.

(A global solution to a local problem, but it shouldn't
 cause any issues anywhere else)
 2020-03-03 21:21:57 +01:00
Mike Fährmann
72122eb9b3
release version 1.13.1 2020-03-01 21:17:29 +01:00
Mike Fährmann
adcd7cb24a
[downloader:http] add another MIME type for '.rar' files (#628) 2020-03-01 20:42:13 +01:00
Mike Fährmann
ce5e2a58fe
[imgbb] update test results 
Image server domain changed from
https://image.ibb.co/ to https://i.ibb.co/
 2020-03-01 20:38:25 +01:00
Mike Fährmann
f117e32910
[danbooru] restore 'popular' functionality 2020-02-29 23:37:53 +01:00
Mike Fährmann
39b48d665b
[hiperdex] use proper name for 'chapter_minor' 2020-02-29 00:18:54 +01:00
Mike Fährmann
8fbbaa54ff
[bcy] fix partial image URLs (#613) 
Images from new posts can have incomplete/partial URLs (1)
without any filename extension when fetching their data from
'/apiv3/user/selfPosts', so now all data gets taken from
'/item/detail/ID' pages.

It is currently unknown how to get the non-watermarked original version
of these images, or if that is possible at all. (2)
Images with a watermark will have their 'filter' metadata field set to
"watermark". For original images this field is an empty string "".

Enabling the 'noop' option will, in addition to the watermarked version,
yield the the '~noop.image' filter version (3),
where 'filter' is set to "noop".

(1) "https://img-bcy-qn.pstatp.com/banciyuan/3ccdff22479c4060aadc86718209b281"
(2) "https://p1-bcy.byteimg.com/img/banciyuan/3ccdff22479c4060aadc86718209b281~tplv-banciyuan-logo-v3:wqnpnLLlhZLlpKfprZTnjotfCuWNiuasoeWFgyAtIEFDR-eIseWlveiAheekvuWMug==.image"
(3) "https://p1-bcy.byteimg.com/img/banciyuan/3ccdff22479c4060aadc86718209b281~noop.image"
 2020-02-28 22:57:10 +01:00
Mike Fährmann
86c00f9e66
[danbooru] move extractor logic from booru.py 2020-02-28 22:53:45 +01:00
Mike Fährmann
1d4a369ea2
update extractor test results 2020-02-27 22:15:40 +01:00
Mike Fährmann
7625912b31
[piczel] improve and update 
- fix tag names
- fix a bug in _pagination()
- parse datetime in 'created_at' as 'date'
- rewrite main loop
- replace user profile test
 2020-02-27 22:13:12 +01:00
Mike Fährmann
ec85bf90de
use context managers in cache.py & add tests 2020-02-25 23:08:47 +01:00
Mike Fährmann
913b8333cc
write DeviantArt refresh-tokens to cache (#616) 
Writing the token is currently disabled by default and must be
enabled with 'extractor.oauth.cache'.

'extractor.deviantart.refresh-token' must be set to '"cache"'
to use the cached token.
 2020-02-25 22:55:11 +01:00
Mike Fährmann
2a4f227e08
warn about expired cookies 2020-02-25 00:34:42 +01:00
Mike Fährmann
34887ae139
fix bugs in DatabaseCacheDecorator.update()/.invalidate() 
- call db.commit() after changes have been made
- remove 'LIMIT 1' from the DELETE statement in invalidate()
    (only available if SQLite3 was compiled with the right flags
     enabled, syntax error otherwise)
 2020-02-23 21:04:36 +01:00
Mike Fährmann
380b693fad
[downloader:http] add more MIME types for '.bmp' files (#621) 2020-02-23 16:51:04 +01:00
Mike Fährmann
4e361b3008
add tests for specific datetime values 2020-02-23 16:48:30 +01:00
Mike Fährmann
80ecb99089
[hitomi] fix extraction 2020-02-22 22:07:21 +01:00
Mike Fährmann
247c9e1416
[vsco] update gallery URL pattern 2020-02-22 21:39:31 +01:00
Mike Fährmann
19ae6f3fc4
update test results 
- twitter:

    Don't test the whole kwdict, only the actual content, since the
    keyword hash changes whenever that user changes his display name.

- khinsider:

    Download host changed
 2020-02-22 03:25:32 +01:00
Mike Fährmann
cc5079c844
[hiperdex] add chapter and manga extractors (closes #606) 2020-02-22 03:09:29 +01:00
Mike Fährmann
64bdec8430
[deviantart] check availability of intermediary URLs (fixes #609) 2020-02-21 03:10:53 +01:00
Mike Fährmann
5607dd3646
[hitomi] follow multiple redirects 2020-02-20 18:22:13 +01:00
Mike Fährmann
6e759fe2f5
reword note for 'extractor.deviantart.refresh-token' (#612) 2020-02-19 23:39:27 +01:00