mikf/gallery-dl
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
2,651 Commits 1 Branch 0 Tags
Commit Graph

1910 Commits

Author SHA1 Message Date
Mike Fährmann
5b844a72b7
[newgrounds] handle embeds without scheme (#1033) 2020-10-15 15:13:54 +02:00
kurumigi
7e0e872f4f
[seiga] Add metadata for single image downloads (#1063) 
* [seiga] Support image metadata.

* [seiga] Update test data.

* [seiga] Fix cookie check.

* [test_cookies] [seiga] Fit test_cookies.py to the last commit.
 2020-10-15 15:13:27 +02:00
Zanny
3ec60e894a
[weasyl] api-key authentication (#1057) 
* [weasyl] support api keys

* [weasyl] document api-key authentication

* [weasyl] usernames can contain ~
 2020-10-15 15:12:09 +02:00
Mike Fährmann
844793847c
update extractor test results 2020-10-11 18:15:41 +02:00
Mike Fährmann
ddd6840509
[behance] fix 'collection' extraction 2020-10-11 18:15:41 +02:00
Mike Fährmann
c5e3971b18
[newgrounds] extract image embeds (closes #1033) 2020-10-11 18:15:40 +02:00
dawidsowa
43b156fb40
[reactor] match URLs without subdomain (#1053) 2020-10-11 18:15:06 +02:00
Mike Fährmann
3ebb174f2c
add missing extractor info when spawning new ones (fixes #1051) 
Not having this information causes the blacklist/whitelist logic to
trigger and prevents things from functioning as intended when using
default settings.

Fixes issues for 8muses, deviantart, exhentai, and mangoxo.
 2020-10-08 14:34:53 +02:00
Mike Fährmann
f9c1684af7
[newgrounds] restore original video URLs (#1042) 2020-10-07 22:53:53 +02:00
Mike Fährmann
73373c06ec
[weibo] handle posts with more than 9 images (closes #926) 
Responses from '/api/container/getIndex' don't list more than
9 images per 'status' object, but the embedded JSON from a
'/detail/<ID>' page does.
 2020-10-06 18:16:08 +02:00
Mike Fährmann
dd1e545597
[hentaifoundry] rename GalleryExtractor to PicturesExtractor 2020-10-04 22:53:23 +02:00
Mike Fährmann
c874071f5a
[kissmanga] remove module 2020-10-04 22:46:41 +02:00
Mike Fährmann
93e04bf9a9
[500px] update query hashes 2020-10-03 19:25:28 +02:00
Mike Fährmann
844502cad5
update extractor test results 2020-10-03 19:24:19 +02:00
Mike Fährmann
fad7748b6b
[xvideos] fix 'title' extraction 2020-10-01 22:04:14 +02:00
Mike Fährmann
5b927c15df
[newgrounds] fix video extraction (closes #1042) 2020-10-01 20:14:16 +02:00
Mike Fährmann
bdc6c8f074
improve message for 'oauth:deviantart' etc (closes #989) 2020-09-29 21:25:24 +02:00
Mike Fährmann
430b6d6e2e
[twitter] extend 'retweets' option (closes #1026) 
Setting 'retweets' to '"original"' will use metadata from the
original retweeted Tweets, and not from the Retweet entry.
 2020-09-28 23:03:35 +02:00
Mike Fährmann
b9bdd2c564
[hentaifoundry] add support for stories (closes #734) 2020-09-27 02:27:40 +02:00
Mike Fährmann
9a9d1924d8
[hentaicafe] add 'manga_id' metadata field (closes #1036) 
This field is only available when using a non-foolslide URL
like '/hc.fyi/9874' or '/hazuki-yuuto-summer-blues/'
 2020-09-26 14:34:48 +02:00
Mike Fährmann
cc4ac80302
[weasyl] add 'favorite' extractor (#1032) 2020-09-26 13:09:03 +02:00
Mike Fährmann
e9cc719497
[weasyl] update and simplify 
- simplify 'pattern' regexps
- parse 'posted_at' as 'date'
- use unaltered 'title' ({title!l:R /_/} to lowercase and replace spaces)
 2020-09-26 02:10:45 +02:00
Mike Fährmann
6514312126
[nijie] add 'include' option (closes #1018) 2020-09-25 18:18:35 +02:00
Mike Fährmann
0d43456323
[hentaifoundry] add 'include' option 2020-09-25 18:18:03 +02:00
Zanny
ebb7737b9b
Weasyl Extractor (#977) 
* weasyl extractor

* @kattjevfel suggested changes

* @mikf changes
 2020-09-25 15:18:21 +02:00
Mike Fährmann
aeb0d32333
[twitter] improve twitpic extraction (fixes #1019) 
- ignore twitpic.com/photos/… URLs
- ignore empty image URLs
 2020-09-22 22:22:35 +02:00
Mike Fährmann
7cd383c0f9
update extractor test results 2020-09-20 21:54:39 +02:00
Mike Fährmann
1e313d5b84
implement 'sleep-request' option 2020-09-20 20:28:17 +02:00
Mike Fährmann
c43b3894be
[myhentaigallery] update and fix extraction (#1001) 
- extract more metadata
- match "/show/" URLs
- complete test results
- fix missing images for lines starting with " <img"
- fix missing comma in supportedsites.py
 2020-09-17 18:14:23 +02:00
choeronline
05b9ac8d37
[myhentaigallery] add extractor (#1001) 
* adds support for myhentaigallery

* fixes linting issues in myhentaigallery extractor
 2020-09-17 17:32:54 +02:00
Mike Fährmann
2626629117
[danbooru] handle posts without 'id' (fixes #1004) 2020-09-16 21:35:27 +02:00
Mike Fährmann
cc1fb0b4ea
[500px] update query hash 2020-09-16 01:26:31 +02:00
Mike Fährmann
da87a5fb7e
[exhentai] fix accessing config before main constructor 
bug introduced with 055c32e0

Making 'Extractor.config()' quite  a bit faster is worth the "cost"
of having to set _cfgpath in exhentai constructors, I think.
 2020-09-15 18:09:50 +02:00
Mike Fährmann
f5b7ae01c1
update extractor test results 2020-09-15 18:07:08 +02:00
Mike Fährmann
136df52d1f
[deviantart] support watchers-only/paid deviations (#995) 2020-09-15 16:03:46 +02:00
Mike Fährmann
055c32e0f7
precompute extractor config paths 2020-09-14 22:06:54 +02:00
Mike Fährmann
231dd4c800
accumulate postprocessor objects (#994) 
Instead of one 'postprocessors' setting overwriting all others lower
in the hierarchy, all postprocessors along the config path will now
get collected into one big list.

For example '--mtime-from-date' will therefore no longer cause
other postprocessor settings in a config file to get ignored.
 2020-09-14 21:51:55 +02:00
Mike Fährmann
3108e85b89
[worldthree] remove extractors 
http://www.slide.world-three.org/ hasn't been accessible for a long time.
 2020-09-11 18:12:57 +02:00
Mike Fährmann
8fed3eb8cb
[jaiminisbox] remove extractors 
https://jaiminisbox.com/post.html
 2020-09-11 18:09:35 +02:00
Mike Fährmann
dcf3ad7eef
[furaffinity] update download URL extraction (fixes #988) 
support the new 'd2.facdn.net' subdomain
 2020-09-11 13:23:57 +02:00
Mike Fährmann
3918b69677
remove 'extractor.blacklist' context manager 2020-09-11 13:17:35 +02:00
Mike Fährmann
2b8d57f0ab
[twitter] support '/intent/user?user_id=…' URLs (#980) 2020-09-08 23:17:50 +02:00
Mike Fährmann
a3b473bd2f
[twitter] support specifying users by ID (#980) 
by using 'id:…' as their screen name, i.e.
https://www.twitter.com/id:2976459548/media
instead of
https://twitter.com/supernaturepics/media

The user ID can, for example, be obtained from the output of
$ gallery-dl -j --range 1 https://twitter.com/<screen-name>
 2020-09-08 22:56:52 +02:00
Mike Fährmann
a0d916ed41
[exhentai] update wait time before original image download (#978) 
depend on 'wait-max', don't use a hard-coded value
 2020-09-07 23:48:28 +02:00
Mike Fährmann
f6fd449b59
reduce wait time growth rate from exponential to linear 
Waiting for 2**N seconds after each error grows too fast.
Simply waiting N seconds seems far more reasonable.
 2020-09-06 22:38:25 +02:00
Mike Fährmann
bc48514d84
[aryion] get post ID via gallery-item (fixes #981, closes #982) 
this even works when fetching post IDs from '/latest.php?id='
 2020-09-06 22:17:23 +02:00
Mike Fährmann
799ca07fc8
[imgur] update 
- fix image/album detection for galleries
- use new API endpoints for image/album data
 2020-09-06 21:11:32 +02:00
Mike Fährmann
7876a03ece
[tumblr] create directories for each post (fixes #965) 
This changes the identifiers for directory format string fields.
Everything blog related is now inside a 'blog' object
and not at the "base level" anymore.

E.g. '{name}' for directories is now '{blog[name]}'
(or '{blog_name}', since that is also available)
 2020-08-31 21:58:20 +02:00
Mike Fährmann
d50f3b333a
update extractor test results 2020-08-30 20:55:22 +02:00
Mike Fährmann
0f55b8e80a
[exhentai] fix type check from dbbbb21 (#940) 
'bool' is a subclass of 'int', and therefore
'isinstance(self.limits, int)' also returns True when
'self.limits' has a boolean value
 2020-08-30 20:51:22 +02:00