Mike Fährmann
f2000a69aa
implement 'image-unique' and 'chapter-unique' options ( #303 )
...
The default value for both is 'false', i.e. duplicate URLs are NOT
ignored.
The previous behavior was to always ignore duplicate URLs to make
'--abort-on-skip' work properly when new images where added to the
beginning of a collection while gallery-dl is running.
2019-06-29 22:50:17 +02:00
Mike Fährmann
40da44b17f
Merge branch 'v1.9.0'
2019-06-29 15:39:52 +02:00
Mike Fährmann
9a216a6c6c
release version 1.8.7
2019-06-28 21:04:00 +02:00
Mike Fährmann
7a99e85943
[kissmanga] fix download URLs and file extensions
...
The current Blogspot image URLs hosted on Kissmanga end with an
"invalid" query parameter (/000.png&upx=...), which doesn't get
recognized by 'spliturl()' and 'parseurl()' as such and gets therefore
included in the 'extension' field from 'text.nameext_from_url()'.
2019-06-28 20:34:43 +02:00
Mike Fährmann
055102431f
[hitomi] handle Game CG galleries with scenes ( fixes #321 )
2019-06-27 20:25:40 +02:00
Mike Fährmann
a9c89085fb
[instagram] implement login support ( #195 )
2019-06-26 23:58:47 +02:00
Mike Fährmann
f1b0c2bf5c
[downloader:ytdl] forward cookies to youtube-dl
...
to be able to download private videos from Twitter, Instagram, etc.
2019-06-26 19:32:07 +02:00
Mike Fährmann
7856e5e7dc
]deviantart] "fix" scraps extraction
2019-06-25 18:18:12 +02:00
Mike Fährmann
b1985d6579
test default format strings during extractor result tests
...
A missing value or an invalid "syntax" for a format replacement field
will raise an exception.
2019-06-25 18:12:32 +02:00
Mike Fährmann
082cb24acd
[pururin] fix extraction
...
Missing metadata information would lead to unnecessary exceptions.
2019-06-24 22:27:50 +02:00
Mike Fährmann
98554cbab8
[mangoxo] fix login
2019-06-24 21:57:17 +02:00
Mike Fährmann
108963d138
[imagefap] include Referer headers
2019-06-24 21:31:29 +02:00
Mike Fährmann
e314621366
[nsfwalbum] fix default directory_fmt ( #287 )
2019-06-24 18:29:54 +02:00
Mike Fährmann
95b1e4c3c0
implement R<old>/<new>/ format option ( #318 )
2019-06-23 22:45:44 +02:00
Mike Fährmann
18a1f8c6cd
[vanillarock] add post and tag extractors ( closes #254 )
2019-06-23 22:45:36 +02:00
Mike Fährmann
f0c5093812
[nsfwalbum] add album extractor ( closes #287 )
2019-06-23 22:45:07 +02:00
Mike Fährmann
15e4ddf46d
implement custom logging formatter
...
supports custom log message formats for each loglevel and, by
extension, custom ANSI codes and colors for errors and warnings
(#304 )
2019-06-21 20:17:58 +02:00
Mike Fährmann
61e413d85d
[hentaifoundry] stop disabling IPv6 addresses
...
The rogue address mentioned in a138d58 is no longer included in the DNS
results for www.hentai-foundry.com.
2019-06-21 20:03:14 +02:00
Mike Fährmann
76ae9957c2
[deviantart] force legacy version for single deviations
...
Let's see how long this works ...
DeviantArt is rolling out a new version of their website, including a
new internal and potentially usable API (rewrite incoming, yay).
The issue with the new layout is that it doesn't include the "old"
UUIDs for single deviations, i.e. mapping a numeric deviation ID to its
UUID counterpart is impossible with the new layout.
2019-06-20 19:26:15 +02:00
Mike Fährmann
70713f0f28
fix extractor result tests
2019-06-20 18:12:36 +02:00
Mike Fährmann
db3f52881a
add 'mtime' option
2019-06-20 17:19:44 +02:00
Mike Fährmann
ee4d7c3d89
update downloader.find() and related code
...
Instead of replacing 'https' with 'http' for every URL in
'get_downloader()', this now only happens once during downloader
initialization. Also unit tests.
2019-06-20 16:59:44 +02:00
Mike Fährmann
f4ba98771d
use Last-Modified header to set file modification time
...
(#236 , #277 )
2019-06-19 23:16:32 +02:00
Mike Fährmann
179d112083
[downloader] overhaul http and text modules
...
Get rid of the modular structure and simplify/specialize those modules.
2019-06-19 22:56:11 +02:00
Mike Fährmann
a01f99728c
[postprocessor:zip] delete empty archives when done ( #316 )
2019-06-19 18:14:33 +02:00
Mike Fährmann
520c8ba106
[hentaicafe] extract 'tags' and 'artist' metadata ( closes #238 )
...
These metadata fields will only be filled in when using a top-level
URL, because that's the only place this information is available. Using
a Foolslide URL (1) will leave these fields empty.
(1) https://hentai.cafe/manga/read/.../en/0/1/ "
2019-06-18 14:30:26 +02:00
Mike Fährmann
b51baa9a4b
[hitomi] fix empty language detection; parse datetime
2019-06-17 20:02:58 +02:00
Mike Fährmann
258e8b2060
[deviantart] small code improvements
2019-06-17 19:49:50 +02:00
Mike Fährmann
a77340c647
[keenspot] fix extraction for "TwoKinds"
2019-06-17 19:49:39 +02:00
Mike Fährmann
03e6876fbe
[instagram] provide 'description' metadata ( #310 )
2019-06-16 21:54:01 +02:00
Mike Fährmann
b171befa87
implement 'parse_unicode_escapes()'
2019-06-16 21:47:24 +02:00
Mike Fährmann
3a36a0fa1e
release version 1.8.6
2019-06-14 21:11:58 +02:00
Mike Fährmann
ec3e8601f1
[slickpic] add user extractor ( #249 )
2019-06-14 18:55:56 +02:00
Mike Fährmann
97ef416218
[8muses] support multi-page listings ( #305 )
2019-06-14 18:48:22 +02:00
Mike Fährmann
f5961ac968
[deviantart] download deviations with no 'content' field
...
Some deviations (possibly only from sta.sh sources) are downloadable
(i.e. 'is_downloadable' is true and /deviation/download/ works), but
have no 'content' or similar in their JSON representation.
(fixes #307 )
2019-06-13 21:14:12 +02:00
Mike Fährmann
4e07f99e3e
[mangoxo] change token message level to debug
...
The login page currently doesn't provide and require a login token
(logging in works without a token), so printing a warning during
each login is unnecessary.
2019-06-13 21:09:11 +02:00
Mike Fährmann
d997c10320
[8muses] add album extractor ( #305 )
2019-06-10 22:17:46 +02:00
Mike Fährmann
e05a96db5e
[deviantart] rename 'stash' to 'extra' ( #302 )
...
'stash' is already used as a name for the StashExtractor and therefore
expected to be a dictionary.
2019-06-10 21:05:25 +02:00
Mike Fährmann
2184e3a86b
[slickpic] add album extractor ( #249 )
2019-06-09 21:59:22 +02:00
Mike Fährmann
c23bf263fe
[deviantart] rename 'external' to 'stash' ( #302 )
...
restrict extracted URLs to ones from https://sta.sh/...
2019-06-09 11:16:02 +02:00
Mike Fährmann
c73c2cda50
[pornhub] add gallery & user extractor ( #282 )
2019-06-07 16:31:20 +02:00
Mike Fährmann
7c6cb908f9
[xhamster] update test results
2019-06-07 16:28:49 +02:00
Mike Fährmann
035b850e82
update postprocessor entries in example config
...
- use whitelists
- add ugoira example (#299 )
2019-06-07 13:47:02 +02:00
Mike Fährmann
2fb85178da
[deviantart] add 'external' option ( #302 )
...
If a description is available, this will extract URLs from the
description text and try to find Extractors for them.
2019-06-06 18:53:50 +02:00
Mike Fährmann
f85e42cffc
[deviantart] fix --range for deviation & stash extractor
2019-06-06 18:45:10 +02:00
Mike Fährmann
40c7eb3424
[livedoor] improve extraction ( fixes #301 )
2019-06-06 15:22:27 +02:00
Mike Fährmann
62335b9015
[paheal] adjust test results
2019-06-05 11:42:01 +02:00
Mike Fährmann
aa1ca4ed35
[shopify] skip deleted products ( #175 )
...
Product pages which return a 4xx status code will now be skipped instead
of raising an exception.
2019-06-05 11:40:54 +02:00
Mike Fährmann
096009367b
[xhamster] add gallery & user extractor ( #281 )
2019-06-05 11:11:51 +02:00
Mike Fährmann
208202b962
[tumblr] improve error handling ( #297 )
...
In some cases Tumblr's API responds with an HTML document.
Trying to decode it as JSON would raise an uncaught exception.
2019-06-04 14:02:17 +02:00
