mikf/gallery-dl
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
1,948 Commits 1 Branch 0 Tags
Commit Graph

141 Commits

Author SHA1 Message Date
Mike Fährmann
38d97f3da6
[deviantart] add debug message about API credentials (#424) 2019-09-25 21:20:55 +02:00
Mike Fährmann
80c2104fb5
[deviantart] fix 429 handling if 'fatal' is False (closes #424) 2019-09-25 21:16:35 +02:00
Mike Fährmann
01bc7adadc
[deviantart] improve journal detection (#419) 
Some journal-like posts are not reported to be journals (isJournal
is set to False), even though they have a textContent field.

https://www.deviantart.com/gliitchlord/art/brashstrokes-812942668
 2019-09-14 22:45:22 +02:00
Mike Fährmann
6e12907de6
[deviantart] improve handling of private deviations (#414) 
- don't try to call '/deviation/metadata' with an empty list of
  deviation ids
- print a warning when detecting private deviations without having
  a 'refresh-token'
 2019-09-10 16:09:03 +02:00
Mike Fährmann
dedea3b4db
[deviantart] fix journal creation (#400) 2019-08-30 18:50:04 +02:00
Mike Fährmann
efb64ad031
[deviantart] generate filenames (#392, #400) 2019-08-29 10:09:21 +02:00
Mike Fährmann
49f6d7176d
[deviantart] restore filenames (#392) 
<title>_by_<user>_<id> --> <title>_by_<user>-<id>
 2019-08-23 22:02:03 +02:00
Mike Fährmann
63daa68d67
[deviantart] improvements (#392) 
- consistent 'filename' entries, at least as far as possible
  - GIFs and SWFs don't have a <title>_by_<artist>_<id> anywhere in
    their metadata
  - Generating <id> (from 'deviationid'?) might be something that needs
    to be figured out, so we can build those filenames ourselves
- better code structure etc.
- tests for videos, archives, and flash animations
 2019-08-23 12:27:19 +02:00
Mike Fährmann
30d6e284b0
[deviantart] use NAPI for artworks and scraps (#392) 
TODO:
- journal downloads
- test for all media types
 2019-08-21 23:56:06 +02:00
Mike Fährmann
423f68f585
[deviantart] fix scraps extraction (closes #376) 2019-08-11 16:06:15 +02:00
Mike Fährmann
f4bc75e854
fix rate limit handling for OAuth APIs (#368) 2019-08-03 13:43:00 +02:00
Mike Fährmann
3957d27d79
[deviantart] add 'quality' option (#369) 2019-08-03 11:40:35 +02:00
Mike Fährmann
5d968412ca
[deviantart] case-insensitive folder name matching (fixes #343) 2019-07-19 18:05:31 +02:00
Mike Fährmann
fdec59f8e2
replace extractor.request() 'expect' argument 
with
- 'fatal': allow 4xx status codes
- 'notfound': raise NotFoundError on 404
 2019-07-05 00:42:16 +02:00
Mike Fährmann
7856e5e7dc
]deviantart] "fix" scraps extraction 2019-06-25 18:18:12 +02:00
Mike Fährmann
76ae9957c2
[deviantart] force legacy version for single deviations 
Let's see how long this works ...

DeviantArt is rolling out a new version of their website, including a
new internal and potentially usable API (rewrite incoming, yay).

The issue with the new layout is that it doesn't include the "old"
UUIDs for single deviations, i.e. mapping a numeric deviation ID to its
UUID counterpart is impossible with the new layout.
 2019-06-20 19:26:15 +02:00
Mike Fährmann
258e8b2060
[deviantart] small code improvements 2019-06-17 19:49:50 +02:00
Mike Fährmann
f5961ac968
[deviantart] download deviations with no 'content' field 
Some deviations (possibly only from sta.sh sources) are downloadable
(i.e. 'is_downloadable' is true and /deviation/download/ works), but
have no 'content' or similar  in their JSON representation.

(fixes #307)
 2019-06-13 21:14:12 +02:00
Mike Fährmann
e05a96db5e
[deviantart] rename 'stash' to 'extra' (#302) 
'stash' is already used as a name for the StashExtractor and therefore
expected to be a dictionary.
 2019-06-10 21:05:25 +02:00
Mike Fährmann
c23bf263fe
[deviantart] rename 'external' to 'stash' (#302) 
restrict extracted URLs to ones from https://sta.sh/...
 2019-06-09 11:16:02 +02:00
Mike Fährmann
2fb85178da
[deviantart] add 'external' option (#302) 
If a description is available, this will extract URLs from the
description text and try to find Extractors for them.
 2019-06-06 18:53:50 +02:00
Mike Fährmann
f85e42cffc
[deviantart] fix --range for deviation & stash extractor 2019-06-06 18:45:10 +02:00
Mike Fährmann
f1893b2b5b
[deviantart] add 'folders' option (#276) 2019-05-30 17:28:12 +02:00
Mike Fährmann
f837ea98cb
[deviantart] don't call 'extend()' on folders (fixes #271) 2019-05-20 16:24:13 +02:00
Mike Fährmann
51e0e92429
[deviantart] fix GIF downloads (#242) 
The "original" download URL for GIF animations is only a preview version
of the original file.
 2019-05-08 15:43:43 +02:00
Mike Fährmann
9544683d56
[deviantart] provide 'date' metadata (#232) 2019-04-29 17:30:24 +02:00
Mike Fährmann
d6ddb74cde
update test results 
- deviantart: 'index' is now an integer
- flickr: image file with lower quality
- paheal: image server name changed
- rule34: post got deleted
 2019-04-12 09:59:48 +02:00
Mike Fährmann
9587aea98f
[deviantart] don't rewrite URLs for newer deviations 
The '/intermediary/' trick stopped working for recently posted
deviations, but it still appears to be functional for older ones.
 2019-04-11 10:37:01 +02:00
Mike Fährmann
5ec55ec4fc
[deviantart] improve URLs for non-downloadable deviations 2019-03-21 15:37:22 +01:00
Mike Fährmann
c7a6b0ed90
[deviantart] add 'metadata' option (#189) 2019-03-21 14:49:42 +01:00
Mike Fährmann
a2af2d2965
adjust cache maxage values 2019-03-14 22:21:49 +01:00
Mike Fährmann
13e0f2a78f
[deviantart] add 'scraps' extractor (closes #168) 2019-03-01 14:13:34 +01:00
Mike Fährmann
c7b8421333
[deviantart] don't match 'www' as a potential username 2019-02-15 16:38:29 +01:00
Mike Fährmann
61741d7333
provide type information for Queue messages 
Child extractors are now directly constructed with Extractor.from_url()
if the extractor class is known beforehand, instead of using
extractor.find() and searching through all possible extractor classes.
 2019-02-12 21:32:32 +01:00
Mike Fährmann
2e516a1e3e
store the full original URL in Extractor.url 2019-02-12 18:46:48 +01:00
Mike Fährmann
4b1880fa5e
propagate 'match' to base extractor constructor 2019-02-11 13:31:10 +01:00
Mike Fährmann
6284731107
simplify extractor constants 
- single strings for URL patterns
- tuples instead of lists for 'directory_fmt' and 'test'
- single-tuple tests where applicable
 2019-02-08 13:45:40 +01:00
Mike Fährmann
fa7fa2f8ff
[deviantart1 update tests] 2019-01-01 15:39:34 +01:00
Mike Fährmann
6c71e9cf5d
[deviantart] add separate 'sta.sh' extractor (#113) 
- supports multiple stashed deviations per page
- explicitly mentions sta.sh support on supportedsites.rst
 2018-12-26 18:56:57 +01:00
Mike Fährmann
7471933d5f
use extractor.request for all other API calls 
- deviantart
- pawoo
- pixiv
- reddit
 2018-12-22 14:42:23 +01:00
Mike Fährmann
7e2d6bcd62
[deviantart] fix original image downloads 2018-11-10 19:16:10 +01:00
Mike Fährmann
d1f3d32eec
[fallenangels] unescape chapter titles 2018-10-20 18:31:26 +02:00
Mike Fährmann
2221cf97ff
implement 'update()' for caches 2018-10-12 22:18:29 +02:00
Mike Fährmann
d8492df51b
[deviantart] extend functionality of 'original' option 2018-10-11 22:11:49 +02:00
Mike Fährmann
1532d1b690
fix 'range' tests and update a few test results 2018-10-08 23:53:58 +02:00
Mike Fährmann
e066f35118
update extractor tests 2018-09-21 11:25:56 +02:00
Mike Fährmann
0232d80cec
[deviantart] convert 'published_time' to int (fixes #108) 
The 'published_time' field (a timestamp) changed from integer to string
and caused journal creation to fail.
 2018-09-13 19:52:01 +02:00
Mike Fährmann
a493fed376
[deviantart] fix journal creation if no 'username' is set 2018-08-31 17:38:12 +02:00
Mike Fährmann
4313c95bc9
improve error message for OAuth2 authentication 2018-08-11 23:54:25 +02:00
Mike Fährmann
5f27cfeff6
[deviantart] remove prefer-public option 
All API requests now always use a public token and only switch to
a private token for pagination results if `refresh-token` is set
and less deviations than requested were returned.
 2018-07-26 19:43:46 +02:00