mikf/gallery-dl
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
2,723 Commits 1 Branch 0 Tags
Commit Graph

87 Commits

Author SHA1 Message Date
Mike Fährmann
9b2e5f72d6
[exhentai] update image URL parsing (#1094) 2020-11-02 15:28:54 +01:00
Mike Fährmann
3ebb174f2c
add missing extractor info when spawning new ones (fixes #1051) 
Not having this information causes the blacklist/whitelist logic to
trigger and prevents things from functioning as intended when using
default settings.

Fixes issues for 8muses, deviantart, exhentai, and mangoxo.
 2020-10-08 14:34:53 +02:00
Mike Fährmann
da87a5fb7e
[exhentai] fix accessing config before main constructor 
bug introduced with 055c32e0

Making 'Extractor.config()' quite  a bit faster is worth the "cost"
of having to set _cfgpath in exhentai constructors, I think.
 2020-09-15 18:09:50 +02:00
Mike Fährmann
a0d916ed41
[exhentai] update wait time before original image download (#978) 
depend on 'wait-max', don't use a hard-coded value
 2020-09-07 23:48:28 +02:00
Mike Fährmann
0f55b8e80a
[exhentai] fix type check from dbbbb21 (#940) 
'bool' is a subclass of 'int', and therefore
'isinstance(self.limits, int)' also returns True when
'self.limits' has a boolean value
 2020-08-30 20:51:22 +02:00
Mike Fährmann
dbbbb21180
[exhentai] add ability to specify custom image limit (#940) 2020-08-17 22:29:20 +02:00
Mike Fährmann
cd9de613a2
[exhentai] adjust image limit costs (#940) 
Each original file costs 10 points per 10^6 bytes,
not 10 per 2^20 == 1048576 bytes.
 2020-08-15 18:19:33 +02:00
Mike Fährmann
ecaecc4064
[exhentai] add 'domain' option (#897) 2020-07-17 22:17:46 +02:00
Mike Fährmann
6b373cb7e2
[exhentai] restrict default directory name length (#545) 2019-12-27 21:29:00 +01:00
Mike Fährmann
4409d00141
embed error messages in StopExtraction exceptions 2019-10-28 16:39:49 +01:00
Mike Fährmann
1848788970
update test results etc 2019-09-08 11:33:35 +02:00
Mike Fährmann
9ae58a6b3e
[exhentai] update image limit checks 
- adjust cost of original images
- delay limit initialization until gallery and first image page have
  been requested and all cookies are available
 2019-09-07 13:29:01 +02:00
Mike Fährmann
fa60109e97
[exhentai] don't use e-hentai.org for exhentai URLs 2019-08-02 21:10:09 +02:00
Mike Fährmann
beb4fab2e6
[exhentai] improve limit and error handling (#360) 
- check image limit before opening the first gallery or image page
- prevent any further exhentai extractors from running after the image
  limit has been reached
 2019-07-30 22:58:35 +02:00
Mike Fährmann
81b35ed3cb
[exhentai] catch more error states (#356, #360) 
- warn on MPV-enabled galleries
- catch parsing errors for gallery pages and image info
- write page content to debug output
 2019-07-29 16:54:31 +02:00
Mike Fährmann
6ce22f606b
[exhentai] update login procedure and tests 
Logging in now follows the natural login flow that also happens in a
browser more closely and collects more cookies than just ipb_member_id
and ipb_pass_hash.

Test URLs have been updated and now point to the e-hentai.org domain.
 2019-07-28 16:51:05 +02:00
Mike Fährmann
dc73d02d87
[exhentai] always use e-hentai.org as domain + set nw cookie 2019-07-28 10:54:17 +02:00
Mike Fährmann
fdec59f8e2
replace extractor.request() 'expect' argument 
with
- 'fatal': allow 4xx status codes
- 'notfound': raise NotFoundError on 404
 2019-07-05 00:42:16 +02:00
Mike Fährmann
1c36e65e9b
[exhentai] choose site version depending on input URL (#278) 
Use e-hentai.org as root and cookiedomain if the input URL is from
e-hentai (or g.e-hentai), use exhentai.org otherwise.
 2019-05-31 15:34:39 +02:00
Mike Fährmann
1f7fa9dc8e
[exhentai] update data extraction code 
- parse 'date' to datetime object
- use 'text.extract_from()'
 2019-05-08 15:44:29 +02:00
Mike Fährmann
5398bfbd69
[exhentai] fix search and favorite extraction 
removes basically all metadata, but that can be compensated for with the
right search query. writing "parsers" for all 4 possible views that have
been introduced in the latest changes is too much of a hassle ...
 2019-03-28 16:22:02 +01:00
Mike Fährmann
a2af2d2965
adjust cache maxage values 2019-03-14 22:21:49 +01:00
Mike Fährmann
5530871b5a
change results of text.nameext_from_url() 
Instead of getting a complete 'filename' from an URL and splitting that
into 'name' and 'extension', the new approach gets rid of the complete
version and renames 'name' to 'filename'. (Using anything other than
{extension} for a filename extension doesn't really work anyway)

Example: "https://example.org/path/filename.ext"

before:
- filename : filename.ext
- name     : filename
- extension: ext

now:
- filename : filename
- extension: ext
 2019-02-14 16:07:17 +01:00
Mike Fährmann
61741d7333
provide type information for Queue messages 
Child extractors are now directly constructed with Extractor.from_url()
if the extractor class is known beforehand, instead of using
extractor.find() and searching through all possible extractor classes.
 2019-02-12 21:32:32 +01:00
Mike Fährmann
2e516a1e3e
store the full original URL in Extractor.url 2019-02-12 18:46:48 +01:00
Mike Fährmann
4b1880fa5e
propagate 'match' to base extractor constructor 2019-02-11 13:31:10 +01:00
Mike Fährmann
6284731107
simplify extractor constants 
- single strings for URL patterns
- tuples instead of lists for 'directory_fmt' and 'test'
- single-tuple tests where applicable
 2019-02-08 13:45:40 +01:00
Mike Fährmann
dd358b4564
improve cookie handling during logins 2019-01-30 17:09:32 +01:00
Mike Fährmann
134487ffb0
[exhentai] stop extraction if image limit is exceeded (#141) 
can be turned off with the `exhentai.limits' option
 2019-01-26 18:40:39 +01:00
Mike Fährmann
e868fb4393
[exhentai] improve gallery extraction 
- match image page URLs and extract galleries from that point onward
- add a few more metadata entries: 'parent', 'visible', 'cost'
 2019-01-26 18:23:25 +01:00
Mike Fährmann
2ffc105887
[exhentai] extract tag metadata 2019-01-15 18:08:17 +01:00
Mike Fährmann
2801a0d997
[exhentai] skip "Content Warning" page when not logged in 
(closes #97)
 2018-08-16 09:17:22 +02:00
Mike Fährmann
b8c97d2295
use 'extractor.request()' for more HTTP requests 2018-06-25 23:40:59 +02:00
Mike Fährmann
017188d268
improve extractor.request() 
Replace the 'fatal' parameter with 'expect', which is a list/range
of HTTP status codes >= 400 that should also be accepted.
 2018-06-18 16:29:56 +02:00
Mike Fährmann
7a58151566
fix util.parse_bytes invocations 
(should be text.parse_bytes)
 2018-05-10 22:07:55 +02:00
Mike Fährmann
cc36f88586
rename safe_int to parse_int; move parse_* to text module 2018-04-20 14:53:21 +02:00
Mike Fährmann
64d7c85b55
[exhentai] improve metadata 
- add 'width', 'height' and 'size' (in bytes) for each image
- change the former 'size' and 'size_units' into 'gallery_size'
 2018-04-03 18:59:53 +02:00
Mike Fährmann
52d41c41e7
[exhentai] add extractor for favorited galleries 2018-03-27 18:58:42 +02:00
Mike Fährmann
63cc2599c4
[exhentai] add extractor for search results 2018-03-27 16:50:47 +02:00
Mike Fährmann
34873dbd90
set 'archive_fmt' values 
These are going to be used to create an unique id for each image.
 2018-02-01 15:30:49 +01:00
Mike Fährmann
e6814aebe2
add 'extractor.*.user-agent' config option 2017-11-15 14:01:33 +01:00
Mike Fährmann
9fc1d0c901
implement and use 'util.safe_int()' 
same as Python's 'int()', except it doesn't raise any exceptions and
accepts a default value
 2017-09-24 15:59:25 +02:00
Mike Fährmann
6f30cf4c64
change keyword names to valid Python identifiers 
This commit mostly replaces all minus-signs ('-') in keyword names with
underscores ('_') to allow them to be used in filter-expressions. For
example 'gallery-id' got renamed to 'gallery_id'.

(It is theoretically possible to access any variable, regardless of its
name, with 'locals()["NAME"]', but that seems a bit too convoluted if
just 'NAME' could be enough)
 2017-09-10 22:20:47 +02:00
Mike Fährmann
c0755a4d5e
[exhentai] revert login-method to its old version (#37) 
Additional cookies don't seem to help and have to be manually set
anyway. The older method is more likely to succeed, so I'd rather
use this one.
 2017-08-29 22:10:38 +02:00
Mike Fährmann
3ee39ffd93
[exhentai] update login procedure (#37) 
This new version behaves pretty much exactly like a browser would and
caches all cookies sent to it and not just "ipb_member_id" and
"ipb_pass_hash".
 2017-08-28 21:03:32 +02:00
Mike Fährmann
2d0dfe9d56
[exhenai] init headers before login and detect sadpanda 
- also debug-logs html after failed login
- #37
 2017-08-25 16:44:59 +02:00
Mike Fährmann
915a0137de
improve 'extractor.request' 
- add 'fatal' argument
- improve internal logic and flow
- raise known exception on error
- update exception hierarchy
 2017-08-05 16:11:46 +02:00
Mike Fährmann
7aa9fa796a
code cleanup and fixes 2017-07-25 14:59:41 +02:00
Mike Fährmann
808f67ba7d
use 'cookiedomain' for cookies set by object-config-values 
otherwise these cookies would not be picked up by the
_check_cookies() method.
 2017-07-22 15:43:35 +02:00
Mike Fährmann
0610ae5000
skip login if cookies are present 2017-07-17 10:33:36 +02:00