Mike Fährmann
885bd4cbe2
[readcomiconline] extract comic metadata
2017-09-18 19:18:24 +02:00
Mike Fährmann
cebf800a7f
[foolfuuka] add support for more sites ( #18 )
...
- https://arch.b4k.co
- https://archive.whatisthisimnotgoodwithcomputers.com
- https://archive.yeet.net
Notes:
- The name "whatisthisimnotgoodwithcomputers" is way too long ...
- archive.yeet.net is out of date and also blocked by 4chan servers
- newest threads are 2 weeks old
- using "https://archive.yeet.net " as Referer header results in
"403 Forbidden" when accessing 4chan
2017-09-16 21:36:16 +02:00
Mike Fährmann
84d4450410
[fallenangels] extract manga metadata
2017-09-15 20:51:40 +02:00
Mike Fährmann
f32b1a0292
[imgyt] fix extraction
2017-09-14 15:04:32 +02:00
Mike Fährmann
4ad903b797
[warosu] fix extraction
2017-09-14 14:57:40 +02:00
Mike Fährmann
b84f48dfa5
[batoto] extract manga metadata
2017-09-14 14:55:57 +02:00
Mike Fährmann
4ceb176c6b
[foolslide] extract manga metadata
...
enables chapter filtering for
- https://kobato.hologfx.com/
- https://jaiminisbox.com/
- https://reader.kireicake.com/
- https://powermanga.org/
- https://reader.seaotterscans.com/
- http://sensescans.com/
- http://www.slide.world-three.org/
2017-09-12 16:44:38 +02:00
Mike Fährmann
24e5f154a4
[deviantart] update test results
...
API responses now contain proper https:// URLs and their image download
server is now "orig00.deviantart.net" for all images.
2017-09-12 16:38:57 +02:00
Mike Fährmann
0dedbe759c
enable '--chapter-filter'
...
The same filter infrastructure that can be applied to image URLS now
also works for manga chapters and other delegated URLs.
TODO: actually provide any metadata (currently supported is only
deviantart and imagefap).
2017-09-12 16:19:00 +02:00
Mike Fährmann
31cd5b1c1d
[luscious] detect high-load responses
2017-09-12 15:46:21 +02:00
Mike Fährmann
470bbe9d8c
fix smaller stuff
...
- change filename option in example config file
- adapt default filename format for mangafox
- remove unnecessary newline
[skip ci]
2017-09-11 17:07:29 +02:00
Mike Fährmann
6f30cf4c64
change keyword names to valid Python identifiers
...
This commit mostly replaces all minus-signs ('-') in keyword names with
underscores ('_') to allow them to be used in filter-expressions. For
example 'gallery-id' got renamed to 'gallery_id'.
(It is theoretically possible to access any variable, regardless of its
name, with 'locals()["NAME"]', but that seems a bit too convoluted if
just 'NAME' could be enough)
2017-09-10 22:20:47 +02:00
Mike Fährmann
54c0715135
allow users to set their own API access_tokens/client_ids
2017-09-09 17:50:19 +02:00
Mike Fährmann
49c7e70c10
[acidimg] add image extractor
2017-09-09 15:19:18 +02:00
Mike Fährmann
9b21d3f13c
add '--filter' command-line option
...
This allows for image filtering via Python expressions by the same
metadata that is also used to build filenames (--list-keywords).
The usually shunned eval() function is used to evaluate
filter-expressions, but it seemed quite appropriate in this case and
shouldn't introduce any new security issues, as any attacker that could do
> gallery-dl --filter "delete-everything()" ...
could as well do
> python -c "delete-everything()"
2017-09-08 17:52:00 +02:00
Mike Fährmann
00420ff202
[booru] consistent order for "popular" results
2017-09-06 12:33:19 +02:00
Mike Fährmann
83cf1e1d6d
[sankaku] unescape image URLs
2017-09-04 17:41:11 +02:00
Mike Fährmann
f98e3e8002
[luscious] fix tag extraction
2017-09-01 16:29:52 +02:00
Mike Fährmann
65997d835b
replace popular/ranking tests with older ones
...
Metadata of several year old lists shouldn't change as much as it
would for newer ones, which makes metadata-comparisons of the output
of build_testresult_db.oy easier.
2017-08-31 15:09:18 +02:00
Mike Fährmann
be30fb2f98
add common config category for boorus and foolslide
2017-08-29 22:42:48 +02:00
Mike Fährmann
c0755a4d5e
[exhentai] revert login-method to its old version ( #37 )
...
Additional cookies don't seem to help and have to be manually set
anyway. The older method is more likely to succeed, so I'd rather
use this one.
2017-08-29 22:10:38 +02:00
Mike Fährmann
3ee39ffd93
[exhentai] update login procedure ( #37 )
...
This new version behaves pretty much exactly like a browser would and
caches all cookies sent to it and not just "ipb_member_id" and
"ipb_pass_hash".
2017-08-28 21:03:32 +02:00
Mike Fährmann
88a386977e
[booru] add "popular" extractors for more sites
...
- konachan.com
- behoimi.org
- e621.net
2017-08-26 23:08:52 +02:00
Mike Fährmann
07214f4007
[booru] place subcategories into base classes
2017-08-26 22:27:55 +02:00
Mike Fährmann
60a888a1e4
[foolfuuka] add common config category
...
All FoolFuuka based 4chan-archive extractors can now be configured using
their own config keys (extractor.<category>) as well as a common shared
one (extractor.foolfuuka).
2017-08-25 22:53:37 +02:00
Mike Fährmann
47bcf53ec1
implement support for additional unit test result types
...
- "pattern" matches all resulting URLs against the given regex
- "count" allows to specify the amount of returned URLs
2017-08-25 22:01:14 +02:00
Mike Fährmann
2d0dfe9d56
[exhenai] init headers before login and detect sadpanda
...
- also debug-logs html after failed login
- #37
2017-08-25 16:44:59 +02:00
Mike Fährmann
c7ec103e15
[batoto] fix extraction of chapter URLs
2017-08-25 16:34:42 +02:00
Mike Fährmann
18e6ed1c7e
[booru] add extractors for "Popular" images
2017-08-24 21:29:22 +02:00
Mike Fährmann
f7cdfd4c25
add a simplified version of 'parse_qs'
...
This version only returns a dict of plain string to string key-value
pairs and ignores multiple values for the same query variable.
2017-08-24 20:55:58 +02:00
Mike Fährmann
3b21e0703c
[deviantart] allow distinction between users and groups ( #26 )
...
This is done by prepending "group-" to an extractor's subcategory
if the URL belongs to a group ("folder" becomes "group-folder" and
so on). This changes the configuration-path being used and is also
reflected in the output of '--list-keywords'.
2017-08-22 20:15:13 +02:00
Mike Fährmann
e61a3a56d1
[hentai2read] fix and update keywords
...
Added the "author" keyword and changed the name of a few others to be
consistent with other manga/chapter extractors.
2017-08-22 15:01:47 +02:00
Mike Fährmann
c45770331a
use 'str.partition()'
...
The (r)partition method is always faster then split() or any other
method that has been replaced in this commit.
2017-08-21 18:29:50 +02:00
Mike Fährmann
017a72f448
[pixiv] improve input validation
2017-08-21 17:53:27 +02:00
Mike Fährmann
dcf42c5e89
[pixiv] add extractor for ranking lists
2017-08-20 20:21:52 +02:00
Mike Fährmann
4ea82ea556
[warosu] add thread extractor
2017-08-18 19:54:07 +02:00
Mike Fährmann
9aa95fba8c
[deviantart] adapt download URLs to use https
...
Even though DeviantArt is "completely switching over to HTTPS"[1],
every URL contained in an API response is still using HTTP
[1] https://danlev.deviantart.com/journal/DeviantArt-Is-Switching-To-HTTPS-697996906
2017-08-16 12:17:50 +02:00
Mike Fährmann
02e89700fc
[foolfuuka] ensure sorted posts
2017-08-13 14:29:26 +02:00
Mike Fährmann
8bcf88bff7
[flickr] fix extraction
...
This issue was only noticeable with older Python versions, as these
don't exhibit a consistent ordering of dict keys.
2017-08-12 21:41:10 +02:00
Mike Fährmann
004456d5d5
properly update the config-dictionary
...
When using 2 or more config files, the values of the second would
improperly overwrite nested dictionaries of the first one.
The new method properly combines these nested dictionaries as well.
2017-08-12 20:07:27 +02:00
Mike Fährmann
cfa479fab5
update error message for unspecified exceptions
...
- ask user to report unexpected errors, which usually indicate
extractor failure
- handle OSErrors separately (permissions, disk full, etc)
- revert 30eef52
2017-08-10 16:35:46 +02:00
Mike Fährmann
7e936e9c06
[luscious] simplify and remove dead code
2017-08-08 19:26:13 +02:00
Mike Fährmann
0245a0ba5f
fix extraction and update test results
...
- fixes for hbrowse, imgyt, imgcandy, hosturimage
- test updates for deviantart, gfycat
2017-08-08 19:11:13 +02:00
Mike Fährmann
abd7c559cd
[yonkouprod] remove module
...
Every manga chapter on this site has been removed.
2017-08-07 18:32:14 +02:00
Mike Fährmann
da7219ba74
[kisscomic] remove module
...
Image links on this site are dead.
2017-08-07 18:28:35 +02:00
Mike Fährmann
852e7acd31
[twitter] ignore "Promoted Tweets"
2017-08-06 13:43:08 +02:00
Mike Fährmann
915a0137de
improve 'extractor.request'
...
- add 'fatal' argument
- improve internal logic and flow
- raise known exception on error
- update exception hierarchy
2017-08-05 16:11:46 +02:00
rachmadani haryono
dcd573806e
chg: dev: fix error ( #32 )
...
* fix: dev: error
* fix: dev: AttributeError when getting artist
* fix: dev: typo on luscious parser
2017-08-04 15:01:10 +02:00
Mike Fährmann
c4713404c8
[directlink] improve URL pattern
2017-08-02 21:06:49 +02:00
Mike Fährmann
d443822fdb
[luacious] get correct image URLs ( fixes #33 )
...
Instead of using thumbnail URLs and modifying them the extractor now
goes through every single image-page and gets its download URL from
there.
2017-08-02 19:58:13 +02:00
