Xonshiz/comic-dl
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

Manga searching capabilites added.

Browse Source 
You can now Search and download manga straight from this tool. Check out the updated ReadMe to know more about this feature.
This commit is contained in:
Xonshiz 2017-12-28 08:08:40 +05:30
parent 4886184bd9
commit 8513670e91
14 changed files with 646 additions and 28 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

1
.gitignore vendored
View File

@ -72,3 +72,4 @@ comic_dl/build/__main__/warn__main__.txt
*.old2
comic_dl/build/__main__/xref-__main__.html
*.xml___jb_tmp___
comic_dl/Manga_Eden_Data.json

3
Changelog.md
View File

@ -63,4 +63,5 @@
- Added Support for [readcomics.website](http://www.readcomics.website/) (#63) [2017.12.27]
- Fix For #64 [2017.12.27]
- Fix For #66 [2017.12.27]
- Added MangaFox.la (mangafox.me redirects to .la now) [2017.12.27]
- Added MangaFox.la (mangafox.me redirects to .la now) [2017.12.27]
- Added Feature to Search and Download Manga from MangaEden [2017.12.28]

85
ReadMe.md
View File

@ -1,11 +1,13 @@
[![N|Solid](https://raw.githubusercontent.com/Xonshiz/comic-dl/master/Images/Icon.png)](https://github.com/Xonshiz/comic-dl)
# Comic-DL | [![Build Status](https://travis-ci.org/Xonshiz/comic-dl.svg?branch=master)](https://travis-ci.org/Xonshiz/comic-dl) [![Documentation Status](https://readthedocs.org/projects/comic-dl/badge/?version=latest)](http://comic-dl.readthedocs.io/en/latest/?badge=latest) | [![Donate](https://img.shields.io/badge/Donate-PayPal-green.svg)](https://www.paypal.me/xonshiz)
Comic-dl is a command line tool to download Comics and Manga from various Manga and Comic sites easily. Idea from [youtube-dl](https://github.com/rg3/youtube-dl)
Comic-dl is a command line tool to download Comics and Manga from various Manga and Comic sites easily. You can search Manga from this tool as well. Idea from [youtube-dl](https://github.com/rg3/youtube-dl)
> Don't overuse this script. Support the developers of those websites by disabling your adblock on their site.
>Advertisments pay for the website servers.
> Searching and downloading that manga is supported via MangaEden's API : http://www.mangaeden.com/api/
## Table of Content
* [Supported Sites](https://github.com/Xonshiz/comic-dl/blob/master/Supported_Sites.md)
@ -17,6 +19,7 @@ Comic-dl is a command line tool to download Comics and Manga from various Manga
* [Python Support](#python-support)
* [Windows Binary](#windows-binary)
* [List of Arguments](#list-of-arguments)
* [Using The Search](#using-the-search)
* [Youtube Tutorial](https://www.youtube.com/watch?v=TmQYhLHEZxA)
* [Usage](#usage)
* [Windows](#windows)
@ -147,12 +150,92 @@ Currently, the script supports these arguments :
--convert Tells the script to convert the downloaded Images to PDF or anything else. (Supported Values : pdf, cbz) (Default : No) [By default, script will not convert anything.]
--keep Tells the script whether to keep the files after conversion or not. (Supported : No, False) (Default : Yes/True) [By default, images will be kept even after conversion.]
--quality Tells the script about the image quality you want to download. (Supported Values : low/bad/worst/mobile/cancer) [By default, images will be downloaded in Highest Quality Available. No need to provide any option.]
-find, --search Searches for a manga through the Manga Eden Database.
-ml, --manga-language Selects the language for manga. 0 is English (Default) and 1 is Italian.
-sc, --skip-cache Forces to skip cache checking.
-cid, --chapter-id Takes the Chapter ID to list all the chapters in a Manga.
-fd, --force-download Forces download of chapters, when using comic-dl's search function.
-pid, --page-id Takes the Page ID to download a particular "chapter number" of a manga.
```
#### Note :
1.) Some websites like bato.to don't let you view some pages if you're not logged in. You'll have to create an account and pass the login information to the script via `-p` and `-u` arguments.
2.) Since omgbeaupeep is uh... well, you just need to pass the absolute chapter numbers in the range section for that. For eg : Check out [Richie Rich](http://www.omgbeaupeep.com/comics/Richie_Rich/647/). If you want to download first 600 episodes, you would pass : --range 001-600. Just check the URLs for those chapters and pass accordingly.
## Using The Search
In the updated of version 2017.12.28, searching is also available. This is a rather confusing approach though, so carefully read this section.
When you search via this tool, you will get the list of Manga and their respective unique IDs, that you will later use to download those Manga. Firstly, you will search for a Manga, it'll show it's unique ID, which you will copy and then pass into the tool again, it will list all the chapters listed in that particular Manga. The tool will then ask whether you want to download all the chapters belonging to that Manga. You can type in "Yes", "Y", "N" or "No" accordingly.
### How To Find A Manga:
To search for a Manga, you need to use `-find` or `--search` argument followed by Manga Name.
```
Windows Binary Command : `comic_dl.exe -find "<name_of_manga>"`
Python Command : `__main__.py -find "<name_of_manga>"`
```
For Example : If we wish to search for "One Piece", we wil use this : `comic_dl.exe -find "One Piece"`
This will show something like this :
```
Manga Name --> Manga ID
------------------------
One Piece: Wanted! --> 4e70ea60c092255ef7006726
One Piece (Databook) --> 5218b0ef45b9ef8b83731b00
One Piece x Toriko --> 4e70ea75c092255ef7006ee2
One Piece dj - Boukyaku Countdown --> 55a19e2b719a1609004ad1f3
One Piece --> 4e70ea10c092255ef7004aa2
One Piece Party --> 566d9611719a1697dd8cf79a
One Piece dj - Tears Will Surely Turn into Strength --> 55a19e31719a1609004ad1f7
One Piece dj - Lotus Maker --> 55a19e2e719a1609004ad1f5
One Piece dj - Three Days of Extreme Extravagance --> 55a19e34719a1609004ad1f9
```
As you can see, all the Manga matching the name show up, along with their unique IDs. You need to note these IDs down, if you want to download any of these Manga.
Here, for sake of an example, we'll take "One Piece" Manga and its ID is : "4e70ea10c092255ef7004aa2".
#### Note :
* When ever you search/find a Manga, comic_dl makes a "Manga_Eden_Data.json" file, which more or less serves as a Cache. It'll always reference the cache file for the next 24 hours. However, if you don't want it to use that cache file, just pass `--skip-cache` argument along with your command, and it will ignore the cache completely and fetch fresh resources and overwrite the older cache to update it.
* By default, the tool searches for only Manga translated in English Language. But, if you want to search for Manga translated in Italian, you can pass this argument : `--manga-language 1`.
### Getting List Of Chapters For A Manga:
So, now that you have the Manga's unique ID (mentioned above), you can now use that ID to get list of all the chapters for that Manga, or can even download those chapters directly.
So, to list all the chapters of "One Piece", we will pass its ID with the argument `--chapter-id`. The command will be:
```
Windows Binary Command : `comic_dl.exe --chapter-id "<unique_id_of_manga>"`
Python Command : `__main__.py --chapter-id "<unique_id_of_manga>"`
```
Our example command for One Piece would be : `comic_dl.exe --chapter-id "4e70ea10c092255ef7004aa2"`
This will return all the chapters, along with their unique IDs, which can be later used to download a separate chapter.
```
Chapter Number --> Chapter ID
-----------------------------
761.5 --> 54ad50d045b9ef961eeeda2e
714.5 --> 5552a262719a163d21dc7125
2 --> 4efe1d2ac0922504a300001a
127.5 --> 54ad15c445b9ef961eee798b
4 --> 4efe1d20c092250492000014
379.5 --> 5372485a45b9ef6a97744417
217.5 --> 54ad1f3245b9ef961eee826b
```
#### Note:
* If you use this command, it'll just list the chapters and then ask whether you want to download the chapters or not. If you wish to download the chapters without asking, just pass `--force-download` option along with the main command line. Script will NOT ask you anything. It'll list the chapters and start downloading them.
* If you wish to download only a few chapters in a range, you can do so by giving the good old `--range` command. If you pass this argument, the script will not ask you whether you want to download the chapters or not. You will not need `--force-download` option, if you are using `--range` already.
* Sorting is NOT supported in this, yet. YET!
### Download A Chapter:
You can download all the chapters of a Manga, as stated in the previous step. But, if you wish to download a particular chapter, then you need to get the unique ID of the chapter (mentioned above) and then download that chapter separately.
You need to use `--page-id "<unique_id_of_chapter>"` argument.
```
Windows Binary Command : `comic_dl.exe --page-id "<unique_id_of_chapter>"`
Python Command : `__main__.py --page-id "<unique_id_of_chapter>"`
```
Our example command for One Piece, chapter 2 would be : `comic_dl.exe --page-id "4efe1d2ac0922504a300001a"`
#### Note:
* If you download the chapter separately, you will need to provide the `Manga Name` and `Chapter Number` yourself. Because MangaEden's API doesn't list those values in their JSON reply (weird).
## Youtube Tutorial
[![Check The YouTube Tutorial](https://img.youtube.com/vi/TmQYhLHEZxA/0.jpg)](https://www.youtube.com/watch?v=TmQYhLHEZxA)

2
comic_dl/__main__.py
View File

@ -2,6 +2,8 @@
# -*- coding: utf-8 -*-
from comic_dl import ComicDL
import sites
import manga_eden
# import comic_dl
import sys

2
comic_dl/__version__.py
View File

@ -1,4 +1,4 @@
#!/usr/bin/env python
# -*- coding: utf-8 -*-
__version__ = "2017.12.27.a"
__version__ = "2017.12.28"

109
comic_dl/comic_dl.py
View File

@ -9,6 +9,7 @@ import platform
import honcho
import os
import time
import manga_eden
class ComicDL(object):
@ -18,15 +19,37 @@ class ComicDL(object):
parser.add_argument('--version', action='store_true', help='Shows version and exits.')
parser.add_argument('-s', '--sorting', nargs=1, help='Decides downloading order of chapters.')
parser.add_argument('-dd', '--download-directory', nargs=1, help='Decides the download directory of the comics/manga.')
parser.add_argument('-rn', '--range', nargs=1, help='Specifies the range of chapters to download.', default='All')
parser.add_argument('--convert', nargs=1, help='Tells the script to convert the downloaded Images to PDF or anything else.')
parser.add_argument('--keep', nargs=1, help='Tells the script whether to keep the files after conversion or not.', default='True')
parser.add_argument('-dd', '--download-directory', nargs=1,
help='Decides the download directory of the comics/manga.')
parser.add_argument('-rn', '--range', nargs=1,
help='Specifies the range of chapters to download.', default='All')
parser.add_argument('--convert', nargs=1,
help='Tells the script to convert the downloaded Images to PDF or anything else.')
parser.add_argument('--keep', nargs=1,
help='Tells the script whether to keep the files after conversion or not.', default='True')
parser.add_argument('--quality', nargs=1,
help='Tells the script which Quality of image to download (High/Low).', default='True')
parser.add_argument('-i', '--input', nargs=1, help='Inputs the URL to anime.')
parser.add_argument('-find', '--search', nargs=1, help='Searches for a manga through the Manga Eden Database.')
parser.add_argument('-ml', '--manga-language', nargs=1,
help='Selects the language for manga.', default='0')
parser.add_argument('-sc', '--skip-cache', nargs=1,
help='Forces to skip cache checking.', default='0')
parser.add_argument('-cid', '--chapter-id', nargs=1,
help='Takes the Chapter ID to list all the chapters in a Manga.')
parser.add_argument('-pid', '--page-id', nargs=1,
help='Takes the Page ID to download a particular "chapter number" of a manga.')
parser.add_argument("-fd", "--force-download", help="Forces download of chapters, when using comic-dl's"
" search function.",
action="store_true")
parser.add_argument("-v", "--verbose", help="Prints important debugging messages on screen.",
action="store_true")
logger = False
@ -41,7 +64,8 @@ class ComicDL(object):
print("\n***Starting the script in Verbose Mode***\n")
try:
os.remove("Error_Log.log")
except:
except Exception as VerboseError:
print(VerboseError)
pass
logging.basicConfig(format='%(levelname)s: %(message)s', filename="Error_Log.log", level=logging.DEBUG)
logging.debug("Arguments Provided : %s" % args)
@ -52,9 +76,74 @@ class ComicDL(object):
logging.debug("Python Version : %s (%s)" % (platform.python_version(), platform.architecture()[0]))
logger = True
if args.search:
start_time = time.time()
manga_eden.mangaSearch.MangaSearch(search_string=str(args.search[0]),
manga_language=args.manga_language[0], skip_cache=args.skip_cache[0])
end_time = time.time()
total_time = end_time - start_time
print("Total Time Taken To Search : %s" % total_time)
print("API Provided By Manga Eden : http://www.mangaeden.com/")
sys.exit()
if args.chapter_id:
force_download = False
if args.force_download:
force_download = True
if type(args.range) == list:
args.range = args.range[0]
if not args.sorting:
args.sorting = ["ascending"]
elif args.sorting:
print("Sorting not supported in this section yet.")
if not args.convert:
args.convert = ["None"]
if not args.keep:
args.keep = ["True"]
if not args.download_directory:
args.download_directory = [os.getcwd()]
start_time = time.time()
manga_eden.mangaChapters.MangaChapters(chapter_id=args.chapter_id[0],
download_directory=args.download_directory[0],
conversion=args.convert[0], delete_files=args.keep[0],
chapter_range=args.range, sorting_order=args.sorting[0],
force_download=force_download)
end_time = time.time()
total_time = end_time - start_time
print("Total Time Taken To Search : %s" % total_time)
print("API Provided By Manga Eden : http://www.mangaeden.com/")
sys.exit()
if args.page_id:
if not args.convert:
args.convert = ["None"]
if not args.keep:
args.keep = ["True"]
if not args.download_directory:
args.download_directory = [os.getcwd()]
start_time = time.time()
manga_eden.mangaChapterDownload.MangaChapterDownload(page_id=args.page_id[0],
download_directory=args.download_directory[0],
log_flag=logger, conversion=args.convert[0],
delete_files=args.keep[0])
end_time = time.time()
total_time = end_time - start_time
print("Total Time Taken To Download : %s" % total_time)
print("API Provided By Manga Eden : http://www.mangaeden.com/")
sys.exit()
if args.input is None:
print("I need an Input URL to download from.")
print("Run the script with --help to see more information.")
if not str(args.search).strip():
print("I need an Input URL to download from.")
print("Run the script with --help to see more information.")
else:
if not args.sorting:
args.sorting = ["ascending"]
@ -71,10 +160,14 @@ class ComicDL(object):
start_time = time.time()
honcho.Honcho().checker(comic_url=str(args.input[0]).strip(), current_directory=os.getcwd(),
sorting_order=args.sorting[0], logger=logger, download_directory=args.download_directory[0], chapter_range=args.range, conversion=args.convert[0], delete_files=args.keep[0], image_quality=args.quality[0])
sorting_order=args.sorting[0], logger=logger,
download_directory=args.download_directory[0],
chapter_range=args.range, conversion=args.convert[0],
delete_files=args.keep[0], image_quality=args.quality[0])
end_time = time.time()
total_time = end_time - start_time
print("Total Time Taken To Complete : %s" % total_time)
sys.exit()
@staticmethod
def version():

27
comic_dl/honcho.py
View File

@ -54,7 +54,8 @@ class Honcho(object):
current_directory=current_directory, sorting_order=sorting,
log_flag=log_flag, download_directory=download_directory,
chapter_range=chapter_range, conversion=kwargs.get("conversion"),
delete_files=kwargs.get("delete_files"), image_quality=kwargs.get("image_quality"))
delete_files=kwargs.get("delete_files"),
image_quality=kwargs.get("image_quality"))
return 0
elif domain in ["www.comic.naver.com", "comic.naver.com"]:
comicNaver.ComicNaver(manga_url=comic_url, logger=logging, current_directory=current_directory,
@ -103,23 +104,27 @@ class Honcho(object):
delete_files=kwargs.get("delete_files"))
return 0
elif domain in ["www.readcomicbooksonline.net", "readcomicbooksonline.net"]:
readComicBooksOnline.ReadComicBooksOnline(manga_url=comic_url, logger=logging, current_directory=current_directory,
sorting_order=sorting, log_flag=log_flag, download_directory=download_directory,
chapter_range=chapter_range, conversion=kwargs.get("conversion"),
delete_files=kwargs.get("delete_files"))
readComicBooksOnline.ReadComicBooksOnline(manga_url=comic_url, logger=logging,
current_directory=current_directory, sorting_order=sorting,
log_flag=log_flag, download_directory=download_directory,
chapter_range=chapter_range, conversion=kwargs.get("conversion"),
delete_files=kwargs.get("delete_files"))
return 0
elif domain in ["www.readcomics.website", "readcomics.website"]:
readComicsWebsite.ReadComicsWebsite(manga_url=comic_url, logger=logging, current_directory=current_directory,
sorting_order=sorting, log_flag=log_flag, download_directory=download_directory,
chapter_range=chapter_range, conversion=kwargs.get("conversion"),
delete_files=kwargs.get("delete_files"))
readComicsWebsite.ReadComicsWebsite(manga_url=comic_url, logger=logging,
current_directory=current_directory, sorting_order=sorting,
log_flag=log_flag, download_directory=download_directory,
chapter_range=chapter_range, conversion=kwargs.get("conversion"),
delete_files=kwargs.get("delete_files"))
return 0
elif domain in ["www.kissmanga.com", "kissmanga.com"]:
# kissManga.KissManga(manga_url = comic_url, logger = logging, current_directory = current_directory, sorting_order = sorting)
# kissManga.KissManga(manga_url = comic_url, logger = logging,
# current_directory = current_directory, sorting_order = sorting)
print("Under Development!")
return 0
elif domain in ["www.bato.to", "bato.to"]:
# kissManga.KissManga(manga_url = comic_url, logger = logging, current_directory = current_directory, sorting_order = sorting)
# kissManga.KissManga(manga_url = comic_url, logger = logging,
# current_directory = current_directory, sorting_order = sorting)
print("Under Development!")
return 0
else:

3
comic_dl/manga_eden/__init__.py Normal file
View File

@ -0,0 +1,3 @@
import mangaSearch
import mangaChapters
import mangaChapterDownload

97
comic_dl/manga_eden/mangaChapterDownload.py Normal file
View File

@ -0,0 +1,97 @@
#!/usr/bin/env python
# -*- coding: utf-8 -*-
import cfscrape
import requests
import json
import sys
import os
import globalFunctions
class MangaChapterDownload():
def __init__(self, page_id, download_directory, **kwargs):
self.page_id = str(page_id).strip()
self.manga_name = str(kwargs.get("manga_name"))
self.chapter_number = str(kwargs.get("chapter_number"))
self.logging = kwargs.get("log_flag")
self.conversion = kwargs.get("conversion")
self.delete_files = kwargs.get("delete_files")
self.json_content = self.json_download(page_id=self.page_id)
self.image_links = self.link_lookup(json_source=self.json_content)
if self.manga_name == "" or self.chapter_number == "":
try:
self.manga_name = raw_input("Please Enter Manga Name : ")
self.chapter_number = raw_input("Please Enter Chapter Number : ")
except Exception as WrongInputType:
# If python3, then raw_input() won't work.
self.manga_name = input("Please Enter Manga Name : ")
self.chapter_number = input("Please Enter Chapter Number : ")
else:
pass
file_directory = str(self.manga_name) + '/' + str(self.chapter_number) + "/"
directory_path = os.path.realpath(str(download_directory) + "/" + str(file_directory))
if not os.path.exists(directory_path):
os.makedirs(directory_path)
globalFunctions.GlobalFunctions().info_printer(self.manga_name, self.chapter_number)
for image in self.image_links:
globalFunctions.GlobalFunctions().downloader(image_ddl=self.image_links[image],
file_name=str(image) + str(self.image_links[image][-4:]),
referer=None, directory_path=directory_path,
log_flag=self.logging)
globalFunctions.GlobalFunctions().conversion(directory_path, self.conversion, self.delete_files,
self.manga_name, self.chapter_number)
def json_download(self, page_id):
headers = {
'User-Agent':
'Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/56.0.2924.87 Safari/537.36',
'Accept-Encoding': 'gzip, deflate'
}
sess = requests.session()
sess = cfscrape.create_scraper(sess)
search_url = "http://www.mangaeden.com/api/chapter/{0}/".format(page_id)
connection = sess.get(search_url, headers=headers)
if connection.status_code != 200:
print("Whoops! Seems like I can't connect to website.")
print("It's showing : %s" % connection)
print("Run this script with the --verbose argument and report the issue along with log file on Github.")
sys.exit(1)
else:
json_data = connection.content
return json_data
def link_lookup(self, json_source):
image_links = {}
""" the images's urls and sizes of the chapter are received via this API
"""
page_json = json.loads(json_source)
# print(page_json["images"])
list_of_pages = list(page_json["images"])
for page in list_of_pages:
# print(page)
image_links[page[0]] = "https://cdn.mangaeden.com/mangasimg/" + str(page[1])
# Let's sort this dictionary based on the chapter count (KEYS).
sorted(image_links.items(), key=lambda s: s[0])
if image_links:
return image_links
else:
return None

133
comic_dl/manga_eden/mangaChapters.py Normal file
View File

@ -0,0 +1,133 @@
#!/usr/bin/env python
# -*- coding: utf-8 -*-
import cfscrape
import requests
import json
import sys
import mangaChapterDownload
class MangaChapters():
"""Lists all the chapters and their IDs for a particular Manga."""
def __init__(self, chapter_id, download_directory, **kwargs):
self.chapter_id = str(chapter_id).strip()
self.download_range = str(kwargs.get("chapter_range"))
self.sorting_order = str(kwargs.get("sorting_order"))
self.force_download = str(kwargs.get("force_download"))
self.download_confirmation = "no"
self.json_source = self.json_download(chapter_id=self.chapter_id)
self.chapters, self.manga_name = self.id_lookup(json.loads(self.json_source))
if self.chapters:
final_chapter_dict = {}
print("Chapter Number --> Chapter ID")
print("-----------------------------")
for chapter in self.chapters:
print("{0} --> {1}".format(chapter, self.chapters[chapter]))
print("")
if self.download_range != "All":
self.download_confirmation = "yes"
self.force_download = "True"
# If '-fd' command has NOT been passed, we'll ask the user whether they want to download chapters or not.
if self.force_download == "False":
try:
self.download_confirmation = str(
raw_input("Do you want to download all the chapters? : ")).strip().lower()
except Exception as WrongInputType:
# If python3, then raw_input() won't work.
self.download_confirmation = str(
input("Do you want to download all the chapters? : ")).strip().lower()
# If a user has provided RANGE, then obviously, they want to download the chapters.
else:
self.download_confirmation = "yes"
# print("self.download_range : {0}".format(self.download_range))
# print("self.download_confirmation : {0}".format(self.download_confirmation))
if self.download_confirmation in ["yes", "y"]:
if self.download_range != "All":
# -1 to shift the episode number accordingly to the INDEX of it. List starts from 0 xD!
starting = int(str(self.download_range).split("-")[0])
if (str(self.download_range).split("-")[1]).decode().isdecimal():
ending = int(str(self.download_range).split("-")[1])
else:
ending = len(self.chapters)
for chapter in self.chapters:
if chapter >= starting and chapter <= ending:
final_chapter_dict[chapter] = str(self.chapters[chapter])
else:
pass
else:
final_chapter_dict = self.chapters
for chapter in final_chapter_dict:
mangaChapterDownload.MangaChapterDownload(page_id=final_chapter_dict[chapter],
download_directory=download_directory,
manga_name=str(self.manga_name),
chapter_number=str(chapter))
print("Finished Downloading")
sys.exit()
else:
sys.exit()
else:
print("No Chapter Found. Please Double Check the Chapter ID.")
def json_download(self, chapter_id):
headers = {
'User-Agent':
'Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/56.0.2924.87 Safari/537.36',
'Accept-Encoding': 'gzip, deflate'
}
sess = requests.session()
sess = cfscrape.create_scraper(sess)
search_url = "http://www.mangaeden.com/api/manga/{0}/".format(chapter_id)
connection = sess.get(search_url, headers=headers)
if connection.status_code != 200:
print("Whoops! Seems like I can't connect to website.")
print("It's showing : %s" % connection)
print("Run this script with the --verbose argument and report the issue along with log file on Github.")
sys.exit(1)
else:
json_data = connection.content
return json_data
def id_lookup(self, json_source):
chapters = {}
"""Example of a chapter array element:
[
5, # <-- chapter's number
1275542373.0, # <-- chapter's date
"5", # <-- chapter's title
"4e711cb0c09225616d037cc2" # <-- chapter's ID (chapter.id in the next section)
],
"""
list_of_chapters = list(json_source["chapters"])
# print(json_source["title"])
for chapter in list_of_chapters:
chapters[chapter[0]] = str(chapter[-1])
# Let's sort this dictionary based on the chapter count (KEYS).
sorted(chapters.items(), key=lambda s: s[0])
if chapters:
return (chapters, str(json_source["title"]))
else:
return None

113
comic_dl/manga_eden/mangaSearch.py Normal file
View File

@ -0,0 +1,113 @@
#!/usr/bin/env python
# -*- coding: utf-8 -*-
import cfscrape
import requests
import json
import sys
import os
import datetime
class MangaSearch():
"""Lists all the Mangas and their IDs based on user's search string."""
def __init__(self, search_string, manga_language=0, **kwargs):
self.search_string = str(search_string).strip()
self.manga_language = int(manga_language).__str__().strip()
self.cache_skip = str(kwargs.get("skip_cache"))
self.json_source = ""
"""Cache will check whether we have the JSON file already or not.If the file exists and it's not older than 1
day, we can use that to look things up. Otherwise, we need to download the fresh content. And, if anything weird
happens, we still need to fetch the latest content.
"""
# If user wants to force shut cache.
if self.cache_skip != 0:
if self.cache():
try:
with open("Manga_Eden_Data.json", "rb") as read_file:
self.json_source = read_file.read()
# self.json_source = json.load(read_file)
except Exception as FileNotFound:
print("Some Error Occurred : {0}".format(FileNotFound))
# If error occurs, we need to fetch the whole content again.
self.json_source = self.json_download(manga_language=self.manga_language)
else:
self.json_source = self.json_download(manga_language=self.manga_language)
else:
self.json_source = self.json_download(manga_language=self.manga_language)
print("Searching for {0} in the Database...".format(search_string))
if self.json_source and self.search_string:
self.result = self.id_lookup(json_source=self.json_source, user_string=self.search_string)
if self.result:
# print(self.result)
print("")
print("Manga Name --> Manga ID")
print("------------------------")
for manga_name in self.result:
print("{0} --> {1}".format(str(manga_name), str(self.result[manga_name])))
print("")
else:
print("Nothing Found.")
def cache(self):
try:
file_update_time = datetime.datetime.fromtimestamp(
os.path.getmtime(os.getcwd() + os.sep + "Manga_Eden_Data.json")).date()
current_time = datetime.date.today()
if (current_time - file_update_time).days > 1:
return False
else:
return True
except Exception as FileNotFound:
# print("Some Error Occurred : {0}".format(FileNotFound))
return False
def json_download(self, manga_language):
print("Downloading The Latest Data Set...")
headers = {
'User-Agent':
'Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/56.0.2924.87 Safari/537.36',
'Accept-Encoding': 'gzip, deflate'
}
sess = requests.session()
sess = cfscrape.create_scraper(sess)
search_url = "http://www.mangaeden.com/api/list/{0}/".format(manga_language)
connection = sess.get(search_url, headers=headers)
if connection.status_code != 200:
print("Whoops! Seems like I can't connect to website.")
print("It's showing : %s" % connection)
print("Run this script with the --verbose argument and report the issue along with log file on Github.")
sys.exit(1)
else:
json_data = connection.content
# print(json_data)
try:
# Let's save the JSON data
with open("Manga_Eden_Data.json", "wb") as write_file:
write_file.write(json_data)
except Exception as WriteError:
print("Couldn't make Cache : {0}".format(WriteError))
pass
return json_data
def id_lookup(self, json_source, user_string):
found_values = {}
formatted_json = json.loads(json_source)
for x in formatted_json["manga"]:
# print(x)
if str(user_string) in str(x.values()):
found_values[x["t"]] = x["i"]
else:
pass
if found_values:
return found_values
else:
return None

3
comic_dl/sites/acQQ.py
View File

@ -1,3 +1,6 @@
#!/usr/bin/env python
# -*- coding: utf-8 -*-
import re
import globalFunctions
import json

3
docs/Changelog.md
View File

@ -63,4 +63,5 @@
- Added Support for [readcomics.website](http://www.readcomics.website/) (#63) [2017.12.27]
- Fix For #64 [2017.12.27]
- Fix For #66 [2017.12.27]
- Added MangaFox.la (mangafox.me redirects to .la now) [2017.12.27]
- Added MangaFox.la (mangafox.me redirects to .la now) [2017.12.27]
- Added Feature to Search and Download Manga from MangaEden [2017.12.28]

93
docs/index.md
View File

@ -1,11 +1,13 @@
[![N|Solid](https://raw.githubusercontent.com/Xonshiz/comic-dl/master/Images/Icon.png)](https://github.com/Xonshiz/comic-dl)
# Comic-DL | [![Build Status](https://travis-ci.org/Xonshiz/comic-dl.svg?branch=master)](https://travis-ci.org/Xonshiz/comic-dl) [![Documentation Status](https://readthedocs.org/projects/comic-dl/badge/?version=latest)](http://comic-dl.readthedocs.io/en/latest/?badge=latest) | [![Donate](https://img.shields.io/badge/Donate-PayPal-green.svg)](https://www.paypal.me/xonshiz)
Comic-dl is a command line tool to download Comics and Manga from various Manga and Comic sites easily. Idea from [youtube-dl](https://github.com/rg3/youtube-dl)
Comic-dl is a command line tool to download Comics and Manga from various Manga and Comic sites easily. You can search Manga from this tool as well. Idea from [youtube-dl](https://github.com/rg3/youtube-dl)
> Don't overuse this script. Support the developers of those websites by disabling your adblock on their site.
>Advertisments pay for the website servers.
> Searching and downloading that manga is supported via MangaEden's API : http://www.mangaeden.com/api/
## Table of Content
* [Supported Sites](https://github.com/Xonshiz/comic-dl/blob/master/Supported_Sites.md)
@ -17,6 +19,7 @@ Comic-dl is a command line tool to download Comics and Manga from various Manga
* [Python Support](#python-support)
* [Windows Binary](#windows-binary)
* [List of Arguments](#list-of-arguments)
* [Using The Search](#using-the-search)
* [Youtube Tutorial](https://www.youtube.com/watch?v=TmQYhLHEZxA)
* [Usage](#usage)
* [Windows](#windows)
@ -141,18 +144,98 @@ Currently, the script supports these arguments :
-u,--username Indicates username for a website.
-p,--password Indicates password for a website.
-v,--verbose Enables Verbose logging.
--sorting Sorts the download order.(VALUES = asc, ascending,old,new,desc,descending,latest,new)
--sorting Sorts the download order.(VALUES = asc, ascending,old,new,desc,descending,latest,new)
-dd,--download-directory Specifies custom download location for the comics/manga.
-rn,--range Selects the range of Chapters to download (Default = All) [ Ex : --range 1-10 (This will download first 10 episodes of a series)]
--convert Tells the script to convert the downloaded Images to PDF or anything else. (Supported Values : pdf, cbz) (Default : No) [By default, script will not convert anything.]
--keep Tells the script whether to keep the files after conversion or not. (Supported : No, False) (Default : Yes/True) [By default, images will be kept even after conversion.]
--quality Tells the script about the image quality you want to download. (Supported Values : low/bad/worst/mobile/cancer) [By default, images will be downloaded in Highest Quality Available. No need to provide any option.]
--convert Tells the script to convert the downloaded Images to PDF or anything else. (Supported Values : pdf, cbz) (Default : No) [By default, script will not convert anything.]
--keep Tells the script whether to keep the files after conversion or not. (Supported : No, False) (Default : Yes/True) [By default, images will be kept even after conversion.]
--quality Tells the script about the image quality you want to download. (Supported Values : low/bad/worst/mobile/cancer) [By default, images will be downloaded in Highest Quality Available. No need to provide any option.]
-find, --search Searches for a manga through the Manga Eden Database.
-ml, --manga-language Selects the language for manga. 0 is English (Default) and 1 is Italian.
-sc, --skip-cache Forces to skip cache checking.
-cid, --chapter-id Takes the Chapter ID to list all the chapters in a Manga.
-fd, --force-download Forces download of chapters, when using comic-dl's search function.
-pid, --page-id Takes the Page ID to download a particular "chapter number" of a manga.
```
#### Note :
1.) Some websites like bato.to don't let you view some pages if you're not logged in. You'll have to create an account and pass the login information to the script via `-p` and `-u` arguments.
2.) Since omgbeaupeep is uh... well, you just need to pass the absolute chapter numbers in the range section for that. For eg : Check out [Richie Rich](http://www.omgbeaupeep.com/comics/Richie_Rich/647/). If you want to download first 600 episodes, you would pass : --range 001-600. Just check the URLs for those chapters and pass accordingly.
## Using The Search
In the updated of version 2017.12.28, searching is also available. This is a rather confusing approach though, so carefully read this section.
When you search via this tool, you will get the list of Manga and their respective unique IDs, that you will later use to download those Manga. Firstly, you will search for a Manga, it'll show it's unique ID, which you will copy and then pass into the tool again, it will list all the chapters listed in that particular Manga. The tool will then ask whether you want to download all the chapters belonging to that Manga. You can type in "Yes", "Y", "N" or "No" accordingly.
### How To Find A Manga:
To search for a Manga, you need to use `-find` or `--search` argument followed by Manga Name.
```
Windows Binary Command : `comic_dl.exe -find "<name_of_manga>"`
Python Command : `__main__.py -find "<name_of_manga>"`
```
For Example : If we wish to search for "One Piece", we wil use this : `comic_dl.exe -find "One Piece"`
This will show something like this :
```
Manga Name --> Manga ID
------------------------
One Piece: Wanted! --> 4e70ea60c092255ef7006726
One Piece (Databook) --> 5218b0ef45b9ef8b83731b00
One Piece x Toriko --> 4e70ea75c092255ef7006ee2
One Piece dj - Boukyaku Countdown --> 55a19e2b719a1609004ad1f3
One Piece --> 4e70ea10c092255ef7004aa2
One Piece Party --> 566d9611719a1697dd8cf79a
One Piece dj - Tears Will Surely Turn into Strength --> 55a19e31719a1609004ad1f7
One Piece dj - Lotus Maker --> 55a19e2e719a1609004ad1f5
One Piece dj - Three Days of Extreme Extravagance --> 55a19e34719a1609004ad1f9
```
As you can see, all the Manga matching the name show up, along with their unique IDs. You need to note these IDs down, if you want to download any of these Manga.
Here, for sake of an example, we'll take "One Piece" Manga and its ID is : "4e70ea10c092255ef7004aa2".
#### Note :
* When ever you search/find a Manga, comic_dl makes a "Manga_Eden_Data.json" file, which more or less serves as a Cache. It'll always reference the cache file for the next 24 hours. However, if you don't want it to use that cache file, just pass `--skip-cache` argument along with your command, and it will ignore the cache completely and fetch fresh resources and overwrite the older cache to update it.
* By default, the tool searches for only Manga translated in English Language. But, if you want to search for Manga translated in Italian, you can pass this argument : `--manga-language 1`.
### Getting List Of Chapters For A Manga:
So, now that you have the Manga's unique ID (mentioned above), you can now use that ID to get list of all the chapters for that Manga, or can even download those chapters directly.
So, to list all the chapters of "One Piece", we will pass its ID with the argument `--chapter-id`. The command will be:
```
Windows Binary Command : `comic_dl.exe --chapter-id "<unique_id_of_manga>"`
Python Command : `__main__.py --chapter-id "<unique_id_of_manga>"`
```
Our example command for One Piece would be : `comic_dl.exe --chapter-id "4e70ea10c092255ef7004aa2"`
This will return all the chapters, along with their unique IDs, which can be later used to download a separate chapter.
```
Chapter Number --> Chapter ID
-----------------------------
761.5 --> 54ad50d045b9ef961eeeda2e
714.5 --> 5552a262719a163d21dc7125
2 --> 4efe1d2ac0922504a300001a
127.5 --> 54ad15c445b9ef961eee798b
4 --> 4efe1d20c092250492000014
379.5 --> 5372485a45b9ef6a97744417
217.5 --> 54ad1f3245b9ef961eee826b
```
#### Note:
* If you use this command, it'll just list the chapters and then ask whether you want to download the chapters or not. If you wish to download the chapters without asking, just pass `--force-download` option along with the main command line. Script will NOT ask you anything. It'll list the chapters and start downloading them.
* If you wish to download only a few chapters in a range, you can do so by giving the good old `--range` command. If you pass this argument, the script will not ask you whether you want to download the chapters or not. You will not need `--force-download` option, if you are using `--range` already.
* Sorting is NOT supported in this, yet. YET!
### Download A Chapter:
You can download all the chapters of a Manga, as stated in the previous step. But, if you wish to download a particular chapter, then you need to get the unique ID of the chapter (mentioned above) and then download that chapter separately.
You need to use `--page-id "<unique_id_of_chapter>"` argument.
```
Windows Binary Command : `comic_dl.exe --page-id "<unique_id_of_chapter>"`
Python Command : `__main__.py --page-id "<unique_id_of_chapter>"`
```
Our example command for One Piece, chapter 2 would be : `comic_dl.exe --page-id "4efe1d2ac0922504a300001a"`
#### Note:
* If you download the chapter separately, you will need to provide the `Manga Name` and `Chapter Number` yourself. Because MangaEden's API doesn't list those values in their JSON reply (weird).
## Youtube Tutorial
[![Check The YouTube Tutorial](https://img.youtube.com/vi/TmQYhLHEZxA/0.jpg)](https://www.youtube.com/watch?v=TmQYhLHEZxA)