FaceDeer
/
FanFicFare
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

Make CLI available as a pip installable PyPI module--Incomplete.

master
Jim Miller 2015-04-07 22:51:26 -05:00
parent d2af52b0fd
commit 47c1b45b9c
8 changed files with 2766 additions and 2414 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

2
.gitignore vendored
View File

@ -20,3 +20,5 @@ FanFictionDownLoader.zip
FanFicFare.zip
output
build
dist
FanFicFare.egg-info

15
DESCRIPTION.rst Normal file
View File

@ -0,0 +1,15 @@
FanFicFare (FFF)
=======================
FanFicFare is a tool for downloading fanfiction and original stories
from various sites into ebook form.
FanFicFare(FFF) is the renamed successor to
FanFictionDownLoader(FFDL). The project was renamed due to another,
unrelated project sharing the same name.
FanFicFare can download stories from over 100 different fanfiction and
original fiction sites.
FanFicFare can output stories into EPUB (the preferred format), HTML,
plain text and MOBI formats.

202
LICENSE Normal file
View File

@ -0,0 +1,202 @@
Apache License
Version 2.0, January 2004
http://www.apache.org/licenses/
TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
1. Definitions.
"License" shall mean the terms and conditions for use, reproduction,
and distribution as defined by Sections 1 through 9 of this document.
"Licensor" shall mean the copyright owner or entity authorized by
the copyright owner that is granting the License.
"Legal Entity" shall mean the union of the acting entity and all
other entities that control, are controlled by, or are under common
control with that entity. For the purposes of this definition,
"control" means (i) the power, direct or indirect, to cause the
direction or management of such entity, whether by contract or
otherwise, or (ii) ownership of fifty percent (50%) or more of the
outstanding shares, or (iii) beneficial ownership of such entity.
"You" (or "Your") shall mean an individual or Legal Entity
exercising permissions granted by this License.
"Source" form shall mean the preferred form for making modifications,
including but not limited to software source code, documentation
source, and configuration files.
"Object" form shall mean any form resulting from mechanical
transformation or translation of a Source form, including but
not limited to compiled object code, generated documentation,
and conversions to other media types.
"Work" shall mean the work of authorship, whether in Source or
Object form, made available under the License, as indicated by a
copyright notice that is included in or attached to the work
(an example is provided in the Appendix below).
"Derivative Works" shall mean any work, whether in Source or Object
form, that is based on (or derived from) the Work and for which the
editorial revisions, annotations, elaborations, or other modifications
represent, as a whole, an original work of authorship. For the purposes
of this License, Derivative Works shall not include works that remain
separable from, or merely link (or bind by name) to the interfaces of,
the Work and Derivative Works thereof.
"Contribution" shall mean any work of authorship, including
the original version of the Work and any modifications or additions
to that Work or Derivative Works thereof, that is intentionally
submitted to Licensor for inclusion in the Work by the copyright owner
or by an individual or Legal Entity authorized to submit on behalf of
the copyright owner. For the purposes of this definition, "submitted"
means any form of electronic, verbal, or written communication sent
to the Licensor or its representatives, including but not limited to
communication on electronic mailing lists, source code control systems,
and issue tracking systems that are managed by, or on behalf of, the
Licensor for the purpose of discussing and improving the Work, but
excluding communication that is conspicuously marked or otherwise
designated in writing by the copyright owner as "Not a Contribution."
"Contributor" shall mean Licensor and any individual or Legal Entity
on behalf of whom a Contribution has been received by Licensor and
subsequently incorporated within the Work.
2. Grant of Copyright License. Subject to the terms and conditions of
this License, each Contributor hereby grants to You a perpetual,
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
copyright license to reproduce, prepare Derivative Works of,
publicly display, publicly perform, sublicense, and distribute the
Work and such Derivative Works in Source or Object form.
3. Grant of Patent License. Subject to the terms and conditions of
this License, each Contributor hereby grants to You a perpetual,
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
(except as stated in this section) patent license to make, have made,
use, offer to sell, sell, import, and otherwise transfer the Work,
where such license applies only to those patent claims licensable
by such Contributor that are necessarily infringed by their
Contribution(s) alone or by combination of their Contribution(s)
with the Work to which such Contribution(s) was submitted. If You
institute patent litigation against any entity (including a
cross-claim or counterclaim in a lawsuit) alleging that the Work
or a Contribution incorporated within the Work constitutes direct
or contributory patent infringement, then any patent licenses
granted to You under this License for that Work shall terminate
as of the date such litigation is filed.
4. Redistribution. You may reproduce and distribute copies of the
Work or Derivative Works thereof in any medium, with or without
modifications, and in Source or Object form, provided that You
meet the following conditions:
(a) You must give any other recipients of the Work or
Derivative Works a copy of this License; and
(b) You must cause any modified files to carry prominent notices
stating that You changed the files; and
(c) You must retain, in the Source form of any Derivative Works
that You distribute, all copyright, patent, trademark, and
attribution notices from the Source form of the Work,
excluding those notices that do not pertain to any part of
the Derivative Works; and
(d) If the Work includes a "NOTICE" text file as part of its
distribution, then any Derivative Works that You distribute must
include a readable copy of the attribution notices contained
within such NOTICE file, excluding those notices that do not
pertain to any part of the Derivative Works, in at least one
of the following places: within a NOTICE text file distributed
as part of the Derivative Works; within the Source form or
documentation, if provided along with the Derivative Works; or,
within a display generated by the Derivative Works, if and
wherever such third-party notices normally appear. The contents
of the NOTICE file are for informational purposes only and
do not modify the License. You may add Your own attribution
notices within Derivative Works that You distribute, alongside
or as an addendum to the NOTICE text from the Work, provided
that such additional attribution notices cannot be construed
as modifying the License.
You may add Your own copyright statement to Your modifications and
may provide additional or different license terms and conditions
for use, reproduction, or distribution of Your modifications, or
for any such Derivative Works as a whole, provided Your use,
reproduction, and distribution of the Work otherwise complies with
the conditions stated in this License.
5. Submission of Contributions. Unless You explicitly state otherwise,
any Contribution intentionally submitted for inclusion in the Work
by You to the Licensor shall be under the terms and conditions of
this License, without any additional terms or conditions.
Notwithstanding the above, nothing herein shall supersede or modify
the terms of any separate license agreement you may have executed
with Licensor regarding such Contributions.
6. Trademarks. This License does not grant permission to use the trade
names, trademarks, service marks, or product names of the Licensor,
except as required for reasonable and customary use in describing the
origin of the Work and reproducing the content of the NOTICE file.
7. Disclaimer of Warranty. Unless required by applicable law or
agreed to in writing, Licensor provides the Work (and each
Contributor provides its Contributions) on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
implied, including, without limitation, any warranties or conditions
of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
PARTICULAR PURPOSE. You are solely responsible for determining the
appropriateness of using or redistributing the Work and assume any
risks associated with Your exercise of permissions under this License.
8. Limitation of Liability. In no event and under no legal theory,
whether in tort (including negligence), contract, or otherwise,
unless required by applicable law (such as deliberate and grossly
negligent acts) or agreed to in writing, shall any Contributor be
liable to You for damages, including any direct, indirect, special,
incidental, or consequential damages of any character arising as a
result of this License or out of the use or inability to use the
Work (including but not limited to damages for loss of goodwill,
work stoppage, computer failure or malfunction, or any and all
other commercial damages or losses), even if such Contributor
has been advised of the possibility of such damages.
9. Accepting Warranty or Additional Liability. While redistributing
the Work or Derivative Works thereof, You may choose to offer,
and charge a fee for, acceptance of support, warranty, indemnity,
or other liability obligations and/or rights consistent with this
License. However, in accepting such obligations, You may act only
on Your own behalf and on Your sole responsibility, not on behalf
of any other Contributor, and only if You agree to indemnify,
defend, and hold each Contributor harmless for any liability
incurred by, or claims asserted against, such Contributor by reason
of your accepting any such warranty or additional liability.
END OF TERMS AND CONDITIONS
APPENDIX: How to apply the Apache License to your work.
To apply the Apache License to your work, attach the following
boilerplate notice, with the fields enclosed by brackets "[]"
replaced with your own identifying information. (Don't include
the brackets!) The text should be enclosed in the appropriate
comment syntax for the file format. We also recommend that a
file or class name and description of purpose be included on the
same "printed page" as the copyright notice for easier
identification within third-party archives.
Copyright [yyyy] [name of copyright owner]
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.

15
cmds Normal file
View File

@ -0,0 +1,15 @@
(one-time:)
python setup.py register -r https://testpypi.python.org/pypi
rm -rf dist
python setup.py sdist
python setup.py bdist_wheel
twine upload dist/* -r pypitest
Install:
pip install --pre -i https://testpypi.python.org/pypi FanFicFare

4006
cli/defaults.ini → defaults.ini
View File

File diff suppressed because it is too large Load Diff

206
cli/example.ini → example.ini
View File

@ -1,103 +1,103 @@
## This is an example of what your personal configuration might look
## like. Uncomment options by removing the '#' in front of them.
[defaults]
## Some sites also require the user to confirm they are adult for
## adult content. Uncomment by removing '#' in front of is_adult. In
## commandline version, this should go in your personal.ini, not
## defaults.ini.
#is_adult:true
## Don't like the numbers at the start of chapter titles on some
## sites? You can use strip_chapter_numbers to strip them off. Just
## want to make them all look the same? Strip them off, then add them
## back on with add_chapter_numbers. Don't like the way it strips
## numbers or adds them back? See chapter_title_strip_pattern and
## chapter_title_add_pattern.
#strip_chapter_numbers:true
#add_chapter_numbers:true
[epub]
## include images from img tags in the body and summary of stories.
## Images will be converted to jpg for size if possible. Images work
## in epub format only. To get mobi or other format with images,
## download as epub and use Calibre to convert.
#include_images:true
## If not set, the summary will have all html stripped for safety.
## Both this and include_images must be true to get images in the
## summary.
#keep_summary_html:true
## If set, the first image found will be made the cover image. If
## keep_summary_html is true, any images in summary will be before any
## in chapters.
#make_firstimage_cover:true
## Resize images down to width, height, preserving aspect ratio.
## Nook size, with margin.
#image_max_size: 580, 725
## Change image to grayscale, if graphics library allows, to save
## space.
#grayscale_images: false
## Most common, I expect will be using this to save username/passwords
## for different sites. Here are a few examples. See defaults.ini
## for the full list.
[www.twilighted.net]
#username:YourPenname
#password:YourPassword
## default is false
#collect_series: true
[ficwad.com]
#username:YourUsername
#password:YourPassword
[www.twiwrite.net]
#username:YourName
#password:yourpassword
## default is false
#collect_series: true
[www.adastrafanfic.com]
## Some sites do not require a login, but do require the user to
## confirm they are adult for adult content.
#is_adult:true
[www.twcslibrary.net]
#username:YourName
#password:yourpassword
#is_adult:true
## default is false
#collect_series: true
[www.fictionalley.org]
#is_adult:true
[www.harrypotterfanfiction.com]
#is_adult:true
[www.fimfiction.net]
#is_adult:true
#fail_on_password: false
[www.tthfanfic.org]
#is_adult:true
## tth is a little unusual--it doesn't require user/pass, but the site
## keeps track of which chapters you've read and won't send another
## update until it thinks you're up to date. This way, on download,
## it thinks you're up to date.
#username:YourName
#password:yourpassword
## This section will override anything in the system defaults or other
## sections here.
[overrides]
## default varies by site. Set true here to force all sites to
## collect series.
#collect_series: true
## This is an example of what your personal configuration might look
## like. Uncomment options by removing the '#' in front of them.
[defaults]
## Some sites also require the user to confirm they are adult for
## adult content. Uncomment by removing '#' in front of is_adult. In
## commandline version, this should go in your personal.ini, not
## defaults.ini.
#is_adult:true
## Don't like the numbers at the start of chapter titles on some
## sites? You can use strip_chapter_numbers to strip them off. Just
## want to make them all look the same? Strip them off, then add them
## back on with add_chapter_numbers. Don't like the way it strips
## numbers or adds them back? See chapter_title_strip_pattern and
## chapter_title_add_pattern.
#strip_chapter_numbers:true
#add_chapter_numbers:true
[epub]
## include images from img tags in the body and summary of stories.
## Images will be converted to jpg for size if possible. Images work
## in epub format only. To get mobi or other format with images,
## download as epub and use Calibre to convert.
#include_images:true
## If not set, the summary will have all html stripped for safety.
## Both this and include_images must be true to get images in the
## summary.
#keep_summary_html:true
## If set, the first image found will be made the cover image. If
## keep_summary_html is true, any images in summary will be before any
## in chapters.
#make_firstimage_cover:true
## Resize images down to width, height, preserving aspect ratio.
## Nook size, with margin.
#image_max_size: 580, 725
## Change image to grayscale, if graphics library allows, to save
## space.
#grayscale_images: false
## Most common, I expect will be using this to save username/passwords
## for different sites. Here are a few examples. See defaults.ini
## for the full list.
[www.twilighted.net]
#username:YourPenname
#password:YourPassword
## default is false
#collect_series: true
[ficwad.com]
#username:YourUsername
#password:YourPassword
[www.twiwrite.net]
#username:YourName
#password:yourpassword
## default is false
#collect_series: true
[www.adastrafanfic.com]
## Some sites do not require a login, but do require the user to
## confirm they are adult for adult content.
#is_adult:true
[www.twcslibrary.net]
#username:YourName
#password:yourpassword
#is_adult:true
## default is false
#collect_series: true
[www.fictionalley.org]
#is_adult:true
[www.harrypotterfanfiction.com]
#is_adult:true
[www.fimfiction.net]
#is_adult:true
#fail_on_password: false
[www.tthfanfic.org]
#is_adult:true
## tth is a little unusual--it doesn't require user/pass, but the site
## keeps track of which chapters you've read and won't send another
## update until it thinks you're up to date. This way, on download,
## it thinks you're up to date.
#username:YourName
#password:yourpassword
## This section will override anything in the system defaults or other
## sections here.
[overrides]
## default varies by site. Set true here to force all sites to
## collect series.
#collect_series: true

618
cli/downloader.py → fff_internals/cli.py
View File

@ -1,308 +1,310 @@
# -*- coding: utf-8 -*-
# Copyright 2014 Fanficdownloader team
#
# Licensed under the Apache License, Version 2.0 (the 'License');
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an 'AS IS' BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#
from optparse import OptionParser
from os.path import expanduser, isfile, join
from subprocess import call
import ConfigParser
import getpass
import logging
import pprint
import string
import sys
if sys.version_info < (2, 5):
print 'This program requires Python 2.5 or newer.'
sys.exit(1)
if sys.version_info >= (2, 7):
# suppresses default logger. Logging is setup in fanficdownload/__init__.py so it works in calibre, too.
rootlogger = logging.getLogger()
loghandler = logging.NullHandler()
loghandler.setFormatter(logging.Formatter('(=====)(levelname)s:%(message)s'))
rootlogger.addHandler(loghandler)
try:
# running under calibre
from calibre_plugins.fanfictiondownloader_plugin.fff_internals import adapters, writers, exceptions
from calibre_plugins.fanfictiondownloader_plugin.fff_internals.configurable import Configuration
from calibre_plugins.fanfictiondownloader_plugin.fff_internals.epubutils import get_dcsource_chaptercount, get_update_data
from calibre_plugins.fanfictiondownloader_plugin.fff_internals.geturls import get_urls_from_page
except ImportError:
from fff_internals import adapters, writers, exceptions
from fff_internals.configurable import Configuration
from fff_internals.epubutils import get_dcsource_chaptercount, get_update_data
from fff_internals.geturls import get_urls_from_page
def write_story(config, adapter, writeformat, metaonly=False, outstream=None):
writer = writers.getWriter(writeformat, config, adapter)
writer.writeStory(outstream=outstream, metaonly=metaonly)
output_filename = writer.getOutputFileName()
del writer
return output_filename
def main(argv, parser=None, passed_defaultsini=None, passed_personalini=None):
# read in args, anything starting with -- will be treated as --<varible>=<value>
if not parser:
parser = OptionParser('usage: %prog [options] storyurl')
parser.add_option('-f', '--format', dest='format', default='epub',
help='write story as FORMAT, epub(default), mobi, text or html', metavar='FORMAT')
if passed_defaultsini:
config_help = 'read config from specified file(s) in addition to calibre plugin personal.ini, ~/.fanficfare/personal.ini, and ./personal.ini'
else:
config_help = 'read config from specified file(s) in addition to ~/.fanficfare/defaults.ini, ~/.fanficfare/personal.ini, ./defaults.ini, and ./personal.ini'
parser.add_option('-c', '--config',
action='append', dest='configfile', default=None,
help=config_help, metavar='CONFIG')
parser.add_option('-b', '--begin', dest='begin', default=None,
help='Begin with Chapter START', metavar='START')
parser.add_option('-e', '--end', dest='end', default=None,
help='End with Chapter END', metavar='END')
parser.add_option('-o', '--option',
action='append', dest='options',
help='set an option NAME=VALUE', metavar='NAME=VALUE')
parser.add_option('-m', '--meta-only',
action='store_true', dest='metaonly',
help='Retrieve metadata and stop. Or, if --update-epub, update metadata title page only.', )
parser.add_option('-u', '--update-epub',
action='store_true', dest='update',
help='Update an existing epub with new chapters, give epub filename instead of storyurl.', )
parser.add_option('--update-cover',
action='store_true', dest='updatecover',
help='Update cover in an existing epub, otherwise existing cover (if any) is used on update. Only valid with --update-epub.', )
parser.add_option('--force',
action='store_true', dest='force',
help='Force overwrite of an existing epub, download and overwrite all chapters.', )
parser.add_option('-l', '--list',
action='store_true', dest='list',
help='Get list of valid story URLs from page given.', )
parser.add_option('-n', '--normalize-list',
action='store_true', dest='normalize', default=False,
help='Get list of valid story URLs from page given, but normalized to standard forms.', )
parser.add_option('-s', '--sites-list',
action='store_true', dest='siteslist', default=False,
help='Get list of valid story URLs examples.', )
parser.add_option('-d', '--debug',
action='store_true', dest='debug',
help='Show debug output while downloading.', )
options, args = parser.parse_args(argv)
if not options.debug:
logger = logging.getLogger('fff_internals')
logger.setLevel(logging.INFO)
if not options.siteslist and len(args) != 1:
parser.error('incorrect number of arguments')
if options.siteslist:
for site, examples in adapters.getSiteExamples():
print '\n====%s====\n\nExample URLs:' % site
for u in examples:
print ' * %s' % u
return
if options.update and options.format != 'epub':
parser.error('-u/--update-epub only works with epub')
# Attempt to update an existing epub.
chaptercount = None
output_filename = None
if options.update:
try:
url, chaptercount = get_dcsource_chaptercount(args[0])
if not url:
print 'No story URL found in epub to update.'
return
print 'Updating %s, URL: %s' % (args[0], url)
output_filename = args[0]
except Exception:
# if there's an error reading the update file, maybe it's a URL?
# we'll look for an existing outputfile down below.
url = args[0]
else:
url = args[0]
try:
configuration = Configuration(adapters.getConfigSectionFor(url), options.format)
except exceptions.UnknownSite, e:
if options.list or options.normalize:
# list for page doesn't have to be a supported site.
configuration = Configuration('test1.com', options.format)
else:
raise e
conflist = []
homepath = join(expanduser('~'), '.fanficdownloader')
## also look for .fanficfare now, give higher priority than old dir.
homepath2 = join(expanduser('~'), '.fanficfare')
if passed_defaultsini:
configuration.readfp(passed_defaultsini)
if isfile(join(homepath, 'defaults.ini')):
conflist.append(join(homepath, 'defaults.ini'))
if isfile(join(homepath2, 'defaults.ini')):
conflist.append(join(homepath2, 'defaults.ini'))
if isfile('defaults.ini'):
conflist.append('defaults.ini')
if passed_personalini:
configuration.readfp(passed_personalini)
if isfile(join(homepath, 'personal.ini')):
conflist.append(join(homepath, 'personal.ini'))
if isfile(join(homepath2, 'personal.ini')):
conflist.append(join(homepath2, 'personal.ini'))
if isfile('personal.ini'):
conflist.append('personal.ini')
if options.configfile:
conflist.extend(options.configfile)
logging.debug('reading %s config file(s), if present' % conflist)
configuration.read(conflist)
try:
configuration.add_section('overrides')
except ConfigParser.DuplicateSectionError:
pass
if options.force:
configuration.set('overrides', 'always_overwrite', 'true')
if options.update and chaptercount:
configuration.set('overrides', 'output_filename', output_filename)
if options.update and not options.updatecover:
configuration.set('overrides', 'never_make_cover', 'true')
# images only for epub, even if the user mistakenly turned it
# on else where.
if options.format not in ('epub', 'html'):
configuration.set('overrides', 'include_images', 'false')
if options.options:
for opt in options.options:
(var, val) = opt.split('=')
configuration.set('overrides', var, val)
if options.list or options.normalize:
retlist = get_urls_from_page(args[0], configuration, normalize=options.normalize)
print '\n'.join(retlist)
return
try:
adapter = adapters.getAdapter(configuration, url)
adapter.setChaptersRange(options.begin, options.end)
# check for updating from URL (vs from file)
if options.update and not chaptercount:
try:
writer = writers.getWriter('epub', configuration, adapter)
output_filename = writer.getOutputFileName()
noturl, chaptercount = get_dcsource_chaptercount(output_filename)
print 'Updating %s, URL: %s' % (output_filename, url)
except Exception:
options.update = False
pass
# Check for include_images without no_image_processing. In absence of PIL, give warning.
if adapter.getConfig('include_images') and not adapter.getConfig('no_image_processing'):
try:
from calibre.utils.magick import Image
logging.debug('Using calibre.utils.magick')
except ImportError:
try:
import Image
logging.debug('Using PIL')
except ImportError:
print "You have include_images enabled, but Python Image Library(PIL) isn't found.\nImages will be included full size in original format.\nContinue? (y/n)?"
if not sys.stdin.readline().strip().lower().startswith('y'):
return
# three tries, that's enough if both user/pass & is_adult needed,
# or a couple tries of one or the other
for x in range(0, 2):
try:
adapter.getStoryMetadataOnly()
except exceptions.FailedToLogin, f:
if f.passwdonly:
print 'Story requires a password.'
else:
print 'Login Failed, Need Username/Password.'
sys.stdout.write('Username: ')
adapter.username = sys.stdin.readline().strip()
adapter.password = getpass.getpass(prompt='Password: ')
# print('Login: `%s`, Password: `%s`' % (adapter.username, adapter.password))
except exceptions.AdultCheckRequired:
print 'Please confirm you are an adult in your locale: (y/n)?'
if sys.stdin.readline().strip().lower().startswith('y'):
adapter.is_adult = True
if options.update and not options.force:
urlchaptercount = int(adapter.getStoryMetadataOnly().getMetadata('numChapters'))
if chaptercount == urlchaptercount and not options.metaonly:
print '%s already contains %d chapters.' % (output_filename, chaptercount)
elif chaptercount > urlchaptercount:
print '%s contains %d chapters, more than source: %d.' % (output_filename, chaptercount, urlchaptercount)
elif chaptercount == 0:
print "%s doesn't contain any recognizable chapters, probably from a different source. Not updating." % output_filename
else:
# update now handled by pre-populating the old
# images and chapters in the adapter rather than
# merging epubs.
url, chaptercount, adapter.oldchapters, adapter.oldimgs, adapter.oldcover, adapter.calibrebookmark, adapter.logfile = get_update_data(output_filename)
print 'Do update - epub(%d) vs url(%d)' % (chaptercount, urlchaptercount)
if not options.update and chaptercount == urlchaptercount and adapter.getConfig('do_update_hook'):
adapter.hookForUpdates(chaptercount)
write_story(configuration, adapter, 'epub')
else:
# regular download
if options.metaonly:
pprint.pprint(adapter.getStoryMetadataOnly().getAllMetadata())
output_filename = write_story(configuration, adapter, options.format, options.metaonly)
if not options.metaonly and adapter.getConfig('post_process_cmd'):
metadata = adapter.story.metadata
metadata['output_filename'] = output_filename
call(string.Template(adapter.getConfig('post_process_cmd')).substitute(metadata), shell=True)
del adapter
except exceptions.InvalidStoryURL, isu:
print isu
except exceptions.StoryDoesNotExist, dne:
print dne
except exceptions.UnknownSite, us:
print us
if __name__ == '__main__':
main(sys.argv[1:])
# -*- coding: utf-8 -*-
# Copyright 2014 Fanficdownloader team
#
# Licensed under the Apache License, Version 2.0 (the 'License');
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an 'AS IS' BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#
from optparse import OptionParser
from os.path import expanduser, isfile, join
from subprocess import call
import ConfigParser
import getpass
import logging
import pprint
import string
import sys
if sys.version_info < (2, 5):
print 'This program requires Python 2.5 or newer.'
sys.exit(1)
if sys.version_info >= (2, 7):
# suppresses default logger. Logging is setup in fanficdownload/__init__.py so it works in calibre, too.
rootlogger = logging.getLogger()
loghandler = logging.NullHandler()
loghandler.setFormatter(logging.Formatter('(=====)(levelname)s:%(message)s'))
rootlogger.addHandler(loghandler)
try:
# running under calibre
from calibre_plugins.fanfictiondownloader_plugin.fff_internals import adapters, writers, exceptions
from calibre_plugins.fanfictiondownloader_plugin.fff_internals.configurable import Configuration
from calibre_plugins.fanfictiondownloader_plugin.fff_internals.epubutils import get_dcsource_chaptercount, get_update_data
from calibre_plugins.fanfictiondownloader_plugin.fff_internals.geturls import get_urls_from_page
except ImportError:
from fff_internals import adapters, writers, exceptions
from fff_internals.configurable import Configuration
from fff_internals.epubutils import get_dcsource_chaptercount, get_update_data
from fff_internals.geturls import get_urls_from_page
def write_story(config, adapter, writeformat, metaonly=False, outstream=None):
writer = writers.getWriter(writeformat, config, adapter)
writer.writeStory(outstream=outstream, metaonly=metaonly)
output_filename = writer.getOutputFileName()
del writer
return output_filename
def main(argv=None, parser=None, passed_defaultsini=None, passed_personalini=None):
if argv is None:
argv = sys.argv[1:]
# read in args, anything starting with -- will be treated as --<varible>=<value>
if not parser:
parser = OptionParser('usage: %prog [options] storyurl')
parser.add_option('-f', '--format', dest='format', default='epub',
help='write story as FORMAT, epub(default), mobi, text or html', metavar='FORMAT')
if passed_defaultsini:
config_help = 'read config from specified file(s) in addition to calibre plugin personal.ini, ~/.fanficfare/personal.ini, and ./personal.ini'
else:
config_help = 'read config from specified file(s) in addition to ~/.fanficfare/defaults.ini, ~/.fanficfare/personal.ini, ./defaults.ini, and ./personal.ini'
parser.add_option('-c', '--config',
action='append', dest='configfile', default=None,
help=config_help, metavar='CONFIG')
parser.add_option('-b', '--begin', dest='begin', default=None,
help='Begin with Chapter START', metavar='START')
parser.add_option('-e', '--end', dest='end', default=None,
help='End with Chapter END', metavar='END')
parser.add_option('-o', '--option',
action='append', dest='options',
help='set an option NAME=VALUE', metavar='NAME=VALUE')
parser.add_option('-m', '--meta-only',
action='store_true', dest='metaonly',
help='Retrieve metadata and stop. Or, if --update-epub, update metadata title page only.', )
parser.add_option('-u', '--update-epub',
action='store_true', dest='update',
help='Update an existing epub with new chapters, give epub filename instead of storyurl.', )
parser.add_option('--update-cover',
action='store_true', dest='updatecover',
help='Update cover in an existing epub, otherwise existing cover (if any) is used on update. Only valid with --update-epub.', )
parser.add_option('--force',
action='store_true', dest='force',
help='Force overwrite of an existing epub, download and overwrite all chapters.', )
parser.add_option('-l', '--list',
action='store_true', dest='list',
help='Get list of valid story URLs from page given.', )
parser.add_option('-n', '--normalize-list',
action='store_true', dest='normalize', default=False,
help='Get list of valid story URLs from page given, but normalized to standard forms.', )
parser.add_option('-s', '--sites-list',
action='store_true', dest='siteslist', default=False,
help='Get list of valid story URLs examples.', )
parser.add_option('-d', '--debug',
action='store_true', dest='debug',
help='Show debug output while downloading.', )
options, args = parser.parse_args(argv)
if not options.debug:
logger = logging.getLogger('fff_internals')
logger.setLevel(logging.INFO)
if not options.siteslist and len(args) != 1:
parser.error('incorrect number of arguments')
if options.siteslist:
for site, examples in adapters.getSiteExamples():
print '\n====%s====\n\nExample URLs:' % site
for u in examples:
print ' * %s' % u
return
if options.update and options.format != 'epub':
parser.error('-u/--update-epub only works with epub')
# Attempt to update an existing epub.
chaptercount = None
output_filename = None
if options.update:
try:
url, chaptercount = get_dcsource_chaptercount(args[0])
if not url:
print 'No story URL found in epub to update.'
return
print 'Updating %s, URL: %s' % (args[0], url)
output_filename = args[0]
except Exception:
# if there's an error reading the update file, maybe it's a URL?
# we'll look for an existing outputfile down below.
url = args[0]
else:
url = args[0]
try:
configuration = Configuration(adapters.getConfigSectionFor(url), options.format)
except exceptions.UnknownSite, e:
if options.list or options.normalize:
# list for page doesn't have to be a supported site.
configuration = Configuration('test1.com', options.format)
else:
raise e
conflist = []
homepath = join(expanduser('~'), '.fanficdownloader')
## also look for .fanficfare now, give higher priority than old dir.
homepath2 = join(expanduser('~'), '.fanficfare')
if passed_defaultsini:
configuration.readfp(passed_defaultsini)
if isfile(join(homepath, 'defaults.ini')):
conflist.append(join(homepath, 'defaults.ini'))
if isfile(join(homepath2, 'defaults.ini')):
conflist.append(join(homepath2, 'defaults.ini'))
if isfile('defaults.ini'):
conflist.append('defaults.ini')
if passed_personalini:
configuration.readfp(passed_personalini)
if isfile(join(homepath, 'personal.ini')):
conflist.append(join(homepath, 'personal.ini'))
if isfile(join(homepath2, 'personal.ini')):
conflist.append(join(homepath2, 'personal.ini'))
if isfile('personal.ini'):
conflist.append('personal.ini')
if options.configfile:
conflist.extend(options.configfile)
logging.debug('reading %s config file(s), if present' % conflist)
configuration.read(conflist)
try:
configuration.add_section('overrides')
except ConfigParser.DuplicateSectionError:
pass
if options.force:
configuration.set('overrides', 'always_overwrite', 'true')
if options.update and chaptercount:
configuration.set('overrides', 'output_filename', output_filename)
if options.update and not options.updatecover:
configuration.set('overrides', 'never_make_cover', 'true')
# images only for epub, even if the user mistakenly turned it
# on else where.
if options.format not in ('epub', 'html'):
configuration.set('overrides', 'include_images', 'false')
if options.options:
for opt in options.options:
(var, val) = opt.split('=')
configuration.set('overrides', var, val)
if options.list or options.normalize:
retlist = get_urls_from_page(args[0], configuration, normalize=options.normalize)
print '\n'.join(retlist)
return
try:
adapter = adapters.getAdapter(configuration, url)
adapter.setChaptersRange(options.begin, options.end)
# check for updating from URL (vs from file)
if options.update and not chaptercount:
try:
writer = writers.getWriter('epub', configuration, adapter)
output_filename = writer.getOutputFileName()
noturl, chaptercount = get_dcsource_chaptercount(output_filename)
print 'Updating %s, URL: %s' % (output_filename, url)
except Exception:
options.update = False
pass
# Check for include_images without no_image_processing. In absence of PIL, give warning.
if adapter.getConfig('include_images') and not adapter.getConfig('no_image_processing'):
try:
from calibre.utils.magick import Image
logging.debug('Using calibre.utils.magick')
except ImportError:
try:
import Image
logging.debug('Using PIL')
except ImportError:
print "You have include_images enabled, but Python Image Library(PIL) isn't found.\nImages will be included full size in original format.\nContinue? (y/n)?"
if not sys.stdin.readline().strip().lower().startswith('y'):
return
# three tries, that's enough if both user/pass & is_adult needed,
# or a couple tries of one or the other
for x in range(0, 2):
try:
adapter.getStoryMetadataOnly()
except exceptions.FailedToLogin, f:
if f.passwdonly:
print 'Story requires a password.'
else:
print 'Login Failed, Need Username/Password.'
sys.stdout.write('Username: ')
adapter.username = sys.stdin.readline().strip()
adapter.password = getpass.getpass(prompt='Password: ')
# print('Login: `%s`, Password: `%s`' % (adapter.username, adapter.password))
except exceptions.AdultCheckRequired:
print 'Please confirm you are an adult in your locale: (y/n)?'
if sys.stdin.readline().strip().lower().startswith('y'):
adapter.is_adult = True
if options.update and not options.force:
urlchaptercount = int(adapter.getStoryMetadataOnly().getMetadata('numChapters'))
if chaptercount == urlchaptercount and not options.metaonly:
print '%s already contains %d chapters.' % (output_filename, chaptercount)
elif chaptercount > urlchaptercount:
print '%s contains %d chapters, more than source: %d.' % (output_filename, chaptercount, urlchaptercount)
elif chaptercount == 0:
print "%s doesn't contain any recognizable chapters, probably from a different source. Not updating." % output_filename
else:
# update now handled by pre-populating the old
# images and chapters in the adapter rather than
# merging epubs.
url, chaptercount, adapter.oldchapters, adapter.oldimgs, adapter.oldcover, adapter.calibrebookmark, adapter.logfile = get_update_data(output_filename)
print 'Do update - epub(%d) vs url(%d)' % (chaptercount, urlchaptercount)
if not options.update and chaptercount == urlchaptercount and adapter.getConfig('do_update_hook'):
adapter.hookForUpdates(chaptercount)
write_story(configuration, adapter, 'epub')
else:
# regular download
if options.metaonly:
pprint.pprint(adapter.getStoryMetadataOnly().getAllMetadata())
output_filename = write_story(configuration, adapter, options.format, options.metaonly)
if not options.metaonly and adapter.getConfig('post_process_cmd'):
metadata = adapter.story.metadata
metadata['output_filename'] = output_filename
call(string.Template(adapter.getConfig('post_process_cmd')).substitute(metadata), shell=True)
del adapter
except exceptions.InvalidStoryURL, isu:
print isu
except exceptions.StoryDoesNotExist, dne:
print dne
except exceptions.UnknownSite, us:
print us
if __name__ == '__main__':
main()

116
setup.py Normal file
View File

@ -0,0 +1,116 @@
"""A setuptools based setup module.
See:
https://packaging.python.org/en/latest/distributing.html
https://github.com/pypa/sampleproject
"""
# Always prefer setuptools over distutils
from setuptools import setup, find_packages
# To use a consistent encoding
from codecs import open
from os import path
here = path.abspath(path.dirname(__file__))
# Get the long description from the relevant file
with open(path.join(here, 'DESCRIPTION.rst'), encoding='utf-8') as f:
long_description = f.read()
setup(
name="FanFicFare",
# Versions should comply with PEP440. For a discussion on single-sourcing
# the version across setup.py and the project code, see
# https://packaging.python.org/en/latest/single_source_version.html
version="2.2.0.dev6",
description='A tool for downloading fanfiction to eBook formats',
long_description=long_description,
# The project's main homepage.
url='https://github.com/JimmXinu/fanficdownloader',
# Author details
author='Jim Miller',
author_email='retiefjimm@gmail.com',
# Choose your license
license='Apache License',
# See https://pypi.python.org/pypi?%3Aaction=list_classifiers
classifiers=[
# How mature is this project? Common values are
# 3 - Alpha
# 4 - Beta
# 5 - Production/Stable
'Development Status :: 3 - Alpha',
'Environment :: Console',
# Indicate who your project is intended for
'Intended Audience :: End Users/Desktop',
'Topic :: Internet :: WWW/HTTP',
# Pick your license as you wish (should match "license" above)
'License :: OSI Approved :: Apache Software License',
# Specify the Python versions you support here. In particular, ensure
# that you indicate whether you support Python 2, Python 3 or both.
# 'Programming Language :: Python :: 2',
# 'Programming Language :: Python :: 2.6',
'Programming Language :: Python :: 2.7',
# 'Programming Language :: Python :: 3',
# 'Programming Language :: Python :: 3.2',
# 'Programming Language :: Python :: 3.3',
# 'Programming Language :: Python :: 3.4',
],
# What does your project relate to?
keywords='fanfiction download ebook epub',
# You can just specify the packages manually here if your project is
# simple. Or you can use find_packages().
# packages=find_packages(exclude=['contrib', 'docs', 'tests*']),
packages=['fff_internals', 'fff_internals.adapters', 'fff_internals.writers'],
# List run-time dependencies here. These will be installed by pip when
# your project is installed. For an analysis of "install_requires" vs pip's
# requirements files see:
# https://packaging.python.org/en/latest/requirements.html
install_requires=['beautifulsoup4','chardet','six','html5lib'],
# List additional groups of dependencies here (e.g. development
# dependencies). You can install these using the following syntax,
# for example:
# $ pip install -e .[dev,test]
extras_require={
# 'dev': ['check-manifest'],
# 'test': ['coverage'],
},
# If there are data files included in your packages that need to be
# installed, specify them here. If using Python 2.6 or less, then these
# have to be included in MANIFEST.in as well.
package_data={
'defaults.ini': ['defaults.ini'],
'example.ini': ['example.ini'],
},
# Although 'package_data' is the preferred approach, in some case you may
# need to place data files outside of your packages. See:
# http://docs.python.org/3.4/distutils/setupscript.html#installing-additional-files # noqa
# In this case, 'data_file' will be installed into '<sys.prefix>/my_data'
# data_files=[('my_data', ['data/data_file'])],
# To provide executable scripts, use entry points in preference to the
# "scripts" keyword. Entry points provide cross-platform support and allow
# pip to create the appropriate form of executable for the target platform.
entry_points={
'console_scripts': [
'fanficfare=fff_internals.cli:main',
],
},
)