Archiveteam/grab-site
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

README: wrap lines

Browse Source
This commit is contained in:
Ivan Kozik 2018-10-05 16:18:06 +00:00
parent ea4e4eff74
commit d3da0899ad
1 changed files with 6 additions and 3 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

9
README.md
View File

@ -398,9 +398,11 @@ hiding the page content with CSS. You are still likely to get a complete crawl.
#### Subreddits
Use [`--igsets=reddit`](https://github.com/ludios/grab-site/blob/master/libgrabsite/ignore_sets/reddit) and add a `/` at the end of the URL to avoid crawling all subreddits.
Use [`--igsets=reddit`](https://github.com/ludios/grab-site/blob/master/libgrabsite/ignore_sets/reddit)
and add a `/` at the end of the URL to avoid crawling all subreddits.
When crawling a subreddit, you **must** get the casing of the subreddit right for the recursive crawl to work. For example,
When crawling a subreddit, you **must** get the casing of the subreddit right
for the recursive crawl to work. For example,
```
grab-site https://www.reddit.com/r/Oculus/ --igsets=reddit
@ -416,7 +418,8 @@ You can hover over the "Hot"/"New"/... links at the top of the page to see the c
#### Directory listings ("Index of ...")
Use `--no-dupespotter` to avoid triggering false positives on the duplicate page detector. Without it, the crawl may miss large parts of the directory tree.
Use `--no-dupespotter` to avoid triggering false positives on the duplicate
page detector. Without it, the crawl may miss large parts of the directory tree.
#### Very large websites