ryanbr
/
fanboy-adblock
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

Famlan redunancy checker backup

master
fanboynz 2018-10-28 05:08:11 +01:00
parent 67e9476286
commit 06c3362a08
16 changed files with 7652 additions and 0 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

2179
' Normal file
View File

File diff suppressed because it is too large Load Diff

230
famlam-redundancychecker/changelog.html Normal file
View File

@ -0,0 +1,230 @@
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="UTF-8">
<meta http-equiv="Content-Security-Policy" content="default-src 'self';" />
<meta name="viewport" content="width=device-width, initial-scale=1" />
<title>Redundancy check - changelog</title>
<link rel="stylesheet" href="redundantRuleChecker.css" />
</head>
<body id="changelog">
<h3>Changelog</h3>
<p id="topmsg">This page lists the changes to the redundancy checker, if said changes had an effect on the behaviour of the check</p>
<h4>May 23, 2015</h4>
<ul>
<li><b>Fix</b>whitelist filters that start with a protocol no longer imply <samp>$document</samp> in the next version of ABP (example: <samp>@@|http://x.y/</samp>) </li>
<!--li><b>Fix</b>domain check extension: pages that take over 2 minutes to load are not considered unresponsive anymore</li-->
</ul>
<h4>May 17, 2015</h4>
<ul>
<li><b>Modified</b>the future options <samp>$generichide</samp> and <samp>$genericblock</samp> are still ignored, but no longer trigger error messages</li>
<li><b>UI</b>minor changes for mobile browsers</li>
</ul>
<h4>March 9, 2015</h4>
<ul>
<li><b>New</b>domain check tool: add option to export the current results, which can also be used to resume the check at a later moment</li>
<li><b>Fix</b>domain check tool: spaces were removed from filters</li>
</ul>
<h4>March 8, 2015</h4>
<ul>
<li><b>New</b>tool to detect dead, redirected or parked first party domains (requires Chrome)</li>
<li><b>Fix</b>sometimes multiple warnings were shown per rule, instead of only the most important warning (<samp>$domain=a.|b|a.</samp>)</li>
<li><b>Modified</b>tightened the triggering conditions for the warning about a blocking rule being a broken hiding exclusion rule</li>
</ul>
<h4>January 5, 2015</h4>
<ul>
<li><b>New</b>warning if the value of a blocking option contains a <samp>$</samp> which should probably be a <samp>,</samp> (<samp>$domain=abc$image</samp>)</li>
<li><b>Improved</b>warning messages when invalid domain separators are found (<samp>$domain=a||b</samp>)</li>
<li><b>Improved</b>warning messages related to incorrectly used blocking options (<samp>$image=foo</samp> or <samp>$~domain=foo</samp>)</li>
<li><b>Improved</b>warning messages related to overruled blocking options (<samp>$image,~image,script</samp>)</li>
<li><b>Fix</b><samp>$document</samp> rules did not take into account the trailing dot domain variant for blocking rules with <samp>$domain=...</samp> (<samp>@@||abc.def^$document</samp> and <samp>$domain=abc.def</samp>)</li>
<li><b>Fix</b>multiple warnings with shared rules should only be filtered out if all rules are identical (<samp>@@||a^$elemhide</samp> should warn against both <samp>c,a##.adv</samp> and <samp>a,b##.ads</samp> at the same time)</li>
<li><b>Fix</b>warnings for a mutually redundant, equal length rule were dependent on the position of the other rule in the filters (<samp>##.A.\30 .B</samp> and <samp>##.A.B.\30 </samp>)</li>
<li><b>Fix</b><samp>:not(...)</samp> selectors were not checked for conflicts when '<samp>not</samp>' contained capital letters (<samp>##div:NOT(div)</samp>)</li>
<li><b>Fix</b>whitelisted rules containing the <samp>$collapse</samp> option did not make blocking rules without <samp>$collapse</samp> redundant</li>
<li><b>Fix</b><samp>$match-case,~match-case</samp> was treated as <samp>$match-case</samp></li>
<li><b>Fix</b>whitelist rules that imply <samp>$document</samp> were made redundant by rules that do not imply <samp>$document</samp> if no options were present (<samp>@@|http://</samp> was made redundant by <samp>@@http</samp>)</li>
<li><b>Modified</b>ipv6 addresses are now also accepted after a ||, although I don't know how ABP parses them (<samp>||[1234:123:1:12::123]^</samp>)</li>
<li><b>Modified</b>known options are now shown uniformly in warning messages (lowercased, without spaces and using underscores)</li>
</ul>
<h4>July 7, 2014:</h4>
<ul>
<li><b>New</b>warn about rules with contradicting <samp>:nth-*</samp> selectors as they cannot match anything (<samp>##:nth-child(3):nth-child(4)</samp>)</li>
<li><b>New</b>warn about selectors with different values for attribute selectors with the operators <samp>^=</samp> and <samp>|=</samp>, as they cannot match anything (<samp>##[attr^=&quot;cow&quot;][attr|=&quot;dog&quot;]</samp>)</li>
<li><b>New</b>warn about rules containing <samp>:not(...)</samp> pseudo-classes that cannot coexist with other selector types in the same rule (<samp>##:not([class=&quot;ads&quot;]).adv</samp>)</li>
<li><b>New</b>detect redundancies between rules containing <samp>:not(...)</samp> pseudo-classes and rules containing other selector types (<samp>##:not([class=&quot;ads&quot;])</samp> versus <samp>##.adv</samp>)</li>
<li><b>New</b>rules containing <samp>:nth-*</samp> pseudo-classes are compared numerically, instead of literally (<samp>##:nth-child(3)</samp> and <samp>##:nth-child(2n+1)</samp>)</li>
<li><b>New</b>rules containing <samp>:not(...)</samp> pseudo-classes are no longer only literally compared against other <samp>:not(...)</samp> selectors (<samp>##:not([href*=&quot;a&quot;])</samp> and <samp>##:not([href=&quot;ads&quot;])</samp>)</li>
<li><b>Improved</b>the warning messages that tell you about conflicts within a rule are more specific (<samp>##[id=&quot;a&quot;][id^=&quot;b&quot;]</samp>)</li>
<li><b>Improved</b>the warning messages that tell you about redundancies within a rule are more specific (<samp>###ads,div#ads</samp>)</li>
<li><b>Fix</b>attribute selectors with <samp>|=</samp> made attribute selectors with <samp>^=</samp> redundant if the attribute values were identical (<samp>##a[abc^=&quot;abc&quot;]</samp> and <samp>##[abc|=&quot;abc&quot;]</samp>)</li>
<li><b>Fix</b>combined hiding rules with internal redundancies could not be made redundant by non-combined rules (<samp>###ads,#ads</samp> and <samp>###ads</samp>)</li>
<li><b>Fix</b><samp>:not(*)</samp> shouldn't make rules with different namespaces redundant (<samp>##ns|*:not(*)</samp>)</li>
<li><b>Fix</b><samp>:nth-*(An+B)</samp> selectors with <samp>A</samp> or <samp>B</samp> very large numbers still suffered from rounding errors</li>
<li><b>Fix</b>internally ignored <samp>:not(...)</samp> selectors acted as the universal selector <samp>*</samp> (<samp>##:not(:nth-child(9999999999999999))</samp> made <samp>##div</samp> redundant)</li>
<li><b>Modified</b>attribute selector combinations that can't coexist do not match other impossible combinations anymore (<samp>###a#b</samp> and <samp>##[id=&quot;a&quot;][id^=&quot;b&quot;]</samp>)</li>
<li><b>Modified</b>rewrite the way hiding rules are parsed</li>
</ul>
<h4>January 19, 2014:</h4>
<ul>
<li><b>New</b>warn about redundant excluded domains in case the least restrictive rule is site-specific (<samp>~site.com</samp> in <samp>site.com##x</samp> and <samp>~site.com##x.y</samp>)</li>
<li><b>Improved</b>up to 30% increase of speed</li>
<li><b>Removed</b>the tab with ignored rules is removed as it was incomplete and didn't distinguish between fully and partially ignored rules</li>
<li><b>Fix</b><samp>:nth-child(An+B)</samp>/<samp>:nth-last-child(An+B)</samp>/<samp>:nth-of-type(An+B)</samp>/<samp>:nth-last-of-type(An+B)</samp> selectors with <samp>A</samp> or <samp>B</samp> very large numbers suffered from rounding errors</li>
<li><b>UI</b>add documentation about the limitations of the redundancy checker</li>
<li><b>UI</b>fix some typos</li>
</ul>
<h4>January 13, 2014:</h4>
<ul>
<li><b>New</b>warn if <samp>,$</samp> is used to separate rules (example: <samp>$script,$domain=com</samp>)</li>
<li><b>New</b>detect redundancies in hiding rules if a <samp>#@#</samp> rule excluded part of the rule (example: <samp>nl#@##ads</samp> and <samp>###ads</samp> together with <samp>com##div#ads</samp>)</li>
<li><b>Improved</b>all warnings for redundant domains now clearly state which domains can be removed</li>
<li><b>Removed</b>warnings about domain redundancies with non-redundant subdomains (example: <samp>~b.a</samp> in <samp>b.a,~c.b.a#@#ads</samp> and <samp>a,~b.a##ads</samp>)</li>
<li><b>Fix</b><samp>/\|\|x/$document</samp> and similar regex rules with <samp>$document</samp> or <samp>$elemhide</samp> no longer make <samp>$domain=x</samp> or <samp>x##y</samp> redundant</li>
<li><b>UI</b>in case a rule could be made redundant by two rules, and those rules are no redundancies of each other, the shortest rule is now displayed</li>
<li><b>UI</b>tell the user to first check for redundancies again before using the tools, if they modify the contents of the text box</li>
<li><b>UI</b>some changes in the warning messages of simplified syntax hiding rules</li>
</ul>
<h4>November 16, 2013:</h4>
<ul>
<li><b>New</b>warning about <samp>$image,~image,popup</samp>, where <samp>$image,~image</samp> does nothing. ABP now drops unknown options, so those constructions are unneeded</li>
<li><b>New</b>warning about <samp>$~match-case</samp>, since it's the default to be case insensitive (<samp>/ads.$~match-case</samp>)</li>
<li><b>New</b>warn when pseudo-element selectors are present, since they cannot be used for hiding content (<samp>##::before</samp>)</li>
<li><b>New</b>CSS selector identifiers containing escaped Unicode at the end of the identifier may have a trailing space (<samp>##\32 &gt;x</samp> is equal to <samp>##\32&gt;x</samp>)</li>
<li><b>Improved</b>warning about redundant type options if the first type option is an excluded type (<samp>$~script,image,~image</samp>)</li>
<li><b>Improved</b>specify the location of an unnecessary universal CSS selector in warning messages (only the first <samp>*</samp> in <samp>##*.class &gt; *</samp>)</li>
<li><b>Improved</b>more (all?) redundant sequences of complex tree selectors can now be detected (<samp>##a &gt; div div &gt; x</samp> and <samp>##a &gt; div div div &gt; x</samp>)</li>
<li><b>Fix</b>if A is made redundant by B and B is made redundant by a <samp>$document</samp> or <samp>$elemhide</samp> rule C, A must be redundant of C instead of B</li>
<li><b>Fix</b>more warning messages in which the options were shown as lowercased and with all <samp>-</samp> replaced by <samp>_</samp></li>
<li><b>Fix</b>incorrect warnings for <samp>$~elemhide</samp> and <samp>$~document</samp> (<samp>@@/ads.$~elemhide,document</samp>)</li>
<li><b>Fix</b>crash of the redundancy checker if a hiding rule matched nothing and also contained an internal duplicate (<samp>##:not(*):not(*)</samp>)</li>
<li><b>Fix</b>rules with duplicates of <samp>:not(...)</samp> selectors and another comma-separated selector, acted as if <samp>:not(...)</samp> was <samp>*</samp> (<samp>##:not(a):not(a ),x</samp>)</li>
<li><b>Fix</b>CSS selectors with <samp>:lang(x-y)</samp> would skip the selector behind the <samp>:lang(...)</samp> selector (<samp>##:lang(en-gb)&gt;ads</samp>)</li>
<li><b>Modified</b>rewrite the way the options of blocking rules are parsed</li>
<li><b>Modified</b>rewrite the way how rules containing tree selectors ('<samp> </samp>', <samp>&gt;</samp>, <samp>+</samp>, <samp>~</samp>) are matched</li>
<li><b>UI</b>fix typo in warning message for <samp>:not(...)</samp> selectors that match every element (<samp>##:not(:nth-child(-4))</samp>)</li>
</ul>
<h4>September 20, 2013:</h4>
<ul>
<li><b>Improved</b>'same rule, different domains' tool is faster and finds more combinable rules</li>
<li><b>Improved</b>better warning message for rules of the form <samp>@@domain##rule</samp></li>
<li><b>Improved</b>better error message for incorrectly excluded domains (<samp>abc$~domain=site.com</samp>)</li>
<li><b>Fix</b>in the error message <em>unknown option 'X'</em>, any <samp>-</samp> in the option was replaced by <samp>_</samp>, and it was in lowercase (example: <samp>abc$First-Party</samp>)</li>
</ul>
<h4>July 18, 2013:</h4>
<ul>
<li><b>Fix</b>'same rule, different domains' tool suggested combining <samp>a##bar,baz</samp> and <samp>b##bar+baz</samp></li>
<li><b>Fix</b>rare case where the <em>Check for redundant rules</em> button didn't get enabled</li>
</ul>
<h4>June 12, 2013:</h4>
<ul>
<li><b>Fix</b>Punycode-converted domains are considered valid now (<samp>xn----etbbchqbn2afauadx.xn--p1ai##.ads</samp>)</li>
</ul>
<h4>May 31, 2013:</h4>
<ul>
<li><b>Fix</b>tool to find whitelists shouldn't report other whitelists that it matches (<samp>@@x/xml</samp> and <samp>@@/xml$image</samp>)</li>
<li><b>Fix</b>tool to convert hiding into blocking reports wrong results for <samp>\=</samp> in the attribute name (<samp>##[src\=='hello']</samp>)</li>
<li><b>Fix</b>tool to use a less strict matching method matched two characters that would both be matched by <samp>^</samp> (<samp>&amp;adType=</samp> and <samp>?adType=</samp>)</li>
<li><b>Improved</b>code improvements</li>
</ul>
<h4>May 20, 2013:</h4>
<ul>
<li><b>New</b>tool to find hiding rules that could potentially become blocking rules</li>
<li><b>Fix</b>incorrect warning message for rules that start with <samp>*@@</samp>: they should keep their preceding wildcard</li>
<li><b>Improved</b>matching of the <samp>:lang(...)</samp> selector</li>
</ul>
<h4>May 6, 2013:</h4>
<ul>
<li><b>Improved</b>matching of <samp>first/last/nth/only-child</samp> and <samp>first/last/nth/only-of-type</samp> selectors</li>
</ul>
<h4>April 28, 2013:</h4>
<ul>
<li><b>Fix</b>incorrect warning message: selectors that do not match anything, actually match everything when in a <samp>:not()</samp> selector (example: <samp>##:not(:nth-child(-4))</samp>)</li>
<li><b>Fix</b>pseudo-element CSS selectors are not allowed in <samp>:not()</samp> selectors in CSS3 (<samp>##:not(::before)</samp>)</li>
<li><b>Fix</b>combinations of CSS selectors are not allowed in <samp>:not()</samp> selectors in CSS3 (<samp>##:not(##div#ads)</samp>)</li>
<li><b>Fix</b>namespaces in CSS3 are only allowed in front of nodename selectors (<samp>##abc|#def</samp>)</li>
<li><b>Fix</b>in an internal rule redundancy, the most restrictive part is the active part (<samp>##[a^=&quot;b&quot;][a^=&quot;bcd&quot;]</samp> and <samp>##[a^=&quot;bc&quot;]</samp>)</li>
<li><b>Fix</b>incorrect warning message for comma-separated hiding rules for which one part can't match anything (<samp>##div,:not(*)</samp>)</li>
<li><b>Fix</b>do not automatically assume case insensitivity for browser-specific pseudo selectors (<samp>##:moz-whatever(x)</samp> versus <samp>##:moz-whatever(X)</samp>)</li>
<li><b>Modified</b>rules that match nothing (example: <samp>$image,~image</samp>) won't be made redundant by every other rule anymore; a warning is sufficient</li>
<li><b>Modified</b>prefer the message that <samp>donottrack</samp> is deprecated over one that the rule doesn't match anything</li>
<li><b>Modified</b>rewrite the way hiding rules are parsed. This method should be more fail-proof.</li>
<li><b>UI</b>fix typo in warning message</li>
</ul>
<h4>April 14, 2013:</h4>
<ul>
<li><b>Modified</b>reworked the warnings 'management' system so that it always returns the most important warning instead of the last found warning per rule</li>
<li><b>Improved</b>warnings about redundancies within a rule (example: <samp>##[a=&quot;bcd&quot;][a^=&quot;bc&quot;]</samp>)</li>
<li><b>Improved</b>up to 5% increase of speed</li>
<li><b>Fix</b>incorrect warnings about redundancies within a rule if namespaces in attribute selectors were present (<samp>##[ns1|abc=&quot;def&quot;][ns2|abc=&quot;def&quot;]</samp>)</li>
<li><b>Fix</b>disable warnings about selectors that cannot co-exist when a namespace selector is present (<samp>##a|#b[*|id^='b']</samp>), because I'm unsure about the validity of that warning</li>
<li><b>Fix</b>incorrect <em>broken selector</em> warning if a rule contains Unicode characters <samp>00A0</samp> to <samp>0177</samp></li>
<li><b>UI</b>fix bad rule highlighting when a filter with a warning contained '<samp> and </samp>' (example: <samp>##*[ and $=&quot; and &quot;]</samp>)</li>
<li><b>UI</b>warn when a browser lacks some functionality</li>
</ul>
<h4>March 28, 2013:</h4>
<ul>
<li><b>Improved</b>up to 20% increase of speed</li>
</ul>
<h4>March 26, 2013:</h4>
<ul>
<li><b>Fix</b>treat every line containing <samp>*[adblock*]*</samp> as comment</li>
<li><b>Fix</b>hiding rules containing <samp>##tag-or-class-or-id\::known-pseudoselector</samp> are valid CSS (example: <samp>###mysite\::first-child</samp>)</li>
<li><b>Fix</b>duplicate warning if an excluded domain in a hiding rule was also an included domain in a hiding whitelist rule (example: <samp>a,b,~b.a##x</samp> and <samp>a#@#x</samp>)</li>
<li><b>Fix</b>hiding rules excluded by a <samp>#@#</samp> rule do not make other rules redundant anymore (<samp>###ads</samp> and <samp>foo.com#@##ads</samp> and <samp>foo.com###ads.banner</samp>)</li>
<li><b>Fix</b>rules with included domains, excluded subdomains and included sub-subdomains weren't shown as redundant for the last subdomain (<samp>a,~b.a,c.b.a##x</samp> versus <samp>c.b.a##x.y</samp>)</li>
<li><b>Fix</b>simplified hiding rules suggest the <samp>#id</samp> and <samp>.class</samp> syntax, instead of <samp>[id=&quot;the_id&quot;]</samp> and <samp>[class~=&quot;the_class&quot;]</samp></li>
<li><b>Fix</b>warnings involving the same rule twice were removed (<samp>##[id=&quot;x&quot;]</samp> and <samp>##[id=&quot;x&quot;]</samp>)</li>
<li><b>New</b>warn if you have an rule with only excluded domains and a rule with partially redundant included domains (<samp>~b.a##x</samp> vs <samp>z,c.b.a##x</samp>)</li>
<li><b>New</b>if a hiding rule matches a parent element of another hiding rule, the latter is redundant (<samp>###foo</samp> versus <samp>###foo &gt; #bar</samp>)</li>
<li><b>New</b>hiding rules may skip a tree depth if their selector allows so (<samp>###foo &gt; bar &gt; baz</samp> versus <samp>###foo baz</samp>)</li>
<li><b>New</b>warn if you have selectors that can't co-exist (<samp>###id1#id2</samp>, <samp>##[a=&quot;b&quot;][a=&quot;c&quot;]</samp>, ...)</li>
<li><b>New</b>warn if comma-separated hiding rules make each other redundant (<samp>###ads,.xyz#ads</samp>)</li>
<li><b>Improved</b>warn if you have multiple included domains and some of them match (example: <samp>foo,a.bar##a</samp> versus <samp>bar,baz##a</samp>)</li>
<li><b>Improved</b>support for comma-separated hiding rules (<samp>domain.com###ads,#adv</samp>)</li>
<li><b>UI</b>for errors reported (for broken rules, like <samp>##rule]</samp>), an <samp>!</samp> was shown in front of the message</li>
<li><b>UI</b>replace contact link with information link</li>
<li><b>UI</b>disable spellchecking of the filter list input area</li>
</ul>
<h4>January 31, 2013:</h4>
<ul>
<li><b>New</b>add several tools:
<ul>
<li>A similar rules finder tool</li>
<li>Tools to ignore domains and blocking options (replaces the checkbox)</li>
<li>A tool to use a less strict matching algorithm</li>
<li>A tool to find rules which have the same rule, but a different domain</li>
<li>A tool to find the rules that make whitelisting rules necessary, which also displays if no rules could be found for a whitelisting rule</li>
</ul>
</li>
<li><b>New</b><samp>||x</samp> will now also be matched if both <samp>/x</samp> and <samp>.x</samp> are present</li>
<li><b>Modified</b>deprecate <samp>$donottrack</samp>, since it's removed from ABP too</li>
<li><b>Fix</b><samp>//</samp> isn't a regex</li>
<li><b>Fix</b>'<samp> !x</samp>' (with a whitespace in front of it) is a comment, not a blocking rule</li>
<li><b>Fix</b><samp>*!x</samp> and similar will no longer trigger <em>unnecessary preceding wildcard found</em> warnings, since it would become a comment without <samp>*</samp></li>
<li><b>Fix</b><samp>/\|\|x/</samp> and similar regex rules no longer make <samp>||x</samp> redundant</li>
</ul>
<h4>No changelog available before 2013.</h4>
</body>
</html>

128
famlam-redundancychecker/changelog.txt Normal file
View File

@ -0,0 +1,128 @@
January 13, 2014:
-New: warn if ',$' is used to separate rules (example: $script,$domain=com)
-New: detect redundancies in hiding rules if a #@# rule excluded part of the rule (example: nl#@##ads and ###ads together with com##div#ads)
-Improved: all warnings for redundant domains now clearly state which domains can be removed
-Removed: warnings about domain redundancies with non-redundant subdomains (example: ~b.a in b.a,~c.b.a#@#ads and a,~b.a##ads)
-Fix: '/\|\|x/$document' and similar regex rules with $document or $elemhide no longer make '$domain=x' or 'x##y' redundant
-UI: in case a rule could be made redundant by two rules, and those rules are no redundancies of each other, the shortest rule is now displayed
-UI: tell the user to first check for redundancies again before using the tools, if they modify the contents of the text box
-UI: some changes in the warning messages of simplified syntax hiding rules
November 16, 2013:
-New: warning about $image,~image,popup, where $image,~image does nothing. ABP now drops unknown options, so those constructions are unneeded
-New: warning about $~match-case, since it's the default to be case insensitive (/ads.$~match-case)
-New: warn when pseudo-element selectors are present, since they cannot be used for hiding content (##::before)
-New: CSS selector identifiers containing escaped unicode at the end of the identifier may have a trailing space (##\32 >x is equal to ##\32>x)
-Improved: warning about redundant type options if the first type option is an excluded type ($~script,image,~image)
-Improved: specify the location of an unnecessary universal CSS selector in warning messages (only the first * in ##*.class > * )
-Improved: more (all?) redundant sequences of complex tree selectors can now be detected (##a > div div > x and ##a > div div div > x)
-Fix: if A is made redundant by B and B is made redundant by a $document or $elemhide rule C, A must be redundant of C instead of B
-Fix: more warning messages in which the options were shown as lowercased and with all - replaced by _
-Fix: incorrect warnings for $~elemhide and $~document (@@/ads.$~elemhide,document)
-Fix: crash of the redundancy checker if a hiding rule matched nothing and also contained an internal duplicate (##:not(*):not(*) )
-Fix: rules with duplicates of :not(...) selectors and another comma-separated selector, acted as if ':not(...)' was '*' (##:not(a):not(a ),x )
-Fix: CSS selectors with :lang(x-y) would skip the selector behind the :lang(...) selector (##:lang(en-gb)>ads)
-Modified: rewrite the way the the options of blocking rules are parsed
-Modified: rewrite the way how rules containing tree selectors (' ', >, +, ~) are matched
-UI: fix typo in warning message for :not(...) selectors that match every element (##:not(:nth-child(-4)) )
September 20, 2013:
-Improved: 'same rule, different domains' tool is faster and finds more combinable rules
-Improved: better warning message for rules of the form '@@domain##rule'
-Improved: better error message for incorrectly excluded domains (abc$~domain=site.com)
-Fix: in the error message "unknown option 'X'", any '-' in the option was replaced by '_', and it was in lowercase (example: abc$First-Party)
July 18, 2013:
-Fix: 'same rule, different domains' tool suggested combining a##bar,baz and b##bar+baz
-Fix: rare case where the 'Check for redundant rules' button didn't get enabled
June 12, 2013:
-Fix: punycode-converted domains are considered valid now (xn----etbbchqbn2afauadx.xn--p1ai##.ads)
May 31, 2013:
-Fix: tool to find whitelists shouldn't report other whitelists that it matches (@@x/xml and @@/xml$image)
-Fix: tool to convert hiding into blocking reports wrong results for \= in the attribute name ( ##[src\=='hello'] )
-Fix: tool to use a less strict matching method matched two characters that would both be matched by ^ ( &adType= and ?adType= )
-Improved: code improvements
May 20, 2013:
-New: tool to find hiding rules that could potentially become blocking rules
-Fix: incorrect warning message for rules that start with '*@@': they should keep their preceeding wildcard
-Improved: matching of the :lang(...) selector
May 6, 2013:
-Improved: matching of first/last/nth/only-child and first/last/nth/only-of-type selectors
April 28, 2013:
-Fix: incorrect warning message: selectors that do not match anything, actually match everything when in a :not() selector (example: ##:not(:nth-child(-4)) )
-Fix: pseudo-element CSS selectors are not allowed in :not() selectors in CSS3 (##:not(::before) )
-Fix: combinations of CSS selectors are not allowed in :not() selectors in CSS3 (##:not(##div#ads) )
-Fix: namespaces in CSS3 are only allowed in front of nodename selectors (##abc|#def)
-Fix: in an internal rule redundancy, the most restrictive part is the active part (##[a^="b"][a^="bcd"] and ##[a^="bc"] )
-Fix: incorrect warning message for comma-separated hiding rules for which one part can't match anything (##div,:not(*) )
-Fix: do not automatically assume case insensitivity for browser-specific pseudo selectors (##:moz-whatever(x) versus ##:moz-whatever(X) )
-Modified: rules that match nothing (example: $image,~image) won't be made redundant by every other rule anymore; a warning is sufficient
-Modified: prefer the message that donottrack is depricated over one that the rule doesn't match anything
-Modified: rewrite the way hiding rules are parsed. This method should be more fail-proof.
-UI: fix typo in warning message
April 14, 2013:
- Modified: reworked the warnings 'management' system so that it always returns the most important warning instead of the last found warning per rule
- Improved: warnings about redundancies within a rule (example: ##[a="bcd"][a^="bc"])
- Improved: up to 5% increase of speed
- Fix: incorrect warnings about redundancies within a rule if namespaces in attribute selectors were present (##[ns1|abc="def"][ns2|abc="def"])
- Fix: disable warnings about selectors that cannot co-exist when a namespace selector is present (##a|#b[*|id^='b']), because I'm unsure about the validity of that warning
- Fix: incorrect 'broken selector' warning if a rule contains unicode characters 00A0 to 0177
- UI: fix bad rule highlighting when a filter with a warning contained " and " (##*[ and $=" and "])
- UI: warn when a browser lacks some functionality
March 28, 2013:
- Improved: up to 20% increase of speed
March 26, 2013:
- Fix: treat every line containing *[adblock*]* as comment
- Fix: hiding rules containing ##tag-or-class-or-id\::known-pseudoselector are valid CSS (example: ###mysite\::first-child)
- Fix: duplicate warning if an excluded domain in a hiding rule was also an included domain in a hiding whitelist rule (example: a,b,~b.a##x and a#@#x)
- Fix: hiding rules excluded by a #@# rule do not make other rules redundant anymore (###ads and foo.com#@##ads and foo.com###ads.banner)
- Fix: rules with included domains, excluded subdomains and included sub-subdomains weren't shown as redundant for the last subdomain (a,~b.a,c.b.a##x versus c.b.a##x.y)
- Fix: simplified hiding rules suggest the #id and .class syntax, instead of [id="the_id"] and [class~="the_class"]
- Fix: warnings involving the same rule twice were removed (##[id="x"] and ##[id="x"])
- New: warn if you have an rule with only excluded domains and a rule with partially redundant included domains (~b.a##x vs z,c.b.a##x)
- New: if a hiding rule matches a parent element of another hiding rule, the latter is redundant (###foo versus ###foo > #bar)
- New: hiding rules may skip a tree depth if their selector allows so (###foo > bar > baz versus ###foo baz)
- New: warn if you have selectors that can't co-exist (###id1#id2, ##[a="b"][a="c"], ...)
- New: warn if comma-separated hiding rules make each other redundant (###ads,.xyz#ads)
- Improved: warn if you have multiple included domains and some of them match (example: foo,a.bar##a versus bar,baz##a)
- Improved: support for comma-separated hiding rules (domain.com###ads,#adv)
- UI: for errors reported (for broken rules, like "##rule]" ), an "!" was shown in front of the message
- UI: replace contact link with information link
- UI: disable spellchecking of the filter list input area
January 31, 2013:
- New: add several tools:
* A similar rules finder tool
* Tools to ignore domains and blocking options (replaces the checkbox)
* A tool to use a less strict matching algorithm
* A tool to find rules which have the same rule, but a different domain
* A tool to find the rules that make whitelisting rules necessary, which also displays if no rules could be found for a whitelisting rule
- New: ||x will now also be matched if both /x and .x are present
- Modified: depricate $donottrack, since it's removed from ABP too
- Fix: '//' isn't a regex
- Fix: ' !x' (with a whitespace in front of it) is a comment, not a blocking rule
- Fix: '*!x' and similar will no longer trigger 'unnecessary preceeding wildcard found' warnings, since it would become a comment without *
- Fix: '/\|\|x/' and similar regex rules no longer make ||x redundant
No changelog available before 2013.

BIN
famlam-redundancychecker/famlam.tar.gz Normal file
View File

Binary file not shown.

67
famlam-redundancychecker/findWhitelistRules.js Normal file
View File

@ -0,0 +1,67 @@
/*!
This script searches for active whitelisting rules written in the Adblock Plus syntax,
documentated here: http://adblockplus.org/en/filters, and reports them.
Author: Famlam (fam.lam [at] live.nl)
This program is free software: you can redistribute it and/or modify
it under the terms of the GNU General Public License as published by
the Free Software Foundation, either version 3 of the License, or
(at your option) any later version.
This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
GNU General Public License for more details.
*/
"use strict";
var findWhitelistRules = function(lines) {
importScripts("redundant.js");
var ELEMHIDE = /^([^\/\*\|\@\"\!]*?)\#\s*(\@)?\s*\#([^\{\}]+)$/, /**/
BLOCKING = /^(@@)?(.*?)(\$~?[\w\-]+(?:=[^,\s]+)?(?:,~?[\w\-]+(?:=[^,\s]+)?)*)?$/, /**/
i, j,
startWorkerResults = startWorker({filters: lines, modifiers: {matchWhitelist: true, loosely: true}}, false, true),
resultByWhitelist = {},
resultByFilter = {};
// Show all whitelist rules, including the non-matching ones
lines = lines.split("\n");
for (i=0; i<lines.length; i++) {
if (ELEMHIDE.test(lines[i])) {
if (lines[i].match(ELEMHIDE)[2]) {
resultByWhitelist[lines[i]] = [];
}
} else if (lines[i].match(BLOCKING)[1]) {
resultByWhitelist[lines[i]] = [];
}
}
for (i in startWorkerResults.results) {
var splitted = startWorkerResults.results[i].split("\n");
if (ELEMHIDE.test(i)) {
if (!i.match(ELEMHIDE)[2]) {
continue;
}
} else if (!i.match(BLOCKING)[1]) {
continue;
} else {
for (j=splitted.length-1; j>=0; j--) {
if (splitted[j].match(BLOCKING)[1]) {
splitted.splice(i, 1);
}
}
}
resultByWhitelist[i] = resultByWhitelist[i].concat(splitted);
for (j=0; j<splitted.length; j++) {
resultByFilter[splitted[j]] = resultByFilter[splitted[j]] || [];
resultByFilter[splitted[j]].push(i);
}
}
self.postMessage({resultByFilter: resultByFilter, resultByWhitelist: resultByWhitelist});
self.close();
};
this.addEventListener("message", function(e) {
findWhitelistRules(e.data.filters);
}, false);

124
famlam-redundancychecker/generateDomainTable.js Normal file
View File

@ -0,0 +1,124 @@
/*!
This script searches for similar rules written in the Adblock Plus syntax,
documentated here: http://adblockplus.org/en/filters, and reports them.
Author: Famlam (fam.lam [at] live.nl)
This program is free software: you can redistribute it and/or modify
it under the terms of the GNU General Public License as published by
the Free Software Foundation, either version 3 of the License, or
(at your option) any later version.
This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
GNU General Public License for more details.
*/
"use strict";
var extractDomainsFromFilters = function(lines) {
importScripts("redundant.js");
startWorker({filters: ""}, false, true); // import it's "globals"
lines = lines.split("\n");
var i, j, domains, match, line, isThirdParty, firstParty,
knownThirdPartyDomains = [],
domainInfo = {},
ELEMHIDE = /^([^\/\*\|\@\"\!]*?)\#\s*(\@)?\s*\#([^\{\}]+)$/, /**/
B_DOMAINIS = /(?:\,|\$|^)domain\=([^\,]+)/i, /**/
B_THIRDPARTY = /(?:\,|\$)third[_\-]party(?:\,|$)/i,
WHITESPACE_G = /\s+/g, /**/
DOTEND = /\.$/, /**/
SUBDOMAIN = /^.+?(?:\.|$)/, /**/
B_DOMAINMATCH = /^(?:\|\||\|?[\w\-]+\:\/+)([^\^\/\*]+?\.[^\^\/\*]+?)[\^\/]/,
BLOCKING = /^(@@)?(.*?)(\$~?[\w\-]+(?:=[^,\s]+)?(?:,~?[\w\-]+(?:=[^,\s]+)?)*)?$/; /**/
for (i=0; i<lines.length; i++) {
line = lines[i].toLowerCase().replace(WHITESPACE_G, "");
match = line.match(ELEMHIDE);
firstParty = false;
if (match && match[1]) {
domains = match[1].split(",");
firstParty = true;
} else {
match = line.match(BLOCKING);
domains = [];
isThirdParty = false;
if (match[3]) {
if (B_DOMAINIS.test(match[3])) {
domains = match[3].match(B_DOMAINIS)[1].split("|");
firstParty = true;
}
isThirdParty = B_THIRDPARTY.test(match[3]);
}
match = match[2].match(B_DOMAINMATCH);
if (match) {
if (/^(?:\d{1,3}\.){3}\d{1,3}$/.test(match[1])) {
isThirdParty = true;
} else if (!isThirdParty) {
for (j=0; j<domains.length; j++) {
if (match[1].endsWith(domains[j])) {
isThirdParty = false;
break;
}
isThirdParty = true;
}
}
if (!isThirdParty) {
domains.push(match[1]);
firstParty = firstParty || false;
} else {
knownThirdPartyDomains.push(match[1]);
}
}
}
for (j=0; j<domains.length; j++) {
domains[j] = domains[j].replace("~", "").replace(DOTEND, "");
}
domains = domains.unique();
for (j=0; j<domains.length; j++) {
if (domains[j].contains(".")) {
domainInfo[domains[j]] = domainInfo[domains[j]] || {filters: [], parentDomain: null, subdomains: [], unknownParty: !firstParty};
domainInfo[domains[j]].filters.push(lines[i].trim());
}
}
}
for (j=0; j<knownThirdPartyDomains.length; j++) {
knownThirdPartyDomains[j] = knownThirdPartyDomains[j].replace("~", "").replace(DOTEND, "");
}
for (i in domainInfo) {
if (domainInfo[i].unknownParty && knownThirdPartyDomains.contains(i)) {
delete domainInfo[i];
} else {
delete domainInfo[i].unknownParty;
}
}
domains = Object.keys(domainInfo).sort(function(a, b) {
return a.length < b.length ? -1 : 1;
});
for (i=0; i<domains.length; i++) {
match = domains[i].replace(SUBDOMAIN, "");
while (SUBDOMAIN.test(match)) {
if (domainInfo.hasOwnProperty(match)) {
domainInfo[match].subdomains.push(domains[i]);
domainInfo[domains[i]].parentDomain = match;
break;
}
match = match.replace(SUBDOMAIN, "");
}
}
for (i in domainInfo) {
domainInfo[i].subdomains.sort();
domainInfo[i].filters.sort();
}
self.postMessage({results: domainInfo});
self.close();
};
this.addEventListener("message", function(e) {
extractDomainsFromFilters(e.data.filters);
}, false);

132
famlam-redundancychecker/hidingToBlocking.js Normal file
View File

@ -0,0 +1,132 @@
/*!
This script searches for rules that can be changed into blocking rules,
documentated here: http://adblockplus.org/en/filters, and reports them.
Author: Famlam (fam.lam [at] live.nl)
This program is free software: you can redistribute it and/or modify
it under the terms of the GNU General Public License as published by
the Free Software Foundation, either version 3 of the License, or
(at your option) any later version.
This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
GNU General Public License for more details.
*/
"use strict";
var hidingToBlocking = function(lines) {
importScripts("redundant.js");
startWorker({filters: ""}, false, true); // import it's "globals"
var ELEMHIDE = /^([^\/\*\|\@\"\!]*?)\#\s*(\@)?\s*\#([^\{\}]+)$/, /**/
H_NODENAMESELECTOR = /^((?:\-?(?:[_a-z]|[^\u0000-\u009F]|\\[0-9a-f]{1,6}\s?|\\[^0-9a-f])(?:[\-_a-z0-9]|[^\u0000-\u009F]|\\[0-9a-f]{1,6}\s?|\\[^0-9a-f])*|\*)?\|)?((?:\*|\-?(?:[_a-z]|[^\u0000-\u009F]|\\[0-9a-f]{1,6}\s?|\\[^0-9a-f])(?:[\-_a-z0-9]|[^\u0000-\u009F]|\\[0-9a-f]{1,6}\s?|\\[^0-9a-f])*))$/i, /**/
H_ATTRIBUTENAME = /^\[\s*(data|src)\s*\W?\=/i,
WHITESPACE_G = /\s+/g, /**/
COMMA_G = /\,/g, /**/
B_REGEX = /^\/.+\/$/, /**/
H_RELATIVEPATH = /^(?:\.\.?\/)+/,
i,
result = {},
toBlocking = function(line) {
var match = line.match(ELEMHIDE),
parsed = startWorker.parseCSSSelector(match[3]),
priority = 10,
i, s, foundRule, previousFoundRule,
options = [];
if (parsed.length !== 1 || match[2]) {
return;
}
if (match[1]) {
options.push("domain=" + match[1].replace(COMMA_G, "|").toLowerCase());
}
if (parsed[0].length !== 1) {
priority = 6;
}
parsed = parsed[0][0];
for (i=0; i<parsed.length; i++) {
s = parsed[i];
if (H_NODENAMESELECTOR.test(s.toLowerCase())) {
switch (s.toLowerCase().substring(s.indexOf("|") + 1)) {
case "img": {
options.push("image");
break;
}
case "object": case "embed": case "applet": {
options.push("object");
break;
}
case "video": case "audio": case "source": {
options.push("media");
break;
}
case "frame": case "iframe": {
options.push("subdocument");
break;
}
default: {
priority = 6;
}
}
} else if (s[0] === "[") {
if (!H_ATTRIBUTENAME.test(s)) {
priority = 6;
continue;
}
if (foundRule) {
priority = 9;
previousFoundRule = foundRule;
}
if (s.substring(s.indexOf("=") - 1, s.indexOf("=")) === "\\") {
priority = 6;
continue;
}
foundRule = s.substring(0, s.length - 1).substring(s.indexOf("=") + 1).trim();
if (foundRule[0] === "\"" || foundRule[0] === "'") {
foundRule = foundRule.substring(1, foundRule.length - 1);
}
if (encodeURI(foundRule) !== foundRule || foundRule.contains("*") || foundRule.contains("$")
|| (foundRule.contains(":") && foundRule.substring(0, 5) !== "http:" && foundRule.substring(0, 6) !== "https:")) {
priority = 6;
foundRule = previousFoundRule || "";
continue;
}
if (previousFoundRule && previousFoundRule.length > foundRule.length) {
foundRule = previousFoundRule;
}
if (B_REGEX.test(foundRule)) {
foundRule = foundRule + "*";
} else if (foundRule[0] === "!" || (foundRule[0] === "@" && foundRule[1] === "@")) {
foundRule = "*" + foundRule;
}
if (H_RELATIVEPATH.test(foundRule)) {
foundRule = foundRule.replace(H_RELATIVEPATH, "/");
if (foundRule === "/") {
foundRule = previousFoundRule || "";
}
}
} else {
priority = 6;
}
}
if (foundRule) {
result[line] = {
priority: priority,
newRule: (foundRule + (options.length ? "$" + options.sort().join(",") : "")).replace(WHITESPACE_G, "")
};
}
};
lines = lines.split("\n");
for (i=0; i<lines.length; i++) {
if (ELEMHIDE.test(lines[i])) {
toBlocking(lines[i]);
}
}
self.postMessage({results: result});
self.close();
};
this.addEventListener("message", function(e) {
hidingToBlocking(e.data.filters);
}, false);

161
famlam-redundancychecker/limitations.html Normal file
View File

@ -0,0 +1,161 @@
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="UTF-8">
<meta http-equiv="Content-Security-Policy" content="default-src 'self';" />
<meta name="viewport" content="width=device-width, initial-scale=1" />
<title>Redundancy check - limitations</title>
<link rel="stylesheet" href="redundantRuleChecker.css" />
</head>
<body id="limitations">
<h3>Limitations</h3>
<p id="topmsg">This page lists the limitations of this redundancy checker, as far as known by the author</p>
<h4>Known false positives</h4>
<div>
<em>A false positive is a false alarm: a rule is reported as being redundant, while it actually isn't redundant.</em>
<ul>
<li>There are currently no false positives known.</li>
</ul>
</div>
<h4>Known false negatives</h4>
<div>
<em>A false negative is a rule that is redundant (and can be detected as such with the given input), but isn't reported</em>
<ul>
<li>
<b>The options <samp>$third-party</samp> and <samp>$domain=...</samp> are not linked</b><br/>
As a result, the redundancy checker won't find matches between rules of which one contains <samp>$third-party</samp> (or <samp>$~third-party</samp>) and the other one contains a domain that does match the third- or first-party requirement. Examples:
<ul>
<li><samp>||site.com/ads.$~third-party</samp> should have been made redundant by <samp>/ads.$domain=site.com</samp> as both only apply to domain <em>site.com</em></li>
<li><samp>||site.com/ads.$domain=anothersite.com</samp> should have been made redundant by <samp>/ads.$third-party</samp> as both apply on domain <em>anothersite.com</em></li>
<li><samp>||site.com^$domain=~site.com</samp> should have been made redundant by <samp>||site.com^$third-party</samp> as both apply everywhere except for domain <em>site.com</em></li>
</ul>
The cause is that it is not trivial to extract the top level domain. For example: in the case of <em>subdomain.domain.com</em> this would be <em>com</em>, but in case of <em>subdomain.domain.co.uk</em> this would be <em>co.uk</em>.
</li>
<li>
<b><abbr title="Regular expressions that contain at least one character with a non-literal meaning, such as |, ?, \ or )">Complex regular expression</abbr> rules are only checked one-way</b><br/>
As a result, you will not be notified when a regular expression has been made redundant by another rule. Examples:
<ul>
<li><samp>/advert(isement)?/</samp> should have been made redundant by <samp>/ad(v|s)/</samp></li>
<li><samp>/-advert-images?-/</samp> should have been made redundant by <samp>-advert-</samp></li>
</ul>
The cause is that it is almost impossible to rewrite complex expressions in such a form that it can be checked against all possibilities. For example, the simple regex <samp>/\w{3}/</samp> matches every possible filter of at least three successive alphanumeric characters. The regex <samp>/ad(s+|v)?/</samp> matches <samp>ad</samp>, <samp>adv</samp>, <samp>ads</samp>, <samp>adss</samp>, <samp>adsss</samp>, and so on. It is close to impossible to check this against a normal filter or another regex within the duration of a day computing time and without the need for very complex code.
</li>
</ul>
</div>
<h4>Ignored rules</h4>
<div>
<em>Ignored rules are rules that the redundancy checker skips before even trying to find matches. Say, false negatives without the need for a second filter.</em>
<ul>
<li>
<b>Rules with syntax errors</b><br/>
Rules for which it cannot be determined how to interpret them, are ignored. When such a rule is found, an error will be shown. Examples:
<ul>
<li><samp>###.ads</samp></li>
<li><samp>/advert(ising/</samp></li>
</ul>
</li>
<li>
<b>Simplified hiding rules</b></br>
The old, deprecated format of hiding rules is not supported by the tool. When such a rule is found, a warning will be shown telling you how to convert this type of rule to the CSS3 format, if possible. Examples:
<ul>
<li><samp>#div</samp></li>
<li><samp>#div(ads)</samp></li>
<li><samp>#div(class=ads)</samp></li>
</ul>
</li>
<li>
<b>CSS4 hiding rules</b></br>
CSS4 is still work-in-progress and is therefore subject to changes. Furthermore, at this moment most browsers don't fully support all functionality. For that reason, CSS4 selectors are not supported yet. Examples:
<ul>
<li><samp>##[name*=&quot;ads&quot; i]</samp></li>
<li><samp>##div! &gt; #ads</samp></li>
</ul>
</li>
<li>
<b>Rules with unknown options</b></br>
Rules with unknown options are usually the result of typos. However, it could also be an option that has been removed from or has recently been added to Adblock Plus. As it is unknown what the default status of those rules is (<samp>$third-party</samp> defaults to third- and first-party, while <samp>$popup</samp> defaults to disabled), those rules are skipped to prevent incorrect matches. A warning will be shown. Examples:
<ul>
<li><samp>$first-party</samp></li>
<li><samp>$donottrack</samp></li>
<li><samp>$~domain=site.com</samp></li>
<li><samp>$domain=site.com,anothersite.com</samp></li>
</ul>
</li>
<li>
<b>Rules containing the <samp>$sitekey=...</samp> option</b></br>
Rules containing the <samp>$sitekey=</samp> option are silently ignored. The behaviour of those rules is poorly documented, making it hard to determine the behaviour when this rule is combined with other options. Furthermore, there is only one filter list that uses them. Example:
<ul>
<li><samp>@@$sitekey=abcdefghijklmnopqrstuvwxyz</samp></li>
</ul>
</li>
<li>
<b>Rules containing the <samp>$generichide</samp> or <samp>$genericblock</samp> options</b></br>
Rules containing the <samp>$generichide</samp> or <samp>$genericblock</samp> options are silently ignored. These rules are not yet officially released and their behaviour may still change. Example:
<ul>
<li><samp>@@$generichide</samp></li>
</ul>
</li>
<li>
<b>Rules with problematic domains</b></br>
A rule will also be ignored if the domains of a rule are invalid or if a domain is included and excluded at the same time. Warnings are shown in those cases. Examples:
<ul>
<li><samp>$domain=site.com|~site.com</samp></li>
<li><samp>$domain=site..com</samp></li>
<li><samp>$domain=site.com,domain=anothersite.com</samp></li>
<li><samp>site.com,,anothersite.com###ads</samp></li>
<li><samp>site.com/###ads</samp></li>
<li><samp>*.com###ads</samp></li>
</ul>
</li>
<li>
<b>Rules containing JavaScript object properties</b></br>
Rules that contain JavaScript object properties may interfere with the code, thereby changing its behaviour. For that reason, those rules are silently ignored. Examples:
<ul>
<li><samp>constructor##abc</samp></li>
<li><samp>$domain=~con.constructor</samp></li>
<li><samp>__proto__</samp></li>
</ul>
</li>
<li>
<b>Rules containing <samp>:nth-child(An+B)</samp>, <samp>:nth-last-child(An+B)</samp>, <samp>:nth-of-type(An+B)</samp> or <samp>:nth-last-of-type(An+B)</samp> with <samp>A</samp> or <samp>B</samp> very large</b></br>
Due to rounding errors in JavaScript with large numbers, the values of <samp>A</samp> and <samp>B</samp> may differ from the values in the filter. You'll probably never use them anyway, so they are silently ignored. Examples:
<ul>
<li><samp>##:nth-child(590295810358705700001)</samp></li>
<li><samp>##:nth-of-type(10000000000000000000000000n+4)</samp></li>
</ul>
</li>
<li>
<b>Rules containing <samp>[Adblock]</samp> or <samp>[Adblock *]</samp></b></br>
Filter lists have to start with a line containing <samp>[Adblock]</samp> or similar in order to be a valid filter list. As this redundancy checker should also be able to operate with multiple filter lists as input, it is not possible to only filter out the first line, as the second occurrence of this line can indicate the next filter list. Therefore, all rules of this type are silently ignored. Example:
<ul>
<li><samp>[Adblock Plus 2.0]</samp></li>
</ul>
</li>
<!--li>
<b>Rules containing the carriage return sign at the 'wrong place'</b></br>
The redundancy checker contains a lot of regular expressions. In those regexes, the <samp>.</samp> is usually used to match any character. Unfortunately, it doesn't match the carriage return character (<samp>\r</samp>). If this causes an issue, that rule will be skipped silently. Example:
<ul>
<li><samp>:not(#ads&#13;)</samp> (the <samp>\r</samp> is between the <samp>s</samp> and the <samp>)</samp>)</li>
</ul>
</li-->
<li>
<b>Rules that cannot match anything</b></br>
If it can be determined on beforehand that a rule cannot match anything, then that rule will be ignored. Technically, those rules can be made redundant by every filter, but that would only be confusing, so a warning is thrown instead. Examples:
<ul>
<li><samp>##:nth-child(0n-6)</samp></li>
<li><samp>##[class~=&quot;foo bar&quot;]</samp></li>
<li><samp>##[ID]:not([id])</samp></li>
<li><samp>##:not(*)</samp></li>
<li><samp>##::before</samp></li>
<li><samp>$image,~script,~image</samp></li>
<li><samp>$document</samp></li>
<li><samp>||.foo.com^</samp></li>
<li><samp>|://foo.com^</samp></li>
</ul>
</li>
</ul>
</div>
</body>
</html>

100
famlam-redundancychecker/onlyDomainDiffers.js Normal file
View File

@ -0,0 +1,100 @@
/*!
This script searches for identical rules with different domains written in the Adblock Plus syntax,
documentated here: http://adblockplus.org/en/filters, and reports them.
Author: Famlam (fam.lam [at] live.nl)
This program is free software: you can redistribute it and/or modify
it under the terms of the GNU General Public License as published by
the Free Software Foundation, either version 3 of the License, or
(at your option) any later version.
This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
GNU General Public License for more details.
*/
"use strict";
var onlyDomainDiffers = function(lines) {
importScripts("redundant.js");
var ELEMHIDE = /^([^\/\*\|\@\"\!]*?)\#\s*(\@)?\s*\#([^\{\}]+)$/, /**/
BLOCKING = /^(@@)?(.*?)(\$~?[\w\-]+(?:=[^,\s]+)?(?:,~?[\w\-]+(?:=[^,\s]+)?)*)?$/, /**/
B_DOMAINIS = /(?:\,|\$|^)domain\=([^\,]+)/, /**/
STARTCOMMA = /^\,/,
i, j, startWorkerResults, match, key, startWorkerResults2, betterRule, options, modifiedI,
result = {},
linesWithDomains = lines.split("\n");
startWorker({filters: ""}, false, true); // import it's "globals"
// Remove rules without domains/options
for (i=linesWithDomains.length-1; i>=0; i--) {
match = linesWithDomains[i].match(ELEMHIDE);
if (match && !match[1]) {
linesWithDomains.splice(i, 1);
} else if (!match) {
match = linesWithDomains[i].match(BLOCKING);
if (!match[3] || !B_DOMAINIS.test(match[3].toLowerCase())) {
linesWithDomains.splice(i, 1);
}
}
}
startWorkerResults = startWorker({filters: linesWithDomains.join("\n"), modifiers: {ignoreDomains: true}}, false, true);
top:
for (i in startWorkerResults.results) {
betterRule = startWorkerResults.results[i];
for (j=0; j<startWorkerResults.warnings.length; j++) {
if (startWorkerResults.warnings[j].rules.contains(i) || startWorkerResults.warnings[j].rules.contains(betterRule)) {
continue top;
}
}
if (ELEMHIDE.test(i)) {
match = i.match(ELEMHIDE);
startWorkerResults2 = startWorker({filters: betterRule + "\n" + i.replace(match[1], "")}, false, true);
if (startWorker.isEmptyObject(startWorkerResults2.results)) {
continue;
}
key = "#" + (match[2] || "") + "#" + match[3];
result[key] = result[key] || [];
result[key].push(i);
if (!result[key].contains(betterRule)) {
result[key].push(betterRule);
}
} else {
match = i.match(BLOCKING);
options = match[3].substring(1).toLowerCase().split(",");
match[3] = "";
for (j=0; j<options.length; j++) {
if (!B_DOMAINIS.test(options[j])) {
match[3] += "," + options[j];
}
}
match[3] = match[3].replace(STARTCOMMA, "$");
modifiedI = (match[1] || "") + (match[2] || "") + match[3];
startWorkerResults2 = startWorker({filters: betterRule + "\n" + modifiedI}, false, true);
if (startWorker.isEmptyObject(startWorkerResults2.results)) {
continue;
}
key = (match[1] || "") + match[2] + match[3];
result[key] = result[key] || [];
result[key].push(i);
if (!result[key].contains(betterRule)) {
result[key].push(betterRule);
}
}
}
self.postMessage({results: result});
self.close();
};
this.addEventListener("message", function(e) {
onlyDomainDiffers(e.data.filters);
}, false);

2821
famlam-redundancychecker/redundant.js Normal file
View File

File diff suppressed because it is too large Load Diff

275
famlam-redundancychecker/redundantRuleChecker.css Normal file
View File

@ -0,0 +1,275 @@
body {
font-family: arial;
}
textarea, pre, pre * {
font-family: monospace;
}
input {
font-size: 13px;
vertical-align: middle;
}
input:enabled {
cursor: pointer;
}
table th, table td {
text-align: left;
vertical-align: top;
min-width: 120px;
padding: 1px 4px 1px 4px;
}
table {
border-collapse: collapse;
border: 1px solid gray;
}
fieldset {
border-radius: 4px;
margin-top: 20px;
width: 640px;
padding: 5px 10px;
}
legend {
padding: 0px 5px;
font-weight: bold;
}
h3 {
margin-bottom: 0px;
}
#tab_titles {
margin: 20px 0px 0px 20px;
font-size: 0px;
}
#tab_titles > span {
border: 1px solid #777;
display: inline-block;
padding: 3px 10px;
margin: 0px -1px -1px 0px;
font-size: medium;
}
#tab_titles .activeTab {
background-color: #ccc;
cursor: default;
}
#tab_titles > span:not(.activeTab):hover {
background-color: #ccc;
cursor: pointer;
}
#tab_contents {
border: 1px solid #777;
padding: 20px;
font-size: 13px;
min-height: 80px;
white-space: nowrap;
}
#tab_contents > div {
display: none;
}
#tab_contents > .activeTab {
display: block;
}
#tab_contents p {
font-weight: bold;
font-style: italic;
}
#tab_contents pre {
margin-left: 20px;
}
#helplink {
font-size: small;
position: absolute;
right: 10px;
top: 10px;
}
#filters {
width: 100%;
min-width: 640px;
box-sizing: border-box;
display: block;
margin-bottom: 4px;
}
#percent {
margin-left: 30px;
display: inline-block;
width: 200px;
vertical-align: middle;
}
#progressindicator {
display: inline-block;
width: 200px;
font-size: 10px;
text-align: center;
margin-left: -200px;
cursor: default;
}
#toolsdescription {
font-style: normal !important;
font-weight: normal !important;
margin: 1px;
min-height: 48px;
max-width: 100%;
}
#tab_Tools * {
white-space: normal !important;
}
#tab_Tools > input {
min-width: 432px;
text-align: left;
vertical-align: middle;
height: 26px;
background-color: #FFF;
border: 1px solid #777;
padding: 3px 10px;
margin: 0 0 -1px 0;
}
#tab_Tools > input:first-of-type {
border-top-left-radius: 8px;
border-top-right-radius: 8px;
}
#tab_Tools > input:last-of-type {
border-bottom-left-radius: 8px;
border-bottom-right-radius: 8px;
}
#tab_Tools > input:hover {
background-color: #CCC;
border-width: 2px;
padding: 1px 9px;
}
#domainCheckProgress {
margin: 0px 2px 0px 10px;
width: 150px;
}
.minorwarning, .toolsTab, .passedDomainCheck, .pendingDomainCheck, .cancelledDomainCheck {
color: gray;
}
.majorwarning {
color: red;
}
.failedDomainCheck {
color: red;
font-weight: bold;
}
.priority6 {
color: #A0A0A0;
}
.priority7 {
color: #787878;
}
.priority8 {
color: #505050;
}
.priority9 {
color: #282828;
}
.priority10 {
color: #000000;
}
.toolsResults strong {
cursor: pointer;
display: block;
width: 75%;
font-family: monospace;
}
.toolsResults em {
cursor: pointer;
font-family: monospace;
font-size: 12px;
}
.toolsResults .filterCell {
font-family: monospace;
white-space: pre;
}
.toolsResults td[data-indent] {
padding-left: 51px;
}
.toolsResults td[data-indent]::before {
content: "▸";
}
.toolsResults td[data-indent="1"] {
padding-left: 5px;
}
.toolsResults td[data-indent="2"] {
padding-left: 15px;
}
.toolsResults td[data-indent="3"] {
padding-left: 25px;
}
.toolsResults td[data-indent="4"] {
padding-left: 35px;
}
.toolsResults tr:not([data-parent]) {
border-top: 1px solid gray;
}
.toolsResults tbody > .emptynotifier:not(:only-child) {
display: none;
}
.toolsResults pre {
margin-top: 2px;
}
.workerUnsupported {
position: fixed;
top: 0px;
left: 0px;
width: 100%;
background-color: yellow;
border-bottom: 1px solid black;
font-size: 15px;
font-weight: bold;
color: #DD0000;
text-align: center;
z-index: 100000;
white-space: normal;
}
#information {
padding: 12px;
min-width: 500px;
min-height: 250px;
overflow-y: auto;
border: 1px solid black;
border-radius: 8px;
opacity: 0.9;
background: white;
z-index: 1000;
position: absolute;
left: 20%;
top: 20%;
right: 20%;
bottom: 20%;
box-shadow: 1px 1px black;
}
#information ul > li {
margin-bottom: 3px;
font-size: 14px;
}
#topmsg {
margin-top: 0px;
font-size: 14px;
}
.hidden {
display: none !important;
}
/* changelog & limitations */
h4 {
color: #777;
margin: 24px 0px 0px 0px;
}
#changelog ul, #limitations ul {
margin-top: 2px;
font-size: 14px;
}
/* changelog */
#changelog li > b {
margin-right: 4px;
}
/* limitations */
#limitations div > ul > li {
margin-top: 6px;
}
#limitations div {
font-size: 14px;
}
abbr {
cursor: help;
border-bottom: 1px dotted #777;
}

41
famlam-redundancychecker/redundantRuleChecker.html Normal file
View File

@ -0,0 +1,41 @@
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="UTF-8">
<meta http-equiv="Content-Security-Policy" content="default-src 'self';" />
<meta name="viewport" content="width=640, initial-scale=1" />
<title>Redundancy check</title>
<link rel="stylesheet" href="redundantRuleChecker.css" />
<script type="text/javascript" src="redundantRuleChecker.js" defer="defer"></script>
</head>
<body>
<noscript class="workerUnsupported">JavaScript must be enabled on this page.</noscript>
<div id='helplink'>[<a href="#" title="Information">information</a>]</div>
<h3>Redundancy check for Adblock Plus rules</h3>
<p id="topmsg">This script checks if there are any redundant rules in your Adblock Plus filter list. A rule is redundant if the content it blocks, hides or excludes is also blocked, hidden or excluded by another rule.<br/>For example, <em>||somesite.com/ads/</em> has been made redundant by <em>/ads/*</em>. All calculations are performed on your computer, no data is send to the server.</p>
<textarea cols="120" rows="20" id="filters" placeholder="Paste the rules here" spellcheck="false"></textarea>
<input id="btnstart" type="button" disabled="disabled" value="Check for redundant rules"/>
<progress value="0" id="percent" max="100" class="hidden"></progress>
<span id="progressindicator" class="hidden"></span>
<div id="tabs" class="hidden">
<div id="tab_titles"></div>
<div id="tab_contents"></div>
</div>
<div id="information" class="hidden">
<strong>Redundancy check information</strong><br/>
<ul>
<li>Author: Famlam</li>
<li>Contact: <a href="https://forums.lanik.us/viewtopic.php?f=23&amp;t=9041" title="forums.lanik.us" target="_blank">EasyList forum</a> or <em>fam.lam [at] live.nl</em></li>
<li>Changelog: can be found <a href="changelog.html" title="Changelog" target="_blank">here</a></li>
<li>Copyright (C) 2013-2015 Famlam<br/>
This program is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any later version.<br/>
This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details.<br/>
You should have received a copy of the GNU General Public License along with this program. If not, see &lt;<a href="http://www.gnu.org/licenses/" title="GNU GPL" target="_blank">http://www.gnu.org/licenses/</a>&gt;.</li>
<li>This tool aims to help keep the quality of the filter list high, by removing unnecessary filters and warning about potential issues. When using this tool, one should keep in mind that the results of this tool may not be the best results for the purpose of the filter list, and one should thus always proceed with caution and carefully evaluate the suggestions made.</li>
<li>No data you enter on this webpage will be send to any server; all calculations are performed on your local computer</li>
<li>If you find an issue or want to suggest an improvement, please report it via one of the contact addresses. Don't forget to include which browser you use, what filters you were trying to validate, any error messages that occured and any steps necessary to reproduce the issue. A list of known limitations is available <a href="limitations.html" title="List of known limitations" target="_blank">here</a>.</li>
</ul>
<input id="closeinformation" type="button" value="Close"/>
</div>
</body>
</html>

797
famlam-redundancychecker/redundantRuleChecker.js Normal file
View File

@ -0,0 +1,797 @@
"use strict";
var toolsFile;
var createText = function(text, appendTo) {
var el = document.createTextNode(text.replace(/\ {2,}|\ $|^\ /g, function(m) {
return new Array(m.length + 1).join("\u00A0");
}));
if (appendTo) {
appendTo.appendChild(el);
}
return el;
};
var createTag = function(tagname, attributes, properties, appendTo) {
var i, el = document.createElement(tagname);
if (attributes) {
for (i in attributes) {
el.setAttribute(i, attributes[i]);
}
}
if (properties) {
for (i in properties) {
if (i === "textContent") {
createText(properties[i], el);
} else {
el[i] = properties[i];
}
}
}
if (appendTo) {
appendTo.appendChild(el);
}
return el;
};
var switchTab = function(e) {
var i, currentActive = document.querySelectorAll(".activeTab");
for (i=0; i<currentActive.length; i++) {
currentActive[i].classList.remove("activeTab");
}
var target;
if (typeof e === "string") {
target = document.getElementById("tabtitle_" + e);
} else {
target = e.target;
if (target.parentNode.id !== "tab_titles") {
target = target.parentNode;
}
}
target.classList.add("activeTab");
var tabcontent = document.getElementById(target.id.replace("tabtitle", "tab"));
tabcontent.classList.add("activeTab");
};
var addTab = function(id, title, content, index) {
title.id = "tabtitle_" + id;
title.addEventListener("click", switchTab, false);
var tabtitlebar = document.getElementById("tab_titles");
if (index !== undefined && tabtitlebar.children.length > index) {
tabtitlebar.insertBefore(title, tabtitlebar.children[index]);
} else {
tabtitlebar.appendChild(title);
}
content.id = "tab_" + id;
document.getElementById("tab_contents").appendChild(content);
};
var removeTabs = function(id) {
var i, tabIds = id ? [document.getElementById("tabtitle_" + id)] : document.querySelectorAll("[id^='tabtitle_']");
for (i=0; i<tabIds.length; i++) {
document.getElementById("tab_titles").removeChild(tabIds[i]);
var contentarea = document.getElementById(tabIds[i].id.replace("tabtitle", "tab"));
while (contentarea.firstChild) {
// Way faster to first remove all tags inside it
contentarea.removeChild(contentarea.firstChild);
}
contentarea.parentNode.removeChild(contentarea);
}
if (!id) {
document.getElementById("tabs").classList.add("hidden");
}
};
var startWorker = function(script, command, done) {
var btn = document.getElementById("btnstart");
btn.value = "Checking... please wait";
btn.disabled = true;
var prcnt = document.getElementById("percent");
prcnt.classList.remove("hidden");
prcnt.value = 0;
var ind = document.getElementById("progressindicator");
ind.classList.remove("hidden");
ind.textContent = "0%";
var txtbox = document.getElementById("filters");
txtbox.disabled = true;
var tabarea = document.getElementById("tabs");
tabarea.classList.add("hidden");
var worker = new window.Worker(script);
worker.addEventListener("error", function(ex) {
window.alert("The check had to abort due to an error.\nThe error (on line " + ex.lineno + ") was\n" + ex.message);
}, false);
worker.addEventListener("message", function(e) {
if (e.data.progress !== undefined) {
prcnt.value = Math.max(prcnt.value, e.data.progress);
ind.textContent = Math.round(prcnt.value) + "%";
} else {
done(e.data);
btn.value = "Check for redundant rules";
btn.disabled = false;
prcnt.classList.add("hidden");
ind.classList.add("hidden");
txtbox.disabled = false;
tabarea.classList.remove("hidden");
}
}, false);
worker.postMessage(command);
};
var startToolSimilar = function() {
if (document.getElementById("tab_Similarities")) {
switchTab("Similarities");
return;
}
startWorker("similar.js", {filters: toolsFile}, function(e) {
var i, results = e.results;
var tabTitle = createTag("span", {"class": "toolsTab"}, {textContent: "similarities"});
var tabContent = createTag("div");
createTag("p", {}, {textContent: "The following rules look about the same"}, tabContent);
createTag("em", {}, {textContent: "Please note: these results are only based upon the plain text, not on the meaning of it! You should therefore not use them, unless you're very sure what you're doing!"}, tabContent);
createTag("br", {}, {}, tabContent);
var div = createTag("div", {"class": "toolsResults"}, {}, tabContent);
var sortable, b, j, pre, sortfunction = function(a, b) {
if (a[1] === b[1]) {
return b[0] < a[0] ? 1 : -1;
}
return b[1] - a[1] < 0 ? -1 : 1;
};
var clickhandler = function() {
this.nextElementSibling.classList.toggle("hidden");
};
for (i in results) {
sortable = [];
b = createTag("strong", {}, {}, div);
b.addEventListener("click", clickhandler, false);
createText(i + " ", b);
createTag("em", {}, {textContent: " (" + Object.keys(results[i]).length + ")"}, b);
pre = createTag("pre", {"class": "hidden"}, {}, div);
for (j in results[i]) {
sortable.push([j, results[i][j]]);
}
sortable.sort(sortfunction);
for (j=0; j<sortable.length; j++) {
createTag("span", {"class": "priority" + Math.floor(sortable[j][1]*10)}, {textContent: sortable[j][0]}, pre);
createTag("br", {}, {}, pre);
}
}
if (Object.keys(results).length === 0) {
createTag("p", {}, {textContent: "No results found!"}, tabContent);
}
addTab("Similarities", tabTitle, tabContent);
switchTab("Similarities");
});
};
var startToolHideToBlock = function() {
if (document.getElementById("tab_Hidetoblock")) {
switchTab("Hidetoblock");
return;
}
startWorker("hidingToBlocking.js", {filters: toolsFile}, function(e) {
var i, results = e.results;
var tabTitle = createTag("span", {"class": "toolsTab"}, {textContent: "Hiding to blocking"});
var tabContent = createTag("div");
createTag("p", {}, {textContent: "The following rules can become blocking rules"}, tabContent);
createTag("em", {}, {textContent: "Please note: these rules can be converted to blocking rules, but it is not always useful to do so. Some sites for example break when you block the resource, but not if you hide it. Other rules are just too unspecific for blocking rules"}, tabContent);
createTag("br", {}, {}, tabContent);
createTag("br", {}, {}, tabContent);
var sortedKeys = Object.keys(results).sort(function(a, b) {
if (results[a].priority !== results[b].priority) {
return results[a].priority > results[b].priority ? -1 : 1;
}
return a > b ? -1 : 1;
});
for (i=0; i<sortedKeys.length; i++) {
createTag("strong", {"class": "priority" + results[sortedKeys[i]].priority}, {textContent: sortedKeys[i]}, tabContent);
createTag("span", {"class": "priority" + results[sortedKeys[i]].priority}, {textContent: " can be converted to "}, tabContent);
createTag("strong", {"class": "priority" + results[sortedKeys[i]].priority}, {textContent: results[sortedKeys[i]].newRule}, tabContent);
createTag("br", {}, {}, tabContent);
}
if (Object.keys(results).length === 0) {
createTag("p", {}, {textContent: "No results found!"}, tabContent);
}
addTab("Hidetoblock", tabTitle, tabContent);
switchTab("Hidetoblock");
});
};
var startToolLessOptions = function(e) {
var tabid, tabtitle, contenttitle, modifier;
if (e.target.id === "buttonNoOptionsNoDomains") {
modifier = {ignoreOptions: true, ignoreDomains: true};
tabid = "NoOptionsNoDomains";
tabtitle = "ignoring options and domains";
contenttitle = "The following rules are redundant if domains and filter options are ignored:";
} else if (e.target.id === "buttonNoDomains") {
modifier = {ignoreDomains: true};
tabid = "NoDomains";
tabtitle = "ignoring domains";
contenttitle = "The following rules are redundant if domains are ignored:";
} else if (e.target.id === "buttonNoOptions") {
modifier = {ignoreOptions: true};
tabid = "NoOptions";
tabtitle = "ignoring options";
contenttitle = "The following rules are redundant if filter options are ignored:";
} else if (e.target.id === "buttonLoosely") {
modifier = {loosely: true};
tabid = "Loosely";
tabtitle = "loosely";
contenttitle = "The following rules are redundant if the more loose method is used:";
}
if (document.getElementById("tab_" + tabid)) {
switchTab(tabid);
return;
}
startWorker("redundant.js", {filters: toolsFile, modifiers: modifier}, function(e) {
var i, results = e.results;
var tabTitle = createTag("span", {"class": "toolsTab"}, {textContent: tabtitle});
var tabContent = createTag("div");
createTag("p", {}, {textContent: contenttitle}, tabContent);
createTag("em", {}, {textContent: "Please note: these results are based upon modified rules and are not truely redundant! You should therefore not use them, unless you're very sure what you're doing!"}, tabContent);
createTag("br", {}, {}, tabContent);
createTag("br", {}, {}, tabContent);
for (i in results) {
createTag("strong", {}, {textContent: i}, tabContent);
createText(" would have been made redundant by ", tabContent);
createTag("strong", {}, {textContent: results[i]}, tabContent);
createTag("br", {}, {}, tabContent);
}
if (Object.keys(results).length === 0) {
createTag("p", {}, {textContent: "No results found!"}, tabContent);
}
addTab(tabid, tabTitle, tabContent);
switchTab(tabid);
});
};
var startToolOnlyDomainDiffers = function() {
if (document.getElementById("tab_SameRuleDifferentDomain")) {
switchTab("SameRuleDifferentDomain");
return;
}
startWorker("onlyDomainDiffers.js", {filters: toolsFile}, function(e) {
var i, results = e.results;
var tabTitle = createTag("span", {"class": "toolsTab"}, {textContent: "same rule, different domains"});
var tabContent = createTag("div");
createTag("p", {}, {textContent: "The following rules only differ in their domains"}, tabContent);
createTag("em", {}, {textContent: "Please note: you can combine the rules below. It however certainly is not necessary to do so!"}, tabContent);
createTag("br", {}, {}, tabContent);
var div = createTag("div", {"class": "toolsResults"}, {}, tabContent);
var b, j, pre;
var clickhandler = function() {
this.nextElementSibling.classList.toggle("hidden");
};
var keys = Object.keys(results);
keys.sort(function(a, b) {
if (results[a].length === results[b].length) {
return a > b ? 1 : -1;
}
return results[a].length > results[b].length ? -1 : 1;
});
for (i=0; i<keys.length; i++) {
b = createTag("strong", {}, {}, div);
b.addEventListener("click", clickhandler, false);
createText(keys[i] + " ", b);
createTag("em", {}, {textContent: " (" + Object.keys(results[keys[i]]).length + ")"}, b);
pre = createTag("pre", {"class": "hidden"}, {}, div);
results[keys[i]].sort();
for (j=0; j<results[keys[i]].length; j++) {
createTag("span", {"class": "priority10"}, {textContent: results[keys[i]][j]}, pre);
createTag("br", {}, {}, pre);
}
}
if (Object.keys(results).length === 0) {
createTag("p", {}, {textContent: "No results found!"}, tabContent);
}
addTab("SameRuleDifferentDomain", tabTitle, tabContent);
switchTab("SameRuleDifferentDomain");
});
};
var isOpera = function() {
return navigator.userAgent.indexOf("OPR/") > -1;
};
var chromeListener = function(msg) {
var el, i, tr = document.querySelector("tr.pendingDomainCheck[data-domain='" + msg.domain + "']"), top;
if (msg.installed) {
el = document.querySelector(".extensionMissing");
if (msg.incognito) {
el.parentElement.removeChild(el);
} else {
el.textContent = "You need to provide the extension access to incognito pages for it to function correctly. The extension uses incognito pages to determine whether an URL is live, so that no unwanted websites will end up in your search history or leave cookies behind. Furthermore, this way it can fully disable plugins on the websites it opens to prevent security issues, without comprimising your normal browsing on regular tabs. Please visit about://extensions and check the box to allow the extension access to " + (isOpera() ? "private" : "incognito") + " mode. ";
var a = createTag("a", {href: "#", title: "Retry!"}, {textContent: "Retry!"}, el);
a.addEventListener("click", function(e) {
e.preventDefault();
chromeConnect(chromeConnect.maxIndent, true);
}, false);
}
} else if (msg.cancelled) {
tr = document.querySelectorAll("tr.pendingDomainCheck");
for (i=0; i<tr.length; i++) {
tr[i].classList.remove("pendingDomainCheck");
tr[i].classList.add("cancelledDomainCheck");
tr[i].childNodes[1].textContent = "cancelled...";
}
} else if (msg.started) {
tr.childNodes[1].textContent = "checking...";
} else if (msg.passed) {
top = tr.dataset.top;
do {
document.getElementById("domainCheckProgress").value += 1;
tr.classList.remove("pendingDomainCheck");
tr.classList.add("passedDomainCheck");
tr.childNodes[1].textContent = "OK";
tr = document.querySelector("tr.pendingDomainCheck[data-domain='" + tr.dataset.parent + "']");
} while (tr);
if (!document.querySelector("[data-top='" + top + "']:not(.passedDomainCheck)")) {
top = document.querySelectorAll("[data-top='" + top + "']");
window.setTimeout(function() {
var i;
for (i=0; i<top.length; i++) {
top[i].parentElement.removeChild(top[i]);
}
}, 1000);
}
document.getElementById("domainCheckProgress").title = document.getElementById("domainCheckProgress").value
+ "/" + document.getElementById("domainCheckProgress").max;
} else {
document.getElementById("domainCheckProgress").value += 1;
document.getElementById("domainCheckProgress").title = document.getElementById("domainCheckProgress").value
+ "/" + document.getElementById("domainCheckProgress").max;
tr.classList.remove("pendingDomainCheck");
tr.classList.add("failedDomainCheck");
tr.childNodes[1].textContent = msg.msg;
}
if (msg.remaining === 0) {
chromeConnect(chromeConnect.maxIndent - 1, false);
}
};
var chromeConnect = function(maxIndent, firstTime) {
var j, port, els,
domains = [],
extensionID = isOpera() ? "jmcmbnjnjmkfmkgpfppghcagmhofghjh" : "jmcmbnjnjmkfmkgpfppghcagmhofghjh"; // for debugging, use local ID!
//extensionID = "aelckglegkcbdbilkhepjhkhkgeidilg"; //TODO: REMOVE
chromeConnect.maxIndent = maxIndent;
try {
port = chrome.runtime.connect(extensionID);
port.onMessage.addListener(chromeListener);
if (firstTime) {
port.postMessage({prepare: true});
} else if (maxIndent === -1) {
port.postMessage({done: true});
}
} catch(ex) {
return;
}
if (maxIndent > 0) {
els = document.querySelectorAll("tr.pendingDomainCheck > td[data-indent='" + maxIndent + "']");
} else if (maxIndent === 0) {
els = document.querySelectorAll("tr.pendingDomainCheck > td:first-child");
} else {
document.getElementById("domainCheckProgress").classList.add("hidden");
return;
}
for (j=0; j<els.length; j++) {
domains.push(els[j].parentNode.dataset.domain);
}
if (domains.length === 0) {
chromeConnect(maxIndent - 1, false);
return;
}
try {
port.postMessage({domains: domains});
} catch(ex) {}
};
var startToolDomainCheck = function() {
if (document.getElementById("tab_Domaincheck")) {
switchTab("Domaincheck");
return;
}
startWorker("generateDomainTable.js", {filters: toolsFile}, function(e) {
var i,
results = e.results,
isSupported = true;
var tabTitle = createTag("span", {"class": "toolsTab"}, {textContent: "Dead domains"});
var tabContent = createTag("div");
createTag("p", {}, {textContent: "The following table shows whether domains are live, dead, redirected or parked."}, tabContent);
createTag("em", {}, {textContent: "Please note: even if the tool is unable to detect any live web pages on a domain, there might be live pages on the domain or its subdomains!"}, tabContent);
createTag("br", {}, {}, tabContent);
createTag("em", {}, {textContent: "Manually visit the original URL for which the filter was added and use a search engine to be sure a domain is really gone. Third-party domains are ignored."}, tabContent);
if (typeof chrome === "undefined" || !chrome.runtime || !chrome.runtime.connect) {
createTag("div", {"class": "workerUnsupported"}, {textContent: "Unfortunately this tool has to perform actions that web pages are not allowed to do by themselves. Browser extensions have more permissions and can therefore aid in this process. For Chrome/Opera I wrote such an extension. Unfortunately no such extension is available for your browser. Sorry :("}, tabContent);
isSupported = false;
}
createTag("br", {}, {}, tabContent);
createTag("br", {}, {}, tabContent);
var table = createTag("table", {"class": "toolsResults"}, {}, tabContent);
var ttype = createTag("thead", {}, {}, table);
var tline = createTag("tr", {}, {}, ttype);
var th = createTag("th", {}, {textContent: "Domain"}, tline);
createTag("th", {}, {textContent: "Status"}, tline);
createTag("th", {}, {textContent: "Filters"}, tline);
createTag("progress", {value: 0, id: "domainCheckProgress", max: Object.keys(results).length, title: "0/" + Object.keys(results).length}, {}, th);
ttype = createTag("tbody", {}, {}, table);
var domains = Object.keys(results).sort();
var subdomains = [];
for (i=0; i<domains.length; i++) {
if (!results[domains[i]].parentDomain) {
tline = createTag("tr", {"data-domain": domains[i], "class": "pendingDomainCheck"}, {}, ttype);
subdomains = subdomains.concat(results[domains[i]].subdomains.reverse());
createTag("td", {}, {textContent: domains[i]}, tline);
createTag("td", {}, {textContent: "pending..."}, tline);
createTag("td", {"class": "filterCell"}, {textContent: results[domains[i]].filters.join("\n")}, tline);
}
}
var maxIndent = 0;
while (subdomains.length > 0) {
tline = createTag("tr", {"data-domain": subdomains[0], "data-parent": results[subdomains[0]].parentDomain, "class": "pendingDomainCheck"}, {});
var parentRow = ttype.querySelector("tr[data-domain='" + results[subdomains[0]].parentDomain + "']");
parentRow.parentNode.insertBefore(tline, parentRow.nextSibling);
subdomains = subdomains.concat(results[subdomains[0]].subdomains.reverse());
createTag("td", {"data-indent": Number(parentRow.firstChild.dataset.indent || 0) + 1}, {textContent: subdomains[0]}, tline);
maxIndent = Math.max(Number(parentRow.firstChild.dataset.indent || 0) + 1, maxIndent);
createTag("td", {}, {textContent: "pending..."}, tline);
createTag("td", {"class": "filterCell"}, {textContent: results[subdomains[0]].filters.join("\n")}, tline);
subdomains.shift();
}
var trs = ttype.querySelectorAll("tr"), topDomain;
for (i=0; i<trs.length; i++) {
if (!trs[i].dataset.parent) {
topDomain = trs[i].dataset.domain;
}
trs[i].dataset.top = topDomain;
}
tline = createTag("tr", {"class": "emptynotifier"}, {}, ttype);
createTag("td", {colspan: "3"}, {textContent: "All domains in the list seem to be online"}, tline);
addTab("Domaincheck", tabTitle, tabContent);
switchTab("Domaincheck");
var extensionURL = isOpera() ? "data:text/html,the extension for Opera will be added as soon as possible, sorry to keep you waiting :(" : "https://chrome.google.com/webstore/detail/domain-check/jmcmbnjnjmkfmkgpfppghcagmhofghjh";
if (isSupported) {
chromeConnect(maxIndent, true);
var div = createTag("div", {"class": "workerUnsupported extensionMissing hidden"}, {textContent: "Unfortunately this tool has to perform actions that web pages are not allowed to do by themselves. Browser extensions have more permissions and can therefore aid in this process. Therefore, in order to use this tool, you have to install the extension which you can obtain from "}, tabContent);
createTag("a", {href: extensionURL, title: "Install", target: "_blank"}, {textContent: "here"}, div);
createText(". After installing, you also have to give the extension access to the " + (isOpera() ? "private" : "incognito") + " mode via a check box on about://extensions . ", div);
var aRetry = createTag("a", {href: "#", title: "Retry!"}, {textContent: "Retry!"}, div);
aRetry.addEventListener("click", function(e) {
e.preventDefault();
chromeConnect(maxIndent, true);
}, false);
window.setTimeout(function() {
div.classList.remove("hidden");
}, 1234);
}
var aExport = createTag("a", {title: "Export domain check results", href: "#"}, {textContent: "Export results"}, tabContent);
aExport.addEventListener("click", function(e) {
e.preventDefault();
var output = "",
els = document.querySelectorAll(".failedDomainCheck,.cancelledDomainCheck,.pendingDomainCheck");
if (els.length) {
var i,
domains = [],
filters=[],
errors = [],
maxLength = 0;
for (i=0; i<els.length; i++) {
domains.push(els[i].children[0].textContent);
errors.push(els[i].children[1].textContent);
filters = filters.concat(els[i].children[2].textContent.split("\n"));
maxLength = Math.max(maxLength, domains[i].length);
}
for (i=0; i<els.length; i++) {
output += "\n!" + new Array(maxLength - domains[i].length + 1).join(" ") + domains[i] + " => " + errors[i];
}
output += "\n\n\n! Involved filters:\n\n" + filters.sort().filter(function(elem, i, arr) {return elem !== arr[i+1]}).join("\n");
} else {
output = document.querySelector("#tab_Domaincheck .toolsResults .emptynotifier").textContent;
}
window.open("data:text/plain;charset=utf-8," + encodeURIComponent("! Redundancy check domain check results:\n" + output).replace(/\'/g, "%27"));
}, false);
});
};
var startToolWhitelists = function() {
if (document.getElementById("tab_WhitelistsPerRule")) {
switchTab("WhitelistsPerRule");
return;
}
startWorker("findWhitelistRules.js", {filters: toolsFile}, function(e) {
var i, results = e.resultByFilter;
var tabTitle = createTag("span", {"class": "toolsTab"}, {textContent: "whitelists per rule"});
var tabContent = createTag("div");
createTag("p", {}, {textContent: "The following rules have one or more matching whitelists"}, tabContent);
createTag("em", {}, {textContent: "Please note: this is only an indication! If the rule isn't litterally included in the whitelist, it won't be found!"}, tabContent);
createTag("br", {}, {}, tabContent);
var div = createTag("div", {"class": "toolsResults"}, {}, tabContent);
var b, j, pre;
var clickhandler = function() {
this.nextElementSibling.classList.toggle("hidden");
};
var keys = Object.keys(results);
keys.sort(function(a, b) {
if (results[a].length === results[b].length) {
return a > b ? 1 : -1;
}
return results[a].length > results[b].length ? -1 : 1;
});
for (i=0; i<keys.length; i++) {
b = createTag("strong", {}, {}, div);
b.addEventListener("click", clickhandler, false);
createText(keys[i] + " ", b);
createTag("em", {}, {textContent: " (" + Object.keys(results[keys[i]]).length + ")"}, b);
pre = createTag("pre", {"class": "hidden"}, {}, div);
results[keys[i]].sort();
for (j=0; j<results[keys[i]].length; j++) {
createTag("span", {"class": "priority10"}, {textContent: results[keys[i]][j]}, pre);
createTag("br", {}, {}, pre);
}
}
if (Object.keys(results).length === 0) {
createTag("p", {}, {textContent: "No results found!"}, tabContent);
}
addTab("WhitelistsPerRule", tabTitle, tabContent);
switchTab("WhitelistsPerRule");
results = e.resultByWhitelist;
tabTitle = createTag("span", {"class": "toolsTab"}, {textContent: "rules per whitelist"});
tabContent = createTag("div");
createTag("p", {}, {textContent: "The following shows the amount of rules that are involved in the whitelist"}, tabContent);
createTag("em", {}, {textContent: "Please note: this is only an indication! If the rule isn't litterally included in the whitelist, it won't be found!"}, tabContent);
createTag("br", {}, {}, tabContent);
div = createTag("div", {"class": "toolsResults"}, {}, tabContent);
keys = Object.keys(results);
keys.sort(function(a, b) {
if (results[a].length === results[b].length) {
return a > b ? 1 : -1;
}
return results[a].length > results[b].length ? -1 : 1;
});
for (i=0; i<keys.length; i++) {
b = createTag("strong", {}, {}, div);
b.addEventListener("click", clickhandler, false);
createText(keys[i] + " ", b);
createTag("em", {}, {textContent: " (" + Object.keys(results[keys[i]]).length + ")"}, b);
pre = createTag("pre", {"class": "hidden"}, {}, div);
results[keys[i]].sort();
for (j=0; j<results[keys[i]].length; j++) {
createTag("span", {"class": "priority10"}, {textContent: results[keys[i]][j]}, pre);
createTag("br", {}, {}, pre);
}
}
if (Object.keys(results).length === 0) {
createTag("p", {}, {textContent: "No results found!"}, tabContent);
}
addTab("RulesPerWhitelist", tabTitle, tabContent);
});
};
var startRedundancyCheck = function() {
removeTabs();
toolsFile = "\n";
startWorker("redundant.js", {filters: document.getElementById("filters").value}, function(e) {
var i, redundant = e.results, time = e.seconds, warnings = e.warnings;
// REDUNDANCIES TAB
var tabTitle = createTag("span", {}, {textContent: "redundancies (" + Object.keys(redundant).length + ")"});
var tabContent = createTag("div");
createTag("p", {}, {
textContent: "Finished (after " + (time === 1 ? "1 second" : time + " seconds") + ")! " +(Object.keys(redundant).length || "No") + " redundant rule" + (Object.keys(redundant).length === 1 ? "" : "s") + " found!"
}, tabContent);
for (i in redundant) {
if (location.search !== "?nodups" || i.toLowerCase() !== redundant[i].toLowerCase()) {// TEMP, on Fanboys request
createTag("strong", {}, {textContent: i}, tabContent);
createText(" has been made redundant by ", tabContent);
createTag("strong", {}, {textContent: redundant[i]}, tabContent);
createTag("br", {}, {}, tabContent);
}
}
addTab("Redundancies", tabTitle, tabContent);
// CORRECTED FILE TAB
tabTitle = createTag("span", {}, {textContent: "corrected"});
tabContent = createTag("div");
var oldFile = document.getElementById("filters").value.split("\n"), duplicates = {}, newFile = "";
for (i=0; i<oldFile.length; i++) {
if (redundant[oldFile[i]]) {
if (redundant[oldFile[i]] === oldFile[i] && !duplicates[oldFile[i]]) {
duplicates[oldFile[i]] = true;
} else {
continue;
}
}
newFile += oldFile[i] + "\n";
if (oldFile[i] && !/^\s*(?:\!|.*\[Adblock.*\]|\s+$)/i.test(oldFile[i])) {
toolsFile += oldFile[i] + "\n";
}
}
createText("The file without redundancies: ", tabContent);
createTag("a", {href: "#"}, {textContent: "open in a new tab"}, tabContent).
addEventListener("click", function() {
window.open("data:text/plain;charset=utf-8," + encodeURIComponent(newFile).replace(/\'/g, "%27"));
event.preventDefault();
}, false);
createTag("pre", {}, {textContent: newFile}, tabContent);
addTab("Corrected", tabTitle, tabContent);
// WARNINGS TAB
var p, minor, major, minorwarningcount = 0;
tabContent = createTag("div");
if (warnings.length > 1) {
p = createTag("p", {}, {}, tabContent);
createText("The following " + warnings.length + " ", p);
createTag("span", {"class": "majorwarning"}, {"textContent": "errors"}, p);
createText(", warnings or ", p);
createTag("span", {"class": "minorwarning"}, {"textContent": "optimalizations"}, p);
createText(" were encountered while checking the rules:", p);
} else if (warnings.length === 1) {
p = createTag("p", {}, {}, tabContent);
createText("The following ", p);
createTag("span", {"class": "majorwarning"}, {"textContent": "error"}, p);
createText(", warning or ", p);
createTag("span", {"class": "minorwarning"}, {"textContent": "optimalization"}, p);
createText(" was encountered while checking the rules:", p);
} else {
createTag("p", {}, {textContent: "No errors or warnings were encountered while checking the rules"}, tabContent);
}
major = createTag("span", {"class": "majorwarning"}, {}, tabContent);
minor = createTag("span", {"class": "minorwarning"}, {});
for (i=0; i<warnings.length; i++) {
var j, cat = tabContent;
if (warnings[i].priority === "H") {
cat = major;
toolsFile = toolsFile.replace(new RegExp("\n" + warnings[i].rules[0].replace(/\W/g, "\\$&") + "\n", "g"), "\n");
} else if (warnings[i].priority === "L") {
cat = minor;
minorwarningcount++;
}
for (j=0; j<warnings[i].rules.length; j++) {
createTag("strong", {}, {textContent: warnings[i].rules[j]}, cat);
if (j < warnings[i].rules.length-1) {
if (j < warnings[i].rules.length-2) {
createText(", ", cat);
} else {
createText(" and ", cat);
}
}
}
createText(" : " + warnings[i].msg, cat);
createTag("br", {}, {}, cat);
}
tabContent.appendChild(minor);
tabTitle = createTag("span");
createText("warnings (" + (warnings.length - minorwarningcount), tabTitle);
if (minorwarningcount) {
createTag("span", {"class": "minorwarning"}, {textContent: "+" + minorwarningcount}, tabTitle);
}
createText(")", tabTitle);
addTab("Warnings", tabTitle, tabContent, 1);
// TOOLS TAB
tabTitle = createTag("span", {"class": "toolsTab"}, {textContent: "tools"});
tabContent = createTag("div");
createTag("em", {}, {textContent: "The tools below should only be used to gather information. There is no guarantee that any of the results do actually mean something, neither that the results are complete. Therefore, these tools should NOT be used, unless you're very sure what you're doing. Results that are already mentioned in preceding tabs will be ignored here. If you modify the contents of the big text box above, first click 'Check for redundant rules' before using any of the tools."}, tabContent);
createTag("br", {}, {}, tabContent);
createTag("br", {}, {}, tabContent);
var tools = {
"buttonSimilar": {
shortDesc: "Search for rules that look the same",
longDesc: "This option will start a search for rules that look about the same. It does not check the syntax, so you might end up with very weird matches. It however allows you to find rules that are similar, although not exactly the same. The output is a list of all rules that are similar to another rule.",
fn: startToolSimilar
},
"buttonNoOptionsNoDomains": {
shortDesc: "Ignore the domain and rule options when searching for redundancies",
longDesc: "This option will act like the normal redundancy check, except for that it ignores the domains of all types of rules, as well as the options (like $image) for blocking/whitelisting rules.",
fn: startToolLessOptions
},
"buttonNoOptions": {
shortDesc: "Ignore the rule options when searching for redundancies",
longDesc: "This option will act like the normal redundancy check, except for that it ignores the options (like $image) for blocking/whitelisting rules. The option $domain=a|b|c is not ignored.",
fn: startToolLessOptions
},
"buttonNoDomains": {
shortDesc: "Ignore the domains when searching for redundancies",
longDesc: "This option will act like the normal redundancy check, except for that it ignores the domains of all types of rules.",
fn: startToolLessOptions
},
"buttonLoosely": {
shortDesc: "Search for redundancies using a less strict method",
longDesc: "This option will instruct the normal redundancy check to mark a rule as redundant when there is a chance that they are redundant, although not all requirements have been met to be truely redundant.",
fn: startToolLessOptions
},
"buttonOnlyDomainDiffers": {
shortDesc: "Search for equal rules for which only the domain differs",
longDesc: "This option will allow you to find rules for which everything is exactly the same, except for the domain. It'll for example tell you that the rules 'domain1.com###banner' and 'domain2.net###banner' both use the rule '###banner'. The output is a list of rules that share the same rule.",
fn: startToolOnlyDomainDiffers
},
"buttonCheckWhitelists": {
shortDesc: "Find rules that were matched by whitelist rules",
longDesc: "This option allows you to find out how many whitelists you have for a specific rule. It outputs a list of rules and which whitelist rules match them, as well as a list of whitelist rules and which rules have been matched by them.",
fn: startToolWhitelists
},
"buttonHidingToBlocking": {
shortDesc: "Find hiding rules that can be converted to blocking rules",
longDesc: "This option allows you to find hiding rules that can potentially become blocking rules.",
fn: startToolHideToBlock
},
"buttonDomainCheck": {
shortDesc: "Find dead, redirected or parked domains",
longDesc: "This option allows you to find rules for which some domains can be removed because they are either dead, redirected and/or parked. This requires an additional browser extension for Chrome or Opera",
fn: startToolDomainCheck
}
};
var mouseoverfn = function(e) {
document.getElementById("toolsdescription").textContent = tools[e.target.id].longDesc;
};
for (i in tools) {
var btn = createTag("input", {type: "button", id: i, value: tools[i].shortDesc}, {}, tabContent);
btn.addEventListener("click", tools[i].fn, false);
btn.addEventListener("mouseover", mouseoverfn, false);
createTag("br", {}, {}, tabContent);
}
p = createTag("fieldset", {}, {}, tabContent);
createTag("legend", {}, {textContent: "Description"}, p);
createTag("p", {id: "toolsdescription"}, {textContent: "Move your mouse over a button to show the description."}, p);
addTab("Tools", tabTitle, tabContent);
switchTab("Redundancies");
});
};
window.setTimeout(function() {
try {
new Worker("redundant.js").terminate();
var fn = function() {
if (this /*strict mode unsupported*/ || Object.hasOwnProperty("__proto__") /*__proto__ is treated as custom property*/) {
createTag("div", {"class": "workerUnsupported"}, {"textContent": "Your current browser has some limitations. Proceed with caution or use a different browser."}, document.body);
}
};
fn();
var btnstart = document.getElementById("btnstart");
btnstart.addEventListener("click", startRedundancyCheck, false);
btnstart.removeAttribute("disabled");
} catch(ex) {
createTag("div", {"class": "workerUnsupported"}, {"textContent": "Your browser does not support web workers."}, document.body);
}
document.addEventListener("click", function(e) {
var p = e.target;
while (p) {
if (p.id === "information" || p.id === "helplink") {
return;
}
if (p.id === "closeinformation") {
break;
}
p = p.parentNode;
}
document.getElementById("information").classList.add("hidden");
}, false);
document.getElementById("helplink").addEventListener("click", function() {
document.getElementById("information").classList.remove("hidden");
event.preventDefault();
}, false);
}, 0);

221
famlam-redundancychecker/similar.js Normal file
View File

@ -0,0 +1,221 @@
/*!
This script searches for similar rules written in the Adblock Plus syntax,
documentated here: http://adblockplus.org/en/filters, and reports them.
Author: Famlam (fam.lam [at] live.nl)
This program is free software: you can redistribute it and/or modify
it under the terms of the GNU General Public License as published by
the Free Software Foundation, either version 3 of the License, or
(at your option) any later version.
This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
GNU General Public License for more details.
*/
"use strict";
var similarity = function(lines) {
importScripts("redundant.js");
startWorker({filters: ""}, false, true); // import it's "globals"
lines = lines.split("\n");
var i, j, sI, score,
result = {},
ELEMHIDE = /^([^\/\*\|\@\"\!]*?)\#\s*(\@)?\s*\#([^\{\}]+)$/, /**/
BLOCKING = /^(@@)?(.*?)(\$~?[\w\-]+(?:=[^,\s]+)?(?:,~?[\w\-]+(?:=[^,\s]+)?)*)?$/, /**/
MANYSTARS_G = /\*{2,}/g, /**/
B_USELESSFILTERSTART = /^\|\*/, /**/
B_USELESSFILTEREND = /(?:\*+|^)(?:\^+\|?|\|)$/, /**/
B_USELESSSTAR_G = /^\*|\*$/g, /**/
B_STARTWILDCARDPIPE_G = /^(?:\*|\.?\*\\)\|/g, /**/
B_ENDWILDCARDPIPE_G = /\|\.?\*$/g, /**/
H_HIDINGKEYGROUPS = /[\*\|\~\$\^]?\=\s*\"(?:\\.|[^\\\"])*?\"|[\*\|\~\$\^]?\=\s*\'(?:\\.|[^\\\'])*?\'|\-?(?:[_a-z]|[^\u0000-\u0177]|\\[0-9a-f]{1,6}\s?|\\[^0-9a-f])(?:[\-_a-z0-9]|[^\u0000-\u0177]|\\[0-9a-f]{1,6}\s?|\\[^0-9a-f])*|\((?:\\.|[^\\\)])*?\)|\s*[\>\+\~\,]\s*|./gi,
WHITESPACE_G = /\s+/g, /**/
B_REGEX = /^\/.+\/$/, /**/
B_BLOCKINGKEYS = /^\|\||\%[0-9a-f]{2}|[a-z]+|[0-9]+|[^\*\^]|\^\*/g,
cat_elemhide = {},
cat_block = {},
currentChecks = 0,
maxChecks = 0,
nextReport = 0,
reportProgress = function(n) {
currentChecks += n;
nextReport -= n;
if (nextReport < 1) {
nextReport = maxChecks;
self.postMessage({progress: Math.round(currentChecks / maxChecks) / 2});
}
};
var isEmptyObject = function(obj) {
var i;
for (i in obj) {
return false;
}
return true;
};
var get_keys_block = function(line) {
var newLine, i, j,
match = line.replace(WHITESPACE_G, "").match(BLOCKING),
keys = [],
singlekeys = [];
if (B_REGEX.test(match[2])) {
match[2] = match[2].substring(1, match[2].length - 1);
} else {
match[2] = match[2]
.replace(MANYSTARS_G, "*")
.replace(B_USELESSFILTEREND, "*")
.replace(B_USELESSFILTERSTART, "*");
match[2] = match[2]
.replace(B_STARTWILDCARDPIPE_G, "**|")
.replace(B_ENDWILDCARDPIPE_G, "|**")
.replace(B_USELESSSTAR_G, "");
}
newLine = (match[1] || "") + (match[2] || "") + (match[3] ? "$" : "");
if (match[1]) {
keys.push("@@");
}
if (match[2]) {
singlekeys = match[2].toLowerCase().match(B_BLOCKINGKEYS);
for (i=0; i<singlekeys.length; i++) {
for (j=Math.min(i+10, singlekeys.length); j>i; j--) {
keys.push(singlekeys.slice(i, j).join(""));
}
}
}
if (match[3]) {
keys.push("$");
singlekeys = match[3].substring(1).split(",");
for (i=0; i<singlekeys.length; i++) {
if (singlekeys[i][0] !== "~" && !singlekeys.contains("~" + singlekeys[i])) {
keys.push(singlekeys[i]);
}
}
}
return keys.sort(function(a, b) {
if (a.length === b.length) {
return a > b ? 1 : -1;
}
return b.length-a.length < 0 ? -1 : 1;
});
};
var get_keys_hide = function(line) {
var singlekeys, i, j,
match = line.match(ELEMHIDE),
keys = [];
match[1] = match[1].replace(WHITESPACE_G, "").toLowerCase();
match[2] = match[2] ? "#@#" : "##";
if (match[1]) {
keys.push(match[1]);
}
keys.push(match[2]);
singlekeys = match[3].trim().match(H_HIDINGKEYGROUPS);
for (i=0; i<singlekeys.length; i++) {
for (j=Math.min(i+10, singlekeys.length); j>i; j--) {
keys.push(singlekeys.slice(i, j).join(""));
}
}
return keys.sort(function(a, b) {
if (a.length === b.length) {
return a > b ? 1 : -1;
}
return b.length-a.length < 0 ? -1 : 1;
});
};
var tresholdcache = [9, 9, 9, 9, 9, 4, 5, 5, 6, 6];
var get_treshold = function(length) {
if (tresholdcache.length > length) {
return tresholdcache[length];
}
var i, sqrt;
for (i=tresholdcache.length; i<length+1; i++) {
if (i<55) {
// 1 2 3 4 5 6 = 21 = de som minus de x-waarde van de som = 6 minus Math.floor(x/10) maal de (x-1)-waarde van de som = 0
sqrt = Math.sqrt(8*i + 1);
tresholdcache.push(Math.ceil(i - (Math.floor(sqrt/20 - 0.05)*(sqrt + 1) + sqrt)/2));
} else {
// de ingekortte 'som' minus 11 minus 10*floor(x/10)
tresholdcache.push(Math.ceil(i - 11 - 10*Math.floor((i/10 + 4.5)/10)));
}
}
return tresholdcache[length];
};
var get_score = function(rule, similarRule) {
var i, prevScore,
equal = rule.length,
treshold = get_treshold(rule.length);
if (similarRule.length < treshold) {
return 0;
}
for (i=0; i<rule.length; i++) {
if (i > 0 && rule[i] === rule[i-1]) {
equal += prevScore; // prevScore <= 0
continue;
}
prevScore = 0;
if (!similarRule.contains(rule[i])) {
equal -= 1;
prevScore = -1;
if (equal < treshold) {
return 0;
}
}
}
equal = equal/rule.length;
return equal >= 0.6 ? equal : 0;
};
for (i=0; i<lines.length; i++) {
if (ELEMHIDE.test(lines[i])) {
cat_elemhide[lines[i]] = get_keys_hide(lines[i]);
} else {
cat_block[lines[i]] = get_keys_block(lines[i]);
}
}
maxChecks += (Math.pow(Object.keys(cat_elemhide).length, 2) + Math.pow(Object.keys(cat_block).length, 2) + 1) / 200;
for (i in cat_elemhide) {
sI = cat_elemhide[i];
result[i] = {};
for (j in cat_elemhide) {
reportProgress(1);
if (i === j) {
continue;
}
score = get_score(sI, cat_elemhide[j]);
if (score) {
result[i][j] = score;
}
}
}
for (i in cat_block) {
sI = cat_block[i];
result[i] = {};
for (j in cat_block) {
reportProgress(1);
if (i === j) {
continue;
}
score = get_score(sI, cat_block[j]);
if (score) {
result[i][j] = score;
}
}
}
for (i in result) {
if (isEmptyObject(result[i])) {
delete result[i];
}
}
self.postMessage({results: result});
self.close();
};
this.addEventListener("message", function(e) {
similarity(e.data.filters);
}, false);

10
famlam-redundancychecker/test.html Normal file
View File

@ -0,0 +1,10 @@
<!DOCTYPE html>
<html>
<head>
<meta charset="utf-8"/>
<title>Ohai!</title>
</head>
<body>
<h1>Modify me.</h1>
</body>
</html>

366
fanboy-cookiemonster.txt.temp Normal file
View File

@ -0,0 +1,366 @@
[Adblock Plus 2.0]
! Checksum: FwWMlzXT/+2xtFsHlHAmXw
! Title: Fanboy's Cookiemonster List
! Updated: 3 Mar 2018
! This list expires after 4 days
! License: http://creativecommons.org/licenses/by/3.0/
! Homepage: http://www.fanboy.co.nz/
! Reporting Issues: https://github.com/ryanbr/fanboy-adblock/issues
!
! Legal stuff. (T&C's)
! In no event shall Fanboy List, or the list author be liable for any indirect, direct, punitive, special, incidental, or consequential damages whatsoever.
! By downloading or viewing, or using this list, you are accepting these terms and the license.
!
!
! Cookie Messages
-agree-cookies/
-cookie-banner-
-cookie-cnil.
-cookie-consent.
-cookie-filter/$script
-cookie-law-popup.
-cookie-law.$script
-cookie-law/$script
-cookie-message.js
-cookie-msg-bar.
-cookie-notice.
-cookie-policy.
-cookie-popup/
-cookieconsent-
-CookieInfo.
-cookiePolicy.
-cookies-bar/
-cookies-notice/
-cookies-policy.
-eu-cookies/$script
-eucookie.
-privacy-cookie-
.consent-cookie.
.cookie-law.$script
.cookie-policy.
.cookie_law.
.cookieconsent.
.cookiecuttr.
.cookieinfo2.
.cookielaw.js
.cookienotice.
.cookiepolicy.
.cookiesdirective_
.cookiesmessage.
.cookiesPolicy.
.divascookies-
.EuCookieCompliance.
.fscookieconsent.
/__cookie_js.php
/ac_cookiesvalidation/*
/accept-cookie.
/accept-cookies.
/acceptCookies.
/alertcookie.
/alertcookies.
/amCookieApproval.
/angular-cookies-
/asesor-cookies-$~image
/avisCookies.
/aws-cookie/*
/bandeaucookie.
/bauer.cookies.
/bCookiesMessage.
/bgon-cookies-
/cdcookie/*
/checkCookie?
/cipr-cookie-opt-out/*
/cnil-cookies_
/cnil.js
/coCookiesDisclaimer.
/cookie-accept.js
/cookie-alert.
/cookie-bandeau.
/cookie-banner
/cookie-bar-
/cookie-bar.
/cookie-check.js
/cookie-compliance.
/cookie-consent.
/cookie-consent/*
/cookie-control.
/cookie-disclaimer.
/cookie-disclaimer/*
/cookie-eu/*
/cookie-footer-
/cookie-info.
/cookie-law-$~script
/cookie-law.js
/cookie-manager.min.js
/cookie-master.js
/cookie-message.
/cookie-notice-
/cookie-notice.
/cookie-notice/*$~stylesheet
/cookie-notif.
/cookie-notification
/cookie-policy-
/cookie-policy.js
/cookie-policy.min.
/cookie-policy/*
/Cookie-popup-
/cookie-popup.
/cookie-prompt.
/cookie-reglement.
/cookie-richtlinie.
/cookie-terms.
/cookie-warning.
/cookie.policy.
/cookie.response.
/cookie/hi-
/cookie/policy-
/cookie/require-
/cookie/support-
/cookie_acceptance_
/cookie_alert*.js
/cookie_banner-
/cookie_banner.
/cookie_bar.
/cookie_consent.
/cookie_consent/*
/cookie_consent_
/cookie_hint.
/cookie_info.css
/cookie_info.js
/cookie_information_$script
/cookie_law.js
/cookie_law/*
/cookie_legal_$script
/cookie_msg.js
/cookie_notice.
/cookie_notification-
/cookie_notifier/*
/cookie_policy.
/cookie_policy_
/cookie_solution/*
/cookie_sticky_
/cookie_ws.
/cookie_ws/*
/CookieAccept/*
/cookieacceptance.
/cookiealert.
/CookieAlertBootStrapper.
/cookiebadge.
/cookiebadge_
/cookiebanner.
/cookieBanner/*
/cookiebar-
/cookiebar.
/CookieBar/*
/cookiebox.
/cookiec.json
/cookiechecker/*
/cookiechoice.js
/cookiechoices-
/cookiechoices.
/cookieCompliance.js
/cookiecompliance/*$script
/cookiecompliancy.
/cookieConsent-
/cookieconsent.
/cookieconsent/*
/cookieconsent_
/cookiecontrol-
/cookiecontrol.js
/cookiecontroller.js
/cookiecu.
/cookiecutter.js
/cookied.js
/cookiedirective*.js
/cookiedisclaimer.
/CookieDisclaimerUI.
/cookiedisclosure.
/cookieGatekeeper.$~script
/cookiehint.
/CookieInfo.
/cookieInformerBooklet_
/cookiejar/*
/cookielaw.$~stylesheet
/cookielawinfo.$~script
/cookielex/*
/cookiemanagement.js
/cookieMention.
/cookiemessage.
/cookiemessage_
/cookieMetie.
/cookiemonster.mg
/cookieMonster.min.js
/cookiemsg.
/cookienotice.
/cookienotice/*$script
/cookienotif.
/cookieNotification.
/CookieNotification/*
/cookieok.js
/cookiepolicy.
/cookiepolicy/*
/cookiepolicy_
/CookiePolicyAlert_
/cookiepolicybanner.
/cookiepolicydisclaimer.
/cookiePolicyEN.
/cookiePolicyEUPopin_
/cookiepopup?
/CookiePrompter-
/CookieReportsDisclaimer.
/cookies-banner
/cookies-compliant.
/cookies-directive.
/cookies-disclaimer.
/cookies-enabler.
/cookies-footer-
/cookies-info.$~xmlhttprequest
/cookies-legal-
/cookies-monster.js
/cookies-monster/*$script
/cookies-notice.
/cookies-notice/*
/cookies-notification.
/cookies-overlay.
/cookies-policy-
/cookies-policy.js
/cookies-policy/*
/cookies/popup.
/cookies_acceptance.
/cookies_auto-
/cookies_bar.js
/cookies_directive.
/cookies_komunikat.
/cookies_loading.php?
/cookies_policy.js
/Cookies_Pro/*
/CookiesAgreement/*
/cookiesbanner/*
/cookiesConsent.
/cookiescontrol-
/cookiesdisclaimer.
/cookieser.
/CookieSettingsManager?
/cookieslegal.
/cookiesLegales.
/cookiespolicy.
/cookiesPopup.
/cookiesWidget/*
/cookiewarning*.js
/cookiewarning.
/cookiewarning?
/cookiewet.
/divascookies_
/dotbandeaucookie/*
/ecookie_check/*
/ee-cookies-min.
/emea-cookie-
/eprivacy-cookie.
/eu-cookie-
/eu-cookie.
/eu-cookie/*
/EU-cookielaw.
/eu-notice.js
/eu_cookie_$script,stylesheet
/eu_cookies.
/eu_privacy.js
/eu_rp_cookie.
/euc_cookie.js
/eucookie.
/eucookiecompliance.
/eucookiecompliance/*
/EUCookieDirective.
/eucookielaw.
/eucookienotice.
/eucookienotice_
/eucookies.
/eucookiesk.
/eucookieslaw/*$script
/EUCookieUtility.
/eurocookie.
/fp_cookie.
/g4cookies.
/GestionCookiesCNILPlugin/*
/header.cookies.
/hi-cookie-
/icit-cookie-law/*
/ico_cookies.
/icocookies.
/implied-consent.
/implied-consent/*
/impresacookiepolicy/*
/InfoCookies.
/intser_cookies.js
/iubenda_cs.
/jbcookies.
/jjp_eucookie_
/jquery.cookie.js$domain=password-decryptor.com
/jqueryCookieGuard.
/js-cookie-master/*$script
/js/cookiePolicy?
/layer-cookienotice.
/legal/cookies/?
/legalcookie.
/lma-cookie.
/mod_cookiesaccept/*
/mod_jbcookies/*
/mweb.cookie/*$script
/newprivacypolicy.
/ocgcookie?
/oupcookiepolicy_
/plugin_cookie/*
/plugins/cookie-muncher/*
/plugins/CookieInfo/*
/plugins/EuCookie/*
/privacy-cookies.js
/privacycookie/*
/rbicookiepolicy.
/rtiCookieLaw.
/script_cookies.js
/scripts/cookie-
/SetConsentCookie.
/SetCookieBanner.
/sni_cookie_privacy/*
/tm.cookie.js
/toucanCookie.
/trackingProtection/*
/ttgcookieconsent-
/TWCCookies/*
/twitter_cookies.
/type-cookies.
/ubmcookieconsent-
/usecookies.js
/usocookies.
/w034CookieDirective.js
/warnCookies.
/widgets/cookies-
/wkp_cookies.js
/wp-cookie-allow/*$script,stylesheet
/wpmaster-cookie/*
_alert_cookie/
_cookie_bar/
_cookie_cnil.
_cookie_compliance.
_cookie_consent/
_Cookie_Directive-
_cookie_law.js
_cookie_notice-
_cookie_notice.
_cookie_policy.
_cookie_policy_
_cookiebar_
_cookieCgu.
_cookielaw_mip.
_cookieMsgBox/
_cookieNotice.
_CookiePolicy.
_cookies_disclosure/
_cookiesbar.
_cookieschecker.
_cookiewarn.
_eu_cookie_
_eucookie_compliance_
_europecookielaw.
_js__cookie_functions.
_site_cookie_
! Cookie,privacy,legal Messages