You can use URL filters (ctrl+3) to exclude unwanted URLs when you scrape links in “URLs” tab or with Website Crawler tool.


See also:

– How to use regular expressions in URL filters

There are three main URLs filters you should use. That is very enough to perfectly filter links.

1) Place in this filter one or several entries (line by line) to get URLs that contains one of these entries. Check the “all” (a) CheckBox to turn on the “all” mode (URL should contains ALL of these entries).

2) Place in this filter one or several entries (line by line) to get URLs that NOT contains one of these entries.

Note: You can use regular expressions in fields 1 and 2!

3) You can set amount of URLs segments (separated by / char).
For example:
http://website.com – has 1 segment
http://website.com/index.php – has 2 segments
http://website.com/eng/index.php – has 3 segments
…

4) Use this URL filter to delete certain URL parameters.
For example, yo want to transgorm this URL:
website.com/index.php?page=3&offset=300&size=big
In to this URL:
website.com/index.php?size=big
Then place unwanted parametr names in to field 4:
page
offset


(No Ratings Yet)