{"id":100,"date":"2020-01-28T19:19:04","date_gmt":"2020-01-28T19:19:04","guid":{"rendered":"http:\/\/eng.sbfactory.ru\/?p=100"},"modified":"2020-01-29T18:10:22","modified_gmt":"2020-01-29T18:10:22","slug":"how-to-scrape-links-from-website-xml-sitemap","status":"publish","type":"post","link":"https:\/\/eng.sbfactory.ru\/?p=100","title":{"rendered":"How to scrape links from a website XML sitemap"},"content":{"rendered":"<p><!--more--><iframe width=\"560\" height=\"315\" src=\"https:\/\/www.youtube.com\/embed\/InIpPNmvVzQ\" frameborder=\"0\" allow=\"accelerometer; autoplay; encrypted-media; gyroscope; picture-in-picture\" allowfullscreen><\/iframe><br \/>\n<BR><br \/>\nSome websites has XML sitemap which contains URLs you want.<\/p>\n<p>To find out URL of the sitemap type in webbrowser address bar http:\/\/website.com\/robots.txt (where website.com is website address).<\/p>\n<p><img loading=\"lazy\" src=\"http:\/\/eng.sbfactory.ru\/wp-content\/uploads\/2020\/01\/2020-01-28_22-09-03.png\" alt=\"robots.txt\" width=\"518\" height=\"663\" class=\"alignnone size-full wp-image-101\" srcset=\"https:\/\/eng.sbfactory.ru\/wp-content\/uploads\/2020\/01\/2020-01-28_22-09-03.png 518w, https:\/\/eng.sbfactory.ru\/wp-content\/uploads\/2020\/01\/2020-01-28_22-09-03-234x300.png 234w\" sizes=\"(max-width: 518px) 100vw, 518px\" \/><\/p>\n<p><img loading=\"lazy\" src=\"http:\/\/eng.sbfactory.ru\/wp-content\/uploads\/2020\/01\/2020-01-28_22-11-59.png\" alt=\"Sitemap.xml\" width=\"775\" height=\"502\" class=\"alignnone size-full wp-image-102\" srcset=\"https:\/\/eng.sbfactory.ru\/wp-content\/uploads\/2020\/01\/2020-01-28_22-11-59.png 775w, https:\/\/eng.sbfactory.ru\/wp-content\/uploads\/2020\/01\/2020-01-28_22-11-59-300x194.png 300w\" sizes=\"(max-width: 775px) 100vw, 775px\" \/><br \/>\n<BR><br \/>\n<img loading=\"lazy\" src=\"http:\/\/eng.sbfactory.ru\/wp-content\/uploads\/2020\/01\/2020-01-28_22-15-20.png\" alt=\"Content Downloader\" width=\"1072\" height=\"390\" class=\"alignnone size-full wp-image-103\" srcset=\"https:\/\/eng.sbfactory.ru\/wp-content\/uploads\/2020\/01\/2020-01-28_22-15-20.png 1072w, https:\/\/eng.sbfactory.ru\/wp-content\/uploads\/2020\/01\/2020-01-28_22-15-20-300x109.png 300w, https:\/\/eng.sbfactory.ru\/wp-content\/uploads\/2020\/01\/2020-01-28_22-15-20-1024x372.png 1024w\" sizes=\"(max-width: 1072px) 100vw, 1072px\" \/><br \/>\n<BR><br \/>\n<strong>See also:<\/strong><\/p>\n<p> <strong><a href=\"http:\/\/eng.sbfactory.ru\/?p=151\">&#8211; How to use URL filters<\/a><\/strong><br \/>\n<BR><br \/>\n<span id=\"post-ratings-100\" class=\"post-ratings\" data-nonce=\"7e15a137cd\"><img id=\"rating_100_1\" src=\"https:\/\/eng.sbfactory.ru\/wp-content\/plugins\/wp-postratings\/images\/stars\/rating_off.gif\" alt=\"\" title=\"\" onmouseover=\"current_rating(100, 1, '');\" onmouseout=\"ratings_off(0, 0, 0);\" onclick=\"rate_post();\" onkeypress=\"rate_post();\" style=\"cursor: pointer; border: 0px;\" \/><img id=\"rating_100_2\" src=\"https:\/\/eng.sbfactory.ru\/wp-content\/plugins\/wp-postratings\/images\/stars\/rating_off.gif\" alt=\"\" title=\"\" onmouseover=\"current_rating(100, 2, '');\" onmouseout=\"ratings_off(0, 0, 0);\" onclick=\"rate_post();\" onkeypress=\"rate_post();\" style=\"cursor: pointer; border: 0px;\" \/><img id=\"rating_100_3\" src=\"https:\/\/eng.sbfactory.ru\/wp-content\/plugins\/wp-postratings\/images\/stars\/rating_off.gif\" alt=\"\" title=\"\" onmouseover=\"current_rating(100, 3, '');\" onmouseout=\"ratings_off(0, 0, 0);\" onclick=\"rate_post();\" onkeypress=\"rate_post();\" style=\"cursor: pointer; border: 0px;\" \/><img id=\"rating_100_4\" src=\"https:\/\/eng.sbfactory.ru\/wp-content\/plugins\/wp-postratings\/images\/stars\/rating_off.gif\" alt=\"\" title=\"\" onmouseover=\"current_rating(100, 4, '');\" onmouseout=\"ratings_off(0, 0, 0);\" onclick=\"rate_post();\" onkeypress=\"rate_post();\" style=\"cursor: pointer; border: 0px;\" \/><img id=\"rating_100_5\" src=\"https:\/\/eng.sbfactory.ru\/wp-content\/plugins\/wp-postratings\/images\/stars\/rating_off.gif\" alt=\"\" title=\"\" onmouseover=\"current_rating(100, 5, '');\" onmouseout=\"ratings_off(0, 0, 0);\" onclick=\"rate_post();\" onkeypress=\"rate_post();\" style=\"cursor: pointer; border: 0px;\" \/> (No Ratings Yet)<br \/><span class=\"post-ratings-text\" id=\"ratings_100_text\"><\/span><\/span><\/p>\n","protected":false},"excerpt":{"rendered":"","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":[],"categories":[1],"tags":[9,11],"_links":{"self":[{"href":"https:\/\/eng.sbfactory.ru\/index.php?rest_route=\/wp\/v2\/posts\/100"}],"collection":[{"href":"https:\/\/eng.sbfactory.ru\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/eng.sbfactory.ru\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/eng.sbfactory.ru\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/eng.sbfactory.ru\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=100"}],"version-history":[{"count":10,"href":"https:\/\/eng.sbfactory.ru\/index.php?rest_route=\/wp\/v2\/posts\/100\/revisions"}],"predecessor-version":[{"id":185,"href":"https:\/\/eng.sbfactory.ru\/index.php?rest_route=\/wp\/v2\/posts\/100\/revisions\/185"}],"wp:attachment":[{"href":"https:\/\/eng.sbfactory.ru\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=100"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/eng.sbfactory.ru\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=100"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/eng.sbfactory.ru\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=100"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}