{"id":11581,"date":"2020-09-28T22:00:16","date_gmt":"2020-09-28T22:00:16","guid":{"rendered":"https:\/\/wpinitiate.com\/index.php\/2020\/09\/28\/crawlomatic-update-it-is-able-to-respect-the-robots-txt-file-and-to-not-crawl-links-defined-in-it\/"},"modified":"2020-09-28T22:00:16","modified_gmt":"2020-09-28T22:00:16","slug":"crawlomatic-update-it-is-able-to-respect-the-robots-txt-file-and-to-not-crawl-links-defined-in-it","status":"publish","type":"post","link":"https:\/\/wpinitiate.com\/index.php\/2020\/09\/28\/crawlomatic-update-it-is-able-to-respect-the-robots-txt-file-and-to-not-crawl-links-defined-in-it\/","title":{"rendered":"Crawlomatic update: it is able to respect the robots.txt file and to not crawl links defined in it"},"content":{"rendered":"<div class=\"youtubomatic-video-container\"><iframe loading=\"lazy\" width=\"580\" height=\"380\" src=\"https:\/\/www.youtube.com\/embed\/FzSFIgk77VQ?controls=1\" frameborder=\"0\" allowfullscreen><\/iframe><\/div>\n<p>&#8220;Good Crawler&#8221; update today for Crawlomatic: it is able to respect the robots.txt file and to not crawl links defined in it!<br \/>\nRobots.txt is a text file webmasters create to instruct web robots (typically search engine robots) how to crawl pages on their website. The robots.txt file is part of the the robots exclusion protocol (REP), a group of web standards that regulate how robots crawl the web, access and index content, and serve that content up to users. The REP also includes directives like meta robots, as well as page-, subdirectory-, or site-wide instructions for how search engines should treat links (such as \u201cfollow\u201d or \u201cnofollow\u201d).<br \/>\nIn practice, robots.txt files indicate whether certain user agents (web-crawling software) can or cannot crawl parts of a website.<br \/>\nCheck Crawlomatic here: <a href=\"https:\/\/1.envato.market\/crawlomatic\" target=\"_blank\" rel=\"noopener noreferrer\">https:\/\/1.envato.market\/crawlomatic<\/a><br \/>\nRobots meta header more info: <a href=\"https:\/\/developers.google.com\/search\/reference\/robots_meta_tag\" target=\"_blank\" rel=\"noopener noreferrer\">https:\/\/developers.google.com\/search\/reference\/robots_meta_tag<\/a><\/p>\n<p>Subscribe to this channel to get more videos like this (also hit bell notification)!<\/p>\n<p>\u25b6SUBSCRIBE?  Help me reach 5,000 YouTube subscribers and enjoy new videos daily at <a href=\"https:\/\/www.youtube.com\/channel\/UCVLIksvzyk-D_oEdHab2Lgg?sub_confirmation=1\" target=\"_blank\" rel=\"noopener noreferrer\">https:\/\/www.youtube.com\/channel\/UCVLIksvzyk-D_oEdHab2Lgg?sub_confirmation=1<\/a><\/p>\n<p>\u25b6WP PLUGIN PORTFOLIO? &#8211; <a href=\"https:\/\/1.envato.market\/coderevolutionplugins\" target=\"_blank\" rel=\"noopener noreferrer\">https:\/\/1.envato.market\/coderevolutionplugins<\/a><\/p>\n<p>\u25b6[SPECIAL OFFER] GET ALL MY PLUGINS AT ONCE? &#8211; <a href=\"https:\/\/1.envato.market\/bundle\" target=\"_blank\" rel=\"noopener noreferrer\">https:\/\/1.envato.market\/bundle<\/a><\/p>\n<p>\u25b6DONT CLICK THIS &#8211; <a href=\"https:\/\/www.youtube.com\/channel\/UCVLIksvzyk-D_oEdHab2Lgg?sub_confirmation=1\" target=\"_blank\" rel=\"noopener noreferrer\">https:\/\/www.youtube.com\/channel\/UCVLIksvzyk-D_oEdHab2Lgg?sub_confirmation=1<\/a><\/p>\n<p>\u25b6WEBSITE? Check out my website and blog for news and tutorials on how to earn a passive income from the internet and how to build your online business <a href=\"https:\/\/coderevolution.ro\/\" target=\"_blank\" rel=\"noopener noreferrer\">https:\/\/coderevolution.ro\/<\/a><\/p>\n<p>\u25b6JOIN?  Do you like the content I am releasing? Become a member of my website today, to enjoy premium tutorials and content (and other extra perks), at <a href=\"https:\/\/coderevolution.ro\/join-the-site\/\" target=\"_blank\" rel=\"noopener noreferrer\">https:\/\/coderevolution.ro\/join-the-site\/<\/a><\/p>\n<p>\u25b6SUBSCRIBE to my newsletter and as a bonus, download a FREE WordPress plugin (YOUTUBE CAPTION SCRAPER):<br \/>\n<a href=\"https:\/\/www.facebook.com\/CodeRevolution.envato\/app\/100265896690345\/?ref=page_internal\" target=\"_blank\" rel=\"noopener noreferrer\">https:\/\/www.facebook.com\/CodeRevolution.envato\/app\/100265896690345\/?ref=page_internal<\/a><\/p>\n<p>\u25b6JOIN MY FACEBOOK GROUP?<br \/>\n<a href=\"https:\/\/www.facebook.com\/groups\/coderevo\/\" target=\"_blank\" rel=\"noopener noreferrer\">https:\/\/www.facebook.com\/groups\/coderevo\/<\/a><\/p>\n<p>\u25b6TIPS?  See what services I use to power my business online and help me earn as an affiliate at <a href=\"https:\/\/coderevolution.ro\/recommendations\/\" target=\"_blank\" rel=\"noopener noreferrer\">https:\/\/coderevolution.ro\/recommendations\/<\/a><\/p>\n<p>\u25b6COURSES? Enjoy 5+ online courses on my school from Teachable, the online education platform I use: <a href=\"https:\/\/coderevolution.teachable.com\/\" target=\"_blank\" rel=\"noopener noreferrer\">https:\/\/coderevolution.teachable.com\/<\/a><\/p>\n<p>\u25b6UDEMY? I am also an online instructor on Udemy: <a href=\"https:\/\/www.udemy.com\/course\/the-great-affiliate-program-list-course\/?referralCode=D0441C358F4CDBE3EA8F\" target=\"_blank\" rel=\"noopener noreferrer\">https:\/\/www.udemy.com\/course\/the-great-affiliate-program-list-course\/?referralCode=D0441C358F4CDBE3EA8F<\/a><\/p>\n<p>\u25b6GEAR I USE IN MY VIDEOS?<br \/>\nMicrophone: Samson Go Mic (Great quality, Budget price):<br \/>\n<a href=\"https:\/\/amzn.to\/2KBk61G\" target=\"_blank\" rel=\"noopener noreferrer\">https:\/\/amzn.to\/2KBk61G<\/a><\/p>\n<p>&#8212;&#8212;&#8212;&#8212;&#8212;&#8212;&#8212;&#8212;&#8212;&#8212;<\/p>\n<p>\ud83d\udc4d\ud83c\udffc Please help &#038; give the video a like if you enjoyed it!<br \/>\n\u2764\ufe0f Not Yet Subscribed? (love in the house if you do!)<br \/>\n\ud83d\udd14 Hit the notification bell to ensure you get notified!<br \/>\n\u2705 Check my blog: <a href=\"https:\/\/coderevolution.ro\/blog\/\" target=\"_blank\" rel=\"noopener noreferrer\">https:\/\/coderevolution.ro\/blog\/<\/a><br \/>\n\ud83d\udcad Add your comments! Ask any questions! (please be polite)<br \/>\n_________________________________________<br \/>\n\u2705 CAN I HELP YOU OR YOUR BUSINESS \u2753&#8230;  \u00af_(\u30c4)_\/\u00af<br \/>\n\ud83c\udf0d CodeRevolution Membership Area: <a href=\"https:\/\/coderevolution.ro\/join-the-site\/\" target=\"_blank\" rel=\"noopener noreferrer\">https:\/\/coderevolution.ro\/join-the-site\/<\/a><br \/>\n_________________________________________<br \/>\n\u2705 CAN YOU HELP ME TO HELP YOU \u2753&#8230; \u00af_(\u30c4)_\/\u00af<br \/>\n\ud83d\udc96 Please Support Me: <a href=\"https:\/\/www.buymeacoffee.com\/coderevolution\" target=\"_blank\" rel=\"noopener noreferrer\">https:\/\/www.buymeacoffee.com\/coderevolution<\/a><br \/>\n_________________________________________<br \/>\n\u2705 EITHER WAY, CAN WE KEEP IN TOUCH \u2753&#8230;<br \/>\n\ud83d\udd17 Join the CodeRevolution VIP List here \ud83d\udc49\ud83c\udffc <a href=\"https:\/\/www.facebook.com\/CodeRevolution.envato\/app\/100265896690345\/?ref=page_internal\" target=\"_blank\" rel=\"noopener noreferrer\">https:\/\/www.facebook.com\/CodeRevolution.envato\/app\/100265896690345\/?ref=page_internal<\/a><\/p>\n<p>&#8212;&#8212;&#8212;&#8212;&#8212;&#8212;&#8212;&#8212;&#8212;&#8212;<\/p>\n<p>\u25b6SOCIALS!<br \/>\n<a href=\"https:\/\/www.facebook.com\/CodeRevolution.envato\/\" target=\"_blank\" rel=\"noopener noreferrer\">https:\/\/www.facebook.com\/CodeRevolution.envato\/<\/a><br \/>\n<a href=\"https:\/\/www.youtube.com\/c\/SzabolcsKisded_CodeRevolution\" target=\"_blank\" rel=\"noopener noreferrer\">https:\/\/www.youtube.com\/c\/SzabolcsKisded_CodeRevolution<\/a><br \/>\n<a href=\"https:\/\/twitter.com\/code2revolution\" target=\"_blank\" rel=\"noopener noreferrer\">https:\/\/twitter.com\/code2revolution<\/a><br \/>\n<a href=\"https:\/\/www.instagram.com\/coderevolution_envato\/\" target=\"_blank\" rel=\"noopener noreferrer\">https:\/\/www.instagram.com\/coderevolution_envato\/<\/a><br \/>\n<a href=\"https:\/\/www.linkedin.com\/company\/18002194\" target=\"_blank\" rel=\"noopener noreferrer\">https:\/\/www.linkedin.com\/company\/18002194<\/a><br \/>\n<a href=\"https:\/\/pinterest.com\/caddy_lagaristu\/coderevolution\/\" target=\"_blank\" rel=\"noopener noreferrer\">https:\/\/pinterest.com\/caddy_lagaristu\/coderevolution\/<\/a><\/p>\n<p>\u25b6OTHER INTERESTING VIDEOS FROM MY CHANNEL?<br \/>\n&#8211; A Day In The Life Of A WordPress Developer: <a href=\"https:\/\/www.youtube.com\/watch?v=31-xisX1030\" target=\"_blank\" rel=\"noopener noreferrer\">https:\/\/www.youtube.com\/watch?v=31-xisX1030<\/a><br \/>\n&#8211; How Coding Your First WordPress Plugin Looks Like: <a href=\"https:\/\/www.youtube.com\/watch?v=7l65wodnPyM\" target=\"_blank\" rel=\"noopener noreferrer\">https:\/\/www.youtube.com\/watch?v=7l65wodnPyM<\/a><br \/>\n&#8211; How Uploading Your First WordPress Plugin To CodeCanyon Looks Like: <a href=\"https:\/\/www.youtube.com\/watch?v=ah1gReAv-m8\" target=\"_blank\" rel=\"noopener noreferrer\">https:\/\/www.youtube.com\/watch?v=ah1gReAv-m8<\/a><br \/>\n&#8211; Newsomatic Playlist: <a href=\"https:\/\/www.youtube.com\/playlist?list=PLEiGTaa0iBIi-LWwOcsZDnWZcG8FtuYGs\" target=\"_blank\" rel=\"noopener noreferrer\">https:\/\/www.youtube.com\/playlist?list=PLEiGTaa0iBIi-LWwOcsZDnWZcG8FtuYGs<\/a><br \/>\n&#8211; Crawlomatic Playlist: <a href=\"https:\/\/www.youtube.com\/playlist?list=PLEiGTaa0iBIgcqNzVBaoTCS4ws47vNMuQ\" target=\"_blank\" rel=\"noopener noreferrer\">https:\/\/www.youtube.com\/playlist?list=PLEiGTaa0iBIgcqNzVBaoTCS4ws47vNMuQ<\/a><br \/>\n&#8211; Youtubomatic Playlist: <a href=\"https:\/\/www.youtube.com\/playlist?list=PLEiGTaa0iBIhzwV5VhpV7kgUuClLNpP9i\" target=\"_blank\" rel=\"noopener noreferrer\">https:\/\/www.youtube.com\/playlist?list=PLEiGTaa0iBIhzwV5VhpV7kgUuClLNpP9i<\/a><br \/>\n&#8211; Echo RSS Playlist: <a href=\"https:\/\/www.youtube.com\/playlist?list=PLEiGTaa0iBIhKwBdxZMUdHHY4XuXLvctE\" target=\"_blank\" rel=\"noopener noreferrer\">https:\/\/www.youtube.com\/playlist?list=PLEiGTaa0iBIhKwBdxZMUdHHY4XuXLvctE<\/a><\/p>\n<p>\u25b6ENJOY MY VIDEOS? PLEASE DONATE<br \/>\n<a href=\"https:\/\/www.buymeacoffee.com\/coderevolution\" target=\"_blank\" rel=\"noopener noreferrer\">https:\/\/www.buymeacoffee.com\/coderevolution<\/a><\/p>\n<p>To your success,<br \/>\nSzabi &#8211; CodeRevolution.<\/p>\n<p>PS: This description contains affiliate links that allow you to find the items mentioned in this video and support the channel at no cost to you!  If you are like me, you will love using the links when you are ready to research and buy because you will know this helps me continue sharing the best of what I know here for free with you!  Thank you for your support!<\/p>\n","protected":false},"excerpt":{"rendered":"<p>&#8220;Good Crawler&#8221; update today for Crawlomatic: it is able to respect the robots.txt file and to not crawl links defined in it! Robots.txt is a text file webmasters create to instruct web robots (typically search engine robots) how to crawl pages on their website. The robots.txt file is part of the the robots exclusion protocol&hellip;<\/p>\n","protected":false},"author":1,"featured_media":11582,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[1],"tags":[],"class_list":["post-11581","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-wp"],"aioseo_notices":[],"_links":{"self":[{"href":"https:\/\/wpinitiate.com\/index.php\/wp-json\/wp\/v2\/posts\/11581","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/wpinitiate.com\/index.php\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/wpinitiate.com\/index.php\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/wpinitiate.com\/index.php\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/wpinitiate.com\/index.php\/wp-json\/wp\/v2\/comments?post=11581"}],"version-history":[{"count":0,"href":"https:\/\/wpinitiate.com\/index.php\/wp-json\/wp\/v2\/posts\/11581\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/wpinitiate.com\/index.php\/wp-json\/wp\/v2\/media\/11582"}],"wp:attachment":[{"href":"https:\/\/wpinitiate.com\/index.php\/wp-json\/wp\/v2\/media?parent=11581"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/wpinitiate.com\/index.php\/wp-json\/wp\/v2\/categories?post=11581"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/wpinitiate.com\/index.php\/wp-json\/wp\/v2\/tags?post=11581"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}