“Good Crawler” update today for Crawlomatic: it is able to respect the robots.txt file and to not crawl links defined in it!
Robots.txt is a text file webmasters create to instruct web robots (typically search engine robots) how to crawl pages on their website. The robots.txt file is part of the the robots exclusion protocol (REP), a group of web standards that regulate how robots crawl the web, access and index content, and serve that content up to users. The REP also includes directives like meta robots, as well as page-, subdirectory-, or site-wide instructions for how search engines should treat links (such as “follow” or “nofollow”).
In practice, robots.txt files indicate whether certain user agents (web-crawling software) can or cannot crawl parts of a website.
Check Crawlomatic here: https://1.envato.market/crawlomatic
Robots meta header more info: https://developers.google.com/search/reference/robots_meta_tag
Subscribe to this channel to get more videos like this (also hit bell notification)!
▶SUBSCRIBE? Help me reach 5,000 YouTube subscribers and enjoy new videos daily at https://www.youtube.com/channel/UCVLIksvzyk-D_oEdHab2Lgg?sub_confirmation=1
▶WP PLUGIN PORTFOLIO? – https://1.envato.market/coderevolutionplugins
▶[SPECIAL OFFER] GET ALL MY PLUGINS AT ONCE? – https://1.envato.market/bundle
▶DONT CLICK THIS – https://www.youtube.com/channel/UCVLIksvzyk-D_oEdHab2Lgg?sub_confirmation=1
▶WEBSITE? Check out my website and blog for news and tutorials on how to earn a passive income from the internet and how to build your online business https://coderevolution.ro/
▶JOIN? Do you like the content I am releasing? Become a member of my website today, to enjoy premium tutorials and content (and other extra perks), at https://coderevolution.ro/join-the-site/
▶SUBSCRIBE to my newsletter and as a bonus, download a FREE WordPress plugin (YOUTUBE CAPTION SCRAPER):
▶JOIN MY FACEBOOK GROUP?
▶TIPS? See what services I use to power my business online and help me earn as an affiliate at https://coderevolution.ro/recommendations/
▶COURSES? Enjoy 5+ online courses on my school from Teachable, the online education platform I use: https://coderevolution.teachable.com/
▶UDEMY? I am also an online instructor on Udemy: https://www.udemy.com/course/the-great-affiliate-program-list-course/?referralCode=D0441C358F4CDBE3EA8F
▶GEAR I USE IN MY VIDEOS?
Microphone: Samson Go Mic (Great quality, Budget price):
👍🏼 Please help & give the video a like if you enjoyed it!
❤️ Not Yet Subscribed? (love in the house if you do!)
🔔 Hit the notification bell to ensure you get notified!
✅ Check my blog: https://coderevolution.ro/blog/
💭 Add your comments! Ask any questions! (please be polite)
✅ CAN I HELP YOU OR YOUR BUSINESS ❓… ¯_(ツ)_/¯
🌍 CodeRevolution Membership Area: https://coderevolution.ro/join-the-site/
✅ CAN YOU HELP ME TO HELP YOU ❓… ¯_(ツ)_/¯
💖 Please Support Me: https://www.buymeacoffee.com/coderevolution
✅ EITHER WAY, CAN WE KEEP IN TOUCH ❓…
🔗 Join the CodeRevolution VIP List here 👉🏼 https://www.facebook.com/CodeRevolution.envato/app/100265896690345/?ref=page_internal
▶OTHER INTERESTING VIDEOS FROM MY CHANNEL?
– A Day In The Life Of A WordPress Developer: https://www.youtube.com/watch?v=31-xisX1030
– How Coding Your First WordPress Plugin Looks Like: https://www.youtube.com/watch?v=7l65wodnPyM
– How Uploading Your First WordPress Plugin To CodeCanyon Looks Like: https://www.youtube.com/watch?v=ah1gReAv-m8
– Newsomatic Playlist: https://www.youtube.com/playlist?list=PLEiGTaa0iBIi-LWwOcsZDnWZcG8FtuYGs
– Crawlomatic Playlist: https://www.youtube.com/playlist?list=PLEiGTaa0iBIgcqNzVBaoTCS4ws47vNMuQ
– Youtubomatic Playlist: https://www.youtube.com/playlist?list=PLEiGTaa0iBIhzwV5VhpV7kgUuClLNpP9i
– Echo RSS Playlist: https://www.youtube.com/playlist?list=PLEiGTaa0iBIhKwBdxZMUdHHY4XuXLvctE
▶ENJOY MY VIDEOS? PLEASE DONATE
To your success,
Szabi – CodeRevolution.
PS: This description contains affiliate links that allow you to find the items mentioned in this video and support the channel at no cost to you! If you are like me, you will love using the links when you are ready to research and buy because you will know this helps me continue sharing the best of what I know here for free with you! Thank you for your support!