Disallow URLs from crawling with a patern

by Diego Wais   Last Updated May 26, 2019 03:04 AM - source

I have a website with like 100K URLs I want to disallow crawling for all URL that has an ID with this pattern www.site.com/node/sport/category/id but not all that have www.site.com/node/sport/category/ without the ID.

how can I approach this rule on robots.txt?

thanks all



Related Questions


Help with robots meta tag and X-Robots-Tag

Updated December 21, 2016 08:01 AM

Google is ignoring meta tags and robots.txt

Updated August 20, 2019 02:04 AM


Robots meta tag with "noimageindex"

Updated January 11, 2018 17:04 PM