Disallow URLs from crawling with a patern

by Diego Wais   Last Updated May 26, 2019 03:04 AM - source

I have a website with like 100K URLs I want to disallow crawling for all URL that has an ID with this pattern www.site.com/node/sport/category/id but not all that have www.site.com/node/sport/category/ without the ID.

how can I approach this rule on robots.txt?

thanks all



Related Questions


Help with robots meta tag and X-Robots-Tag

Updated December 21, 2016 08:01 AM


Robots meta tag with "noimageindex"

Updated January 11, 2018 17:04 PM

Using the robots file in a different way

Updated March 14, 2017 14:04 PM