#**************************************************************************** # robots.txt # : Robots, spiders, and search engines use this file to detmine which # content they should *not* crawl while indexing your website. # : This system is called "The Robots Exclusion Standard." # : It is strongly encouraged to use a robots.txt validator to check # for valid syntax before any robots read it! # # Examples: # # Instruct all robots to stay out of the admin area. # : User-agent: * # : Disallow: /admin/ # # Restrict Google and MSN from indexing your images. # : User-agent: Googlebot # : Disallow: /images/ # : User-agent: MSNBot # : Disallow: /images/ #**************************************************************************** User-agent: * Disallow: /comments/feed/ Disallow: /feed/ Disallow: /feed/atom/ Disallow: /feed/rss/ Disallow: /rss/ Disallow: /trackback/ Disallow: /wp/ Disallow: /*/comments/feed/$ Disallow: /*/feed/$ Disallow: /*/feed/atom/$ Disallow: /*/feed/rss/$ Disallow: /*/rss/$ Disallow: /*/trackback/$ Disallow: /*&paged=* Disallow: /*&paged=2 Disallow: /*rel=no dofollow Disallow: /?page_id=215" rel=no dofollow Disallow: /?cat* Disallow: /*&cpage=1 Disallow: /?tag* Disallow: /*&cpage=1 Disallow: /?p=1&cpage=1 Disallow: /?feed=comments-rss2