# Robots.txt for main shopping site, allow access to mostly everything # # Discovery Engine uses a crippling DoS style crawl, block it # User-agent: discobot Disallow: / User-agent: * Disallow: /en/checkout Disallow: /ja/checkout Disallow: /en/login Disallow: /ja/login Disallow: /en/search Disallow: /ja/search