User-agent: * Disallow: # All robots will spider the domain