We Don't Serve Your Kind Here.

There's a tiny file on your website that does a big job and most people have not heard of it. It's named "robots.txt" and it's on your site. Open your website URL and add "/robots.txt" to see it (for example, yourcompany.com/robots.txt).

robots.txt contains a short list of rules that defines what bots and AI agents are welcome on your site. It is basically a bouncer at the front door of your website. Make sure to let that sink in. That one file tells these AI systems either "Welcome," or "We don't serve your kind here."

Most businesses have not deliberately set the rules that robots.txt is using. It's almost certainly set to the defaults. The defaults came from your website platform or your hosting provider, and they might be blocking AI systems from visiting your site entirely without you knowing it. All humans who use AI to learn about your products and services are potentially being told "go away" by your robots.txt file.

That matters more now than it ever has. When someone uses ChatGPT or Perplexity or Claude to search for a business like yours, those platforms send out their own agents to gather information. If your robots.txt is blocking those agents, the AI comes back empty, and your business never gets mentioned. But your competitors will if they set up their robots.txt and other key AI-readable information correctly.

Your site might be invisible to AI systems because a file on your own server is telling them to stay out.

Go look at your robots.txt file. See what systems are "allowed" or "disallowed." If there's just a blanket rule that blocks everything, you are potentially turning real customers away.

If you'd like to understand what to do about this and other ways to make your site ready for the AI web, visit foundforai.com.