To get a new website found, make sure search engines and AI crawlers can reach it, tell Google and Bing it exists, and write pages that answer real questions directly. In practice that means checking robots.txt, submitting a sitemap in Google Search Console and Bing Webmaster Tools, turning on IndexNow, and giving every page one clear title and description.
This is the checklist we run on every site we launch, including our own. Most of it takes an afternoon. Ranking takes longer, but nothing ranks until it has been found.
Can search engines and AI crawlers reach your site?
Check this first, because a single line in the wrong file can hide a whole site. Open yourdomain.com/robots.txt in a browser and read what is actually served, not what is in your code.
- robots.txt: it should not contain
Disallow: /for the crawlers you want. Staging settings copied to production are the usual cause. - noindex tags: view the page source and search for
noindex. Many site builders and frameworks add it to preview sites, and it sometimes survives the launch. - One version of the domain:
http,https,wwwand non-wwwshould all redirect to the same address, and each page's canonical tag should point to that address. - Your CDN or host: some add their own rules. On our own site, a Cloudflare setting once added a block for AI crawlers to the top of our robots.txt, on top of the file our code produced. We only spotted it by fetching the live file.
These are the crawlers most worth allowing for a business site:
| Crawler | Company | What it's for |
|---|---|---|
| Googlebot | Google Search, including AI Overviews | |
| Bingbot | Microsoft | Bing search and Microsoft Copilot answers |
| OAI-SearchBot | OpenAI | Showing sites in ChatGPT search results |
| GPTBot | OpenAI | Collecting content for training OpenAI's models |
| ClaudeBot | Anthropic | Collecting content for Anthropic's Claude models |
| PerplexityBot | Perplexity | Showing sites in Perplexity's answers |
Blocking the training crawlers is a fair choice for some publishers. For a business that wants to be recommended, being readable is usually worth more.
How do I tell Google my new site exists?
Verify the site in Google Search Console and submit your sitemap. Google can find a site on its own through links, but this is faster and shows you what it sees.
- Add a Domain property. It covers every subdomain and protocol, and you verify it with one DNS TXT record at your domain registrar.
- Submit your sitemap. Under Sitemaps, enter
sitemap.xml(or whatever your platform generates). Check that it lists only real, live pages. - Request indexing for your key pages. Paste the homepage and your main service pages into URL Inspection and click "Request indexing".
- Check the Page indexing report after a few days. Pages that are "Crawled, currently not indexed" usually need more useful content; "Blocked by robots.txt" or "Excluded by noindex" sends you back to the first section.
If you serve customers locally, set up a Google Business Profile as well. It's covered in step 4 of our technical checklist for starting a business online.
How do I get my site on Bing, and why does it matter?
Add it to Bing Webmaster Tools, which can import a verified site straight from Google Search Console in a couple of clicks. Bing's index also feeds Microsoft Copilot, so it reaches more than Bing's own search page.
- Submit the same sitemap and use "Submit URLs" for your most important pages.
- Turn on IndexNow. It's a simple protocol that tells search engines the moment a page is added or changed. Bing, Yandex and several others use it; Google doesn't, so you still need Search Console. Many CMS plugins and hosts, including Cloudflare, can send IndexNow pings automatically.
What makes a page easy to rank and to quote in AI answers?
A page that answers one clear question, early and plainly, is easier for Google to rank and easier for AI assistants to quote. We write every post and service page to the same rules:
- One main question per page, phrased the way a customer would type or ask it.
- A title of about 60 characters or fewer, with the main words first, and a meta description of 140 to 160 characters that says what the reader gets.
- A direct answer in the first 40 to 60 words. Context and brand voice come after it.
- Headings that are the real questions, each followed by a one- or two-sentence answer, then the detail.
- Real lists and tables for steps and comparisons, with exact numbers, dates and tool names. They're what answer engines extract and cite.
- Structured data for your organisation, articles and FAQs, following Google's structured data guidelines. Google now shows FAQ rich results mainly for government and health sites, but the markup still helps machines read the page.
- First-hand experience and dates. Say what you did and when, and update the date when you change a page.
Being found and being chosen are different problems. Our own PHP2Shell patch guide appeared in Google results 731 times in its first few weeks but earned a single click: it ranked, but its title and description didn't make anyone pick it. Search Console's Performance report shows exactly this, page by page.
Should I add an llms.txt file?
It's optional and cheap. llms.txt is a proposed standard: a plain-text file at your site's root that summarises who you are and links to your key pages for AI tools. It isn't a Google ranking factor and not every AI system reads it.
If you add one, generate it from the same data as your sitemap so the two never drift apart. That's how ours works: a new blog post appears in the sitemap and in llms.txt automatically.
Your launch-week checklist
| Task | Where | Time |
|---|---|---|
| Read the live robots.txt and remove stray noindex tags | Browser, page source | 15 minutes |
| Redirect every domain variant to one address | Host or CDN | 15 minutes |
| Verify the domain and submit the sitemap | Google Search Console | 20 minutes |
| Request indexing for the homepage and key pages | Search Console URL Inspection | 10 minutes |
| Import the site, submit URLs, turn on IndexNow | Bing Webmaster Tools | 15 minutes |
| Check every title, description and first paragraph | Your CMS or code | An hour or two |
| Check indexing and search queries | Search Console, Bing Webmaster Tools | Weekly for the first month |
Need a hand?
We build this into every website and web application we deliver: the sitemap, structured data and llms.txt are generated from the content, so they stay correct as the site grows. If you're launching something new, or your site is live but nobody seems to find it, tell us about it.