Great content does nothing if a crawler cannot reach it, render it or understand which version is the real one. Technical SEO is the plumbing under every ranking and every AI answer that cites you. This checklist follows the path a crawler takes: can it get in, can it read the page, will it index the right URL, and does the page give a good experience once someone arrives.
1. Let crawlers in
Start with robots.txt at the root of the domain. Check that it does not block whole sections by accident, that it lists your sitemap, and that your rules for AI crawlers are deliberate rather than inherited from a template. A single Disallow: / under User-agent: * removes a site from search, and we still find it on staging configs that went live.
Then check status codes. Important pages should return 200, moved pages a single 301 to the final URL, and deleted pages 404 or 410. Redirect chains waste crawl effort and dilute signals, so collapse them to one hop.
- robots.txt returns 200 and lists the sitemap
- No accidental Disallow on key folders
- One-hop 301s, no chains or loops
- Server errors (5xx) monitored, not discovered by users
2. Make the page readable without guesswork
Google renders JavaScript, but rendering is a second step and other crawlers, including several AI assistants, often read only the raw HTML. If your main text, links and headings only appear after scripts run, some readers will see an empty page. Server-side rendering or static generation for marketing pages removes the risk.
Use Search Console's URL Inspection tool to see the rendered HTML Google stored. Google's own guidance on AI search tells site owners to confirm Google sees pages the way users do.
3. Tell search engines which URL is the real one
Every indexable page needs a self-referencing canonical tag, a unique title, a meta description and one H1 that states the topic. Parameter URLs, print views and tracking links should point their canonical to the clean URL.
Keep an XML sitemap of only the canonical, indexable 200 pages, with honest lastmod dates. A sitemap full of redirects or noindex pages teaches crawlers to trust it less.
| Check | Good | Common problem |
|---|---|---|
| Canonical | Self-referencing on every indexable page | Points to a redirected or noindex URL |
| Title | Unique, keyword first, under about 60 characters | Same title across a template |
| H1 | Exactly one, matches search intent | Logo or none at all |
| Sitemap | Only canonical 200 URLs | Includes redirects and 404s |
4. Pass Core Web Vitals on mobile
Google judges Core Web Vitals at the 75th percentile of real visits, separately for mobile and desktop. The targets are LCP within 2.5 seconds, INP within 200 milliseconds and CLS of 0.1 or less.
The usual fixes are unglamorous: compress and size hero images, preload the main font, defer third-party scripts, reserve space for images and embeds, and break up long JavaScript tasks.
5. Describe the business in structured data
Organization, WebSite and BreadcrumbList markup help search engines and AI tools connect your pages to your brand. Add Service, Product or Article markup where the page really is one of those things.
Valid markup makes pages eligible for rich results, but eligibility is not a guarantee, and some types, such as FAQ, now show rich results only for a narrow set of sites.
6. Check AI crawler access and measure
Decide which AI crawlers you allow. Training crawlers and search crawlers are now separate for OpenAI and Anthropic, so you can opt out of training while staying visible in AI search answers.
Finally, measure. Search Console shows clicks, impressions, click-through rate and average position per page and query. XDQ xAgents' AI Visibility agent runs the crawler, robots.txt, sitemap, llms.txt, heading, structured data and canonical checks in this list and turns them into a fix list.
Frequently asked questions
What is technical SEO?
Technical SEO is the work that makes a site easy for search engines and AI crawlers to reach, render, understand and index: robots.txt, status codes, rendering, canonicals, sitemaps, page speed and structured data.
How often should I run a technical SEO audit?
Run a full audit at least every quarter and after every redesign, migration or platform change. Monitor robots.txt, status codes and Core Web Vitals continuously, because one bad deploy can remove pages from search.
What are the Core Web Vitals thresholds?
Largest Contentful Paint of 2.5 seconds or less, Interaction to Next Paint of 200 milliseconds or less, and Cumulative Layout Shift of 0.1 or less, measured at the 75th percentile of page loads.
Does AI search need different technical SEO?
Mostly no. Google's 2026 guide says AEO and GEO do not require fundamentally new approaches. The extra step is deciding which AI crawlers may read your site and making sure your main content is in the HTML.
Sources
- Web Vitals, web.dev (Google), 2026
- Introduction to robots.txt, Google Search Central, 2026
- Understand JavaScript SEO basics, Google Search Central, 2026
- Overview of OpenAI crawlers, OpenAI, 2026
- Google's new guide for AI search: what SEO really needs now, PPC Land, 2026
