What Is Technical SEO? A Practical Guide
What is technical SEO?
Technical SEO is the work that makes a website easy for search engines to crawl, render and index, and comfortable for people to use. It covers robots.txt, XML sitemaps, canonical tags, redirects, status codes, Core Web Vitals, mobile-friendliness, HTTPS and site architecture. It cannot make weak content rank, but technical problems can stop strong content from ranking at all.
Where technical SEO fits: crawling, indexing and ranking
Search engines first crawl a URL, then decide whether to index it, and only then can it rank. Technical SEO gives you controls at each stage:
- Crawling: robots.txt, internal links, XML sitemaps, server response codes and speed.
- Indexing: noindex directives, canonical tags, redirects, duplicate handling and rendering.
- Ranking and experience: HTTPS, mobile usability and Core Web Vitals.
If these ideas are new, read how search engines work first.
robots.txt: controlling crawling
The robots.txt file sits at the root of a host, such as https://www.example.com/robots.txt, and tells crawlers which paths they may fetch. A typical file for an online shop:
User-agent: *
Disallow: /cart/
Disallow: /checkout/
Disallow: /search/
Disallow: /*?sort=
Sitemap: https://www.example.com/sitemap.xmlTwo points cause most confusion. First, Disallow stops crawling, not indexing: if other pages link to a blocked URL, Google can still index the URL, usually shown without a description. Second, robots.txt is public and voluntary, so it is not a security control; protect private areas with authentication.
noindex compared with robots.txt Disallow
| Question | robots.txt Disallow | noindex |
|---|---|---|
| What it controls | Crawling | Indexing |
| Where it lives | The robots.txt file at the site root | A meta robots tag in the HTML, or an X-Robots-Tag HTTP header |
| Can the URL still appear in results? | Yes, sometimes as a bare URL | No, once Google recrawls the page and sees the directive |
| Must Google crawl the page? | No; it is told not to | Yes; otherwise it never sees the directive |
| Typical use | Stop crawling of low-value URL patterns such as internal search | Keep thank-you pages, thin tag pages or staging copies out of results |
<meta name="robots" content="noindex">
X-Robots-Tag: noindexCanonical tags, duplicates and redirects
The same content often exists at several URLs: with and without www, HTTP and HTTPS, with tracking parameters or with a trailing slash. A canonical tag tells search engines which version you prefer:
<link rel="canonical" href="https://www.example.com/mens-running-shoes/">A canonical tag is a hint, not a directive. Google weighs it alongside redirects, internal links, sitemap entries and HTTPS, and it may choose a different canonical if your signals conflict. Keep them consistent: link internally to the canonical URL and list only canonical URLs in your sitemap.
- 301 or 308: permanent moves; the strongest canonical signal.
- 302 or 307: temporary moves; the old URL usually stays canonical.
- 404 or 410: the page is gone; both lead to removal from the index over time.
- Soft 404: a "not found" page that wrongly returns 200; fix it to return a real error code.
XML sitemaps
An XML sitemap lists the canonical, indexable URLs you want search engines to know about. A single sitemap file can hold up to 50,000 URLs or 50 MB uncompressed; larger sites use a sitemap index that points to several files. Google uses lastmod when it is consistently accurate and ignores priority and changefreq.
Submit the sitemap in Search Console and reference it in robots.txt. A sitemap helps discovery, but it does not force Google to index anything.
Page experience: Core Web Vitals, mobile and HTTPS
| Metric | What it measures | Good threshold |
|---|---|---|
| LCP (Largest Contentful Paint) | Loading: when the largest visible element finishes rendering | 2.5 seconds or less |
| INP (Interaction to Next Paint) | Responsiveness: the delay between user interactions and the next visual update, across the visit | 200 milliseconds or less |
| CLS (Cumulative Layout Shift) | Visual stability: unexpected movement of content | 0.1 or less |
INP replaced First Input Delay (FID) as a Core Web Vital in March 2024. Google assesses these metrics at the 75th percentile of real-user field data from the Chrome UX Report, which Search Console and PageSpeed Insights display. Lab tools such as Lighthouse cannot measure INP directly and use Total Blocking Time as a proxy.
Google uses mobile-first indexing, so the mobile version of a page is what it crawls and indexes; content missing on mobile is effectively missing. HTTPS is a lightweight ranking signal and also avoids browser security warnings.
JavaScript, site architecture and a technical checklist
Google renders JavaScript, but rendering may happen after the initial crawl, and links or content that appear only after user interaction may never be seen. Our guide to SEO for JavaScript sites covers rendering in detail. Architecture matters too: important pages should be reachable within a few clicks through plain HTML links.
- robots.txt blocks only what it should, and not CSS or JavaScript needed for rendering.
- Important pages return 200, are indexable and declare a sensible canonical.
- Redirects go straight to the final URL, without chains.
- The XML sitemap lists only canonical, indexable URLs.
- Core Web Vitals pass for most page groups in field data.
- The mobile version contains the same main content as desktop.
A full SEO audit process turns this checklist into a prioritised fix list. These skills develop fastest on real sites, which is how the practical SEO course teaches them, from basics to advanced.
Common mistakes
- Using robots.txt Disallow to try to remove pages from Google's index.
- Blocking CSS or JavaScript files that Google needs to render the page.
- Pointing canonical tags at URLs that redirect, return 404 or carry noindex.
- Leaving redirect chains in place after several site migrations.
- Listing redirected, noindexed or non-canonical URLs in the XML sitemap.
- Still reporting FID in performance reviews instead of INP.
Frequently asked questions
Does crawl budget matter for a small website?
Rarely. Crawl budget mainly concerns very large sites or sites whose content changes rapidly. A site with a few hundred pages is usually crawled fully, so focus on quality and indexability instead.
How do I remove a page from Google quickly?
Use the Removals tool in Search Console to hide it temporarily, which lasts about six months. For permanent removal, add noindex or return a 404 or 410 status, and leave the URL crawlable so Google sees the change.
Can Google index pages that are not in my sitemap?
Yes. Google discovers most pages by following links. A sitemap supports discovery, particularly for new or poorly linked pages, but it is not required for indexing.
Do I need a developer for technical SEO?
You can diagnose most issues yourself with Search Console and a crawler. Fixes involving server configuration, templates or JavaScript rendering usually need a developer, so learning to write clear tickets is a valuable skill.