Technical SEO audit guide: how a website crawler finds what to fix first
Most website audits end as a PDF with three thousand warnings that nobody reads. A useful technical SEO audit is shorter: it finds the problems that stop search engines from crawling, indexing and ranking your pages, puts them in the order that protects the most traffic, and checks that the fixes stay fixed. Here is how to do one, with or without a site audit tool.
What a technical SEO audit covers
A technical audit answers three questions about every page: can a search engine reach it, is it allowed to index it, and can it understand it? Everything else, from redirect chains to missing alt text, is a detail of one of those three.
Content quality and backlinks are separate jobs; the technical audit makes sure neither is wasted.
What is a website crawler?
A website crawler is a program that reads a site the way a search engine does. It starts from the home page and the XML sitemaps, follows every internal link, and records the status code, headers and HTML of each URL it reaches. Search engines run crawlers to discover pages; an SEO website crawler does the same so you can see your site through their eyes.
What makes it an audit is the second step: running checks against everything the crawler collected, and comparing pages with each other. Duplicate titles, orphan pages, redirect chains and canonicals that point to broken URLs only show up when the whole site is crawled, which is why checking one URL at a time misses most technical problems.
Before you crawl: set up the website crawler
A crawl is only as useful as its settings. Decide these before you start:
- Which host and sections. The whole domain, one subdomain or a folder. For a large store, audit the catalog and the blog as separate projects or with include patterns.
- Which user agent. Crawling as Googlebot on mobile shows what Google sees on the mobile-first index; your own bot name is friendlier to server logs.
- Robots rules. Respect robots.txt to see the site as search engines do. Ignore it only on your own site, when you need to check a blocked section.
- JavaScript. If content or links are rendered in the browser, a plain HTML crawl will miss them. Render JavaScript for single-page apps and modern frameworks.
- Speed. A few requests per second is plenty. An audit that slows down the shop is not worth it.
The checks that matter most
Hundreds of checks exist, but they fall into a handful of groups. Work through them in this order, because problems higher up hide the ones below.
Indexability
Pages returning 4xx or 5xx, noindex tags on pages you want in search, canonicals pointing to broken or redirected URLs, and sitemap URLs that are blocked, redirected or noindex. One wrong canonical on a category template can remove hundreds of pages from Google.
Links and redirects
Broken internal links, redirect chains and loops, internal links that point to redirects, orphan pages with no internal links, and pages buried too many clicks from the home page. Internal links are how both users and crawlers find your pages, and how ranking signals flow between them.
Titles, headings and content
Missing, duplicate or truncated titles and descriptions, missing or duplicate H1s, thin pages, near-duplicates and soft 404s. These decide what people see in the results and whether Google considers a page worth indexing.
Speed, security and markup
Slow server responses, heavy HTML, Core Web Vitals of key pages, mixed content and HTTP pages, invalid JSON-LD, hreflang errors on international sites, and site files: robots.txt, XML sitemaps and llms.txt, including whether AI crawlers are blocked.
Prioritise by traffic, not by count
The classic mistake is to sort the issue list by the number of affected pages. Two hundred images without alt text look scarier than two noindex tags, yet the noindex tags may sit on your best-selling category.
Rank every issue by severity × the importance of the pages it touches. Importance is best measured with real clicks from Google Search Console; without it, use the traffic of your tracked keywords from a rank tracker, and without that, internal link weight. A broken link on a page with 2,000 visits a month then beats a cosmetic warning on a page nobody opens.
Look for errors on pages that already rank on positions 1–20. Those are the fixes most likely to show up in traffic within weeks.
Fix, re-crawl and verify
An audit is a loop, not a document. A workable routine looks like this:
- 1Take the top ten issues by impact and hand each one over with the affected URLs and a short explanation of the fix.
- 2Fix template-level problems first: one change to a product template can clear an issue on thousands of pages.
- 3Re-crawl the site, or the affected section, once the changes are live.
- 4Compare the new crawl with the previous one: which issues disappeared, which are new, and what changed on pages that rank.
- 5Record what was fixed and when, so a later traffic change can be traced back to the work.
How often to run an SEO site audit
Weekly for online stores, publishers and any site with regular releases; monthly for small, stable sites. Add an extra crawl after every redesign, migration, CMS or plugin update, and after large content imports. Scheduled crawls with a diff against the previous one catch regressions, like a staging noindex tag that reached production, before they cost rankings.
Running it with Aseoma’s site audit tool
Aseoma’s Site Audit runs more than a hundred checks in twelve groups on every crawled page, ranks the issues by the traffic at risk using your Search Console clicks and tracked keywords, and tests the key pages with PageSpeed Insights in the same run. Each crawl is compared with the previous one, fixed issues are recorded with their dates, and internal link suggestions show where your own copy already mentions a keyword another page ranks for.
To see it on your own site first, run the free SEO audit: up to 100 pages, no sign-up. Agencies can schedule audits for every client and share the reports; see Aseoma for agencies and the plans and prices.