SiteOne Crawler

Find technical website problems and turn a crawl into a readable audit report.

SiteOne Crawler is a free, open-source website crawler for site owners, developers and technical editors. It follows accessible pages and resources, checks technical details and produces an HTML report covering areas such as SEO, performance, accessibility and security. The report is a starting point for investigation: a missing title or broken link is actionable evidence, while an overall score is not a promise of better search rankings.

The project offers a desktop application as well as a command-line tool. The current CLI is a native binary for Windows, macOS and Linux, with x64 and arm64 packages. Basic operation does not require a separate programming-language runtime. Optional browser rendering uses Chromium to inspect JavaScript-generated pages; it requires a compatible browser to be detected or downloaded. Do not confuse the CLI release number with the separate desktop application's version.

Beyond auditing, SiteOne can create offline website copies, export content as Markdown and support automated checks. An offline copy preserves accessible front-end material, not a website's databases or server-side functions. Optional AI analysis connects to a model or provider selected by the operator. Those extra capabilities should be enabled deliberately, with attention to access permissions, server load, data destinations and provider costs.

Features

• Crawl linked pages and resources and identify broken links.
• Generate a local HTML audit report.
• Check technical SEO, performance, accessibility and security signals.
• Export accessible content as an offline copy or Markdown.
• Use optional Chromium rendering for JavaScript-heavy pages.
• Integrate command-line checks into repeatable workflows.

Pros

• Free and open source under the MIT license for the CLI repository.
• Local report workflow does not require uploading the report.
• Desktop and command-line approaches suit different working styles.
• Covers several technical checks in one crawl.

Limits

• Automated findings still need human review and context.
• Crawling makes requests to the target server; permissions and conservative limits matter.
• Optional report upload shares results with the project's infrastructure.
• Optional AI services can move data elsewhere and incur charges.
• Older Linux systems may require the static musl package instead of glibc builds.

FAQ

Is SiteOne Crawler free?

The official project describes it as free. Optional third-party AI providers may have their own charges and conditions.

Does it respect robots.txt?

The documentation says it respects relevant robots.txt restrictions. That does not replace permission to audit a website or sensible request limits.

Can it inspect a JavaScript application?

Optional browser mode renders pages with Chromium, allowing the crawler to inspect the post-JavaScript DOM rather than only the initial HTML response.

Does an offline export back up the entire service?

No. Accessible pages and assets are different from databases, private server logic and live account features.

Sources

https://crawler.siteone.io/
https://crawler.siteone.io/introduction/faq/
https://crawler.siteone.io/installation-and-requirements/ready-to-use-packages/
https://github.com/janreges/siteone-crawler
https://github.com/janreges/siteone-crawler/releases

Official CLI demonstration

The screenshot animation is an older official SiteOne CLI demonstration, not a test by Freewares or a depiction of the current desktop app. Source: https://github.com/janreges/siteone-crawler/blob/main/docs/siteone-crawler-command-line.gif

Image license notice

MIT License

Copyright (c) 2023-2026 Ján Regeš

Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.