Back to Help

Crawl Artifacts

AI-optimized files generated after each crawl — sitemap, sitewide FAQ, and LLM policy manifest.

What are Crawl Artifacts?

After Adventyx crawls your website, it automatically generates three files designed to help AI systems and search engines understand your content better. You can find them on your Website Details page under the Crawl Artifacts card.

ArtifactFileWhat it does
Sitemap LLMsitemap_llm.xmlAn AI-optimized sitemap that points to clean, structured versions of your pages — easier for LLMs to read than raw HTML
Sitewide FAQsitewide_faq.htmlAll your approved FAQs in one page, with FAQ schema markup for rich search results and AI consumption
LLM.txt Manifestllm.txtA policy file that tells AI crawlers (GPTBot, Claude, Gemini, Grok, and others) how they may access and use your content
Tip: Artifacts are regenerated automatically after each crawl and whenever you edit FAQs. You can also click Regenerate on the Sitewide FAQ in Adventyx anytime you want to force a refresh.

Why Crawl Artifacts matter

When someone asks an AI assistant a question about businesses like yours, the assistant needs structured, trustworthy content to form an answer. Crawl Artifacts give AI systems a clean, authoritative view of your site — separate from the visual design of your CMS pages.

Approved FAQs feed into the Sitewide FAQ artifact. See the FAQs guide for how to approve and publish FAQ content.


Option A — WordPress plugin (recommended)

If your site runs on WordPress, the Adventyx Publisher plugin is the lowest-effort path and keeps everything on your own domain.

  1. 1Install the Adventyx Publisher plugin and enter your API key. See the WordPress Plugin guide.
  2. 2The plugin automatically serves your Sitewide FAQ at yoursite.com/sitewide_faq/ and an llm.txt file at your site root (includes your published content).
  3. 3After you approve or edit FAQs in Adventyx, click Refresh in the plugin settings to update the sitewide FAQ page.
Tip: The LLM Sitemap (sitemap_llm.xml) does not auto-sync to WordPress today — download and upload it manually if you want it on your domain.

Option B — Download and host manually

If you prefer to manage files yourself or do not use WordPress:

  1. 1Go to Website Details → Crawl Artifacts in Adventyx.
  2. 2Click Download on llm.txt and sitemap_llm.xml.
  3. 3Upload both files to the root of your website so they are accessible at yoursite.com/llm.txt and yoursite.com/sitemap_llm.xml.
  4. 4Optionally add a line to your robots.txt pointing to the sitemap: Sitemap: https://yoursite.com/sitemap_llm.xml

For the Sitewide FAQ, download the HTML file and either host it as a standalone page or use the WordPress plugin path above.


Option C — Adventyx-hosted links (zero setup)

All three artifacts are also available via Adventyx-hosted links with canonical references back to your real pages. This works without any changes to your website.

  • No installation or file uploads required
  • Artifacts update automatically when Adventyx regenerates them
  • Hosting on your own domain (Options A or B) is generally better for SEO and brand trust
Tip: A practical setup: use the WordPress plugin for the Sitewide FAQ and llm.txt, then download and upload sitemap_llm.xml to your site root once and re-upload after major site changes or recrawls.