Robots.txt

A robots.txt file tells crawlers which URLs they may crawl on your site. It does not directly control indexing; for indexing control, use noindex and related mechanisms. Configure robots rules carefully to avoid blocking important content. See Access your robots.txt file. See How do I prevent specific pages from getting indexed by search engines?.

Related terms

Related terms

  • Index Toggle

    SEO

    A setting controlling whether search engines should index a specific page, useful for excluding drafts or duplicate content. Noindex pages remain accessible but won’t appear in search results. Control indexing in Framer’s page settings for SEO management.

  • Crawling

    SEO

    The process by which search engine bots discover and read web pages by following links and analyzing content. Ensuring your site is crawlable with proper HTML structure, sitemaps, and internal linking improves search visibility. Block sensitive or duplicate pages from crawling using robots.txt directives.

  • Indexing

    SEO

    The process by which search engines add web pages to their database after crawling, making them available in search results. Well-structured sites with quality content get indexed more thoroughly. Monitor indexing status in Google Search Console to ensure pages are discoverable.

  • XML Sitemap

    SEO

    An XML Sitemap is a structured file that lists important URLs and metadata so search engines can crawl and index a website more efficiently.

Academy