Indexing Best Practices

To ensure we can successfully index your site and provide the best possible search experience, follow these best practices to optimize your website's structure and accessibility.

Host a Sitemap at Your Site's Root

A sitemap helps us efficiently locate and index your content.

Best Practices:

• Create an XML sitemap following the Sitemaps.org protocol.

• Post the sitemap at your site root (e.g., https://yourdomain.com/sitemap.xml) for easy discovery.

• Verify that your sitemap is publicly accessible by opening it in a browser.

• Ensure your sitemap is referenced in robots.txt

Ensure URLs Are Publicly Accessible

If your site blocks access, we won’t be able to index it.

Best Practices:

• Test your URLs in Incognito Mode to ensure they load without login credentials

• Ensure no firewall, VPN, or bot protection is blocking access to our crawler

• Remove noindex meta tags from pages that should be indexed

Avoid Redirect Loops

If a page redirects too many times, we may not be able to follow it.

Best Practices:

• Ensure URLs resolve within one or two redirects (e.g., http → https, www → non-www)

• Avoid redirect loops (URL A → URL B → URL A)