# Direqt Crawlers & User Agents

Direqt uses web crawlers (“robots”) and user agents to perform actions for its products, either automatically or triggered by user request. Direqt uses the following `robots.txt` tags to enable webmasters to manage how their sites and content work with our product.

> **User agent headers**\
> `Anomura/1.2 (+https://www.direqt.ai)` 
>
> `Anomura/1.2 (+https://www.direqt-search.com)`
>
> **User agent token**\
> `Anomura`

***

### Overview

**Anomura** is for search. Anomura is used to link to and surface websites in search results in Direqt’s search features. It is **not** used to crawl content to train any models.

To help ensure your site appears in search results, we recommend allowing Anomura in your site’s `robots.txt` file **and** allowing requests from our published IP ranges below.

> **Note:** The full user-agent string will contain either `Anomura/1.2 (+https://www.direqt.ai)` or
>
> `Anomura/1.2 (+https://www.direqt-search.com)`

***

### User agents

| Agent   | Token     | Full user-agent strings                                                                                                      | Purpose                                                                                            |
| ------- | --------- | ---------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------- |
| Anomura | `Anomura` | <p><code>Anomura/1.2 (+<https://www.direqt.ai>)</code></p><p><code>Anomura/1.2 (+<https://www.direqt-search.com>)</code></p> | Direqt search crawler. Discovers links/metadata to surface in search. Not used for model training. |

***

### Control access with `robots.txt`

By default, Anomura follows standard `robots.txt` directives addressed to the `Anomura` token.

#### Allow Anomura everywhere

```ini
User-agent: Anomura
Allow: /
```

#### Block Anomura completely

```ini
User-agent: Anomura
Disallow: /
```

> Blocking Anomura means your content will not appear in Direqt search features.

#### Allow Anomura but block a specific path

```ini
User-agent: Anomura
Disallow: /private/
```

#### Allow Anomura while blocking all other crawlers

```ini
# robots.txt
User-agent: *
Disallow: /

User-agent: Anomura
Allow: /
```

***

### Source IP ranges

Allow traffic from the following IPs to ensure Anomura can access your site when permitted by your `robots.txt` and server configuration.

* **Published IP addresses:**&#x20;
  * `34.56.242.161`      (Hostnam&#x65;**:** `crawler.direqt-search.com`)
  * `35.224.164.246`    (Hostnam&#x65;**:** `crawler.direqt.ai`)

The IP addresses used by the crawler are also listed [here](https://www.direqt.ai/crawler.json).

***

### FAQ

**Does Anomura train Direqt models?**\
No. Anomura is only used to discover and surface links and metadata for Direqt search features. Content fetched by Anomura is not used for model training.

**Does Anomura respect `robots.txt`?**\
Yes. Anomura follows standard `robots.txt` directives for the `Anomura` user agent token.

**Will you add more IPs?**\
If additional IPs or ranges are published, we’ll update this page. Consider allowing by both user-agent and IP where possible.


---

# Agent Instructions: Querying This Documentation

If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter:

```
GET https://docs.direqt-search.com/direqt-bots/direqt-crawlers-and-user-agents.md?ask=<question>
```

The question should be specific, self-contained, and written in natural language.
The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.