> ## Documentation Index
> Fetch the complete documentation index at: https://docs.tryhamsa.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Knowledge base chunking

> How Hamsa turns your documents into searchable pieces, how each chunking method works, how to review and index chunks through the API, and how agents search them on calls.

An agent can't read a whole manual during a call. Instead, Hamsa cuts each knowledge base item into small pieces called **chunks**, turns each chunk into a list of numbers that captures its meaning (an **embedding**), and stores them. When a caller asks something, Hamsa finds the chunks closest in meaning to the question and hands them to the agent.

Good chunks are what make the agent answer correctly. A chunk that holds one complete idea (a question and its answer, one policy, one step) is found when it should be, and gives the agent everything it needs. A chunk that cuts an idea in half, or mixes three topics, is found for the wrong questions or misses the right ones.

This guide explains every step and setting. You don't need to be technical to follow it; the API calls are there when you are ready to use them.

## The journey of an item

```mermaid theme={null}
flowchart LR
  A[Add an item] --> B[Read the text]
  B --> C[Cut into chunks]
  C --> D{autoIndex}
  D -- true --> E[Index]
  D -- false --> R[Review and edit]
  R --> E
  E --> F[Searchable on calls]
```

1. **Add** a text or a file.
2. **Read**: Hamsa takes the text out of the file.
3. **Chunk**: the text is cut into chunks, the way you choose.
4. **Review** (optional): you read the chunks and fix them.
5. **Index**: each chunk is embedded and stored, so it can be found.
6. **Search**: on calls, agents that use the item search its chunks.

## 1. Add an item

[`POST /v1/voice-agents/knowledge-base`](https://docs.tryhamsa.com/api-reference/create-new-knowledge-base-items) takes one or more items:

| Type | What you send | Limits |
| - | - | - |
| `TEXT` | The text itself, in `content` | 50 to 25,000 characters |
| `FILE` | An `https` link to the file, in `itemUrl` | PDF, DOCX, TXT, MD (Markdown), HTML or EPUB |

Each item needs an `itemName` (up to 100 characters). Every item, text or file, counts towards your plan's knowledge base storage, which is shared by every project the owner of this project owns, each member's items included.

### How files are read

| File | How the text is taken out |
| - | - |
| PDF | Read page by page, with text recognition (OCR) for scanned pages. Pages after the 500th are skipped. Page breaks are kept, for `BY_PAGE`. Headings and the reading order of columns are rebuilt from the page layout, so sections keep their titles. |
| DOCX | Headings stay headings, tables become tables, and text in images is recognized. |
| TXT, MD, HTML, EPUB | Titles become headings, lists stay lists, tables stay tables. Page headers and footers are dropped. |

After reading, the text must have at least 50 characters, or the item fails with `READING_FAILURE`.

> [!TIP]
>
> Headings help a lot. Most chunking methods keep each chunk inside one section and start it with the full path of headings it sits under, such as `## Orders > Refunds`, even when a long section is cut into several chunks. So a chunk about "Refunds" under "Orders" carries both words. A document with clear headings gets clearer chunks.

## 2. Two ways to process an item

How you send the request decides the path:

| You send | What happens | Chunks you can review and edit |
| - | - | - |
| No `chunkingConfig`, and `autoIndex` not `false` | Hamsa's standard processing: the item is chunked and indexed in one go | No |
| `chunkingConfig`, or `autoIndex: true` with it | Chunked your way, then indexed right away | Yes, later |
| `autoIndex: false` | Chunked, then it stops at `PENDING_REVIEW` so you can review before indexing | Yes |

To choose a chunking method, or to see and edit chunks, send `chunkingConfig` or `autoIndex: false`.

```bash theme={null}
curl -X POST https://api.tryhamsa.com/v1/voice-agents/knowledge-base \
  -H "Authorization: Token YOUR_SECRET_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "items": [
      {
        "itemName": "Product manual",
        "type": "FILE",
        "itemUrl": "https://example.com/files/product-manual.pdf"
      }
    ],
    "autoIndex": false,
    "chunkingConfig": { "strategy": "SEMANTIC", "targetTokens": 250 }
  }'
```

### Statuses

| Status | Meaning | Searchable |
| - | - | - |
| `PROCESSING` | Being read, chunked or indexed | No |
| `PENDING_REVIEW` | Chunked, waiting for you to review and index | No |
| `PROCESSED` | Indexed | Yes |
| `UPLOAD_FAILURE` | The file couldn't be downloaded or opened | No |
| `READING_FAILURE` | The text couldn't be read or chunked | No |
| `INGESTION_FAILURE` | Indexing failed, or no chunk was stored | No |
| `CANCELLED` | You cancelled it | No |

Read an item's status with [`GET /v1/voice-agents/knowledge-base/{id}`](https://docs.tryhamsa.com/api-reference/endpoint/get-kb-item). Hamsa retries a failed step up to 3 times before marking it failed.

## 3. Choose a chunking method

Set `chunkingConfig.strategy`:

| `strategy` | How it cuts | Best for |
| - | - | - |
| `SEMANTIC` (default) | Keeps a section that fits `targetTokens` as one chunk. A longer section is cut where the topic changes (comparing the meaning of neighbouring sentences), and the pieces are joined back up to `targetTokens`. The slowest. | Most documents: articles, policies, help pages |
| `BY_PARAGRAPH` | Puts whole paragraphs together until the chunk reaches the target size. Never crosses a heading. | Well-written text with clear paragraphs |
| `QA_PAIRS` | One chunk per question and its answer. A question is a line that ends in `?` or `؟`, or starts with `Q`, `Question`, `س` or `سؤال`. The answer runs until the next question or heading. A long answer is split, and each part repeats the question. | FAQs |
| `BY_PAGE` | One chunk per page. A page that's too long is split by paragraph. Each chunk starts with the headings in force where it starts, even from an earlier page. | Files where each page stands alone: slides, brochures, forms |
| `FIXED` | Equal-size pieces that end at the end of a sentence. Ignores meaning. | Long, uniform text like transcripts |
| `LLM_GUIDED` | An AI model reads the text and decides where each chunk should start, and gives it a title. It never rewrites your text. | Text without headings or paragraphs |

### When a method can't be used

The item is chunked with `BY_PARAGRAPH` instead when:

* `QA_PAIRS` finds no questions.
* `BY_PAGE` finds only one page (for example, a `TEXT` item).
* `LLM_GUIDED` can't reach the AI model, or it fails.
* `SEMANTIC` can't measure meaning for a section (only that section is cut by paragraph).

## 4. Size and structure settings

Chunk sizes are measured in **tokens**: small pieces of words. The default 400 tokens is about 270 English words or 180 Arabic words. Every language is counted the same way.

| Setting | Default | Range | What it does |
| - | - | - | - |
| `targetTokens` | `400` | 50 to 1,500 | The size each chunk aims for |
| `minTokens` | `40`, or half of `targetTokens` when that is smaller | 0 to 500 | A chunk smaller than this is joined to a neighbour in the same section, or marked for review |
| `maxTokens` | `1800` | 100 to 1,800 | The hard limit: a longer chunk is cut at a sentence |
| `overlapTokens` | `0` | 0 to 300 | Words repeated from the end of one chunk at the start of the next, so an idea that crosses the cut isn't lost. Every chunk takes one except the first, tables and Q\&A pairs, across sections too. At most half of `targetTokens`, and `0` with `QA_PAIRS`. |
| `respectHeadings` | `true` | | Never cut across a heading, and start each chunk with the full path of headings it sits under |
| `keepAtomicBlocks` | `true` | | Never cut a list, numbered steps or a code block in the middle |
| `splitTables` | `true` | | Cut a large table into groups of rows, each repeating the header row |
| `addContext` | `false` | | Add one or two sentences at the start of each chunk saying where it sits in the document |

A value outside its range is refused with 400 (for example `ChunkingTargetTokensMustBeValid`). Settings that contradict each other are refused with 400 too, never adjusted: `minTokens` and `overlapTokens` can be at most half of `targetTokens` (`ChunkingMinTokensExceedsHalfTarget`, `ChunkingOverlapTokensExceedsHalfTarget`), `maxTokens` must be at least `targetTokens` (`ChunkingMaxTokensBelowTarget`), `minTokens` can't be above `maxTokens` (`ChunkingMinTokensExceedsMaxTokens`), and `QA_PAIRS` takes no `overlapTokens`, since each question and answer has to stand alone (`ChunkingOverlapTokensNotAllowedForQaPairs`). When you leave out `targetTokens`, these are checked against 400. What you send is what runs.

### Where to start

| Content | Settings |
| - | - |
| FAQ | `QA_PAIRS`, `targetTokens` 150 to 250 |
| Manual or policy with headings | `SEMANTIC`, `targetTokens` 200 to 300 |
| Short help articles | `SEMANTIC` with the defaults |
| Slides or one-page sheets | `BY_PAGE` |
| Transcripts | `FIXED`, `targetTokens` 250, `overlapTokens` 40 |
| Text with no structure | `LLM_GUIDED` |

### Adding context (`addContext`)

A chunk like "It costs 50 SAR a month" is hard to find: what costs 50 SAR? With `addContext`, an AI model reads the document (for one over 60,000 characters, its outline and the part around the chunk) and writes a short line for each chunk, in the chunk's language, such as "From the pricing section, about the Premium plan.", and puts it at the start of the chunk. That line is searched along with the chunk. It makes chunking take longer, and it is skipped in previews. A chunk whose line can't be written is saved without one. When no chunk got a line, `contextApplied` is `false` in the item's chunking settings; you can [regenerate](https://docs.tryhamsa.com/api-reference/endpoint/regenerate-kb-item-chunks.md) to try again.

### Marks on chunks

Chunks can come back marked so you know what to check:

| Mark | Meaning |
| - | - |
| Too short | Under `minTokens` and couldn't be joined to a neighbour |
| Oversized | Over the limit |
| Low quality | Scored much lower than the item's other chunks on a quality check. Worth a look; it is kept. |

Exact duplicate chunks are dropped.

With `overlapTokens` above 0, each chunk's `metadata` says whether it got an overlap: `overlapApplied`, with `overlapTokenCount` when it did, or `overlapSkipReason` when it didn't (for example `first_chunk`, `table` or `after_qa_pair`).

## 5. Preview before you commit

Try settings on an item without saving anything (it previews the first 20,000 characters; an item added with standard processing has no stored text to preview): [`POST /v1/voice-agents/knowledge-base/{id}/chunks/preview`](https://docs.tryhamsa.com/api-reference/endpoint/preview-kb-item-chunks).

```bash theme={null}
curl -X POST https://api.tryhamsa.com/v1/voice-agents/knowledge-base/0d3c6a2e-5f1b-4c8e-9a7d-2b4e6f8a1c3d/chunks/preview \
  -H "Authorization: Token YOUR_SECRET_KEY" \
  -H "Content-Type: application/json" \
  -d '{ "chunkingConfig": { "strategy": "QA_PAIRS", "targetTokens": 250 } }'
```

## 6. Review and edit chunks

While an item is `PENDING_REVIEW`, or any time after:

| Task | Endpoint |
| - | - |
| List chunks (up to 100 a page, with search) | [`GET /{id}/chunks`](https://docs.tryhamsa.com/api-reference/endpoint/list-kb-item-chunks) |
| Add a chunk | [`POST /{id}/chunks`](https://docs.tryhamsa.com/api-reference/endpoint/add-kb-item-chunk) |
| Edit a chunk | [`PATCH /{id}/chunks/{chunkId}`](https://docs.tryhamsa.com/api-reference/endpoint/edit-kb-item-chunk) |
| Delete a chunk | [`DELETE /{id}/chunks/{chunkId}`](https://docs.tryhamsa.com/api-reference/endpoint/delete-kb-item-chunk) |
| Many changes at once (all or nothing) | [`POST /{id}/chunks/batch`](https://docs.tryhamsa.com/api-reference/endpoint/batch-edit-kb-item-chunks) |

All paths start with `/v1/voice-agents/knowledge-base`.

| Limit | Value |
| - | - |
| Characters in one chunk | 6,000 |
| Chunks in one item | 5,000 |
| Changes in one batch | 500 |

Edits are refused (422) while the item is `PROCESSING`. After you edit, the item has changes that aren't indexed yet: agents keep finding the old chunks until you index again. Changing only spaces doesn't count as a change.

## 7. Index

[`POST /v1/voice-agents/knowledge-base/{id}/index`](https://docs.tryhamsa.com/api-reference/endpoint/index-kb-item) makes the chunks searchable.

```bash theme={null}
curl -X POST https://api.tryhamsa.com/v1/voice-agents/knowledge-base/0d3c6a2e-5f1b-4c8e-9a7d-2b4e6f8a1c3d/index \
  -H "Authorization: Token YOUR_SECRET_KEY"
```

* Only new and changed chunks are embedded again; deleted chunks are removed from search.
* If nothing changed since the last index, nothing happens.
* A chunk over 2,048 tokens isn't indexed: it is marked with an index error, and the rest of the item is indexed. Shorten it and index again.
* The item is `PROCESSED` when at least one chunk is stored.

## 8. Start over or stop

| Task | Endpoint | What it does |
| - | - | - |
| Re-chunk | [`POST /{id}/chunks/regenerate`](https://docs.tryhamsa.com/api-reference/endpoint/regenerate-kb-item-chunks) | Throws away every chunk and your edits, and cuts the stored text again, with new settings if you send them. Stops for review unless you send `autoIndex: true`. |
| Cancel | [`POST /{id}/cancel`](https://docs.tryhamsa.com/api-reference/endpoint/cancel-kb-item) | Stops an item that is `PROCESSING` and removes what was indexed. The item becomes `CANCELLED`. |

> [!WARNING]
>
> After you regenerate an item that agents already use, they keep finding its old chunks until you index the new ones.

## 9. How agents search on calls

An agent searches only the items you [add to it](https://docs.tryhamsa.com/agents/knowledge-base/overview.md), and only `PROCESSED` items can be added.

The agent's **Agentic RAG** setting decides who starts the search:

| | Agentic RAG off (default) | Agentic RAG on |
| - | - | - |
| Who searches | Hamsa, on each caller turn | The agent itself, through a search tool, when it decides it needs facts |
| What it searches for | What the caller just said, with their previous turn | A search the agent writes, for example "refund window for opened items" |
| Short turns | A turn under 3 words, or only filler such as "نعم" or "ok", isn't searched; recent results are still there | The agent decides |
| Time allowed | 2 seconds. The search starts while the caller is still finishing | 5 seconds |

Either way, every search:

1. Turns the search into an embedding, the same way chunks were embedded. Arabic and English work the same way.
2. Keeps the **3** closest chunks that are at least **75%** similar, and no more than 5 points below the best one.
3. Gives the agent those chunks, each cut to its first **600 characters**.

With Agentic RAG off, chunks found on recent turns stay available for that turn and the next **2**, so a follow-up such as "and for opened items?" still has the facts.

If nothing is close enough, the agent gets no knowledge base text and answers from its prompt.

> [!NOTE]
>
> This is how agents run with the **smart context manager** on, the default for new agents. With it off, Hamsa searches on every caller turn (or when the agent calls the tool), keeps the 5 closest chunks at least 70% similar, gives them whole, and keeps nothing between turns.

> [!TIP]
>
> This is why chunk quality matters: the agent sees at most 3 chunks, and only their first 600 characters. Keep each answer in one chunk and near its start: an answer split across chunks, or buried deep in a long one, may never reach the agent. A `targetTokens` of 150 to 250 keeps chunks within what the agent reads.

## 10. Errors

| Status | Key | What to do |
| - | - | - |
| 400 | `ContentCannotBeEmpty`, `ContentMaxLength` | Keep `TEXT` between 50 and 25,000 characters |
| 400 | `ChunkTextMaxLength` | Keep a chunk under 6,000 characters |
| 400 | `ChunksMaxPerItemExceeded` | An item can hold 5,000 chunks |
| 400 | `KnowledgeBaseItemHasNoChunks` | Add chunks, or regenerate, before indexing |
| 400 | `KnowledgeBaseItemCannotBeRegenerated` | The item has no stored source text to chunk again |
| 400 | `ChunkingMinTokensExceedsHalfTarget`, `ChunkingOverlapTokensExceedsHalfTarget`, `ChunkingMaxTokensBelowTarget`, `ChunkingMinTokensExceedsMaxTokens`, `ChunkingOverlapTokensNotAllowedForQaPairs` | Keep `minTokens` and `overlapTokens` at most half of `targetTokens`, `maxTokens` at least `targetTokens`, `minTokens` not above `maxTokens`, and `overlapTokens` at `0` with `QA_PAIRS` |
| 402 | `FileSizeExceedsMaximumAllowedSize` | The item, or your knowledge base in total, is over your plan's size |
| 403 | `KnowledgeBaseItemNotProcessed` | Index the item before adding it to an agent |
| 403 | `ItemUsedInAgents` | Remove the item from its agents before deleting it |
| 404 | `KnowledgeBaseItemChunkNotFound` | Check the chunk ID |
| 422 | `KnowledgeBaseItemIsProcessing` | Wait until the item is no longer `PROCESSING` |
| 503 | | Hamsa is busy. Try again |

## Examples

> [accordion] A FAQ, reviewed, then indexed
>
> 1. Add it with `"autoIndex": false` and `"chunkingConfig": {"strategy": "QA_PAIRS", "targetTokens": 200}`.
> 2. Wait for `PENDING_REVIEW`, then list the chunks and check each question has its answer.
> 3. Fix or delete chunks, then call `/index`.
> 4. When the item is `PROCESSED`, add it to your agent.

> [accordion] A manual with custom settings
>
> 1. Add the file with `"chunkingConfig": {"strategy": "SEMANTIC", "targetTokens": 250, "overlapTokens": 40, "addContext": true}`.
> 2. It is chunked and indexed in one go; wait for `PROCESSED`.
> 3. List the chunks marked too short or oversized and fix them, then index again.

> [accordion] Updating an item agents already use
>
> 1. Edit the chunks you need to change.
> 2. Call `/index`: only the changed chunks are embedded again.
> 3. Agents find the new text from then on.

## Related

> [card] [Knowledge base in Hamsa Agents](https://docs.tryhamsa.com/agents/knowledge-base/overview.md)
>
> Add items and review chunks in the dashboard.

> [card] [API reference](https://docs.tryhamsa.com/api-reference/create-new-knowledge-base-items.md)
>
> Every knowledge base endpoint.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.