Auto Chunking

Under AI → Auto Chunking you add rules that split the files uploaded to a bucket into text chunks, ready to embed and search. A rule says which files it takes, the collection their chunks go to, and how big a chunk is. Every file uploaded to the bucket, by your app or by the console, is split by the first rule it matches. A bucket without rules is never touched.

The chunks get their vectors from their collection’s own rule, set under Auto Embeddings like for any other collection. So embedding is set in one place: the collection where the vectors live and where you search.

The rule

Open a bucket and Add chunking rule:

Name: written on every chunk as rule.
Condition: which files the rule takes, below. Empty: every file.
Chunks into: the collection of the chunks, <bucket>_chunks by default, created empty on the first file if missing.
Advanced: chunk size in characters, 1000 by default; the overlap taken from the chunk before, 200 by default; the splitter: auto cuts source code at function and class boundaries and the rest as prose, text always cuts as prose, code cuts at boundaries whenever the language is known.

Prose is cut at paragraphs first. A paragraph longer than a chunk is cut at lines, then sentences, then words, and the pieces are joined back up to the chunk size. The overlap is whole sentences or words from the chunk before. A word is never cut. A blank page makes no chunk, and a chunk with fewer than 20 letters and digits, such as a page number or a running header, is joined to its neighbour.

Save writes the whole list on the bucket, as chunking in its metadata. The rules are tried in order and the first one a file matches applies, as in an ACL. The arrows move a rule up or down.

The condition

A JSON object with three keys, each optional; when there are more, all must match.

Key Matches

contentType

the type detected from the file’s content, a string or a list; a trailing matches any suffix: "application/vnd.openxmlformats-officedocument.".

extension

the extension of the file’s name, a string or a list, with or without the dot.

metadata

a MongoDB filter on the file’s metadata, the document your app sends with the upload. Every query operator works except those that run code.

The quick buttons under the field write the common ones: PDF, Office, Text, Markdown, HTML, Source code, By folder, By metadata.

By folder. A bucket has no folders. A folder is part of the path, which your app sends in the file’s metadata.path, next to the name in metadata.filename. Two rules on a bucket docs send the contracts and the design documents to two collections:

{ "chunking": [
    { "name": "legal",
      "filter": { "metadata": { "path": { "$regex": "^legals/" } } },
      "target-collection": "legal_chunks" },
    { "name": "tech",
      "filter": { "extension": ".pdf", "metadata": { "path": { "$regex": "^design/" } } },
      "target-collection": "tech_chunks" } ] }

legals/contratto.pdf goes to legal_chunks, design/project.pdf to tech_chunks. The condition sees only what is in metadata, so upload with the path there.

curl -X POST https://<service>/docs.files \
  -H "Authorization: Bearer <token>" \
  -F file=@contratto.pdf \
  -F 'metadata={"filename": "contratto.pdf", "path": "legals/contratto.pdf"}'

By metadata. A field of your own does the same without a folder: {"metadata": {"area": "legal"}}, uploading with metadata={"filename": "contratto.pdf", "area": "legal"}.

Under the rule

Chunks: how many the rule wrote.
Embedding: the rule of the chunks collection, if it has one, with a link to Auto Embeddings. Without a rule the chunks have no vector. Add embedding opens Auto Embeddings on that collection with text into vector filled in and, for a contextual model, the chunks of a file grouped by fileId, so each vector reflects the rest of its document.

Each chunk carries text, chunkIndex, fileId, source, filename, contentType, the file’s metadata and rule. Replacing or deleting a file replaces or deletes its chunks.

In the background

The upload answers as soon as the file is stored, and the file is split and embedded afterwards: a large PDF takes seconds. The file’s own document shows the progress under chunking: pending right after the upload, then done with the rule applied and the number of chunks, skipped when no rule matched, or failed with the reason in warnings.

GET /docs.files?filter={"_id":{"$oid":"<id>"}}

A service chunks two files at a time on Free, four on Starter and eight on Pro; the others wait their turn.

Under the rules of a bucket, Files shows its queue: how many files are waiting, running, done, skipped and failed; the files waiting or running, oldest first; and the ones that failed, with the reason. While something is waiting or running the list refreshes on its own, and the bucket’s row shows how many are in the queue.

Upload a file to try

Under the rules, choose a file and, optionally, a folder. The file is uploaded to the bucket with its name in metadata.filename and the folder and name in metadata.path, as your app would do. When the chunking is done, the page shows which rule the file matched, its chunks, whether they have a vector, and any warnings. The file stays in the bucket: Delete the file and its chunks removes both.

Upload a folder

Under a bucket, Upload a folder shows a Python script with the service’s URL and the bucket already in it. It uploads every file under a directory, subfolders included, each with its name in metadata.filename and its path from the directory in metadata.path. Put a token in place of <token> and run it with the directory:

python upload.py ./documents

Limits

The chunks of a file are written in one request, with the limits of any request. On a Free service a file is up to 2 MB, and so are its chunks together. On Starter and Pro a file is up to 10 MB, and its chunks up to 5 MB together. When the chunks exceed the limit, the file is stored and its chunks are not, and the upload’s warning says so. Free is for trying; for a real archive, use a paid service. The chunks and their vectors count toward the storage of the plan.

What it costs

Splitting is free. Embedding the chunks is billed by your provider to your account, one embedding per chunk, or one call per file with a contextual model grouped by fileId.