> ## Documentation Index
> Fetch the complete documentation index at: https://docs.ilyama.golain.io/llms.txt
> Use this file to discover all available pages before exploring further.

# Vector dataspaces

> Semantic search over your own content — one row per chunk of text plus its embedding, hybrid dense and keyword retrieval, and the two choices you make once.

A **vector** dataspace holds **searchable text**: one row per chunk, each carrying the chunk's words and the embedding of those words. You search it by meaning rather than by exact wording, and you filter the search with ordinary columns (asset, equipment model, manufacturer, revision).

The API modality is **`vector`**.

Most vector dataspaces are fed by a [document dataspace](/dataspaces/document): you upload a PDF, register it with `vectorize: true`, and the platform splits it into chunks that land here. Indexing is **per file and off by default**, so nothing arrives in this corpus until you ask for it. You can also write chunks yourself for content that is not a file.

→ [All modalities](/dataspaces/overview)

## When to use it

**Use vector when**

* You want to ask a question in plain language and get back the passages that answer it — service manuals, SOPs, fault trees, spec sheets, knowledge-base articles.
* Exact tokens matter alongside meaning: part numbers, error codes, bearing designations. Search runs a keyword channel next to the semantic one for exactly this reason.
* Results must be filterable by what they describe. A torque figure from the wrong revision of a manual is worse than no answer, because it looks right.

**Do not use vector** for ordinary structured records you will filter and aggregate. That is [state](/dataspaces/state) or [mutable](/dataspaces/mutable). Embeddings are for text whose meaning you cannot express as a `WHERE` clause.

**Do not use vector** as your file store. The file itself belongs in a [document](/dataspaces/document) dataspace; this one holds the text pulled out of it.

## Two choices you make once

You pick an **embedding model** and an **embedding dimension** at create. Both are permanent. There is no update path for either, and no in-place migration — changing them means creating a new dataspace and re-indexing your content into it.

<Warning>
  **Vectors from two models are not comparable, and comparing them does not fail.** A corpus embedded with one model and searched with another returns a ranked list of confident nonsense: every distance is a valid number, no error is raised anywhere, and the only symptom is that results quietly stop being relevant. Pick the model deliberately, and treat `embedding_model` as part of the dataspace's identity.
</Warning>

| Field             | Rule                                                                                           |
| ----------------- | ---------------------------------------------------------------------------------------------- |
| `embedding_model` | Required. The model name your provider knows, for example `text-embedding-3-small`. Immutable. |
| `embedding_dims`  | Required. The width of the vectors that model produces. Must be **1–2000**. Immutable.         |

### Why the ceiling is 2000

The index that makes semantic search fast cannot be built above **2000 dimensions**. Above that the column still stores your vectors and every search still returns correct results — by scanning every chunk in the dataspace, one at a time. Nothing errors. The symptom is that search is fine on a hundred documents and unusably slow on ten thousand.

Create rejects anything outside `1..2000` up front, so you cannot reach that state through the API. If you want a model whose native width is larger, use a provider option that shortens the output (many models support a requested dimension) and declare the shortened width here.

Choosing a width **below** your model's native output is fine as long as the provider actually produces that width. The vectors you store and the vectors you search with must be the same size and come from the same model.

## Choose an embedding provider

The dataspace needs credentials for the service that turns text into vectors. Supply them as a **Golain integration account** — register the account once for your organization, then point any number of vector dataspaces at it.

The account's provider must declare the **`embed`** capability, and that is checked when you create the dataspace, not on first use. A dataspace that cannot embed would happily accept documents, chunk them, and then never become searchable, with nothing in the response to say so.

```bash theme={null}
curl -sS -X POST "$API/organizations/integrations/accounts" \
  -H "Authorization: Bearer $TOKEN" \
  -H "ORG-ID: $ORG_ID" \
  -H "Content-Type: application/json" \
  -d '{
    "provider_kind": "openai",
    "provider_version": 1,
    "name": "Embeddings (OpenAI)",
    "capabilities": ["embed"],
    "auth_config": { "secrets": { "api_key": "REPLACE_WITH_YOUR_API_KEY" } }
  }'
```

```json theme={null}
{
  "ok": 1,
  "data": {
    "account_id": "6a1f0a4c-8b21-4a55-9a0e-2b7c1f9d3e41",
    "status": "active",
    "auth_ref": { "scope_type": "account", "scope_id": "6a1f0a4c-8b21-4a55-9a0e-2b7c1f9d3e41", "names": ["api_key"] }
  }
}
```

Pass that `account_id` as `embedding_account_id` when you create the dataspace. The key is sealed in the platform's secret store and is never returned by any read endpoint.

`openai` is the provider kind available as an account today. Its `base_url` is settable per account through `endpoint_config`, so an Azure OpenAI deployment or any OpenAI-compatible gateway works through the same path.

<Note>
  **Running your own embedding service?** Set `embedding_provider: "byoe"` together with an HTTPS `embedding_endpoint` instead of `embedding_account_id`. Set one or the other — both is rejected, and so is neither. Naming a credential-bearing provider inline is also rejected: keys belong in an integration account, where they can be rotated and verified in one place rather than copied into every dataspace that needs them.
</Note>

## Backing store

Vector dataspaces are **Postgres**, using the same keyed-row shape as [state](/dataspaces/state): one live row per identity, last-write-wins on `version`, deletes as tombstones.

| Physical detail         | Behavior                                                                                                       |
| ----------------------- | -------------------------------------------------------------------------------------------------------------- |
| Engine                  | Postgres with the `pgvector` extension                                                                         |
| Identity                | `(document_id, chunk_ordinal)` — one row per chunk of one document                                             |
| Layout                  | **`typed` only.** The schema is fixed; there is no overflow.                                                   |
| Indexes                 | A vector index for the semantic channel and a full-text index for the keyword channel, both maintained for you |
| Storage class / tiering | **Not applicable.** `storage_class` and `hot_cold` are rejected.                                               |
| Updates                 | Re-parsing a document overwrites its chunks in place rather than doubling them                                 |

Tiering does not apply because embeddings have no time axis to age along, and an aged-out embedding is one that searches would silently stop finding.

Vector stays `typed` while [document](/dataspaces/document) does not, and the asymmetry is deliberate. Search names these columns literally, and every filter has to be a real indexed column: a predicate inside an overflow blob is stored, readable, and unable to narrow an index scan — which is the whole point of the modality. Fields of your own belong on the **document** row, which is `hybrid` for exactly that reason, and reach a chunk through the columns already copied onto it.

<Note>
  On a self-hosted deployment, `pgvector` must be installed on your Postgres server. If it is not, creating a vector dataspace fails with a message naming the extension — every other modality is unaffected.
</Note>

## The fixed schema

You do **not** declare `fields` for a vector dataspace. The schema is fixed by the platform, and fields you send are ignored: the search endpoint names these columns directly, so a column you added would be one nothing writes and nothing reads.

| Column                            | What it holds                                                                                                                                                                     |
| --------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `document_id`                     | Which document this chunk came from. Half of the row's identity.                                                                                                                  |
| `chunk_ordinal`                   | Position within that document. The other half of the identity.                                                                                                                    |
| `chunk_text`                      | The chunk as it appears in the source. This is what is searched and what you show the reader.                                                                                     |
| `context_prefix`                  | Generated context (manufacturer, model, title, revision, nearest heading) prepended before indexing. Kept in its own column so it can be regenerated without re-parsing the file. |
| `page_from`, `page_to`            | Citation range, when the source carries page information.                                                                                                                         |
| `asset_id`                        | Which asset the source document describes.                                                                                                                                        |
| `equipment_model`, `manufacturer` | What the source document describes.                                                                                                                                               |
| `revision`, `effective_from`      | Which version of the source document this text came from.                                                                                                                         |

The last five are **copied onto every chunk** rather than looked up from the document at query time. That is what makes filtered semantic search fast: a filter on a column of the same row can narrow an index scan, a filter that requires a join cannot.

Every column is nullable. Chunks are written in stages, and a missing value must not reject the row.

The embedding itself is a platform-managed column. It does not appear in the schema you read back, and you never write it directly.

### The chunk row key

You never build one. A chunk's row key is derived server-side from its `document_id` and `chunk_ordinal`, which is what makes re-parsing a document overwrite its chunks in place instead of accumulating a second copy beside them. Supply those two columns and the platform does the rest. The `natural_key` you see on a search result is that derived value — treat it as opaque, and address a chunk by `document_id` and `chunk_ordinal`.

<Note>
  **`page_from` and `page_to` come back empty for chunks the platform parsed for you.** The conversion step works in structured text, which carries headings but not page numbers, so it has none to record. Cite by `context_prefix` — it names the manufacturer, model, title, revision, and nearest heading — and treat the page columns as yours to fill when you [write chunks yourself](#writing-chunks-yourself).
</Note>

## How chunks get their embeddings

Chunks land **without** an embedding and are embedded by a background job shortly afterwards. This is the normal state immediately after a parse, not a fault: embedding a 500-page manual inline would put hundreds of provider calls inside one request, where a single hiccup loses the whole document.

Until a chunk is embedded, it is findable by the keyword channel and invisible to the semantic one. In practice one freshly uploaded document is fully searchable within a minute or two; a bulk import of hundreds of files takes proportionally longer, because the fill works through a bounded number of chunks at a time rather than flooding your provider.

If chunks stay unembedded indefinitely, the provider credential is the first thing to check — a revoked or rate-limited API key stalls the fill with no change visible on the dataspace itself.

## Search

```
POST /projects/{project_id}/dataspaces/{dataspace_id}/search
```

Search is a `POST` even though it reads nothing but the corpus: the body can carry a query embedding, which is hundreds of floats that do not belong in a URL or in every access log.

You need `can_read_data` on the dataspace. Without it — or if the dataspace is not `vector` — you get a **`404`**, not a `403`.

```bash theme={null}
curl -sS -X POST "$API/projects/$PROJECT_ID/dataspaces/$VECTOR_DS_ID/search" \
  -H "Authorization: Bearer $TOKEN" \
  -H "ORG-ID: $ORG_ID" \
  -H "Content-Type: application/json" \
  -d '{
    "query": "hydraulic pump will not build pressure",
    "mode": "hybrid",
    "top_k": 5,
    "equipment_model": "HPU-450",
    "revision": "C"
  }'
```

| Field             | Default  | Notes                                                                                                                          |
| ----------------- | -------- | ------------------------------------------------------------------------------------------------------------------------------ |
| `query`           | —        | Required, always. The keyword channel scores words, not vectors, so it needs the text even when you supply your own embedding. |
| `mode`            | `hybrid` | `hybrid`, `dense`, or `lexical`.                                                                                               |
| `top_k`           | `10`     | Maximum 100. Asking for more returns 100 rather than an error.                                                                 |
| `candidate_k`     | `50`     | How many results each channel produces before they are merged. Raised to at least `top_k` if you set it lower.                 |
| `query_embedding` | —        | Optional. Supply it only if you embedded the query yourself with the same model. Normally the platform embeds it for you.      |

### Modes

| Mode      | What it does                                                                | When to use it                                                                                                                                          |
| --------- | --------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `hybrid`  | Runs both channels and merges the two rankings. **This is the one to use.** | Almost always. Technical text is full of exact tokens — part numbers, error codes — which is precisely what embeddings blur and keyword matching nails. |
| `dense`   | Semantic channel only.                                                      | Evaluating the embedding in isolation.                                                                                                                  |
| `lexical` | Keyword channel only.                                                       | Hunting a known exact token, and the automatic fallback when a query cannot be embedded.                                                                |

Merging uses rank, not score. A semantic distance and a keyword relevance score are different units, and a chunk that only one channel found is exactly what hybrid search exists to surface.

The keyword channel understands quoted phrases and `-term` negation, so `"HPU-450" -decommissioned` works the way a technician typing into a search box expects.

### Filters

Every filter applies to **both** channels, so a filtered hybrid search cannot leak an unfiltered result in through one side.

| Filter                            | Effect                                                                                                                                                                                      |
| --------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `asset_id`                        | Only chunks from documents describing this asset.                                                                                                                                           |
| `equipment_model`, `manufacturer` | Only chunks from documents describing this equipment.                                                                                                                                       |
| `document_id`                     | Search within one document.                                                                                                                                                                 |
| `revision`                        | Only chunks from this revision of the source document. **This is the one that isolates a single edition.**                                                                                  |
| `effective_at`                    | Excludes anything not yet in force at that moment — a chunk whose `effective_from` is later than it. A chunk with no `effective_from` counts as always in force rather than being excluded. |

<Warning>
  **`effective_at` does not hide superseded editions.** It drops revisions that have not taken effect yet; it does not pick the newest of the ones that have. A search with `effective_at` set still returns chunks from every earlier revision that was ever in force. To search exactly one edition, pass `revision`. Track which revision is current for a piece of equipment on your side, and pass it.
</Warning>

Leaving both empty searches **every** revision you have ever loaded. For a maintenance query that is usually the wrong answer, and it is a wrong answer that looks right.

### When search quietly falls back

The platform normally embeds your query server-side, because the provider credential lives in the secret store and your client does not have it. When that embedding call fails:

| Requested mode      | What happens                                                                                                                                |
| ------------------- | ------------------------------------------------------------------------------------------------------------------------------------------- |
| `dense`             | **`502`** with the reason. You explicitly asked for semantic search; running something else instead would be worse than saying no.          |
| `hybrid`, `lexical` | Falls back to the keyword channel and **still returns results**, with `mode: "lexical"` and a `degraded` string in the response saying why. |

Always read `mode` and `degraded` off the response rather than assuming you got what you asked for. Thin results from a keyword fallback and thin results from a genuinely empty corpus look identical otherwise.

`degraded: "embedding is not configured on this deployment"` means the platform could not reach its secret store to open your provider key — an operator problem, not a query problem.

### Response

```json theme={null}
{
  "ok": 1,
  "data": {
    "mode": "hybrid",
    "items": [
      {
        "document_id": "3f8b1c2a-9d4e-5f60-8a71-2b3c4d5e6f70",
        "chunk_ordinal": 42,
        "chunk_text": "If the pump does not build pressure, verify the relief valve setting before replacing the cartridge…",
        "context_prefix": "Acme — HPU-450 — Service Manual — revision C — 4.3 Pressure faults",
        "page_from": null,
        "page_to": null,
        "revision": "C",
        "effective_from": "2026-01-15T00:00:00Z",
        "asset_id": "b21c8f30-77a4-4a11-9c5e-0f2d6a8b4c19",
        "source_id": "3f8b1c2a-9d4e-5f60-8a71-2b3c4d5e6f70",
        "natural_key": "…",
        "score": 0.0325
      }
    ]
  }
}
```

`score` ranks items **within one response**. It is not a similarity percentage and is not comparable between two searches, so do not threshold on it — take the top *n* instead.

`source_id` and `natural_key` are the platform's row identifiers, and `natural_key` is the [derived chunk key](#the-chunk-row-key). Use `document_id` and `chunk_ordinal` when you need to point back at a chunk.

Show the reader `chunk_text`, and use `context_prefix`, `revision`, and the page range where you have one to say where it came from. An answer a technician cannot trace back to a source is an answer they cannot act on.

## Writing chunks yourself

You do not need a document dataspace. To index text that is not a file — ticket resolutions, maintenance notes, a wiki export — write chunk rows directly through the batch row API and let the background job embed them:

```bash theme={null}
curl -sS -X POST "$API/projects/$PROJECT_ID/dataspaces/$VECTOR_DS_ID/rows/batch" \
  -H "Authorization: Bearer $TOKEN" \
  -H "ORG-ID: $ORG_ID" \
  -H "Content-Type: application/json" \
  -d '{
    "rows": [
      {
        "source_id": "3f8b1c2a-9d4e-5f60-8a71-2b3c4d5e6f70",
        "data": {
          "document_id": "3f8b1c2a-9d4e-5f60-8a71-2b3c4d5e6f70",
          "chunk_ordinal": 0,
          "chunk_text": "Replaced the relief valve cartridge; pressure recovered to 210 bar.",
          "context_prefix": "Acme — HPU-450 — work order 88213",
          "equipment_model": "HPU-450",
          "manufacturer": "Acme",
          "asset_id": "b21c8f30-77a4-4a11-9c5e-0f2d6a8b4c19"
        }
      }
    ]
  }'
```

Rules that matter:

* `document_id` and `chunk_ordinal` are the row's identity, and the row key is derived from them server-side — do not send one. Supply a stable `document_id` per logical source and number the chunks from `0`, and re-writing a source replaces its chunks instead of doubling them.
* `document_id` must be a UUID. Generate one per source and keep it.
* Fill `equipment_model`, `manufacturer`, `revision`, `effective_from`, and `asset_id` yourself. Nothing copies them onto hand-written chunks, and an unfilled column cannot be filtered on.
* Only declared columns are accepted. An extra key rejects the batch rather than being absorbed — there is no overflow on this modality.

Keep chunks meaningful rather than uniform. A chunk that ends mid-table answers nothing, and a chunk that spans a whole document embeds to an average of everything it contains.

To retire a source, tombstone its chunks with a predicate delete on `document_id`:

```bash theme={null}
curl -sS -X DELETE "$API/projects/$PROJECT_ID/dataspaces/$VECTOR_DS_ID/rows" \
  -H "Authorization: Bearer $TOKEN" \
  -H "ORG-ID: $ORG_ID" \
  -H "Content-Type: application/json" \
  -d '{"where": {"document_id": "3f8b1c2a-9d4e-5f60-8a71-2b3c4d5e6f70"}}'
```

Tombstoned chunks stop answering searches immediately. Writing the same identity again with a higher version brings the chunk back, so a re-index after a delete behaves the way you would want.

## Create

```bash theme={null}
curl -sS -X POST "$API/projects/$PROJECT_ID/dataspaces" \
  -H "Authorization: Bearer $TOKEN" \
  -H "ORG-ID: $ORG_ID" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "manual_chunks",
    "modality": "vector",
    "embedding_model": "text-embedding-3-small",
    "embedding_dims": 1536,
    "embedding_account_id": "6a1f0a4c-8b21-4a55-9a0e-2b7c1f9d3e41"
  }'
```

`$API` is the [platform base URL](/api-reference/introduction) (production: `https://api.ilyama.golain.io/core/api/v1`).

Create this **before** the [document dataspace](/dataspaces/document) that will feed it. A document dataspace names its vector dataspace at create and cannot be repointed afterwards, and without that pairing every attempt to register a file with `vectorize: true` is rejected.

## Related

* [Document dataspaces](/dataspaces/document) — files, uploads, and the end-to-end walkthrough
* [Dataspaces overview](/dataspaces/overview)
* [State](/dataspaces/state) — the keyed-row shape vector dataspaces share
* [Schema and layouts](/dataspaces/schema)
* [API introduction](/api-reference/introduction)