vector.
Most vector dataspaces are fed by a document dataspace: you upload a PDF, register it with vectorize: true, and the platform splits it into chunks that land here. Indexing is per file and off by default, so nothing arrives in this corpus until you ask for it. You can also write chunks yourself for content that is not a file.
→ All modalities
When to use it
Use vector when- You want to ask a question in plain language and get back the passages that answer it — service manuals, SOPs, fault trees, spec sheets, knowledge-base articles.
- Exact tokens matter alongside meaning: part numbers, error codes, bearing designations. Search runs a keyword channel next to the semantic one for exactly this reason.
- Results must be filterable by what they describe. A torque figure from the wrong revision of a manual is worse than no answer, because it looks right.
WHERE clause.
Do not use vector as your file store. The file itself belongs in a document dataspace; this one holds the text pulled out of it.
Two choices you make once
You pick an embedding model and an embedding dimension at create. Both are permanent. There is no update path for either, and no in-place migration — changing them means creating a new dataspace and re-indexing your content into it.Why the ceiling is 2000
The index that makes semantic search fast cannot be built above 2000 dimensions. Above that the column still stores your vectors and every search still returns correct results — by scanning every chunk in the dataspace, one at a time. Nothing errors. The symptom is that search is fine on a hundred documents and unusably slow on ten thousand. Create rejects anything outside1..2000 up front, so you cannot reach that state through the API. If you want a model whose native width is larger, use a provider option that shortens the output (many models support a requested dimension) and declare the shortened width here.
Choosing a width below your model’s native output is fine as long as the provider actually produces that width. The vectors you store and the vectors you search with must be the same size and come from the same model.
Choose an embedding provider
The dataspace needs credentials for the service that turns text into vectors. Supply them as a Golain integration account — register the account once for your organization, then point any number of vector dataspaces at it. The account’s provider must declare theembed capability, and that is checked when you create the dataspace, not on first use. A dataspace that cannot embed would happily accept documents, chunk them, and then never become searchable, with nothing in the response to say so.
account_id as embedding_account_id when you create the dataspace. The key is sealed in the platform’s secret store and is never returned by any read endpoint.
openai is the provider kind available as an account today. Its base_url is settable per account through endpoint_config, so an Azure OpenAI deployment or any OpenAI-compatible gateway works through the same path.
Running your own embedding service? Set
embedding_provider: "byoe" together with an HTTPS embedding_endpoint instead of embedding_account_id. Set one or the other — both is rejected, and so is neither. Naming a credential-bearing provider inline is also rejected: keys belong in an integration account, where they can be rotated and verified in one place rather than copied into every dataspace that needs them.Backing store
Vector dataspaces are Postgres, using the same keyed-row shape as state: one live row per identity, last-write-wins onversion, deletes as tombstones.
Tiering does not apply because embeddings have no time axis to age along, and an aged-out embedding is one that searches would silently stop finding.
Vector stays
typed while document does not, and the asymmetry is deliberate. Search names these columns literally, and every filter has to be a real indexed column: a predicate inside an overflow blob is stored, readable, and unable to narrow an index scan — which is the whole point of the modality. Fields of your own belong on the document row, which is hybrid for exactly that reason, and reach a chunk through the columns already copied onto it.
On a self-hosted deployment,
pgvector must be installed on your Postgres server. If it is not, creating a vector dataspace fails with a message naming the extension — every other modality is unaffected.The fixed schema
You do not declarefields for a vector dataspace. The schema is fixed by the platform, and fields you send are ignored: the search endpoint names these columns directly, so a column you added would be one nothing writes and nothing reads.
The last five are copied onto every chunk rather than looked up from the document at query time. That is what makes filtered semantic search fast: a filter on a column of the same row can narrow an index scan, a filter that requires a join cannot.
Every column is nullable. Chunks are written in stages, and a missing value must not reject the row.
The embedding itself is a platform-managed column. It does not appear in the schema you read back, and you never write it directly.
The chunk row key
You never build one. A chunk’s row key is derived server-side from itsdocument_id and chunk_ordinal, which is what makes re-parsing a document overwrite its chunks in place instead of accumulating a second copy beside them. Supply those two columns and the platform does the rest. The natural_key you see on a search result is that derived value — treat it as opaque, and address a chunk by document_id and chunk_ordinal.
page_from and page_to come back empty for chunks the platform parsed for you. The conversion step works in structured text, which carries headings but not page numbers, so it has none to record. Cite by context_prefix — it names the manufacturer, model, title, revision, and nearest heading — and treat the page columns as yours to fill when you write chunks yourself.How chunks get their embeddings
Chunks land without an embedding and are embedded by a background job shortly afterwards. This is the normal state immediately after a parse, not a fault: embedding a 500-page manual inline would put hundreds of provider calls inside one request, where a single hiccup loses the whole document. Until a chunk is embedded, it is findable by the keyword channel and invisible to the semantic one. In practice one freshly uploaded document is fully searchable within a minute or two; a bulk import of hundreds of files takes proportionally longer, because the fill works through a bounded number of chunks at a time rather than flooding your provider. If chunks stay unembedded indefinitely, the provider credential is the first thing to check — a revoked or rate-limited API key stalls the fill with no change visible on the dataspace itself.Search
POST even though it reads nothing but the corpus: the body can carry a query embedding, which is hundreds of floats that do not belong in a URL or in every access log.
You need can_read_data on the dataspace. Without it — or if the dataspace is not vector — you get a 404, not a 403.
Modes
Merging uses rank, not score. A semantic distance and a keyword relevance score are different units, and a chunk that only one channel found is exactly what hybrid search exists to surface.
The keyword channel understands quoted phrases and
-term negation, so "HPU-450" -decommissioned works the way a technician typing into a search box expects.
Filters
Every filter applies to both channels, so a filtered hybrid search cannot leak an unfiltered result in through one side.
Leaving both empty searches every revision you have ever loaded. For a maintenance query that is usually the wrong answer, and it is a wrong answer that looks right.
When search quietly falls back
The platform normally embeds your query server-side, because the provider credential lives in the secret store and your client does not have it. When that embedding call fails:
Always read
mode and degraded off the response rather than assuming you got what you asked for. Thin results from a keyword fallback and thin results from a genuinely empty corpus look identical otherwise.
degraded: "embedding is not configured on this deployment" means the platform could not reach its secret store to open your provider key — an operator problem, not a query problem.
Response
score ranks items within one response. It is not a similarity percentage and is not comparable between two searches, so do not threshold on it — take the top n instead.
source_id and natural_key are the platform’s row identifiers, and natural_key is the derived chunk key. Use document_id and chunk_ordinal when you need to point back at a chunk.
Show the reader chunk_text, and use context_prefix, revision, and the page range where you have one to say where it came from. An answer a technician cannot trace back to a source is an answer they cannot act on.
Writing chunks yourself
You do not need a document dataspace. To index text that is not a file — ticket resolutions, maintenance notes, a wiki export — write chunk rows directly through the batch row API and let the background job embed them:document_idandchunk_ordinalare the row’s identity, and the row key is derived from them server-side — do not send one. Supply a stabledocument_idper logical source and number the chunks from0, and re-writing a source replaces its chunks instead of doubling them.document_idmust be a UUID. Generate one per source and keep it.- Fill
equipment_model,manufacturer,revision,effective_from, andasset_idyourself. Nothing copies them onto hand-written chunks, and an unfilled column cannot be filtered on. - Only declared columns are accepted. An extra key rejects the batch rather than being absorbed — there is no overflow on this modality.
document_id:
Create
$API is the platform base URL (production: https://api.ilyama.golain.io/core/api/v1).
Create this before the document dataspace that will feed it. A document dataspace names its vector dataspace at create and cannot be repointed afterwards, and without that pairing every attempt to register a file with vectorize: true is rejected.
Related
- Document dataspaces — files, uploads, and the end-to-end walkthrough
- Dataspaces overview
- State — the keyed-row shape vector dataspaces share
- Schema and layouts
- API introduction