Skip to main content
A document dataspace is a collection of files: manuals, SOPs, spec sheets, drawings, signed purchase orders, calibration certificates. It holds one row per file, recording where the bytes are and what the file describes. The bytes themselves live in object storage, not in the row. The API modality is document. Storing a file is the whole job by default. You can additionally opt an individual file into chunking and embedding with vectorize: true, which makes its text semantically searchable through a paired vector dataspace. That choice is per file, not per dataspace, so one collection can hold manuals worth searching and delivery notes that are not. → All modalities

​
When to use it

Use document when
  • You have files that describe your equipment and you want them findable by what they describe: asset, model, manufacturer, revision.
  • You want somewhere to keep the attachments a record accumulates — a signed PO against an order, a calibration certificate against an instrument — and to hand someone a link to the bytes later.
  • You want some of those files searchable by meaning. Pair the dataspace with a vector dataspace and register the files worth searching with vectorize: true.
  • The same file is attached to many assets. Documents are content-addressed, so uploading identical bytes twice stores one copy.
Do not use document for the extracted text itself. That is a vector dataspace. Do not use document as general blob storage for firmware, images, or backups. It is a catalogue of files that describe things, with an optional parse pipeline attached — not a bucket.

​
The dataspace is the collection; a document is one file in it

This is the distinction most likely to trip you up, because both are named with the word “document” and both have an id. You do not create a dataspace per file. A project typically has a handful of document dataspaces and thousands of documents inside them.

​
The document’s key has three spellings

One value, three places, because it is a request field, a URL segment, and a stored column: Pick something stable and human-meaningful: hpu-450-service-manual, sop/lockout-tagout, po/2026-00814. Percent-encode it in paths if it contains slashes or spaces. Re-registering the same key is a new revision of that document, not a second document. Do not key on the file’s hash: that would make every revision a new document and leave the superseded one live and searchable beside the current one, which is the failure mode revisions exist to prevent.

​
Vectorizing is opt-in

vectorize defaults to false. Register a file without it and the platform stores the bytes and writes the row, and does nothing else: no conversion, no chunking, no embedding. The document’s parse_state is skipped. That default is about cost, and about what a document dataspace actually holds. Parsing a file costs a format conversion plus one embedding call per chunk — hundreds of provider calls for a 500-page manual. Most files in a document dataspace are attachments nobody will ever ask a question of: a signed PO, a calibration certificate, a photo of a nameplate. Indexing those buys nothing, and a project that uses documents purely as an attachment store never needs a vector dataspace at all. Set vectorize: true on the files worth searching, and only those:
skipped is a distinct state from pending, deliberately. skipped means nobody asked for this file to be indexed. pending means somebody did and it has not happened yet. Collapsing them would make every stored attachment look exactly like a stalled pipeline, and you would have no way to tell which one you were looking at.
vectorize: true needs a chunk corpus, and is rejected without one. If the dataspace has no chunk_dataspace_id, registration fails with a message saying so rather than accepting a file that could never become searchable. Create the vector dataspace first and pair it at create — chunk_dataspace_id is fixed for the life of the document dataspace, so there is no repointing it afterwards.

​
Backing store

Document dataspaces are Postgres, using the same keyed-row shape as state: one live row per document, last-write-wins on version, deletes as tombstones.

​
The schema

The platform declares the columns it reads itself. You do not pass fields for those — anything you send is replaced — and you cannot remove them. Every column is nullable, because the row is written in stages — it exists before the bytes land.

​
Your own fields

The built-in columns cover what the platform itself reads. What you need to record alongside a file is yours to decide, and you attach it with a metadata object on the register call:
Those keys come back under metadata on every read. There is no special metadata column and no separate metadata store: your keys go into the row like any other field, and the dataspace’s ordinary overflow storage holds the ones that are not built-in columns. That is exactly why document defaults to hybrid — on a typed layout an undeclared column orphans the whole row, so your own field would not merely be untyped, it would take the document with it. metadata is an envelope rather than loose top-level keys because the register call also carries arguments that are not row columns (sha256, vectorize, multipart). Nesting your fields means a field of yours named multipart can never be mistaken for a platform flag. Keys must be made of letters, digits, underscores, and dots.
A metadata key that collides with a built-in field is rejected. metadata.title and the real title would be two values for one thing with no rule about which wins, so registration fails with a message naming the key. Set the built-in directly instead — title, revision, asset_id, equipment_model, manufacturer, language, and the rest of the table above are all top-level fields on the register call. Row meta columns (source_id, source_type, event_ts, ingest_ts, overflow) and doc_key are refused for the same reason.
On a typed document dataspace there is no overflow, so metadata comes back empty and undeclared keys are refused. Choose typed only when you have declared every field you intend to send.

​
The two states

Upload and parse are tracked separately because they are two different things that fail for different reasons. A document sitting at awaiting_upload means the bytes never arrived — usually an upload that was abandoned, or a presigned URL that expired before the client got round to using it.
A parse that does not complete leaves the document at pending, not failed. So pending is the state to watch: a document that has been uploaded for a long time and is still pending is the symptom of a parse that did not run or did not finish, and the usual cause is a missing or wrong content_type at registration. The row is safe to re-drive once that is fixed. List with ?parse_state=pending to find them.

​
Registering a document

You send the document’s identity, its SHA-256, and whatever you know about it. You get back an upload target. The row is written before your bytes exist. A client that vanishes mid-upload leaves a visible row at awaiting_upload rather than nothing at all.
Send content_type. Files are stored under their hash, so the stored object has no filename and no extension. The content type you declare is how the platform knows what kind of file it is parsing, and it is what gives a download its extension. Register a PDF without one and the upload succeeds, the parse runs, and it produces no text — a failure that surfaces late and points nowhere.

​
Content addressing and deduplication

The object key is the file’s SHA-256, so identical bytes are stored once. The same service manual attached to forty assets is one object and forty rows. When the bytes are already present, the response comes back with already_uploaded: true and no presigned URL. Skip the upload entirely.

​
Send put_headers verbatim

The response’s put_headers are part of the upload’s signature. Send every one of them, exactly as returned, with nothing added, dropped, or changed. Alter or omit any and the object store rejects the upload with a signature mismatch.
That is deliberate rather than fussy. The digest and the content type are signed into the URL, so the store itself refuses bytes that do not hash to what you declared and refuses to store the file under a content type you did not declare. It is the store enforcing your claim, not the platform checking afterwards and hoping. Presigned upload URLs are short-lived — the response tells you how long, in expires_in_seconds. If yours expires, register the document again to mint a new one; re-registering the same key is not a duplicate.

​
Reading a document dataspace

Three endpoints cover the collection: list it, read one document, and get a link to the bytes.
All three need can_read_data on the dataspace. Without it — or if the dataspace is not document — you get a 404, not a 403, so a caller who may not read the dataspace cannot learn that it exists.

​
List the documents

Newest change first: results are ordered by updated_at descending, then by key.
total counts everything matching your filters, not just the page, so you can drive a pager from one call.

​
Read one document

The response is {"item": …} with the same fields as a list row. This is what you poll while a document moves through its two states.

​
Download the bytes

You get a short-lived presigned link, not a proxied stream. The object store serves the bytes directly, so a 400 MB manual never travels through the API, and the link opens in a browser tab with no Authorization header attached. Treat it the way you would treat the bytes: it is a bearer capability to read one object, which is why it expires. A document whose bytes have not landed has nothing to link to, so this returns 404 while upload_state is still awaiting_upload.
Use filename. Objects are stored under their content hash, with no extension. Hand the browser the url alone and the download lands as a 64-character hex blob that nothing will open. filename is built from your document key plus the extension implied by content_type — another reason to send one.
Document dataspaces also expose the generic row surface (…/dataspaces/{id}/rows) that state uses, where {row_key} is the document key. Prefer the documents endpoints: they return the derived document_id and your metadata, and they leave out row meta columns that mean nothing to a caller holding a document key.

​
One asset, many documents

This is the grouping the modality exists for, and it is worth being concrete. A u-blox positioning module has a datasheet, a reference design, a set of application notes, and a command reference. That is four separate documents. It is one asset. It is one dataspace. Register each one with the same asset_id:
Then pull the whole set back with one call:
Put what the documents have in common in the built-in columns, and what distinguishes them in metadata. asset_id, equipment_model, manufacturer, and revision are built in precisely because they are the filters that also have to work on search: they are copied onto every chunk, and a metadata key is not.

​
End to end: from an API key to a searchable PDF

This is the whole flow for a file you do want to search. It assumes $API, $TOKEN, $ORG_ID, and $PROJECT_ID are set — see the API introduction.
1

Connect an embedding provider

Register an integration account for your organization. Its provider must declare the embed capability.
Keep account_id. The key is sealed in the platform’s secret store and is never readable again.
2

Create the vector dataspace

The corpus your chunks land in. The model and dimension are permanent — see vector dataspaces.
3

Create the document dataspace, paired to it

chunk_dataspace_id is the vector dataspace’s id from the previous step. It must be a vector dataspace in the same project, and it is fixed for the life of this dataspace.
Skip chunk_dataspace_id and the dataspace still works as a file store — but vectorize: true is then rejected on every registration, and nothing in it can ever become searchable.
4

Register the PDF, asking for it to be indexed

Hash the file first. The API takes hex or base64; the upload header wants base64, and the response gives you both in the right places.vectorize: true is the part that matters here. Leave it out and you get a stored PDF and nothing to search.
If already_uploaded is true, there is no presigned_put_url and no upload to do — jump to the next step.
5

Upload the bytes

Send every header from put_headers, exactly as returned.
A 403 with a signature message here almost always means a header was changed, dropped, or added by your HTTP client. Many clients set their own Content-Type on a PUT unless you stop them.You do not tell the platform the upload finished. The object store does.
6

Wait for it to become searchable

Poll the document. It moves awaiting_upload → uploaded, then pending → parsed.
parse_state: parsed means the chunks have been written. They are searchable by keyword immediately, and semantically once the background job has embedded them — usually a minute or two for one document. See how chunks get their embeddings.If it is still pending long after upload_state reached uploaded, the parse has not completed; check content_type first. If it is skipped, you did not pass vectorize: true.
7

Search it

Search the vector dataspace, not the document one.
A result carries the document_id it came from, so you can close the loop: look the document up, then call download to hand the reader the PDF itself.Check mode on the response. If it comes back lexical when you asked for hybrid, a degraded field says why — see when search quietly falls back.
The common case is shorter, needs no vector dataspace, and costs nothing to index. Register the file, upload it, hand out a link.
1

Register it without vectorize

Upload the bytes to presigned_put_url exactly as above. When the object store reports them, upload_state becomes uploaded and parse_state stays skipped — final, and correct. Nothing was parsed, chunked, or embedded, and no provider call was made.
2

Hand someone the file

Put url behind a button in your own UI and save it as filename. The key contains a slash, so it is percent-encoded as po%2F2026-00814 in the path — and dropped from the suggested filename, which has no directories to make.
To find every attachment on an asset later, list with ?asset_id=. To audit what you have never indexed, list with ?parse_state=skipped.

​
Loading a revision

To publish a new edition of a document you already have, register the same key with the new file’s hash, a new revision, and a new effective_from. The document row updates in place, the new bytes are uploaded, and if the document is registered with vectorize: true its chunks are overwritten rather than added to. Then pass revision on your searches so a superseded procedure cannot surface as a current one. effective_at alone will not do it — it hides revisions that are not yet in force, not ones that have been replaced. See filters.
Chunks are overwritten by position, and surplus ones are not swept. If the new edition produces fewer chunks than the edition it replaces, the extra chunks from the previous parse stay in the corpus, still carrying the old revision. A search filtered to the new revision will not return them. A search that is not filtered by revision can.
If that matters — a procedure was removed rather than reworded, say — clear the document’s chunks before re-registering it. Tombstone them by document_id on the vector dataspace, then register the new edition:

​
Create

Drop chunk_dataspace_id for a pure file store. Add "layout": "typed" if you want undeclared metadata keys refused instead of absorbed. $API is the platform base URL (production: https://api.ilyama.golain.io/core/api/v1).