Möbius
Guides

Held datasets

Bring data into Möbius, see what happens to each row, take certified data out, and retire what you no longer need.

A held dataset is one where Möbius stores the records itself, not just the governance layer around them. This guide follows a held dataset from entry to exit. For how held differs from in place, see custody modes.

Choose a storage kind

When you create a dataset (Tables → Create Table, Step 1: Möbius-stored), Step 2 asks for a storage type. Pick the one that matches the shape of your data:

Storage typeHoldsChoose it when
StructuredRow-shaped records in a dedicated table with typed columns and indexesYour data is tabular and stable: orders, customers, ledgers
DocumentEach record as a nested JSON objectPayloads are nested or change shape: webhook bodies, API responses
VectorEmbeddings plus metadata, for similarity searchYou are building a retrieval store for an AI assistant

Structured is stricter: column types are inferred more conservatively (see below). Vector datasets have their own create flow, covered in Vector datasets.

Bring data in

Upload a file

  1. On a held dataset, select Import. (When creating a new dataset you can also start from the Upload CSV tab, or choose Define schema only and import later.)
  2. Set Source to File upload and choose a CSV file. Set the delimiter and whether the first row holds headers.
  3. Pick a Mode: Append inserts every row; Upsert matches on a Key column you name.
  4. Turn on Create Missing Columns if the file may carry new fields. New fields default to text.
  5. Select Dry Run to see per-row errors without writing anything, then Run Import.

The file is processed as a background job; a banner on the dataset tracks its progress.

How columns are inferred. When you create a dataset from a file, Möbius samples up to the first 1,000 rows and proposes a type for each column — boolean, number, date, email, URL, or text, in that order of preference. On a Structured dataset a type must match at least 80% of sampled values, otherwise the column falls back to text with a warning. A column with no empty values in the sample is proposed as required; one fully-unique column (preferring id, uuid or email) is proposed as unique. Headers that collide with a reserved system column are renamed and you are told before you confirm. Short or long rows are flagged too.

You can also send rows directly as a JSON array (the Rows (JSON array) box on the Import page, or the import endpoint on the REST API). Large batches return a job id and are checked in the background.

Ingest from your own S3 or S3-compatible bucket

Connecting a bucket needs a workspace admin, and is behind the AWS S3 tenant capability control.

  1. Go to Integrations → Data Integrations and open AWS S3 Pipelines (an IAM role you grant Möbius, identified by the external ID shown) or S3-Compatible Storage (MinIO, Cloudflare R2, Wasabi, Backblaze B2 and similar — an endpoint URL plus an optional access key).
  2. To import one object by hand, choose Import → Source: S3 object, pick the connection, and give the object key and format (CSV, JSONL or Parquet).

To have objects flow in automatically, add an ingest endpoint on the AWS S3 Pipelines page:

  1. Under a saved bucket, open Ingest endpoints and add one. Set the Target table, the Mode (append, or upsert with an Upsert key), the Format, an optional Prefix override, and for CSV the delimiter and header setting.
  2. Select Rotate token to issue the endpoint's credential. The token is shown once and never stored in readable form — copy it straight into your secret store.
  3. Point your bucket's event notification (or any caller) at the Endpoint URL, sending the token in the x-mobius-ingest-token header. The body is a standard S3 event notification, or { "bucketName": "...", "objectKey": "..." }.

Objects outside the endpoint's prefix are skipped, a repeat delivery of the same object is a no-op, and each eligible object becomes its own import job. To rotate the credential, select Rotate token again: the old token stops working immediately, and the rotation is recorded in the audit trail (the token itself never is).

Ingest from a public URL

On Import, set Source to Public URL (unattested), paste an https:// URL, and pick the format. Möbius fetches the file anonymously — it never sends credentials — and refuses URLs that carry embedded credentials, signed query strings, or resolve to a private address. Your workspace's network policy can restrict which hosts are allowed. Downloads are capped (500 MB by default). URL ingest is unavailable on Möbius Enclave, which has no outbound internet path.

After the fetch, Möbius records the source's provenance in the audit trail: the URL, a SHA-256 of the content, its size, the fetch time, the row count, and the fact that the source is unattested.

Register and replace a source document

On a vector dataset, a source document binds the chunks you store to the document they came from. Each document has an external key you choose, and a version that increases each time you replace it.

  • Register — POST /api/tables/{id}/source-documents with externalKey and the embeddingModel your chunks were made with. It returns the document id and its new version.
  • Replace — PUT the same path with externalKey, embeddingModel, and rows: the full, re-chunked and re-embedded new version. Möbius retires every chunk of the previous version and writes the new ones in one step. Replacing the whole document is the only way to update it; individual chunks are not patched.

Both calls need write access to the dataset. The model you declare must match the model the dataset is pinned to, or the call is refused.

What happens to each row on write

Every row written to a Structured or Document dataset passes the row-level admission gate: the dataset's automated quality checks are evaluated against that row before it lands. The outcome depends on the enforcement posture dial in effect for the dataset:

  • Accepted — the row passes and is written.
  • Held — with the dial on enforcing, a failing row is not written. It is kept on the dataset's Rejected Rows page with the check it failed, its severity (hard or soft fail), the reason, the surface it came from, and when.
  • Written with a flag — with the dial on advisory, a failing row is written but flagged, and a copy is kept on Rejected Rows for the record.
  • Not checked — with the dial off, or before the dataset has a contract, rows are written without checks, and the skip itself is recorded.

The import response counts and explains each batch: accepted lists the rows written, and rejected lists each refused row's index, the check, its severity, and the reason. Rows land in the sandbox until the dataset is certified, and in the certified zone after that.

Correct and replay refused rows

A held row is never edited in place. Fix the cause instead, then replay:

  1. If the check itself was wrong, correct it in the dataset's contract. Then open Rejected Rows and select Replay all. Each held row is re-checked, unchanged, against the current checks: rows that now pass are written and leave the list; rows that still fail stay, with their replay count increased. Large sets replay in the background.
  2. If the data was wrong, fix it at the source and import those rows again, then Delete the stale rejection.

Replay needs the steward role (or higher) and is refused while the dial is off or the dataset has no contract. Rows written with an advisory flag are already stored, so they can only be deleted from the list, not replayed.

Create and fill a vector dataset

Vector datasets are behind the vector-tables tenant capability control.

  1. Go to Vector in the sidebar and select Create Vector Table. Set the embedding dimension if you know it. The dataset is pinned to one embedding model when it is created; every later write must use that model.
  2. Open the dataset's Import page. Either:
    • Manual JSON — paste a JSON array of rows, each with an embedding array of numbers, or
    • Document Pipeline — upload a PDF, DOCX, Markdown or plain-text file, or give a web page URL. Möbius chunks it (chunk size 512 and overlap 64 by default, both adjustable) and embeds it with the pinned model.

Vector rows always append. Each write is checked against the pinned model and the dataset's embedding dimension. The row-level quality checks do not run on embeddings today; each write records that the check was skipped.

Masking at read. To search, open the dataset and select Chat: the assistant retrieves the closest chunks for your question. Chunk content in a column classified as personal data is masked by its field policy. Metadata is reduced to the standard keys (such as document_type, source_url, embedding_model, chunk_index, total_chunks); any other keys you stored are withheld at read, and the response names which were withheld.

Take data out

Export a CSV

On a dataset's data view, select Export CSV. The export:

  • contains the full dataset as currently filtered and sorted, not just the visible page;
  • includes only rows and columns you could already read, and only for datasets in environments you have access to;
  • is refused, not truncated, above 100,000 rows;
  • writes one entry to the audit trail.

Export to your S3 bucket

For larger or recurring exports, add an export to a saved bucket on the AWS S3 Pipelines page: choose a format (CSV, JSONL or Parquet), a source dataset or decision view, an optional destination key, and a schedule type (Manual, Event-based or Scheduled). For Scheduled, enter a cron expression in UTC. Select Save. Only certified datasets that are approved for exports can be sent, and the dataset's environment must match the bucket's. Managing exports needs the Admin role.

  • Export now runs the saved export once, straight away, with the same checks as a scheduled run. The job appears in the job monitor. A Manual export runs only when you press it.
  • Disable stops an export, including its schedule, until you Enable it again.
  • Every column classed personal or sensitive is masked in the exported file. See How Möbius masks personal data.

If your deployment hasn't switched exports on yet, saving a scheduled export shows "Scheduled exports are not running on this deployment yet; the schedule is saved and starts when they are enabled." under the schedule. The schedule is kept, and it starts once exports are switched on. Until then, Export now is refused with a message saying so.

When it stops

Every scheduled run checks the source dataset first. If the dataset is not certified at that moment, nothing is sent: the export pauses, the run is recorded in the audit trail, and the dataset's steward and the person who added the bucket are notified once ("scheduled export paused"). You don't need to restart it. Once the dataset is certified again, the next scheduled run sends as normal. An export also pauses the same way, and resumes on its own, while your workspace's network boundary blocks the bucket or the AWS S3 integration is turned off for your workspace. Pausing on an uncertified dataset can't be turned off.

Write back to a warehouse

Open the dataset's Settings → Governance and find Writeback. Add a writeback per target warehouse connector and choose an export strategy:

StrategyWhat it does
FullTruncates the target and inserts all certified rows
DeltaInserts or updates rows changed since the last export
Near-realtimeExports as certified changes arrive; an optional batching window (up to 900 seconds) groups them
ScheduledExports on a cadence you set, no more often than every 15 minutes

Select Run now to export immediately, or pause and resume a writeback from its row. Only certified rows are sent, and the connector's environment must match the dataset's. Writeback is not available for vector datasets.

Read through the Data API

Applications read certified held data through the Data API on a scoped token — see the REST API reference.

Delete and recover

Nothing in a held dataset is erased on delete; it is retired.

  • Datasets — delete from the dataset header or Settings → Danger Zone. If a token is still bound to the dataset, revoke it first; the delete is blocked until you do. Integrations reading the dataset get 404 from then on.
  • Columns — a removed column is deactivated, not dropped, so its data stays. Removal is refused while a decision view still uses the column.
  • Rows — a deleted row is marked deleted and kept. The audit entry names the columns, never the values.

A deleted dataset can be recovered during the workspace retention window. A workspace admin sets the window (1–365 days) under Table Retention Policy on the workspace governance settings page. To recover, call POST /api/tables/{id}/recover as a user who could delete the dataset. After the window passes, the dataset is purged the next time it is accessed.

On this page