OriginChain docs
schema · full-text

Full-text search.

Full-text search answers "which documents mention these words, and which mention them most". It is an inverted index scored with BM25 — the same ranking family Lucene and Elasticsearch use — so rare words count for more than common ones and short documents beat long ones on the same term.

Reach for it when the user typed words and you want the words to matter: product search, log and ticket search, or the keyword half of a hybrid retrieval stack. When meaning matters more than wording, use vector search; when you know the predicate exactly, use SQL.

Every operation below is shown in cURL, Python, TypeScript and Go. Where an SDK does not wrap something the tab says so and shows the raw call.

1

Before you start.

Every example uses the shop.orders table from the quickstart, full-text indexing its notes column.

indexing is explicit — inserting a row does not index it

This is the single most common surprise on this page. Writing a row through the rows endpoint or SQL does not put anything into the full-text index. You POST each document to the FTS endpoint yourself, and you do it again whenever the text changes. Nothing in the schema TOML marks a column as searchable, because the index is keyed on a free-form (table, field) pair rather than on your schema at all.

The :table and :field path segments are opaque strings. They must match exactly between the write and the read — index into shop.orders/notes and search shop.orders/note and you get an empty result, not an error. Registering the row schema anyway is what lets you turn a doc_id back into a row.

schemas/orders.toml
# Nothing in the schema marks a field as full-text indexed - there is no
# such flag. An FTS index comes into existence the first time you POST a
# document to a (table, field) pair. Registering the row schema is still
# worth it: it is what lets you take a doc_id from a search hit and read
# the whole row back with SQL.

namespace   = "shop"
table       = "orders"
primary_key = ["id"]

[[columns]]
name = "id"
ty   = "str"
required = true

[[columns]]
name = "customer"
ty   = "str"

[[columns]]
name = "amount_cents"
ty   = "i64"

[[columns]]
name = "status"
ty   = "str"

[[columns]]
name = "notes"          # the field we will full-text index
ty   = "str"

[[columns]]
name = "placed_ms"
ty   = "u64"
2

Index a document.

One document, one call. Use the row's primary key as the doc_id so a hit maps straight back to a row.

POST /v1/tenants/:tenant/fts/:table/:field
curl -X POST "https://$OC_HOST/v1/tenants/$OC_TENANT/fts/shop.orders/notes" \
  -H "Authorization: Bearer $OC_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "doc_id": "01JTRX9KQ3YH8K2WMX0F5JZAB7",
    "text":   "rush delivery, signed by recipient"
  }'
# → 201 Created, empty body
  • Returns 201 Created with an empty body.
  • The write is synchronous and atomic — postings, document length, token set and corpus statistics all land in one batch. The document is searchable the moment the call returns; a crash mid-call leaves every record or none.
  • Re-indexing the same doc_id replaces the previous version cleanly. There is no separate "update" or "delete from index" call — write it again.
indexing a nested json document

If your text is spread across a nested object, the /json variant walks it for you. Dotted paths select what to index; string arrays under a listed path flatten one level. No SDK wraps this variant.

# Walk a nested document and index its string leaves.
curl -X POST "https://$OC_HOST/v1/tenants/$OC_TENANT/fts/shop.orders/notes/json" \
  -H "Authorization: Bearer $OC_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "doc_id": "01JTRX9KQ3YH8K2WMX0F5JZAB7",
    "json": {
      "note":     "rush delivery",
      "shipping": { "instructions": "signed by recipient" },
      "tags":     ["priority", "insured"]
    },
    "paths": ["note", "shipping.instructions", "tags"]
  }'
# Omit "paths" to index every string leaf in the document.
3

Search — a single term.

Search is a GET on the same path you indexed to, with the query in ?q=. mode=bm25 is what you want when you care about ranking.

GET /v1/tenants/:tenant/fts/:table/:field?q=…
curl "https://$OC_HOST/v1/tenants/$OC_TENANT/fts/shop.orders/notes?q=delivery&mode=bm25&k=10" \
  -H "Authorization: Bearer $OC_TOKEN"
response — mode=bm25
[
  { "doc_id": "01JTRX9KQ3YH8K2WMX0F5JZAB7", "score": 9.4213 },
  { "doc_id": "01JTRX9KQ3YH8K2WMX0F5JZAB9", "score": 6.1077 }
]

A bare array of { doc_id, score }, highest score first. k caps the list and defaults to 10.

the response shape changes with the mode

This endpoint returns four different shapes depending on the parameters. boolean and phrase give you a bare array of doc_id strings; bm25 gives an array of objects; adding highlight or facets wraps it all in an object with a hits key; and explain=true returns a scoring report instead. The TypeScript SDK models this as a union you have to narrow yourself.

4

What the query syntax really is.

This is worth being blunt about, because it is easy to assume otherwise: ?q= is not a query language. There is no parser. The string is tokenized into words — Unicode word segmentation, then lowercased — and every token becomes a term. All punctuation is discarded.

The practical consequence is that operators you might type do not work, and fail silently rather than erroring. rush AND delivery searches for three terms — rush, and, delivery. Quotes around a phrase are simply dropped.

You might try What actually happens
rush AND delivery Searches rush, and, delivery. Use mode=boolean, which already ANDs.
rush OR delivery Searches three terms. Use mode=bm25, which already ORs.
NOT cancelled / -cancelled Negation is not available here at all. Use must_not in the DSL.
"signed by recipient" Quotes are discarded. Use mode=phrase.
deliv* The * is discarded. Prefix and wildcard queries exist only in the DSL.
delivary~1 This one works. ~N is the single inline operator the query surface honours. See fuzzy.

How multiple terms combine — it depends on the mode.

This is the important distinction, and it is not configurable:

mode Multi-term meaning Returns
boolean AND — every term must be present Unranked doc_id array, lexicographic. Unboundedk is ignored.
bm25 OR — any term matches, more/rarer terms score higher Ranked {doc_id, score}, capped at k.
phrase Adjacent and in order Unranked doc_id array. k is ignored.
boolean — every term required
mode=boolean (the default)
# mode=boolean is the DEFAULT when ?mode= is omitted.
# Every term must be present. Returns a bare array of doc_ids, unranked.
curl "https://$OC_HOST/v1/tenants/$OC_TENANT/fts/shop.orders/notes?q=rush+delivery" \
  -H "Authorization: Bearer $OC_TOKEN"
response
["01JTRX9KQ3YH8K2WMX0F5JZAB7", "01JTRX9KQ3YH8K2WMX0F5JZAC1"]
an unknown mode falls back to boolean, silently

?mode=ranked, ?mode=BM25 or any typo is not an error — it takes the default branch and runs a boolean AND. If you get back an array of bare strings when you expected scores, check the spelling of mode first. The accepted values are lowercase boolean, bm25 and phrase.

the dashboard's search box is different

The console workbench accepts a one-line table:field terms shorthand in Search mode. That colon syntax is parsed by the console, which then calls the endpoint documented here — it is not something the engine understands. Don't put shop.orders:notes into ?q=. See Run queries from the dashboard.

5

Phrase queries.

mode=phrase requires the terms to appear adjacent and in the order given. The quotes you would type in another search engine are not syntax here — the mode is the switch.

mode=phrase
# Terms must appear adjacent, in this order. Quotes are NOT syntax -
# they would simply be discarded by the tokenizer. Use mode=phrase.
curl "https://$OC_HOST/v1/tenants/$OC_TENANT/fts/shop.orders/notes?q=signed+by+recipient&mode=phrase" \
  -H "Authorization: Bearer $OC_TOKEN"

Phrase results come back as an unranked array of doc_id strings — position matching selects the documents, but this mode does not score them. If you need ranking as well as adjacency, use the phrase clause inside the DSL, which selects on positions and then ranks with BM25.

6

Fuzzy matching.

Two ways in, both bm25-only: a whole-query budget via ?fuzzy=N, or a per-term ~N suffix. Putting a ~ anywhere in q switches the query onto the fuzzy path automatically.

fuzzy search
# Whole-query budget via ?fuzzy= (bm25 only, 0-3)
curl "https://$OC_HOST/v1/tenants/$OC_TENANT/fts/shop.orders/notes?q=delivary&mode=bm25&fuzzy=1" \
  -H "Authorization: Bearer $OC_TOKEN"

# Or per-term inline with ~N. A bare ~ means distance 2.
curl "https://$OC_HOST/v1/tenants/$OC_TENANT/fts/shop.orders/notes?q=delivary~1+recipient&mode=bm25" \
  -H "Authorization: Bearer $OC_TOKEN"
  • Edit distance is capped at 3. Higher is a 400: fuzzy edit_distance 5 exceeds MAX_EDIT_DISTANCE 3.
  • A bare term~ with no number means distance 2. term~0 is an exact match.
  • Each term expands to at most 50 dictionary candidates.
  • Distance 1 catches most real typos. Distance 2 and 3 expand aggressively and will pull in unrelated words — measure before shipping either.
  • Only the Python SDK exposes a fuzzy parameter. In TypeScript and Go, put ~N in the query string.
7

Ranking, scoring and explain.

mode=bm25 scores each document as the sum of its query terms' contributions. Each contribution combines three things:

  • Inverse document frequency. A term in few documents is worth more than one in many. This is why a rare word dominates a query.
  • Term frequency, with diminishing returns. The fifth occurrence adds much less than the second. k1 controls how fast that saturates.
  • Length normalisation. A hit in a short document counts for more than the same hit buried in a long one. b controls how strongly.

Defaults are k1 = 1.2 and b = 0.75 — the standard Lucene values. On the ?q= surface they are fixed; they can only be overridden through the DSL's params object. Scores are relative within one result set — never compare a score across two different queries.

The explain parameter.

Add explain=true to a bm25 query to get the full arithmetic instead of the hits. This is a query parameter on the search route — there is no separate explain endpoint.

curl "https://$OC_HOST/v1/tenants/$OC_TENANT/fts/shop.orders/notes?q=rush+delivery&mode=bm25&k=5&explain=true" \
  -H "Authorization: Bearer $OC_TOKEN"
response
{
  "query_terms": ["rush", "delivery"],
  "n_total":     1204,
  "avgdl":       11.6,
  "k1":          1.2,
  "b":           0.75,
  "hits": [
    {
      "doc_id": "01JTRX9KQ3YH8K2WMX0F5JZAB7",
      "score":  9.4213,
      "terms": [
        { "term": "rush",     "df": 42,  "idf": 3.361, "tf": 1.0,
          "doc_len": 5, "contribution": 5.9102 },
        { "term": "delivery", "df": 310, "idf": 1.362, "tf": 1.0,
          "doc_len": 5, "contribution": 3.5111 }
      ]
    }
  ]
}

n_total is the corpus size and avgdl the average document length — the two corpus statistics the formula needs. Per hit, terms is sorted by contribution descending, so the first entry is the term that actually won the document its place. Length normalisation is folded into contribution; doc_len is the raw token count.

  • Explain works only with mode=bm25, and overrides highlight and facets if you pass them together.
  • It always reports the default k1 and b — you cannot explain a custom-tuned score, and the DSL endpoint has no explain of its own.
  • Explain takes the exhaustive scoring path, so it is slower than the equivalent ranked query. It is a debugging tool, not a production one.
  • No SDK exposes explain. Call the endpoint directly.
8

Highlights and facets.

Both features need the document's text stored, which the plain index call does not do. Send it to the /doc variant instead, along with any facet values you want to aggregate on.

# Store the doc text plus per-facet values. Required before highlight=true
# or facets= will return anything.
curl -X POST "https://$OC_HOST/v1/tenants/$OC_TENANT/fts/shop.orders/notes/doc" \
  -H "Authorization: Bearer $OC_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "doc_id": "01JTRX9KQ3YH8K2WMX0F5JZAB7",
    "text":   "rush delivery, signed by recipient",
    "facets": { "status": ["paid"], "channel": ["web"] }
  }'

Then ask for them at query time:

curl "https://$OC_HOST/v1/tenants/$OC_TENANT/fts/shop.orders/notes?q=rush&mode=bm25&k=5&highlight=true&facets=status,channel" \
  -H "Authorization: Bearer $OC_TOKEN"
response — note the object wrapper
{
  "hits": [
    {
      "doc_id": "01JTRX9KQ3YH8K2WMX0F5JZAB7",
      "score":  9.4213,
      "highlights": { "notes": ["<em>rush</em> delivery, signed by recipient"] }
    }
  ],
  "facets": {
    "status":  [ { "value": "paid", "count": 12 }, { "value": "refunded", "count": 3 } ],
    "channel": [ { "value": "web",  "count": 11 }, { "value": "app",      "count": 4 } ]
  }
}
python
res = db.fts.search(
    "shop.orders", "notes", "rush",
    mode="bm25", k=5, highlight=True, facets=["status", "channel"],
)
for hit in res.hits:
    print(hit.doc_id, hit.score, hit.highlights)
for value, bucket in res.facets.items():
    print(value, [(b.value, b.count) for b in bucket])
  • Highlights come back as raw <em> markup around matched terms. Escape or sanitise before rendering.
  • Facets aggregate; they do not filter. facets=status tells you the distribution across the hit set — it does not narrow it. For real filtering use the DSL.
  • At most 1000 distinct values are tracked per facet field.
  • Only the Python SDK wraps highlight and facets; the /doc write is not wrapped by any SDK.
9

Filters and multi-field — the JSON DSL.

Everything the ?q= surface can't do — boolean composition, negation, filters, prefix and wildcard matching, multi-field search with weights, custom k1/b — lives in a JSON query DSL at POST /fts/:table/_search. Note the path takes a table only; fields are named inside the query.

curl -X POST "https://$OC_HOST/v1/tenants/$OC_TENANT/fts/shop.orders/_search" \
  -H "Authorization: Bearer $OC_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "doc_values_field": "notes",
    "query": {
      "bool": {
        "must":     [ { "match":  { "field": "notes", "query": "rush delivery" } } ],
        "should":   [ { "term":   { "field": "notes", "value": "signed", "boost": 2.0 } } ],
        "must_not": [ { "term":   { "field": "notes", "value": "cancelled" } } ],
        "filter":   [ { "numeric_range": { "field": "amount_cents", "gte": 10000 } } ]
      }
    },
    "top_k":  20,
    "params": { "k1": 1.2, "b": 0.75 }
  }'
response
{
  "total": 431,
  "hits": [
    { "doc_id": "01JTRX9KQ3YH8K2WMX0F5JZAB7", "score": 14.8802 },
    { "doc_id": "01JTRX9KQ3YH8K2WMX0F5JZAC1", "score": 11.2044 }
  ]
}

total is the match count before top_k truncation, so you can render "showing 20 of 431" without a second query. top_k defaults to 10.

fields are a named key, not a dynamic one

If you know Elasticsearch, this is the one difference that will trip you up. Where ES writes {"term": {"notes": "rush"}}, this DSL writes {"term": {"field": "notes", "value": "rush"}}. The field is always an explicit field key.

Clause types.

match_all, match_none, term, terms, match, phrase, prefix, wildcard, regex, range, numeric_range, fuzzy, exists, multi_match, bool, constant_score and function_score.

  • bool scores must plus any matching should. filter gates without contributing score; must_not excludes.
  • match defaults to OR across its terms; pass "operator": "and" to require them all.
  • Every clause takes a boost, and boosts multiply down the tree — a boost: 2.0 clause inside a boost: 3.0 bool contributes 6×.
  • minimum_should_match accepts an integer, a negative integer ("all but N"), or a percentage string like "75%".
  • regex supports literals, character classes, repetition, alternation and grouping — but not back-references, look-around or named groups.

Multi-field search with weights.

# Search several indexed fields at once, with per-field weights.
curl -X POST "https://$OC_HOST/v1/tenants/$OC_TENANT/fts/shop.orders/_search" \
  -H "Authorization: Bearer $OC_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "query": {
      "multi_match": {
        "fields": [
          { "field": "notes",    "boost": 2.0 },
          { "field": "customer", "boost": 1.0 }
        ],
        "query": "rush delivery",
        "type":  "best_fields",
        "tie_breaker": 0.3
      }
    },
    "top_k": 20
  }'

best_fields (the default) takes the best-scoring field plus tie_breaker × each other match; most_fields sums them all. tie_breaker defaults to 0.0 — pure winner-takes-all — and must be within [0, 1]. Every field you name must be independently indexed, or the request 404s.

numeric filters need doc_values_field

numeric_range, field_value_factor and decay functions read per-document values that only exist if you wrote them as facets through the /doc endpoint. You must also name the source with a top-level doc_values_field. Omit it and the request is refused with a 400 rather than quietly matching nothing — a deliberate choice, since a silent empty result is indistinguishable from "no matches".

no SQL integration

Full-text search is not reachable from POST /sql. There is no MATCH() function and no way to put a text predicate in a WHERE clause. To combine the two, search first and then query the returned doc_ids with SQL.

Two sibling endpoints share the DSL: POST /fts/:table/_aggs for aggregations and POST /fts/:table/_suggest for prefix suggestions. Because of these routes, _search, _aggs and _suggest are reserved and cannot be used as field names. No SDK wraps the DSL endpoints.

10

Analysis, synonyms and stopwords.

The analyzer is fixed: Unicode word segmentation, then lowercase. That is the whole pipeline, and there is no parameter to change it.

no stemming today

Searching deliver will not match delivery or delivered — they are three distinct terms. A stemmer covering eighteen languages exists inside the engine, but it is not selectable through the API, so today's behaviour is exact-token matching. Work around it with fuzzy matching, a synonym class, or a prefix clause in the DSL.

What you can configure, per (table, field) pair, is a synonym map and a stopword list. Both apply at index and query time, so installing either after you have indexed documents means re-indexing them to get consistent behaviour.

POST …/synonyms · POST …/stopwords
# Synonym classes - applied at BOTH index and query time.
# Re-installing replaces the whole map.
curl -X POST "https://$OC_HOST/v1/tenants/$OC_TENANT/fts/shop.orders/notes/synonyms" \
  -H "Authorization: Bearer $OC_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{ "synonyms": { "delivery": ["shipment", "dispatch"] } }'

# Stopwords - dropped at BOTH index and query time.
curl -X POST "https://$OC_HOST/v1/tenants/$OC_TENANT/fts/shop.orders/notes/stopwords" \
  -H "Authorization: Bearer $OC_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{ "stopwords": ["the", "a", "an", "and", "by", "of", "with"] }'

Each install replaces the whole map or list — there is no incremental add. A term may have at most 32 synonyms. Only the Python SDK wraps these two calls.

11

Limits and gotchas.

Limit Value
Hits per query10,000
Default k / top_k10
Max fuzzy edit distance3
Fuzzy expansions per term50
DSL query nesting depth32
DSL clauses per query1024
Synonyms per term32
Distinct values per facet field1000
Max query string lengthno limit
Pagination / offsetnot supported
there is no pagination

No offset, no from, no cursor. You get the top k and that is all. To show page two, raise k and slice client-side — and remember boolean and phrase mode ignore k entirely and return every match, which is why a broad boolean query on a large corpus can trip the result-size cap and return 413.

top_k bounds the response, not the work

A small k does not make a broad query cheap. The engine scores every matching document and ranks afterwards, so match_all or a very common single term over a large corpus is expensive however few results you ask for. A block-max optimisation skips provably-losing blocks for ranked queries, but it declines to engage in several cases — including whenever you request explain — and falls back to exhaustive scoring.

an unindexed field is an empty result on ?q=, a 404 on the DSL

The two surfaces disagree here. _search refuses an unindexed field with a 404 explaining that an unindexed field is refused "rather than answered with an empty result you could not tell from 'nothing matched'". The ?q= route has no such check and returns an empty array, so a typo in the path looks exactly like a genuine miss.

honest scale guidance

Measured on the published benchmark: 50,000 documents → 587 MB on disk, ranked p99 1.8 ms. 200,000 documents → 2.3 GB on disk, p99 13 ms. Cost is roughly 12 KB per document, and resident memory grows linearly with the corpus — projected around 25 GB at a million documents. Latency is production-grade at these sizes; memory, not speed, is what will bound you. Size the instance's RAM against your document count, and treat a million documents on a single instance as the point where you should be talking to us about sharding.

what needs the add-on

Indexing documents and mode=boolean searches work on any instance. Ranked bm25 and phrase queries, and all three DSL endpoints, require the Full-Text Pro add-on — without it the call returns 402 naming the add-on. Enable it from Billing → Add-ons.

caps refuse rather than truncate

When a query exceeds an expansion or clause limit the engine returns 400 with a message naming the limit and suggesting a fix — it never silently returns a partial result set. A truncated expansion would drop matching documents without telling you, so the refusal is deliberate.

Related.