# Searching entities

Query sets of entities over REST — direct vs async modes, predicates, pagination, and historical reads.

<FromTheBinary topic="search" />
<FromTheBinary topic="predicates" />

Cyoda exposes search over REST for any query that returns a set of
entities. Use it when you need more than one entity back, when the filter
goes beyond a single id lookup, or when you want to scope by workflow
state. For single-entity reads, stay on the CRUD endpoints in
[working with entities](/build/working-with-entities/); for cross-entity
analytics, use [SQL](/build/analytics-with-sql/); for event-driven
compute, use [gRPC compute nodes](/build/client-compute-nodes/). For
counts and aggregates without the entity bodies, jump to
[grouped statistics](#grouped-statistics).

## Two query modes

Cyoda splits search into **Immediate** and **Background** modes. Pick by
expected result size and urgency; the filter grammar is identical.

- **Immediate** (API term: `direct`) — synchronous. The request streams
  matching entities back as NDJSON. It is **bounded-or-fail**: `limit`
  caps the *matched set*, and a query matching more than that fails
  outright rather than returning part of the answer. Use it when you
  know the filter produces a bounded, small set — a UI list, a lookup,
  a small report.
- **Background** (API term: `async`) — queued. The request returns a
  job handle; poll it, then page the results. Result size is
  **unbounded**. On the Cassandra-backed tier (Cyoda Cloud, or a
  licensed Enterprise install), `async` runs distributed across
  the cluster: for a fixed query shape, throughput scales roughly
  linearly with the number of nodes.

A direct search whose matched set exceeds the effective `limit` returns
`400 SEARCH_RESULT_LIMIT`. It does **not** return the first `limit`
entities. This is deliberate: a truncated page was indistinguishable
from a complete one, so callers silently processed partial data.

- `limit` defaults to **1000** when omitted, and caps at **10000**.
- `limit=0` — which once meant "unbounded" — is rejected with
  `400 BAD_REQUEST`, as is any value below 1. gRPC rejects `limit < 1`
  identically.
- A matched set exactly equal to `limit` succeeds.

When you hit it, narrow the condition, raise `limit` within the maximum,
or move the query to `async`.

The decision tree is short:

- Small bounded result, UI-facing → `direct`.
- Might be large, can tolerate a second or two of queuing; exports,
  reports, batch jobs → `async`.
- Hitting `SEARCH_RESULT_LIMIT` or the request timeout → `async`.
- Ordered top-N over a large model → `async` (see
  [sorting](#sorting-results)).

## A direct search

Filter by a combination of entity fields and workflow state:

```bash
curl -X POST 'http://localhost:8080/api/search/direct/orders/1?limit=200' \
  -H 'Content-Type: application/json' \
  -H "Authorization: Bearer $TOKEN" \
  -d '{
    "type": "group",
    "operator": "AND",
    "conditions": [
      { "type": "lifecycle", "field": "state",
        "operatorType": "EQUALS", "value": "submitted" },
      { "type": "simple", "jsonPath": "$.customerId",
        "operatorType": "EQUALS", "value": "CUST-7" }
    ]
  }'
```

The path is `/api/search/direct/{entityName}/{modelVersion}`, and the request
body is the condition document itself — there is no wrapper object. `limit`,
`pointInTime` and `trackingRead` are **query parameters**, not body fields.

The response is `application/x-ndjson`: one complete entity envelope per line,
each carrying `data` plus a `meta` block with the id, model key, state, and
timestamps.

```
{"type":"ENTITY","data":{"customerId":"CUST-7"},"meta":{"id":"74807f00-…","state":"submitted","creationDate":"2026-03-01T10:00:00.000000000Z"}}
{"type":"ENTITY","data":{"customerId":"CUST-7"},"meta":{"id":"89abc100-…","state":"submitted","creationDate":"2026-03-02T11:30:00.000000000Z"}}
```

Because it is a stream, parse it line by line rather than buffering the whole
body as a single JSON document.

## An async search

Submit the search to `/api/search/async/{entityName}/{modelVersion}` and
capture the handle:

```bash
curl -X POST http://localhost:8080/api/search/async/orders/1 \
  -H 'Content-Type: application/json' \
  -H "Authorization: Bearer $TOKEN" \
  -d '{
    "type": "lifecycle",
    "field": "state",
    "operatorType": "EQUALS",
    "value": "submitted"
  }'
```

The response is a bare UUID string — the job id. Poll its status until the job
reports `SUCCESSFUL`, then page the results:

```
GET /api/search/async/{jobId}/status
GET /api/search/async/{jobId}?pageNumber=0&pageSize=1000
GET /api/search/async/{jobId}?pageNumber=1&pageSize=1000
```

The status response carries `searchJobStatus` (`RUNNING`, `SUCCESSFUL`,
`FAILED`, `CANCELLED`, or `NOT_FOUND`), `entitiesCount` for the total match
count, and `expirationDate`. Fetching results before the job is `SUCCESSFUL`
returns `400 BAD_REQUEST`.

Results come back as a paged JSON object — `content` plus a `page` block with
`number`, `size`, `totalElements` and `totalPages` — not as a stream. Job
results expire 24 hours after creation; a single `jobId` can be paged
repeatedly until then.

Async search never returns `SEARCH_RESULT_LIMIT`: it snapshots the whole
matched set and pages over it. An out-of-range `pageSize` or `pageNumber` is a
plain `400 BAD_REQUEST`.

### Cancelling a job

If a job is no longer needed — the user navigated away, a replacement
query was submitted, the deployment is shutting down — cancel it rather
than letting it run to completion:

```bash
curl -X PUT http://localhost:8080/api/search/async/{jobId}/cancel \
  -H "Authorization: Bearer $TOKEN"
```

Cancellation is cooperative: in-flight work is stopped at the next safe
point and any partial results for that `jobId` are discarded.

## The condition DSL

The request body is a **`Condition`** document. Every condition carries a
`type` discriminator. The same DSL is used by workflow and transition criteria,
so what you learn here transfers directly.

**`simple`** — match a JSONPath into the entity payload:

```json
{ "type": "simple", "jsonPath": "$.amount",
  "operatorType": "GREATER_OR_EQUAL", "value": 1000 }
```

**`lifecycle`** — match entity metadata rather than payload. `field` is one of
`state`, `creationDate`, `lastUpdateTime`, `transitionForLatestSave` (alias
`previousTransition`), `transactionId`, `id`:

```json
{ "type": "lifecycle", "field": "state",
  "operatorType": "EQUALS", "value": "submitted" }
```

**`group`** — combine conditions. `operator` is `AND` or `OR`; these are the
only two, and `NOT` is not supported. Groups nest to a maximum depth of 50:

```json
{
  "type": "group",
  "operator": "AND",
  "conditions": [
    { "type": "lifecycle", "field": "state", "operatorType": "EQUALS", "value": "submitted" },
    { "type": "simple", "jsonPath": "$.amount", "operatorType": "GREATER_OR_EQUAL", "value": 1000 }
  ]
}
```

**`array`** — match positional values in a JSON array, where `null` matches any
value at that index:

```json
{ "type": "array", "jsonPath": "$.laureates",
  "values": ["John", null, "Hopfield"] }
```

**`function`** — delegate the predicate to a compute node. The condition is
dispatched as an `EntityCriteriaCalculationRequest`, exactly like a workflow
criterion:

```json
{
  "type": "function",
  "function": {
    "name": "my-criteria-fn",
    "config": {
      "calculationNodesTags": "approval-service",
      "attachEntity": true,
      "responseTimeoutMs": 30000
    }
  }
}
```

`function.name` and `config.calculationNodesTags` are required for routing;
`attachEntity` defaults to `true` and `responseTimeoutMs` to `30000`.

`function` is the one condition type that never translates to a storage-plugin
filter. It always runs as a **post-filter with in-memory entity loading**, so
every candidate entity is materialised and dispatched to your compute node.
Combine it with a selective `simple` or `lifecycle` condition in an `AND` group
so the pushdown narrows the population first — a bare `function` condition over
a large model means one callout per entity.

`operatorType` also accepts the spellings `operator` and `operation`. The
available operators are:

| Group | Operators |
|---|---|
| Comparison | `EQUALS`, `NOT_EQUAL`, `GREATER_THAN`, `GREATER_OR_EQUAL`, `LESS_THAN`, `LESS_OR_EQUAL` |
| Range | `BETWEEN`, `BETWEEN_INCLUSIVE` — `value` must be a two-element `[low, high]` array |
| Presence | `IS_NULL`, `NOT_NULL` |
| String | `CONTAINS`, `STARTS_WITH`, `ENDS_WITH` and their `NOT_*` inverses |
| Case-insensitive | `IEQUALS`, `INOT_EQUAL`, `ICONTAINS`, `INOT_CONTAINS`, `ISTARTS_WITH`, `INOT_STARTS_WITH`, `IENDS_WITH`, `INOT_ENDS_WITH` |
| Pattern | `LIKE` (anchored glob), `MATCHES_PATTERN` (anchored RE2 regex) |

An operator outside this list is rejected with `400 BAD_REQUEST`, and the
error detail includes the canonical list. `IS_CHANGED` and `IS_UNCHANGED` are
change-generation operators, not search predicates — cyoda-go does not
implement them.

An empty body (`{}`), or any body without a `type`, is rejected with
`400 BAD_REQUEST`. To match every entity in the model, send a vacuous `AND`
group:

```json
{ "type": "group", "operator": "AND", "conditions": [] }
```

An `AND` group with no conditions is `true`; an `OR` group with no conditions
is `false`.

## Predicate semantics

Search and workflow criteria run on **one shared evaluation kernel**, so a
predicate means the same thing wherever you write it. The kernel implements
Cyoda Cloud's model; `cyoda help predicates` is the exhaustive reference.

**Comparison is type-directed and same-type.** An operand is parse-tested
against the field's declared type(s), so the JSON number `30` and the string
`"30"` behave identically. There is no cross-type coincidental matching: an
operand that parses only as a number never matches a string-stored value.
Numbers compare at arbitrary precision, so values beyond 2^53 are exact.

**Absent and null fields never match a binary operator — including
negatives.** `NOT_EQUAL`, `NOT_CONTAINS`, `INOT_*` and the rest are
null-guarded to non-match rather than evaluated as `!positive`. A missing field
does not satisfy `NOT_EQUAL` just because it fails `EQUALS`. Use `IS_NULL` and
`NOT_NULL` to test presence — they are the only operators that do.

**`LIKE` is a real anchored glob.** `%` matches any run of characters, `_`
matches exactly one, and `\` escapes a literal `%`, `_` or `\`. The match is
whole-string and case-sensitive. `MATCHES_PATTERN` is a whole-string-anchored
RE2 regex; RE2 diverges from Java's dialect on constructs such as
backreferences and some lookaround.

**Temporal fields compare chronologically**, not lexically, across
`LOCAL_DATE`, `LOCAL_DATE_TIME`, `LOCAL_TIME`, `ZONED_DATE_TIME`, `YEAR` and
`YEAR_MONTH`. The `creationDate` and `lastUpdateTime` meta fields accept a
*coarser* operand that upscales — `"2024"` or `"2024-09"` are valid against a
full timestamp.

**Validation is parse-based**, evaluated against the target model:

- `400 CONDITION_TYPE_MISMATCH` — the operand parses into none of the field's
  declared types. Comparison and range operators only; string operators and
  presence tests carry no operand-type constraint.
- `400 INVALID_FIELD_PATH` — the path is unknown to the model, or it names a
  pure container (an object) and you used a scalar operator. Navigate to a
  scalar leaf instead. A path seen as *both* an object and a scalar across
  entities stays searchable via its scalar type. `IS_NULL`/`NOT_NULL` are
  exempt, since they test presence rather than a value.
- `400 INVALID_CONDITION` — a `null` operand on a binary or range operator, a
  range operand that is not a two-element array, or an object operand.

There is no operator-versus-type rejection: `CONTAINS` on a numeric field is a
valid request that simply evaluates to a non-match.

cyoda-go v0.8.3 converged two predicate evaluators that had drifted apart, so
the same query could previously return different results depending on which
one ran. Aligning them changed observable behaviour:

- cross-type coincidental matches no longer occur;
- negatives no longer match null or absent fields;
- `LIKE` is a real glob on every backend, where SQL backends previously
  neutered the wildcards;
- `BETWEEN_INCLUSIVE` performs an inclusive range check, where
  `Searcher`-backed stores previously fell through to a regex evaluation;
- validation is parse-based, so some queries that used to fail now succeed,
  and some that used to pass are now rejected.

Re-check any query whose correctness you depend on, particularly negations and
`LIKE` patterns.

## Historical reads with `pointInTime`

Every search accepts a `pointInTime` parameter to run against the world
as it existed at a given timestamp. Each entity maintains a history of
revisions; point-in-time queries return results using the entity state
that was current at the specified timestamp. The result is the set of
entities that would have matched, using the revision active at that
time.

The rule is canonical across every storage engine and read path: **inclusive of the requested instant** (`<=`), compared at native precision with no millisecond round-up.

`pointInTime` is a **query parameter** (RFC 3339), not a body field:

```bash
curl -X POST 'http://localhost:8080/api/search/direct/orders/1?pointInTime=2026-03-01T00:00:00Z' \
  -H 'Content-Type: application/json' \
  -H "Authorization: Bearer $TOKEN" \
  -d '{
    "type": "lifecycle", "field": "state",
    "operatorType": "EQUALS", "value": "submitted"
  }'
```

This is the primary way to answer audit and regulatory questions from
REST — *what did this customer's open orders look like at quarter
close?* The Trino surface exposes the same capability as a column named
`point_time` (snake-case, matching SQL convention); for the analytical
form, see [`point_time` in analytics](/build/analytics-with-sql/).

## Sorting results

Both search endpoints (direct and async) accept sort keys. Over HTTP, add one or more `sort` query parameters using the grammar `[@]path[:asc|desc]`:

- a bare dotted path sorts on a **scalar data field** (`amount`, `customer.tier`);
- an `@`-prefix sorts on a **meta field** — one of `state`, `creationDate`, `lastUpdateTime`, `transitionForLatestSave`, `transactionId`, `id`;
- direction defaults to `asc`.

```
POST /api/search/direct/orders/1?sort=@state&sort=amount:desc
```

Ordering is **canonical across every backend**: text by byte order, numeric by IEEE-754 double, bool as `false < true`. Meta date fields order chronologically; temporal *data* paths are ordered as text, which for ISO-8601 values coincides with chronological order. Note that this is the **sort** contract — predicate *comparison* uses the arbitrary-precision, type-directed rules described under [predicate semantics](#predicate-semantics). Absent or null values sort last, and the entity id is always the final tiebreaker. An unsortable, array, or unknown path is rejected with `400 INVALID_FIELD_PATH`, as is exceeding the sort-key cap set by `CYODA_SEARCH_MAX_SORT_KEYS` (default `16`; see the [configuration reference](/reference/configuration/#vars-search)). Over gRPC the same capability is expressed as a structured `orderBy` array.

Sorting a direct search does not let you take the top N of a large
population. Because direct search is bounded-or-fail, the *whole matched set*
must fit within `limit` before ordering is applied — so `sort` plus a small
`limit` over a large model returns `400 SEARCH_RESULT_LIMIT` rather than the
first N rows.

Run these as async searches, which snapshot the full matched set and page over
it in order.

### Paging (async)

- `pageSize` and `pageNumber` are query parameters on `/search/async/{jobId}`; they apply at result-fetch time, not at job submission. `pageNumber` is zero-indexed.
- A completed `jobId` is stable for its 24-hour retention window — page reads are idempotent.

## Searching inside a transaction

A search issued inside an active transaction is **read-your-own-writes
correct**: it sees that transaction's uncommitted writes, without falling back
to a full-model scan. The memory and SQLite backends overlay the transaction
buffer on the committed stream; PostgreSQL runs the query natively on the
transaction's own connection.

By default the search is a plain snapshot read that records nothing. Pass
`trackingRead=true` to opt into recording the entities the search **returns**
into the transaction's read-set, so that a concurrent commit touching any of
them aborts your transaction with `409 Conflict` at commit time:

```
POST /api/search/direct/orders/1?trackingRead=true
```

The protection is entity-level, the same as a `GetAll` read. Neither setting
protects against **phantoms** — an entity that starts matching your predicate
after the snapshot was taken is not detected either way. The flag is ignored
outside a transaction, and async search does not expose it at all, since it
runs detached.

## Performance notes

- Scope by `state` or a high-selectivity field first — the workflow
  state is indexed on every entity and is almost always the right
  first predicate.
- Prefer `async` as soon as the result set might be thousands of
  entities; the distributed execution on the Cassandra tier makes it
  cheaper per entity than a series of `direct` pages.
- Avoid open-ended `pointInTime` scans across every revision — anchor
  the query at a specific timestamp or a short window.
- On PostgreSQL, supported predicates push down into SQL — JSONB extraction plus numeric, range, and string comparisons run in the database. Non-pushable operators (regex, case-insensitive) are post-filtered as rows stream. This removes full-model scans and per-document decode; it is a constant-factor win, not a JSON-path index (indexing queried paths remains a separate operational step). SQLite already does this; the in-memory backend filters in memory by design.
- Pushdown is a **narrowing optimization only** — the in-process kernel remains authoritative for every match decision, so results never diverge between backends. Pushdown applies inside transactions too, so an in-transaction search no longer degrades into a full `GetAll` scan.
- Boolean conditions on PostgreSQL are pushed down correctly as of v0.8.3. Previously an `EQUALS`/`NOT_EQUAL` against a boolean field returned `500`, because the planner bound a raw Go `bool` against a text-typed JSON extraction that the driver cannot encode.
- A condition the plugin cannot translate falls back to in-memory filtering after a full scan. If that residual scan examines more rows than the backend's budget allows, it fails with `400 SCAN_BUDGET_EXHAUSTED` — narrow the query or add an indexable predicate rather than retrying.

## Grouped statistics

When you want **counts and aggregates** rather than the entities
themselves — *how many orders are in each state? what is the total
amount per country?* — use the grouped-statistics query instead of
paging a search result and summing client-side. It returns one row per
group and never the underlying entity bodies, so it stays cheap over
large populations.

```http
POST /entity/stats/{entityName}/{modelVersion}/query
```

```bash
curl -X POST http://localhost:8080/api/entity/stats/orders/1/query \
  -H 'Content-Type: application/json' \
  -H "Authorization: Bearer $TOKEN" \
  -d '{
    "groupBy": ["state", "$.country"],
    "aggregations": [
      { "op": "sum", "field": "$.amount", "as": "totalAmount" },
      { "op": "avg", "field": "$.amount" }
    ],
    "condition": {
      "type": "simple",
      "jsonPath": "$.channel",
      "operation": "EQUALS",
      "value": "web"
    }
  }'
```

The request has four parts:

- **`groupBy`** (required) — an ordered list of dimensions. Each entry is
  either the literal `state` (the workflow state) or a `$.`-prefixed
  JSONPath into the entity payload. The result `groupKey` is ordered to
  match.
- **`aggregations`** (optional) — per-group numeric aggregates. Each is
  `{ "op", "field", "as" }` where `op` is one of `sum`, `avg`, `min`,
  `max`, `stdev`, `field` is a JSONPath, and `as` is an optional alias
  (defaults to `op(field)`, e.g. `sum($.amount)`). Every group also
  carries a `count` whether or not you request aggregations.
- **`condition`** (optional) — a predicate that restricts the population,
  using the **same** [condition DSL](#the-condition-dsl) as search.
- **`limit`** (optional) — caps the number of buckets returned; must be
  ≤ the server's `CYODA_STATS_GROUP_MAX` (default 10000).

The response is one bucket per group:

```json
[
  {
    "groupKey": [
      { "field": "state", "value": "submitted" },
      { "field": "$.country", "value": "GB" }
    ],
    "count": 31,
    "aggregations": { "totalAmount": 48210.0, "avg($.amount)": 1555.16 }
  }
]
```

Aggregation values are numeric, or `null` when a group has no eligible
inputs.

### Point-in-time statistics

Like search, the query accepts a `pointInTime` to aggregate the world
**as it existed at a past instant** — without standing up a derived
counter entity that processors must maintain on every transition. Add it
to the request body:

```json
{
  "groupBy": ["state"],
  "pointInTime": "2026-03-31T23:59:59Z"
}
```

This answers end-of-quarter and audit roll-up questions directly — *how
many claims were pending at midnight on the last day of Q1?* When
omitted, the query runs against the current consistency time.

## Where to go next

- [REST API reference](/reference/api/) — authoritative search payload
  schema, operator grammar, status and result endpoints.
- [Working with entities](/build/working-with-entities/) — single-entity
  CRUD and transitions; the CRUD page for reference on the same
  surface.
- [Analytics with SQL](/build/analytics-with-sql/) — heavy analytical
  work, cross-entity joins, historical scans via `point_time`.
- [Entities and lifecycle](/concepts/entities-and-lifecycle/) — the
  audit/history model behind `pointInTime`.