Skip to content
Settings

Searching entities

Cyoda exposes search over REST for any query that returns a set of entities. Use it when you need more than one entity back, when the filter goes beyond a single id lookup, or when you want to scope by workflow state. For single-entity reads, stay on the CRUD endpoints in working with entities; for cross-entity analytics, use SQL; for event-driven compute, use gRPC compute nodes. For counts and aggregates without the entity bodies, jump to grouped statistics.

Cyoda splits search into Immediate and Background modes. Pick by expected result size and urgency; the filter grammar is identical.

  • Immediate (API term: direct) — synchronous. The request streams matching entities back as NDJSON. It is bounded-or-fail: limit caps the matched set, and a query matching more than that fails outright rather than returning part of the answer. Use it when you know the filter produces a bounded, small set — a UI list, a lookup, a small report.
  • Background (API term: async) — queued. The request returns a job handle; poll it, then page the results. Result size is unbounded. On the Cassandra-backed tier (Cyoda Cloud, or a licensed Enterprise install), async runs distributed across the cluster: for a fixed query shape, throughput scales roughly linearly with the number of nodes.

The decision tree is short:

  • Small bounded result, UI-facing → direct.
  • Might be large, can tolerate a second or two of queuing; exports, reports, batch jobs → async.
  • Hitting SEARCH_RESULT_LIMIT or the request timeout → async.
  • Ordered top-N over a large model → async (see sorting).

Filter by a combination of entity fields and workflow state:

Terminal window
curl -X POST 'http://localhost:8080/api/search/direct/orders/1?limit=200' \
-H 'Content-Type: application/json' \
-H "Authorization: Bearer $TOKEN" \
-d '{
"type": "group",
"operator": "AND",
"conditions": [
{ "type": "lifecycle", "field": "state",
"operatorType": "EQUALS", "value": "submitted" },
{ "type": "simple", "jsonPath": "$.customerId",
"operatorType": "EQUALS", "value": "CUST-7" }
]
}'

The path is /api/search/direct/{entityName}/{modelVersion}, and the request body is the condition document itself — there is no wrapper object. limit, pointInTime, trackingRead, timeoutMillis and sort are query parameters, not body fields.

The response is application/x-ndjson: one complete entity envelope per line, each carrying data plus a meta block with the id, model key, state, and timestamps.

{"type":"ENTITY","data":{"customerId":"CUST-7"},"meta":{"id":"74807f00-…","state":"submitted","creationDate":"2026-03-01T10:00:00.000000000Z"}}
{"type":"ENTITY","data":{"customerId":"CUST-7"},"meta":{"id":"89abc100-…","state":"submitted","creationDate":"2026-03-02T11:30:00.000000000Z"}}

Because it is a stream, parse it line by line rather than buffering the whole body as a single JSON document.

Submit the search to /api/search/async/{entityName}/{modelVersion} and capture the handle:

Terminal window
curl -X POST http://localhost:8080/api/search/async/orders/1 \
-H 'Content-Type: application/json' \
-H "Authorization: Bearer $TOKEN" \
-d '{
"type": "lifecycle",
"field": "state",
"operatorType": "EQUALS",
"value": "submitted"
}'

The response is a bare UUID string — the job id. Poll its status until the job reports SUCCESSFUL, then page the results:

GET /api/search/async/{jobId}/status
GET /api/search/async/{jobId}?pageNumber=0&pageSize=1000
GET /api/search/async/{jobId}?pageNumber=1&pageSize=1000

The status response carries searchJobStatus (RUNNING, SUCCESSFUL, FAILED, CANCELLED, or NOT_FOUND), entitiesCount for the total match count, and expirationDate. Fetching results before the job is SUCCESSFUL returns 400 BAD_REQUEST.

Results come back as a paged JSON object — content plus a page block with number, size, totalElements and totalPages — not as a stream. Job results expire 24 hours after creation; a single jobId can be paged repeatedly until then.

Async search never returns SEARCH_RESULT_LIMIT: it snapshots the whole matched set and pages over it. An out-of-range pageSize or pageNumber is a plain 400 BAD_REQUEST.

The server translates the condition before it stores the job. A condition that no backend can execute is refused at submission. The server then streams the results to storage as the scan runs.

A fixed worker pool serves the submissions. When the workers and the submit queue are both full, submission fails with a retryable 503 SEARCH_QUEUE_FULL.

Variable Default Meaning
CYODA_SEARCH_ASYNC_WORKERS 8 Jobs that execute at the same time on this node
CYODA_SEARCH_ASYNC_QUEUE 256 Submissions that wait for a worker
CYODA_SEARCH_ASYNC_MAX_PER_TENANT 8 One tenant’s share of queued and running jobs

A running job records its liveness every CYODA_SEARCH_JOB_HEARTBEAT_INTERVAL (default 15s). It starts to do this at submission, not at the start of the scan.

If the owning node stops before the job reaches a terminal status, a reaper takes over. The reaper claims the job when the heartbeat has been silent for CYODA_SEARCH_JOB_STALE_AFTER (default 5m). It deletes the partial results of the lost executor. It then runs the job again on a live node, at the pointInTime stored with the job. The client sees a longer RUNNING period, and the job ends SUCCESSFUL.

A reaped executor that recovers cannot write to a result set that another node has taken over.

A job ends FAILED in three cases:

  • The search itself failed.
  • The job lost its executor CYODA_SEARCH_JOB_MAX_ATTEMPTS times (default 3).
  • The model’s schema became unreadable between submission and execution.

A graceful shutdown releases its in-flight jobs immediately for reclaim. A released claim does not count against the attempt cap, so a rolling restart of any length is safe.

If a job is no longer needed — the user navigated away, a replacement query was submitted, the deployment is shutting down — cancel it rather than letting it run to completion:

Terminal window
curl -X PUT http://localhost:8080/api/search/async/{jobId}/cancel \
-H "Authorization: Bearer $TOKEN"

Cancellation is cooperative. The server stops the work at the next safe point and deletes any partial results for that jobId.

Cancellation succeeds only while the job is RUNNING. A job that is already SUCCESSFUL, FAILED or CANCELLED returns 400 SEARCH_JOB_ALREADY_TERMINAL, with currentStatus and snapshotId in the response properties.

The request body is a Condition document. Every condition carries a type discriminator. The same DSL is used by workflow and transition criteria, so what you learn here transfers directly.

simple — match a JSONPath into the entity payload:

{ "type": "simple", "jsonPath": "$.amount",
"operatorType": "GREATER_OR_EQUAL", "value": 1000 }

lifecycle — match entity metadata rather than payload. field is one of state, creationDate, lastUpdateTime, transitionForLatestSave (alias previousTransition), transactionId, id:

{ "type": "lifecycle", "field": "state",
"operatorType": "EQUALS", "value": "submitted" }

group — combine conditions. operator is AND, OR or NOT, case-sensitive; anything else is 400 INVALID_CONDITION. Groups nest to a maximum depth of 50:

{
"type": "group",
"operator": "AND",
"conditions": [
{ "type": "lifecycle", "field": "state", "operatorType": "EQUALS", "value": "submitted" },
{ "type": "simple", "jsonPath": "$.amount", "operatorType": "GREATER_OR_EQUAL", "value": 1000 }
]
}

array — match positional values in a JSON array. The path addresses the array’s elements, so it must carry a trailing [*]; a bare path is 400 INVALID_FIELD_PATH. A null entry tests nothing at that index:

{ "type": "array", "jsonPath": "$.laureates[*]",
"values": ["John", null, "Hopfield"] }

Each non-null entry is a positional test, and the server combines the tests with AND. The example matches when element 0 equals "John" and element 2 equals "Hopfield". A values array of only null entries matches every entity.

function — delegate the predicate to a compute node. This clause is criteria-only. Criteria and search share one condition DSL, but search does not accept a function clause. The server rejects a function clause at any depth in a search, async-search, grouped-statistics or conditional-delete body with 400 INVALID_CONDITION. See workflows and processors for its use as a transition criterion.

operatorType also accepts the spellings operator and operation. The available operators are:

Group Operators
Comparison EQUALS, NOT_EQUAL, GREATER_THAN, GREATER_OR_EQUAL, LESS_THAN, LESS_OR_EQUAL
Range BETWEEN, BETWEEN_INCLUSIVE — value must be a two-element [low, high] array
Presence IS_NULL, NOT_NULL
String CONTAINS, STARTS_WITH, ENDS_WITH and their NOT_* inverses
Case-insensitive IEQUALS, INOT_EQUAL, ICONTAINS, INOT_CONTAINS, ISTARTS_WITH, INOT_STARTS_WITH, IENDS_WITH, INOT_ENDS_WITH
Pattern LIKE (anchored glob), MATCHES_PATTERN (anchored RE2 regex)

The server rejects an operator outside this list, and a missing operatorType, with 400 INVALID_CONDITION. The error detail lists the valid operators. Every surface that carries a condition uses this code, including workflow import.

IS_CHANGED and IS_UNCHANGED are change-generation operators, not search predicates. cyoda-go does not implement them.

A jsonPath is JSON Path. The server checks it at the API boundary, before anything executes. One scanner enforces this grammar for search conditions, workflow criteria, groupBy and aggregation fields. A sort key uses the same segment charset, with one difference: its $. leader is optional.

jsonPath = "$." segment ( "." segment )*
segment = name subscript*
name = 1*( ALPHA / DIGIT / "_" / "-" ) ; ASCII only
subscript = "[" ( "*" / 1*DIGIT ) "]" ; the index must fit an int32

The $. leader is required. A bare amount is not a path, and it is not an alias for $.amount. The server rejects these paths with 400 INVALID_FIELD_PATH:

  • A bare name, such as amount.
  • An empty path.
  • An empty or trailing segment, such as $..a or $.a..
  • Bracket-quoted property access, such as $['x'] or $.a["b"]. Write $.x.
  • Any character outside the segment set.

The syntax of a path decides what the path addresses.

Path Addresses On {"tags": []}
$.tags The array itself NOT_NULL is true
$.tags[*] Every element IS_NULL and NOT_NULL are both false
$.tags[0] One position IS_NULL is true

A leaf on $.tags[*] is true when some element satisfies it. For example, $.tags[*] EQUALS "red" selects the entities whose tags contains "red".

Write one [*] for each array level, as in $.matrix[*][*] and $.orders[*].lines[*].sku. A trailing [*] over an array of objects under a scalar operator is 400 INVALID_FIELD_PATH. The element has no scalar form, so address the leaf instead: write $.items[*].sku, not $.items[*].

The server rejects every other bracket form with 400 INVALID_FIELD_PATH: unclosed or unmatched ($.a[, $.a]), no field name before it ($.[0]), empty ($.a[]), negative or signed ($.a[-1]), a slice ($.a[0:2]), a union ($.a[0,1]), a filter expression ($.a[?(@.x)]), whitespace inside ($.a[ 0]), or an index too large for a signed 32-bit integer. The server scans the full path, so it also rejects text after a valid subscript, such as $.a[0]b or $.a[0];DROP.

A field declared in more than one kind stays searchable through each kind. The server accepts a path that is valid for at least one declared branch. For each entity, the predicate applies to the branch that the entity’s data uses. An entity on a different branch does not match. It is not an error.

For a field declared as both string and array-of-string, $.a EQUALS "A" selects the scalar-shaped entities. $.a[*] EQUALS "A" selects the array-shaped entities. The server accepts both conditions.

This grammar does not apply to lifecycle metadata. A lifecycle condition names a meta field directly.

NOT inverts the answer of its child condition. NOT(c) is true when c is false.

NOT takes exactly one child condition. Zero children, or two or more, is 400 INVALID_CONDITION. To negate a group, nest the group inside the NOT:

{
"type": "group",
"operator": "NOT",
"conditions": [
{ "type": "simple", "jsonPath": "$.tags[*]", "operatorType": "EQUALS", "value": "red" }
]
}
  • Over a wildcard path, NOT is a universal quantifier. The leaf beneath it stays existential. NOT($.tags[*] EQUALS "red") matches when no element equals "red". $.tags[*] NOT_EQUAL "red" matches when some element is different. For {"tags": ["red", "blue"]} the first is false and the second is true.
  • NOT matches an empty list, an explicit null and an absent field. The wrapped leaf is false in all three states. On an entity with no x, NOT($.x EQUALS "A") matches. Both $.x EQUALS "A" and $.x NOT_EQUAL "A" do not match.
  • The server does not apply De Morgan’s laws to NOT. NOT(EQUALS) is not NOT_EQUAL. NOT(IS_NULL) is not NOT_NULL. The server does not distribute a NOT group over its children.

There is no “every element” operator, and NOT does not build one. On {"tags": ["red", null]}, NOT($.tags[*] NOT_EQUAL "red") looks like “every tag is red”, and it matches: the leaf is false at the null element, so the NOT is true. A null element makes any such construction report success.

NOT is residual-only. No backend pushes a NOT into its own query language. The in-process kernel evaluates it.

Search and workflow criteria run on one shared evaluation kernel, so a predicate means the same thing wherever you write it. The kernel implements Cyoda Cloud’s model; cyoda help predicates is the exhaustive reference.

Comparison is type-directed and same-type. An operand is parse-tested against the field’s declared type(s), so the JSON number 30 and the string "30" behave identically. There is no cross-type coincidental matching: an operand that parses only as a number never matches a string-stored value. Numbers compare at arbitrary precision, so values beyond 2^53 are exact.

An operator that can never be satisfied still answers. The server accepts an operand that parses into at least one declared type, then compares it against each stored-value type family. Where no stored value of that type can equal the operand, a positive operator answers non-match and a negative operator answers match.

For example, $.n NOT_EQUAL 12.5 on a field declared INTEGER returns every entity that holds a number at n, because no integer equals 12.5. Take care with a negative operator in a conditional delete: it selects this wider set.

Absent and null fields never match a binary operator — including negatives. NOT_EQUAL, NOT_CONTAINS, INOT_* and the rest are null-guarded to non-match rather than evaluated as !positive. A missing field does not satisfy NOT_EQUAL just because it fails EQUALS. Use IS_NULL and NOT_NULL to test presence — they are the only operators that do. NOT is not a binary operator, so this guard does not apply to it. See negation.

LIKE is a glob. The server matches the operand directly. It does not translate the operand into a regular expression, so a regex metacharacter has no special meaning. Three characters are special:

Character Matches
% Any sequence of characters, including an empty sequence and a newline
_ Exactly one character (one UTF-8 rune), including a newline
\ Escapes the next character to its literal form

So \d matches the letter d, and \\ matches one backslash. The match is whole-string anchored and case-sensitive. Wildcards match rune by rune, and the server compares every other character byte by byte. A pattern that ends with an unpaired \ is rejected. To match a literal backslash at the end, write \\.

MATCHES_PATTERN is a whole-string-anchored RE2 regular expression. RE2 is different from the Java dialect for some constructs, such as backreferences and some lookaround. The operand must parse on its own, and it must also compile when the server anchors it.

Temporal fields compare chronologically, not lexically, across LOCAL_DATE, LOCAL_DATE_TIME, LOCAL_TIME, ZONED_DATE_TIME, YEAR and YEAR_MONTH. The creationDate and lastUpdateTime meta fields accept a coarser operand that upscales — "2024" or "2024-09" are valid against a full timestamp.

Validation is parse-based, evaluated against the target model:

  • 400 CONDITION_TYPE_MISMATCH — the operand parses into none of the field’s declared types. Comparison and range operators only; string and pattern operators and presence tests carry no operand-type constraint.
  • 400 INVALID_FIELD_PATH — the path is outside the grammar, is unknown to the model, or names a container with no scalar form: a bare object path, or a [*] path over an array whose elements are only ever objects. Navigate to a scalar leaf instead. A path declared in more than one kind stays searchable through its scalar kind — see how a field path is written. IS_NULL/NOT_NULL are exempt from the container rejection, since they test presence rather than a value.
  • 400 INVALID_CONDITION — a structural or shape failure rather than a path or type one: an unknown or missing operatorType, a null, object or complex operand on a binary or range operator, a range operand that is not a two-element array, a malformed LIKE/MATCHES_PATTERN operand, a string or pattern operator on a temporal meta field, an array clause on a bare path, or a function clause at any depth.
  • 500 — the server could not load or parse the model’s schema, so it could not check the condition against the schema. The response carries a ticket id and no result set. A condition that contains only lifecycle clauses needs no schema, and the server answers it normally.

There is no operator-versus-type rejection on a data path. CONTAINS against a numeric field is a valid request, and it evaluates to a non-match. The exception is the two temporal meta fields. A string or pattern operator on creationDate or lastUpdateTime is 400 INVALID_CONDITION.

Every search accepts a pointInTime parameter to run against the world as it existed at a given timestamp. Each entity maintains a history of revisions; point-in-time queries return results using the entity state that was current at the specified timestamp. The result is the set of entities that would have matched, using the revision active at that time.

The rule is canonical across every storage engine and read path: inclusive of the requested instant (<=), compared at native precision with no millisecond round-up.

pointInTime is a query parameter (RFC 3339), not a body field:

Terminal window
curl -X POST 'http://localhost:8080/api/search/direct/orders/1?pointInTime=2026-03-01T00:00:00Z' \
-H 'Content-Type: application/json' \
-H "Authorization: Bearer $TOKEN" \
-d '{
"type": "lifecycle", "field": "state",
"operatorType": "EQUALS", "value": "submitted"
}'

This is the primary way to answer audit and regulatory questions from REST — what did this customer’s open orders look like at quarter close? The Trino surface exposes the same capability as a column named point_time (snake-case, matching SQL convention); for the analytical form, see point_time in analytics.

Both search endpoints (direct and async) accept sort keys. Over HTTP, add one or more sort query parameters using the grammar [@]path[:asc|desc]:

  • a bare dotted path sorts on a scalar data field (amount, customer.tier);
  • an @-prefix sorts on a meta field — one of state, creationDate, lastUpdateTime, transitionForLatestSave, transactionId, id;
  • direction defaults to asc.
POST /api/search/direct/orders/1?sort=@state&sort=amount:desc

Ordering is canonical across every backend: text by byte order, numeric by IEEE-754 double, bool as false < true. Meta date fields order chronologically; temporal data paths are ordered as text, which for ISO-8601 values coincides with chronological order. Note that this is the sort contract — predicate comparison uses the arbitrary-precision, type-directed rules described under predicate semantics. Absent or null values sort last, and the entity id is always the final tiebreaker. An unsortable, array, or unknown path is rejected with 400 INVALID_FIELD_PATH, as is exceeding the sort-key cap set by CYODA_SEARCH_MAX_SORT_KEYS (default 16; see the configuration reference).

A sort key is a scalar path over the same segment charset as a condition path, with one difference: the $. leader is optional. $.year:desc and year:desc are the same key.

The server rejects an array subscript or projection, such as $.items[*].name or $.items[0].name, with 400 INVALID_FIELD_PATH. An ordering needs one scalar value per entity.

Over gRPC the same capability is a structured orderBy array, over the same grammar and the same resolver.

  • pageSize and pageNumber are query parameters on /search/async/{jobId}; they apply at result-fetch time, not at job submission. pageNumber is zero-indexed.
  • A completed jobId is stable for its 24-hour retention window — page reads are idempotent.

A search issued inside an active transaction is read-your-own-writes correct: it sees that transaction’s uncommitted writes, without falling back to a full-model scan. The memory and SQLite backends overlay the transaction buffer on the committed stream; PostgreSQL runs the query natively on the transaction’s own connection.

By default the search is a plain snapshot read that records nothing. Pass trackingRead=true to opt into recording the entities the search returns into the transaction’s read-set, so that a concurrent commit touching any of them aborts your transaction with 409 Conflict at commit time:

POST /api/search/direct/orders/1?trackingRead=true

The protection is entity-level, the same as a GetAll read. Neither setting protects against phantoms — an entity that starts matching your predicate after the snapshot was taken is not detected either way. The flag is ignored outside a transaction, and async search does not expose it at all, since it runs detached.

The server translates every condition into a backend query. A condition it cannot translate is 400 INVALID_CONDITION, or 400 INVALID_FIELD_PATH for a path failure. The same rule applies to search, conditional delete and grouped statistics, so all three answer a given condition the same way.

No backend meters the rows a search examines, and the server sets no time budget. It bounds search results, not search time. A direct search is bounded-or-fail on limit. An async search bounds neither duration nor result count. Two controls bound the time:

  • timeoutMillis on a direct search.
  • Cancellation on an async job.

If you supply neither, the search is unbounded. The one server-side exception is an operator-configured ceiling on the async scan, CYODA_POSTGRES_SEARCH_STATEMENT_TIMEOUT.

POST /api/search/direct/orders/1?timeoutMillis=5000

timeoutMillis has no default. When the time elapses, the server stops the search and fails the request with 408 SEARCH_TIMEOUT. It returns no partial results.

The server rejects timeoutMillis with 400 BAD_REQUEST on a non-positive value, and on a request that joins an open transaction.

  • Scope by state or a high-selectivity field first — the workflow state is indexed on every entity and is almost always the right first predicate.
  • Prefer async as soon as the result set might be thousands of entities; the distributed execution on the Cassandra tier makes it cheaper per entity than a series of direct pages.
  • Avoid open-ended pointInTime scans across every revision — anchor the query at a specific timestamp or a short window.
  • On PostgreSQL, supported predicates push down into SQL — JSONB extraction plus numeric, range, and string comparisons run in the database. Non-pushable operators (regex, case-insensitive) are post-filtered as rows stream. This removes full-model scans and per-document decode; it is a constant-factor win, not a JSON-path index (indexing queried paths remains a separate operational step). SQLite already does this; the in-memory backend filters in memory by design.
  • Pushdown is a narrowing optimization only — the in-process kernel remains authoritative for every match decision, so results never diverge between backends. Pushdown applies inside transactions too, so an in-transaction search does not degrade into a full-model scan.
  • Boolean conditions on PostgreSQL are pushed down correctly as of v0.8.3. Previously an EQUALS/NOT_EQUAL against a boolean field returned 500, because the planner bound a raw Go bool against a text-typed JSON extraction that the driver cannot encode.
  • A NOT, and a wildcard path under a scalar operator, are residual. The kernel re-checks them as candidate rows stream. Put a selective pushable predicate in the same AND group to narrow the population first.

When you want counts and aggregates rather than the entities themselves — how many orders are in each state? what is the total amount per country? — use the grouped-statistics query instead of paging a search result and summing client-side. It returns one row per group and never the underlying entity bodies, so it stays cheap over large populations.

POST /entity/stats/{entityName}/{modelVersion}/query
Terminal window
curl -X POST http://localhost:8080/api/entity/stats/orders/1/query \
-H 'Content-Type: application/json' \
-H "Authorization: Bearer $TOKEN" \
-d '{
"groupBy": ["state", "$.country"],
"aggregations": [
{ "op": "sum", "field": "$.amount", "as": "totalAmount" },
{ "op": "avg", "field": "$.amount" }
],
"condition": {
"type": "simple",
"jsonPath": "$.channel",
"operation": "EQUALS",
"value": "web"
}
}'

The request has four parts:

  • groupBy (required) — an ordered list of dimensions. Each entry is either the literal state (the workflow state) or a $.-prefixed JSONPath into the entity payload. The result groupKey is ordered to match. Duplicate entries are 400 DUPLICATE_GROUP_BY.
  • aggregations (optional) — per-group numeric aggregates. Each is { "op", "field", "as" } where op is one of sum, avg, min, max, stdev, field is a JSONPath, and as is an optional alias (defaults to op(field), e.g. sum($.amount)). Every group also carries a count whether or not you request aggregations.
  • condition (optional) — a predicate that restricts the population, using the same condition DSL as search.
  • limit (optional) — caps the number of buckets returned; must be ≤ the server’s CYODA_STATS_GROUP_MAX (default 10000).

The response is one bucket per group:

[
{
"groupKey": [
{ "field": "state", "value": "submitted" },
{ "field": "$.country", "value": "GB" }
],
"count": 31,
"aggregations": { "totalAmount": 48210.0, "avg($.amount)": 1555.16 }
}
]

Aggregation values are numeric, or null when a group has no eligible inputs.

Both groupBy and an aggregation field follow the same path grammar as a search condition, but they do not accept array subscripts. A group key must be a single scalar. The server rejects $.items[*] and $.items[0] with 400 INVALID_GROUP_BY_PATH for a groupBy entry, or 400 INVALID_AGGREGATION_FIELD for an aggregation.

There is also no dotted form for an array position. $.items.0 names a field called 0, not an element.

The server validates each path against the model. A path that names an undeclared field is rejected.

Like search, the query accepts a pointInTime to aggregate the world as it existed at a past instant — without standing up a derived counter entity that processors must maintain on every transition. Add it to the request body:

{
"groupBy": ["state"],
"pointInTime": "2026-03-31T23:59:59Z"
}

This answers end-of-quarter and audit roll-up questions directly — how many claims were pending at midnight on the last day of Q1? When omitted, the query runs against the current consistency time.