Skip to content
Settings

SEARCH_QUEUE_FULL — async-search submission was refused for capacity

cyoda-go version 0.8.4

SEARCH_QUEUE_FULL — the node refused an async-search submission because its capacity for one is already committed.

HTTP: 503 Service Unavailable. Retryable: yes.

POST /api/search/async/{entityName}/{modelVersion} runs on a bounded worker pool: a fixed number of workers drain a fixed-capacity queue. Submission fails fast with this error rather than blocking the request or spawning an unbounded goroutine per submission. The rejected submit leaves nothing behind — no job row, no id to poll.

Two guards raise it, both retryable, both meaning “back off and retry”:

  • Node capacity — no free worker and the submit queue at capacity. Sized by CYODA_SEARCH_ASYNC_WORKERS (default 8) and CYODA_SEARCH_ASYNC_QUEUE (default 256).
  • Tenant share — your tenant already holds its full share of this node’s in-flight jobs. Sized by CYODA_SEARCH_ASYNC_MAX_PER_TENANT (default: the worker count; 0 disables the cap). This bound keeps one tenant from consuming the whole pool.

The queue drains as in-flight jobs complete; a client that backs off and retries will typically succeed. Frequent occurrences mean sustained submission volume exceeds what the pool is sized to absorb — raise CYODA_SEARCH_ASYNC_QUEUE for burst tolerance, or CYODA_SEARCH_ASYNC_WORKERS for sustained throughput (bounded by the storage backend’s connection budget: each running job holds a scan connection plus, per chunk, a save connection).

See cyoda help config (Search internals) for the variables.

  • errors
  • errors.SEARCH_SHARD_TIMEOUT
  • errors.STORAGE_UNAVAILABLE
  • config
  • cyoda help errors — Every error response from the Cyoda REST API carries a structured errorCode in the properties object. Multiple codes may share the same HTTP status. Programmatic handling keys on errorCode, not HTTP status.
  • cyoda help errors SEARCH_SHARD_TIMEOUT — Distributed search fans out to multiple shards in parallel. If any shard does not return results before the search timeout expires, the job is marked failed and this error is returned. Occurs under high load, during partial cluster degradation, or with expensive queries.
  • cyoda help errors STORAGE_UNAVAILABLE — Raised in three cases, all transient:
  • cyoda help config — Environment variables beat default values. The _FILE suffix variant takes precedence over the plain variable when both are set — for example, CYODA_POSTGRES_URL_FILE=/etc/secrets/db-url wins over CYODA_POSTGRES_URL. There are no command-line flags for configuration values; env vars are the sole configuration surface.