Worker roles and scaling¶

Deploy the same reviewed backend artifact in two process roles. Start HTTP replicas with python -m koinon.serve and worker replicas with python -m koinon.worker. The API role is the default. If setting KOINON_PROCESS_ROLE explicitly, set api or worker for that process rather than in a shared environment file. A conflicting entrypoint or native scheduler override fails startup.

The complete worker enables native scheduling and runs conversation generation, index jobs, attachment extraction and cleanup, corporate profile synchronization, document ingestion, structured ingestion, callback delivery and retention. API replicas do not run these consumers. Workers do not construct the HTTP application, register controllers or enable MCP session storage. Keep koinon.knowledge.enabled true for native knowledge operations; compatibility HTTP exposure is independently controlled by koinon.legacy.enabled.

Durable execution and shutdown¶

Document and structured workers discover pending or expired jobs from their named control datasource every second, then claim and process one job per domain. Delivery does not depend on an in-process notification reaching another replica. The worker reuses persisted inputs and existing native pipelines. It restores only server-captured canonical identity, originating credential ID, source scope and exact model-selection revisions. It checks current native authorization before opening models and before committing stage output. It never stores a bearer token in execution context. Historical jobs lacking provable context fail with an explicit unresolved-authority result; they do not acquire a broader service identity.

Claims use existing attempt and start-time generations. Every stage transaction and terminal update verifies the generation, scope and lease. Structured callback rows reserve a distinct delivery generation before the claim transaction exits. Only the current reservation can send or settle a retry. Current job identity and originating credential must still authorize delivery. Document webhooks retain their existing best-effort delivery semantics and check current authority and attempt ownership immediately before sending. These checks fence a stale worker after another replica reclaims its job.

On SIGTERM or SIGINT, stop claiming work, allow a bounded drain and then cancel remaining work before closing shared resources. Use a platform termination grace longer than the configured drain and retain a finite hard termination deadline. A killed process may leave an unknown remote outcome; do not describe callback delivery as exactly once. Expired claims are recoverable and stale finalizers cannot overwrite a replacement attempt.

Investigating a failed conversation¶

Use the conversation run identifier to correlate its recorded status with worker logs. A failed knowledge tool emits Knowledge tool failed with the run identifier, tool-call identifier and tool name. A failure in the surrounding execution emits Conversation execution failed with the same run identifier.

The diagnostic contains exception types and the module, function and line where they occurred. It retains at most eight related exceptions and four frames per exception. Exception messages, local variables, source text, file paths, questions and provider credentials are excluded. The conversation keeps its existing failure or cancellation state; recording a diagnostic does not retry a tool or publish an answer. Read these logs through your deployment's restricted operations access and check the corresponding source revision before interpreting line numbers.

Capacity and demand¶

Keep at least one worker initially, including during periods with no interactive demand, so periodic profile and retention work continues. Scale API replicas from HTTP demand independently. Tune worker demand targets and replica limits using measured queue age, model latency, provider limits and database saturation.

The internal typed WorkerBacklog.snapshot() observation returns bounded counts for ten queues: conversation, index, attachment, profile, document ingestion, structured ingestion, callback, communications, bulk intake and bulk read-only queries. Each includes ready, active, retry, oldest ready age and declared per-replica capacity. Demand is ready plus active work, including jobs waiting for a provider. A missing datasource reports unavailable rather than an empty queue. No identifiers, prompts, source contents or credentials are returned, and this observation adds no public route or grant.

One native Bulkhead backs a shared process admission service. koinon.operations.admission.worker_capacity defaults to 1 root heavy operation across conversation, index, attachment, profile, document ingestion, structured ingestion, callback delivery, communications and bulk intake/query supervision. A worker acquires capacity before claiming a lease or loading its job payload and keeps it through execution and settlement. Waiting does not increment attempts or spend a lease. Admission is FIFO with at most max_waiters (default 64) and wait_timeout_seconds (default 30); a full or timed-out wait leaves durable work unclaimed. Shutdown rejects waiting and new roots while active work drains. Nested calls reuse their live root reservation; detached calls cannot reuse it after that root exits.

A worker capacity of 1 applies to each process: two worker replicas can admit two root operations. Shared database claim quotas may impose a lower limit for a particular queue.

API and MCP native model-bound operations use a separate koinon.operations.admission.api_capacity, also defaulting to 1. Conversation consumer concurrency still uses koinon.operations.worker_concurrency (1 through 16). Index and attachment each process one job at a time, profile synchronization up to two, and durable ingestion one per domain. Managed callbacks claim and deliver one admitted row at a time, with a bounded number of rows visited per poll. The effective capacity for any queue is bounded by both its own limit and the shared process limit. WorkerBacklog.snapshot() reports the shared process_capacity separately; do not sum per-queue capacities.

These limits count root operations, not bytes or every nested provider request. A single operation can contain bounded native parallel subtasks. File limits, parser subprocess limits, DuckDB settings, object prefetch and concurrent HTTP request bodies require their own bounds and measured sizing. Periodic retention and reconciliation remain separate maintenance. A capacity of 1 does not guarantee a 1 GiB memory footprint or qualify an instance size. Multiple queues may use the same provider, and aggregate provider demand grows with the replica count; there is no distributed provider quota. Set replica maxima and provider limits accordingly. Including active work avoids a false zero metric but does not guarantee that a platform retains every busy replica; graceful drain and fencing remain necessary.

Bulk consumers remain explicit opt-ins. Set PYFLY_KOINON_KNOWLEDGE_BULK_WORKER_ENABLED=true and PYFLY_KOINON_KNOWLEDGE_BULK_QUERY_WORKER_ENABLED=true on worker replicas that consume these queues. Both default to false. The API role forces file-configured flags off and rejects enabled native environment overrides. Each bulk supervisor acquires the shared heavy-work permit before claiming, retaining it until its exact child has stopped and drained. Claimed children use an API resource-only context with scheduling and both bulk consumers disabled; they do not start an HTTP listener or another consumer.

Bulk demand includes due queued jobs, strictly expired running leases, and live running jobs. Exhausted attempts and expired query batches still require a claim to settle their terminal state. Live running jobs remain demand even after cancellation until their lease ends. Per-database koinon.knowledge.bulk.max_workers_global and per-tenant quotas still bound claims across all replicas. A global limit of one serializes bulk execution even when the autoscaler adds replicas; choose limits together with replica bounds, database connections, CPU and provider budgets.

For a PostgreSQL autoscaler, grant only a reviewed counts-only observation or restricted function. Query the primary, document-control and structured-control datasources separately when they are different databases. Do not grant raw job or private execution-envelope reads to the scaler. When a deployment deliberately shares one physical database, an operator may expose a fixed-search-path counts function over the actual queue tables. Use the separate counts-only PostgreSQL installer after migrations. Different physical databases require a bounded external aggregate strategy; the installer rejects that topology.

SQL sessions, durable object storage and accepted private attachment storage must be shared across replicas. Process-local scratch storage must survive no longer than its job and must not be the durable source of queued bytes. Follow Production deployment, Storage configuration and Backup and recovery. Source and SQLite tests do not qualify PostgreSQL lock behavior, installed artifacts, a live provider or a managed rolling deployment.