Self-hosted workers

Mango is converging on one provider-neutral execution boundary: the control plane assigns a Session through Environment Work, and an operator-run worker executes tools in infrastructure selected by that operator. Docker, Daytona, Modal, Cloudflare, and Vercel are worker deployment choices, not Environment types and not control-plane provider values.

Mango control plane
  Work queue + Session events + tool results
                 |
Mango SDK worker layer
  WorkPoller + EnvironmentWorker + SessionToolRunner
                 |
Operator launcher
  Docker / Daytona / Modal / Cloudflare / Vercel
                 |
Generic sandbox infrastructure

Product decision

The current OSS product supports only self_hosted execution. Mango does not currently operate an Anthropic-style cloud Environment or shared hosted sandbox fleet. An operator may choose a commercial compute or sandbox service for their worker; that remains self-hosted from Mango's trust-boundary perspective because the operator owns the account, credentials, policy, and lifecycle. A future Mango-hosted product would be a separate product and trust- boundary decision, not a hidden provider value in this protocol.

The control plane must not import provider SDKs or expose provider-specific fields on Environment or Session resources. A provider example may use a generic API to create a sandbox, upload the same Mango runner, start it with the claimed Work identity, and tear it down. The runner, not the provider, owns the Mango protocol: event recovery, permission gates, heartbeat, lease loss, tool result submission, and Stop.

Earlier reference scope (2026-09-07)

The public Claude cookbook at main commit a97b9a2dc300635f0c26b5e05d0b54bbe0279ee5 and the current public Go, Python, and TypeScript SDK sources at commits de6914c544629b14a67c0695ce147edae6a291e0, 62de60b27d04f0927a0ccf0f2610597fafcfab6a, and ba14b1f4fdf2e840a7b32297965342a099f6201d were reviewed again on 2026-09-07. The cookbook reference set is Docker, Cloudflare Containers, a pure Cloudflare Worker variant, Modal, Daytona, and Vercel. Those implementations confirm the separation above: compute platforms expose generic container, process, filesystem, or volume primitives; the provider-neutral SDK/CLI worker owns the managed-agent protocol and Session Memory synchronization.

Mango will use the same separation without treating that list as a provider compatibility promise:

• Docker is the first required end-to-end reference and the OSS development default.
• Daytona, Modal, Cloudflare Containers, and Vercel are independent launcher examples after the shared runner exists.
• The pure Cloudflare Worker example is useful for the lower-level runner API, but its in-isolate fake filesystem is not equivalent to a Linux sandbox and must not be advertised as one.
• E2B, CubeSandbox, and OpenSandbox are outside this cookbook-alignment slice. A future example needs its own user/operator reason rather than inheriting an old compiled adapter.

Examples stay outside contract and system-test harnesses. Credential-free unit tests cover shared worker behavior; provider live checks remain explicit and opt-in.

Current behavior alignment

This table is a behavioral audit, not a compatibility claim. CMA's current security guide recommends passing a Work item's per-Session secret only to that Session sandbox, while its SDK retains an Environment-key fallback. Mango makes the narrower path mandatory: it will not pass a standing credential into a Session container merely to copy the cookbook script.

Assessment on 2026-09-30

This review starts from Mango runtime commit 7f4e2b5, its OpenAPI, startup code, and executable tests. It refreshes the earlier boundary comparison; it does not turn every external difference into implementation work.

CMA's self-hosted sandbox guide keeps orchestration and durable services at Anthropic and moves execution to the operator. Mango also hosts its own control plane, database, object storage, Memory, and scheduling. A sound SDK worker design does not establish that Mango's independently implemented storage and recovery are safe.

Compare the same execution mode

The current official SDK references are Go v1.76.0 (ad865dfa3d1a8d2f4a7ad0d072011e811e9957a9), Python v1.9.0 (a7285e919ab79998d9380b3b57f6315b7860b8d8), and TypeScript sdk-v0.129.0 (bf2058689f845dfb10e59bd9ebeb5cb4e9318a9d). Paired reads covered the Work resources and worker/Poller/runner responsibilities, heartbeat during input preparation, and Memory/Skill setup. The cookbook was reviewed at d7265d6ae994ccd8429db0594b000073b2f9ad43. Reference implementations were not executed or imported.

The cookbook Docker script still forwards an Environment key and discards the item secret, while the current security guide describes passing the item credential to its own sandbox. Follow the actual lifecycle and Mango trust boundary, not one script mechanically. Mango already requires the scoped item credential and keeps standing keys outside containers.

Completed work that must not be scheduled again

The approval bug was a missed application boundary. Model retry recovery and exhaustion tests already existed in runtime commit 0914ea4 (PR #215). PR #226 added a full-history reader that returned on every session.error; its fixture covered restart and ambiguous results but no historical provider retry. Normal live runs did not force such a failure. PR #230 adds independently authored recovered/active retry cases and keeps exhausted, terminal, and unknown errors fatal for this single-turn tutorial. That fixes the example without replacing the runtime's existing recovery.

Remaining work and evidence limits

The active-upload startup race identified in the September assessment is now addressed by upload recovery across API processes. Ordinary Files and Skills commit renewable database leases. Cleanup atomically claims only ended or expired uploads, and periodic scans collect crashes even when the replacement API started before lease expiry. Ready resources survive lost completion responses. Independent object cleanup guards retain late writes after the original metadata is gone, including writer crashes after publication. Non-reused revisions protect newer guards from stale acknowledgements. Unknown remote writes keep a guard and are revisited by bounded, rotating scans.

Two independently pooled services sharing real PostgreSQL and SeaweedFS now exercise upload-versus-reconcile on both sides of object publication, expiry, reused time-based Skill Versions, deletion outages, and completion-response loss. The earlier one-service concurrent lifecycle test remains useful but cannot establish those interleavings alone. This resolves the selected storage boundary; broader multi-replica rollout and production operation remain separate evidence requirements.

Generated MCP File publication now uses unique internal write keys under the stable File ID. Atomic receipt/key checks fence stale publication; independent guards track unknown writes through ready publication, Session/File deletion, and restart. Two-pool PostgreSQL/SeaweedFS tests cover delayed accepted requests, superseded writers, lost commit/read responses, and deletion outages. This extends the storage recovery evidence without establishing broader rollout readiness. Next, prioritize a versioned self-hosted alpha bundle, matched native SDK artifacts, and a demonstrated backup/restore procedure for database, objects, Memory, and encryption keys. Kubernetes distribution and worker rollout/versioning require their own operational acceptance; a Docker demo or CMA sandbox feature list does not prove them. Additional worker languages and private MCP remain separately selectable product work. Complete large MCP text retention now follows the File lifecycle described above. Cloud/OpenSandbox does not gate the current self-hosted scope. Capabilities remains the product inventory.

Lifecycle and security invariants

Web execution ownership

A self-hosted Session must be able to use the configured model endpoint's Web Search/Fetch alongside the six sandbox tools. The Environment selects where shell and file operations run; it does not move provider-native Web tools into the external worker. The current Messages adapter already supports those Web tools and preserves their opaque responses in the durable model transcript.

Acceptance criteria for this slice:

• The self-hosted path declares enabled Web tools as provider-native tools. Self-hosted Bash retains its persistent-shell contract.
• Only shell/file and custom calls may request an external result. Provider Web calls never create an external pending-action barrier; a malformed ordinary client call to a provider-owned Web tool is rejected before tool execution.
• Web tools retain the existing always_allow restriction. The current model adapter cannot suspend a provider-native call for a Mango approval.
• Tests verify mixed Web/worker requests, correlated external results, lossless Web transcript recovery, and failures without dispatching Environment Work.

This slice does not add a Web provider, new model credentials, domain filters, an external worker Web executor, or automatic File/Git transfer. Endpoints that do not support the current native Web declarations must use agents with those tools disabled. Full Docker/control-plane recovery and the default deployment cutover follow separately.

Work lifecycle

• Poll is a tentative claim; Ack must complete before execution is handed off.
• A worker that loses the heartbeat lease stops executing and must not submit a successful result afterward.
• Normal worker shutdown cancels an executing tool, gives its error result a bounded independent send window, then Stops the Work. Lease loss remains a hard fence: the old owner cancels locally without submitting any result or issuing Stop.
• Re-delivery and process restart are normal. Event IDs and persisted pending actions, not in-memory seen sets, determine whether a result is outstanding.
• A Work poller does not Stop acknowledged items. The composed EnvironmentWorker owns heartbeat, permanent-input Fail, and final Stop while SessionToolRunner owns only the Session event/tool loop; an ambiguous Ack or invalid Ack response is left for TTL reclaim.
• Poll rotates an unpredictable per-claim sessions_token inside the Work secret payload. The worker switches to it for heartbeat, Fail, Stop, Session events, and pinned immutable inputs after Ack; reclaim invalidates the old token and closes an established event stream. Lease TTL is capped at five minutes, and a graceful Stop can retain execution access for no longer than that current TTL. Ack continues to use the supervisor credential and every non-Poll Work response redacts the payload.
• The Session credential may submit tool results only. It cannot manufacture user input, approval decisions, or persistent system context.
• The same credential may read only Memory Stores attached to its frozen Session, and may mutate only read_write attachments. Every Memory mutation rechecks the live Work lease in the same PostgreSQL transaction as the write, so reclaim cannot race a previously authorized request.
• No model-provider key or broad server credential belongs in a sandbox.
• Mango authorizes supervisor Poll/Ack and queue Stats with an Environment key. The operator issues, lists, and revokes keys through mango api-key; revocation fences claims/Acks without cancelling already-Acked Work leases. Only the item token crosses into the Session sandbox. The supervisor still controls its Docker host and remains trusted infrastructure.

Incremental delivery

1. Added a Mango Go SDK WorkPoller over the existing Work endpoints. It handles poll, Ack, drain, reclaim, and cancellation, but neither stops Work, creates sandboxes, nor executes tools.
2. Added the CMA-shaped Work secret payload and per-Session bearer scope. Heartbeat, Stop, Session event execution, and pinned immutable inputs can use the item token; reclaim invalidates it before another worker starts.
3. Added a provider-neutral single-Session SessionToolRunner. It connects the stream before paginated history reconciliation, dispatches local and custom tools, honors durable confirmation gates, retries ambiguous result writes, and stops on terminal events, end-turn idle, cancellation, or lease loss.
4. Added a provider-neutral EnvironmentWorker composition. It performs a successful conditional heartbeat before tool execution, keeps heartbeating in parallel, derives result retry bounds from the lease TTL, cancels on lease loss, and force-Stops ordinary exits. It requires the per-Work token after Ack and never falls back to the supervisor's standing key. Environment keys now restrict the supervisor to its own queue; Docker host access still requires trusted operators.
5. Added a standalone Docker launcher. A trusted supervisor polls and Acks, creates one hardened container per Work item, gives it only the Work secret, and retains one named workspace volume per Session. A real Docker test covers Poll through Stop, Skill preparation before dispatch, and a second activation reading both its re-prepared Skill and the first activation's persistent file.
6. Added the persistent self-hosted Bash lifecycle. One PTY-backed shell keeps cwd, environment variables, and background jobs within an activation; explicit restart, per-call timeout, cancellation recovery, bounded output, and bounded shutdown are SDK-owned rather than Docker-specific.
7. Added provider-neutral Memory preparation. The worker downloads attached Stores before constructing tools, exposes writable and read-only roots, reconciles with SHA-256 preconditions after tool calls, performs a final sync or cancellation-safe push-only flush, and removes only trusted folders it created. Docker supplies a bounded /mnt/memory tmpfs; no provider logic enters the SDK lifecycle.
8. Keep Web Search/Fetch on the model endpoint. Only the six shell/file tools belong to the self-hosted built-in result protocol. Provider responses and error blocks survive external-result waits and orchestration-worker restart in the durable transcript.
9. Verified the Docker worker against real authenticated HTTP, PostgreSQL, Temporal, NATS, and Docker. The system test restarts the Temporal execution worker while an acknowledged container waits on an always_ask approval, then completes that activation and proves a second Work container can read the same Session workspace. Focused real-Docker tests retain deeper Skill, Memory, shell, cancellation, and lease-renewal coverage; PostgreSQL and provider-neutral worker tests retain lease-loss fencing coverage. Keeping those fault cases focused avoids one timing-heavy combinatorial test while the system test verifies that the real boundaries compose. An opt-in live smoke uses the same fixture for one real-model-selected Bash call without making external credentials a CI dependency.
10. Completed the public product convergence: omitted Environment config defaults to self_hosted and all first-party examples use that execution path; the old cloud path, compiled provider registry, File/Git Session Resources, and output-publication exception were removed directly from /v1 before a stable release.
11. Add thin provider examples one at a time. Each must use the same runner and document persistence, cancellation, resource limits, network policy, and restart behavior.

CMA's self-hosted guide rejects File/Git resource mounts and leaves input staging and deliverable retrieval to the operator. Its SDK worker prepares Skills and Memory, not automatic File/Git inputs or output publication. Mango may add those conveniences for an independently selected user workflow, but they are not CMA self-hosted parity requirements. The API therefore rejects File/Git attachments explicitly.

A future operator-managed sandbox service may use the same Work and runner boundary, with a Mango-managed launcher owning provisioning and reclamation. That possibility does not require retaining the current provider registry or adding an unused cloud abstraction now.

The control plane now has one Environment execution path. Additional launchers remain independent follow-up work, not a second runtime architecture.