OpenJCockpit

Configuration-driven agents + A2A — operations guide (SPEC-002)

This guide is self-contained: a developer can reproduce the local demo with it without any prior context. Progress and evidence per step: IMPLEMENTATION_PROGRESS.md.

Since the management/execution split the configuration-driven agents, the inbound A2A endpoints, the openjcockpit.agent-definitions.* settings, the LLM routes and agent-definitions/ run in embabel-agent-runtime (services/embabel-agent-runtime, port 8092 locally), no longer in embabel-agent-service. Wherever embabel-agent-service appears below as the A2A server or agent runtime, read embabel-agent-runtime. Tenant bindings (agent_tenant_bindings), workflow runs and OPA decisions remain owned by embabel-agent-service; the runtime reads them via /internal/agent-runtime/v1/**. See docs/readme/concepts/agent-runtime.md.

1. What it does

A2A client ──► LiteLLM A2A Gateway (/a2a/customer-support)
                 │  x-litellm-api-key: Bearer <virtual key>
                 │  x-a2a-customer-support-authorization: Bearer <Keycloak JWT>  → forwarded as `authorization`
                 ▼
embabel-agent-service  POST /a2a/{agentId}  (A2A 0.3, JSON-RPC, message/send only)
  ├─ JWT validation (existing resource server) → tenant from claim `tenant-claim`
  ├─ AgentDefinitionResolver (1× per run)
  │    PostgreSQL agent_tenant_bindings (tenant ↔ agent, tool allowlist, mandatory guardrails, pins, fallback permission)
  │    source: REMOTE (LiteLLM, UNSUPPORTED in v1.101.2) → LAST_KNOWN_GOOD → LOCAL agents/<id>.md
  │    skills (skills/<id>/SKILL.md), tools (server registry ∩ allowlist), guardrails (executor required), model alias
  │    ⇒ immutable ResolvedAgentDefinition + PreparedAgent
  ├─ ConfiguredDefinitionAgent (Embabel @Agent/@Action)
  │    ai.withLlmByRole(<model alias>) + AgentPromptContributor + native Embabel Skills (materialised per run)
  │    + SnapshotBoundTool (call-time snapshot and OPA check) + Embabel GuardRails + strict ToolNotFoundPolicy
  └─ LLM call via LiteLLM model gateway (model = alias, e.g. customer-chat; policy openjcockpit-agent-pii)

Important: for its model calls the Embabel service calls the LiteLLM model route (/v1/chat/completions), never its own a2a/<agent> route — there is no loop.

2. Configuration (openjcockpit.agent-definitions.*)

Property Env var Default Meaning
enabled AGENT_DEFINITIONS_ENABLED false Whole feature on/off. Off = no bean active at all.
local-root AGENT_DEFINITIONS_LOCAL_ROOT – Directory containing agents/ and skills/. Required when enabled.
tenant-claim AGENT_DEFINITIONS_TENANT_CLAIM tenant_id JWT claim holding the tenant id. Missing → refused.
source-priority – REMOTE,LAST_KNOWN_GOOD,LOCAL Order of sources.
mandatory-guardrails – [] Guardrails every agent must have, regardless of its definition.
fallback.on-unavailable AGENT_DEFINITIONS_FALLBACK_ON_UNAVAILABLE true Fallback on UNAVAILABLE (plus per-agent permission).
fallback.on-not-found AGENT_DEFINITIONS_FALLBACK_ON_NOT_FOUND false Fallback on NOT_FOUND (plus per-agent fallback_on_not_found).
cache.ttl AGENT_DEFINITIONS_CACHE_TTL 1h Validity of a last-known-good snapshot.
cache.directory AGENT_DEFINITIONS_CACHE_DIR – Directory for snapshots; empty = no last-known-good source.
cache.max-entries – 256 Maximum number of snapshots.
litellm.enabled AGENT_DEFINITIONS_LITELLM_ENABLED false LiteLLM as metadata source + gateway guardrail verification.
litellm.base-url AGENT_DEFINITIONS_LITELLM_BASE_URL http://localhost:4000 Proxy root (without /v1).
litellm.api-key LITELLM_AGENT_REGISTRY_KEY – Key for GET /v1/agents and POST /policies/resolve. Never in YAML.
litellm.connect-timeout / read-timeout …_CONNECT_TIMEOUT / …_READ_TIMEOUT 3s / 10s  
litellm.gateway-guardrails.<id> – – Guardrail id → LiteLLM guardrail name; only active after confirmation via /policies/resolve.
litellm.gateway-model-aliases – [] Model aliases routed as their own LiteLLM model (one Embabel LLM service per alias).
litellm.model-gateway-url / -key LITELLM_BASE_URL / LITELLM_VIRTUAL_KEY   OpenAI-compatible gateway for those aliases.
a2a.enabled A2A_ENABLED false Inbound A2A endpoint + Agent Card. Requires enabled.
a2a.public-base-url A2A_PUBLIC_BASE_URL – URL that appears in the Agent Card.

Every model alias in a definition must be an Embabel role: embabel.models.llms.<alias>=<alias> (aliases containing a hyphen cannot be set as an env var — use a profile/extra YAML, see §9).

3. Local directories

agent-definitions/
  agents/customer-support.md          # YAML frontmatter (id, version, description, model, skills,
                                      # optionalSkills, tools, guardrails) + Markdown instructions
  skills/customer-lookup/SKILL.md     # Agent Skills format: name (= directory name), description, …
  skills/customer-lookup/references/  # text resources (.md/.txt/.json/.yaml/.yml/.csv)

Rules: ids are kebab-case (max 64); unknown or duplicate frontmatter keys, symlinks, .., files > 256 KB and skill packages > 1 MB are refused. scripts/ is never read or executed. allowed-tools in a skill and text in instructions never grant tools.

4. PostgreSQL mapping

Migration src/main/resources/db/migration/V3__create_agent_tenant_bindings.sql was approved (2026-10-14) and runs with the regular Flyway migrations on every database. Local seed (docker-compose only, via SPRING_FLYWAY_LOCATIONS=classpath:db/migration,classpath:db/seed-local): db/seed-local/R__seed_local_agent_bindings.sql binds the 4 seeded agents to tenant local (model agent-chat, guardrail pii-protection, no tools); insert-if-missing, changes are preserved. Note: once the seed has been applied, Flyway expects the location db/seed-local on every start of that database (otherwise validation fails on a “missing” repeatable migration).

Table Key Content
agent_tenant_bindings (tenant_id, agent_id) enabled, model_alias (pin), pinned_version, fallback_allowed, fallback_on_not_found
agent_tenant_binding_tools (tenant_id, agent_id, tool_id) tool allowlist
agent_tenant_binding_guardrails (tenant_id, agent_id, guardrail_id) additional mandatory guardrails

Example: docs/agent-integration/demo-seed.sql.

5. Fallback matrix

Remote status fallback_allowed Config Result
FOUND – – remote (and stored as last-known-good)
UNSUPPORTED – – next source (the source does not exist)
UNAVAILABLE yes on-unavailable=true last-known-good → local
UNAVAILABLE no, or config off – error UNAVAILABLE
NOT_FOUND yes + fallback_on_not_found on-not-found=true last-known-good → local
NOT_FOUND otherwise – error NOT_FOUND
UNAUTHORIZED / INVALID – – always an error, never a fallback

Last-known-good: expired → skipped + deleted; corrupt/hash mismatch → skipped + deleted. Every selected definition goes through the same validation (binding, tools, guardrails, model) against the current state; a snapshot therefore cannot bring back revoked permissions. Local definitions are a fallback for configuration, not for an unreachable model gateway.

6. Guardrails

Id Executor Coverage Fail-closed
pii-protection PiiProtectionGuardrail (Embabel UserInputGuardRail + AssistantMessageGuardRail, CRITICAL) LLM input and output of the run yes: Embabel throws GuardRailViolationException → A2A error GUARDRAIL_VIOLATION
<gateway-id> (config) LiteLlmPolicyGuardrailExecutor LLM calls to the model alias via LiteLLM yes: only operational when POST /policies/resolve {"model": alias} lists the LiteLLM guardrail in effective_guardrails
tool access ToolAccessPolicy (bind time) + SnapshotBoundTool → OpaToolCallAuthorizer (call time) every tool call yes; OPA off = allowlist only

A guardrail id without an operational executor blocks the run (GUARDRAIL_MISSING). A policy ID that appears only in a prompt or YAML without an executor does not count. LiteLLM side (infrastructure/litellm/config.yaml): guardrail openjcockpit-pii (litellm_content_filter, nl_bsn_contextual/iban/credit_card, BLOCK, pre+post), policy openjcockpit-agent-pii attached to model customer-chat; for A2A traffic via litellm_params.guardrails in the agent registration.

7. A2A registration with LiteLLM

Files: infrastructure/litellm/a2a/customer-support.agent.yaml (config) and .json (API payload). Pinned protocol version: A2A 0.3 (protocolVersion: "0.3.0", transport JSONRPC). Supported: message/send (synchronous, result = Message). Not supported (explicit error): message/stream (-32004), tasks/* (-32001), push notifications (-32004).

Options (both require approval for a shared/live instance):

  1. Merge the agents: block into infrastructure/litellm/config.yaml and restart LiteLLM.
  2. curl -X POST $LITELLM_URL/v1/agents -H "Authorization: Bearer $LITELLM_MASTER_KEY" -H 'Content-Type: application/json' -d @infrastructure/litellm/a2a/customer-support.agent.json

LiteLLM requirements: v1.101.2 (pinned in docker-compose.yml), a2a-sdk present in the image, and the client key must have access to the agent. The caller sends its Keycloak JWT as x-a2a-customer-support-authorization: Bearer <jwt>; LiteLLM forwards it as authorization.

8. Health checks

9. Local stack: docker compose up --build

One command, everything seeded locally:

docker compose up -d --build
./scripts/a2a-roundtrip.sh        # calls the 4 seeded agents via the LiteLLM A2A Gateway (:4100)

One seed service local-seed (infrastructure/local-seed/seed-local.sh; runs on every up, idempotent, no compose --profile needed):

Part Script Result
Keycloak A2A client keycloak-a2a-client.sh confidential client openjcockpit-a2a (client_credentials; token contains tenant_id=local), secret A2A_CLIENT_SECRET
LiteLLM model key litellm-keys.sh embabel-agent-runtime (LITELLM_VIRTUAL_KEY), incl. agent-chat, customer-chat
LiteLLM MCP-only key litellm-keys.sh embabel-agent-service-mcp (LITELLM_AGENT_SERVICE_KEY): git MCP server only, for checkpoint commits
The LiteLLM A2A key litellm-keys.sh openjcockpit-a2a (LITELLM_A2A_KEY): allowed_routes: ["/v1/agents","agent_inference_routes"], grants for the 4 agents
Cleanup litellm-keys.sh deletes keys from earlier setups (embabel-agent-catalog, openjcockpit-a2a-client, embabel-agent-service)

One Spring profile local (application-local.yml, activated by compose with SPRING_PROFILES_ACTIVE=local): Flyway incl. db/seed-local (tenant local bindings), agent definitions + A2A on, model aliases, the agent catalogue and the A2A provider. Compose only passes container paths and secrets.

One A2A client per A2A provider. openjcockpit.a2a-providers (a list; locally one entry type: litellm) → A2AProviderClients creates exactly one A2AProviderClient per provider (LiteLlmA2AProviderClient): listAgents() (catalogue), agentCard(name) (Details), sendMessage(name, text, callerAuthorization) (A2A message/send via the gateway; the caller JWT is passed along as x-a2a-<agent>-authorization). The same client provides the LiteLLM connection for gateway guardrail checks. There are no other LiteLLM/A2A HTTP clients any more; the separate agent-catalog.litellm and agent-definitions.litellm connections have been removed (agent-definitions.litellm now only contains model gateway settings).

Calling it manually: get a token via curl -d grant_type=client_credentials -d client_id=openjcockpit-a2a -d client_secret=openjcockpit-a2a-local-secret http://localhost:8080/realms/openjcockpit/protocol/openid-connect/token and POST http://localhost:4100/a2a/<agent> with x-litellm-api-key: Bearer sk-openjcockpit-a2a-local and x-a2a-<agent>-authorization: Bearer <token>.

9a. Agent catalogue in the dashboard (Workflow design → Agents)

GET /api/agent-catalog (embabel-agent-service, relayed by ai-control-service) returns:

In the LiteLLM registration (config.yaml) the url deliberately stays the Docker-internal address http://embabel-agent-runtime:8082/a2a/<id> (that is where LiteLLM forwards to). Config agents are read from config.yaml when LiteLLM starts; after changes docker compose restart litellm is enough.

Managing A2A services (Workflow designer → Configuration)

The A2A services block shows all providers with live status and number of agents (GET /api/a2a-services):

9b. Workflows consist of references to catalogue agents

A workflow only refers to agents from the central agent catalogue (Workflow design → Agents); the built-in pipeline (requirement … evidence, realisation, spec publication) has been removed. The workflow is the only execution engine: every run starts via a workflow (POST /api/workflows/{id}/start) or a git-workspace job that starts a workflow.

10. Test commands

# everything (from the repo root)
mvn -o -pl services/embabel-agent-service,services/embabel-agent-runtime -am test
# SPEC-002 only (the runtime now owns definitions and inbound A2A)
mvn -o -pl services/embabel-agent-runtime -am test -Dsurefire.failIfNoSpecifiedTests=false \
  -Dtest='nl.metafactory.agentruntime.definition.**.*Test,nl.metafactory.agentruntime.a2a.*Test'
# live roundtrip (requires a running stack, see §9)
LITELLM_KEY=… A2A_JWT=… ./scripts/a2a-roundtrip.sh

11. Limitations