# model-runtime-agent

**Repository Path**: cloudshift/model-runtime-agent

## Basic Information

- **Project Name**: model-runtime-agent
- **Description**: No description available
- **Primary Language**: Unknown
- **License**: Not specified
- **Default Branch**: master
- **Homepage**: None
- **GVP Project**: No

## Statistics

- **Stars**: 0
- **Forks**: 0
- **Created**: 2026-08-02
- **Last Updated**: 2026-08-02

## Categories & Tags

**Categories**: Uncategorized

**Tags**: None

## README

# Model Runtime Agent

`model-runtime-agent` is a node-side control-plane agent for model workloads. It reports runtime inventory and consumes terminal grants; it does not execute ASR, LLM, or other model inference itself.

## Current Control Plane Compatibility

The current Model Hub API remains ASR-namespaced while the protocol is generalized. Configure the Agent with the existing endpoint:

```sh
MODEL_RUNTIME_CONTROL_PLANE_URL=http://127.0.0.1:3000/api/internal/asr-runtime
MODEL_RUNTIME_AGENT_TOKEN=replace-with-agent-token
MODEL_RUNTIME_AGENT_ID=model-runtime-local-01
MODEL_RUNTIME_ENVIRONMENT_ID=local
INVENTORY_SOURCE=file
MODEL_RUNTIME_INVENTORY_FILE=/etc/model-runtime-agent/inventory.json
CONFIG_POLL_INTERVAL=30s
MODEL_RUNTIME_HEARTBEAT_INTERVAL=15s
```

Run it with:

```sh
go run ./cmd/model-runtime-agent
```

## Docker

Build the Agent image from this directory:

```sh
docker build -t registry.example.com/model-runtime-agent:0.1.0 .
docker push registry.example.com/model-runtime-agent:0.1.0
```

Run it on the same Docker network as Model Hub. Mount the inventory file read-only and point the control-plane URL at the Model Hub service name, not `127.0.0.1`:

```sh
docker run -d --name model-runtime-agent \
  --restart unless-stopped \
  --network modelhub_default \
  --env-file .env \
  -e MODEL_RUNTIME_CONTROL_PLANE_URL=http://modelhub:3000/api/internal/asr-runtime \
  -e MODEL_RUNTIME_INVENTORY_FILE=/etc/model-runtime-agent/inventory.json \
  -v "$PWD/inventory.local.json:/etc/model-runtime-agent/inventory.json:ro" \
  registry.example.com/model-runtime-agent:0.1.0
```

## ASR Executor Docker

The ASR Executor image is built from the sibling `model-asr-executor` repository. Its production target includes `/app/models`; use the code-validation target only for CI checks.

```sh
cd ../model-asr-executor
docker build --target production \
  -t registry.example.com/asr-executor:0.1.0-cuda12 .
docker push registry.example.com/asr-executor:0.1.0-cuda12
```

For Runtime mode, run the image with GPU access and these required values. `ASR_EXECUTOR_TOKEN` is the executor token expected by the Runtime client.

```sh
docker run -d --name asr-executor \
  --restart unless-stopped \
  --gpus all \
  --network modelhub_default \
  -e ASR_EXECUTION_MODE=runtime \
  -e ASR_EXECUTOR_ID=asr-docker-01 \
  -e ASR_COMPUTE_POOL_ID=asr-local \
  -e ASR_CONTROL_PLANE_URL=http://modelhub:3000/api/internal/asr-runtime \
  -e ASR_EXECUTOR_TOKEN=replace-with-executor-token \
  -e ASR_NATS_URL=nats://modelhub-asr-nats:4222 \
  -e ASR_NATS_SUBJECT=asr.runtime \
  -e ASR_MODEL_ID=paraformer-large-zh \
  -e ASR_MODEL_VERSION=1 \
  registry.example.com/asr-executor:0.1.0-cuda12
```

Replace the network, service hostnames, image registry, credentials, and model values for the target environment. The Executor needs a Docker host with NVIDIA Container Toolkit installed.

## Inventory Sources

`MODEL_RUNTIME_INVENTORY_SOURCE` defaults to `file`, which reads the JSON array at `MODEL_RUNTIME_INVENTORY_FILE`. Each instance requires `instance_id`, `compute_pool_id`, `runtime_type`, `resource_version`, and `state`.

Set `MODEL_RUNTIME_INVENTORY_SOURCE=control_plane` to poll `GET /v1/agents/{agent_id}/inventory-config` with the configured agent bearer token. `MODEL_RUNTIME_CONFIG_POLL_INTERVAL` defaults to `30s`. The endpoint returns the desired inventory inside the Control Plane response envelope:

```json
{
  "data": {
    "desired_inventory": [
    {
      "instance_id": "pod-1",
      "compute_pool_id": "gpu-a",
      "runtime_type": "kubernetes",
      "resource_version": "1",
      "state": "ready"
    }
    ]
  }
}
```

The Agent maps `desired_inventory` into heartbeat instances. It keeps reporting the last successfully decoded configuration if a later poll fails. This only controls reported inventory; it does not create, stop, or otherwise manage Docker or Kubernetes workloads.

## Terminal Scope

The client supports one-time Terminal Grant consumption and control-plane session auditing. Interactive terminal I/O is intentionally not implemented until the control plane exposes an authenticated mTLS stream contract; this Agent never opens Kubernetes, Docker, or SSH access based on browser input.