Self-hosted model operations
Workstation Model Service Runbook
Build and operate a bounded local model service on one workstation without confusing locality with security, privacy, or production readiness.
- Format
- Playbook
- Level
- Intermediate
- Audience
- Practitioner, Developer, Operator
- Owner
- Project42 Editorial
- Review cadence
- Every 60 days
- Prerequisites
- An inventory of CPU, accelerator, memory, storage, operating system, and power constraints; An approved immutable model artifact with license and provenance review; A local-user, data, network, and retention boundary
Bound the local service before loading weights
Name the intended user, workload, data classes, prohibited inputs, network exposure, context and output ceilings, expected concurrency, and whether the service may invoke tools. Local execution removes neither the need for authorization nor the risk of prompts, outputs, caches, logs, model files, or tool effects reaching people and systems that should not receive them.
Inventory the exact device and runtime path: operating-system build, drivers, execution provider, runtime or server version, model revision and digest, quantization or conversion, tokenizer and chat template, gateway or client adapter, and configuration. Record model-card limitations and license conditions. Reject mutable aliases and unexplained converted artifacts.
Qualify performance, isolation, and restart behavior
Measure model load time, first-output and completion latency distributions, memory and accelerator use, sustained temperature or throttling, disk growth, concurrency, cancellation, malformed requests, and out-of-memory behavior with the real request contract. Reserve capacity for the operating system and other required work; a model that fits once can still destabilize the device under long context, parallel requests, or swap pressure.
Bind the endpoint only where intended, require caller authorization when more than one trust zone can reach it, keep secrets outside prompts and repositories, minimize retained content, and verify that sleep, reboot, runtime failure, and model reload do not silently change identity or policy. Production claims require a supported owner, monitored service objectives, backups, patching, and tested recovery beyond one developer session.
Task: [LOCAL WORKLOAD AND USER OUTCOME]
Scope: [DEVICE, USERS, DATA, NETWORK EXPOSURE, TOOLS, AND RETENTION]
Permissions: [MODEL/LICENSE, DEVICE ADMIN, DATA, ENDPOINT, AND RELEASE AUTHORITY]
Exact build: [OS, DRIVER, EXECUTION PROVIDER, RUNTIME, MODEL REVISION/DIGEST, QUANTIZATION, TEMPLATE, AND CONFIG]
Capacity limits: [CONTEXT, OUTPUT, CONCURRENCY, MEMORY, STORAGE, TEMPERATURE, AND DEADLINE]
Data controls: [INPUT/OUTPUT/CACHE/LOG ALLOWLIST, RETENTION, ACCESS, AND DELETION]
Verification: [IDENTITY, QUALITY/SAFETY, LATENCY DISTRIBUTION, SATURATION, REBOOT, AND NEGATIVE ACCESS TESTS]
Stop conditions: [UNVERIFIED ARTIFACT, UNAUTHORIZED LISTENER, DATA LEAK, THROTTLING, OOM, CRITICAL EVAL FAILURE, OR IDENTITY DRIFT]
Recovery: [STOP LISTENER, CANCEL/RECONCILE WORK, RESTORE PINNED BUILD AND POLICY, VERIFY BEFORE REOPENING]Expected evidence and verification
Expected evidence includes device and driver inventory, artifact license and digest, runtime compatibility, endpoint and access boundary, representative evaluation results, load and thermal observations, capacity limits, data-handling controls, reboot test, known-good manifest, owner, and review date. Keep measurements dated because drivers, runtimes, model revisions, and device conditions change.
Verify the identity returned by the service rather than trusting a file name. Stop admission on artifact mismatch, unauthorized exposure, protected-data leakage, repeated out-of-memory or thermal failure, critical evaluation failure, or an unrecoverable restart. Close the listener, preserve minimal evidence, reconcile unfinished requests or tool effects, restore the pinned build, and repeat identity, access, evaluation, and load checks before use.