AI Extension · Inference

Run a Local LLM

Use local models for cost control, privacy, offline workflows, and model optionality, not just because they feel free.

Run with a harness

Compare Hosted API, Ollama, and vLLM against the same eval cases before judging quality.

Local LLM Experiment Prompt

Copy and use

"I want to experiment with a Local LLM in this project. Create a comparison plan for Ollama and vLLM. [Project] - Current LLM call path: [OpenAI SDK/direct HTTP/other] - Feature to test: [e.g. document QA, summarization, classification] - Local machine/server: [Mac/GPU server/none] - Important criteria: [quality/cost/speed/privacy/offline] [Output requirements] 1) Ollama quick experiment steps 2) vLLM server experiment steps 3) OpenAI-compatible base URL switching 4) eval harness criteria 5) rollback criteria to Hosted API"

Copy this prompt and paste it into your AI chat.

Environment Switch Example

.env.local

OPENAI_BASE_URL=http://localhost:11434/v1
OPENAI_API_KEY=ollama
AI_MODEL=llama3.1

Execution Checklist

0/5 done

Lock test featureLimit to one feature and eval set

Run Ollama smoke testVerify local OpenAI-compatible endpoint

Run vLLM server testRecord GPU server, port, and model name

Switch base URLChange model provider through environment variables

Write rollback criteriaFallback when quality, latency, or error rate misses target

TL;DR

Use this section as a quick baseline for local LLM.
Replace branch names, secrets, URLs, and thresholds with your team-specific values.

Prerequisites

Confirm repository and runtime access permissions.
Verify baseline commands (`install`, `build`, `test`) for local/CI.
Ensure environment variables, secrets, and deployment targets are up to date.

Steps

Lock the change goal and scope in one sentence.
Apply the core local LLM changes in minimal commits.
Capture validation logs and immediately update checklist/docs.

Validation

Confirm build/test pass reproducibly on the current branch.
Run manual sample checks on critical user/operation flows.
Cross-check ko/en docs and links for intent consistency.

Troubleshooting

Capture logs, request payloads, and environment values first on failures.
Narrow down by permissions, env vars, routing, and build artifacts.
Record temporary mitigation and root-cause fix separately.

References

Local LLM

Ollama/vLLM selection criteria

AI Harness

Evaluate model-switching quality

Last updated: June 22, 2026 · Version: v0.0.1

Send Feedback

Opens a new issue page with your message.

Open GitHub Issue