网页 综合 更新时间 2026-10-24 19:18:41

Local LLM Studio: chat and a speed check that stay on the laptop - DEV Community

自动蜘蛛 dev.to 正文约 6000 字
Local LLM Studio: chat and a speed check that stay on the laptop - DEV Community
Hacktoberfest Open-Source AI Challenge Week 3: Slow and Steady Submission 🦥
This is a submission for the Hacktoberfest Open-Source AI Challenge Week 3: Slow and Steady
I wanted a local model for the boring stuff: drafting a reply I do not want in someone else's log, checking whether a screenshot is readable, and finding out if a 3B model is actually usable on this laptop. What I had instead was three apps, three ports, and a terminal I had to babysit.
Local LLM Studio is the slow version of that workflow. It sits on 127.0.0.1 and talks to whatever open model server is already running.
Local LLM Studio is a small private front end for a model server on your own machine. It speaks the OpenAI-compatible /v1 API, so the same install works with Ollama, LM Studio, and llama.cpp's llama-server.
Four jobs, and then it gets out of the way:
The person who gets to be lazier is anyone who keeps a local model around for private drafts and then loses twenty minutes remembering which port is which. Studio binds to 127.0.0.1:8000. Chats, uploaded images, and benchmark rows stay in a ./data folder next to the app. There is no account and no telemetry.
A loopback server URL is saved immediately. A private-network URL is saved only after you confirm it, because the prompt then leaves the machine. Public cloud hosts, including https://api.openai.com/v1, are rejected.
There is no deployed link, and that is the product. Studio listens only on loopback, and the URL check turns a public host away.
Open http://127.0.0.1:8000. On Windows, localhost often resolves to IPv6 while Studio listens on IPv4, so use the numeric address.
MIT licensed. The repo is tusharv/local-llm.
Local LLM Studio is a private chat, model list, vision check, and speed benchmark for a model server running on your own machine. It speaks the OpenAI-compatible /v1 API, so it works with Ollama, LM Studio, llama.cpp, and any other local server that implements those routes. Ollama is a preset that can also pull a model, delete a model, and read native vision flags.
Studio binds to 127.0.0.1:8000. Chats, images, and benchmark rows stay in ./data. It does not install or start Ollama, LM Studio, or llama.cpp. The first run prints the commands for your operating system. You copy them into a terminal yourself.
The same Studio install is used on macOS, Windows, and Linux.
A plain explanation of why Studio exists, and who it is for, is at https://tusharv.github.io/local-llm/.
You need Python 3.11 or newer and Node.js 20 or newer.
The privacy claim is a URL check in studio/url_policy.py. Loopback is saved immediately. A private LAN address needs an explicit confirm, because the prompt then leaves the machine. A public host is refused and the database is left unchanged.
The open-source AI in this project is the model server, not an agent framework. Studio ships no weights. It assumes you already run an open-weight model through one of three local runtimes:
Suggested Ollama pulls in the UI include Llama 3.2, Qwen 2.5 Coder, and Gemma 3 4B. Those are starting points for an 8 GB or 16 GB machine. Any other tag in the Ollama library works the same way.
The API is Python 3.11, FastAPI, and uvicorn. Chat, images, and benchmark rows go into SQLite under ./data. The UI is React and Vite, built once and served by the same process. python -m studio --dev runs the API and Vite together while the interface is changing.
Ollama gets a second, native client on the same port. Its OpenAI routes hide pull progress, delete, and the vision capability flag, so those calls use Ollama's own HTTP API. For every other server, vision comes from a known family name, a one-shot vision probe, or an override you set.
Benchmarks are a fixed suite, version 2, with three profiles: a general mix, code generation, and one image-generation prompt. Generated code is shown as text and never executed. Image URLs are never fetched. Repeats are 1 or 3. Reports show median response time and effective throughput: output tokens divided by the full request duration, including the wait for the first token.
Two write-ups shaped those choices. Theo Slater built a desktop chat app that talks only to Ollama, and wrote that an OpenAI-compatible API should have been there from the first day: I Built a Desktop Chat App for Running Local LLMs Offline. Studio starts from that API, so LM Studio and llama.cpp are a URL change. Sam Hartley is blunt that local models still lose on long reasoning, stale knowledge, and finicky vision: I Replaced All My Cloud AI With Local Models — Here's What Actually Broke. The benchmark page exists so that failure shows up on your own hardware.
A closed chat API can answer the prompt. It cannot be the thing this project is for.
The weights are open, so Llama, Qwen, and Gemma can be swapped with no new vendor SDK and no per-token bill. The servers are open, so the same client reaches three runtimes. Inference stays on the machine, so a transcript about a half-finished idea stays in ./data.
A hosted API would also make the speed number meaningless. Tokens per second is a memory bandwidth number, as @lowspecai put it: each token reads the weights, so the ceiling is how fast your RAM or VRAM can be read. A leaderboard from someone else's GPU does not say whether llama3.2:3b is pleasant on this laptop. The same prompt, run locally with the warmup thrown away, does.
Open weights are also why the lazy version can be picky. If a 3B model rambles, or a vision model calls a screenshot a flowchart, delete it and pull another one.
The model does not do the job. It removes the errand around the model.
Before Studio, trying one local model meant remembering the port, remembering whether this runtime wants ollama run or a loaded model in a Developer tab, pasting the prompt, then doing it again for the next model because nothing kept the transcript. Checking vision meant attaching an image and reading a confused paragraph. Checking speed meant watching the terminal and guessing.
With Studio, th
访问原文 返回搜索 https://dev.to/tusharv/local-llm-studio-chat-and-a-speed-check-that-stay-on-the-laptop-5a5
本页为82搜索生成的网页快照,内容来自站内索引;演示数据的原文链接仅作示例。