网页
综合
更新时间 2025-07-29 02:36:03
Offline AI on Mobile: Tackling Whisper, LLMs, and Size Roadblocks - DEV Community
自动蜘蛛
dev.to
正文约 3848 字
Posted on Jul 29, 2025
So here’s the thing—I’m not a mobile dev by trade.
I mostly work on backend systems, build APIs, and mess around with servers and machine learning. One of my pet projects has been a web-based, AI-powered speech coaching assistant — it listens to your voice, analyses your pacing and fluency, rephrases hesitant speech into confident phrasing, and even simulates conversations.
The vision? To bring that same assistant to mobile — fully offline, no server required.
No network required. No server to talk to. Just a pocket-sized fluency mentor that could listen to your voice, give you feedback, rephrase your sentences, and even roleplay a conversation—all on-device.
How hard could it be?
Well… let’s talk about that.
I already had a working version on the web: a FastAPI backend running Whisper for transcription and Ollama serving phi-3 for LLM tasks like pacing advice, rephrasing, and generating roleplay responses.
The frontend just sent API calls and got back results. Everything was modular, clean, and fast.
So naturally, I thought, “Let’s just reuse that backend and ship it with the mobile app.”
This was my first dead end.
Turns out, you can’t just run a Python web server inside a mobile app. iOS and Android are not happy to host FastAPI, nor do they allow launching binaries like Ollama from inside the app bundle. Even trying to force it with tools like BeeWare was met with sandbox restrictions, resource constraints, and general disapproval from the mobile OS gods.
Okay, cool. Lesson learned.
With the backend out of the picture, I decided to take a new approach:
This was where things got interesting.
I started with transcription, and honestly—it was a dream.
I used whisper.rn, which wraps whisper.cpp into a clean React Native module. A few lines of code and a bundled tiny Whisper model (ggml-tiny.en.bin), and I was transcribing audio directly on-device.
No cloud calls. No API keys. Just a clean stream of local inference. ✅
Since there was no backend to call anymore, I had to rethink how to express all the logic—fluency checking, pacing suggestions, rephrasing, etc.
Instead of calling POST /fluency, I would now just send this prompt to the LLM:
Same for rephrasing and roleplay:
With Whisper humming along, it was time to add the LLM that would handle rephrasing, analysis, and roleplay. I chose phi-3-mini-4k-instruct, quantized and all.
I copied the model into:
As soon as I tried building the app, Gradle exploded with this error:
Turns out, Android’s build system tries to compress assets during packaging. It attempted to allocate a massive array in memory to compress the model file—and just crashed.
Even if it hadn’t crashed, Google Play has size limits:
I wired up JNI to call llama.cpp from React Native, but the integration just... doesn’t return anything.
My C++ run_prompt() seems to be getting the input—but nothing is returned back to JavaScript. Possibly model path issues, maybe asset handling, maybe JNI bugs. It’s on my list.
To use a .gguf model from native C++, you can’t just fopen() it directly from assets.
Still need to implement that.
This isn’t done—but it’s real.
I’ve got offline Whisper working, I’ve shifted all backend logic into prompt templates, and I’ve scaffolded the native pieces to run an LLM offline.
Next stop? Making the LLM actually respond.
If you're building something similar, don’t be afraid to dig in. Just maybe… don’t start with a 7.1 GB model 😉
Infact a good read and getting to learn more things in this llm space.
DEV Community — A space to discuss and keep up software development and manage your software career
Built on Forem — the open source software that powers DEV and other inclusive communities.
Made with love and Ruby on Rails. DEV Community © 2016 - 2026.
We're a place where coders share, stay up-to-date and grow their careers.
本页为82搜索生成的网页快照,内容来自站内索引;演示数据的原文链接仅作示例。