Introduction
When I first explored running local LLMs inside Flutter, I had a lot of questions:
- Why are people using C++ libraries for AI?
- Why not just write everything in Dart?
- What exactly is llama.cpp?
- What are .so, .dll, and .dylib files?
- Why does Flutter suddenly need native code?
If you're coming from a Flutter-only background, all of this initially feels very unfamiliar. Most Flutter developers rarely touch:
- native memory
- C/C++
- shared libraries
- compilers
- ABI architectures
- or systems-level programming
But once you understand the architecture, things become surprisingly logical. This article documents my journey building a proof-of-concept meeting summarizer with local LLM inference.
A Quick Disclaimer
This article is not intended to convince anyone to replace existing Flutter local AI packages.
In fact, if your goal is simply to integrate local AI into an application, a well-maintained package may be the fastest way to ship.
My goal here was different.
I wanted to understand what happens beneath those abstractions, how Flutter communicates with native AI runtimes, and what it takes to build that bridge myself.
That curiosity ultimately led me down the Dart FFI and llama.cpp rabbit hole.
POC support scope: The working LLM path in this repository currently targets Android arm64-v8a. The platform descriptions below explain how FFI libraries generally work; the other native llama bridge integrations like iOS or desktop OS’s are not implemented in this POC.
The Problem Statement
Before diving into implementation details, let's start with a simple question.
Why go through all the complexity of Flutter FFI and llama.cpp when we could simply host a GGUF model on a Python FastAPI server and consume it like any normal API?
That's exactly what I wondered initially.
The answer is that local AI solves a completely different problem.
- Server-based AI depends on internet connectivity, introduces latency, incurs infrastructure costs, and sends user data to external servers
- Local AI via llama.cpp runs the model directly on the device itself, enabling:
- Offline LLM inference after the model has been downloaded
- Better privacy because prompts and model output do not need to leave the device
- Lower latency (no network round trips)
- No backend costs (computation happens on user's device)
For applications like meeting summarization, note taking, personal assistants, or enterprise environments with strict privacy requirements, these benefits can be extremely valuable.
That said, local AI isn't always the right choice.
If your application requires very large models, centralized company knowledge, or continuous model updates, a cloud-based architecture may still make more sense.
The important thing is understanding the trade-offs and choosing the right approach for your use case.
Why Are AI Libraries Usually Written in C++?
As Flutter developers, we're used to solving everything in Dart. So why do most AI runtimes seem to be written in C++, Rust, CUDA, or other low-level technologies?
The answer is performance.
Large Language Models spend most of their time performing:
- Matrix multiplications
- Tensor operations
- Memory-intensive computations
- CPU and GPU optimizations
Languages like Dart are excellent for application development, but they're not designed for squeezing every bit of performance from modern hardware.
C++ gives developers direct control over:
- Memory management
- CPU instructions
- Threading
- SIMD optimizations
- GPU integrations
That's why projects such as TensorFlow, PyTorch, ONNX Runtime, and llama.cpp all rely heavily on native code under the hood. Even many Python AI frameworks are actually thin wrappers around C++ implementations.
What Is llama.cpp?
At a high level, llama.cpp is an inference engine for running Large Language Models locally. Think of it as an AI inference engine optimized for local hardware.
It takes:
- Model weights (the trained parameters from the LLM, in GGUF format)
- Tokens (numerical representations of text)
- Prompts (user input)
And performs all the mathematical operations required to generate text.
Some reasons llama.cpp has become so popular include:
- Runs directly on CPUs
- Supports GPU acceleration
- No Python dependency
- Optimized for mobile and embedded devices
- Supports quantized models
- Cross-platform support
Understanding GGUF Models
When working with llama.cpp, you'll frequently come across GGUF files.
GGUF (GGML Universal File) is a model file format designed to store model tensors and metadata for efficient loading by GGML-based inference engines.
GGUF can store a model’s weights in different levels of precision. Quantization is the process of making those weights smaller by reducing their precision. So, GGUF can store either the original high-precision weights or quantized versions like Q4.
For example, Q4 quantization can reduce a 1B model from roughly 2 GB to roughly 800 MB on disk while maintaining acceptable quality i.e., small enough to be practical on some phones.
Without formats like GGUF, running local LLMs on mobile devices would be far less practical.
One thing I learned quickly is that choosing the right model is often more important than choosing the right framework.
A smaller, well-quantized model that runs smoothly on target devices is usually more valuable than a larger model that struggles with memory and performance constraints.
Existing Flutter Packages for Local AI
Before building my own integration, I explored some of the existing Flutter packages available for running local models.
Some examples include:
- flutter_gemma
- fllama
- llama_cpp_dart
These alternatives can be a great starting point if your goal is simply to get local AI working quickly.
So why didn't I just use one of them?
For this project, my goal wasn't only to run a model locally. I wanted to understand:
- How models are loaded
- How memory is managed
- How Dart communicates with native code
- How token streaming works
- How worker isolates fit into the picture
- What actually happens inside the FFI layer
Building the bridge myself gave me a much deeper understanding of the architecture and helped me appreciate the amount of work these packages abstract away.
If your objective is shipping a feature quickly, evaluating existing packages first is absolutely a sensible approach.
If your objective is learning how local AI actually works inside Flutter, building a small FFI bridge yourself is an incredibly valuable exercise.
Flutter + llama.cpp Architecture
Each layer has a specific responsibility.
- Flutter - Handles UI, state management, and user interactions.
- Provider / Services - Contains application logic and coordinates model operations.
- Worker Isolate - Keeps heavy inference work away from the UI thread.
- Dart FFI - Acts as the bridge between Dart and native code.
- Native Bridge - Exposes a stable C API that Flutter can call.
- Llama.cpp - Loads models and performs inference.
- GGUF Model - Contains the trained weights that generate responses.
Understanding Native Libraries
One thing that confused me initially was seeing files like:
libllama_bridge.so
llama_bride.dll
libllama_bridge.dylib
When we used to install games on Windows, many saw files like: game.dll, audio.dll, etc.
Those .dll files are compiled native libraries. They contain optimized code for:
- Rendering
- Audio processing
- Physics simulation
- Networking
- Gameplay systems
The main game executable loads them and uses their functionality.
Platform-Specific Names
Flutter integrations using llama.cpp can follow the same pattern:
Instead of physics.dll, a bridge might be packaged as:
- Android/Linux: libllama_bridge.so (Shared Object)
- macOS/iOS: Symbols linked directly into the process or a .dylib where platform rules allows it
- Windows: llama_bridge.dll
The native library does the heavy work, and Flutter communicates with it via FFI.
Please note that in our POC we have only covered Android arm64 build configs.
How Flutter Talks to C++: Dart FFI
This is where Dart FFI comes in.
FFI stands for: Foreign Function Interface
Which means: "Allow Dart to call functions written in another language." At the Dart-to-native boundary, FFI can call native C APIs without:
- Flutter Platform channels
- Platform-channel message serialization or
- Message passing
Official Dart FFI Documentation
FFI vs Platform Channels: Why Not Platform Channels?
Flutter already provides Platform Channels, so why introduce FFI?
The answer comes down to performance.
AI workloads involve:
- Frequent native calls
- Large amounts of memory
- Continuous token generation
- Performance-sensitive operations
FFI allows Dart to call native functions directly without serialization overhead. A simple FFI binding looks like this:
DynamicLibrary.open() only loads the library. lookupFunction() resolves an exported C symbol and converts it to a callable Dart function. Only then can Dart invoke the native bridge. The repository collects these typed lookups in _BridgeBindings.
Key Concepts That Finally Made Everything Click
Opaque Pointers
One concept that initially felt strange was opaque pointers.
Instead of exposing the actual C++ model object to Dart, native code simply returns a handle.
Dart doesn't know what's inside the session.
It simply passes the handle back whenever it wants to load a model, generate tokens, or destroy the session.
This keeps the object layout and memory management on the native side. It also creates an explicit ownership contract: every successful bridgeSessionCreate() must eventually be paired with bridgeSessionDestroy(), and pointers returned by allocating native functions must be released with the matching native free function.
Streaming Callbacks
One of the coolest parts of the implementation was token streaming.
Instead of waiting for the entire response, llama.cpp generates tokens one by one.
Every generated token flows back into Flutter through this callback. Pointer.fromFunction() converts a Dart callback into something the native C code can call. The userData pointer tells the callback which token stream to send the generated tokens to, so the tokens can appear one by one in the typing effect.
One important limitation is that the callback must run on the same Dart isolate thread. In this POC, that works because generation and the callback happen on the same worker thread. If the native library calls the callback from a different thread, we should use NativeCallable.listener instead.
Worker Isolates
AI inference is computationally expensive.
Running inference on Flutter's main Isolate would quickly make the UI feel sluggish or frozen. To avoid that, I moved all native interactions into a dedicated Worker Isolate.
The UI remains responsive while the model generates tokens in the background.
The Isolate lets the main Dart thread and the worker communicate by sending messages. The main isolate sends a command to the worker, the worker runs the blocking native/FFI work, and then sends back token, completion, or error messages. FFI talks directly to the native code, avoiding platform-channel overhead, while the Isolate keeps the heavy work away from the UI thread.
The POC I Built - Meeting Summarizer
Architecture
The proof of concept I built follows a fairly simple flow.
The user records a conversation.
The app has two transcription options: recorded audio uses local Whisper, while live speech uses the device’s speech recognition. It prefers offline recognition but falls back to online recognition if offline isn’t supported.
That transcript is then passed to the local LLM through the FFI bridge.
As the model generates a summary, tokens are streamed back to Flutter and displayed in real time.
The LLM inference process happens locally on the device.
No server required for LLMs.
Preview of what I build
Challenges I Faced
While the final architecture looks clean, getting there wasn't always smooth.
Model Downloading and Distribution
One thing that surprised me initially was that llama.cpp doesn't ship models - you download GGUF files, store them, and verify checksums. Model management quickly becomes a product concern rather than just an engineering concern.
This was probably one of the first moments where I realized local AI introduces a completely different set of challenges compared to calling a hosted API.
Memory Consumption
The Llama 3.2 1B Q4_K_M model file used in this POC is roughly 800 MB. Total runtime memory is higher and varies with context size, KV cache, runtime buffers, backend allocations, and device. Bigger isn’t better on a phone. Choosing a model becomes a balancing act between:
- Quality
- Speed
- RAM usage
- Storage size
FFI Debugging Can Be Painful
A normal Dart error usually gives a helpful stack trace.
A native crash often gives something like:
SIGSEGV
Segmentation FaultAnd that's it.
Most of my debugging time was spent tracking:
- Invalid pointers
- Memory ownership issues
- Double frees
- Native lifecycle bugs
FFI is powerful, but it also removes some of the safety nets Flutter developers are used to.
Device Variability
Fast on one phone, slow on another. Testing across multiple devices becomes extremely important when working with local AI.
Local AI Is Not Always Better
This was perhaps the biggest lesson.
It's easy to get excited about running models completely offline.
But local AI isn't a universal replacement for cloud AI.
Cloud-based solutions still have advantages:
- Access to larger models
- Easier updates
- Centralized knowledge
- Less device storage usage
The best solution depends entirely on the problem you're trying to solve.
When Would I Actually Use This?
One question I kept asking myself throughout the project was: "This is cool, but would I use it in a real application?"
I think the answer depends on the use case.
Great Candidates
- Meeting summarization
- Voice notes
- Personal journals
- Offline assistants
- Enterprise privacy-focused applications
- Travel apps with limited connectivity
Probably Better on the Cloud
- Large-scale document analysis
- Complex RAG systems
- Company-wide knowledge assistants
- Very large models requiring significant compute
Local AI is powerful, but it isn't a universal replacement for cloud-hosted AI.
Project Repository
The complete source code for this proof of concept is available on GitHub:
GitHub Repository:
The repository contains:
- Flutter application code
- Dart FFI bindings
- Native C/C++ bridge implementation
- Worker isolate architecture
- Token streaming implementation
- Model loading lifecycle management
- Meeting summarization proof of concept
If you're interested in exploring the implementation details beyond what is covered in this article, feel free to check out the repository.
Key Learnings From This Project
Memory Management Matters
Dart uses garbage collection; native C/C++ memory must be released manually to avoid leaks and crashes.
Whenever native memory is allocated, it must eventually be released. Ignoring this quickly leads to crashes or memory leaks.
Isolates Are Essential
Inference can take several seconds depending on the model and device. Running everything on the UI isolate is not a realistic option.
Streaming Creates Better UX
Showing tokens as they're generated makes the application feel much faster and more responsive.
Native Boundaries Should Be Simple
Keeping a small, stable API between Dart and C++ significantly reduces debugging complexity.
Don't Fight Existing Ecosystems
Building an inference engine from scratch would be an enormous undertaking.
Using llama.cpp allowed me to focus on application development rather than reinventing years of optimization work.
Conclusion
Before starting this project, Flutter FFI and local AI felt intimidating.
There were many unfamiliar concepts, from native libraries and memory management to callbacks and worker isolates.
But once I understood the architecture, it became clear that each layer has a very specific purpose.
The stack I ended up with looks like this:
- Flutter for UI
- Dart for application logic
- FFI for interoperability
- C++ for performance-critical operations
- llama.cpp for inference
- GGUF models for local execution
The result is a working POC on android arm64: speech -> transcript -> streamed local summary, with LLM inference that runs directly on the device without depending on cloud infrastructure.
The learning curve is definitely steeper than consuming a REST API, but the capabilities you unlock - private, offline, low-latency AI running entirely on a user's device - make the journey worthwhile.
Building this POC showed how much Flutter can do when it reaches beyond Dart into native code.
For more on that kind of cross-platform and native integration work, explore our Mobile Engineering practice.
