In Olares 1.12.6, local LLMs are moving to a decoupled engine + instance architecture.
In 1.12.5, you had two options: install a pre-packaged model from the Market, or install the Ollama base engine and pull models manually via the command line. The first limited your options to what was published. The second bypassed the platform—leaving you with no visibility into model metrics, no lifecycle coordination, and no dedicated API endpoints.
Model Console fixes this. You now install a base inference engine once, then deploy and manage your model instances directly within Olares, with full system integration.
A console for every instance
Every deployed model now gets its own entry on your Launchpad. Opening it brings up a dedicated Model Console, giving you direct visibility and control over that specific instance:
- API Endpoints: Ready-to-copy OpenAI and Anthropic compatible routes.
- Status Visibility: Live download progress and service readiness.
- Hardware Metrics: Real-time GPU residency and performance charts.
- Configuration Details: Quick access to your runtime settings.
Bring your own models
Under the hood, a new unified sidecar architecture supports the most common inference engines—Ollama, vLLM, llama.cpp, and SGLang—and handles the heavy lifting. This means you are no longer limited to pre-packaged models. If the weights are out there, you can run them (as long as they fit in your VRAM). You can pull models via:
- Hugging Face (Full repos or single GGUF files)
- Ollama Library
- Direct HTTPS URLs (with resume support)
How it works
1. Install a base LLM engine from the Market.
2. Create a model instance and point it to your preferred source.
3. Access its dedicated console straight from your Launchpad.
4. Copy the API endpoint and plug it into frontends like LobeChat or Open WebUI.
Or, skip the UI entirely: You can ask an Agent to deploy a local LLM for you using the Olares CLI and Agent Skills! See below to get started:
What model would you run?
What open-weight model have you been waiting to self-host on Olares now that you can pull from anywhere? Let us know in the replies below!