Host local large language models with Engine Base apps

Olares v1.12.6 introduces Model Console, a platform that manages the full lifecycle of local large language models (LLMs). This platform provides four Engine Base applications, each built on a different inference engine: Ollama Engine Base, vLLM Engine Base, llama.cpp Engine Base, and SGLang Engine Base.

Choose the base app for the engine you want, clone it to deploy a model, then run and manage that model from its dedicated console.

Before you start

• Your Olares system has been upgraded to v1.12.6 or later.

• If you want to skip manual configuration and quickly try out a model, install a pre-built model app from Market instead. These apps package the recommended model-and-engine combinations validated for Olares.

  Check available pre-built model apps

  • Qwen3.6-27B (llama.cpp)
  • Qwen3.6-35B-A3B (llama.cpp)
  • Qwen3.6-27B MTP (llama.cpp)
  • Qwopus3.6-27B MTP (llama.cpp)
  • Gemma-4-12B (vLLM)
  • Qwen3.5-9B (SGLang)
  • Gemma 4 26B (Ollama)
  • Ornith 35B (Ollama)

Locate Engine Base apps

1. Open Market and search for "Engine Base".

   Four engine base apps appear: vLLM Engine Base, SGLang Engine Base, Ollama Engine Base, and llama.cpp Engine Base.

   

2. Choose the engine base that fits your needs. Each one is optimized for a different inference scenario:

   Engine Base	When to choose
   llama.cpp Engine Base	Choose llama.cpp when you are running lightweight GGUF models or deploying with limited GPU memory. It is the recommended engine on Olares One.
   Ollama Engine Base	Choose Ollama when you want to get started quickly with broad model compatibility. It pulls models automatically using native model tags, making it ideal for chat and embedding tasks.
   SGLang Engine Base	Choose SGLang when you need efficient structured generation or advanced reasoning optimizations.
   vLLM Engine Base	Choose vLLM when you need high-throughput serving of Hugging Face models under heavy concurrent load.

Create a new model instance

An Engine Base app serves as a template. To run a model, you must first clone the base app into an independent running instance.

1. Select the base app that matches your preferred inference engine, and then click View on it. For example, llama.cpp Engine Base.

2. Click Create to initialize a new instance.

   

3. Select the hardware accelerator for the instance to run on, and then click Confirm.

4. Specify the instance identity settings:

   • New app name: Enter a unique name for the instance. This name is displayed as the app name in Market and Settings. For example, Qwen3.6-35B-A3B.
   • Shortcut name for [client]: Enter a unique shortcut name for the instance. This name is displayed on the Launchpad. For example, qwen3.6-35b-a3b.

5. Click Create to proceed to the environment configuration.

Configure engine environment variables

After creating the instance, the configuration window opens. Define where your engine pulls the model, how much memory it uses, and what capabilities it exposes to other client apps.

1. In the Configure environment variables for [New-app-name] window, fill in the following details according to the target model and engine.

2. Click Confirm to save the configuration and start the instance installation.

   An Instances panel appears on the right side of the page, showing the installation progress. Once the setup completes, the instance's operation button changes to Open, indicating that the base service is running. A model app with the same name also appears on the Launchpad.

   

   INFO

   Model instances created from Engine Base apps show a From template tag next to the app name. You can see this tag when viewing the app in Market or Settings.

   

Update environment variables later

To change these variables after installation, go to Olares Settings > Applications > [App-Name] > Manage environment variables. Click the edit icon next to a variable, update its value, save your change, and then click Apply.

Monitor deployment and configure the model service

Open the built-in model console to track the model download, confirm the model and engine readiness, configure client access, and inspect GPU usage and performance.

1. Locate the model instance in the Instances panel on the Engine Base app details page, or find it on the Launchpad.

2. Open it to launch the dedicated model console.

   The console opens on the Status tab by default, and the model files start downloading automatically.

3. Under Service status, track the readiness of the model and the engine:

   • Model: Shows Ready after the files are downloaded and verified.
   • Engine: Shows Running after the inference service is online.

   

4. When the engine shows Running, configure how client apps reach the service.

   • Connection source: Select where the client runs.
     • Apps in Olares: For apps running in Olares.
     • Devices on your network: For devices on the same local network.
     • Remote: For access over the public internet, which requires you to enable the VPN in LarePass first.
   • API format: Select the API style that matches your client: Ollama, OpenAI-Compatible, or Anthropic-Compatible.
   • Base URL: Copy the URL to use for client app connections.
   • Supported endpoints: Expand this list to see every endpoint the selected API format exposes, with its HTTP method, path, and purpose.

5. Select the Configuration tab to review the model's details:

   

   • Model: Shows the model name, mode, and the capability tags.
   • Parameters: View the engine parameters. Expand Advanced parameters for the full set, and switch the view between Form and Raw.

6. In the GPU residency section, click Detect, and then:

   • Check the mode: Confirm where the model is running.
     • Full GPU: The entire model runs on the GPU. This is the fastest state, and is expected when you selected the GPU accelerator during installation.
     • CPU or Split: Part or all of the model runs on the CPU, which makes inference slower.
       • If you chose the CPU accelerator during installation, CPU is expected.
       • If you chose the GPU accelerator, review your [ENGINE]_REQUIRED_GPU_MEMORY setting and engine arguments.
   • Check the memory usage: Review the VRAM, KV cache used, and GPU memory utilization to see how much memory the model occupies and how much memory is left for longer contexts or more concurrent requests.

7. In the Performance section, click Run test to measure two response-speed metrics.

   Use them to compare quantization levels, context sizes, or engine arguments, and to verify that a change actually improved speed before you use it:

   • TTFT (Time To First Token): How long you wait before the first word appears. A lower value means the model responds faster.
   • Cold start: How long the engine takes to load the model from scratch, for example after a restart. A lower value means the model is ready to serve sooner.

Connect client apps to the model service

Once the model instance is running, any client app that speaks the OpenAI-compatible API can connect to it through the base URL.

The following example uses OpenCode as the client.

1. In the model console, go to the Status tab. Under Service status:

   • Connection source: Select Apps in Olares, because OpenCode runs in Olares.
   • API format: Select OpenAI-Compatible.
   • Copy the Base URL and note down the Model name.

2. In OpenCode, click settings in the bottom-left corner, select Providers, then scroll down and select Connect next to Custom Provider.

3. Enter the following details:

   • Provider ID: A unique identifier for this provider. For example, olares-llm.
   • Display name: The name shown in the provider list. For example, Olares LLM.
   • Base URL: The Base URL you copied from the model console.
   • Models:
     • Model ID: Your MODEL_NAME. For example, Qwen3.6-35B-A3B.
     • Display Name: The name shown for this model. For example, Qwen3.6 35B A3B.

4. Click Submit to save the configuration. The provider appears in the provider list.

5. Run a task to test the connection. This example uses the Olares skills to deploy an app to Olares.

   a. At the top, click the Search field and select Toggle terminal to open a terminal.

   b. Log in to the Olares CLI to use the built-in Olares skills. Replace alice123@olares.com with your own Olares ID.

   olares-cli profile login --olares-id alice123@olares.com

   c. When prompted, type your Olares password and press Enter. The input stays hidden.

   d. If two-factor authentication is enabled on your Olares, the CLI prompts you for a two-factor code for the Olares ID. Enter the 6-digit code from LarePass and press Enter.

   e. Below the chat box, select Big Pickle to open the model selector, and select Qwen3.6 35B A3B from the list.

   f. Send a task. The example below uses dockersamples/101-tutorial, a beginner-friendly Docker tutorial web app.

   Deploy this app to Olares: https://github.com/dockersamples/101-tutorial

   g. Follow any prompts that appear until the deployment finishes. You can then find the app on the Launchpad and in My Olares.

   

References

Engine tuning arguments

Use the ENGINE_ARGS variable to add custom settings that adjust memory usage, context limits, and processing behaviors. Separate multiple arguments with spaces. Select your inference engine below to view some commonly used tuning arguments.

Recommended models and parameters

The following per-engine recommendations are validated best practices. Use them as a starting point, then adjust for your hardware.