Build a local knowledge base with AnythingLLM

AnythingLLM is an open-source, all-in-one AI application that lets you chat with documents using Retrieval-Augmented Generation (RAG). It supports multiple LLM providers and vector databases, all running locally on your Olares device.

Learning objectives

In this guide, you will learn how to:

• Install AnythingLLM on Olares.
• Configure the chat model and embedding provider in AnythingLLM.
• Create a workspace and upload documents to build a knowledge base.
• Query your knowledge base using natural language.

Prerequisites

Before you begin, you need:

• An Olares device with sufficient disk space and memory.

• Admin privileges to install apps from Market.

• The following models:

  Model type	Model	How to get it
  Chat	Qwen3.6-27B (llama.cpp)	Install from Market
  Embedding	all-MiniLM-L6-v2	The built-in embedder from AnythingLLM

Optional: Use a different model

You can use a different model size or provider instead of the one listed above:

• Install a different model app from Market.
• Create a model instance from Engine Base apps to bring your own model.

If your AI agent app has the Olares CLI and Agent Skills installed, ask it to deploy the model and skip the manual setup. For example:

Deploy qwen3.5:9b on my Olares using the Ollama Engine Base.

Install AnythingLLM

1. Open Market and search for "AnythingLLM".

   

2. Click Get, then Install, and wait for installation to complete.

Get model connection details

How model connections work

A standalone model on Olares runs as a separate service from the client app. To connect them, the client needs the exact Model name and a Base URL that matches the API format it expects.

You can get both values from the model's console. For more details, see Connect AI apps.

For Qwen3.6-27B (llama.cpp):

1. Open the model app from Launchpad. Its Model Console opens automatically.

2. Wait until Model shows READY and Engine shows RUNNING.

   

3. Under Model, copy the Model name exactly as shown.

4. Under Engine:

   a. Connection source: Select Apps in Olares.

   b. API format: Select OpenAI-Compatible.

   c. Copy the provided Base URL exactly as shown.

Configure AnythingLLM

Configure the chat model and embedding provider. These settings become the system defaults for all workspaces.

Set up the chat model

1. Open AnythingLLM from Launchpad.

2. On the home page, click the Open settings icon in the bottom-left.

3. In the left sidebar, select AI Providers > LLM, and then select Generic OpenAI as the LLM provider.

4. In Base URL, paste the Base URL from the Qwen3.6-27B (llama.cpp) Model Console.

5. In Selected Model, select unsloth/Qwen3.6-27B-GGUF:Q4_K_M.

   

6. Click Save changes. The "LLM preferences saved successfully" message is displayed.

Set up the embedding model

1. In the left sidebar, select Embedder, and then select AnythingLLM Embedder as the embedding provider.

2. In Model Preference, select all-MiniLM-L6-v2 as the embedding model.

   

3. Click Save changes. The "Embedding preferences saved successfully" message is displayed.

Create a workspace

1. Click AnythingLLM in the upper-left corner to return to the home page.

2. Click add_2 next to the search bar.

   

3. In the New Workspace window, name your workspace such as My test, and then click Save.

Upload documents

1. Click upload next to the workspace name to open the document manager.

   

2. Upload your documents by uploading files or by submitting links. The uploaded documents and webpages are displayed in the My Documents panel.

   

3. In the My Documents panel, select the uploaded documents, and then click Move to Workspace to add them to the newly created workspace.

   

4. Click Save and Embed to start embedding.

   This might take a few minutes depending on the number of documents. When the embedding finishes, the "Workspace updated successfully" message is displayed.

Query your knowledge base

Ask questions about your documents.

1. Return to the workspace chat view.

2. Send your question through the chat. For example:

   Olares supports backup or not

3. AnythingLLM retrieves relevant sections from your documents and generates an answer based on the content.

   

Learn more

• Host local models with Engine Base apps
• Official AnythingLLM documentation