Barefoot AI HubTHE BAREFOOT FREELANCER
Start learning
▣ Local AI · Lesson 4 of 4

18Local AI Apps: Open WebUI, LM Studio & Private RAG

Give yourself or a whole office a private ChatGPT-style workspace with document chat — Open WebUI, LM Studio, AnythingLLM and Jan — and deploy it for a client.

⏱ 25 min📶 Intermediate🧪 3-question check

What you'll learn

  • Install Open WebUI on top of Ollama
  • Chat with your own documents privately
  • Compare LM Studio, AnythingLLM, Jan and GPT4All
  • Plan a private AI setup for a small office

Open WebUI: a private ChatGPT for your team

Open WebUI is a self-hosted web interface for Ollama (and any OpenAI-compatible API) with chat history, multiple users, document upload (RAG), model switching, web search and more.

terminalbash
docker run -d -p 3000:8080 \
  --add-host=host.docker.internal:host-gateway \
  -v open-webui:/app/backend/data \
  --name open-webui --restart always \
  ghcr.io/open-webui/open-webui:main

Open http://localhost:3000. The first account you create becomes the admin. Ollama models appear automatically.

terminalbash
pip install open-webui
open-webui serve          # then open http://localhost:8080

Requires a supported Python version (see the Open WebUI docs).

  1. Chat with a model

    Pick a model in the top-left dropdown (e.g. qwen3:8b) and start chatting. Everything stays on your machine.

  2. Chat with a document

    Click + in the message box, upload a PDF or DOCX, and ask about it. Open WebUI chunks and embeds it for retrieval.

  3. Create a knowledge base

    Workspace → Knowledge: upload your SOPs, price lists and FAQs once. Reference it in any chat with #.

  4. Build a custom assistant

    Workspace → Models: pick a base model, add a system prompt and attach a knowledge base — e.g. “HR Policy Assistant”.

  5. Add users

    Admin Panel → Users: invite colleagues and set roles. Each person gets private chats.

Desktop apps: no terminal required

AppBest forHighlights
LM StudioNon-technical users; exploring modelsSearch and download models in-app, shows what fits your hardware, local OpenAI-compatible server
AnythingLLMDocument chat and simple agentsWorkspaces per project, drag-and-drop docs, built-in agent skills, works with Ollama or its own runtime
JanOffline, open-source ChatGPT alternativeClean UI, local models, optional cloud models
GPT4AllOlder or modest laptopsRuns on ordinary CPUs, LocalDocs for private document chat
Open WebUITeams and officesMulti-user, admin controls, knowledge bases, browser-based

Making private document chat accurate

SettingRecommendation
Embedding modelUse a dedicated embedding model (e.g. nomic-embed-text) rather than the default if answers miss obvious passages.
Chunk sizeSmaller chunks (~500 tokens) for FAQs and policies; larger for narrative documents.
Top-KRetrieve 4–8 chunks; too many dilutes the answer.
Chat model contextIncrease context length so retrieved chunks and your question both fit.
Document qualityClean, text-based PDFs work best. Scanned images need OCR first.

Blueprint: private AI for a small office

A service you can sell (Lesson 23): a law office, clinic or accounting firm that can't send documents to cloud AI.

HARDWAREMini PC or Mac mini32–64 GB RAM
RUNTIMEOllamaqwen3:14b + nomic-embed-text
INTERFACEOpen WebUIusers, knowledge bases
ACCESSOffice LAN / Tailscaleno public exposure
CAREMonthly maintenanceupdates, backups, new models

  1. Scope the use cases

    E.g. “summarize case files”, “draft letters from templates”, “Q&A over the office manual”. Collect sample documents.

  2. Size the hardware

    Pick the model that passes your test questions, then choose hardware with headroom (see Lesson 15).

  3. Install & harden

    Ollama + Open WebUI in Docker, auto-start on boot, admin account secured, sign-ups disabled, access only via LAN or Tailscale.

  4. Load knowledge & build assistants

    Create knowledge bases and custom models per workflow with clear system prompts.

  5. Train the team (1–2 hours)

    Prompting basics, what not to rely on, how to verify answers.

  6. Maintain

    Monthly updates, backups of the Open WebUI data volume, model upgrades, usage review.

Key takeaways

  • Open WebUI turns Ollama into a multi-user, private ChatGPT with document chat.
  • LM Studio, AnythingLLM, Jan and GPT4All offer local AI without a terminal.
  • Good private RAG depends on embedding model, chunking, top-K and context size.
  • A private office AI setup is a concrete, sellable service for privacy-sensitive clients.

Knowledge check

0 / 3

Q1Who becomes admin in a fresh Open WebUI install?

Q2Best app for a non-technical person to explore local models?

Q3Private RAG misses obvious answers. First thing to check?

Finished this lesson?Your progress is saved in this browser.