Setting up Local AI - private, offline, free

Running AI locally means everything stays on your computer. Your resume, your job descriptions, your career data - none of it goes online. It works without internet and costs nothing to run.

If that sounds complicated, don't worry. Think of it like installing any other app. We'll walk you through the whole thing.

Is my computer ready?

Before we dive in, let's make sure your computer can handle it. Local AI needs a bit more memory than a regular app.

A simple friendly checklist:

• ✓ 16GB of RAM or more
• ✓ About 5GB of free storage space
• ✓ macOS, Windows, or Linux

Not sure about your computer's specs? Visit canirun.ai - it's a free tool that checks your computer and tells you exactly which AI models you can run. Takes about 30 seconds.

A note for lower-spec machines: If your computer has less than 16GB of RAM, local AI will feel slow and frustrating. The good news: Groq's free API gives you the same privacy-first experience without any hardware requirements. Set up Groq instead →

Apple Silicon Macs (M1, M2, M3): You're in great shape. Apple Silicon handles local AI better than almost any other consumer hardware. If you have 16GB, you'll have a smooth experience.

Step 1: Install Ollama

Ollama is the free app that lets your computer run AI models. Think of it like a player app - Ollama is the player, and the AI model is the content.

Download Ollama at ollama.com. Install it like any other app. Once it's running, you'll see a small icon in your menu bar (Mac) or system tray (Windows).

Step 2: Download an AI model

Now you need to give Ollama an AI model to run. We recommend starting with Llama 3.1 8B - it's the best balance of quality and performance for Quillon's features.

The Easy Way (if you're comfortable with a terminal): Open your terminal and run:

ollama pull llama3.1

This downloads the model automatically. It's about 4-5GB so give it a few minutes depending on your connection.

What's actually happening: Ollama downloads a compressed version of the model that's optimized for your hardware. You don't need to configure anything - it picks the right version automatically.

Model options:

Model	What it's like	Command
Llama 3.1 8B ⭐	Best quality for most tasks	ollama pull llama3.1
Mistral 7B	Slightly faster, similar quality	ollama pull mistral
Llama 3.2 3B	For older/slower computers	ollama pull llama3.2

Step 3: Connect to Quillon

Once Ollama is running and your model is downloaded:

1. Open Quillon
2. Go to Settings → AI Configuration
3. Select Local AI (Ollama)
4. Your downloaded model should appear in the dropdown - select it
5. Hit Save

You'll see the green "AI Operational" indicator when everything is connected.

Troubleshooting

"Quillon can't find Ollama" Ollama needs to be running in the background. On Mac, look for the Ollama icon in your menu bar. On Windows, check the system tray. If you don't see it, open the Ollama app again.

"It's working but really slow" Local AI speed depends on your hardware. If it feels too slow for your workflow, Groq's free API is a faster alternative that's just as easy to set up. Switch to Groq →

"I'm getting memory errors" The model may be too large for your available RAM. Try closing other apps first. If the problem persists, try a smaller model (ollama pull llama3.2) or switch to Groq.