The first time I tried llama.cpp

llama.cpp was one of the first inference servers available for running an LLM model. At the time, I wasn’t super impressed with AI but they were better than the old fashioned auto complete. I didn’t think a subscription was worth paying for but if I could run my own, i’d take it.

When I first got to the llama.cpp repo on github, running it still required compiling it from source code. Annoying but it wasn’t the end of the world. What actually stopped me was that there wasn’t something like hugging face available to pull models from. The models that were around were the hundreds of billion parameter versions or I could write code to create my own from training data.

Ollama is a powerful AI model serving platform that makes it easy to run large language models locally. In this post, we’ll walk through setting up Ollama with Open WebUI in a Docker containerized environment, including GPU acceleration support. Visit Ollama’s website for more information.

What is Ollama?

Ollama is an AI platform that provides an easy way to run large language models locally. It offers:

• Easy model management
• GPU acceleration support
• A clean API for model interactions
• Containerized deployment

What is Open WebUI?

Open WebUI provides a web-based interface for interacting with Ollama models, making it easy to use AI models without needing to write code or deal with complex command-line interfaces. Visit the Open WebUI GitHub repository for more information.

Ollama is a powerful AI model serving platform that makes it easy to run large language models locally. In this post, we’ll walk through setting up Ollama with Open WebUI in a Docker containerized environment, including GPU acceleration support. Visit Ollama’s website for more information.

What is Ollama?

Ollama is an AI platform that provides an easy way to run large language models locally. It offers:

• Easy model management
• GPU acceleration support
• A clean API for model interactions
• Containerized deployment

What is Open WebUI?

Open WebUI provides a web-based interface for interacting with Ollama models, making it easy to use AI models without needing to write code or deal with complex command-line interfaces. Visit the Open WebUI GitHub repository for more information.

I previously wrote on how to run Dex using docker. In this article, I will create nearly the same setup using docker-compose. This will allow you to run, stop, and restart the setup from a central location.

Setting up the docker-compose file

You’ll want to make sure that you have the same Dex config file from the previous article but with one small change. The issuer field should be changed to http://dex:5556/dex to match the service name that will be used in the docker-compose file. If the issuer field is not changed, the example app will try to find Dex within the container it is running in and will not be able to reach it. Changing it to http://dex:5556/dex will allow the example app to reach Dex through the network that docker-compose creates.

Dex is a tool created to make setting up and managing authentication methods easy. It is a single sign-on and identity management tool that can be used to authenticate users against a variety of backends. It is a great tool to use when you want to add authentication to your application without having to worry about managing the users yourself but still gives you the option if you want to. This post will show you how to run Dex locally to decide if it is the right tool for you. This does not cover how to integrate Dex into your application or integration with other authentication providers but I will cover that in a future article.

Installing the driver

sudo apt install broadcom-sta-dkms

Configuring the connection

Open /etc/netplan/00-installer-config-wifi.yaml and make sure it has something like the following:

# This is the network config written by 'subiquity'
network:
  version: 2
  wifis:
    wlp6s0:  # The network found with `ls /sys/class/net`
      access-points:
        {SSID}: # SSID
          password: {Password} # Password
      dhcp4: true

Close the file and run