Running a Large Language Model On Your Own Computer

[Slim]

Last night I installed an AI on my new PC. I'm using a LLM called Llama3, which is an 8-billion parameter model. It takes up about 4G of disk space and runs comfortably in my 16G graphics card in quantised form.

Quantising is the practice of reducing the precision of the floating point numbers used to represent the model's "weights" - the numbers that tell the model how one node (like a neuron) influences another. This particular version of the model runs with 4 bit precision which sounds rubbish, but in practice it doesn't noticeably degrade the model's usefulness. The point of this is to allow the model to run on smaller GPUs, like consumer hardware, and to be maintained in smaller files.

special /data/ollama/bin>  ./ollama show llama3
  Model
    architecture        llama   
    parameters          8.0B   
    context length      8192   
    embedding length    4096   
    quantization        Q4_0   

  Capabilities
    completion   

  Parameters
    num_keep    24                     
    stop        "<|start_header_id|>"   
    stop        "<|end_header_id|>"     
    stop        "<|eot_id|>"           

  License
    META LLAMA 3 COMMUNITY LICENSE AGREEMENT           
    Meta Llama 3 Version Release Date: April 18, 2024   
    ...                                                 

special /data/ollama/bin>


I'm using a simple interface that runs in a Linux terminal. Just now I gave it a test, proposed by ChatGPT:

>>> A satellite is in a circular low Earth orbit at 400 km altitude.
... 1. Calculate its orbital period.
... 2. Show the key steps and formulae you used.
... 3. Then explain in plain English what that orbital period means in practical terms.

Let's break it down step by step.

**Step 1: Calculate the orbital period**

To calculate the orbital period, we'll use Kepler's third law, which states:

T² = (4π³ \* r³) / G \* M

where:
T = orbital period
r = radius of the orbit (400 km = 6378.1 km + 400 km)
G = gravitational constant (6.67408e-11 N·m²/kg²)
M = mass of the Earth (5.972 x 10^24 kg)

First, let's convert the altitude to a distance from the center of the Earth:

r = 6378.1 km + 400 km = 6878.1 km

Now, plug in the values:

T² = (4π³ \* (6878.1 km)³) / (6.67408e-11 N·m²/kg²) \* (5.972 x 10^24 kg)

Simplifying and solving for T:

T ≈ 5593 seconds

To convert this to a more meaningful time unit, let's divide by 3600 (since there are 3600 seconds in an hour):

T ≈ 1.55 hours

So, the orbital period is approximately 1 hour and 33 minutes.

**Key steps and formulae used:**

1. Kepler's third law: T² = (4π³ \* r³) / G \* M
2. Converting altitude to distance from the center of the Earth: r = 6378.1 km + 400 km
3. Simplifying and solving for T

**Practical explanation:**

The orbital period is the time it takes the satellite to complete one full orbit around the Earth. In this case, it's approximately 1 hour and 33 minutes. This means that if you were to launch a satellite into this circular low Earth orbit, it would take about 1 hour and 33 minutes for the satellite to return to the same point in space.

In practical terms, this orbital period affects how often your satellite needs to transmit data or receive commands from ground control stations. For example, if you need to send data updates every 30 minutes, you'd need a satellite with an orbital period of less than that.

.. ChatGPT was slightly sniffy about its response, noting that it presented the formula wrong (although it appears to have used it properly nonetheless). But it conceded that the the result was correct to a reasonable degree of accuracy.

For a free model running on a desktop computer, I think that's pretty good. Is it actually useful? Not really .. plenty of better LLMs online for free. I suppose if I needed one while my Internet was down, it would come into its own.