Skip to content

How to AI

Input Interface

You will need:

  • Terminal emulator software (free)

Then send your keystrokes via the terminal to the remote machine, where you can practice in a virtual environment using a real-world controller.

Hardware

  • A remote server to run Linux, an internet connection and forwarding (optional: Tailscale)
  • Cloud instance(s) -- If you do non-trivial amounts of AI a local machine will be significantly cheaper than the cloud over time. Scaling in the cloud is useful, but you should build a setup with dedicated hardware (e.g. below) for local development and testing.
  • GPU(s) to run code -- 16GB cards can be had for under $400 (eg. NVidia 5060 Ti). Modern workstation motherboards allow you to connect up to 4-6 at a time for running problems in parallel. 32GB (eg 2x) can be done cheaply with a gaming-style motherboard if you are on a budget. If you are a gamer looking to splurge on a single GPU then a 5090 is your best bet.

Current config

  • 6 x 4060 Ti 16GB setup w/ 5995x

Over the past year, memory requirements for many models are now creeping up into the 48GB-80GB+ range, 70B models in particular. New frontier models require even larger amounts of RAM if you want to run them locally. Setting up dual slot server/HEDT CPUs with ~1TB of cheap(er) RAM is probably the cheapest way to run large models locally without investing in a GPU cluster.

  • 2 x 7900 XTX w/ 2455X

If all you want to do is run ollama then this is an interesting choice. ROCm has come a long way forward since the RDNA2 series but is still generationally behind NVidia on the software front for AI development and edge compute for deployment in the field. Apple (MLX) and AMD (Ryzen 7) are bringing shared memory approaches to market, but their software stack is behind.

  • On-device

Being able to run models on a device like Jetson Xavier, Nano or Thor is an interesting way to work with AI in the real world. Having said that, the best way to bootstrap embedded development is to get a model working on a higher-end system and then figuring out how to bring it to a resource-constrained device.

Self study

IRL

  • A laptop to run terminal emulator (your choice)
  • Portable internet in some form -- hack the planet
  • Random papers, pens, coffee (required)