Most of us spend a surprising amount of time and money putting together computers. We research parts, compare benchmarks, debate whether the next upgrade is worth it, and eventually build something we’re pretty happy with. Then we use it for work, browsing the internet, gaming, and a handful of everyday tasks.
And for a good portion of the day, all that hardware sits idle.
I’ve always wondered what else those machines could do. That curiosity led me to running large language models locally, and eventually to OpenLLMWorks, a project that turned out to be a lot more involved than I expected.
It started with Ollama
My introduction to local LLMs came through Ollama. The idea that I could download a model and run it on hardware I already owned was fascinating. I didn’t have to start with a brand-new computer or a high-end GPU. I could experiment with what was sitting around the house and see how far it would take me.
That raised all kinds of questions. Could an older GPU run a useful model? How much did video memory matter? Would a faster card make a meaningful difference, or just produce a more impressive number on a benchmark?
I’ve always liked finding another use for existing hardware. We put a lot into these systems, and I like the idea of getting more out of them than the usual mix of productivity, entertainment, and web browsing.
Local AI felt like a new reason to start experimenting.
I thought I was building a simple benchmark
Once I started trying different hardware, I wanted a consistent way to compare results. My original thinking was pretty straightforward: run a test, collect a few numbers, and see how one system stacks up against another.
That sounded manageable.
Then I started thinking about what would make those numbers useful to anyone besides me. If different people run different models, settings, or versions of the software, are we actually comparing the same thing? What happens when a test fails? How do you collect enough information to understand the result? And how do you keep a public leaderboard from becoming a collection of numbers nobody can verify?
Suddenly, I wasn’t just building a website with a table of benchmark results. I needed a benchmark runner, a repeatable testing process, a way to collect submissions, and a workflow for reviewing and publishing them.
What I thought would be a small project became an education in how much work sits behind software that looks simple from the outside.
Building software with AI
The other experiment was seeing how far I could take the project with ChatGPT.
I’m comfortable around technology, but I’m not going to pretend I’m a seasoned software developer. Building a website, a benchmark runner, and the systems connecting them would have been a pretty ambitious undertaking for me without AI assistance.
ChatGPT made it possible to work through ideas, write code, troubleshoot errors, and gradually put the pieces together. But I learned fairly quickly that asking AI to build something isn’t the same as receiving a finished, dependable product.
Sometimes a feature worked on the first try. Other times, fixing one issue exposed another. There were moments when the code looked perfectly reasonable but didn’t behave the way we expected. Getting something to run was only part of the job. Understanding whether it was doing the right thing was another challenge entirely.
I’m sure there are platforms that can automate more of this process, but cost is part of the equation for a personal experiment. I wanted to see what I could accomplish with the tools available to me, without turning a hobby into an expensive development project.
More than anything, the experience has given me a deeper appreciation for developers who built reliable software long before AI coding assistants were available. I went in expecting to learn about GPUs and language models. I ended up learning quite a bit about testing, validation, deployment, and the patience required to make things work consistently.
Seriously, respect.
How fast is fast enough?
It’s easy to get caught up in chasing performance. I understand the appeal of owning the newest GPU and seeing just how fast it can go. That’s part of what makes hardware exciting.
But I’m increasingly interested in the middle ground. I don’t need the absolute fastest system, and I don’t want something painfully slow either. I want to know what delivers a good experience without spending more than necessary on hardware and electricity.
A benchmark might report 4,000 tokens per second. That sounds incredible, but what does it actually mean for the person sitting in front of the computer? Is that prompt processing or text generation? Does it make a noticeable difference to the task they’re trying to complete?
I’m still figuring that out.
VRAM capacity matters, especially when determining whether a model fits on a GPU. But it’s not the entire story. Architecture, memory bandwidth, software support, and the kind of workload all play a role. We’re still collecting results, so I don’t want to jump to conclusions about which hardware offers the best balance.
What interests me is the trade-off. If a more expensive GPU finishes a task much faster but consumes more power, is it worth it? Maybe. If I’m running an interactive assistant, speed might matter a lot. If I’m processing something in the background while doing other things, perhaps waiting a few extra minutes is perfectly reasonable.
I want OpenLLMWorks to help make those trade-offs easier to understand. Not just which GPU wins, but what kind of performance people can realistically expect from the hardware they already have.
Beyond trivia questions
There’s a bigger question behind all of this: what am I actually going to do with local AI?
Right now, I can run models, have conversations, ask questions, and experiment. That’s fun, but I’m hoping we eventually get well beyond trivia, reminders, and chat.
I’d love to have AI running at home that could help build things for my family. Maybe that’s a small website or application designed around our particular needs. It wouldn’t have to attract visitors or become a product. It would just need to be useful to the people in our house.
Or imagine taking a 3D scan of an object, asking AI to help turn it into something printable, and using a 3D printer to make it. Another example: my dishwasher stops working. Instead of spending hours searching through forum posts and videos, I could upload a video of the problem and ask AI to help me figure out what to check.
Of course, those are aspirations, not promises about what my current local setup can reliably accomplish. An AI-generated repair suggestion still needs verification, especially when electricity, water, or other hazards are involved. And turning a scan into a usable 3D print takes more than generating a convincing picture.
Still, the possibilities are exciting. Even if a local model needs several minutes to analyze something and produce a useful answer, that could be a worthwhile trade if the alternative is spending an evening searching for the right information.
And once again, I come back to the hardware question. How powerful does a computer need to be to make those experiences practical? Could an older system do the job if I’m willing to wait? How much will improvements in models and inference software change the answer?
Those are questions I’d like to keep exploring.
Why make the results public?
I could have tracked my own experiments in a spreadsheet. That would have been enough to satisfy some of my curiosity.
But I’ve always admired crowdsourced projects. I like when people share what they’ve learned, including the roadblocks, because those experiences can help someone else get a little further. A failed attempt can be just as useful as a successful one if it saves another person hours of frustration.
That’s why I want OpenLLMWorks to be a community resource rather than just a record of my own machines. The more people contribute results from different hardware, the more interesting and useful the comparisons can become.
That comes with a challenge, though. Community data only becomes valuable when there’s enough consistency to make comparisons meaningful. It’s one reason I’ve spent more time than expected working on the benchmark process and submission workflow.
My hope is to build something useful in the spirit of tools like CrystalDiskMark, but focused on local AI and shaped by community participation.
Keeping a record of progress
I also want OpenLLMWorks to be more than a snapshot of which GPU is fastest today.
Imagine looking back several years and seeing how the same class of hardware performs as inference software improves and models become more efficient. Or comparing what an affordable home computer can accomplish now with what it could do a few generations earlier.
Think about the original iPhone compared with today’s smartphones. The change isn’t only about faster processors. Hardware, software, and the things people expect their devices to do have all evolved together.
Local AI may have a similarly interesting progression. Hardware that seems limited today might become more useful as models and software improve. To see that clearly, though, we’d need to preserve enough information about the benchmark versions and test conditions to make historical comparisons fair.
That’s part of the longer-term goal: an open record of how local AI performance changes over time, not just a leaderboard that forgets yesterday’s results.
Still figuring it out
OpenLLMWorks is very much a work in progress. There’s more hardware to test, more data to collect, and plenty I still need to learn about building and maintaining the platform.
I started because I wanted to see what my existing computers could do. Along the way, I found myself asking a bigger question: what could we accomplish if useful AI didn’t always require the newest hardware, an expensive subscription, or a server somewhere else?
I don’t have the answer yet. But that’s a pretty good reason to keep experimenting.
A little note: OpenLLMWorks is a personal side project of mine. If you’re curious about running AI locally, want to explore the benchmarks, or have hardware you’d like to contribute results from, I’d love for you to check it out at OpenLLMWorks.com.