Thanks! I was running some models on my RX 9070XT, but only Ollama works flawlessly. I couldn’t make llama.Cpp to run Gemma4 or the newer Qwen - maybe I’m hitting that incompatibility, but probably it’s the skill issue.
P40 doesn’t look very appealing. 32 GB V100 costs about the same as 2xP40, less VRAM in total, but it’s faster, will use less power.
But I’m not sure if I follow you on the PCIe… If I run a model that spans multiple GPUs, doesn’t PCIe bandwidth matter?
Thank you! This is really helpful. 32 GB V100 or pair of 5060ti s looks very interesting, and about the same price. Does running multiple GPUs require any special hardware? I mean apart from the motherboard with 2+ PCIe x16 slots?
Not this one. It IS a new shiny one, they coat $300-$400 more on eBay than in stores due to limited supply.
Nice! You kinda answered my next question already with this web tool. I was curious if you are getting any useful results from the model itself without feeding it with good data first or relying on hardcoded tools. 4b model must be really dumb for anything even little complicated. I see you recommend to run two models - is it in parallel or the router can control backend and switch models?
This looks awesome! Can you share the real life use cases for this? What are you using it for?
Wow, I didn’t think you were running 176GB worth of GPUs on a consumer board! I don’t have an extra board, and my gaming PC that has 9070XT is not a good basis for multi GPU build - it has a cheap mATX motherboard with too few slots and lanes. So it’s going to be a new build. Used EPYC boards look interesting for that.