@kaosbreed

So, I've been doing a little poking around myself, as having 32gb vram means I could run the LLM on the CUDA option, if it were set up properly, so --- "Change (CUDA\d+) to ((?:CUDA|ROCm)\d+) in llama linux runtime.py and bundle the Lemonade SDK ROCm llama-server binary" - dunno if that's helpful at all, but it has worked for me at least, running a 27b LLM on CPU was taking.... around 2-3 hours to do a big prompt, after changing these settings, same prompt, about 5 minutes, possible you may want to add that yourself at some point, or something along the same lines for running the LLM on GPU, s

Running LLM on gpu

Yue 2

architecture