So, I've been doing a little poking around myself, as having 32gb vram means I could run the LLM on the CUDA option, if it were set up properly, so --- "Change (CUDA\d+) to ((?:CUDA|ROCm)\d+) in llama_linux_runtime.py and bundle the Lemonade SDK ROCm llama-server binary" - dunno if that's helpful at all, but it has worked for me at least, running a 27b LLM on CPU was taking.... around 2-3 hours to do a big prompt, after changing these settings, same prompt, about 5 minutes, possible you may want to add that yourself at some point, or something along the same lines for running the LLM on GPU, since it unloads after finishing and you have all your vram back for generating That said, this is on linux obviously, the pipeline may be different for windows, but hopefully it's helpful anyway.
AboutMaestro AMDyoyokits/Maestro-AMD
So, I've been doing a little poking around myself, as having 32gb vram means I could run the LLM on the CUDA option, if it were set up properly, so --- "Change (CUDA\d+) to ((?:CUDA|ROCm)\d+) in llama_linux_runtime.py and bundle the Lemonade SDK ROCm llama-server binary" - dunno if that's helpful at all, but it has worked for me at least, running a 27b LLM on CPU was taking.... around 2-3 hours to do a big prompt, after changing these settings, same prompt, about 5 minutes, possible you may want to add that yourself at some point, or something along the same lines for running the LLM on GPU, since it unloads after finishing and you have all your vram back for generating That said, this is on linux obviously, the pipeline may be different for windows, but hopefully it's helpful anyway.