Local Ai
OrangePi AI Studio Pro - Qwen3.5-122B-A10B
https://preview.redd.it/wbq8ullnbafh1.png?width=1409&format=png&auto=webp&s=e6d2fe2b1c87c724bc64003c25f917dcee53260f I finally got round to tweaking this, with a bit of help from GLM5.2. The trick to
https://preview.redd.it/wbq8ullnbafh1.png?width=1409&format=png&auto=webp&s=e6d2fe2b1c87c724bc64003c25f917dcee53260f I finally got round to tweaking this, with a bit of help from GLM5.2. The trick to getting it running with vLLM (which I couldn't get anything really out of before) was when I realized we could write a stub to to implement the rtGetDevMsg to return device capabilities (basically we fake a response from the card) - this is need to get torch_npu running properly on the device. With that I can finally use vLLM with this, making it actually useful. I finally got round to tweaking this, with a bit of help from GLM5.2. The trick to getting it running with vLLM (which I couldn't get anything really out of before) was when I realized we could write a stub to to implement the rtGetDevMsg to return device capabilities (basically we fake a response from the card) - this is need to get torch_npu running properly on the device. With that I can finally use vLLM with this, making it actually useful. submitted by /u/StillVeterinarian578 [link] [comments]
Related
Source: r/LocalLLaMA | 2026-07-25