Naw, it sounds perfectly reasonable to me. I’ve used RoonServer on high end PC desktops since 2015 and have had fantastic response times and rock (pun intended) solid performance.
IMHO, there is a world of difference between using best of breed computer parts (power supply, motherboard, memory) and what you get in a NUC.
Here is the link to the server I built this Aug.