It can take a long while to load and play a local track. And it seems to take longer when Roon servers are likely hit harder (times when it is likely a lot of people are using it).
My OS and Roon DB are in an internal SSD, local music in an SSD drive connected directly over USB. There’s no hardware reason why this would take so long, and indeed it doesn’t in Audirvana.
Setup is mini i7, 16GB RAM, 250gb internal SSD with Catalina + Roon + Roon DB, 4TB external SSD over USB. Regular CPU load at 5%.