I have exactly the same issue with a similar set up (Mac mini M1). Thread here https://community.roonlabs.com/t/please-suggestions-to-solve-time-lag-on-track-and-album-changes/198147
In my case the issue is “time sensitive”: gaps are longer the longer the previous song has played AND the longer the Roon session. Restarting Roon (I do it every day) solves the issue momentarily, but crawls back.
I’ve been suffering from this issue for a long time, interacted with support many times, seems to have gotten better with latest versions of Roon…but still is there