FWIW, I’ve been having something similar that I’m chasing.
For me, turning off the “auto-optimize” feature of my Unifi networking gear seems to have reduced the frequency with which I get a long delay at the beginning of a track, but it still does happen. I’m working on cleaning up my library so I can run the suggested test on backing up the database and trying fresh. No idea if you’re facing similar issues, but this may elicit some ideas: