I set up the same on a RPi 3B+ and it did not keep up for 20 minutes
But there is good news. For the Librespot Entrypoint I investigated how I can update to the latest version of Liquidsoap, the application that creates the web radio stream, and saw an improvement in stability and a decrease in CPU load. I now performed the same update for the CD Entrypoint and have not seen the issue while I already played multiple CDs. Before I can release this update I have to adapt the extension to the latest changes in Liquidsoap, but it has my attention and an update will come.