I had the same problem recently. The solution is to click More on Home > Listen Later, and then you have a menu for each item, even the removed one, where you can still delete it.
RoonLabs will look for a neater solution and created an internal ticket: