The best thing you can do is remove complexity and build it back up until the point it breaks. This firewall has a somewhat complex set of knobs which can be turned. A few of those knobs enable “intelligence” by trying to automagically tune and/or protect the network in a dynamic way. This requires the network is properly modeled and that model is part of the firewall configuration. Of course, I’m guessing here as there isn’t nearly enough information to diagnose why the discovery becomes unstable. If I was walking in to diagnose this I’d start with drawing out the network (requires full set of configuration) and then looking through the logs to identify what the devices were trying to be “intelligent” about.
As far as what transport and port is needed? It’s not entirely straight forward:
This is why its best, and most reliable, to not put any kind of firewall, filtering, shaping, or other intelligence between Roon Core, Remote, Zones, etc.
Oh… and I guess I should mention the obvious…
How many Wifi networks do you have that the iOS device knows about? When you have this issue is the device on the right Wifi network?