Have you made sure that your firmware is up-to-date?
I also had issues with a modem router access point all in a single unit. But I have not had issues since I went to an erro mesh network. You might try using a Wi-Fi analyzer to check your signal strength. But as has been said a wired connection would be much more reliable.
Also, remember that the LS50W converts all inputs to 24/192 in its DAC and the USB accepts up to 192kHz and the Toslink only accepts up to 96kHz so it may help to match the output of your source to the KEF input specs.