Is your Denon AVR being used in a home cinema setup, with its HDMI output being connected to a TV?
If so, try physically unplugging the HDMI connection between the AVR HDMI output, and the TV HDMI input, and see whether the Nucleus HDMI output can be set up and work with the Denon.
In my case, I ran into HDMI handshake issues when I replaced our old Panasonic LCD TV with a new Samsung QLED TV. In the end I had to invest in a Dr. HDMI device to sit between my ROCK NUC and my Denon 3808 AVR. Not cheap, but it does mean that I’m no longer plagued by HDMI handshake issues.