To the OP:
While there is an orthodox answer (see below), there is no universally correct answer to your question. While I don’t have a Nucleus, I do have a Marantz 8802A pre-pro, so I can relate to your situation.
Let’s start with the basics: you’re getting sound out over HDMI, so that’s a good start. What type of files are you playing with Roon? FLAC? MP3? AAC? Apple Lossless (ALAC)? WAV? AIFF? Something else?
As other replies have indicated, your Marantz should accept up to 192/24 PCM over HDMI. It will take up to 7.1 channels, if you like multichannel music. You can configure this by clicking on the gear icon next to the Nucleus HDMI output on the Audio Settings page. If you’re playing WAV/AIFF/FLAC/ALAC, you’re aiming to get the signal path indicator (next to the skip backwards playback control) to glow purple. This indicates that Roon has a transparent, bit-perfect path to your pre-pro. Glowing green indicates that Roon (or some other software) is modifying the audio before it gets to your pre-pro. Amber/orange, if I remember correctly, is what Roon shows if you’re playing a lossily compressed file, e.g., MP3 or AAC. Ignore the discussion of DSD for now; set Roon to convert it to PCM.
If you’ve gotten the purple signal path indicator on a losslessly compressed or uncompressed track, how does it sound to you? If the answer is good, then you can either declare victory for the time being, or continue to explore, but exploring will entail additional purchases. If you have dropouts, clicking noises, or similar very obvious problems, then more work is needed.
For various technical reasons, HDMI isn’t the audio interface you’d design if video wasn’t an issue. If you wish to continue exploring, the next simplest approach would be purchasing a USB/SPDIF converter, USB & SPDIF cables (I won’t touch the great cable debate here), and use one of the SPDIF inputs on your pre-pro. You may or may not hear a difference, but if you hear a very dramatic difference, then it’s probably related to a pre-pro setting. If it sounds the same, a little better, or a little worse, then you’re probably making a fair comparison.
The orthodox answer is to purchase a streamer with a SPDIF output to use with your pre-pro. DIY solutions start at ~$100, packaged solutions in the $300-400 range, and go up (sometimes way up) from there.
Good luck!