Only getting 24-bit/96kHz from my Denon 3803

I have a 2018 Onkyo RZ 740 and it is limited to 96/24 for SPDIF both optical and coax and even with at 96 optical struggles at times. HDMI accepts 192/24 and the DAC can even do more. The limitation are hidden somewhere in the manual.
I would not be surprised one bit that your vintage receiver is limited the same way.