176.4kHz is exactly 4x the CD rate 44.1kHz. Why do you think it’s strange?
true!
I no longer think it’s strange 
It is the most logical/common result of converting DSD64 file data to PCM.
Other than 88.2 which is also very common.
but I am not playing DSD, no matter what id the file sampling rate, no matter if I eneble or disable upsampling, and how I configure it everything comes to DAC as 176.4.
If 44.1 and 88.2 show up as 176.4, then something is upsampling. Sounds like you’re running Max PCM rate Power of 2 in MUSE.
What device are you using as the streamer connected to the DAC? Show you Roon signal path.
Roon core on Win 10 Surface fanless USB to RMI ADI-Pro.
Situation escalated somewhat because when USB was connected I was not able to change my default upsampling rate on my DAC.
The problem was that no matter what I set up as upsampling rate in Muse it was upsampled to 176.4. Even the tracks in 24/192 native format.
Updating the USB driver for RME ADI-2 Pro and switching to ASIO from WASPI fixed this.
This setup, USB in particular, is for travel, so I am not that concerned. My main streamer is either Pi2AES or Holo Red with AES EBU to the DAC and I never had problems of this kind.
Roon was most likely OK, upsampling to the only rate the DAC was willing to accept, taking into account that the device clock was managed by Windows through USB. I avoid USB like a plague, and for good reasons.
Sounds like you were using WASAPI but not in exclusive mode and Windows system audio set to 176.4 this would result in what you say. If exclusive was turned on in Roons settings then it should have worked the same. ASIO bypasses system audio processing much like setting Exclusive mode to on for WASAPI.
No, the audio device was set to exclusive mode. End I could see it Roon was upsampling to 176.4.
Strange, never had that with my ADI 2 DAC but then never used Windows.