From the manual:
To play 11.2MHz DSD recordings with TEAC HR Audio Player, open the Configure window and set the “Decode mode” to “DSD Native.”
To play formats other than 11.2MHz DSD, you can set this to either “DSD over PCM” or “DSD Native” as you like.
Native DSD requires an ASIO driver – which are rare for macOS/OSX (I believe Exasound has one). By default it does DoP only – limiting your DAC to DSD128.
I suspect that the TEAC HR app uses some special trickery that works in-app only. Until TEAC provides an OS level driver, I’m afraid you’re stuck with DoP on your Mac – and DSD128.