I think the problem here is the Phantom has an internal delay for its DSP. So despite the Pi being in sync with the Mac as both are RAAT it doesn’t take into account the device delay. Perhaps Roon could give options for custom delays when grouping such devices?
Meanwhile I suppose you could put a delay in both channels of the D120 in firmware to try and match the Phantom. Devialet may be able to tell you what it is? Or it may not be consistent in which case it would be practically impossible.