I guess their answer would be to buy a Roon Nucleus and that ROCK / Roon OS is a DIY solution offered for free, but without any hands on support.
Anyway…that’s all kind of irrelevant to this conversation…
Is there a good reason you don’t want the NUC to be both the server & endpoint? Doing that would almost certainly be the easiest solution to get you up and running with a truly headless setup.
Why make life more complex…
Personally I’d try that first, it might be that once you have that working you no longer feel there’s a need to install a sever elsewhere. You’ll also have a better understanding of how Roon / Roon OS works by doing that.
Otherwise your only option is to install the regular Roon Server package on your PC under whichever OS you prefer (Windows or Linux) and get that OS to boot up headless. But getting that OS to boot up headless is outside the scope of Roon, as it’s a OS thing.
As mentioned above, Roon OS is only designed to to run on selected NUC models, it might install on other machines but isn’t official supported and even community support will be hard to come by due to all the unknown factors involved. But you don’t need to use Roon OS to run Roon Server, it’'s just a nicely packaged, low-maintenance, self-updating way of doing so, as it doesn’t require a separate host operating system like Windows. It also happen to the the same OS that Roon’s Nucleus uses.
[1] If your dead set on separating them out, then I’d probably make the NUC the server with Roon OS installed (and put it in another room) and then get a small, low-cost endpoint (without any moving parts or a fan) like a RPi with Ropiee installed and connect that to my DAC. Although the only real benefit to doing this is that you no longer have to have a NUC on display in your HiFI rack with potential fan noise although the fan on the NUC rarely kicks in during normal use.