USB is asynchronous so it should not care about any drifts or inaccuracy, since the DAC will ask for data when it need it and use its own clock. If SPDIF was used it might matter though, since the DAC is forced to use the overall speed of the sender (even if it internally reclocks and smooths out inaccuracies in shorter timeframes).
What can make a difference is how electronically noisy the server is, since that noise can (and often will) traverse through USB or ethernet and affect the DAC. The noise might not be caused by the actual server though, but originate from mains power, RFI and so on.