To expand on the answer above a little bit:
System Output is going to make Roon’s output function like any other application – audio is sent to the OSX mixer, where it’s mixed with other sources (Youtube, system sounds, etc).
In many cases, the audio will be resampled during this process, so this mode is more about convenience than critical listening. For example, if you switch your default output from the Macbook’s speakers to a bluetooth device, Roon’s output will go with it, because it’s simply playing to the default OSX mixer.
Built-In Output is pointing Roon at an actual audio device regardless of your OSX audio defaults. This means you can set it to run in Exclusive Mode, meaning Roon takes control of the device and should play bit-perfect in whatever formats the device supports. This is how you’d want to play for critical listening.