Check to see what scaling you are applying. It maybe you are scaling things up so much on the TV that there is not enough space to run Roon windowed. On the PC I have attached to the TV, I set the input to 1920 x 1080 and that is large enough so I do not have to do any scaling to read things from my lounge chair.
As Daniel suggested above, can you confirm the scaling settings on the Core machine? If you set it to 100% do you see this same message or do things work for you?