Some sample special characters (I hope they display well in your browser):
ä Ä ö Ö ü Ü à À ç é É è È ê Ê * @ ? ! _ ` ~ { [ ] }
AFAIK does Roon still require SMB v2. If SMB v3 is available it might use that instead or upgrade the connection (or whatever). Enable SMB v2 support in your QNAP and try again.
There were several user reports in the past that SMB v3 only support on the server side doesn’t work.