xdm mode (among other things) forces full screen and disables those settings in the UI. However it only disables them in the tlclient options dialog. It is still possible to toggle full screen from the context menu, and the options dialog in vncviewer.