This is because tlclient has code paths for updating Windows and !Windows, but OS X doesn't have xdg-open which tlclient tries to use for the update procedure. This should be fixed so that the automatic updates work on OS X as well.
Add a APPLEINSTALLER to the config as well.
This one seems fairly straight-forward to unbreak, but could take a few hours extra if we want to add some extra polish to the user experience.
After a bit of discussion and investigation I've found that a limitation of our current way of updating is that the default web browser needs to launch the downloaded file after it has finished downloading it, and the behavior of Safari on Mac is to just download the file and put it into the default Downloaded folder. (There are a few exceptions - Safari has a "safe files list", but support varies between versions. Using open on a url containing a dmg does not trigger any special behavior in Safari either, they're just downloaded.) This means that we're leaving the user in the middle of a process, with no clues how to proceed with updating. Therefore, lets refer users to the download page, as with the Linux clients. The idea is to have the user in control the entire upgrade process. I'm willing to reconsider this approach if we can use. This also means that we really don't need the APPLEINSTALLER configuration key, so the code I've written assumes that the DEFAULTINSTALLER will point to a web site with sufficient links and instructions. All of this has been commited as r26204.
Seems like I forgot to end a sentence in my last comment. I'm willing to reconsider this approach if we can use a library such as neon or libcurl for downloading the client instead of offloading it to the OS, or make this process more automatic in other ways.
Tested using client build 3937, works as expected both with a client configured with UPDATE_MANDATORY and without.