When a thinlocal job arrives on an agent that doesn't have the relevant client connection, it must respool to the correct agent. To do this it needs a hostname to connect to. Unfortunately we do not have a suitable one for this purpose. Instead we have two other ones:
a) agenthost - Specified as the hostname the master uses to connect to agents
b) termserv_hostname - Specified as the hostname the client uses to connect to agents
Currently we use b). This is probably not the best choice as it is likely to be a NAT:ed address that might not work properly from the agent's side of the firewall. I'm amazed we haven't seen any complaints about this yet.
It is probably better to use a). Although not explicitly specified for agent to agent communication, it is at least implied to be addresses internal to the data center.
We could also add a third name, explicitly for agent to agent connections, but that seems messy.
Fourthly, we could change the architecture so that an agent respools at the master first, since we have a well defined path from agent to master, and then back to another agent.
We should also change our documentation to reflect that agenthost is used for communication between all servers, not just from the master to the agents.