vsmserver and vsmagent have a list of approved hosts that are allowed to communicate with them. To respect DNS changes to these hosts, we look up the hosts on each connection. Normally, a bit of delay shouldn't be a problem, as we generally have a timeout of 40 seconds for XML-RPC requests. The issue is that the DNS requests are synchronous, so the delays accumulate. Every XML-RPC request will have to wait for the DNS lookup for all requests before it. But the timeouts still keep ticking. So it doesn't require that many requests until things start failing. The issue could also affect many other things as everything in the respective service gets paused and delayed.
When we wrote this code, there weren't any good asynchronous versions available. But now that we've switched to asyncio, there is one available: https://docs.python.org/3/library/asyncio-eventloop.html#asyncio.loop.getaddrinfo Note that the caller of the relevant code also expects a synchronous result, so some redesign of this entire section of code will be needed.
