If you umount a local drive and there is still something accessing it, then the kernel will not fully let go of it. It gets removed from /proc/mounts, but not /proc/fs/nfsfs/volumes. You can also see "server not responding" in dmesg.
The problems start when you connect again and get the same port number. The server believes it's the same NFS server (same address and port), so it will try to reconnect things for a while. This causes a very long and annoying delay before giving up and moving on with the session start up.
Not sure what to do about it though. Random ports? Fiddle some more with the timeout delays? Drop -l from the umount and have just -f? Or simply move to FUSE?