Comment 1 Henrik Andersson 2015-06-02 14:57:54 CEST

This is not a regression, same issue is reproducible using 4.2.0 and 4.3.0 client.

Steps to reproduce:

 1. Configure local drive redirection and connect to a ThinLinc session

 2. Start a terminal window and go into ~/thindrives/<share>/

 3. Using the Explorer on the windows client, browse to exported directory and 
    create a file.

 4. List the files within the mounted export and the new file created in step 3 
    is not visible.

However, you can run 'stat filename.txt' so some data is sent over nfs protocol.

Comment 2 Henrik Andersson 2016-08-29 16:31:16 CEST

Support issue 21040 had the same problem using ThinLinc client 4.6.0 on Win10 platform. I couldn't reproduce at first. There is a corner case to this issue. You need to export a directory directly under C: eg. `C:\thinlincshare` to hit this bug.

Comment 3 Henrik Andersson 2016-09-01 12:06:40 CEST

(In reply to comment #2)
> Support issue 21040 had the same problem using ThinLinc client 4.6.0 on Win10
> platform. I couldn't reproduce at first. There is a corner case to this issue.
> You need to export a directory directly under C: eg. `C:\thinlincshare` to hit
> this bug.

This information is invalid. I am now in a state that i can not have a working case at all. I have tested with different ThinLinc servers (NFS client version). With different configurations for caching. Non producing a working environment.

I do see that a 'ls' within a directory does not send a readdir nfs operation. This means that the client seems to see that they already have the correct information in the cache.

'ls' does an stat nfs operation on current directory. I have verified that the statbuf contains proper information such as st_mtime before and after a change within a directory.

The problem is not reproducible using a linux client. So the suspected parts are the win32 backend.

We should look into on what premises the kernel nfs client actual invalidates cached data and performes a readdir nfs operation.

Comment 5 Pierre Ossman 2019-05-21 16:06:44 CEST

Adding or removing files on Windows changes mtime, but not ctime. But on a Linux system both mtime and ctime are changed. This might explain why this only happens on Windows.

Note that this behaviour is not fully supported by the documentation for stat() and friends. It doesn't say anything specific about ctime and directories, but it does state that it should be updated when the inode is updated (as opposed to the contents). However no fields reported by stat change when a file is added or removed.

The Linux nfs man page also states that only mtime is examined:

>       To detect when directory entries have been added or removed on the
>       server, the Linux NFS client watches a directory's mtime.  If the
>       client detects a change in a directory's mtime, the client drops all
>       cached LOOKUP results for that directory.  Since the directory's
>       mtime is a cached attribute, it may take some time before a client
>       notices it has changed.  See the descriptions of the acdirmin,
>       acdirmax, and noac mount options for more information about how long
>       a directory's mtime is cached.

Comment 6 Pierre Ossman 2019-05-21 16:09:36 CEST

There is a mount option called "lookupcache" which has this as part of the description:

>                      If none is specified, the client revalidates both
>                      types of directory cache entries before an application
>                      can use them.  This permits quick detection of files
>                      that were created or removed by other clients, but can
>                      impact application and server performance.

However I'm still seeing the bug even with lookupcache=none specified.

Not even "noac" works, which is documented as turning off all metadata caching.

Comment 7 Pierre Ossman 2019-05-21 18:57:03 CEST

Updating ctime seems to work, and I'm seeing some suspicious parts of the Linux kernel to confirm this.

I filed a bug with them:

https://bugzilla.kernel.org/show_bug.cgi?id=203665

But it's not realistic to wait for a fix to roll out to every system out there. We'll have to work around this.

Comment 8 Pierre Ossman 2019-05-23 12:22:38 CEST

Windows doesn't have "change time", so the plan is to go ahead and report modification time in the ctime field. That should be close enough.

Related to this is the handling of creation verifiers. If a file is created exclusively (O_EXCL), then there is some extra magic involved to handle possible retransmissions of the create command.

The client provides a random "verifier" to the server, and the server is supposed to store this in persistent storage. The recommended (and de-facto) method is to store this information in atime and mtime. The client will update these to sane values once the create has succeeded.

However FAT does not have enough space for the verifier in those fields. So instead we started using the creation time. We also chopped off some bits as there still wasn't enough room.

This happened to work smoothly because Windows reports creation time in the ctime field. If we start putting modification time in there then we'll need to change things. We probably want to rethink using creation time at all, because we don't have a good way of setting it to something sane once we're done.

Comment 9 Pierre Ossman 2019-05-23 12:25:41 CEST

So the idea for the verifier is to reduce it from 64 bits down to something that fits in atime and mtime even on FAT. This adds some risk for collision, but we should not be under such a load that it's really something to worry about*.

Fortunately we already have a good hash function in unfs with low collision risk, namely FNV1a. It gives us 32 bits out, which is small enough that it fits in atime and mtime.

* As a side note, not even the Linux kernel perfectly preserves the verifier. It chops off the highest bit in atime and mtime as having it set confuses some systems.

Comment 10 Pierre Ossman 2019-05-23 12:32:57 CEST

Whilst testing this, I discovered that _stat64i() doesn't work correctly on Windows. It will report times that are off by one hour in some cases. Not entirely sure of the conditions, but right now it is misreporting every time that is during winter time. GetFileTime() does not suffer from this fortunately, so we should probably use that instead.

As a fun note, "dir" is also reporting things incorrectly so you have to use Explorer to see proper file timestamps.

Comment 11 Pierre Ossman 2019-05-23 15:29:00 CEST

Argh! More fun obstacles!

Windows likes to store things in local time rather than UTC on FAT disks. That means it has to convert things back and forth as the API is using UTC. For the access time field that means things might shift around, as it has a resolution of whole days and cropping to just a date can then cause it to shift.

So it means we need to have some margin for these conversions when we store things. Fortunately we still have bits to spare...

Comment 12 Pierre Ossman 2019-05-23 16:49:32 CEST

There is some magic in fh_decomp() where it fakes mtime in some cases so that the NFS client will notice changes to a directory. I would assume that this also needs to consider ctime to work with current Linux kernels.

It was added because FAT doesn't track mtime for the root directory, so any files added or removed there won't get noticed.

But during testing everything works just fine. I guess modern Linux kernels have realised that an mtime of 0 means "unknown" and will always recheck the directory.

That code also seems incomplete as it will only trigger if the FAT root is also the exported directory. If I export the parent directory of where the FAT drive is mounted then the workaround never triggers.


Since things seem to work in practice I'm not touching this code and focusing on the specific Windows bug.

Comment 13 Pierre Ossman 2019-05-23 16:50:50 CEST

A thing to note is that the verifier problem mentioned earlier is a FAT problem, not a Windows problem. And FAT drives are common on Linux as well, so that fix should probably be made generic.

Comment 17 Pierre Ossman 2019-05-24 12:03:00 CEST

Should work well now. Check the acceptance criteria for things to test.

Comment 18 Peter Åstrand 2019-05-28 13:25:07 CEST

Tested with build 6138:

* Files added or removed on the client should show up right away on the server
Windows NTFS: OK
Windows FAT32: OK
Linux FAT32: OK
Linux ext4: OK
macOS: OK

* atime and mtime should be shown correctly on the server
Windows NTFS: OK
Windows FAT32: OK
Linux FAT32: OK
Linux ext4: OK
macOS: OK

* ctime should be shown correctly on the server with a Linux or macOS client
Windows NTFS: -
Windows FAT32: -
Linux FAT32: OK
Linux ext4: OK
macOS: OK

* ctime should be shown the same as mtime on the server with a Windows client
Windows NTFS: OK
Windows FAT32: OK
Linux FAT32: -
Linux ext4: -
macOS: -

* open() with O_EXCL should work fine with any client, even when exporting crappy fs like FAT
Windows NTFS: OK
Windows FAT32: OK
Linux FAT32: OK
Linux ext4: OK
macOS: OK

Comment 19 Pierre Ossman 2019-06-10 14:17:59 CEST

Something is broken. It is no longer possible to export root directories using NTFS. It gives you a "access denied" on the server when trying to access the directory.

Comment 20 Pierre Ossman 2019-06-10 15:07:47 CEST

The problem doesn't seem to be the root directory itself, but rather that there is a file in there that I don't have access to. So when we do READDIR, we also stat() that file to get the inode. But a stat() call now involves getting the FILETIMEs, which are failing with ERROR_ACCESS_DENIED.

Not sure why though since explorer and dir can both access this just fine.

Comment 22 Pierre Ossman 2019-06-10 16:21:53 CEST

Works better now.

This completely changes the calls used on Windows, so the Windows client needs to be re-tested. Test exporting C:\ to check for the latest found bug.

Comment 23 Peter Åstrand 2019-06-11 12:21:51 CEST

* Files added or removed on the client should show up right away on the server
Windows NTFS: OK
Windows FAT32: Not OK

Apparently, the time of a folder is not updated when files in that folder are added or removed.

Comment 24 Pierre Ossman 2019-06-11 12:58:23 CEST

Apparently this is per design:

"Note: The design and behavior of the FAT file system is different with regards to the modified time stamp. On a FAT file system, the modified date of a folder does not change if the contents of the folder change."

https://web.archive.org/web/20080219020154/http://support.microsoft.com/kb/299648


Looks like we need to fix a general workaround on Windows...

Comment 25 Pierre Ossman 2019-06-12 09:43:22 CEST

Note that FAT is also partially broken on Linux, even though I stated this before:

(In reply to comment #12)
> 
> But during testing everything works just fine. I guess modern Linux kernels
> have realised that an mtime of 0 means "unknown" and will always recheck the
> directory.
> 

Apparently I didn't test carefully enough. It seems like adding or removing directory works, probably because the link count of the root directory changes. But regular files are not noticed.

So we may have to make this workaround a bit more general than just Windows.

Comment 27 Peter Åstrand 2019-06-17 14:11:17 CEST

Tested with build 6157:

* Files added or removed on the client should show up right away on the server
Windows NTFS: OK
Windows FAT32: OK
Linux FAT32: OK
Linux ext4: OK
macOS: OK

* atime and mtime should be shown correctly on the server
Windows NTFS: OK
Windows FAT32: OK
Linux FAT32: OK
Linux ext4: OK
macOS: OK

* ctime should be shown correctly on the server with a Linux or macOS client
Windows NTFS: -
Windows FAT32: -
Linux FAT32: OK
Linux ext4: OK
macOS: OK

* ctime should be shown the same as mtime on the server with a Windows client
Windows NTFS: OK
Windows FAT32: OK
Linux FAT32: -
Linux ext4: -
macOS: -

* open() with O_EXCL should work fine with any client, even when exporting
crappy fs like FAT
Windows NTFS: OK
Windows FAT32: OK
Linux FAT32: OK
Linux ext4: OK
macOS: OK