1174294Sobrien NFS Attribute Caching OS Problems and Amd 2174294Sobrien Last updated September 18, 2005 3174294Sobrien 4174294Sobrien* Summary: 5174294Sobrien 6174294SobrienSome OSs don't seem to have a way to turn off the NFS attribute cache, which 7174294Sobrienbreaks the Amd automounter so badly that it is not recommend using Amd on 8174294Sobriensuch OS for heavy use, not until this is fixed. 9174294Sobrien 10174294Sobrien 11174294Sobrien* Details: 12174294Sobrien 13174294SobrienAmd is a user-level NFSv2 server that manages automounts of all other file 14174294Sobriensystems. The kernel contacts Amd via RPCs, and Amd in turn performs the 15174294Sobrienactual mounts, and then responds back to the kernel's RPCs. Every kernel 16174294Sobriencaches attributes of files, in a cache called the Directory Name Lookup 17174294SobrienCache (DNLC), or a Directory Cache (dcache). 18174294Sobrien 19174294SobrienAmd manages its namespace in the user level, but the kernel caches names 20174294Sobrienitself. So the two must coordinate to ensure that both namespaces are in 21174294Sobriensync. If the kernel uses a cached entry from the DNLC, without consulting 22174294SobrienAmd, users may see corruption of the automounter namespace (symlinks 23174294Sobrienpointing to the wrong places, ESTALE errors, and more). For example, 24174294Sobriensuppose Amd timed out an entry and removed the entry from Amd's namespace. 25174294SobrienAmd has to tell the kernel to purge its corresponding DNLC entry too. The 26174294Sobrienway Amd often does that is by incrementing the last modification time 27174294Sobrien(mtime) of the parent directory. This is the most common method for kernels 28174294Sobriento check if their DNLC entries are stale: if the parent directory mtime is 29174294Sobriennewer, the kernel will discard all cached entries for that directory, and 30174294Sobrienwill re-issue lookup methods. Those lookups will result in 31174294SobrienNFS_GETATTR/NFS_LOOKUP calls sent from the kernel down to Amd, and Amd can 32174294Sobrienthen properly inform the kernel of the new state of automounted entries. 33174294Sobrien 34174294SobrienIn order to ensure that Amd is "in charge" of its namespace without 35174294Sobrieninterference from the kernel, Amd will try to turn off the NFS attribute 36174294Sobriencache. It does so by using the NFSMNT_NOAC flag, if it exists, or by 37174294Sobriensetting various "cache timeout" fields in struct nfs_args to 0 (acregmin, 38174294Sobrienacregmax, acdirmin, or acdirmax). 39174294Sobrien 40174294SobrienWe have released a major new version of am-utils, version 6.1, in June 2005. 41174294SobrienSince then, a lot of people have experimented with Amd, in anticipation of 42174294Sobrienmigrating from the very old am-utils 6.0 to the new 6.1. For a couple of 43174294Sobrienmonths since the release of 6.1, we have received reports of problems with 44174294SobrienAmd, especially under heavy use. Users reported getting ESTALE errors from 45174294Sobrientime to time, or seeing automounted entries whose symlinks don't point to 46174294Sobrienwhere it should be. After much debugging, we traced it to a few places in 47174294SobrienAmd where it wasn't updating the parent directory mtime as it should have; 48174294Sobrienin some places where Amd was indeed updating the mtime, it was using a 49174294Sobrienresolution of only 1 second, which was not fine enough under heavy load. We 50174294Sobrienfixed this problem and switched to using a microsecond resolution mtime. 51174294Sobrien 52174294SobrienAfter fixing this in Amd, we went on to verify that things work for other 53174294SobrienOSs. When we got to test certain BSDs, we found out that they always cache 54174294Sobriendirectory entries, and there is no way to turn it off completely. 55174294SobrienSpecifically, if we set the ac{reg,dir}{min,max} fields in struct nfs_args 56174294Sobrienall to zero, the kernel seems to cache the entries for a default number of 57174294Sobrienseconds (something like 5-30 seconds). On some OSs, setting these four 58174294Sobrienfields to 0 turns off the attribute cache, but not on some BSDs. We were 59174294Sobrienable to verify this using Amd and a script that exercises the interaction of 60174294Sobrienthe kernel's attrcache and Amd. (If you're interested, the script can be 61174294Sobrienmade available.) 62174294Sobrien 63174294SobrienWe then experimented by setting the ac{reg,dir}{min,max} fields in struct 64174294Sobriennfs_args all to 1, the smallest non-zero value we could. When we ran the 65174294SobrienAmd exercising script, we found that the value of 1 reduced the race between 66174294Sobrienthe DNLC and Amd, and the script took a little longer to run before it 67174294Sobriendetected an incoherency. That makes sense: the smaller the DNLC cache 68174294Sobrieninterval is, the shorter the window of vulnerability is. (BTW, the man 69174294Sobrienpages on some OSs say that the ac{reg,dir}{min,max} fields use a 1 second 70174294Sobrienresolution, but experimentation indicated it was in 0.1 second units.) 71174294Sobrien 72174294SobrienClearly, setting the ac{reg,dir}{min,max} fields to 0 is worse than setting 73174294Sobrienit to 1 on those OSs that don't have a way to turn off the attribute cache. 74174294SobrienSo the current workaround I've implemented in am-utils is to create a 75174294Sobrienconfiguration parameter called "broken_attrcache" which, if turned on, will 76174294Sobrienset these nfs_args fields to 1 instead of 0. I wish I didn't have to create 77174294Sobriensuch ugly workaround features in Amd, but I've got no choice. 78174294Sobrien 79174294SobrienThe near term solution is for every OS to support a true 'noac' flag, which 80174294Sobriencan be added fairly easily. This'd make Amd work reliably. 81174294Sobrien 82174294SobrienThe long term solution is to implement Autofs support for all OSs and to 83174294Sobriensupport it in Amd. Currently, Amd supports autofs on Solaris and Linux; 84174294SobrienFreeBSD is next. Still, we found that even with autofs support, many 85174294Sobriensysadmins still prefer to use the good 'ol non-autofs mode. 86174294Sobrien 87174294Sobrien 88174294Sobrien* Confirmed Status 89174294Sobrien 90174294SobrienThis is the confirmed status of various OSs' vulnerability to this attribute 91174294Sobriencache bug. We are slowly checking the status of other OSs. The status of 92174294Sobrienany OS not listed is unknown as of the date at the top of this file. 93174294Sobrien 94174294Sobrien** Not Vulnerable (support a proper "noac" flag): 95174294Sobrien 96174294SobrienSun Solaris 8 and 9 (10 probably works fine) 97174294SobrienLinux: 2.6.11 kernel (2.4.latest probably works fine) 98174294SobrienFreeBSD 5.4 and 6.0-SNAP001 (older versions probably work fine) 99174294SobrienOpenBSD 3.7 (older versions probably work fine) 100174294Sobrien 101174294Sobrien** Vulnerable (don't support a proper "noac" flag natively): 102174294Sobrien 103174294SobrienNetBSD 2.0.2 (older versions are also probably affected) 104174294Sobrien 105174294SobrienNote: NetBSD has promised to support a noac flag hopefully after 2.1.0 is 106174294Sobrienreleased (maybe in 3.0 or 2.2). In the mean time, you can apply one of 107174294Sobrienthese two kernel patchs to support a 'noac' flag in NetBSD 2.x or 3.x: 108174294Sobrien ftp://ftp.netbsd.org/pub/NetBSD/misc/christos/2x.nfs.noac.diff 109174294Sobrien ftp://ftp.netbsd.org/pub/NetBSD/misc/christos/3x.nfs.noac.diff 110174294SobrienAfter applying this patch and rebuilding your kernel, reboot with the new 111174294Sobrienkernel. Then copy the new nfs.h and nfsmount.h from /sys/nfs/ to 112174294Sobrien/usr/include/nfs/, and finally rebuild am-utils from scratch. 113174294Sobrien 114174294Sobrien** Testing 115174294Sobrien 116174294SobrienWhen you build am-utils, a script named scripts/test-attrcache is built, 117174294Sobrienwhich can be used to test the NFS attribute cache behavior of the current 118174294SobrienOS. You can run this script as root as follows: 119174294Sobrien 120174294Sobrien# make install 121174294Sobrien# cd scripts 122174294Sobrien# sh test-attrcache 123174294Sobrien 124174294SobrienIf you run this script on an OS whose status is known (and not listed 125174294Sobrienabove), please report it to am-utils@am-utils.org, so we can record it in 126174294Sobrienthis file. 127174294Sobrien 128174294SobrienSincerely, 129174294SobrienErez. 130