gain. Configure is big enough as it is, we don't need any more useless
baggage in it.
-
-Erez & Ion
+(18) NetBSD 2.0.2, FreeBSD 4.6, OpenBSD 3.7, and quite possibly most other
+ BSDs and other OSs (as of September 2005)
+
+Some BSD kernels don't have a way to turn off the NFS attribute cache. They
+don't have a 'noac' mount flag, and setting various cache timeout fields in
+struct nfs_args doesn't turn off the attribute cache; instead, it sets the
+attribute cache timeout to some internal hard-coded default (usually
+anywhere from 5-30 seconds). If Amd cannot turn off the NFS attribute
+cache, under heavy Amd usage, users could get ESTALE errors from automounted
+symlinks, or find that those symlinks point to the wrong place. One
+workaround which would minimize this effect is to set auto_attrcache=1 in
+your amd.conf, but it doesn't eliminate the problem! The best solutions are
+(1) to use Amd in Autofs mode, if it's supported in your OS, and (2) talk to
+your OS vendor to support a true "noac" flag. See README.attrcache for more
+details.
+
+Erez & the am-utils team.
+2005-09-18 Erez Zadok <ezk@cs.sunysb.edu>
+
+ * README.attrcache: new file documenting in detail OS bugs
+ relating to attribute caching, which can affect Amd's reliability
+ under heavy load.
+
+ * doc/Makefile.am: install README.attrcache file as attrache.txt
+ file on am-utils Web site.
+
+ * Makefile.am (EXTRA_DIST): include README.attrcache in distro.
+
+ * BUGS, NEWS, doc/am-utils.texi (auto_attrcache Parameter),
+ scripts/amd.conf.5, scripts/amd.conf-sample: document attribute
+ cache bugs on some OSs.
+
2005-09-17 Erez Zadok <ezk@cs.sunysb.edu>
* libamu/xutil.c (clocktime): newly implemented routine. Uses
LSM.am-utils \
MIRRORS.txt \
MIRRORS.html \
+ README.attrcache \
README.autofs \
README.ldap \
README.osx \
Finally, we now update the mtime in places that were never updated before
(when a whole map is flushed or a single entry times out).
+Warning: some OSs, we discovered, are incapable of turning off their
+NFS attribute cache entirely. This means that Amd cannot work fully
+reliability on these systems, not under heavy load. This is documented in
+detail in the README.attrcache file with this distribution.
+
- minor new ports:
i386-pc-linux-deb3.1
powerpc-apple-darwin8.2.0
--- /dev/null
+ NFS Attribute Caching OS Problems and Amd
+ Last updated September 18, 2005
+
+* Summary:
+
+Some OSs don't seem to have a way to turn off the NFS attribute cache, which
+breaks the Amd automounter so badly that it is not recommend using Amd on
+such OS for heavy use, not until this is fixed.
+
+
+* Details:
+
+Amd is a user-level NFSv2 server that manages automounts of all other file
+systems. The kernel contacts Amd via RPCs, and Amd in turn performs the
+actual mounts, and then responds back to the kernel's RPCs. Every kernel
+caches attributes of files, in a cache called the Directory Name Lookup
+Cache (DNLC), or a Directory Cache (dcache).
+
+Amd manages its namespace in the user level, but the kernel caches names
+itself. So the two must coordinate to ensure that both namespaces are in
+sync. If the kernel uses a cached entry from the DNLC, without consulting
+Amd, users may see corruption of the automounter namespace (symlinks
+pointing to the wrong places, ESTALE errors, and more). For example,
+suppose Amd timed out an entry and removed the entry from Amd's namespace.
+Amd has to tell the kernel to purge its corresponding DNLC entry too. The
+way Amd often does that is by incrementing the last modification time
+(mtime) of the parent directory. This is the most common method for kernels
+to check if their DNLC entries are stale: if the parent directory mtime is
+newer, the kernel will discard all cached entries for that directory, and
+will re-issue lookup methods. Those lookups will result in
+NFS_GETATTR/NFS_LOOKUP calls sent from the kernel down to Amd, and Amd can
+then properly inform the kernel of the new state of automounted entries.
+
+In order to ensure that Amd is "in charge" of its namespace without
+interference from the kernel, Amd will try to turn off the NFS attribute
+cache. It does so by using the NFSMNT_NOAC flag, if it exists, or by
+setting various "cache timeout" fields in struct nfs_args to 0 (acregmin,
+acregmax, acdirmin, or acdirmax).
+
+We have released a major new version of am-utils, version 6.1, in June 2005.
+Since then, a lot of people have experimented with Amd, in anticipation of
+migrating from the very old am-utils 6.0 to the new 6.1. For a couple of
+months since the release of 6.1, we have received reports of problems with
+Amd, especially under heavy use. Users reported getting ESTALE errors from
+time to time, or seeing automounted entries whose symlinks don't point to
+where it should be. After much debugging, we traced it to a few places in
+Amd where it wasn't updating the parent directory mtime as it should have;
+in some places where Amd was indeed updating the mtime, it was using a
+resolution of only 1 second, which was not fine enough under heavy load. We
+fixed this problem and switched to using a microsecond resolution mtime.
+
+After fixing this in Amd, we went on to verify that things work for other
+OSs. When we got to FreeBSD 4.6 and NetBSD 2.0.2, we found out that they
+always cache directory entries, and there is no way to turn it off
+completely. Specifically, if we set the ac{reg,dir}{min,max} fields in
+struct nfs_args all to zero, the kernel seems to cache the entries for a
+default number of seconds (something like 5-30 seconds). On some OSs,
+setting these four fields to 0 turns off the attribute cache, but not on
+these two BSDs. We were able to verify this using Amd and a script that
+exercises the interaction of the kernel's attrcache and Amd. (If you're
+interested, the script can be made available.)
+
+We then experimented by setting the ac{reg,dir}{min,max} fields in struct
+nfs_args all to 1, the smallest non-zero value we could. When we ran the
+Amd exercising script, we found that the value of 1 reduced the race between
+the DNLC and Amd, and the script took a little longer to run before it
+detected an incoherency. That makes sense: the smaller the DNLC cache
+interval is, the shorter the window of vulnerability is. (BTW, the man
+pages on some OSs say that the ac{reg,dir}{min,max} fields use a 1 second
+resolution, but experimentation indicated it was in 0.1 second units.)
+
+Clearly, setting the ac{reg,dir}{min,max} fields to 0 is worse than setting
+it to 1 on those OSs that don't have a way to turn off the attribute cache.
+So the current workaround I've implemented in am-utils is to create a
+configuration parameter called "broken_attrcache" which, if turned on, will
+set these nfs_args fields to 1 instead of 0. I wish I didn't have to create
+such ugly workaround features in Amd, but I've got no choice.
+
+The near term solution is for FreeBSD to support a true 'noac' flag, which
+can be added fairly easily. This'd make Amd work reliably.
+
+The long term solution is to implement Autofs support for all OSs and to
+support it in Amd. Currently, Amd supports autofs on Solaris and Linux;
+FreeBSD is next. Still, we found that even with autofs support, many
+sysadmins still prefer to use the good 'ol non-autofs mode.
+
+
+* Confirmed Status
+
+This is the confirmed status of various OSs' vulnerability to this attribute
+cache bug. We are slowly checking the status of other OSs. The status of
+any OS not listed is unknown as of the date at the top of this file.
+
+** Not Vulnerable (support a proper "noac" flag):
+
+Sun Solaris 8 and 9 (10 probably works fine)
+Linux: 2.6.11 kernel (2.4.latest probably works fine)
+
+** Vulnerable (don't support a proper "noac" flag):
+
+NetBSD 2.0.2 (older versions also probably affected)
+FreeBSD 4.6 (older versions also probably affected)
+OpenBSD 3.7 (older versions also probably affected)
+
+Note: NetBSD has promised to support a noac flag hopefully after 2.1.0 is
+released (maybe in 3.0 or 2.2).
+
+
+Sincerely,
+Erez.
$(AMU_HTML_TOP)/y2k.txt: $(top_srcdir)/README.y2k
install -c -m 644 $< $@
+install-attrcache: $(AMU_HTML_TOP)/attrcache.txt
+$(AMU_HTML_TOP)/attrcache.txt: $(top_srcdir)/README.attrcache
+ install -c -m 644 $< $@
+
install-ldap: $(AMU_HTML_TOP)/ldap.txt
$(AMU_HTML_TOP)/ldap.txt: $(top_srcdir)/README.ldap
install -c -m 644 $< $@
@subsection @t{auto_attrcache} Parameter
@cindex auto_attrcache Parameter
-(type=numeric, default=0). Specify in seconds, the (kernel-side) NFS
-attribute cache timeout for @i{Amd}'s own automount points. A value
-of 0 turns off attribute caching, meaning that @i{Amd} will be
-consulted via a kernel-RPC each time someone stat's the mount point
+(type=numeric, default=0). Specify in seconds (or units of 0.1
+seconds, depending on the OS), what is the (kernel-side) NFS attribute
+cache timeout for @i{Amd}'s own automount points. A value of 0 is
+supposed to turn off attribute caching, meaning that @i{Amd} will be
+consulted via a kernel-RPC each time someone stat()'s the mount point
(which could be abused as a denial-of-service attack).
-@emph{WARNING}: if you set this option to any non-zero value,
-especially a large value, and you get ESTALE errors on your particular
-OS, then set this value back to 0 seconds.
+
+@emph{WARNING}: @i{Amd} depends on being able to turn off the NFS
+attribute cache of the client OS. If it cannot be turned off, then
+users may get ESTALE errors or symlinks that point to the wrong
+places; this is more likely under heavy use of @i{Amd}. Therefore,
+under normal circumstances, this parameter should remain set to 0, to
+ensure that the attribute cache is indeed off.
+
+Unfortunately, some kernels (e.g., certain BSDs) don't have a way to
+turn off the NFS attribute cache. Setting this parameter to 0 is
+supposed to turn off attribute caching entirely, but unfortunately it
+does not; instead, the attribute cache is set to some internal
+hard-coded default (usually anywhere from 5-30 seconds). If you
+suspect that your OS doesn't have a reliable way of turning off the
+attribute cache, then it is better to set this parameter to the
+smallest possible non-zero value (set auto_attrcache=1 in your
+@code{amd.conf}). This will not eliminate the problem, but reduce the
+risk window somewhat. The best solutions are (1) to use Amd in Autofs
+mode, if it's supported in your OS, and (2) talk to your OS vendor to
+support a true @samp{noac} flag. See the
+@uref{http://www.am-utils.org/docs/am-utils/attrcache.txt,README.attrcache}
+document for more details.
@c ----------------------------------------------------------------
@node auto_dir Parameter, cache_duration Parameter, auto_attrcache Parameter, Global Parameters
nfs_retransmit_counter_udp = 22
nfs_retry_interval_tcp = 4
nfs_retransmit_counter_tcp = 5
-# NFS attribute cache value for Amd's automount points (seconds)
+# NFS attribute cache value for Amd's automount points
+# (some OSs use this parameter in units of seconds, and others in 0.1 seconds)
+# See README.attrcache for important information about this parameter.
auto_attrcache = 0
# (amd -w)
dismount_interval = 120
.TP
.BR auto_attrcache " (numeric, default=0)"
-Specify in seconds, the (kernel-side) NFS attribute cache timeout for Amd's
-own automount points. A value of 0 turns off attribute caching, meaning
-that Amd will be consulted via a kernel-RPC each time someone stat's the
-mount point (which could be abused as a denial-of-service attack). Warning:
-if you set this option to any non-zero value, especially a large value, and
-you get ESTALE errors on your particular OS , then set this value back to 0
-seconds.
+Specify in seconds (or units of 0.1 seconds, depending on the OS), what is
+the (kernel-side) NFS attribute cache timeout for @i{Amd}'s own automount
+points. A value of 0 is supposed to turn off attribute caching, meaning
+that @i{Amd} will be consulted via a kernel-RPC each time someone stat()'s
+the mount point (which could be abused as a denial-of-service attack).
+Warning: some OSs are incapable of turning off the NFS attribute cache
+reliably. On such systems, Amd may not work reliably under heavy load. See
+the README.attrcache document in the Am-utils distribution for more details.
.TP
.BR auto_dir " (string, default=/a)"
projects / am-utils-6.1.git / commitdiff