=inotify= file limits can be confusing/misleading

2025-01-16T10:59:00Z

`inotify` file limits can be confusing/misleading

I've recently been watching a build for a package on my server fail inconsistently ~50% of the times I build updates, and I was confused as to why it was missing the binary cache in the first. Oh fun, how reproduceable. I spent an hour or two yesterday reading the code and the failing tests and realized it was because the test was stress-testing inotify system calls:

  Traceback (most recent call last):
    File "/nix/store/zv1kaq7f1q20x62kbjv6pfjygw5jmwl6-python3-3.12.7/lib/python3.12/threading.py", line 1075, in _bootstrap_inner
      self.run()
    File "/build/source/src/documents/tests/test_management_consumer.py", line 30, in run
      self.cmd.handle(directory=settings.CONSUMPTION_DIR, oneshot=False, testing=True)
    File "/build/source/src/documents/management/commands/document_consumer.py", line 251, in handle
      self.handle_inotify(directory, recursive, options["testing"])
    File "/build/source/src/documents/management/commands/document_consumer.py", line 294, in handle_inotify
      inotify = INotify()
                ^^^^^^^^^
    File "/nix/store/3ziqbc4xcs58hhh5srx7pfl2n9mwj22g-python3.12-inotifyrecursive-0.3.5/lib/python3.12/site-packages/inotifyrecursive/inotifyrecursive.py", line 31, in __init__
      inotify_simple.INotify.__init__(self)
    File "/nix/store/1qv923rjqijj7nbhhm9k1bz53jh9pb3a-python3.12-inotify-simple-1.3.5/lib/python3.12/site-packages/inotify_simple.py", line 91, in __init__
      FileIO.__init__(self, _libc_call(_libc.inotify_init1, flags), mode='rb')
                            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    File "/nix/store/1qv923rjqijj7nbhhm9k1bz53jh9pb3a-python3.12-inotify-simple-1.3.5/lib/python3.12/site-packages/inotify_simple.py", line 39, in _libc_call
      raise OSError(errno, os.strerror(errno))
  OSError: [Errno 24] Too many open files

inotify is an API that allows your process to be notified when files are accessed, changed, deleted, etc. Because inotify queues can use a fair bit of memory, Linux implements specific interfaces to limit these calls:

The following interfaces can be used to limit the amount of kernel memory consumed by inotify:
/proc/sys/fs/inotify/max_queued_events The value in this file is used when an application calls inotify_init(2) to set an upper limit on the number of events that can be queued to the corresponding inotify instance. Events in excess of this limit are dropped, but an IN_Q_OVERFLOW event is always generated.
/proc/sys/fs/inotify/max_user_instances This specifies an upper limit on the number of inotify instances that can be created per real user ID.
/proc/sys/fs/inotify/max_user_watches This specifies an upper limit on the number of watches that can be created per real user ID.

I found a Nixpkgs GitHub Issue about the build failure where folks saw the "too many open files" and assumed it was ulimit configuration issues on the build hosts that stumped even the NixOS Super Posters, but it's this other more obscure limit that nevertheless raises the same Errno when it's hit. Classic Linux! These values are set to arbitrarily low values by default, and I remembered that the inotify watcher in the The Arcology Project's FastAPI prototype bumped up against these limits when I deployed it way back when. Classic Linux!

There are two machines in /etc/nix/machines, my Framework 13 Laptop and My Homelab Build. Presumably the build works on one but not the other, but I no longer explicitly set these values, so something in NixOS itself must be, this is easy enough to check:

shell:results drawer

pushd ~/arroyo-nix
grep -ri fs.inotify.max_user_watches
pushd ~/Code/nixpkgs
grep -ri fs.inotify.max_user_watches

:results: ~/arroyo-nix ~/org ~/Code/nixpkgs ~/arroyo-nix ~/org nixos/modules/services/misc/graphical-desktop.nix: "fs.inotify.max_user_watches" = lib.mkDefault 524288; nixos/modules/virtualisation/lxd.nix: "fs.inotify.max_user_watches" = 1048576; nixos/modules/virtualisation/incus.nix: "fs.inotify.max_user_watches" = lib.mkOverride 1050 1048576; # override in case conflict nixos/modules/services/x11/xserver.nix :END:

So it's set to a higher value by enabling LXD (which I believe Waydroid does) but also by enabling any graphical desktop. So the package would build on my laptop but not my server or any "stock" NixOS server... a 50-50 shot.

sudo sysctl -w fs.inotify.max_user_watches=524288
sudo sysctl -w fs.inotify.max_user_instances=524288

I set this temporarily on my server and the build ran reliably 5 times though the tests are sooooooo freakin' slow... I hope the Hydra instance that publishes the binary cache gets around to something like this, rather than disabling the tests that validate that document consumption in this document scanning/processing service works properly....

It can be set "for good" with this NixOS configuration:

nix

boot.kernel.sysctl = {
  "fs.inotify.max_user_instances" = 524288;
  "fs.inotify.max_user_watches" = 524288;
};

Count 502s from each URL from web access logs

2023-11-01T21:00:00Z

Count 502s from each URL from web access logs

shell

sudo tail -n 10000 /var/log/nginx/access.log | awk '$10 == "502" {print $10 " " $1 " " $8}' | sort | uniq -c | sort -n

Shell Grymoire

=inotify= file limits can be confusing/misleading

inotify file limits can be confusing/misleading

Count 502s from each URL from web access logs

Count 502s from each URL from web access logs

`inotify` file limits can be confusing/misleading