Since some weeks I got a strange error message from the Nextcloud client on startup: Changes in synchronized folders could not be tracked reliably. At first, I ignored it, but I noticed Nextcloud wouldn’t sync local changes reliably to the server. After I lost a git commit (yes, I sync some git repos over Nextcloud), I took the issue seriously.

I knew this was related to a Linux feature called inotify. With it, you can monitor files or directories and get notified by the kernel whenever the selected events happen on those elements. I searched the Nextcloud client and nixpkgs changelogs, but didn’t find anything related to my problem. Then I searched in the Nextcloud client source code and found the dialog I saw. However, that didn’t help me either.

By pure coincidence, I noticed a separate program (qt5ct) logging a failure relating inotify: inotify_add_watch(/home/user/.config/qt5ct) failed: (No space left on device) That was the information I needed. The issue was not rooted in Nextcloud itself, but rather a Linux kernel thing. Manually starting a inotifywait command let to the following insightful message:

Failed to watch /tmp; upper limit on inotify watches reached!
Please increase the amount of inotify watches allowed per user via `/proc/sys/fs/inotify/max_user_watches'.

It’s a limit in the kernel that’s reached! And it even provides me the path to see and change it. At the same time I discovered inotify-info in nixpkgs. Its output was also very enlightening:

------------------------------------------------------------------------------
INotify Limits:
  max_queued_events    16,384
  max_user_instances   128
  max_user_watches     227,675
------------------------------------------------------------------------------
       Pid Uid        App                                   Watches  Instances
      2001 1000       .nextcloud-wrapped                    227,533          5
      2712 1000       firefox                                    16          1
      1993 1000       dbus-daemon                                10          1
      [...]
------------------------------------------------------------------------------
Total inotify Watches:   227633
Total inotify Instances: 21
------------------------------------------------------------------------------

It confirms that the system has (almost) reached the max_user_watches limit. After raising the limit with sysctl fs.inotify.max_user_watches=1048576 and restarting the Nextcloud client, the message didn’t pop up again. I had solved my issue. The client had 295,041 watches after raising the limit. Finally, I preserved this change by setting boot.kernel.sysctl."fs.inotify.max_user_watches" = 1048576; in my NixOS config. This change might lead to increased memory consumption, as each watch needs about 1KB. The 1,048,576 watches therefore might lead to about 1 GB of memory.

During debugging, I looked at the output of inotify-info on my laptop. It had a limit half of the desktop (as it has half of the RAM), but the open watches was at about 95,000, so comfortably below the limit. I assume some git repos with a huge amount of small files I have on my desktop (e.g. nixpkgs) are the culprit of this limit. They are inside a synchronized folder, but exempt from synchronization due to poor performance. I know inotify-tools provides a recursive option. However, the syscalls don’t reflect that. Therefore, the client might prevent watching excluded subdirectories of synced folders. That’d help at least me staying below the inotify watch limit.