Disaster recovery after a partition restore
Unforeseen Consequences
After restoring a root partition / from backup, I found myself unable to start X as my regular user. The curious part? It worked fine when root’s X session was running first. Something was initializing the GPU when root logged in, and my user session was benefiting from it.
The first clue was sudo not working properly. Man, this is not looking good, I must have missed the flag that preserves the mode when making the root partition / backup.
sudo: /usr/bin/sudo must be owned by uid 0 and have the setuid bit set
Don’t panic yet, its time to investigate. I hope you have your root password nearby, we are going to need it.
The symptoms
Other than sudo being broken, the next clue was nvidia-persistenced.service. This daemon keeps the NVIDIA driver loaded and ensures device nodes exist at boot. Checking its status revealed the problem:
systemctl status nvidia-persistenced.service
× nvidia-persistenced.service - NVIDIA Persistence Daemon
Loaded: loaded (/usr/lib/systemd/system/nvidia-persistenced.service; enabled; preset: disabled)
Active: failed (Result: exit-code) since Sat 2026-01-17 22:12:57 CET; 1min 39s ago
Process: 727 ExecStart=/usr/bin/nvidia-persistenced --user nvidia-persistenced (code=exited, status=1/FAILURE)
Jan 17 22:12:57 arch nvidia-persistenced[730]: Failed to query NVIDIA devices. Please ensure that the NVIDIA device files (/dev/nvidia*) exist, and that user 143 has read and write permissions for those files.
Jan 17 22:12:57 arch nvidia-persistenced[730]: Shutdown (730)
Jan 17 22:12:57 arch systemd[1]: nvidia-persistenced.service: Failed with result 'exit-code'.
I don’t recognize that user, but it being such a small number it must be a system user. Lets find out:
id 143
uid=143(nvidia-persistenced) gid=143(nvidia-persistenced) groups=143(nvidia-persistenced)
The service was failing because /dev/nvidia* device nodes didn’t exist at boot. A classic chicken-and-egg problem: the persistence daemon needs device nodes to start, but the device nodes need something else to create them first.
How NVIDIA device nodes are created
On Linux, NVIDIA device nodes are typically created by nvidia-modprobe. This utility is called by udev rules when the driver loads:
cat /usr/lib/udev/rules.d/60-nvidia.rules
# Device nodes are created by nvidia-modprobe, which is called by the nvidia DDX.
# In case the DDX is not started, the device nodes are never created, so call
# nvidia-modprobe in the udev rules to cover that case.
ACTION=="add", DEVPATH=="/bus/pci/drivers/nvidia", RUN+="/usr/bin/nvidia-modprobe -c 0 -u"
The key here is that nvidia-modprobe needs to create device nodes owned by root. To do this as a non-root process (like early boot services), it requires the setuid bit to be set.
Finding the culprit
Let’s check the permissions on nvidia-modprobe:
ls -la /usr/bin/nvidia-modprobe
-rwxr-xr-x 1 root root 47568 Jan 17 19:42 /usr/bin/nvidia-modprobe
There it is. The permissions show -rwxr-xr-x — no setuid bit. It should be -rwsr-xr-x (note the s in the owner execute position).
Without setuid, nvidia-modprobe cannot create device nodes when called by non-root processes or during early boot. The backup restoration had stripped this special permission.
The
setuidbit allows a program to run with the permissions of its owner (root in this case), regardless of who executes it. This is essential for utilities that need elevated privileges for specific operations.
The fix
Restoring the setuid bit is straightforward:
chmod u+s /usr/bin/nvidia-modprobe
Verify the permissions are correct:
ls -la /usr/bin/nvidia-modprobe
-rwsr-xr-x 1 root root 47568 Jan 17 19:42 /usr/bin/nvidia-modprobe
Now restart the persistence daemon:
systemctl restart nvidia-persistenced.service
systemctl status nvidia-persistenced.service
● nvidia-persistenced.service - NVIDIA Persistence Daemon
Loaded: loaded (/usr/lib/systemd/system/nvidia-persistenced.service; enabled; preset: disabled)
Active: active (running) since Sun 2026-01-18 10:01:19 CET; 40min ago
Invocation: 1b79f56787fa46fc8702624579ba9518
Process: 730 ExecStart=/usr/bin/nvidia-persistenced --user nvidia-persistenced (code=exited, status=0/SUCCESS)
Main PID: 733 (nvidia-persiste)
Tasks: 1 (limit: 76756)
Memory: 628K (peak: 2M)
CPU: 6ms
CGroup: /system.slice/nvidia-persistenced.service
└─733 /usr/bin/nvidia-persistenced --user nvidia-persistenced
After a reboot, X started correctly as my regular user without needing root’s session first.
Alternative fix: reinstall the package
If you prefer to restore all default permissions at once, you can reinstall the nvidia-utils package:
pacman -S nvidia-utils
This will restore any permissions that were lost during the backup restoration, including the setuid bit on nvidia-modprobe.
OK… Who’s not dead? Sound off!
If sudo and nvidia-modprobe lost their permissions, chances are other files did too.
You could now go and check every package with zcat and check the mode on the mtree.
zcat /var/lib/pacman/local/nvidia-utils-590.48.01-2/mtree | grep nvidia-modprobe
./usr/bin/nvidia-modprobe time=1767476285.0 mode=4755 size=47568 sha256digest=42f922246c930d373c543cb7c4c5fa3d254720d807c2e21ea9c7995c4ec2f020
Rather than playing mouse and cat for days with individual binaries, we can ask pacman to verify all installed packages.
Pacman’s -Qkk flag performs an extended check that includes file permissions, ownership, and modification times pacman(8) [2].
Since all my modification times are not messed up after the backup, I filter those out, leaving only those I’m interested about.
pacman -Qkk 2>&1 | grep -E "(Permissions mismatch|GID mismatch)" | grep -v "Modification time"
This command may take a while, but once finished the output will look something like this for each package:
warning: nvidia-utils: /usr/bin/nvidia-modprobe (Permissions mismatch)
You can extract the unique package names and save them to a file:
pacman -Qkk 2>&1 | grep -E '(Permissions|UID|GID)' | cut -d: -f2 | sort -u > /tmp/broken-packages.txt
Review the list if you want, then pass it to pacman to reinstall everything with correct permissions:
pacman -Sy $(cat /tmp/broken-packages.txt)
This will download fresh copies and restore the correct permissions for all affected files.
If you see any warnings from pacman while installing about permissions:
warning: directory permissions differ on /var/games/
filesystem: 755 package: 775
warning: directory permissions differ on /var/spool/mail/
filesystem: 1755 package: 1777
warning: directory permissions differ on /var/named/
filesystem: 750 package: 770
You can fix those using the package permissions with chmod:
chmod 775 /var/games/
chmod 1777 /var/spool/mail/
chmod 770 /var/named/
Lessons learned
Backup restorations can silently strip special permissions like setuid bits. These permissions are easy to overlook because the files themselves appear intact. The file contents are still there, so its not a catastrophe.
When doing backups and restoring from them, don’t forget using flags that preserve permissions like rsync -a, tar -p or cp -p.
For NVIDIA specifically, remember that nvidia-modprobe requires setuid to function correctly at boot. Without it, device nodes won’t be created until something with root privileges accesses the GPU.