execvp: Permission denied and rpm -qV to the rescue

Sometimes bad things happen even to good systems. In the case the box become unresponsive and had to be kicked. When the host (a VM) came back it lacked network capabilities and presented the following error on the console:


execvp: Permission denied

Digging through log files, I found the following in /var/log/boot.log:


Aug 30 08:46:32 bad-box ifup: execvp: Permission denied
Aug 30 08:46:32 bad-box network: Bringing up loopback interface: failed
Aug 30 08:46:32 bad-box ifup: execvp: Permission denied
Aug 30 08:46:32 bad-box network: Bringing up interface eth0: failed

That was odd and some googling showed similar problems even related to network interfaces, but nothing that was concrete. I decided that likely a file was corrupted and proceeded along those lines.

Here is where one of the nicer features in Redhat's RPM system came in handy.

That feature is the ability to verify the installed packages and how the installed files may differ from the original package content.

I figured it had something to do with startup scripts or kernels and so I found a list of interesting packages by running


rpm -qa | grep linux
rpm -qa | grep ip

and found that iputils and initscripts were good choices. I also tried some others, but that would only make this tale more tedious.

I then ran the verification command


[root@bad-box log]# rpm -qV iputils initscripts
S.5....T c /etc/rc.d/rc.local
SM5....T /etc/sysconfig/network-scripts/ifdown
SM5....T /etc/sysconfig/network-scripts/ifup
[root@bad-box log]# less /etc/sysconfig/network-scripts/ifdown
[root@bad-box log]# ls -l /etc/sysconfig/network-scripts/ifdown
---------- 1 root root 0 Jun 30 11:45 /etc/sysconfig/network-scripts/ifdown
[root@bad-box log]# ls -l /etc/sysconfig/network-scripts/ifup
---------- 1 root root 0 Jun 30 11:45 /etc/sysconfig/network-scripts/ifup
[root@bad-box log]#

Since it seemed odd that the files /etc/sysconfig/network-scripts/ifdown
and /etc/sysconfig/network-scripts/ifup were modified, I took a closer look and found them to be of zero size. As a comparison I examined another system and found that there the files were symlinks:


[root@good-box ~]# ls -l /etc/sysconfig/network-scripts/ifdown
lrwxrwxrwx 1 root root 20 Mar 9 21:17 /etc/sysconfig/network-scripts/ifdown -> ../../../sbin/ifdown
[root@good-box ~]# ls -l /etc/sysconfig/network-scripts/ifup
lrwxrwxrwx 1 root root 18 Mar 9 21:17 /etc/sysconfig/network-scripts/ifup -> ../../../sbin/ifup
[root@good-box ~]#

After that, I only had to replace the broken files with the proper symlinks and the system once again booted with all it's networking abilities.

Personally I think it would be great if all package management systems had the verification option. I love that feature.

\@matthias