Power Outage During a Fedora Update

This past weekend I was updating my Fedora 30 laptop, when the battery ran out of power and my laptop shutoff--very inconvenient. I had no idea whether or not that was going to be a problem, but, since I am writing this post, you can guess that it was a problem. Basically, I received the Fedora booting screen, but it got to some point and the progress wheel was no longer spinning. I left it for some time and it didn't ever fully boot up. Further, no matter what option I used on the boot menu, I received the same result.

Now What?

At that point I had a few options, first, reinstall Fedora without touching my user data (I've had some success with that) or, second, see if there is some way of repairing Fedora. Of course, the latter option would save a lot of time and effort, so I pursued that course of action.

With a Google search for power failure during fedora upgrade, I found a Fedora bug report that discussed a problem someone reported when they had a power outage during a full dnf system-upgrade, which can be used to upgrade Fedora from one major version to another. I have successfully used dnf system-upgrade on lots of computers, but, thankfully, I've never experienced what happens when the power goes up in the middle of the OS upgrade. There looks like there are several useful comments in the article if that is the problem you are experiencing.

Repairing a Broken System with DNF

I mention the article, because it provided a reference to a nice blog post entitled Repair of broken system with DNF-2.0. Be aware that the link in the previous sentence will probably be incorrect in the future since the blog seems to not provide static links to articles, as best as I can tell. You should still seek out the article on the DNF Team's Blog when it moves.

LUKS and Fedora 31 Xfce Spin

Following the lead of the article, I booted the system with a Fedora 31 Xfce Spin installation USB stick to see what I could do. The first issue I had to deal with is that my laptop's SSD is encrypted with LUKS. I have dealt with accessing LUKS-encrypted volumes under GNOME (it seems like the GNOME Disks utility has been convenient in the past for this), but I wondered if the Fedora 31 Xfce installer had the utilities to access a disk encrypted by LUKS. To my pleasant surprise, it was built into the desktop displayed by the installer. There was an icon with the label "1.0 TB Encrypted" on the desktop.

Encrypted Hard Drive Icon

On a whim, I double-clicked on the drive icon and a handy dialog box popped up, asking for the disk password.

LUKS password dialog box

When I provided the password, I received a notice that it wasn't able to mount the drive.

Mount Failure Notice Xfce

At first, this was disconcerting, but I noticed that a drive icon with "fedora-root" as the label showed up.

fedora-root icon

Double clicking on that icon, the root filesystem for the Fedora 30 machine was mounted without hunting for any cryptic command-line commands!

Mounted fedora-root in file
manager

As noted in the above image, the root filesystem was mounted at /run/media/liveuser/fedora-root.

Fixing the Packages with DNF

Following the suggestions of Repair of broken system with DNF-2.0, I then dropped into a terminal window and became root (sudo -i). At this point, I could execute the following command to check the status of the system:

dnf --installroot /run/media/liveuser/fedora-root check

which returned:

grub2-common-1:2.02-87.fc30.noarch is a duplicate with grub2-common-1:2.02-88.fc30.noarch
grub2-efi-ia32-1:2.02-87.fc30.x86_64 is a duplicate with grub2-efi-ia32-1:2.02-88.fc30.x86_64
grub2-efi-ia32-cdboot-1:2.02-87.fc30.x86_64 is a duplicate with grub2-efi-ia32-cdboot-1:2.02-88.fc30.x86_64
grub2-efi-x64-1:2.02-87.fc30.x86_64 is a duplicate with grub2-efi-x64-1:2.02-88.fc30.x86_64
grub2-efi-x64-cdboot-1:2.02-87.fc30.x86_64 is a duplicate with grub2-efi-x64-cdboot-1:2.02-88.fc30.x86_64
grub2-pc-1:2.02-87.fc30.x86_64 is a duplicate with grub2-pc-1:2.02-88.fc30.x86_64
grub2-pc-modules-1:2.02-87.fc30.noarch is a duplicate with grub2-pc-modules-1:2.02-88.fc30.noarch
grub2-tools-1:2.02-87.fc30.x86_64 is a duplicate with grub2-tools-1:2.02-88.fc30.x86_64
grub2-tools-1:2.02-87.fc30.x86_64 is obsoleted by grub2-tools-1:2.02-88.fc30.x86_64
grub2-tools-1:2.02-87.fc30.x86_64 is obsoleted by grub2-tools-efi-1:2.02-88.fc30.x86_64
grub2-tools-1:2.02-87.fc30.x86_64 is obsoleted by grub2-tools-extra-1:2.02-88.fc30.x86_64
grub2-tools-1:2.02-87.fc30.x86_64 is obsoleted by grub2-tools-minimal-1:2.02-88.fc30.x86_64
grub2-tools-efi-1:2.02-87.fc30.x86_64 is a duplicate with grub2-tools-efi-1:2.02-88.fc30.x86_64
grub2-tools-extra-1:2.02-87.fc30.x86_64 is a duplicate with grub2-tools-extra-1:2.02-88.fc30.x86_64
grub2-tools-minimal-1:2.02-87.fc30.x86_64 is a duplicate with grub2-tools-minimal-1:2.02-88.fc30.x86_64
java-latest-openjdk-1:13.0.1.9-2.rolling.fc30.x86_64 is a duplicate with java-latest-openjdk-1:13.0.2.8-1.rolling.fc30.x86_64
java-latest-openjdk-headless-1:13.0.1.9-2.rolling.fc30.x86_64 is a duplicate with java-latest-openjdk-headless-1:13.0.2.8-1.rolling.fc30.x86_64
libsolv-0.7.10-1.fc30.x86_64 is a duplicate with libsolv-0.7.11-1.fc30.x86_64
libxcrypt-4.4.10-2.fc30.x86_64 is a duplicate with libxcrypt-4.4.11-1.fc30.x86_64
libxcrypt-compat-4.4.10-2.fc30.x86_64 is a duplicate with libxcrypt-compat-4.4.11-1.fc30.x86_64
libxcrypt-devel-4.4.10-2.fc30.x86_64 is a duplicate with libxcrypt-devel-4.4.11-1.fc30.x86_64
nss-3.48.0-1.fc30.x86_64 is a duplicate with nss-3.49.0-1.fc30.x86_64
nss-mdns-0.14.1-3.fc30.x86_64 is a duplicate with nss-mdns-0.14.1-5.fc30.x86_64
nss-softokn-3.48.0-1.fc30.x86_64 is a duplicate with nss-softokn-3.49.0-1.fc30.x86_64
nss-softokn-freebl-3.48.0-1.fc30.x86_64 is a duplicate with nss-softokn-freebl-3.49.0-1.fc30.x86_64
nss-sysinit-3.48.0-1.fc30.x86_64 is a duplicate with nss-sysinit-3.49.0-1.fc30.x86_64
nss-tools-3.48.0-1.fc30.x86_64 is a duplicate with nss-tools-3.49.0-1.fc30.x86_64
nss-util-3.48.0-1.fc30.x86_64 is a duplicate with nss-util-3.49.0-1.fc30.i686
nss-util-3.48.0-1.fc30.x86_64 is a duplicate with nss-util-3.49.0-1.fc30.x86_64
pcre2-10.33-16.fc30.x86_64 is a duplicate with pcre2-10.34-4.fc30.x86_64
pcre2-utf16-10.33-16.fc30.x86_64 is a duplicate with pcre2-utf16-10.34-4.fc30.x86_64
selinux-policy-3.14.3-53.fc30.noarch is a duplicate with selinux-policy-3.14.3-55.fc30.noarch
selinux-policy-targeted-3.14.3-53.fc30.noarch is a duplicate with selinux-policy-targeted-3.14.3-55.fc30.noarch
tigervnc-license-1.10.1-1.fc30.noarch is a duplicate with tigervnc-license-1.10.1-2.fc30.noarch
tigervnc-server-minimal-1.10.1-1.fc30.x86_64 is a duplicate with tigervnc-server-minimal-1.10.1-2.fc30.x86_64
tzdata-2019c-1.fc30.noarch is a duplicate with tzdata-2019c-2.fc30.noarch
tzdata-java-2019c-1.fc30.noarch is a duplicate with tzdata-java-2019c-2.fc30.noarch
virtualbox-guest-additions-6.1.0-1.fc30.x86_64 is a duplicate with virtualbox-guest-additions-6.1.2-1.fc30.x86_64
xen-libs-4.11.3-2.fc30.x86_64 is a duplicate with xen-libs-4.11.3-3.fc30.x86_64
xen-licenses-4.11.3-2.fc30.x86_64 is a duplicate with xen-licenses-4.11.3-3.fc30.x86_64
zchunk-libs-1.1.4-1.fc30.x86_64 is a duplicate with zchunk-libs-1.1.5-1.fc30.x86_64

So, it looks like the package update process had installed the newer packages but hadn't yet removed the old ones. The ones that were probably keeping the system from booting were the grub packages.

After some trial and error (and based on the DNF blog post), I manually removed the older packages one by one, for example:

dnf --installroot /run/media/liveuser/fedora-root remove zchunk-libs-1.1.4-1.fc30.x86_64

This was because the recommended command:

dnf --installroot /run/media/liveuser/fedora-root remove --duplicates

did not work.

After removing all of the older packages, the dnf check seemed to be clear.

I did have a situation where a newer package (pcre2-utf16-10.34-4.fc30.x86_64) did not appear to be properly installed. To fix this, as the DNF blog post said, I ran:

dnf --installroot /run/media/liveuser/fedora-root reinstall pcre2-utf16-10.34-4.fc30.x86_64

and that seemed to fix the problem.

To see what other packages had problems I was also able to run the following as well:

rpm -Va --root /run/media/liveuser/fedora-root

and looked for files from packages that looked very broken. If there appears to be a file with a lot of problems, you can run:

rpm -q -f --root /run/media/liveuser/fedora-root <path to file>

to see what package owns the file. I think the <path to file> in this case is probably relative to the root directory in normal use, e.g., /etc/hosts instead of /run/media/liveuser/fedora-root/etc/hosts. I am writing this part from memory. If it doesn't work with the "normal" absolute path, maybe try with "as-mounted" absolute path instead.

In those cases, running dnf reinstall on suspect packages seems like a good approach.

Note that for this entire rescue, I did have Internet connectivity for the Live Fedora image that I was running so that dnf could actually pull down files from repositories.

As another recommendation, if you have to remove package for some reason and want to re-install it, you don't necessarily have to use the full version in the package name. For example, you can install the package with just the base package name like this:

dnf --installroot /run/media/liveuser/fedora-root install pcre2-utf16.x86_64

or

dnf --installroot /run/media/liveuser/fedora-root install pcre2-utf16

This way you don't have to type in as much information. For the dnf remove operations above, though, you must use full name of the file so you can specify the older package.

Moment of Truth

Once I got no errors when running dnf check and addressed the problem with the pcre2-utf16.x86_64 package with a dnf reinstall, it was time to reboot the laptop. Considering that grub boot loader packages were in the process of being updated, I was a suspicious that the whole rescue attempt would work, but I rebooted and, lo and behold, the laptop booted as normal. I think the fact that the laptop had installed the new packages and was in the process or removing the old packages might have helped my chances.

Anyway, I am impressed with the capabilities of both dnf and rpm for this sort of rescue operation and I was able get on with my normal work with relatively little effort. Definitely a positive Fedora experience.