Recovering Fedora 30 after a Power Outage During an Update
Power Outage During a Fedora Update
This past weekend I was updating my Fedora 30 laptop, when the battery ran out of power and my laptop shutoff–very inconvenient. I had no idea whether or not that was going to be a problem, but, since I am writing this post, you can guess that it was a problem. Basically, I received the Fedora booting screen, but it got to some point and the progress wheel was no longer spinning. I left it for some time and it didn’t ever fully boot up. Further, no matter what option I used on the boot menu, I received the same result.
Now What?
At that point I had a few options, first, reinstall Fedora without touching my user data (I’ve had some success with that) or, second, see if there is some way of repairing Fedora. Of course, the latter option would save a lot of time and effort, so I pursued that course of action.
With a Google search for power failure during fedora upgrade, I
found a Fedora bug
report that
discussed a problem someone reported when they had a power outage
during a full dnf system-upgrade, which can be used to upgrade
Fedora from one major version to
another.
I have successfully used dnf system-upgrade on lots of computers,
but, thankfully, I’ve never experienced what happens when the power
goes up in the middle of the OS upgrade. There looks like there are
several useful comments in the article if that is the problem you are
experiencing.
Repairing a Broken System with DNF
I mention the article, because it provided a reference to a nice blog post entitled Repair of broken system with DNF-2.0. Be aware that the link in the previous sentence will probably be incorrect in the future since the blog seems to not provide static links to articles, as best as I can tell. You should still seek out the article on the DNF Team’s Blog when it moves.
LUKS and Fedora 31 Xfce Spin
Following the lead of the article, I booted the system with a Fedora 31 Xfce Spin installation USB stick to see what I could do. The first issue I had to deal with is that my laptop’s SSD is encrypted with LUKS. I have dealt with accessing LUKS-encrypted volumes under GNOME (it seems like the GNOME Disks utility has been convenient in the past for this), but I wondered if the Fedora 31 Xfce installer had the utilities to access a disk encrypted by LUKS. To my pleasant surprise, it was built into the desktop displayed by the installer. There was an icon with the label “1.0 TB Encrypted” on the desktop.
On a whim, I double-clicked on the drive icon and a handy dialog box popped up, asking for the disk password.
When I provided the password, I received a notice that it wasn’t able to mount the drive.
At first, this was disconcerting, but I noticed that a drive icon with “fedora-root” as the label showed up.
Double clicking on that icon, the root filesystem for the Fedora 30 machine was mounted without hunting for any cryptic command-line commands!
As noted in the above image, the root filesystem was mounted at
/run/media/liveuser/fedora-root.
Fixing the Packages with DNF
Following the suggestions of Repair of broken system with
DNF-2.0, I then
dropped into a terminal window and became root (sudo -i). At this
point, I could execute the following command to check the status of
the system:
dnf --installroot /run/media/liveuser/fedora-root check
which returned:
grub2-common-1:2.02-87.fc30.noarch is a duplicate with grub2-common-1:2.02-88.fc30.noarch
grub2-efi-ia32-1:2.02-87.fc30.x86_64 is a duplicate with grub2-efi-ia32-1:2.02-88.fc30.x86_64
grub2-efi-ia32-cdboot-1:2.02-87.fc30.x86_64 is a duplicate with grub2-efi-ia32-cdboot-1:2.02-88.fc30.x86_64
grub2-efi-x64-1:2.02-87.fc30.x86_64 is a duplicate with grub2-efi-x64-1:2.02-88.fc30.x86_64
grub2-efi-x64-cdboot-1:2.02-87.fc30.x86_64 is a duplicate with grub2-efi-x64-cdboot-1:2.02-88.fc30.x86_64
grub2-pc-1:2.02-87.fc30.x86_64 is a duplicate with grub2-pc-1:2.02-88.fc30.x86_64
grub2-pc-modules-1:2.02-87.fc30.noarch is a duplicate with grub2-pc-modules-1:2.02-88.fc30.noarch
grub2-tools-1:2.02-87.fc30.x86_64 is a duplicate with grub2-tools-1:2.02-88.fc30.x86_64
grub2-tools-1:2.02-87.fc30.x86_64 is obsoleted by grub2-tools-1:2.02-88.fc30.x86_64
grub2-tools-1:2.02-87.fc30.x86_64 is obsoleted by grub2-tools-efi-1:2.02-88.fc30.x86_64
grub2-tools-1:2.02-87.fc30.x86_64 is obsoleted by grub2-tools-extra-1:2.02-88.fc30.x86_64
grub2-tools-1:2.02-87.fc30.x86_64 is obsoleted by grub2-tools-minimal-1:2.02-88.fc30.x86_64
grub2-tools-efi-1:2.02-87.fc30.x86_64 is a duplicate with grub2-tools-efi-1:2.02-88.fc30.x86_64
grub2-tools-extra-1:2.02-87.fc30.x86_64 is a duplicate with grub2-tools-extra-1:2.02-88.fc30.x86_64
grub2-tools-minimal-1:2.02-87.fc30.x86_64 is a duplicate with grub2-tools-minimal-1:2.02-88.fc30.x86_64
java-latest-openjdk-1:13.0.1.9-2.rolling.fc30.x86_64 is a duplicate with java-latest-openjdk-1:13.0.2.8-1.rolling.fc30.x86_64
java-latest-openjdk-headless-1:13.0.1.9-2.rolling.fc30.x86_64 is a duplicate with java-latest-openjdk-headless-1:13.0.2.8-1.rolling.fc30.x86_64
libsolv-0.7.10-1.fc30.x86_64 is a duplicate with libsolv-0.7.11-1.fc30.x86_64
libxcrypt-4.4.10-2.fc30.x86_64 is a duplicate with libxcrypt-4.4.11-1.fc30.x86_64
libxcrypt-compat-4.4.10-2.fc30.x86_64 is a duplicate with libxcrypt-compat-4.4.11-1.fc30.x86_64
libxcrypt-devel-4.4.10-2.fc30.x86_64 is a duplicate with libxcrypt-devel-4.4.11-1.fc30.x86_64
nss-3.48.0-1.fc30.x86_64 is a duplicate with nss-3.49.0-1.fc30.x86_64
nss-mdns-0.14.1-3.fc30.x86_64 is a duplicate with nss-mdns-0.14.1-5.fc30.x86_64
nss-softokn-3.48.0-1.fc30.x86_64 is a duplicate with nss-softokn-3.49.0-1.fc30.x86_64
nss-softokn-freebl-3.48.0-1.fc30.x86_64 is a duplicate with nss-softokn-freebl-3.49.0-1.fc30.x86_64
nss-sysinit-3.48.0-1.fc30.x86_64 is a duplicate with nss-sysinit-3.49.0-1.fc30.x86_64
nss-tools-3.48.0-1.fc30.x86_64 is a duplicate with nss-tools-3.49.0-1.fc30.x86_64
nss-util-3.48.0-1.fc30.x86_64 is a duplicate with nss-util-3.49.0-1.fc30.i686
nss-util-3.48.0-1.fc30.x86_64 is a duplicate with nss-util-3.49.0-1.fc30.x86_64
pcre2-10.33-16.fc30.x86_64 is a duplicate with pcre2-10.34-4.fc30.x86_64
pcre2-utf16-10.33-16.fc30.x86_64 is a duplicate with pcre2-utf16-10.34-4.fc30.x86_64
selinux-policy-3.14.3-53.fc30.noarch is a duplicate with selinux-policy-3.14.3-55.fc30.noarch
selinux-policy-targeted-3.14.3-53.fc30.noarch is a duplicate with selinux-policy-targeted-3.14.3-55.fc30.noarch
tigervnc-license-1.10.1-1.fc30.noarch is a duplicate with tigervnc-license-1.10.1-2.fc30.noarch
tigervnc-server-minimal-1.10.1-1.fc30.x86_64 is a duplicate with tigervnc-server-minimal-1.10.1-2.fc30.x86_64
tzdata-2019c-1.fc30.noarch is a duplicate with tzdata-2019c-2.fc30.noarch
tzdata-java-2019c-1.fc30.noarch is a duplicate with tzdata-java-2019c-2.fc30.noarch
virtualbox-guest-additions-6.1.0-1.fc30.x86_64 is a duplicate with virtualbox-guest-additions-6.1.2-1.fc30.x86_64
xen-libs-4.11.3-2.fc30.x86_64 is a duplicate with xen-libs-4.11.3-3.fc30.x86_64
xen-licenses-4.11.3-2.fc30.x86_64 is a duplicate with xen-licenses-4.11.3-3.fc30.x86_64
zchunk-libs-1.1.4-1.fc30.x86_64 is a duplicate with zchunk-libs-1.1.5-1.fc30.x86_64
So, it looks like the package update process had installed the newer
packages but hadn’t yet removed the old ones. The ones that were
probably keeping the system from booting were the grub packages.
After some trial and error (and based on the DNF blog post), I manually removed the older packages one by one, for example:
dnf --installroot /run/media/liveuser/fedora-root remove zchunk-libs-1.1.4-1.fc30.x86_64
This was because the recommended command:
dnf --installroot /run/media/liveuser/fedora-root remove --duplicates
did not work.
After removing all of the older packages, the dnf check seemed to be clear.
I did have a situation where a newer package
(pcre2-utf16-10.34-4.fc30.x86_64) did not appear to be properly
installed. To fix this, as the DNF blog post said, I ran:
dnf --installroot /run/media/liveuser/fedora-root reinstall pcre2-utf16-10.34-4.fc30.x86_64
and that seemed to fix the problem.
To see what other packages had problems I was also able to run the following as well:
rpm -Va --root /run/media/liveuser/fedora-root
and looked for files from packages that looked very broken. If there appears to be a file with a lot of problems, you can run:
rpm -q -f --root /run/media/liveuser/fedora-root <path to file>
to see what package owns the file. I think the <path to file> in
this case is probably relative to the root directory in normal use,
e.g., /etc/hosts instead of
/run/media/liveuser/fedora-root/etc/hosts. I am writing this part
from memory. If it doesn’t work with the “normal” absolute path,
maybe try with “as-mounted” absolute path instead.
In those cases, running dnf reinstall on suspect packages seems like
a good approach.
Note that for this entire rescue, I did have Internet connectivity for
the Live Fedora image that I was running so that dnf could actually
pull down files from repositories.
As another recommendation, if you have to remove package for some reason and want to re-install it, you don’t necessarily have to use the full version in the package name. For example, you can install the package with just the base package name like this:
dnf --installroot /run/media/liveuser/fedora-root install pcre2-utf16.x86_64
or
dnf --installroot /run/media/liveuser/fedora-root install pcre2-utf16
This way you don’t have to type in as much information. For the dnf remove operations above, though, you must use full name of the
file so you can specify the older package.
Moment of Truth
Once I got no errors when running dnf check and addressed the
problem with the pcre2-utf16.x86_64 package with a dnf reinstall,
it was time to reboot the laptop. Considering that grub boot loader
packages were in the process of being updated, I was a suspicious that
the whole rescue attempt would work, but I rebooted and, lo and
behold, the laptop booted as normal. I think the fact that the laptop
had installed the new packages and was in the process or removing the
old packages might have helped my chances.
Anyway, I am impressed with the capabilities of both dnf and rpm
for this sort of rescue operation and I was able get on with my normal
work with relatively little effort. Definitely a positive Fedora
experience.