ObjectiveOur objective is to ensure that updating the operating system will run smoothly and without errors.
Operating System and Software Versions
- Operating system: Red Hat Enterprise Linux 6+
RequirementsPrivileged access to the systems
- # - requires given linux commands to be executed with root privileges either directly as a root user or by use of
- $ - given linux commands to be executed as a regular non-privileged user
IntroductionKeeping the system up to date is an every day task for a sysadmin, as well as a desktop user. By applying the latest (stable) available software on the system we can take advantage of the latest features, and will be more protected from security issues and hopefully suffer less from bugs. To update the system you will need configured
yumrepositories that act as the source of the updated software.
If you sit next to the machine that is running the operating system to be updated, you can easily act if something goes wrong during update, like checking the output on the terminal, or boot to a live system if the upgraded one does not return from reboot - but this is not always the case. Think of a datacenter with hundreds or thousands of (virtual) machines, or simply a physical PC that you have to upgrade remotely.
There are simple steps we can perform to prepare the system for upgrade, and possibly clear any problem that would endanger a successful update.
Update processWhen performing an unconditional update (meaning "update all"),
yumwill fetch all metadata from the available repositories, and calculate all packages to be upgraded against the
rpmdatabase that contains all the metadata about packages installed on the system.
The update process also calculates all dependencies of the upgraded packages, may replace old packages, and remove old kernel images according to its configuration. The number of kernel images to keep is set in the
/etc/yum.confconfiguration file, and is 3 by default:
After all the needed modifications calculated,
yumprovides an extensive list of all the packages to be upgraded, removed, or installed for dependencies, the same way it does when installing or upgrading specific packages.
In an interactive update session
yumwill provide a summary of the packages to be modified, as well as calculation on the size of data need to be downloaded for the upgrade as shown below:
After examining the results, we can decide if we start the update, or cancel it. As yum will update everything it can find updates for, we may want to remove unneeded packages beforehand. We may also notice a package marked for update that we are version-locked with that need to be excluded from the upgrade.
After approval yum will download all new packages, and install/update them one by one. When completed, it will check the integrity of the installed/updated packages, clean up unneeded files. It also provides feedback during the process, providing a line of text for each step, as well as an exit code that hints if the upgrade was successful, or if some problem arose. It will also cancel the update process if a problem rises that seems critical from the consistent system perspective - but there are times when it is too late already, so preventing update problems from happening is a better approach.
yum cacheFrom the process described above we could guess that we need some disk space for the update process:
- The metadata of all configured repositories need to be stored until the calculation of all packages (and their dependencies) to be updated finishes.
rpmpackages that constitute the update itself need to be stored locally until installed properly.
yum cacheis only needed during the update, but can take up substantial disk space. The default location for this cache is in the
/var/cache/yumdirectory. Needless to say that if there isn't enough space to store all data needed, the update process will fail. Some unfinished downloads will be dropped, but not all space may be freed, which ends up having a system failed the update and have it's volume containing
Many installations store their
/vardirectory on a volume dedicated to logging, as the default place for logfiles is
/var/logon most distros, and most well-behaving applications will stop working or even crash if they can't write their logfiles. So filling up the volume they are writing to is a bad thing.
The more packages need to be upgraded, and the more repositories we have, the more space the update will occupy temporary. To calculate this space from update to update is hard, but can be tested with the dry run solution described later if we have a test machine with the exact software content. For a real time example, updating from RHEL 7.1 to 7.5 (desktop installation with Gnome) may take up 4 GB of cache space, but installation of a few fixes to a system that is only one or two months out of date will take up only a few MB.
To check how much space we have, we can use the
In the above example we have 4.4 GB of free space, which will be enough given that the server was updated only a few months ago. To free up space a trivial step would be to clear the
# df -h /var/ Filesystem Size Used Avail Use% Mounted on /dev/mapper/vg_sys-var 6.0G 1.7G 4.4G 28% /var
yum cachealready stored (maybe at the last update). To check how much space the a cache occupies at the moment, we can use
The above numbers are in MB, so the
# du -mcd 1 /var/cache/yum 1103 /var/cache/yum/x86_64 1103 /var/cache/yum 1103 total
yum cachein this example takes up about 1 GB of disk space and occupies most of the space on the
Clearing the cacheWe can clear the whole cache with the following command:
yum clean all
yumnotifies us in the above command's output on RHEL 7 versions, there may be orphaned data from removed or disabled repositories, which will most likely happen after minor release ugrades, in which case we can safely clear the data by hand:
We may get more space for the update by clearing other data stored on the volume, like compressing/deleting old logfiles, moving large files to other volumes, or extending the volume size.
rm -rf /var/cache/yum/*
Moving the cacheTo work on with the possibilities of
yum, if we are really low on disk space, can't clear anything further, and can't add more space to the volume, we can move the location of the
yum cacheto another volume with more free space. We can configure the cache location in the
yum.confconfiguration file mentioned above. Consider the default setting:
By changing the path before
$basearchthe next yum operation will work with the same directory structure, but on a different path - hopefully with more free space for the upgrade. We can also move the cache to another volume by moving the whole directory:
And creating a symlink at the original location that points to the new place:
mv /var/cache/yum /extended_data_volume/
It is wise to know that the update will not fail on a trivial error such as low disk space. On a large system sysadmins deploy monitoring tools like Nagios that can report low disk space on all machines, making this step much less time consuming and error prone.
ln -s /extended_data_volume/yum /var/cache/yum
Network errorsIf there are problems with connectivity between the repositories and the machine performing the update, the update may fail. This can only happen at the metadata, or the new rpms download stage, and will not break the system. You can start the update process again when the network issue is solved.
On the other hand, if the update is initialized from an interactive session, on network outage the connection may break, leaving the updating machine without admin to answer the questions
yummay ask. If the package install/update stage already started, it will continue unattended, and may fail or complete if it would otherwise do. After reconnection the process can be followed in the
Yum dry runAside insufficient disk space and network problems, the update in many cases can fail on unresolved package dependencies. These need to be solved with tools that can calculate and handle package dependencies, but it would be useful to know there will be problems before the actual update (and therefore not wasting the always too short downtime of the system). To get this valuable information we can run the update process as it would run the actual update, but stop before any actual package downloading, installing or updating have taken place.
Around Redhat 6.6 a new option was introduced that will cause
yumto assume "No" to every question that comes up during update - including the approval before the actual package manipulation stage, and as a consequence no actual interaction is needed execute a dry run:
This can be the ideal tool to provide a dry run of the coming update, including packages to be upgraded, and any errors what may occur. Consider the following simple
yum update --assumeno
The above script can be executed automatically and will provide a text report of the dry run, as well as an overall exit code indicating any problems. The output does not need to be saved on the local file system. The target of the output redirection can be a network file system, or the report can be posted to some central reporting server, may be gathered by other scripts or applications. The reports can be published and distributed among other IT departments for approval, this way everyone involved can see exactly what packages will be updated, and to what version.
#!/bin/bash yum update --assumeno &> $(hostname).yum.dryrun.$(date '+%Y-%m-%d').out exit $?
The dry run can be scheduled to run on a given time frame (maybe at night to impact the system's performance less) with
cron, or executed from a central source with a puppet setup. The exit code can also be stored and processed by monitoring or
facter, to aggregate the possible results of the upcoming upgrade before proceeding.
ConclusionEven with one or a few computers, we should gather information before starting an update of the whole operating system, just to be on the safe side. One day there will be a problem, and it is much less stressing if you can solve it before it has impact on the actual job of a given machine. On a larger scale, it is simply not possible to sit next to each server or desktop and support it with your presence in the hopes that this will help the update to run flawlessly.
By knowing the stages of the update process, the pitfalls, as well as the solution to them is essential for successful updates. Starting your whole infrastructure's next update stage with the confidence that there will be no problems is to do it with style.