Creating a package repository on Linux : Fedora and Debian

Introduction

This article at is the logical continuation of our PXE article, because after reading this you will be able to network boot AND actually install the distribution of your choice. But there are other uses of creating your own repository. For example, bandwidth. If you manage a network and all the systems (or some) are running the same distribution, it’s easier for you to just rsync in conjunction with a nearby mirror and serve updates yourself. Next, maybe you have some packages created by you that your distro won’t accept in the main tree, but the users find them useful. Get a domain name, set up a webserver and there you go. We will not detail the setup of a webserver here, just basic installation tasks and the basic setup of a repository for Fedora or Debian systems. Hence you are expected to have the necessary hardware (the server and the necessary network equipment, depending on the situation) and some knowledge about Linux and webservers. So, let’s start.

NOTE:This article was moved from our previous domain linuxcareer.com.

Creating a repository on Fedora systems

Installing the tools

Fedora has a tool called createrepo which simplifies the task at hand. So, all we need to install is that and httpd as the webserver:

 # yum install createrepo httpd 

Setting up the repositories

Now, after setting up your webserver, we will assume that the root directory is ar /var/www. We have to create the necessary directories in an organized matter (feel free to adjust to taste if necessary or just follow the official layout):

 
 # cd /var/www/html
 # mkdir -p fedora/15/x86_64/base
 # mkdir fedora/15/x86_64/updates

That’s it for now. All we have to do is rsync to the created folders and make sure we have plenty of space available:

 # rsync -avrt rsync://ftp.heanet.ie/pub/fedora/linux/releases/15/Everything\
/x86_64/os/Packages/ /var/www/html/fedora/15/x86_64/base

Now use createrepo for the base folder :

 # createrepo /var/www/html/fedora/15/x86_64/base

This is mandatory, as it will create the repodata directory that yum needs when using your repository. Now let’s repeat the same step as above, but this time we will get the updates:

 # rsync -avrt rsync://ftp.heanet.ie/pub/fedora/linux/\
updates/15/x86_64/ /var/www/html/fedora/15/x86_64/updates

In the end, we recommend you check if httpd is set to start on boot and also use cron to get updates on a regular basis:

 # systemctl enable httpd.service
 # crontab -e

Remember that the rsync command to be added is the second one, the one related to updates and that systemctl is only available on Fedora 15 or above. Use ntsysv or chkconfig on older Fedora systems.

Client setup

You must tell the machines that will get updates from your server where to find them, so we start by creating the .repo files :

# this will be base-lan.repo
[base-lan]
name=Fedora $releasever - $basearch
failovermethod=priority
baseurl=http://192.168.1.2/fedora/$releasever/$basearch/base
enabled=1
# Make sure you disable the official .repo files with enabled=0
gpgcheck=0

# this will be updates-lan.repo
[updates-lan]
name=Fedora $releasever - $basearch - Updates
failovermethod=priority
baseurl=http://192.168.1.2/fedora/$releasever/$basearch/updates
gpgcheck=0

Now, just do a

 # yum update

and you are ready to go.

Package signing

As pointed out by one of our readers, one should be aware of security issues when installing packages. Software may be downloaded from compromised servers and may contain malicious executables. Yum (and apt, zypper and other package management systems) overcomes this problem by using GPG keys. We talked about mirroring a Fedora repository. These packages are already signed, and the keys can be found in /etc/pki/rpm-gpg. If you ever used an official Fedora repo as a client before enabling your local repositories, that directory will already contain the necessary keys. If not, the keys can be downloaded from getfedora.org/keys/. Now, we have to alter our .repo files to enable gpgcheck and tell yum where the keys are.

# These are the only lines that need to be changed
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-fedora

If you’re using a local custom packages repository, yum will complain that your custom packages aren’t signed. You can either use the yum flag –nogpgcheck if you’re the mirror/repository maintainer and you only serve packages to your organization, or, the secure way, sign the custom packages as well. This is because the server holding the custom/local repository may be also compromised. So you will have to create a GPG key on the server and use rpm to sign the custom package:

$ gpg --gen-key
$ gpg --list-sigs

Create RPM package gpg key

As you can see, the USERID is in our case “Linux Career <user@linuxcareer.com>”. Now to make the key public:

 $ gpg --armor --export "USERID" > my.key.file.asc
 $ gpg --keyserver pgp.mit.edu --send-key "USERID"

Of course, your USERID will differ, so alter the info accordingly. Take note that on Fedora 16, on which we tested this, the executable is named gpg2 instead of gpg.

We just need to create a .rpmmacros file in the home directory of the user who will sign the packages, and put the following in there:

%_signature gpg
%_gpg_name USERID
%_gpgbin /usr/bin/gpg2

The command to sign a package, now that all is set, will be

 $ rpm --addsign name_of_package.rpm

Now the client downloading from your custom repo will use ‘rpm –import $key’ in order to be able to download those custom packages.

Creating a repository on Debian systems

Installing

Since Debian’s repository structure is more convoluted, you’ll see that it takes a little more work on the server side, but less on the client side. At all times, there will be three sections: stable, testing and unstable (not counting experimental) which each have three components depending on how packages are licensed: main, contrib and non-free. It’s your decision to make what part of the distribution you want mirrored, but it’s our duty to warn you: Debian has way more packages to offer than Fedora, so the disk space requirements will grow significantly. There are a lot of tools you can use for creating a custom repository with your own custom packages, but we will stick to official packages for now. So, we’ll get back on our setup for the PXE article and create a local repo for the installation. We will need a webserver, so let’s install it:

 # aptitude install apache2

Make sure that Apache is configured and started before you continue.

Server setup

The default root directory, just like in Fedora, is /var/www, so let us create a debian directory in there:

 # mkdir /var/www/debian

The Debian folks recommend ftpsync, a collection of perl scripts meant to help you get what you need onto your local mirror. Of special interest is the --exclude option, since you don’t want to get all the contents of a Debian archive (only amd64, only main and contrib, only squeeze, without CDs, etc.). If you want to create a repository to use after installation, just point your /etc/apt/sources.list to the directory containg the packages (you already have a working model there) and that’s all. For example:

 deb http://192.168.1.2/debian squeeze main contrib

But let’s see in detail what you need to download if you don’t fancy using ftpsync. Debian (and Ubuntu, and probably other Debian-derivatives) have a package named apt-utils, which offers, among others, the apt-ftparchive program that we will use for our custom repository. So…

 # aptitude install apt-utils

will get the necessary tool installed on your system. We already have the base directory created on our webserver, so we will need subdirectories customized for our needs:

 # cd /var/www/debian
 # mkdir -p pool/main
 # mkdir  pool/contrib
 # mkdir -p dists/squeeze/main/binary-amd64
 # mkdir -p dists/squeeze/contrib/binary-amd64
 # mkdir .cache

Now that we have the directory structure in place, let’s create the necessary configuration files to help apt-ftparchive find and index our software. Please note that you can use this setup to mirror official Debian packages or create a repository with your own packages, as the steps are same.

The first file of the two we will need to create (both will live in /var/www/debian) is named apt-release.conf.

 # cd /var/www/debian
 # $editor apt-release.conf

The contents, related to our needs as shown above, would be like this:

APT::FTPArchive::Release::Codename "squeeze";
APT::FTPArchive::Release::Origin "linuxcareer.com";
APT::FTPArchive::Release::Components "main contrib";
APT::FTPArchive::Release::Label "Linuxcareer.com Debian Repo";
APT::FTPArchive::Release::Architectures "amd64";
APT::FTPArchive::Release::Suite "squeeze";

You can also use apt-ftparchive to generate config files based on the command-line arguments. Use whatever approach you prefer.

The second configuration file is named apt-ftparchive.conf and its’ contents would look like this:

    Dir {
     ArchiveDir ".";
     CacheDir "./.cache";
    };

    Default {
     Packages::Compress ". gzip bzip2";
     Contents::Compress ". gzip bzip2";
    };

    TreeDefault {
     BinCacheDB "packages-$(SECTION)-$(ARCH).db";
     Directory "pool/$(SECTION)";
     Packages "$(DIST)/$(SECTION)/binary-$(ARCH)/Packages";
     Contents "$(DIST)/Contents-$(ARCH)";
    };

    Tree "dists/squeeze" {
     Sections "main contrib";
     Architectures "amd64";
    }

As you can see, the syntax is very much self-explanatory regarding both files.

For the sake of example, we will now download a .deb from a Debian mirror in order to properly illustrate our idea.

 # cd /var/www/debian/pool/main
 # wget -c ftp://ftp.heanet.ie/mirrors/ftp.debian.org/debian/pool/main/\
p/patch/patch_2.6.1.85-423d-3_amd64.deb

Now let’s generate the contents (this will have to be repeated every time you add or remove packages).

 # cd /var/www/debian
 # apt-ftparchive generate apt-ftparchive.conf
 # apt-ftparchive -c apt-release.conf release dists/squeeze > \
dists/squeeze/Release

These actions did what is called “building the repository”. Now, as instructed above, add a line to your sources.list and you can have access to your software repository. Should you need to become a Debian mirror and still don’t fancy ftpsync, use rsync with the remote directory named pool/$section and go get yourself a coffee or something. Also, use a mirror, don’t overload ftp.debian.org, please.

Package signing

If you want to use a CD/DVD/Blu-Ray image to serve content to your clients, the Release file on the optical media images isn’t signed by default. But if you serve by rsync’ing a mirror’s content, chances are you don’t have to do anything. If you have a custom repository, here’s how to do it. First, as in the Fedora example, generate the GPG key:

 $ gpg --gen-key

Now, because of bug #639204 in debsign (last update this August), it would seem that we will have to take an alternative route. Since Debian packages are basically just ar archives, we will use the lower-level way to sign our package(s):

 $ ar x package_name.deb
 $ cat debian-binary control.tar.gz data.tar.gz > tempfile
 $ gpg -abs -o _gpgorigin tempfile
 $ ar rc package_name.deb _gpgorigin debian-binary control.tar.gz data.tar.gz

So, what we did here was eXtract the .deb file with ar, concatenate its’ contents to a temporary file (mind the order), sign that file then recompose the .deb to its’ original state. Now we need to export the GPG key (as you can see, the process isn’t that different from the one applied on Fedora).

 $ gpg --export -a > mydebsign.asc

Now let’s extract the key for further usage:

 $ gpg --fingerprint

Remember the last four groups in the key fingerprint (as seen below), as those will be the key ID, which we’ll use later.

On the client machine, make sure you have debsig-verify installed, then you can create a place for the key:

 # mkdir /usr/share/debsig/keyrings/$key_id

As you can see in the screenshot, our example key ID is 8760C540B4FC5C21. Now let’s import the key:

 # gpg --no-default-keyring --keyring \
    /usr/share/debsig/keyrings/$key_id/debsign.gpg --import mydebsign.asc

Now, here comes the tricky part: we will need a policy file for the keys. The language used is XML, but no need to worry: in /usr/share/doc/debisg-verify/examples you’ll find a file named generic.pol that can be copied somewhere to be edited and renamed. An example of such file could look like so:

<Policy xmlns="http://www.debian.org/debsig/1.0/">

<Origin Name="Linux Career" id="8760C540B4FC5C21"
Description="Package offered by Linux Career"/>

<Selection>
<Required Type="origin" File="debsign.gpg" id="8760C540B4FC5C21"/>
</Selection>

<Verification MinOptional="0">
<Required Type="origin" File="debsign.gpg" id="8760C540B4FC5C21"/>
</Verification>

</Policy>

What you see above is only the essential part of the policy file. After checking with the example and making the necessary alterations, save this file to /etc/debsig/policies/$key_id/$policy_name.pol. After this step, if you followed the steps correctly, you can use debsig-verify with the package name as an argument to check your downloaded packages. Thanks to PurpleFloyd for his helpful article on this.

Client setup

So, let’s start our client machine, making sure it’s set up to boot from network and, when you are asked to choose a mirror, select “Enter information manually”. Enter your server’s IP, then the location relative to /var/www (debian, in our case) and you should be ready to install.

Conclusion

One can never stress enough the importance of saving bandwidth, even on a small network. Of course there are other advantages to a local mirror approach, like serving customized software for your company (special patches applied or just changes to fit the company’s needs better) or serving your piece of software packaged for your favorite distribution.