The total traffic of ftp.fau.de increased from 1.89 PB in 2016 to 2.51 PB in 2017, by 33%. 203 TB alone was accounted to one of our newest additions from late may 2017: lineageos, a free and open-source android distribution and successor of cyanogenmod (which was discontinued).

The total IPv6 traffic increased significantly, to 22% (0.55 PB) of all traffic in 2017, from 14% in 2016 and 11% in 2015.

We had a little downtime today between around 12:10 and 13:20. About 5 minutes of this downtime were planned - we simply wanted to reboot the machine. Unfortunately, the machine did not come back up after the reboot, and it took a while to figure out what the problem was.

As it turns out, the machine was unable to mount the huge volume with all our mirrors on it during boot. Manually mounting the disk failed as well because the device just wasn’t there. In lvdisplay the volume was listed as unavailable, and trying to set it available with lvchange failed with the message /usr/sbin/cache_check: execvp failed: No such file or directory Check of pool bigdata/ftpcachedata failed (status:2). Manual repair required!

Recently, we’ve frequently reached the I/O capacity of our RAID array during peak hours, meaning that increasingly often downloads were not limited by network speed, but by how fast our disks could deliver the data.

While we use a hardware RAID6 which does deliver pretty decent read speeds, we’re still using traditional magnetic hard drives, not SSDs. While these arrays easily deliver more than a Gigabyte per second when reading sequentially, this performance drops rapidly the more random the I/O gets, i.e. the more different files are requested at the same time. Of course, due to the nature of our FTP, which provides mirrors for a whole bunch of different projects, we do get a large amount of random I/O.

It seems I haven’t posted the traditional annual mirror stats for 2016 yet. Well, lets fix that: Here are the most used mirrors on ftp.fau.de in 2016:

Comparing this list to last year, the first thing one notices is the new entry on rank 3: OSMC generates a steady amount of traffic, with visible peaks whenever they do a release. They were not in the 2015 top ten because we only started mirroring it in Q4 of 2015.

People operate mirrors out of different reasons, this could be purely altruistic to support open source communities, or to reduce incoming traffic by providing local clients with an internal package source, or to tilt the in/out ratio of the internet uplink. What ever the reasons are, they are all associated with the usage of outbound traffic and local storage size, in addition to occupation of other hardware and administrative resources. We will consider traffic and storage requirements as the costs and outgoing traffic as the profit (to the general public or an organization) a mirror service provides.

Here are the most used mirrors in 2015:

Kiwix on rank 2 has only been mirrored since March 2015, so it generated that amount of traffic in only 9 and a half months - it might well take over the top spot next year. The Ubuntu release ISOs dropped out of the Top 10 for IPv4. As is to be expected, the CCC conference recordings generate extremely peaky traffic after CCC events - setting a new record high with 11.3 TB in just one day on December 30 (after 32C3). It might even have made a little bit more, but unfortunately the webserver ran out of threads because someone apparently distributed web-torrent-files with 64 KB of chunksize.

Since we moved the ftp to the (then) new hardware in October 2013, we had been using XFS as the filesystem on which we store all our mirror trees. All data resided in one single large XFS filesystem of 35 TB. That worked rather well until we updated the operating system on the machine in January.

After the update, the machine behaved extremely unstable - and in crass contrast to its 460 days uptime before the update. Sometimes all file I/O would stop for a few minutes (up to 30), and then suddenly continue as if nothing had happened. Those hangs happened in average twice a day. Sometimes the machine would also lock up completely and need to be reset. During the hangs, the machine would log many of the following error-messages to the kernel log:

While we publish daily stats for all of our mirrors at https://ftp.fau.de/cgi-bin/show-ftp-stats.cgi, there is currently no aggregation over a whole year. So here are the most used mirrors over the whole year:

The list is slightly different for IPv6.

Since the beginning of September, we’ve been creating experimental GeoIP-statistics to see where the users of ftp.fau.de come from. We’re drawing nice maps from the results. You can find these stats from our general ftp statistics page - for days where GeoIP-stats are available, you will find a link to them at the bottom of the page. Here is an example:

As you can see, unsurprisingly most of our users come from Germany. Besides the map there is also a table that show the respective percentages per country - in the map above, Germany accounts for about 50 percent of all accesses. Note that is counting the number of bytes transferred, not the number of accesses.

Over the last 2.5 months we have added a few new mirrors. Amongst those are:

• CCC conference recordings by FeM. Here you can find recordings of almost all talks from the yearly Chaos Communication Congresses. This mirror also takes part in their official mirror network for these recordings under http://mirror.fem-net.de/CCC/
• CentOS. CentOS is a free rebuild of the Redhat Enterprise Linux packages (without the support of course).
• Eclipse, an IDE
• EPEL, Extra Packages for Enterprise Linux - EPEL packages a lot of additional software for Redhat Enterprise Linux or its clones
• the Qt project. It develops Qt, which is a great multiplatform application framework. There is a commercial company backing the project (currently Digia) that also offers commercial variants of the framework. Only the non-commercial releases are distributed over the mirror-network in which we're taking part here. This is now one of the three mirrors that causes the most traffic on our FTP.
• The Document Foundation - TDF is responsible for the development of LibreOffice
• Trinity Desktop Environment. The TDE project is a fork of KDE3 and continues to develop a productive desktop environment that does not put looks above functionality.
• the VideoLAN project, which produces amongst other things a great free media player that plays almost everything (VLC media player). This is now another one of the three mirrors that causes the most traffic on our FTP.
• XBMC, a free media center software