ZAP-Hosting outage on 17.01.2021

In the 10.5 year history of ZAP-Hosting, the largest network outage of the main location "Frankfurt am Main -> Eygelshoven" occurred on Sunday, 17.01.2021. We will inform you about the background and causes in the following. At the end of the blog entry you will find a 10 € ZAP credit compensation voucher.

Currently, all German ZAP hosting networks are announced and routed in the Interxion data center in Frankfurt am Main via Combahton. This is also where the extremely powerful DDoS filtering, optimized for game servers, takes place. Combahton then pushes the "clean traffic" over a British Telecom line to Eygelshoven, NL, where most of our servers (incl. website etc.) are located.
The link of British Telecom, which is rented via i3d.net, failed on Sunday around 08:00 (UTC +1) in the morning. As we heard later, it was unannounced maintenance work of British Telecom. It is already the second time that British Telecom did not announce maintenance work or the said provider did not pass on the maintenance work.

In such a case, redundancy via Amsterdam should provide a remedy. For this purpose, we have a tunnel in the Eyhelshoven data center that restores the VLAN via Amsterdam to Frankfurt in the event of a failure. However, the tunnel is limited to 10gbit/s, which has not been sufficient for a few months, as ZAP has experienced very strong growth in recent months.
For this reason, the line was overloaded beyond measure and there were massive packet losses for several hours.

Instead of routing in Frankfurt, we will bring the required network hardware and routers to Eygelshoven this week and announce all networks here. This will allow us to control the routing directly from Eygelshoven and route both the direct link to Frankfurt and via Amsterdam with one BGP session each.

This way, both links are active at the same time, and if one link fails, the user notices almost nothing. Thus, a limited backup tunnel does not have to become active first, as is currently the case.