Maintenance report of April 28th 2017
A. The short version
On April 27th 2017, we replaced one of Piwigo.com main servers. The replacement itself was successful. No downtime. The read-only mode has lasted only 7 minutes, from 6:00 to 6:07 UTC.
While sending the notification email to our clients, we encountered difficulties with Gmail users. Solving this Gmail issue made the website unavailable for a few users and maybe an hour. Everything was back to normal in a few hours. Of course, no data has been lost during this operation.
The new server and Piwigo are now good friends. They both look forward to receive version 2.9 in the next days 😉
B. Additional technical details
The notification message had already been sent to the first 390 users when we realized emails sent to Gmail addresses were returned in error. Indeed Gmail now asks for a “reverse DNS IPv6”. Sorry for this very technical detail. We already had it on the old server so we added it on the new server. And then start the problems… Unfortunately the new server does not manage IPv6 the same way. A few users, on IPv6, told us they only see “Apache2 Debian Default Page” instead of their Piwigo. Here is the timeline:
- 6:00 the upgrade starts, switch to read-only mode
- 6:07 web traffic redirected to the new server
- 7:40 the new server is doing fine, we start to notify users
- 7:52 Gmail errors, we stop the notification
- 8:31 we add an IPv6 on the new server
- 10:05 following a few users feedback, we remove the IPv6
Unfortunately adding or removing an IPv6 is not an immediate action. It relies on the “DNS propagation” which may take a few hours, depending on each user.
We took the rest of the day to figure out how to make Gmail accept our emails and web visitors see your Piwigo. Instead of “piwigo.com”, we now use a sub-domain of “pigolabs.com” (Pigolabs is the company running Piwigo.com service) with an IPv6 : no impact on web traffic.
We also have a technical solution to handle IPv6 for web traffic. We have decided not to use it because IPv6 lacks an important feature, the FailOver. This feature, only available on IPv4, let us redirect web traffic from one server to another in a few seconds without worrying about DNS propagation. We use it when a server fails and web traffic goes to a spare server.
In the end, the move did not go so well and we sweat quite a this friday, but everything came back to normal and the “Apache2 Debian Default Page” issue eventually affected only a few people!
