Under Investigation Unplanned Maintenance
RESOLVED: Unscheduled Failover of production and qual F5 Load Balancers
Yesterday evening (10/11) between 5 pm and 5:20 pm, maintenance was conducted on the F5 Load Balancers. This maintenance was expected to have no customer visible impact. This morning (10/12), we discovered an unforeseen issue occurred as a result of yesterday's maintenance. To correct the issue, we conducted an unscheduled failover of both the production and qual F5 Load Balancers this morning between 7:23 am and 7:25 am. Most load-balanced services would not have been impacted by either yesterday's maintenance or this morning's failover event. A few services that use long-held connections may have been disconnected, but they would have been able to reconnect immediately.
Latest Update: 2019-01-12 06:02:28 - We believe the issue is now resolved.