At 00:50 on 11/12/2017 (AEST), Conetix staff were notified to an issue affecting some sites. This issue meant that some sites were't available or were slow through certain links.
Initial diagnosis suggested a possible upstream carrier fault and work began to further narrow down the affected systems. At 01:20, it was determined the issue wasn't from external networks. Diagnosis then began in the core network, however all systems were showing no fault found based on our 24/7 monitoring systems. Deep level traces were then performed to isolate the cause of the issue. At 01:50, a fault with one of the firewall's Intrusion Prevention System (IPS) services was pinpointed, causing new connections to be queued rather than processed.
This service was restarted and the fault resolved by 02:10 11/12/17, at which point all connections were able to pass through without issue.
Conetix is continuing to investigate the underlying root cause to determine why it occurred and ensure the big is addressed to prevent it occurring again in the future.