All Barracuda Email Security Service customers received a letter from Stephen Pao, GM Security Business, providing an in-depth analysis of the root cause of the outage and a post-mortem of our response.
At this time, all systems are back and running with no mail loss.
The outage was caused by a denial-of-service (DoS) attack designed to keep SMTP connections alive at layer 4 as long as possible. While no servers actually failed, our network infrastructure was clogged with fake network traffic preventing email delivery after scanning.
We restored services by routing Barracuda Email Security Service traffic through other, unaffected portions of our network. Once email flow was restored, we were able to characterize the attack traffic and block the sources. At the same time, we hardened and increased the capacity of the Barracuda Email Security Service infrastructure to prevent this situation in the future.
Our systems have been fully operational since we implemented these measures.
For a more detailed post-mortem of the attack and our response, please refer to the full email from Stephen Pao which may be found on the Barracuda Email Security Service forum outage thread here: https://community.barracudanetworks.com/forums.php?url=/topic/24559-barracuda-email-security-service-outage-information/
Update: The following letter was sent to customers by Stephen Pao, GM Security Business
Dear Barracuda Email Security Service customer,
On behalf of Barracuda Networks, I apologize for any inconvenience caused due to recent service disruptions with the Barracuda Email Security Service.
The Barracuda Email Security Service initially experienced a minor outage on Monday, December 22, 2014 at 2:41PM PST, which was resolved by 5:50 PM PST that same afternoon. During this time, email for a small portion of Barracuda Email Security Service customers experienced delays in delivery, with no data loss. However, at 10:45 PM PST, the service experienced another interruption that affected a broader segment of customers. The problem was mitigated by reconfiguring network traffic through additional servers, and by 10:10 AM on December 23, 2014, mail flow was restored and the system started processing the email backlog. Also, there is no evidence of data loss.
At this time, mail flow should be back to normal. We continue to actively monitor the situation and investigate the root cause of the disruptions.
Barracuda Networks remains committed to providing the highest quality of service to our customers. On this occasion, I realize that we did not meet the high standards we set for ourselves. The disruption in service and the delay in identification was well outside of our uptime and response time targets. This has my personal attention, and I am committed towards having our team isolate the root cause, improve our monitoring systems, and expand our network infrastructure to avoid this situation in the future.
In addition, we are working on a process to implement a one-time service credit for Barracuda Email Security Service customers who were affected by today’s outage. We will update you about this process shortly.
Our first priority is to ensure uninterrupted mail flow. If you are still experiencing any issues with undelivered mail, please contact our Technical Support team on the phone at +1 (408) 342-5300 for immediate assistance.
You can also directly email me your concerns and feedback at email@example.com.
Thank you for your ongoing support as a valued Barracuda customer.
GM, Security Business
Update: As of 2:30pm PST, Barracuda Email Security Service has been restored for most customers and is processing mail. Some intermittent issues may remain and the service is in the process of working through the backlog. If you are still experiencing delays, please file a support ticket by clicking here.
Update: As of 12:10 PM PST, Barracuda has added additional network infrastructure and made routing changes to mitigate the problem. At the moment, emails are being processed at a faster rate.
There is no evidence of data loss at the moment and we will continue to provide updates through the investigation. Please check back here for updates.
Published 12-23-2014 11:30am PST
At approximately 11:00PM PST, Barracuda started experiencing service disruptions across a broad segment of the Barracuda Email Security Service. Affected customers may see a delay in email traffic or an inability to access the Barracuda Email Security Service UI. We are aware of the issue and are actively working to resolve it. We will provide additional information as it becomes available on the following sites:
Customers have contacted us to report the issue resulting in above normal call volume to our support center. To avoid call delays, we recommend the following:
Determining if you have been impacted:
- If you have an SNMP trap, you would have been notified that email is being queued.
- If you do not have an SNMP trap configured, you may have seen deferrals in your outbound email queues.
We recommend you configure SNMP monitoring, such as using this OID to be notified if no email has been sent through the system.
Notifying Barracuda customer support
To notify Barracuda customer support to open a ticket: https://login.barracudanetworks.com/support/newcase
We apologize for the disruption in service as Barracuda is committed to resolving this as quickly as possible.