Posted by: Oliver Wai, product marketing manager
As you probably heard from our previous blog posting, Barracuda Networks suffered a breach from a SQL Injection attack on the weekend of April 8. While the overall impact of the breach turned out to be relatively minor (only contact names, including names and emails), such an event always involves a post-mortem. As is often the case in events such as data breaches or data center outages, there is never one single error that leads to the outage or attack but rather a series of interrelated errors that ultimately results in a failure or vulnerability that can be exploited. Taken individually, each event is usually accounted for by the organization and there are redundancies in place to handle any failure issues. However when taken together, the unexpected – in this case an attack on our site – occurs. In analyzing the attack, we observed:
- In the rush to continually add timely and fresh content to the corporate Web site, a few mistakes were made in the PHP code.
- Code vulnerability scanning of the affected part of the Web site was scheduled but had not yet occurred.
- The Web Application Firewall that was put in place to harden the Web site was put into Passive Mode by human error during a maintenance window.
So while there were redundancies in place to secure our Web site, an unfortunate confluence of events last weekend left a vulnerability in our Web site exposed; this resulted in the SQL injection attack by a group we believe to be originating in Malaysia. The upside? Since the Barracuda Web Application Firewall was still inspecting traffic even in Passive Mode, it gave us a detailed audit trail of the SQL Injection probe and the subsequent attack. This gave us the necessary forensics to quickly analyze the breach, contain the damage and reach out to those affected.
Analyzing the Attack
From our Barracuda Web Application Firewall logs we determined that there were two clients used to probe and attack the barracudanetworks.com Web site:
Using the information reported by the Barracuda Web Application Firewall, we were able to quickly filter and find the corresponding entries on our Web server logs:
(NOTE: the Web server logs use Greenwich Mean Time (GMT) whereas the Web Application Firewall uses Pacific Daylight Time (PDT) zone)
Drilling down into details of each entry on the Barracuda Web Application Firewall logs gives us clues on the attackers and the tools used in the attack:
The first attack started at 5:07pm PDT on April 9 and had an IP address of 115.134.249.15 which resolved to somewhere near Kuala Lumpur, Malaysia. This confirms online reports of the hacks originating from Malaysia. We also noticed that the attackers launched the attacks using a modified version of a pentest tool designed by “white hats” to probe Web sites for SQL injection vulnerabilities. This also seems to corroborate reports that the hackers responsible for the attacks hung out on “white hat” online communities. Looking at our Web server logs, we also see the same entries, enabling us to trace down what was attempted and what succeeded on our backend systems.
(NOTE: the Web server logs use GMT whereas the Web Application Firewall uses PDT)
From the recorded logs, it was clear that the first attacker used the automated tool to recursively crawl through the barracudanetworks.com Web site and blindly injected a series of SQL commands against each input parameter to find potential vulnerabilities. The SQL Injection tool finds the first vulnerability at 5:16pm PDT but continues to probe the Web site. At 8:10pm PDT a second client using the IP address of 87.106.220.57 joined the attack. The second IP address resolved to a server in Germany but it is unclear at this time if the server was a relay point or if it was a second attacker. Nevertheless, activities from the second IP were recorded and logged by the Barracuda Web Application Firewall:
Below is the screenshot of the corresponding Web server log:
(NOTE: the Web server logs use GMT whereas the Web Application Firewall uses PDT)
From the logs captured in the Barracuda Web Application Firewall, it seems that the attacker used the second client to launch manual attacks against discovered vulnerabilities while the primary attack script continued to scan the Web site for vulnerabilities. Ultimately, the attackers focused their efforts on a single line of weakness in a peripheral Web page where the input parameters were not properly sanitized. Here is the pseudo-code of the underlying vulnerability:
<?=Foo_Function( $_GET['parameter'] )?> //Takes user input
By not sanitizing the input value, it gave the attackers the ability to inject SQL commands into the HTML input parameter to attack the underlying database.
All developers are taught to never trust user inputs and that all inputs must be sanitized before sending it to underlying servers. However, what you can see from this example is that it is often not obvious to the naked eye that there is anything wrong with the code. This is why in addition to using defensive coding, Barracuda Networks also uses code scanners and our own Web Application Firewall to guard against possible vulnerabilities. Unfortunately in a Web site of tens of thousands of lines of code, all it takes is a single mistake. We have since fixed the code to protect against future attacks by adding a single line of code to sanitize the inputs on the affected page:
$parameter = @is_sanitized($_GET['parameter']) ? $_GET[' parameter '] : 0;
<?=Foo_Function($parameter)?>
From Vulnerability to Breach
Once the attackers found the vulnerable page, they attempted to steal the database user accounts. Over the next 10 hours, they tried a number of different attacks in an attempt to break into the underlying database but failed each time. At 3:06am PDT, the attackers changed strategy to focus on the underlying database schema. This proved to be a correct strategy and by 3:19am PDT the first set of database records containing contact email addresses was stolen.
Barracuda Networks discovered the breach at 10:30am PDT and the Barracuda Web Application Firewall was re-enabled to Active Mode at 10:39am PDT. Once in place, the Barracuda Web Application Firewall immediately blocked all subsequent attacks from the 115.134.249.15 IP address. The attacker continued to cycle through attacks against the remaining pages for the next few hours, even when the Barracuda Web Application Firewall blocked all of the attacks. This seems to confirm that an automated pentest tool was used to blindly inject SQL commands. In all, a total of 110,892 SQL injection commands from both attacking IP addresses were sent against 175 URLS at a rate of 42 per minute.
In tracing the Web Firewall and Access logs on the Barracuda Web Application Firewall, we determined that the attackers compromised a Marketing database and stole two sets of records containing a total of 21,861 names and emails. However since there were a number of duplicates between the two sets and the fact that many of the entries were from users who are no longer with the original organizations, the number of affected users is substantially lower.
Any breach is a serious issue and we have reached out to the affected users documenting what has happened and any necessary precautions that they may need to take in response. We believe that the users affected by the breach are at minimal risk. We do not store any sensitive information in our Marketing database other than names and email addresses. Moreover, since Barracuda Networks primarily uses this data to send emails on upcoming events, Webinars, or other corporate news, the risk of spear-phishing is low as all communications are one-directional and informational in nature. Finally since most users are existing Barracuda Spam & Virus Firewall customers, the vast majority of potential spam would likely be blocked regardless.
Conclusion
In hindsight, it was clear the Barracuda Web Application Firewall would have been able to detect and protect our Web site from the recent SQL Injection attack that occurred. However the reality of the situation is that with most breaches, the weak link is typically not with the technology itself but rather with the human element and the processes associated with security. Unfortunately attackers today have more sophisticated tools at their disposal to find victims. They can now automate the tedious task of finding vulnerabilities and focus solely on the “last mile” once a vulnerability is detected. What this means to the rest of us is that attacks will likely become more common and affect a much wider range of organizations.
The silver lining to this experience was that it helped us to demonstrate the effectiveness of the Barracuda Web Application Firewall in providing the necessary protection and auditing capabilities to defend against SQL injection attacks. The Barracuda Web Application Firewall was able to identify the SQL injection attack and would have blocked the attack if it had been placed in Active Mode. Nevertheless even in Passive Mode, the Barracuda Web Application Firewall was able to gather detailed forensic information that we used to investigate, contain and audit the affected systems. Using this data, we were able to quickly identify how the attacks occurred, what was breached and who we needed to reach out to after the incident.
While we have definitely advised customers on the risks of not securing their Web applications and we certainly have heard the worst-case scenarios from our customers as a vendor, we did not imagine that we would find ourselves having first-hand experience with such a scenario. We learned some valuable lessons in this situation and we hope that our story serves as evidence of how important it is to harden and secure your Web applications.
