Earlier today, we notified some of our customers of a security vulnerability we discovered in Mandrill’s infrastructure. At this time, we’re confident that no customer data was compromised as a result of the vulnerability, but we feel it’s our responsibility to let our customers know exactly what happened and what we’re doing about it.
Since this issue only applies to Mandrill accounts that were used between February 6 and March 10, we emailed everyone who may have been affected. If you didn’t get an email from us today, that means you weren’t affected. For other Mandrill users and affected users who choose to notify their own customers about the vulnerability, this post explains what happened.
On March 10, we discovered evidence that automated attempts were made against Mandrill's internal logging servers in an effort to use them in a botnet. Analysis of the servers that were impacted, including network traffic logs and files present on the servers, indicates that these attempts were unsuccessful. There are no signs that the servers were targeted to access the data stored on them.
We investigated the issue and found that the opportunity for this attack stemmed from a firewall change we made on February 20 in order to more granularly control access to some of Mandrill's servers. Parts of Mandrill's infrastructure are hosted with Amazon Web Services (AWS), and we use EC2 Security Groups to control access. One change was made to a security group that contained more servers than we intended to affect. As a result, a cluster of servers hosting Mandrill's internal application logs was made publicly accessible instead of allowing internal-only access.
What we did about it
As soon as we discovered this vulnerability on March 10, we took several immediate steps to mitigate the potential impact:
- We disabled impacted servers and took them offline, backing up log files and data for further investigation. These servers will not be used in any way going forward.
- We updated all of Mandrill's internal credentials, including SSH keys and our own API keys/passwords for external services, according to our internal security protocols.
- We audited and updated the firewall rules in use for all of our AWS security groups to ensure they were correctly limiting access, and that no other servers were impacted by the earlier change.
We're also making changes to our internal processes for managing security groups and firewall rules. That includes placing all security group rules into version-controlled configuration files with a tool to apply those rules via the EC2 API. This will allow us to more rigorously vet any rules or rule changes in advance, including more automated testing and robust audit logs for any proposed and approved changes. We're implementing additional regular, automated audits of our security groups as well.
Which accounts were affected and how
There's no evidence that any customer data was queried or exported, but unfortunately, we can't completely rule out the possibility of access. So, we're being paranoid and letting our customers know the worst-case scenario. Although it's extremely unlikely, if we assume the attackers were able to access information stored on the servers when the firewall rules were changed, the following data about affected Mandrill account could have been accessed:
Internal logs with basic log data about emails sent between February 6 and March 10. These logs include sender address, recipient address, and subaccount used (if any), but do not include custom metadata or message content.
Only for those who sent via Mandrill’s SMTP integration between February 13 and March 10: API keys used for SMTP messages sent during that time period.
If you did not send using Mandrill between February 6 and March 10, then you didn’t receive an email notification from us because your account could not have been affected.
What you can do
If you sent between February 6 and March 10 but did not use Mandrill’s SMTP integration, then you don’t need to make any changes to your account.
If you sent using Mandrill’s SMTP integration between February 13 and March 10, please deactivate all API keys for your account and generate new ones. Although there's no evidence that your API key was exposed or accessed, we strongly recommend this as a precaution.
Whether or not your account may have been affected by this vulnerability, we encourage all of our customers to use the following several security-related features. They’re all free with your Mandrill account:
- Alerts to notify you when a new IP is used to access your account or your account information is updated
- API key restrictions by IP or API call, based on how and where the API key is being used
- Two-factor authentication using Yubikey, Google Authenticator, or SMS
- IP whitelisting for limiting web-based access to your account
- Account Security overview, which includes a list of IPs used to access your account, the method of access, and first and last date of access
Finally, to our customers: We are deeply sorry for our error. We're committed to making the necessary changes to prevent something like this from happening again. Keeping our customers' data secure is integral to everything we do at Mandrill, and we're using this incident to identify and prioritize improvements to our internal systems and processes. We take this stuff extremely seriously, and as always, we thank you for trusting us to send your email.
If you have any questions, feel free to email firstname.lastname@example.org.