We spent most of our time this week adjusting the backend systems to improve the performance and consistency of the application. This is the kind of thing that can be difficult to explain simply, but let's give it a shot.
Even better sending performance
Last week, we made a series of changes to improve our sending speed. We've continued to see better performance and fewer spikes since those changes were made, but we saw places where we could do even more. With SMTP servers all over the world, there were some edge cases where those servers couldn't get new messages to us as quickly as users were sending them, causing messages to be delayed by whole seconds - clearly unacceptable.
We replaced our main SMTP delivery system with a one that scales better with increased concurrency and uses fewer resources. Here's the graph from our status page that shows the results:
After the change, the average message sent via SMTP is being delivered about 120ms (or 10%) faster, but the real win is that large bulk loads sent over SMTP are being processed as fast as possible, and the queues are much smaller.
Faster webhook delivery
Continuing the theme of "make things faster", we took a hard look at how quickly our webhooks deliver events to your servers. Processing of webhooks was designed to be more cautious than fast. We batch events together so we make as few requests as possible, and back off entirely if we see your server start giving us errors. We updated the system to be more optimistic - delivering events faster if we see your server responding quickly. Median time it takes to send an event to your server is now between one and two seconds instead of a minute or more. This should make custom business logic based on webhooks more reliable and closer to real-time.
The rejection blacklist is what Mandrill uses to make sure bad emails don't go out. If an email you send bounces or triggers a spam complaint, we automatically add the recipient to this blacklist and stop delivering to them for a while. This is usually what you want to happen to avoid sending to addresses that are having issues or have indicated they don't want more emails. In some circumstances, though, this is a terrible thing to do. Imagine if you're sending emails to yourself with order notifications or application error emails. Your receiving email provider has an issue and bounces one of your messages - now important messages are being thrown away, maybe for days. We've added a whitelist to avoid having certain email addresses added to the rejection blacklist. Any email address that is whitelisted will be sent every email, even if there are bounces.
Now, before you start thinking of ways to add your entire customer database to the whitelist (you naughty hypothetical people, you), there are some downsides to this. The rejection blacklist is one of the ways Mandrill protects itself and helps to protect you from sending emails that will bounce or generate spam complaints. If you're regularly sending email that normally would be rejected, we'll dock your reputation and hourly quota. For small whitelists and addresses that hardly ever bounce, the impact isn't big enough that most users would even notice. But use it sparingly.