How emails are sent from Domus

Domus allows companies to send millions of emails to their customers which is a significant undertaking, and one that we take seriously.

Recently we changed the mechanism by which we send event notifications and match emails. This post describes the new setup, the reasoning behind the decision, and a roadmap for future changes that we hope to make.

Summary

Email systems are getting more and more restrictive on the email that they will accept from Software as a Service (Saas) suppliers like us that send out lots of emails on behalf of our clients.  We used to send out emails as if you yourself had sent them, but this makes us more and more liable to have our mail server blacklisted as they think we are 'faking' your email address. 

In addition, not all of our customers are managing their email addresses well so they leave in addresses even though they never work.  Every 'bounced' email affects our sending email reputation and makes it more likely that we will get blacklisted.

In order to mitigate the above effects we have made two changes to the way that we send Match Emails and Event Notifications:

• Emails are sent from the address messaging-service@domus.net
• We capture and monitor all bounced emails on your behalf and if they are deemed invalid we mark the customer record so that no more emails can be sent to that address.
  • Any customer marked as DoNotEmail will have their email address greyed and crossed out on the Customer Summary and Document Summary pages so you can easily see this.
  • In a recent change (as of 6th November 2018), an Activity is also added to the customer record to indicate that the system has detected the bounce and marked the customer as DoNotEmail

If you wish to know more about this in detail, please read on. Be warned - this is a fairly technical topic so take a deep breath.

Setting the Scene

The email landscape has changed dramatically over the past 5 years, and mail servers and email ISPs have all developed unique practices to try and shield their users from spam and other unsolicited email that currently floods the internet.

Over the past 10 years, several standards and de-facto practices have evolved to help mail servers make decisions on which emails to trust:

• Sender Policy Framework (SPF) - a 10 year old standard that lets the owner of a mail domain (e.g. gmail.com) specify which mail servers are allowed to send email

• DomainKeys Identified Mail (DKIM) - a slightly newer standard that allows the sender of an email to digitally sign their email to prove that it came from them

• Grey listing - a heuristic that some receiving mail servers use to slow down the receipt of email, on the theory that spammers won’t attempt to queue and retry messages

• Spam filtering - a set of algorithms that receiving mails servers can run on email to try and determine if the content of the email is “spammy”

• Blacklisting - various databases of IP addresses or email addresses that are known to send spam

• Domain based Message Authentication, Reporting & Conformance (DMARC) - a recent specification that combines SPF and DKIM to allow email senders to prove that the mail was sent by them

We realise that our users absolutely, positively want email that they send to be received by their customers - their cashflow and business depend on it. Even though email is by design an unreliable communication mechanism, we are trying to make it as robust and reliable as possible for our users to send email to their customers.

Our Previous Approach

Our previous approach to sending emails from within Domus was to “impersonate” or send email “on behalf” of the user currently logged in to Domus. e.g. If joe.bloggs@kingsestateagents.com had signed up to Domus, then we would happily send out email that looked like this:

From: Joe Bloggs <joe.bloggs@kingsestateagents.com>

To: A Customer <a.customer@customer.com>

Subject: We have found properties for you!

In 2006, this was a fairly reliable solution, and most mail servers would happily accept these emails and deliver them to the recipient. In the subsequent years, it was possible for Joe Bloggs to add an SPF record to their kingsestateagents.com domain to indicate that Domus was able to send email on their behalf. This worked reasonably well for a few more years.

The Problem

It is now no longer possible for Saas (Software as a Service) vendors to reliably send email “on behalf” of their customers.

If a recipient mail server receives an email like the above example, the mail server will perform some checks to try and decide whether this email was legitimately sent by Joe Bloggs. One of the indications the server will use is the IP address of the server that delivered the message to it. If the server does not look like a legitimate server for Joe Bloggs’ ISP (kingsestateagents.com), then that’s a negative mark against the email. Traditionally a combination of reverse DNS lookups and SPF would be used to perform this check.

These days, setting SPF records is not sufficient to prove the authenticity of an email, and this example email illustrates the reason why:

Envelope Sender: joe.bloggs@evilspammer.net

From: Joe Bloggs <joe.bloggs@kingsestateagents.com>

To: A Customer <a.customer@customer.com>

Subject: We have found properties for you!

The “Envelope Sender” address is never displayed to regular email users, but it is actually the address that SPF uses to test the authenticity of an email. In this example, if this email message arrived from a server that was legitimately run by “evilspammer.net”, then technically that’s all that is needed to be legitimate from an SPF perspective - even though the email displayed in the recipient’s mail client doesn’t indicate it comes from evilspammer.net!

For this reason, SPF is no longer the solution to sending email “on behalf” of customers like it was when we launched Domus a few years ago.

This has been confirmed by email experts and in fact, they confirmed what we were seeing, which is that there’s no way to reliably send email while pretending to be someone else. The previous method of sending Domus emails out by impersonating the Domus user was untenable, and we would see continuing increases in the numbers of emails that would fail to deliver.

Problems with SPF

We’ve already discussed how SPF on it’s own is not sufficient, and that it relies on the verification of a part of the email message that’s not actually displayed to an end user.

Some mail servers would also run SPF like tests against the visible “From” address that displays in the email, even though that’s not strictly part of the SPF standard, effectively asking the question: “Is Domus's email server allowed to send email from kingsestateagents.com?”

In order to pass this test, every Domus customer would need to create an SPF record in their DNS to allow Domus’s mail servers to send email on their behalf.

For some Domus customers that are IT savvy, and in control of their own DNS, this was indeed possible, and over the past few years some have managed to do this successfully.

However, for most Domus customers this is infeasible:

• Customers that had their own domain name (e.g. MyCompany.com) but weren’t able to change DNS, had a large company with strict IT policies. These customers aren’t easily able to create SPF records.

• Customers that use SPF, but didn’t include Domus’s servers in their SPF records may be hardfailing all impersonated mail from other sources.

• Corporate customer’s mail servers would often reject Domus emails that they sent to themselves, even if SPF was correctly set up. E.g. a corporate mail server for MyCompany.com would say “I’ve recieved mail from domus.net’s mail server’s that is pretending to be from MyCompany.com. I know that I’m the only mail server for MyCompany.com, so this is obviously spam” and throw it away.

• Creating valid SPF records is actually hard. One only needs to look around the internet and on our own Community to see that people don’t understand the nuances of SPF, and a simple mistake in configuration can stop delivery of all your company’s email. This goes against our mantra of being the “World’s easiest estate agent system”.

The Solution

We absolutely want to deliver email for our customers, to your customers.

Candidate solutions:

• Continue impersonating email

  • Unreliable, not recommended by experts.

  • Requires every Domus user to configure SPF correctly, even though that won’t guarantee delivery.

• Send email via our customer’s email servers

  • Technically infeasible (we’d have to collect login credentials, connect to hundreds of thousands of authenticated mail servers to send email, and deal with the myriad of different ways that could fail).

  • We’d be outsourcing the “email delivery” issues to our customers, rather than being responsible for delivering mail correctly on their behalf.

• Deliver email with a .domus.net “From” address

  • We can guarantee that messages are correctly authenticated.

  • We control the full infrastructure for the email sending.

  • We can track deliverability and bounces.

Ultimately, we made the decision to choose the third option, and change the way that we deliver emails, starting with the most important systems first.

The change we made in the last month changed the format of our emails to look like this:

Envelope Sender: bounce@domus.net

From: Joe Bloggs <messaging-service@domus.net>

Reply-To: Joe.Bloggs@kingsestateagents.com

To: A Customer <a.customer@customer.com>

Subject: We have found properties for you!

The Individual Changes

The “Envelope Sender” address is a domus.net address. This is the domain that is used in SPF to validate that our mail server is allowed to send this email. This address is not displayed in user’s email clients, and the unique part at the beginning of the email address allows us to track bounces and failures and update customer records so they receive no more emails.

The “From” email address is no longer configurable and is set to messaging-service@domus.net

The Reply-To address is whatever the sending address used to be (e.g. Joe.Bloggs@kingsestateagents.com). This is the address that any customer replies to your emails will be directed to and displayed to them, if they press the Reply button in their mail client.

Our testing showed that in many mail clients, the recipient of this email would see nothing untoward, and simple actions like reading and replying to emails would not display the “domus.net” mail addresses.

In some mail clients, the new email format means that recipients can now see “domus.net” in the emails that you send. This is an unavoidable side-effect of this technical decision, and it was not made for marketing reasons!

The reason that some mail clients display “domus.net”, is that those clients want to show the user the information they have used to make the “trust decision” that has lead to the email not being marked as spam. Our new email format leverages domus.net’s trust and email delivery mechanism, so the mail client wants to make it clear to the user who receives the email the reason why it has appeared in their inbox.

Bounce messages

There are several different types of replies that might happen to an email.

If a user is performing the reply, usually by clicking the “Reply” button in their mail client, then the reply will go to the “Reply-To” address, and arrive directly in the original sender’s inbox. If a user is using an old mail client that doesn’t understand the “Reply-To” address, or they manually reply to the “From” address of message-service@domus.net, then their reply won’t be sent to the original sender, but will rather be sent to the unmonitored mailbox hosted by Domus.

We don’t expect that this will happen too often, but in those situations that it does happen we want the experience for your customers to be understandable. So what we’ve done is have a robot “Auto Reply” to all of those messages with an “Oops!” message asking them to re-send their email to you rather than Domus. This mailbox is unmonitored, and we don’t read any of the emails that are erroneously sent there. Some time after the auto-reply is sent back to the original sender, the email will get deleted from our servers.

The other types of replies that will come from an email are sent automatically by the recipient’s mail server, and called “Bounces”.

Bounce messages will usually happen in one of two ways:

• An immediate failure

• A slower failure, where the recipient mail server initially accepts the mail message but then later decides it won’t deliver (e.g. the username doesn’t exist, or their mailbox is full).

We receive these bounce message to the Envelope Sender address of bounce@domus.net and then process them to mark the email address on our system as invalid by setting the customer to DoNotEmail.

Our Roadmap

Changing the way we send email is difficult, but also risky. We absolutely want to avoid making our customers’ emails fail to deliver. To this end, we’re gradually phasing in our adoption of the changes to our email sending process.

Implemented:

Step 1: Monitor email delivery by redirecting bounces to bounce@domus.net

Step 2: Change the from address of outgoing emails to a messaging-service@domus.net address.

Next steps:

Step 3: Move to a new email service using Amazon AWS instead of our existing email service which has had problems with load and blacklisting.

Step 4: Implement DKIM signing of domus.net email

This will allow us to cryptographically sign all emails, so that a recipient mail server knows that they definitely came from a Domus server and aren’t spam.

Step 5: Change all remaining email processes within Domus to use the new email system.

The end result of these steps will be that we’ll have a much higher quality of email that we’re sending, recipient mail servers will have a much higher confidence in the email they receive.

Frequently Asked Questions

Is it possible to have emails come from our mail domain rather than messaging-service@domus.net?

Once we have moved to Amazon AWS, it will be possible for us to provide instructions on changes to your DNS that will allow you to revert to sending from your own email addresses once we have verified the DNS changes have been put in place correctly. Please send an email to support@domus.net if you would be interested in this.

Some mail clients don’t display our company information any more.

This is because these tools are looking at the displayed “From” address to determine which company has sent the email. Unfortunately there is no way we can change the way we deliver email to allow these products to treat email originating from the Domus mail servers as coming from you.

Does Domus track the deliverability of our email?

Using Amazon AWS we are able to see overall how many emails are sent and how many bounces are occurring, but no other tracking is possible.

Can we have bounce messages returned to my email address?

As mentioned in the Bounce section above, this is not currently possible. It is something that we might look to implement in the future.

I use another SaaS product, and it is able to send email on my behalf. Why can’t Domus?

Our review of other SaaS applications, and the ways they send email on behalf of the users (impersonate), shows two main approaches:

• Incorrect emailing practices, where impersonated email will fall foul of the SPF or spam rules we’ve discussed above.

• Correct emailing practices, but which require the customer to set up SPF and DKIM records in their DNS, and don’t allow email impersonation until the SaaS provider has confirmed these settings are correct.