How to send e-mail in the modern age.
Introduction
Several of the big names in free e-mail have published their own "solution" to the SPAM problem. Some of them are simple, but difficult in real world circumstances. For example, some require that the HELO line for the mail server match the DNS entry for the IP sending the e-mail. This seems simple and harmless, but is a problem for companies that host virtual domains.
You should advise your recipients to add you to their address book, as many ESPs use this as a whitelist. Or, tell your customers that with many ESPs you will find the legitimate mail in the SPAM folder and SPAM in the INBOX because you did not want to pay their bribe.
Note: the only reliable way of sending e-mail is to pay each and every ESP a fee for them to pass your e-mail through without filtering. All other techniques still leave you at the mercy of over-aggressive SPAM filters. The logic is something like this: If 85% of all e-mail is SPAM, and all of it is blocked, then the filter is 85% accurate. But we will let some through to maintain our client base.
There is definitely a tiered Internet where e-mail is concerned. It seems that big providers and spammers are the only once that can afford these fees. But in theory the spammers will pay their fee, the ESP will receive a lot of negative feedback, and the spammer will loose their ability to send. The problem with this logic is that you are asking the people who receive a large check to cut of a source of future revenue.
Hotmail and Yahoo have mandated their own schemes. The purpose of this article is to explore those mechanism.
Sender ID/SPF
The idea is to verify that a sender of e-mail is allowed to send mail from the domain for which they are sending. It is assumed that the person controlling DNS for a domain can be trusted to say which IP addresses can send e-mail. The DNS administrator adds a TXT record to the DNS for the domain listing who can send e-mail.
More information can be found at Microsoft's SenderID portal or at OpenSPF.org.
The first problem for many of us is that our DNS provider does not support adding TXT records. This can be overcome by simply hosting your own DNS.
I decided to use bind 9 because I could easily yum install it. As it turns out, the caching name server that ships with Fedora was a good start for setting up a name server in my situation. My desktop machine is also my web server, mail server, and now DNS server.
My current /etc/named.conf shows how I've updated my named.conf for the best of both worlds. I still enjoy the caching name server for my local computers, and for external computers the domains I host are served. Windows boxes on your network will enjoy the speed up in DNS queries. It is important that zones be set up to avoid an external party from using your DNS server as a forwarder as this could lead to an undo amount of work for your DNS server.
The zone file for eder.us shows the SPF record for the domain.
You need to make sure your MTA is configured to respond with your domain name when the receiving e-mail server logs in to check it.
You need to make sure that a reverse look-up of your IP address returns your domain name. This is not something you can generally do with your DNS server, but is something that your ISP can sometimes be sweet talked into doing for you.
There are some really good tools at DNSStuff.com for checking DNS records, timing, SPF, and much more.
There are two wizards, one at Microsoft and the other at OpenSPF.org. The both seem to generate the same SPF record for me, but it is probably wise to generate both and compare.
Once you've updated your DNS and waited for the Internet to synchronize your changes, you are ready for the first test. Send an e-mail to check-auth@verifier.port25.com. A report will be sent back to you telling you what passes and what fails. Do not worry about the DomainKeys check as this can be configured later. You need to pass the mail-from check and the PRA check before you continue.
You should receive a reply back from port25 that looks something like this once you have both SPF and DomainKeys working.
Despite the framework that SPF employs, Hotmail will not do a DNS query to verify your SPF record until you tell them that you have one. Nor with they check your mail server for the HELO line. Send an e-mail to senderid@microsoft.com with your domain in the body. You can send several domains by putting one on each line.
All two full working days between when you send this e-mail and when you should be able to send to Hotmail correctly. You should receive a confirmation e-mail within a couple of hours, but it doesn't get propagated through their servers for a couple of days.
Note: now you are in the door, but you must still pass a rigorous SPAM filter. Make sure the body of your e-mail is at least 512 bytes and contains none of the words that spam filters tend to stick on. If SpamAssassin gives you a really low score, you have a decent chance of sending to Hotmail.
Domain Keys
Domain Keys is owned/patented/mandated by Yahoo.
Additional information can be found at Yahoo and implementations can be found at SourceForge.
Domain Keys is much more difficult to implement, but fortunately less important. I know of no ESP that currently blocks or penalizes e-mail that does not have it.
Domain Keys is a more comprehensive method of authentication. Like SPF, it relies on DNS to authorize a domain to send mail, but it is more subtle. The DNS record does not list the IP addresses of hosts that can send mail, but instead relies on private key distribution. If you have the key, you must be authorized to send e-mail. The public key, used to verify this fact, is published in DNS.
Domain Keys also authenticates an e-mail message using a cryptographic signature. Then entire message along with the headers is passed through a canonicalization process that should be tolerant of typical e-mail server mangling and then using RSA to sign the SHA1 sum of the canonicalized message. An e-mail header is generated and prepended to the message.
Because Domain Keys actually adds a header to a message, the MTA must be updated to support this functionality.
DomainKeys extensions exist for Sendmail, QMail, Postfix and many other MTAs, but installing them may require an endless search for CPAN perl modules that are still in alpha.
After two days of trying to get one of them working, I read the DomainKeys specifications and decided it was pretty simple. I use SquirrelMail and Postfix. In two hours I had a basic DomainKeys signer written in PHP. I integrated the signer into SquirrelMail and began testing.
In corresponding with other Squirrelmail developers, I found that my approach was somewhat flawed because I read the entire message into memory and used the OpenSSL functions to sign it. This becomes a problem when a large attachment cause it to exceed PHP's memory limit set in the config file for PHP.
I rewrote the signer to use an SHA1 algorithm that supported adding data in smaller chunks.
The advantage of this approach is that no perl daemon is needed to be running in the background. The disadvantage is that all e-mail must be sent through SquirrelMail. For me this is not a problem as I consider all native clients (especially ones from companies famous for their security problems) to be insecure.
To add DomainKeys to your DNS server, you must first generate public and private keys for your domain. Replace {your_domain.com} with your domain in the following commands.
mkdir /etc/domain_keys
openssl genrsa -out /etc/domain_keys/{your_domain.com}.private.key 1024
openssl rsa -in /etc/domain_keys/{your_domain.com}.private.key -pubout -out /etc/domain_keys/{your_domain.com}.public.key
Add this to your BIND zone file for the domain:
_domainkey.{your_domain.com}. IN TXT "o=-; r=postmaster@{your_domain.com}; t=y"
beta._domainkey.{your_domain.com}. IN TXT "k=rsa; p={your_public_key}"
Replace {your_public_key} with the contents of /etc/domain_keys/{your_domain.com}.public.key with the BEGIN line, END line and line feeds removed.
See my zone file for an example.