Sending Email From Your Web Site
March 09, 2010 — Code
If you’ve ever run a web site which sends email to its users you may have found yourself on the opposite end of the spam problem. That is, receiving servers think your emails are spam and refuse to accept them. If you’re sending customer invoices, order confirmations, or product download instructions, this is a serious problem. You may be tempted to use an external service for sending emails, and this may be a good solution in your case, but I would argue that any well-configured web server should be able to successfully send emails. What about the output of cron jobs? Or developer notifications when errors occur? Web servers need to communicate with us, and *nix systems are naturally inclined to do it through email. So how to we make it work?
There is a lot of advice out there on how to make sure messages are delivered properly (and if you’ve visited some of the URLs strewn about your mail server logs you may be wondering if any of it is worthwhile). Some actions you can take are very helpful. Some, like getting yourself off Comcast’s blacklist, are recipient-specific and probably not worth your time. Some are completely useless. In the following paragraphs I’ll summarize the advice that has worked for me, in roughly descending order of importance. Please note that I am not an expert on mail server admininstration, just a web developer who needs to send some emails!
1. Avoid Sending Spam
This may sound either obvious or irrelevant, but it’s really important to be familiar with the ways you could be sending spam without even knowing it.
Don’t Send Email To People Who Don’t Want It
Much spam filtering these days is done by techniques (such as Bayesian classification) which rely on users reporting received messages as spam or not-spam. If you don’t want your messages marked as spam, you should avoid sending them to people who might do so. You should also provide clear instructions for un-subscribing from your mailing list (and have an actual working un-subscribe procedure).
Configure Your MTA Properly
Before you install an MTA (Message Transfer Agent, eg: Sendmail, Postfix), make sure you’re familiar with basic configuration of the software, and the implications of running an MTA. Running an MTA is not like running a web server. You can cause real problems for yourself and others (see below). Be sure to read more than one tutorial on setting up your particular MTA, and don’t forget “obvious” things like setting the proper hostname/mailname so your messages come from the correct domain.
Don’t Be an Open Relay
An open relay is an MTA which can be used by anyone to send mail to any address. Unfortunately this is the default configuration for some MTAs. If you are an open relay it’s just a matter of time until a spammer finds you and uses your server to send thousands of emails, getting you blacklisted in the process. Repeat: you will certainly be blacklisted if you are an open relay.
Backscatter are bounced emails, usually that were never intended to be successfully delivered. Suppose a spammer wants to send an email to firstname.lastname@example.org. If they send an email to a non-existent user on your system that appears to be from email@example.com (forged sender address), your server will “return” that message to the victim. Clever, eh?
Easy Solution to Open Relay and Backscatter Problems
If your server’s main function is to be a web server, and it doesn’t host any user accounts that need to send email, I recommend simply rejecting non-local connections on port 25 (SMTP). I do this in Postfix with this line at the top of
127.0.0.1:smtp inet n - n - - smtpd
This ensures that you’re not even wasting CPU cycles communicating over malicious connections. If you need to accept SMTP connections, use authentication and make sure your HELO response matches your hostname.
2. Remove Your Server From Spamhaus’s Blocklists
Start by checking if you’re on Spamhaus’s blocklists. Chances are your server’s IP address is on the Policy Block List (PBL). If so, don’t get mad at Spamhaus. The PBL is a block list, not a black list, and Spamhaus isn’t telling everyone that you’re a spammer. Because most computers connected to the Internet don’t send email, all IP addresses are on the PBL by default. If you want to send email you simply get yourself de-listed (it doesn’t take long). This is a good thing. The only unfortunate part is that most people don’t know about it.
If your IP address is on the SBL or XBL you may have a problem. Before doing anything else, go back to step #1 and make sure you’re not sending spam. If you’re absolutely certain that your server is not sending spam you can try to get yourself removed from Spamhaus’s list, or contact your hosting provider for a new IP address (you may have inherited a previous customer’s problem).
3. Configure Reverse DNS Lookups
Not configuring reverse DNS is nearly equivalent in effect to being listed in Spamhaus’s PBL: many servers simply will not accept your messages. Look for reverse DNS configuration in your hosting provier’s control panel. To configure, just map your server’s IP address to your site’s domain name. Simple, yet effective.
4. Format Messages Properly
This is just common sense. There is a standard format for email (RFC 2821: Internet Message Format), so you should adhere to it. Some highlights from the RFC:
- each line should be no more than 78 and must be no more than 998 characters
- each line must be terminated with a CRLF
- headers must contain only US-ASCII characters (ASCII 32-126 inclusive)
You should also follow W3C HTML standards if you’re sending HTML email.
5. Create SPF Records
This is an important step for delivering messages to Google accounts. An SPF record is a TXT record in your domain’s zone configuration which authorizes servers to send mail from your domain. For example, to authorize only the servers named in your A and MX records, create a TXT record with this content:
"v=spf1 a mx ~all"
Be sure to include the quotes. If you need a different configuration you can use OpenSPF’s wizard to generate the record for you. I was able to confirm that my SPF record was working by noting the following header in an email sent from my server:
Received-SPF: pass (google.com: domain of firstname.lastname@example.org designates xxx.xxx.xxx.xxx as permitted sender) client-ip=xxx.xxx.xxx.xxx;
and the following in some spam which appeared to be from my domain:
Received-SPF: softfail (google.com: best guess record for domain of transitioning email@example.com does not designate yyy.yyy.yyy.yyy as permitted sender) client-ip=yyy.yyy.yyy.yyy;
I use Google Apps for email, and I imagine that if you use a different service which doesn’t care about SPF records, you will see no such confirmation. If anyone knows of a general method for checking SPF configuration could you please explain it in a comment?
6. Send All Messages From the Same Address
Google says you should do this, presumably so that you can be more accurately followed by address-based reputation tracking mechanisms.
7. Sign Email With DomainKeys or DKIM
I haven’t actually set up my MTA with DomainKeys or DKIM, but my guess is that they will be help when delivering messages to Yahoo! accounts. Basically, they provide cryptographic verification of an email’s origin. DomainKeys and DKIM are two different standards (you only need to implement one).
8. Use ‘Precedence: bulk’ for Bulk Emails?
This one is a little confusing. Google recommends using the Precedence header while RFC 2076 recommends against it. Ostensibly setting
Precedence: bulk lets recipient servers know that your email is not urgent and therefore helps the Internet operate more smoothly. You might think that this would identify your email as spam, but think about it: would a spammer really mark their messages as bulk email?
Use or not, as you see fit.
Sending email from a web server can be daunting, and repeated failure can be discouraging. I hope the above tips are helpful to you. You can check your status as a sender on Cisco’s SenderBase and try to move from Poor to Good. You can go from Poor to Neutral in just a few days, so don’t despair.
More CodeNov 06, 2008 Form Builders in Rails
Oct 06, 2009 Simulate Has Many Through HABTM
Dec 09, 2009 Single Table Inheritance in Rails
Jan 18, 2010 Anatomy of a Ruby Web Application
Apr 13, 2011 What the ɊȱɁͲ Is UTF-8? A Character Encoding Primer
Jun 29, 2009 Punting on Feedzirra Dependencies
Sep 15, 2009 Build Web Applications In Static HTML
Sep 21, 2009 How to Cause Useful Accidents