SPF is harmful. Adopt it.

You've come to this page because you've said something similar to the following:

SPF ("sender permitted from" a.k.a. "sender policy framework") is a scheme designed to prevent forgery of SMTP-based Internet mail and thus prevent unsolicited bulk mail ("spam"). AOL has already adopted it.

This is the Frequently Given Answer to such statements.

(You can find different approaches to this answer on John Levine's WWW page, in an article by Steven M. Bellovin, on David Woodhouse's WWW page, and on Brad Knowles' WWW page. By the way, whilst he agrees with what is said here about DNS security, take Brad Knowles' DNS server comparison, that he then refers to, with a large sackful of salt.)

SPF is harmful. The architectural ramifications of it are so extensive and will have such significant changes on the ways that people can access and can use Internet mail, that it would actually be less costly to switch to an entirely new architecture such as IM2000 Internet mail than it would be to switch to SPF and deal with all of its consequences properly.

Many of those architectural ramifications have either been incompletely addressed or not addressed at all as yet. Moreover, SPF usurps the meaning of an existing and widely used DNS resource record type for its own purposes, and has not yet been assigned its own actual resource record type. Anyone adopting SPF right now (which means actually adopting it, rather than merely paying it lip service) is adopting a scheme that can at best be described as woefully incomplete.

Most people who have analysed SPF in detail have come to the conclusion that it is a deeply flawed scheme that should be avoided outright.

On the gripping hand, maybe the fact that SPF is so damaging to the SMTP-based Internet mail architecture is a good thing. In the battle against unsolicited bulk mail, we have concentrated upon the wrong problem time after time, with mechanisms that address the wrong thing and that don't address the actual "unsolicited" and "bulk" qualities of undesirable mail. SMTP has become less usable, more patchy, and more balkanised with each new bodge. Perhaps the adoption of SPF will turn out to be the straw that finally breaks the camel's back, and that thus finally forcibly weans us off this bad habit of addressing the wrong problem.

So perhaps SPF is a deeply flawed scheme that should be adopted.

Ironically: SPF is also a good counter to one objection to IM2000 Internet mail, namely that it involves changing the structure of the mail system. If people sending mail and mail hosting companies are clearly willing to accept the massive structural changes that SPF will entail, they will be willing to accept the smaller structural changes that IM2000 Internet mail will entail.

Paying SPF lip service

10,000 domains cannot be wrong! claim the SPF marketing people. (That is actually quite a small number, of course. To gain perspective, note that in 2002-12-17 there were 22,009,173 domains under com. alone.) But publishing SPF data (aside from further entrenching the unauthorised hijacking by SPF of an existing DNS resource record type) is merely the paying of lip service to SPF.

And, ironically, research has shown that most of the domains that are publishing such SPF data are owned by UBM senders, proving, as if any more proof were needed, that we are still dancing the same old foolish dance of concentrating upon the wrong problem, changing something that is not directly related to "unsolicited" and "bulk" and seeing the unsolicited bulk mail change to match.

To properly adopt SPF, rather than just to pay it lip service, it is also necessary to configure one's SMTP Relay server to look up and to process the SPF data for all mail received.

The adoption rate of those who are actually adopting SPF properly, and not just paying it lip service, is rather more difficult to measure. The UBM senders publishing SPF data probably haven't adopted SPF fully, of course. Interestingly, though, many proponents of SPF have fallen silent when asked whether, in addition to paying lip service to SPF, they have also configured their SMTP Relay servers to check SPF data, leading to the conclusion that it is quite probable that many of the SPF proponents in those 10,000 domains are paying mere lip service to SPF too.

Some of the flaws in SPF

The flaws in SPF are numerous and severalfold.

SPF breaks pre-delivery forwarding.

SMTP-based Internet mail is, by design, a "store and forward" architecture. Mail is transported from machine to machine, stored and then forwarded on, until it finally reaches its destination.

SPF runs fundamentally counter to this architecture. The primary design of SPF is that the owner of a domain name publishes a list of SMTP Relay client machines associated with that domain, and that SMTP Relay servers reject mail with any such envelope sender mailbox names that does not come from an SMTP Relay client that is on that list. The architecture that SPF thus actually mandates is one where mail is delivered directly from originator to recipient, with no "store and forward" hops in between.

This is directly at loggerheads with the "store and forward" nature of SMTP-based Internet mail.

SPF is at loggerheads with RFC 1123

RFC 1123 describes simple mail exploders, aliases where the mail system replaces a particular envelope recipient with zero or more other envelope recipients and then forwards the message on to those new recipients. Many MTSes have such features. (For examples: This is the operation of forwarding instructions in .qmail files and of /etc/aliases in Sendmail.) Many people employ such features, moreover. (For examples: Companies will have "system-wide" alias mailboxes such as all-sales-representatives. Users will choose to forward all mail from one of their mailboxes to another.)

SPF breaks this. Simple mail exploders don't change the envelope sender. (Indeed, in the case of system-wide aliases, there's no user account under whose aegis the forwarding can be said to have occurred, and so no reasonable value to change the envelope sender to.) By declaring that mail with a particular envelope sender can only legitimately come from SMTP Relay clients that the domain name owner designates, SPF prevents the recipients of mail message from forwarding them on to further mailboxes, in the very "store and forward" manner that the SMTP-based Internet mail system is designed to do.

In essence, SPF allows domain name owners to impose an unwaranted degree of control over what recipients can do with their own mail, eliminating outright a common mechanism that recipients have heretofore had and have widely employed.

SPF is purported to come with a scheme called "SRS" that supposedly fixes this problem. But it doesn't. SRS is not even fully thought through, yet. It has massive problems of its own:

The only way to "fix" this problem with SPF is, as some of its proponents have asserted, to simply outlaw pre-delivery forwarding in toto and to presumptively declare the fact that SPF breaks it therefore to be a non-problem. According to those proponents, pre-delivery forwarding is simply wrong in concept and all forwarding must be post-delivery, i.e. performed by an MUA re-injecting a delivered message into the system as a new message with a new envelope declaring it to be from the forwarder.

The outright elimination of pre-delivery forwarding is, of course, a major change to the way that people use the mail system.

SPF is at loggerheads with RFC 974 and RFC 2821

One of the aspects of the "store and forward" design of SMTP-based Internet mail is that of fallback mail exchangers. SPF stops fallback mail exchangers from working.

By declaring that mail with a particular envelope sender can only legitimately come from SMTP Relay clients that the domain name owner designates, SPF prevents fallback mail exchangers from forwarding messages on to primary mail exchangers, because of course the domain name owners of all of the various envelope sender mailboxes haven't designated the SMTP Relay clients on the fallback mail exchanger as being legitimate sources of such mail.

Having every domain name owner list the SMTP Relay client sides of every fallback mail exchanger in the world in xyr SPF data is of course a completely unfeasible solution to this problem, because of scalability problems alone.

So therefore every SMTP Relay server that adopts SPF has to disable its use for (the SMTP Relay client sides of) every fallback mail exchanger for every domain that it accepts mail for. The consequences of this are at the very least twofold:

The same problem applies to organisations where the SMTP Relay server seen by the outside world is not the final mailhost, but instead mail is relayed internally across the organisation's own network. Again, the use of SPF validation on the parts of the internal SMTP Relay servers has to be disabled, yielding holes in the system, and the same consequences occur.

SPF hijacks existing DNS mechanisms.

Rather than creating a new DNS resource record type for the new data to be published in the public DNS database, SPF unilaterally redefines the meaning of an existing DNS resource record type, TXT, for its own purposes. This creates a conflict between SPF and all existing uses of TXT resource records.

SPF gives ISPs a "lock-in" weapon against their customers.

In the SMTP-based Internet mail system, the nature of the system allows people to "roam". People may employ their own mailbox names as the envelope sender, whatever place they may be submitting the mail into the system at. For examples:

SPF breaks this, providing ISPs (that provide mail hosting services to their customers) with a draconian lock-in weapon to wield against their customers. An ISP can employ SPF to prevent its customers from submitting mail, using those customers' own mailbox names as the envelope senders, from elsewhere other than the ISP itself, preventing the customer from submitting mail directly, via another ISP, or as a guest of another system.

SPF is useless for several entire classes of people.

Even leaving aside the desirability of doing so, there isn't even a way with SPF to publish reliable or meaningful SPF data for entire classes of people.

SPF is useless for SMTP Relay clients with dynamically-assigned IP addresses.

Whilst it is impractical and a bad idea (because it allows mail to be intercepted or to be lost) for an SMTP Relay server to listen on a dynamically assigned IP address, there is no similar prohibition on SMTP Relay clients connecting from dynamically assigned IP addresses. However, SPF is useless for people with such SMTP Relay clients.

Supposedly, the SPF "ptr" mechanism is there for dealing with publishing data about such SMTP Relay clients. However, this is just the flawed "double-reverse lookup" mechanism in a paper-thin disguise. It's a half-baked idea that is conceptually flawed and that doesn't actually work.

SPF is useless for roaming SMTP Relay clients.

A person may own an entire domain and wish to submit mail, using mailbox names in xyr own domain as the envelope senders, directly from a system with a dynamically-assigned IP address (such as, for example, a portable system that connects via multiple ISPs).

Ironically, the official SPF answer to the question of what SPF data to publish in these circumstances is, effectively, to not adopt SPF:

If you run a personal domain, you can either not publish SPF records at all, or set up "v=spf1 +all" for your domain, and you'll be able to send mail from your laptop no matter where you are.

SPF relies upon DNS for security, but DNS isn't a security service.

SPF places far too much trust in the DNS. DNS lookups are liable to attack by spoofed responses. (Many people are surprised when they learn how easy this is.)

Moreover, even aside from the potential for spoofing, by relying upon the DNS for SMTP Relay client validation information SPF relies upon attacker-supplied data, a foolish design for any sort of validation system. It's trivial for malicious senders to create throwaway domain names with published DNS data, that they supply, declaring any SMTP Relay clients that they like to be (as far as SPF is concerned) "legitimate". And there's no shortage of such domain names to use once and then throw away.

SPF is vulnerable to race conditions during database changes.

DNS is a distributed database. Resource record sets are cached in proxy DNS servers around the world. Changing information published in the DNS database involves careful management, in the shape of employing the "time to die" features of content DNS server softwares, if one is to avoid the case where some proxy DNS servers will have old information and others will have new information. Some content DNS server softwares simply do not have such features, and the case is thus unavoidable.

This means that there is a race condition whenever the (SPF-defined) "legitimacy" of an SMTP Relay client is revoked by changing the published DNS data. During the TTL period of the old TXT resource record set, some areas of the world will still consider that SMTP Relay client to be (what SPF defines to be) "legitimate".

SPF creates new categories of third class citizenship.

It's ironic, given its prior history of bodges to SMTP that AOL is presented as the champion of SPF.

AOL is already famous for dividing the Internet up into three classes of citizens, placing the customers of all other ISPs into the third class and refusing to have any dealings with them for mail service. SPF is just more of the same sort of discrimination. With SPF, the third class of citizens with whom AOL will refuse to have dealings is, of course, those who aren't locked into their ISPs for mail service.

SPF doesn't actually address unsolicited bulk mail at all.

"Sent by an SMTP Relay client that one doesn't expect" is not the same as "unsolicited bulk". Blocking mail with the former quality won't block mail with the latter.

Moreover: Consider the case of the unsolicited bulk mail generated by Microsoft Worms, for example. Microsoft Worms run on infected machines, and using the same mail submission tools that the machine's owner uses to submit normal mail, they mail themselves to other people. SPF does not and cannot distinguish such mail traffic generated by Microsoft Worms from normal mail traffic sent by the machine's owner. It's submitted to the same submission service, using the same user credentials, and travels along the same route, as all other mail.

SPF marketing

But SPF isn't an anti-UBM tool and was never portayed as such! goes the cry.

Someone appears to have forgotten to tell the SPF webmaster and Meng Weng Wong himself that, then.

SPF hands Verisign its next unwelcome "innovation" on a platter.

We have already witnessed that Verisign is prepared to put into action schemes to make itself more money at the expense of other Internet entities. It has already once made a land grab claiming all previously unowned *.com. and *.net. subdomains as its own, in order to sell advertising rights on web pages that it had set up associated with all of those domains, at the expense of many other Internet users using many other features of Internet.

SPF hands Verisign another nice little money spinner, in the form of enabling it to specify what SPF data are published for unowned domains by simply adding TXT resource record sets to its wildcards, and to sell the rights to have their SMTP Relay clients properly validated by SPF, for huge numbers of previously unowned domain names, to the highest bidding UBM sender.

Some ISPs already have "pink contracts" with UBM senders, essentially providing them with services in exchange for a cut of the take. And Verisign is already attempting to bring its *.com. and *.net. wildcards back into service so that vast numbers of people are directed to its web servers with the advertising space once again. It seems likely, therefore, that Verisign would find this additional money spinner very difficult to resist.

© Copyright 2004,2011 Jonathan de Boyne Pollard. "Moral" rights asserted.
Permission is hereby granted to copy and to distribute this web page in its original, unmodified form as long as its last modification datestamp is preserved.