Intelligent Message Filter, Content Filter, can do more...

Video Tutorial

Home
Anti-Spam
Understanding NDRs and other DSN Reports

Understanding NDRs and other DSN Reports

Alexander Zammit

Software Development Consultant. Involved in the development of various Enterprise software solutions. Today focused on Blockchain and DLT technologies.

What is a DSN?

NDRs are just one type of Delivery Status Notification emails. As the name implies, the notifications report on the status of email delivery. The standard is covered by RFCs 3461, 3462, 3463 and 3464 (see references for links).

I have myself dug into these in the process to beef up the IMF Tune v4.2 DSN and DN filtering capabilities. RFC 3461 defines a number of SMTP extensions, allowing email senders to request the generation of DSNs. RFCs 3462, 3463 and 3464 focus more on how DSNs are formatted and their content. Our discussion will mostly refer to the latter 3.

A DSN may report on any of these delivery events:

Failed Delivery (NDR)
Successful Delivery
Delayed Delivery due to temporary failures
Relaying to systems that may not support DSN generation
Address expansion causing the delivery to multiple recipients

As emails travel from sender to recipient, any of these delivery events may arise. For sure, each email should lead to at least one such event, most often the one for successful delivery. However, of course we do not receive a DSN for each submitted email. Instead unless the sender specifically requests it, reports are only generated for failed and possibly for delayed delivery.

In Outlook 2007 we can request for full DSN reporting per email by selecting 'Request a Delivery Receipt' under Tracking. Earlier Outlook version and other email clients also provide the same functionality.

In the above screen shot it is worth noticing the 'Request a Read Receipt' checkbox. Read/Deletion Receipts are part of the Disposition Notifications DN standard. These use a very similar reporting format but are not DSNs.

Who Generates DSNs

As an email finds its way to the recipient mailbox it travels through a number of email servers. Delivery responsibility changes hands each time the email is relayed from one server to another. In this manner the server accepting to deliver an email becomes also responsible for generating DSNs.

This point is important in understanding the challenge raised by NDR spam and is worth stressing. Consider a case where spam is sent to some domain whose server accepts responsibility to deliver the email. The server later discovers to be unable to complete delivery, most often because of a non-existing recipient address. It thus needs to report back by generating an NDR. However the spammer spoofed the sender address, causing the NDR to go to some other address. Thus an unrelated recipient ends up receiving the NDR.

Since the NDR sender is not the spammer, RBLs listing IPs of known spammers cannot be employed directly to filter DSNs. I suggest you to check the article Anti-Spam Golden Rules – Reject if You Can. This explains how rejecting an email is the correct way for a server not to accept responsibility for an email, freeing itself from generating any DSNs.

Inside DSNs

The DSN RFCs describe how these report emails are to be formatted. I will avoid excessive MIME technicalities. However having a general understanding can certainly be handy.

To begin let's have look at an NDR as shown in Outlook Express. This is useful as OE does not perform any special processing when displaying DSNs, allowing us to see how the information is organized.

Points to note:

The DSN email subject reads 'Delivery Status Notification (Failure)'. It is very common for DSNs to have such a generic subject. One could identify a large number of DSNs just from the subject, although there are better ways to do such identification.
The DSN email body reads: 'This is an automatically generated...'. Here we have the human readable report, giving a basic description of why the DSN was generated.
OE shows two email attachments. One is the original email itself. The second named ATT00013.dat is the dump of a more detailed DSN report.
Attaching the entire original email to a DSN is optional. A server may choose to only include the original email headers leaving out the message content. Alternatively the original message may be omitted altogether.
The strange name of the second attachment (ATT00013.dat) is automatically generated by OE. Depending on the email client in use this may not be visible, even though all RFC compliant servers will include it on generating an NDR. Here is what this attachment looks like. There is some interesting stuff here and will return to this shortly.

It is now interesting to see the same NDR as shown in Outlook:

Quite obviously the NDR looks different. The subject now reads 'Undeliverable: February 77% OFF'. Outlook must have dug this from the original email. Although more informative, this is not the true DSN email subject.

The body is also different from the one shown in OE. Here Outlook constructs a body combining the information within the DSN report (that OE showed as ATT00013.dat attachment) and the original message.

Finally Outlook does not show any attachments. Instead in this case it enables the Send Again button, from which we could get access to the Original Message ready to be resent.

The more user friendly presentation of DSNs in Outlook can be a bit misleading. For example consider the use of some filtering software to block emails by body keywords. One could easily be tempted to copy-paste some text from the body shown at Outlook. However since this is not the true email body, the filtering might not work. In fact this was one area where we worked on IMF Tune v4.2 to render filtering such emails more intuitive.

Having looked at the DSN from OE and Outlook it is time to close the circle and look at the raw email.

Here I numbered the most interesting parts and more details follow.

Point 1 - DSNs have a NULL address as Return-Path '<>'. This is meant to avoid the generation of any automated responses to the DSN.

Point 2 - The Content-Type header is the best way to identify DSNs and other report types. The multipart/report media type identifies a report email. The header parameter report-type=delivery-status is what identifies the report to be a DSN.

Points 3, 4 - Confirm that OE is the one showing the original DSN subject and body.

Points 5, 6 - Here we have the report that OE showed in a separate attachment named ATT00013.dat

Points 7, 8 - Here we have the original message also shown as an attachment in OE.

Delivery-Status Report

The delivery-status report shown as ATT00013.dat is worth revisiting. DSNs will store this in a MIME part having a content-type of message/delivery-status. This content-type is an excellent way for identifying DSNs.

A DSN reports on the delivery status for one or more recipients. The delivery-status report is composed of a number of header blocks. The topmost block includes headers common to all recipients. Following that, a number of header blocks follow, one per recipient.

Here I will list the header names specified by RFC 3464.

Standard per-message headers:
Original-Envelope-id
Reporting-MTA
DSN-Gateway
Received-from-MTA
Arrival-Date

Standard per-recipient headers:
Original-Recipient
Final-Recipient
Action
Status
Remote-MTA
Diagnostic-Code
Last-Attempt-Date
Final-Log-id
Will-Retry-Until

RFC 3464 includes a detailed description of each of these headers. Here are some of the most useful.

Original-Recipient/Final-Recipient - The optional Original-Recipient identifies the recipient to which the email was originally addressed. The required Final-Recipient header identifies the address to which the email was being delivered on generating the DSN. This may be different from the Original-Recipient as the address might undergo transformations as it travels to its destination. In our earlier example we had:

Final-Recipient: rfc822;user@dummydomain.abc

The initial part of the header value "rfc822", identifies the address type. In this manner it allows DSNs to cater for email systems supporting different address formats.

Action - This header identifies the DSN type. It could have any value from: failed, delayed, delivered, relayed and expanded. In our example we looked at an NDR, thus the action value was failed:

Action: failed

Status - Here we find a code giving more specific delivery status information. This is composed of 3 numbers separated by periods formatted as Class.Subject.Detail. Class can be set to 2 - Success, 4 - Persistent Transient Failure or 5 - Permanent Failure. The Subject and Detail portions are up to 3 digits each and add a lot more status specific information. Here is the Status value taken from our example:

Status: 5.5.0

Referring to the RFC the value signifies a Permanent Failure described as "undefined protocol status". This is not so informative, but other possible Status codes are also defined.

Diagnostic-Code - This header returns the diagnostics returned by the email transport. Referring to our example we find an SMTP rejection response

Diagnostic-Code: smtp;550 user@dummydomain.abc... No such user

Final Tips

DSNs are an integral part of our email system. Understanding how these are generated and the information delivered is important for us to make the best use out of them.

NDR Spam exploits servers that too hastily accept responsibility for emails they cannot deliver. Immediate email rejection takes us a long way in freeing the internet from such spam. However for the moment, we did better learn how to identify, and separate legitimate NDRs from spam...