Included in Exchange 2007 and 2010, pipeline tracing is one of the features available for troubleshooting email transport issues. This gives us multiple dumps of emails as they flow through the various transport agents running on Edge/Hub Transport servers. In this manner we get a snapshot of the email just before and after each of the agents completes processing.
Although not specific to anti-spam, pipeline tracing is especially useful when analyzing how anti-spam and other email hygiene agents are handling and modifying emails.
Something about Agents
Before we start with the tracing examples let's have a quick look at the configuration side. Since pipeline tracing is intended to troubleshoot transport agents it's a good idea to start from the list of installed agents. From the shell run:
Get-TransportAgent
This is what I had on my Exchange 2010 test Hub Transport server (Note: This article also applies to Exchange 2007):
To follow this article it is not necessary for you to have exactly the same set of agents. However it would help if you follow the examples on a transport server that has the Exchange anti-spam agents installed. For details on the anti-spam agents check The Exchange 2007 Content Filter Agent. This article is applicable to both Exchange 2007 and 2010.
Configuring Pipeline Tracing
Pipeline tracing is configured and enabled on transport servers using the cmdlet:
Set-TransportServer
There are three settings that can be configured. Firstly we specify a sender address, whose emails are to be traced. Effectively this is a filter that is matched against the SMTP envelope sender address (MAIL FROM). If the filter is matched, the email snapshots are dumped to disk. This filter is critically important since tracing all emails could quickly exhaust the available disk resources.
Here I configure sender@adminstop.com as the pipeline tracing sender address on the HAMRUN transport server:
Set-TransportServer HAMRUN -PipelineTracingSenderAddress sender@adminstop.com
The next setting is the directory where the data is to be dumped. If not set this defaults to:
<Program Files>\Microsoft\Exchange Server\V14\TransportRoles\Logs\PipelineTracing\MessageSnapshots
To change the path to d:\pipeline on the HAMRUN transport server:
Set-TransportServer HAMRUN -PipelineTracingPath "d:\pipeline"
Lastly we need to enable pipeline tracing:
Set-TransportServer HAMRUN -PipelineTracingEnabled $True
To switch it off, rerun the last cmdlet with the flag set to $false.
Test 1 and 2: Tracing Non-Delivery Reports NDRs
In this article I will use telnet to submit test emails. This allows us to directly control the envelope sender address. If you need a refresher on how to submit emails using telnet check Telnet Port 25.
Tracing NDRs and other server generated emails is done by setting the PipelineTracingSenderAddress to an empty sender. Here is how I setup pipeline tracing for the HAMRUN transport server:
Set-TransportServer HAMRUN -PipelineTracingSenderAddress "<>"
Set-TransportServer HAMRUN -PipelineTracingEnabled $True
Configured in this manner, any email with an empty sender will be traced (not just those generated by Exchange). To confirm this submit an email using the command sequence that follows. From the command prompt:
Note: 192.168.0.60 is the IP of the test server HAMRUN. The Exchange email domain is wtest-dom1.local. You will need to replace these with your own values.
telnet 192.168.0.60 25
ehlo test.adminstop.com
MAIL FROM: <>
RCPT TO: user1@wtest-dom1.local
DATA
From: sender@adminstop.com
To: user1@wtest-dom1.local
Subject: Testing Pipeline Tracing
This is a test email
bye
.
Under the pipeline tracing folder you should find a new directory with the snapshots of this email. But before looking at this directory let's try a more interesting email. I will now send an email to an invalid recipient address so as to generate an NDR.
Note: For this to work make sure you don't have Recipient Filtering | 'Block messages sent to recipients that do not exist in the directory' selected. Otherwise the email would be rejected immediately and no NDR is generated.
Here is the command sequence of the next email:
telnet 192.168.0.60 25
ehlo test.adminstop.com
MAIL FROM: sender@adminstop.com
RCPT TO: WrongAddress@wtest-dom1.local
DATA
From: sender@adminstop.com
To: WrongAddress@wtest-dom1.local
Subject: Sending Email to Wrong Address
Sending Email to Wrong Address
bye
.
Note how the MAIL FROM is no longer an empty sender. On my test server there is no mailbox with the address WrongAddress@wtest-dom1.local. So Exchange will respond to this with an NDR that will be caught by pipeline tracing.
This is what my tracing destination directory looked like:
If you open any of the eml files you will see the NDR itself:
The multiple copies of the same email are what this tracing functionality is all about. It shows how the email change as it starts and finishes processing for each of the transport agents. Very often the differences are minimal. In other cases the changes are more substantial. It all depends on what changes the transport agents perform for that particular email. We will take a closer look at these EML files in the second part of this article.
Final Tips
Today we introduced transport agent pipeline tracing. We saw how this is configured and produced our first traces. In today's examples we captured an NDR showing how this functionality can be used to sniff emails generated by Exchange itself. In the second part we will immediately start with more examples, applying pipeline tracing to agent troubleshooting. We will also look at the snapshots and gain a better understanding of the information contained in the trace output.
References
Learning Transport Agent Pipeline Tracing Hands-on (Part 2)
Enable Pipeline Tracing
Using Pipeline Tracing to Diagnose Transport Agent Problems
Telnet Port 25
The Exchange 2007 Content Filter Agent