Server applications like MS Exchange can be very resource intensive. Resource consumption can be anticipated to a certain extent. For example given the maximum mailbox size one could calculate the necessary HDD space required by the store. The necessary processing power and RAM could also be computed from vendor data and experience.
However even if a server is commissioned by the book, guaranteeing the machine is always able to handle its load is nearly impossible. A server functioning perfectly could run into an extraordinary load peak that puts under test its true limits. This is what makes server monitoring so important.
Exchange 2007 built new monitoring capabilities right into the product core. Its new transport now includes a real-time monitoring feature referred to as Back Pressure and is implemented by both Hub and Edge Transport server roles. This monitors the key resources the transport is dependent on. If a resource shortage is detected, Back Pressure kicks in reducing the amount of load handled. This equates in stopping the delivery of some messages in an attempt to retain minimal functionality avoiding further resource depletion. Administration can then correct the situation without loosing the transport altogether. Of course if no action is taken, full Back Pressure could take effect blocking message flow completely.
A Close Encounter with Back Pressure
It did not take me long to discover Back Pressure. My virtual machines are often well below the recommended requirements, especially if installed for taking article screenshots.
When Back Pressure goes active the first thing you will notice are stuck messages. Since Exchange is based on the SMTP transport, we can see how Back Pressure rejects SMTP connections. Telneting Exchange on port 25 we got the rejection response:
450 4.3.1 Insufficient System Resources
The event IDs 15001, 15002 and 15003 logged by MSExchangeTransport are more informative. Here is an example:
Event Type: Warning
Event Source: MSExchangeTransport
Event Category: ResourceManager
Event ID: 15002
Date: 17/07/2007
Time: 13:53:35
User: N/A
Computer: BANG
Description:
The resource pressure is constant at High. Statistics:
Queue database and disk space ("C:\Program Files\Microsoft\
Exchange Server\TransportRoles\data\Queue\mail.que") = 68%
[High] [Normal=45% MediumHigh=47% High=49%]
Queue database logging disk space ("C:\Program Files\Microsoft\
Exchange Server\TransportRoles\data\Queue\") = 68% [Normal]
[Normal=89% MediumHigh=91% High=93%]
Version buckets = 1 [Normal] [Normal=40 MediumHigh=60 High=100]
Private bytes = 14% [Normal] [Normal=71% MediumHigh=73%
High=75%]
Physical memory load = 50% [limit is 94% to start dehydrating
messages.]
Inbound mail submission from other Hub Transport servers, the
Internet, the Pickup directory, the Replay directory, and the
Mailbox server, if it is on a Hub Transport server, has stopped.
Loading of e-mail from the queuing database, if available,
continues.
For more information, see Help and Support Center at
http://go.microsoft.com/fwlink/events.asp
This event shows the consumption level of the 5 resource vectors monitored. Here is the description for each of these taken from TechNet (see links below):
- Free space on the hard disk drive that stores the message queue database.
- Free space on the hard disk drive that stores the message queue database transaction logs.
- The number of uncommitted message queue database transactions that exist in memory.
- The memory that is used by the EdgeTransport.exe process.
- The memory that is used by all processes.
Consumption is reported in terms of percentages and tested against the configured limits. In all 3 limits are considered, Normal, Medium and High. Back Pressure will gradually start to reduce resource consumption as these limits are exceeded. For an explanation on how the default limits are computed and on the various Back Pressure stages check the references section.
Relieving the Pressure
Hub and Edge Transport server read their Back Pressure and various other settings from EdgeTransport.exe.config file. This is an XML file located under the Exchange bin directory. From here we could change the Back Pressure limits and even disable it altogether. However this is not wise. Back Pressure is a good feature, and we should make sure the necessary resources are available.
The file is composed of a series of name value pairs. Each setting is held within an 'Add' XML element. Some of the Back Pressure settings follow. However note that this file contains many other settings not related to this feature.
EnableResourceMonitoring - Turn on/off Back Pressure
ResourceMonitoringInterval - Amount of time between subsequent resource level tests
PercentageDatabaseDiskSpaceUsedNormalThreshold, PercentageDatabaseDiskSpaceUsedMediumThreshold, PercentageDatabaseDiskSpaceUsedHighThreshold - Normal, Medium and High thresholds for transport message queue disk space
For a full list of these settings see the references. Here our goal is not that of tweaking the monitoring settings, but is rather that of providing the necessary resources.
As a minimum we should make sure the transport has a good amount of disk space, memory and processing power. Keep in mind the machine resources are being shared between all applications running on the machine. The machine may be running multiple Exchange roles. Third party message hygiene applications will also find the transport to be a good place where to plug. All of these will add-up rendering the resource requirements harder to predict without proper testing.
Thanks to the 64-bit address space we should have plenty of memory extensibility. Exchange will by default put all databases on the installation drive. So hard disk space could be more of an issue. Luckily relocating the transport message queue database and transaction logs are an XML element away.
Create the directory where the new transport database/logs are to be held. Exchange won't create the directory tree for you.
Assign Full Control permissions to the Network Service, System and Administrators Group over the new directory location.
Open EdgeTransport.exe.config located under the Exchange bin directory.
Edit the XML elements for QueueDatabasePath and QueueDatabaseLoggingPath. The former identifies the queue database path and the latter identifies that for the transaction logs.
By default both point to:
<Exchange dir>\ TransportRoles\data\Queue
We will now change these to the new disk location, for example:
<add key="QueueDatabasePath" value="D:\Exchange\Queue" />
<add key="QueueDatabaseLoggingPath" value="D:\Exchange\Queue" />
Be careful when editing XML files. It is best to use an appropriate XML editor that ensures the XML is well formed.
Restart the Microsoft Exchange Transport service.
Exchange will now create new queue and transaction logs. The old files won't be deleted. After verifying that the new files were indeed created, then old files can be deleted manually. Here is the list of files involved:
Message Queue:
Mail.que and Trn.chk
Transaction Logs:
Trn.log, Trntmp.log, Trnnnn.log, Trnres00001.jrs, Trnres00002.jrs, and Temp.edb
Alternatively we could move the files to the new location. In this case instead of restarting the Microsoft Exchange Transport service in one step, we would first stop, move the files and then start the service again.
Final Tips
Back Pressure adds some native resource monitoring to the Exchange Hub and Edge transports. This is "must have" functionality for such a business critical application. Tweaking the Back Pressure settings is possible but not recommended. Instead we should make sure to provide the necessary resources.
If the machine keeps hitting the limits it is likely we have an overloaded server. Exchange allows for running the transport role on multiple boxes. In this manner the load can be split bringing resource consumption to normal levels again.
References
Understanding Back Pressure
Managing the Queue Database
How to Change the Location of the Queue Database