A Guide to Back Pressure in Microsoft Exchange Server

For most Exchange administrators the first time they encounter the concept of “back pressure” is when they see this error:

452 4.3.1 Insufficient system resources

They might see it for the first time in a non-delivery report, an SMTP error log from an application, a telnet session, or the queue viewer on another Exchange server.

In this article:

• An overview of Transport service resource monitoring
• Customizing back pressure thresholds
• Detecting back pressure
  • Monitoring Transport queues
  • Monitoring event logs
  • Monitoring protocol logs

Microsoft Exchange Transport Service Resource Monitoring

Back pressure is the name for a condition that an Edge Transport or Hub Transport server is in when it is in an overloaded state and is actively refusing some or all further connection attempts from other systems.

The overloaded state is based on a series of resource utilization metrics:

• Free disk space on the drive(s) that store the message queue database and logs
• Uncommitted queue database transactions in memory
• Memory utilization by the EdgeTransport.exe process (the Microsoft Exchange Transport service)
• Overall memory utilization for the server

Each of those metrics is measured individually, and as such each is individually capable of causing the server to go into a back pressure state. There are two different levels of back pressure. as well as the condition where no over-utilization is occurring, so in total there are three resource utilization conditions that your Edge or Hub Transport servers can be in:

• Normal – all is well and the server is performing its role as intended (assuming you haven’t modified the back pressure settings to mask a genuine problem – more on that later)
• Medium – a resource is moderately over-utilized and the server begins limiting some connection types. Typically internal email flow remains functional while email from external or non-Exchange sources will be rejected.
• High – a resource is severely over-utilized. The server ceases to accept any new connections.

For disk space metrics the back pressure condition causes messages to be rejected. However for memory utilization metrics, before rejecting connections the server will first take actions to attempt to relieve the conditions.

For example, the server will perform garbage collection (reclaiming memory from unused objects) or flush the server’s DNS cache.

If after a certain number of polling intervals (which vary depending on the metric involved) the utilization is still above threshold, then the server will begin rejecting new connections as it does with disk space utilization.

Ultimately the problems you will actually notice are delays or a total lack of message delivery.

Customizing Back Pressure Thresholds

The metrics used by Transport server resource monitoring to trigger back pressure are based on a combination of configurable settings as well as fixed algorithms.

The configurable settings are stored in the EdgeTransport.exe.config file, located in the following directories by default:

• Exchange 2007: C:\Program Files\Microsoft\Exchange Server\Bin
• Exchange 2010: C:\Program Files\Microsoft\Exchange Server\V14\Bin
• Exchange 2013: C:\Program Files\Microsoft\Exchange Server\V15\Bin

If you installed Exchange Server to a different location then you will find the config file in the \bin folder of your installation directory. In Exchange 2010 and 2013 this is referenced by the environment variable $env:exchangeinstallpath.

EdgeTransport.exe.config

The first two values control whether resource monitoring is enabled, and at what interval resource monitoring is performed.

Though the option exists it is not recommended to actually disable resource monitoring. Instead the recommended approach is to identify the cause of resource over-utilization and correct it.

The same goes for basically all of the configurable settings for resource monitoring. Rather than spend time tuning the settings you should resolve the underlying issue, for example by adding disk or memory capacity to the server, or by adding additional servers to assist with overall email traffic load.

If you do happen to want to dive into the specific metrics and thresholds you can read more about them here:

• Understanding Back Pressure (Exchange 2007)
• Understanding Back Pressure (Exchange 2010)

Note: there is no Exchange 2013 documentation for this yet but my understanding at this stage is that it is largely the same as Exchange 2010 when it comes to resource monitoring.

Detecting Back Pressure

Back pressure can go undetected in your Exchange environment for quite some time if you are not monitoring for it. A Transport server can slip in and out of back pressure hundreds of times in a day and you could be completely unaware that it is happening if other servers manage to handle the email traffic load well enough that your customers don’t complain about delayed message delivery.

If you already have an Exchange-aware or event-based monitoring system running in your environment then then you are probably already monitoring for the signals I am about to describe, or can do so relatively easily with the monitoring system you have. For everyone else I will make a few suggestions for ways to detect these conditions.

Note that not all of these are practical to use for real time, proactive monitoring and alerting. But used as part of your regular server capacity monitoring they can be used to detect emerging capacity problems before they begin to seriously impact your environment.

Monitoring Transport Queues

Back pressure on a Transport server in one site can cause email to queue on Transport servers in other sites. Therefore by monitoring queue lengths you can detect the signs of back pressure.

You can use the Get-Queue cmdlet to manually check Transport queues on the local Transport server or on a remote server.

[PS] C:\>Get-Queue

Identity                 DeliveryType          Status MessageCount NextHopDomain
--------                 ------------          ------ ------------ -------------
HO-EX2010-MB1\866        SmtpRelayWithinAdSite Retry  3            hub version 8
HO-EX2010-MB1\Submission Undefined             Ready  0            Submission

One way to automate the checking of queues is with my Test-ExchangeServerHealth.ps1 script. However it is more suited to running as a scheduled report at a specific time of day, not as a continuous monitoring and alerting script (though you could customize it be one).

Get-Queue doesn’t always tell you the full story though. This is the Get-Queue output on an Exchange 2007 server that is in back pressure.

[PS] C:\>Get-Queue

Identity                DeliveryType Status MessageCount NextHopDomain
--------                ------------ ------ ------------ -------------
HO-EX2007-MB1\Submis... Undefined    Ready  0            Submission

Nothing appears wrong based on that. However in the event log:

Log Name: Application
Source: MSExchangeMailSubmission
Date: 25/08/2012 10:35:25 PM
Event ID: 1009
Task Category: MSExchangeMailSubmission
Level: Warning
Keywords: Classic
User: N/A
Computer: HO-EX2007-MB1.exchangeserverpro.net
Description:
The Microsoft Exchange Mail Submission Service is currently unable to contact any Hub Transport servers in the local Active Directory site. The servers may be too busy to accept new connections at this time.

The above event log entry is the Mailbox server’s mail submission to the local Hub Transport server in the site (which happens to be the same server in this particular case) being rejected due to back pressure on the Transport server.

Another option is to use Perfmon to monitor the aggregate delivery queue on each of your Transport servers in real time.

Aggregate Delivery Queues on Transport Servers (Nothing to see here)

Monitoring Event Logs

Even if you don’t have an event-based monitoring system you can still monitor event logs using PowerShell and the Get-EventLog cmdlet.

Back pressure conditions are logged to the Application event log under a series of event IDs:

• Event ID 15004: Increase in the utilization level for any resource (eg from Normal to Medium)
• Event ID 15005: Decrease in the utilization level for any resource (eg from High to Medium)
• Event ID 15006: High utilization for disk space (ie critically low free disk space)
• Event ID 15007: High utilization for memory (ie critically low available memory)

For example to search for instances of event ID 15004 in the past 24 hours you can run the following PowerShell command:

PS C:\> Get-EventLog -ComputerName ho-ex2007-mb1 -LogName Application -After (Get-Date).AddDays(-1) | where {$_.EventID -eq "15004"}

   Index Time          EntryType   Source                 InstanceID Message
   ----- ----          ---------   ------                 ---------- -------
   93560 Aug 25 22:24  Warning     MSExchangeTransport    2147760796 Resource pressure increased from Normal to High...
   93535 Aug 25 22:09  Warning     MSExchangeTransport    2147760796 Resource pressure increased from Normal to Medi...

A more detailed report can be generated with some scripting.

• PowerShell Script: Check Hub Transport Servers for Back Pressure Events

Monitoring Protocol Logs

When an Edge or Hub Transport server rejects a connection due to back pressure it uses a 4.x.x SMTP status code.

Using a similar Log Parser query to the one I shared in my previous article on reporting SMTP error codes, you can scan your protocol logs for evidence of back pressure events.

This command when run from the same directory that has the protocol logs in it will produce a report of any 4.x.x SMTP error codes.

"C:\Program Files (x86)\Log Parser 2.2\logparser.exe" "SELECT data as [Status Code],Count(*) as Hits FROM *.log WHERE data LIKE '4%' GROUP BY data ORDER BY Hits DESC" -i:CSV -nSkipLines:4 -rtp:-1

The results will appear similar to this (if any such errors are present).

Status Code                                                                Hits

-------------------------------------------------------------------------- -----

452 4.3.1 Insufficient system resources                                    19791

421 4.7.0 Too many errors on this connection, closing transmission channel 1298
421 4.4.1 Connection timed out                                             526
454 4.7.0 Temporary authentication failure                                 289
451 4.7.0 Timeout waiting for client input                                 32

You may be more interested in the per-day stats, in which case this query can be used:

"C:\Program Files (x86)\Log Parser 2.2\logparser.exe" "SELECT TO_LOCALTIME(TO_TIMESTAMP(EXTRACT_PREFIX(TO_STRING([#Fields: date-time]),0,'T'), 'yyyy-MM-dd')) AS Date, COUNT(*) AS Hits from *.log where (data LIKE '4%') GROUP BY Date ORDER BY Date ASC" -i:CSV -nSkipLines:4 -rtp:-1

The results will look similar to this:

Date       Hits
---------- -----
2012-08-08 12
2012-08-09 93
2012-08-10 72
2012-08-11 7
2012-08-12 48
2012-08-13 146
2012-08-14 460
2012-08-15 36
2012-08-16 389
2012-08-17 10345
2012-08-18 105
2012-08-19 252
2012-08-20 9631
2012-08-21 43
2012-08-22 133
2012-08-23 80
2012-08-24 57
2012-08-25 21
2012-08-26 6

Statistics:
-----------
Elements processed: 13181205
Elements output:    19
Execution time:     123.13 seconds (00:02:3.13)

This highlights the importance of taking samples of your server stats on a regular basis. While the above example makes it pretty easy to spot the significant increase in errors on two particular days, if your logging data did not cover a long enough time span then you may need to rely on previous benchmarks to spot problems.

It also highlights the importance of having diagnostic logging such as protocol logging turned on in advance of a problem occurring, so that you can begin to troubleshoot with all data immediately available to you.

Summary

Back pressure can cause serious problems in an Exchange Server environment due to the interruptions it causes to message delivery. Be sure to check your Transport servers for signs of back pressure, and take steps to resolve the underlying issues.