I specifically mention Domain Controllers twice there because both of these very common scenarios introduce the serious risk of a “USN rollback” condition occurring (USN stands for “update sequence number”). If you want to get deeply technical with the concept you can read this article from Microsoft:

How to detect and recover from a USN rollback in Windows Server 2003

If you just want the summary version, basically a USN rollback condition can occur when the Active Directory database is restored to an earlier version in an improper fashion. Microsoft makes available methods for restoring Active Directory databases such that the Domain Controller can properly resynchronise with its replication partners afterwards. Restoring in an improper fashion, such as restoring a DC using an earlier Ghost or Acronis image, or rolling back to an earlier snapshot of a virtualised DC, will cause a USN rollback condition to occur.

A Simple USN Rollback Scenario

You can create a USN rollback condition by deliberately performing one of the restoration methods mentioned above. Here I have created two virtualised Domain Controllers named TESTDC1 and TESTDC2, both in the testing.local domain. Looking at the servers I can see that they appear to be in a healthy state of replication. Active Directory Users and Computers shows the same user objects that I created have replicated between the servers.

Replmon.exe indicates successful replication is occurring.

Running DCDiag.exe /q (the /q switch suppresses all output except for errors, so if there is no output there is no errors) indicates all is well.

Next I shut down TESTDC2 and make a copy of the virtual hard disk. I then boot TESTDC2 again, and confirm once more that replication is healthy. I can then make a few changes to Active Directory to demonstrate the problems with USN rollback. Using the Active Directory Users and Computers console I create the user object User3 while connected to TESTDC1, and the user object User4 while connected to TESTDC2. As you can see here the user objects appear in the Active Directory of the Domain Controllers, but are yet to replicate between the two servers.

In a real world environment some event might occur such as a hardware failure on TESTDC2, or simply a human decision to roll the server back to the last image or snapshot. I shut down TESTDC2, remove the current virtual hard disk, and copy back the virtual hard disk file from before. As soon as I boot TESTDC2 again everything starts to go bad.

Administrators might first become aware of the problem when they notice that changes they make in the course of their day are not replicating to all the domain controllers. For example, I notice that User3 is appearing on TESTDC1 but not on TESTDC2, even after several hours of waiting. If I attempt to force replication between the two servers in Active Directory Sites and Services I receive an error. A similar error is also now appearing in Replmon, and the Directory Services Event log is showing some critical errors.

The event ID to look out for in this scenario is 2095. The full details of this event are as follows.

Event Type: Error
Event Source: NTDS Replication
Event Category: Replication
Event ID: 2095
Date: 1/06/2007
Time: 4:40:20 PM
User: NT AUTHORITY\ANONYMOUS LOGON
Computer: TESTDC2
Description:
During an Active Directory replication request, the local domain controller (DC) identified a remote DC which has received replication data from the local DC using already-acknowledged USN tracking numbers.

Because the remote DC believes it is has a more up-to-date Active Directory database than the local DC, the remote DC will not apply future changes to its copy of the Active Directory database or replicate them to its direct and transitive replication partners that originate from this local DC.

If not resolved immediately, this scenario will result in inconsistencies in the Active Directory databases of this source DC and one or more direct and transitive replication partners. Specifically the consistency of users, computers and trust relationships, their passwords, security groups, security group memberships and other Active Directory configuration data may vary, affecting the ability to log on, find objects of interest and perform other critical operations.

To determine if this misconfiguration exists, query this event ID using http://support.microsoft.com or contact your Microsoft product support.

The most probable cause of this situation is the improper restore of Active Directory on the local domain controller.

User Actions:
If this situation occurred because of an improper or unintended restore, forcibly demote the DC.

Remote DC:
d63ef566-f3a9-4700-ae27-a5c5ac7c9fe0
Partition:
DC=testing,DC=local
USN reported by Remote DC:
16435
USN reported by Local DC:
16387
For more information, see Help and Support Center at http://go.microsoft.com/fwlink/events.asp.

There are also instances of event ID 2013.

Event Type: Error
Event Source: NTDS General
Event Category: Service Control
Event ID: 2103
Date: 1/06/2007
Time: 4:40:20 PM
User: NT AUTHORITY\ANONYMOUS LOGON
Computer: TESTDC2
Description:
The Active Directory database has been restored using an unsupported restoration procedure.

Active Directory will be unable to log on users while this condition persists. As a result, the Net Logon service has paused.

User Action
See previous event logs for details.

For more information, see Help and Support Center at http://go.microsoft.com/fwlink/events.asp.

You definitely want to know about these errors when they occur. If you are running any kind of monitoring system that scrapes event logs and alerts you for certain events then these are two you want to be alerted for. If you are not paying attention this problem can surface and go unnoticed for quite some time. Your admins might just be scratching their heads a little as to why some odd behaviour is occurring in Active Directory. Meanwhile your server event logs are overwriting older events and may remove this crucial evidence, as happened to a customer of ours.

If you do not have the benefit of seeing those events in the Directory Services Event Log there are some other clues you can look out for. Firstly the repadmin.exe command can help identify the state of replication on the Domain Controller.

C:\>repadmin /options
repadmin running command /options against server localhost
Current DC Options: (none)

If the output is as above, then replication is not explicitly disabled on the Domain Controller. Note that a Global Catalog server will show an “IS_GC” option as being active instead of “(none)”. However if the output is as follows then replication has been disabled on the Domain Controller.

C:\>repadmin /options
repadmin running command /options against server localhost
Current DC Options: DISABLE_INBOUND_REPL DISABLE_OUTBOUND_REPL

More evidence is if the NetLogon service is in a “paused” state on the Domain Controller.

C:\>sc query netlogon
SERVICE_NAME: netlogon
TYPE : 20 WIN32_SHARE_PROCESS
STATE : 7 PAUSED
(STOPPABLE, PAUSABLE, IGNORES_SHUTDOWN))
WIN32_EXIT_CODE : 0 (0x0)
SERVICE_EXIT_CODE : 0 (0x0)
CHECKPOINT : 0x0
WAIT_HINT : 0x0

If you attempt to restart NetLogon and re-enable replication with Repadmin while the USN rollback condition is still in effect then the event IDs 2095 and 2013 will appear again in the Directory Services Event Log, which gives you further evidence of the issue. The final clue is by checking the USN that each Domain Controller believes is correct for itself and its replication partners.

On TESTDC1:
C:\>repadmin /showutdvec testdc1 dc=testing,dc=local
Caching GUIDs.
..
Default-First-Site-Name\TESTDC2 @ USN 16435 @ Time 2007-06-01 16:37:49
Default-First-Site-Name\TESTDC1 @ USN 14272 @ Time 2007-06-01 16:52:08

On TESTDC2:
C:\>repadmin /showutdvec testdc2 dc=testing,dc=local
Caching GUIDs.
..
Default-First-Site-Name\TESTDC2 @ USN 16409 @ Time 2007-06-01 16:52:49
Default-First-Site-Name\TESTDC1 @ USN 14146 @ Time 2007-06-01 16:04:22

The condition you are looking for is if the direct replication partners have a higher USN for the Domain Controller than the Domain Controller has for itself. In the above output you can see that TESTDC2 has a USN for itself of 16409, whereas TESTDC1 has an USN for TESTDC2 of 16435.

More Complex USN Rollback Scenarios

In the simple scenario above is it relatively easy to conclude that TESTDC2 is the cause of the USN rollback condition occuring. However in an environment with more than two Domain Controllers the evidence can present in different ways. Using the example of our customer, the virtualised Domain Controller that had been rolled back to an earlier snapshot was not showing event ID 2095, its NetLogon service was running, and it did not have inbound and outbound replication disabled. However both of its replication partners were showing those symptoms.

By analysing the USN numbers in the output of the “repadmin /showutdvec” commands on each Domain Controller it was ultimately shown that the virtualised Domain Controller was still the one causing the USN rollback condition.

Because of this type of variance in the real world it is important to investigate the situation carefully and assess all of the available information before making a decision as to how to proceed with resolving the issue.

Recovering from a USN Rollback

The Microsoft article mentioned at the start of this post contains instructions as to how to recover from USN rollbacks.

1. Remove Active Directory from the server causing the USN rollback condition. If you try to run DCPromo.exe on the server you will receive an error.

In order to demote the server you need to run DCPromo.exe with the /forceremoval switch. This is a last resort option for removing a Domain Controller when it cannot be removed by the conventional method.

2. Shut down the demoted server.

3. On a healthy Domain Controller, clean up the metadata of the demoted Domain Controller. This is explained in detail in this Microsoft article.

How to remove data in Active Directory after an unsuccessful domain controller demotion

The enhanced NTDSUtil application in Windows Server 2003 SP1 and above allows you to remove the metadata.
C:\>ntdsutil
ntdsutil: metadata cleanup
metadata cleanup: connections
server connections: connect to server testdc1
Binding to testdc1 ...
Connected to testdc1 using credentials of locally logged on user.
server connections: quit
metadata cleanup: select operation target
select operation target: list domains
Found 1 domain(s)
0 - DC=testing,DC=local
select operation target: select domain 0
No current site
Domain - DC=testing,DC=local
No current server
No current Naming Context
select operation target: list sites
Found 1 site(s)
0 - CN=Default-First-Site-Name,CN=Sites,CN=Configuration,DC=testing,DC=local
select operation target: select site 0
Site - CN=Default-First-Site-Name,CN=Sites,CN=Configuration,DC=testing,DC=local
Domain - DC=testing,DC=local
No current server
No current Naming Context
select operation target: list servers in site
Found 2 server(s)
0 - CN=TESTDC1,CN=Servers,CN=Default-First-Site-Name,CN=Sites,CN=Configuration,D
C=testing,DC=local
1 - CN=TESTDC2,CN=Servers,CN=Default-First-Site-Name,CN=Sites,CN=Configuration,D
C=testing,DC=local
select operation target: select server 1
Site - CN=Default-First-Site-Name,CN=Sites,CN=Configuration,DC=testing,DC=local
Domain - DC=testing,DC=local
Server - CN=TESTDC2,CN=Servers,CN=Default-First-Site-Name,CN=Sites,CN=Configurat
ion,DC=testing,DC=local
DSA object - CN=NTDS Settings,CN=TESTDC2,CN=Servers,CN=Default-First-Sit
e-Name,CN=Sites,CN=Configuration,DC=testing,DC=local
DNS host name - TESTDC2.testing.local
Computer object - CN=TESTDC2,OU=Domain Controllers,DC=testing,DC=local
No current Naming Context
select operation target: quit
metadata cleanup: remove selected server
Transferring / Seizing FSMO roles off the selected server.
Removing FRS metadata for the selected server.
Searching for FRS members under "CN=TESTDC2,OU=Domain Controllers,DC=testing,DC=
local".
Deleting subtree under "CN=TESTDC2,OU=Domain Controllers,DC=testing,DC=local".
The attempt to remove the FRS settings on CN=TESTDC2,CN=Servers,CN=Default-First
-Site-Name,CN=Sites,CN=Configuration,DC=testing,DC=local failed because "Element
not found.";
metadata cleanup is continuing.
"CN=TESTDC2,CN=Servers,CN=Default-First-Site-Name,CN=Sites,CN=Configuration,DC=t
esting,DC=local" removed from server "testdc1"
metadata cleanup: quit
ntdsutil: quit
Disconnecting from testdc1...

You must manually remove the records in the Forest and Domain DNS zones for the demoted Domain Controller, and remove if from the list of Name Servers for each of the zones.

Finally you must remove the demoted server from Active Directory Sites and Services.

4. If the demoted server held FSMO roles you can seize them with NTDSUtil.

Using Ntdsutil.exe to transfer or seize FSMO roles to a domain controller

5. Turn on the demoted server.

6. Promote the server to a Domain Controller again using DCPromo (if you wish it to have this role again).

7. If the server was a Global Catalog readd this role.

8. Restore FSMO roles to the server (if applicable).

9. Restore the System State (optional). This is only applicable if there was a previous, valid System State backup of the server from before the USN rollback condition occured in which there are Active Directory changes that you require to be restored.

Once the server is fully restored you can check that replication is occuring again with Replmon. You can also observe whether changes to the Active Directory are replicating properly by performing tests such as creating a new user object and waiting for it to replicate between servers.

Finally the Event Log, Repadmin output and the state of the NetLogon service of all of your Domain Controllers can be checked as well.

C:\>repadmin /options
repadmin running command /options against server localhost
Current DC Options: (none)

C:\>sc query netlogon
SERVICE_NAME: netlogon
TYPE : 20 WIN32_SHARE_PROCESS
STATE : 4 RUNNING
(STOPPABLE, PAUSABLE, IGNORES_SHUTDOWN))
WIN32_EXIT_CODE : 0 (0x0)
SERVICE_EXIT_CODE : 0 (0x0)
CHECKPOINT : 0x0
WAIT_HINT : 0x0

Once the recovery is complete you may not be out of the woods yet. As you may have noticed in the screenshots above the User4 object no longer exists in the Active Directory. This is because the object was never able to replicate outbound from TESTDC2 and was therefore lost when the server was forced demoted. You should be aware that aside from objects completely disappearing, some other strange issues may pop up after the recover, such as users reporting that their new password no longer works, but their old password does. In the case of our customer one of the Domain Controllers immediately threw some SAM errors into the Event Log due to a replication conflict for a particular computer account. As a result the computer account was deleted automatically by the Domain Controller, a change which then replicated across the network and required the computer in question to be rejoined to the domain.

As you can see the USN rollback condition is a very serious situation that threatens the integrity of your Active Directory environment. Aherence to proper backup and restore processes for your Active Directory, and caution when dealing with projects that involve virtualising Domain Controllers, can help you avoid this condition in your network. However if you do unfortunately experience this problem in your production environment, careful analysis of the evidence and a clear recovery process as demonstrated here can get your environment back into a healthy condition again.

Related posts:

• Recovering a single Domain Controller from a USN Rollback
• Exchange 2007 ADPrep Error Executing Ldifde.exe to Import Schema File
• Exchange 2010 Setup Error – The Exchange Server is in an Inconsistent State
• Exchange 2000 and Windows Server 2008 Domain Controllers
• Exchange 2010 Deployment: Preparing Active Directory

This article Event ID 2095 and The USN Rollback Adventure is © 2007 ExchangeServerPro.com

Get more Exchange Server tips at ExchangeServerPro.com