How to Reseed a Failed Database Copy in Exchange Server 2013

When you’re running an Exchange 2013 database availability group you will eventually need to deal with a failed database copy that needs to be reseeded.

Database copies may be in a failed state due to a variety of reasons, such as a hardware failure on the underlying storage system. After resolving the root cause of a failed database copy you then need to reseed the copy, which is a process that creates a new copy of the database by replicating the data from another DAG member that hosts a healthy copy.

To demonstrate how to reseed a failed database copy I’ve first caused a failure of one of my databases, which can be seen here in the output of the Get-MailboxDatabaseCopyStatus cmdlet.

[PS] C:\>Get-MailboxDatabaseCopyStatus *

Name                      Status
----                      ------
Mailbox Database 1\E15MB1 Healthy
Mailbox Database 2\E15MB1 FailedAndSuspended
Mailbox Database 1\E15MB2 Mounted
Mailbox Database 2\E15MB2 Mounted

If you’re looking for guidance to resolve failed content indexes in your database availability group, see PowerShell Tip: Fix All Failed Exchange Database Content Indexes.

The reseed operation can be started using either the Exchange Admin Center or the Exchange Management Shell (PowerShell).

Preparing to Reseed a Database Copy

There are a number of considerations when reseeding a database copy.

First, the time required for the reseed to complete will depend on the size of the database and the network performance between the source and destination servers.

By default the reseed will use the DAG member hosting the active database copy as the source.

If the database is 500Gb in size, that is 500Gb of database that needs to be copied across the network, plus the transaction log files and the content index for that database. This can add up to a lot of data that needs to travel across your network.

If the DAG members exist only within a single site connected by high speed LAN then this will not likely be a concern.

exchange-2013-reseed-from-active

However if the DAG members exist in multiple sites across a WAN then it may be more of an issue.

Fortunately, you can specify a source server for the database reseed, which allows you to select a server that has better connectivity to reseed from, such as another DAG member within the same site as the server with the failed database copy.

exchange-2013-reseed-from-passive

The options for selecting a reseed source will be demonstrated below.

Reseeding a Database Copy Using the Exchange Admin Center

Open the Exchange Admin Center and navigate to Servers -> Databases. Select the database that has the failed copy.

exchange-2013-reseed-failed-database-copy-01

On the database copy that is shown as failed click the Update link.

exchange-2013-reseed-failed-database-copy-02

You can click Browse and specify a source server if necessary, otherwise click Save to reseed from the server that hosts the active database copy.

exchange-2013-reseed-failed-database-copy-03

Wait for the reseed operation to complete.

exchange-2013-reseed-failed-database-copy-04

When the reseed operation has finished the database copy should now be healthy.

Reseeding a Database Copy using the Exchange Management Shell

We can also perform reseeds using the Update-MailboxDatabaseCopy cmdlet.

The database to reseed is entered in the format “Database Name\Server Name”, for example:

[PS] C:\>Update-MailboxDatabaseCopy "Mailbox Database 2\E15MB1"

To specify a source server for the reseed use the -SourceServer parameter as well.

[PS] C:\>Update-MailboxDatabaseCopy "Mailbox Database 2\E15MB1" -SourceServer EXMB3

If you receive an error message that log files already exist in the transaction log path for the database you can use the -DeleteExistingFiles parameter to tell the Exchange server to remove those files before beginning the reseed.

[PS] C:\>Update-MailboxDatabaseCopy "Mailbox Database 2\E15MB1" -DeleteExistingFiles

Finally, for long reseeds where you do not want to leave your Exchange Management Shell open, or when scripting a reseed and you don’t want the script to have to wait for the reseed to complete, you can use the -BeginSeed parameter.

[PS] C:\>Update-MailboxDatabaseCopy "Mailbox Database 2\E15MB1" -BeginSeed

Of course those parameters can be used in conjunction with each other, for example:

[PS] C:\>Update-MailboxDatabaseCopy "Mailbox Database 2\E15MB1" -DeleteExistingFiles -BeginSeed -SourceServer E15MB3

Monitoring Database Copy Health

If you do not already have a monitoring solution in place for your Exchange 2013 DAG I encourage you to try my Get-DAGHealth.ps1 or Test-ExchangeServerHealth.ps1 scripts.

Comments

  1. Derek says

    Hi Paul,

    I have a fairly new 2013 Exchange setup. I have noticed the following after rebooting one of the DAG members and was wondering if this behavior is expected, in particular the copy queue length? The server that was rebooted has the passive copy and the copy queue length was at 0 before the reboot but now it’s huge. I had a similar instance with the other DAG member when I rebooted it, the only difference was the contentindexstate was in a failed status. I tried updating only the catalog per one of your docs but it continued to stay in a failed state. After about 24 of no intervention on my part it fixed itself. Just trying to find out if the symptoms here are expected or if I have bigger issues.

    Thanks as always for sharing your knowledge. Your documentation has saved me on several occasions!

    Identity : mailbox\myserver1
    Id : mailbox\myserver1
    Name : mailbox\myserver1
    DatabaseName : mailbox\myserver1
    Status : Healthy
    LastStatusTransitionTime : 10/23/2013 3:15:07 PM
    MailboxServer : myserver1
    ActiveDatabaseCopy :myserver2
    ActiveCopy : False
    ActivationPreference : 2
    StatusRetrievedTime : 10/23/2013 3:17:09 PM
    WorkerProcessId : 8004
    ActivationSuspended : False
    ActionInitiator : Service
    ErrorMessage : The timestamp indicating the age of the last log generated on the active
    database copy is ’10/23/2013 3:16:54 PM’, which is stale or missing. As a
    safeguard, the system set the copy queue length to a large value to prevent
    losses greater than the automatic database mount dial setting.
    ErrorEventId :
    ExtendedErrorInfo :
    SuspendComment :
    RequiredLogsPresent :
    SinglePageRestore : 0
    ContentIndexState : Healthy
    ContentIndexErrorMessage :
    ContentIndexBacklog : 0
    ContentIndexRetryQueueSize : 0
    ContentIndexMailboxesToCrawl :
    ContentIndexSeedingPercent :
    ContentIndexSeedingSource :
    CopyQueueLength : 9223372036854501780
    ReplayQueueLength : 35
    ReplaySuspended : False
    LatestAvailableLogTime : 10/23/2013 10:14:37 AM
    LastCopyNotificationedLogTime : 10/23/2013 10:14:37 AM
    LastCopiedLogTime : 10/23/2013 10:14:37 AM
    LastInspectedLogTime : 10/23/2013 10:14:37 AM
    LastReplayedLogTime : 10/23/2013 9:25:56 AM
    LastLogGenerated : 9223372036854775807

  2. Paul says

    Seems like our remote Database over the WAN crashes much more often and reseeding it is a huge undertaking because the DB is so big, takes about a week.

    What are some of the causes of this? In the event viewer I get a bunch of corrupt log errors.

    At ’1/31/2014 5:15:48 AM’ the Microsoft Exchange Information Store Database ‘LWGDB01′ copy on this server encountered a serious I/O error. A lost write was detected. Consult the event log on the server for “ExchangeStoreDb” or “MSExchangeRepl” events that may contain more specific information about the failure.

    Information Store – LWGDB01 (5092) LWGDB01: Database E:\LWGDB01\LWGDB01.edb: Page 5675453 (0x005699bd) failed verification due to a timestamp mismatch. The ‘before’ timestamp persisted to the log record was 0x177b4100 but the actual timestamp on the page was 0x177b4082. The ‘after’ update timestamp 0x177b4101 that would have updated the on page timestamp. Recovery/restore will fail with error -566. If this condition persists then please restore the database from a previous backup. This problem is likely due to faulty hardware “losing” one or more flushes on this page sometime in the past. Please contact your hardware vendor for further assistance diagnosing the problem.

    • Paul says

      I ran a eseutil /r /l E00 F:\logdatabase. It fixed my time and said I had no logs required. However, the database in the DAG is still suspended. Is there a way to bring it back online without reseeding?

      • says

        You can resume a suspended database copy. If it keeps suspending itself automatically that indicates a problem, and there’ll be event log entries explaining why it was suspended. Often the situation involves a full reseed for a database copy that is too far out of sync with the other copies.

    • says

      Storage hardware issue perhaps. Sounds pretty messy and worth opening support calls with Microsoft or your storage vendor (or both).

      If you want to minimise reseed times use smaller databases (ie break up a large database into multiple smaller databases).

Leave a Reply

Your email address will not be published. Required fields are marked *