How to Reseed a Failed Mailbox Database Copy in Exchange Server 2010

When a mailbox database copy has failed in an Exchange Server 2010 Database Availability Group (DAG) it may be necessary to reseed the mailbox server with the failed database copy.

Exchange 2010 DAG with a Failed Database Copy

Exchange 2010 DAG with a Failed Database Copy

To reseed the database copy launch the Exchange Management Shell on the server that is in a failed state.

First we need to suspend replication for the mailbox database copy on this server.  Use the following command, specifying the mailbox database in the format “<mailbox database name>\<server name>”.

[PS] C:\>Suspend-MailboxDatabaseCopy -Identity "Mailbox Database 01\EX2"

Confirm
Are you sure you want to perform this action?
Suspending mailbox database copy "Mailbox Database 01" on server "EX2".
[Y] Yes  [A] Yes to All  [N] No  [L] No to All  [?] Help (default is "Y"): y

The mailbox database copy status will now change from “Failed” to “Failed and Suspended”.

Exchange 2010 Mailbox Database in Failed and Suspended State

Exchange 2010 Mailbox Database in Failed and Suspended State

Next we reseed the database with a new copy by issuing the following command.

[PS] C:\>Update-MailboxDatabaseCopy -Identity "Mailbox Database 01\EX2" -DeleteExistingFiles

The duration of the seeding process will depend on the size of the database and the speed of the network. When the update has completed it will automatically resume replication for the database copy. If you want to prevent automatic resume of replication use this command instead.

[PS] C:\>Update-MailboxDatabaseCopy -Identity "Mailbox Database 01\EX2" -DeleteExistingFiles -ManualResume
About Paul Cunningham

Paul is a Microsoft Exchange Server MVP and publisher of Exchange Server Pro. He also holds several Microsoft certifications including for Exchange Server 2007, 2010 and 2013. Find Paul on Twitter, LinkedIn or Google+, or get in touch for consulting/support engagements.

Comments

  1. Saved my day! :-)

  2. hi,

    can u tell me how to bring back PAM role of my 4-node cluster – after some problem with network i have operational cluster but cmdlet Get-DatabaseAvailabilityGroup showing empty PAM and OS

    –MC

    • Hi ZEFel,

      To see the PAM and Operational Servers fields populated you need to include the -status parameter with Get-DatabaseAvailabilityGroup

      ie

      Get-DatabaseAvailabilityGroup -Status | fl

  3. Dave Purscell says:

    Any way to limit bandwidth consumption during the reseed? Last time I did this I took down the VPN for several days. I really don’t care how long it takes to reseed if it can do so in the background.

  4. Dave Purscell says:

    Is there a way to pause the reseed process, so that we can continue it later during off hours.

    Based on progress measurements, it appears the reseed will take another 32 hours to complete. We are performing it on the weekend with minimal staff, but we are consuming 100% of the VPN bandwidth.

  5. Hein Traag says:

    Just wanted to pop in and say THANK YOU! We recently moved to a DAG environment and after a copy action two databases failed. Was able to get them going again with this how-to.

    Thanks again!

  6. David Fletcher says:

    Thanks for your help. This is a great site that gives real solutions to problems that I am having.

    This a little weird. Why does the update database copy work when I do it from the EMS but from the EMC, it always goes back into failed state? Do you do everything from the EMS?

    • I do most things from the shell. But the console gives you pretty much all the options you need for running a reseed, so I’m not sure why its failing for you. The event log should tell you more.

      • Dave Purscell says:

        David is correct. I just did another reseed this weekend. When tried from the EMC it failed (after 6+ hours) due to a single missing log file. I ran the update copy again from the EMS and it went perfectly. Perhaps some parameter which is not being sent from the EMC.

  7. Jack Schweigel says:

    We have to move all our Mbx Db on VMware from RDMs to VMDKs – 4 servers with 4 DB each = 16 DB. One DB active per server. Is there a variation of this process we could use? Something like – Add a new VMDK disk for each DB, incorporate into DAG, replicate, remove old (RDM) replicate DB and disks? Would need some specifics on that or a similar process.

  8. You sir are a lifesaver, panic/crisis averted!

  9. Richard Cunningham says:

    Hi Paul, just investigating some weirdness which has happened in our exchange env, would the loss of FSW cause a failed & suspended status on our DBs?

    Basically, ETS had disappeared from the local admins group where the FSW was hosted, which caused the cluster to not be able to contact it… this coincided with our DBs going into a failed state…

    • If quorum was lost then your databases will go offline. The loss of the FSW alone normally wouldn’t cause that, but I don’t know the full detail of your DAG.

      I do hear from people who have an undetected FSW issue, and find that their databases go offline when they’re doing patching/rebooting of the mailbox servers in the DAG.

  10. James Branch says:

    Awesome… this saved my day! Thanks Paul

    James

  11. Paul..

    This proces did not work for me….

    even after the Reseeding I still get the “failed suspended”

    is there any harm in going into EMC and remove the “failed suspended” passive copy from the server..

    will this impact the Active copy of the DB

    thanks

    Dave

  12. Paul, just a dumb question, it is the same to run it from the active or passive node? Or should be on a specific one? If you could tell me why i will be appreciated! thanks!

  13. Hello,
    Thanks you for your post. I Still have one question. Is it ok to execute the command if I only have 30 Gb of free space for a 200Gb database?

    Do you know if the
    Update-MailboxDatabaseCopy -DeleteExistingFiles

    first delete the problematic database and after reconstruct them.

    Thanks you very much

    Alain

  14. Andrew Roe says:

    I tried this process and I get the following error:

    A source-side operation failed. Error There isn’t enough free disk space to perform this operation.. [Database: DB9, Server: server1.acmecorp.com]

    it seems like it is not deleting the existing files first? would there be an issue with removing some files manually then re-trying the command or should I blow away the whole copy and create a new copy from scratch (750GB database)

    • Reseed includes the database file, content index files, and transaction logs. How much free space do you have on your DB and log drives for the source and destination servers?

  15. chander says:

    Your Article is SuperB, got me out of a jam. Same issue, Failed Status.
    Fyi for anyone interested, while running the command to Re-Seed I rec. this error:

    Error: A database backup is already in progress, Please verify that no other seeding or incremental reseeding operations are started for this database, and then try the operation again by rerunning the Update-MailboxDabaseCopy cmdlet…

    There was a backup by commvault running at the time of this reseeding, re-ran this command during the day and Voila this Article worked wonders.

  16. How to make the service replication running? the copy status in EX1 shows disconnected & resynchronizing.

    Failure mesage : A serve-side administrative operation has failed. The microsoft exchange replcation service may not be running on server EX01.___. com

    Specific RPC erro message : Error 0x6d9 (There are no more endpoints available from the endpoint mapper) From cli_GetCopyStatusEx2

  17. I have a cross site DAG all is well in each site in terms of copies, test-replicationhealth. When i go to add copy in other site i am getting source-side communication errors. socket operation was attempted and unreachable on DAG nic IP for the source server.

    any help would be great! thanks

  18. Not sure exactly which process is triggering this, but every attempt to reseed a large database was getting interrupted hourly with a communications error. I’ve attempted from both active and passive copies of the database, as well as putting the seeding source into maintenance mode first. Nothing helped and this was the error:

    A source-side operation failed. Error An error occurred while performing the seed operation. Error: Communication was terminated by server ‘MB1′: Data could not be read because the communication channel was closed..

    Smaller .edb files that could copy in less than an hour succeeded every time, but the larger ones that took over an hour would fail 100% of the time. Resuming was not an option when this error happened.

    Based on the event logs, I was able to figure out the timing coincided with Microsoft Exchange VSS writer attempting to do an hourly backup. Event 4088 complains it cannot backup database because the copy is in the ‘SeedingSource’ state, followed by event 2045 VSS Witer failed with error 80070015 when checking volume dependencies.

    This only happened for the seeding database, all other databases backed up fine. However, as soon as this occurred the seeding was fatally interrupted and event 4104 triggered on the source server “Seeding from the passive copy for database to the passive copy on server X was cancelled … because the copy configuration changed on the seeding source’.

    It is definitely related to logs being replayed. However, this is supposed to be suspended based on the description of Event 4102 (“Passive copy of database on this server is starting a seed … Log replay ill be suspended for this copy and will be automatically resumed when seeding is complete.”) However, I have circular logging enabled, so this is my guess, as we run no third-party software in this environment but YMMV. I’m not in a position to turn that off at this time to confirm.

    The fix for us was to disable the Microsoft Software Shadow Copy Provider and Volume Shadow Copy services while seeding the logs. When the backup/replay routine was called, VSS would output errors 13 and 8193 to the event log (both related to being unable to start due to service disabled) instead of doing the above. We let two cycles of the log replay get skipped until the seeding completed (2h:30m in our environment) and then switched both services back to manual and let everything catch up naturally.

    This may not work for you in a snapshot-heavy/VSS-required installation or when available bandwidth will take you days to seed – so it would be interesting to know a more elegant way of handling this issue.

  19. Aoa dear,
    please tell me one question my exchange 2010 is properly configured all rules are set, all user are using in 2 database . first DB1 is 750 users and 2nd DB1 is 750 …. one DB1 are working good but the anpther DB are show error in webmail did you have any good solution.

  20. Gaurav K. Passi says:

    Hi Paul,

    Once again, great and very helpful article!!!

    I’m running into a problem while seeding one of the databases to the passive server.

    We’ve 2 node multisite exchange 2010 dag. We’ve a dedicated 100 mbps connection between the sites.
    We’ve 4 mailbox dabases which are off different sizes. 2 small databases replicated properly and are online. 3rd database was of 75 gb in size and went into ‘Failed & Suspended’ mode after creating a copy. I was able to fix it using this very helpful article.

    My 4th database is of 195 gb and even after running this command, it’s still going into ‘Failed & Suspended’ mode. I’ve reseeded it twice but it doesn’t go into the ‘Healthy’ status even though the seeding completes without any issues.

    Can you think of any reason why this would happen. Can you please suggest a solution to fix this issue?

    Thanks in advance :)

    • Gaurav K. Passi says:

      Sorry, forget to mention that the ‘Copy Queue Length’ is about 642,000 and it didn’t go down after the full backup of DAG.

      My ping response to the other site is 28ms.

      I’ve ran the command to test the replication and it don’t show any problems:

      [PS] C:\Windows\system32>Test-ReplicationHealth

      Server Check Result Error
      —— —– —— —–
      EXC01 ClusterService Passed
      EXC01 ReplayService Passed
      EXC01 ActiveManager Passed
      EXC01 TasksRpcListener Passed
      EXC01 TcpListener Passed
      EXC01 ServerLocatorService Passed
      EXC01 DagMembersUp Passed
      EXC01 ClusterNetwork Passed
      EXC01 QuorumGroup Passed
      EXC01 FileShareQuorum Passed

      • Check your event logs. There should be some more info being logged when the DB copy status changes.

        • Gaurav K. Passi says:

          Hi Paul!

          Thanks for getting back so quickly :)

          I went through the event logs and it shows that the seeding process completed properly:

          Information event id: 4135, MSExchangeRepl:
          Seeding operation from the active copy of database ‘New York’ to the passive copy of the database on server EXC02 is complete.

          Then there is this event log following earlier one:

          Error Event ID: 4113, MSExchangeRepl:
          Database redundancy health check failed.
          Database copy: New York
          Redundancy count: 1

          The details of the event logs says:

          Error: Passive copy ‘New York\EXC02′ is not in a good state. Status: FailedAndSuspended.

          ErrorMessage: The required log file 621253 for New York\EXC02 is missing on the active copy. If you removed the log file, please replace it. If the log file is lost, the database copy will need to be reseeded using Update-MailboxDatabaseCopy.

          I reseeded the database again and it still went into ‘Failed & Suspended’ mode.

          How can I have my database copy fully replicated and working without dismounting the original database and manually copying it over as a file to the passive server?

        • I’ve seen a similar issue in the past, and the solution was to run a full backup of the database in question then try and reseed again. The full backup updates the log checkpoint and truncates the logs and should therefore resolve the missing log file issue.

        • Gaurav K. Passi says:

          Thanks!!! I’ll try it and will update you on the status :)

        • Gaurav K. Passi says:

          It worked this time. I just completed the reseeding of database copy and it’s healthy!!!

          Thanks alot for your help Paul :)

          One last thing. Are there any advanages or disadvantages in enabling ‘Circular Logging’ on the databases now that they’re fully replicated?

          The reason that I’m asking this is because we do ‘Full Exchange’ backups every night and if the circular logging is enabled then the changes are already been committed to the databases and it’s being replicated to the passive server also. Is it still worth to have the circular logging turned off.

        • For a DAG circular logging works differently to a standalone mailbox server/database.

          Some reading:
          http://blogs.technet.com/b/scottschnoll/archive/2011/06/27/circular-logging-and-mailbox-database-copies.aspx

  21. Hi there
    Please check the “Cluster Service” if it’s not started, this solution will not work.
    In my case, this was the problem why i couldn’t reseed the failed database copy.

  22. Anees Ur Rehman says:

    Hi Paul,

    I am facing a problem with DAG

    I have 2 Mailbox servers in San Francisco and DAG is working fine on both servers.
    Now I have to implement DAG on a Server which is in New York. Everything is ok. Servers are accessible. I am able to ping the DAG from every server. There is vpn tunnel between sf and ny zone.

    But when I started the replication it gives the following error from ecp
    Passive Failed and Suspended
    Copy queue length: 8067
    Content index state: Suspended

    Then I started to reseed from EMS, I got following error

    The seeding operation failed. Error: An error occurred while performing the seed operation. Error: Communication was terminated by server ‘SanFrMailBoX1′: Data could not be read because the communication channel was closed. [Database: SFDataBase, Server: newyorkmbx.xyz.com]
    + CategoryInfo : InvalidOperation: (:) [Update-MailboxDatabaseCopy], SeedInProgressException
    + FullyQualifiedErrorId : [Server=newyorkmbx,RequestId=cb11f9f1-4157-4e3a-a056-f35cf61db5ba,TimeStamp=2/7/2014 6:36:50 AM] 6079FA2A,Microsoft.Exchange.Management.SystemConfigurationTasks.UpdateDatabaseCopy
    + PSComputerName : newyorkmbx.xyz.com

    Please suggest.

    Thanks in advance

    • Hi Anees,

      Facing same error for DR copy across the WAN link. Please share if you have managed to identify the cause if the error while seeding.

      Thanks,
      Sumit

  23. What is the purpose of : MailboxDatabaseReseed.ps1 ?
    Is it doing the same job ?

  24. Paul Paul Paul Paul Your the MAN…! Blesses you saved my day
    cheers

    regards,
    imran from afghanistan

  25. Philip Kitheka says:

    Hi Paul,
    I am facing a problem with DAG in exchange 2010 sp1 environment.

    I have 2 Mailbox servers having a couple of mailbox databases (active/passive). i have been trying the reseeding operation on DTDB1 database without success. below is the error. the db size is 201.6GB. kindly assist.

    A source-side operation failed. Error An error occurred while performing the seed operation. Error: An error occurred while processing a request on server ‘EXCH-DRS-SVR02′. Error: An I/O error occurred while attempting to access file ‘I:\Program Files\ExchangeHome\Mailbox\DTDB1\DTDB1.edb’ on source server EXCH-DRS-SVR02. Error: SendSeedingDataInternal: PerformDatabaseRead fails with code 0xFFFFFC18. Expected 65536 bytes read 0.

    Copy Status: Failed and Suspended
    Content index state: failed
    Copy queue length (logs): 4718233
    Replay queue length (logs): 0

    Please suggest.

    Thanks in advance

  26. Hello Experts,

    My reseed has been running for days! 1 TB DB over wan link.. Can I run backup while reseed in progress?

    Thanks
    Tarra

    • Nope, reseed and VSS backups can’t run at the same time.

      1TB database size is a risk if you’re having to reseed across a WAN. I would prefer multiple, smaller databases in that situation.

  27. Hi – Your solutions worked fine but after multiple attempts as for some reason it was failing because of Win Rm service going down. Thanks a lot for your blog. keep writing :)

    on side note, do you know is there any way i can replace existing FILE witness server with new server ?
    we have two Exchange server running DAG and one file witness server (server 2003) and file server is failing for past few months and i was thinking to replace with latest 2008R2 File server.

    Thanks again

    • You can change the FSW any time using Set-DatabaseAvailabilityGroup. You just need to set up the permissions first (ie add Exchange Trusted Subsystem to local admins group on the new FSW) and also be aware that in Failover Cluster management tool the old UNC path will still appear in the name of the FSW resource.

  28. Adam Fowler says:

    Great instructions. Although I didn’t use the -manualresume switch, it didn’t automatically resume replication. I manually resumed and it worked perfectly.

Leave a Comment

*

We are an Authorized DigiCert™ SSL Partner.