Cannot Activate Database Copy: Content Index Catalog Files in Failed State

When activating a database copy in an Exchange Server 2010 Database Availability Group it may fail with an error message that catalog index files are in a failed state.

exchange-2010-cannot-activate-database

——————————————————–
Microsoft Exchange Error
——————————————————–
Cannot activate database copy ‘Activate Database Copy…’.

Activate Database Copy…
Failed
Error:
An Active Manager operation failed. Error The database action failed. Error: An error occurred while trying to validate the specified database copy for possible activation. Error: Database copy ‘MB-HO-03′ on server ‘HO-EX2010-MB2.exchangeserverpro.net’ has content index catalog files in the following state: ‘Failed’.. [Database: MB-HO-03, Server: HO-EX2010-MB1.exchangeserverpro.net]

An Active Manager operation failed. Error An error occurred while trying to validate the specified database copy for possible activation. Error: Database copy ‘MB-HO-03′ on server ‘HO-EX2010-MB2.exchangeserverpro.net’ has content index catalog files in the following state: ‘Failed’..

When you view the copy status of the mailbox database the content index is in a failed state.

[PS] C:\>Get-MailboxDatabaseCopyStatus | fl name, contentindexstate

Name              : MB-HO-01\HO-EX2010-MB1
ContentIndexState : Healthy

Name              : MB-HO-04\HO-EX2010-MB1
ContentIndexState : Healthy

Name              : MB-HO-03\HO-EX2010-MB1
ContentIndexState : Failed

Name              : MB-HO-02\HO-EX2010-MB1
ContentIndexState : Healthy

There are two likely causes of this issue.

  1. The content index has failed due some error or fault occurring
  2. Indexing has been disabled for that database (a common configuration for databases hosting journal mailboxes)

Fixing Failed Content Indexes

To resolve the issue update the content index on the server on which it has failed.

[PS] C:\>Update-MailboxDatabaseCopy "MB-HO-03\HO-EX2010-MB1" -CatalogOnly

The content index should now be in a healthy state.

[PS] C:\>Get-MailboxDatabaseCopyStatus | fl name, contentindexstate

Name              : MB-HO-01\HO-EX2010-MB1
ContentIndexState : Healthy

Name              : MB-HO-04\HO-EX2010-MB1
ContentIndexState : Healthy

Name              : MB-HO-03\HO-EX2010-MB1
ContentIndexState : Healthy

Name              : MB-HO-02\HO-EX2010-MB1
ContentIndexState : Healthy

After the content index is healthy you can attempt to activate the database copy again.

Performing Database Switchovers When Indexing is Disabled

If indexing has been intentionally disabled you can still perform a database switchover using the Exchange Management Shell, by running Move-ActiveMailboxDatabase with the -SkipClientExperienceChecks switch.

[PS] C:\>Move-ActiveMailboxDatabase MB-HO-03 -ActivateOnServer HO-EX2010-MB2 -SkipClientExperienceChecks

Assuming no other issues exist with the database the move should complete successfully.

Comments

  1. Jeff says

    Hi,

    Thanks for detailing that out; however my databases are stuck in a permanent ‘crawling’ state and cannot run the -catalogonly command.

    Any suggestions would be greatly appreciated.

  2. Mike says

    This isn’t working for me. The command completes without errors, but the state of the database doesn’t go back to healthy, and the server keeps logging ExchangeStoreDB 123 events (which says to run the Update-MailboxDatabaseCopy command to reseed the catalog). This has affected two databases in the past two days. The first one was a new database, so I fixed it by deleting and recreating it, but that’s not a long-term solution. Any other ideas? I’ll keep searching. Thanks.

  3. Mike says

    I found two other articles concerning the same problem, with slightly different solutions, which sounded promising…

    http://msexchangeguru.com/2010/12/01/cannot-activate-database-copy-content-index-catalog-files-in-failed-state/

    http://blogs.msdn.com/b/pepeedu/archive/2010/12/14/exchange-2010-database-copy-on-server-has-content-index-catalog-files-in-the-following-state-failed.aspx

    Unfortunately, neither of these could resolve my problem. In the end, I had to just stop the Microsoft Exchange Search Indexer service on both nodes, then delete the Catalog folder, and restart the service. I just hope this doesn’t happen too often.

    Mike

  4. Jon says

    Hey Paul,

    Thanks for this info, it has worked both times I have had this issue. One question I have that I haven’t been able to find anywhere is what causes this to happen in the first place and how can I avoid it?

      • RobK says

        Noone is very sure about why that happens. We’ve seen it on nearly empty databases more often than on filled ones (for God’s sake!), and see it happen when the GC is not available in a Failover Cluster.

        Greetings,

        Rob

  5. talishka says

    Get-MailboxDatabaseCopyStatus | where {$_.contentindexstate -eq ‘Failed’} | Update-MailboxDatabaseCopy -CatalogOnly

  6. Elliston says

    Worked for me after changing my database paths and the copy wouldnt become active. Thanks – really great to have helpful people in the commuinity.

  7. Amit says

    Hey,

    Did you have information on failed index state as error message ” The Time stamp of the Catalog is too old for database”

    Please us know,

    Thanks,
    Amit

  8. Rax says

    if there are more than 1 mailbox server in to Exchange environment then you need to run following commnad to know actually which copy of the databse having content indexing issue.

    get-mailboxserver |Get-MailboxDatabaseCopyStatus | fl name, contentindexstate

    Based on the result, it will give us that which databse on which server is having status fail of content indexing

  9. Amit says

    Hi,

    Could you have details on contentindex state failed on active node database server, As of we reseed only passive copy with active node. In case its active node database server how it possible to make its contentindex state failed to healthy

    does any one have answer ?

    Thanks,
    Amit

  10. Rax says

    in that case, you need to go to the Exchange script folder where you have Active copy of DB with contect index Failed . you may find it is in default location which is C:\Program Files\Microsoft\Exchange Server\V14\scripts

    after going in to Script directory through EMS, run below command

    .\ResetSearchIndex.ps1

    It will ask you to press enter for continue. so say YES.

    Once it done, intial state of Contectindexing will show as crawling. later it will turn in to Healthy state.

    Cheers!

  11. Amit says

    When ran reset index for mailbox databases, seems this error
    Stop-Service : Service ‘Microsoft Exchange Search Indexer (MSExchangeSearch)’ cannot be stopped due to the following error: Cannot open MSExchangeSearch service on computer ‘.’.
    At C:\Program Files\Microsoft\Exchange Server\V14\Scripts\ResetSearchIndex.ps1:105 char:13
    + stop-service <<<< MSExchangeSearch
    + CategoryInfo : CloseError: (System.ServiceProcess.ServiceController:ServiceController) [Stop-Service],
    ServiceCommandException
    + FullyQualifiedErrorId : CouldNotStopService,Microsoft.PowerShell.Commands.StopServiceCommand

    I guess search service not respoding for reset,

    Any idea ?

    Thanks,
    Amit

  12. Ian Salgado says

    Hi,

    I have a Database which has a Catalog Index = FAILED

    The database IS NOT part of a DAG. It is a stand alone DB which is use for archiving emails (archiving function in Exchange 2010)

    The DB is currently in a DISMOUNT state because of the CATALOG INDEX FAILED & cannot be mount.

    I tried applying a command such as:

    Update-MailboxDatabaseCopy “Mailbox Database 01\EX1″ -CatalogOnly

    It does not work as the DB is not in a DAG.

    I have also try the following command:

    [PS] D:\Program Files\Microsoft\Exchange Server\V14\Scripts>.\Troubleshoot-CI.ps1 -Database “Mailbox Database 01\EX1″ –Action DetectAndResolve

    However, I did not have any luck with this one either.

    How can i correct this catalog index & hence be able to mount the DB ?

    Thank You in advance

    Ian Salgado

  13. Paul taylor says

    I have opened up a powershell command window as administrator, I have then typed get-mailboxserver |Get-MailboxDatabaseCopyStatus | fl name, contentindexstate and I get the following error

    get-mailboxserver : The term ‘get-mailboxserver’ is not recognized as the name of a cmdlet, function, script file, or
    operable program. Check the spelling of the name, or if a path was included, verify that the path is correct and try
    again.
    At line:1 char:1
    + get-mailboxserver |Get-MailboxDatabaseCopyStatus | fl name, contentindexstate
    + ~~~~~~~~~~~~~~~~~
    + CategoryInfo : ObjectNotFound: (get-mailboxserver:String) [], CommandNotFoundException
    + FullyQualifiedErrorId : CommandNotFoundException

    I am just learning Powershell so why wont it work?

  14. Ciprian Bacanu says

    Hi Paul,
    thanks for this post. I wonder what’s the solution when there’s only one server i.e. only one database.
    I noticed index state was failed, I reseed search index but perf mon crawler is telling me there is one mailbox to be seeded. This lasts from the 1st of November. Also, I have error id 4087:
    Failed to move active database ‘Mailbox Database DB1′ from server ‘ExSrv1′. Move comment: Database failover initiated by Responder ‘SearchLocalCopyStatusEscalate’ target resource ‘Mailbox Database DB1′

    Error: An Active Manager operation failed. Error: The database action failed. Error: This operation is not supported on standalone mailbox servers.

    Could you tell me what’s going on? Will crawling process stop? If not, how could I find the mailbox?

    Best regards,
    Ciprian

        • Ciprian Bacanu says

          Tarek, hello,
          Unfortunately I didn’t find out what was the mailbox that caused the trouble so I started out and created a new database where I moved some of mailboxes (I took reverse the size they had). No joy after rendex. I created a new one and moved another ones. At the third one I noticed that after reindex databases were healthy. So, I moved back users mailboxes and delete the temporary databases.
          I don’t have a lot of users and a had also a lot of space available – take care that moving a mailbox to another database does not free the space.
          I managed to find out the maibox looking the migration logs.

          Ciprian

  15. Mohammad says

    Hi Paul
    It’s one of the best Website which I used forever.
    Thank you for your short and good guides to resolve problems

    Best Regards

  16. Praveen says

    Hi Paul

    I have content index state on one DB as crawling for both the active and the passive noe. Any help would be appreciated

  17. Upendra says

    Hi Paul,

    Really it is very useful.

    I have one question , Can you explain how does replication works in DAG, if suppose if any log file got corrupted in passive copy in DAG , how identify and how will resolve it.

    Highly appreciated your quick response.

    Thanks,
    Upendra.

  18. Jeff says

    Hi Paul,

    Have you seen any issues with Exchange 2010 SP3 in EMC >> Server Config >> Mailbox. Under Database Copies > Copy Status not updating correctly with a refresh.

    I end up logging into another Exchange server just to see the current correct copy status.

    Thanks,
    Jeff

  19. RobK says

    Incredible to see how MS does nothing to address this little problem – the work on the overnext problem … erm… Exchange version seems to be more important than to auto-fix problems that are obvious; be it in the ource or by auto-healing.

    Sad to see that a company which claims to be powerful is so weak – but to be honest, I did not expect more…

    Rob

Leave a Reply

Your email address will not be published. Required fields are marked *