Exchange Server Pro » Disaster Recovery

Google Discontinues Message Continuity

Paul Cunningham — Fri, 03 Feb 2012 12:29:55 +0000

Google has announced that they are discontinuing Google Message Continuity.

Google Message Continuity (Source: http://www.google.com/postini/continuity.html)

Originally launched as a disaster recovery and business continuity solution for Microsoft Exchange customers it received a lukewarm reception from Exchange admins, and according to a Google blog post did not see significant uptake.

In December 2010 we launched an email disaster recovery product for enterprise customers that use Google’s cloud to back up emails originally sent or received in an on-premise, Microsoft Exchange system. In the time since we launched, we’ve seen hundreds of businesses sign up for it. By comparison, in that same time, we’ve seen millions of businesses move entirely to the cloud with Google Apps, benefitting from disaster recovery capabilities built directly into Apps. Going forward we’ve decided to focus our efforts on Google Apps and end support for GMC.

Exchange 2010: Are Passive Database Copies Included in Backups of DAG Members?

Paul Cunningham — Sun, 06 Nov 2011 04:23:45 +0000

In my article on backup error event id 2137 and what it means for Exchange 2010 a reader asked whether backups of Exchange 2010 DAG members include both the active and passive database copies.

Update: according to Microsoft, “All database copies can be backed up using an Exchange-aware, Volume Shadow Copy Service (VSS)-based backup application. However, the built-in support for Windows Server Backup is for active copies only. You can’t use Windows Server Backup to back up passive copies.” The rest of the article below has been left unedited, however be aware that the example demonstrated is of an unsupported scenario

The short answer is yes, however there are some important considerations to be aware of. From Microsoft TechNet:

Passive mailbox database copies are backed up using a separate VSS writer in the Microsoft Exchange Replication service. The Microsoft Exchange Replication service VSS Writer doesn’t support restores. Although you can back up a passive mailbox database copy using Microsoft System Center Data Protection Manager or a third-party Exchange-aware VSS-based application, you can’t perform a VSS restore directly to a passive mailbox database copy. However, you can perform a VSS restore to an alternate location, suspend replication to the passive copy, and then copy the database and log files from the alternate location to the location of the passive database copy in the file system.

To demonstrate this, here you can see the mailbox databases MB-HO-* that are currently active on server HO-EX2010-MB2. They have no LastFullBackup timestamp because they have never been backed up.

[PS] C:\>Get-MailboxDatabase -status | ft name,mounted*,lastfull* -auto

Name     MountedOnServer                     Mounted LastFullBackup
----     ---------------                     ------- --------------
MB-HO-01 HO-EX2010-MB2.exchangeserverpro.net    True
MB-HO-02 HO-EX2010-MB2.exchangeserverpro.net    True
MB-HO-03 HO-EX2010-MB2.exchangeserverpro.net    True
MB-BR-01 BR-EX2010-MB.exchangeserverpro.net     True

On server HO-EX2010-MB1 I ran a mailbox server backup using Windows Server Backup.

After the backup completed the following can be seen. First, the LastFullBackup time stamp has not been written.

[PS] C:\>Get-MailboxDatabase -status | ft name,mounted*,lastfull* -auto

Name     MountedOnServer                     Mounted LastFullBackup
----     ---------------                     ------- --------------
MB-HO-02 HO-EX2010-MB2.exchangeserverpro.net    True
MB-HO-01 HO-EX2010-MB2.exchangeserverpro.net    True
MB-HO-03 HO-EX2010-MB2.exchangeserverpro.net    True

Next, the disk that the backups were written to contain VHD files matching the size of the data stored on the backed up volumes. For example, the F: (Data) drive holds about 2Gb of data, and the VHD for that volume is about the same size.

Because we can mount the VHD as a disk on the server we have the opportunity to inspect the contents inside of it.

We can see that even though it was not the active databases at the time of the backup, the EDB file for database MB-HO-01 (and others) was still backed up by the server.

[PS] C:\>Get-ChildItem H:\ -Recurse | where {$_.name -like "*.edb"} | select name

Name
----
HO-PF-01.edb
MB-HO-01.edb
MB-HO-02.edb
MB-HO-03.edb
PF-HO-01.edb

Next I’ll run another backup of HO-EX2010-MB1, but this time the mailbox database MB-HO-01 has been made active on HO-EX2010-MB1.

This time the LastFullBackup time stamp is written to the database.

[PS] C:\>Get-MailboxDatabase -status | ft name,mounted*,lastfull* -auto

Name     MountedOnServer                     Mounted LastFullBackup
----     ---------------                     ------- --------------
MB-HO-01 HO-EX2010-MB1.exchangeserverpro.net    True 11/6/2011 1:40:19 PM
MB-HO-02 HO-EX2010-MB2.exchangeserverpro.net    True
MB-HO-03 HO-EX2010-MB2.exchangeserverpro.net    True
MB-BR-01 BR-EX2010-MB.exchangeserverpro.net     True

So what do we learn from this?

Reminder, the below should be read in conjunction with the notice at the start of the article, as backing up passive database copies using Windows Server Backup is not a supported scenario

Yes, passive database copies stored on a volume are backed up by Windows Server Backup
Yes, this allows backups of passive database copies to be restored, although not directly
Yes, this would allow a backup of a passive database copy to be used for disaster recovery of a database
No, the backup time stamp is not written when a passive database copy is backed up, so any monitoring of backups that involves checking that time stamp needs to take that into consideration

Get more Exchange Server tips at ExchangeServerPro.com

Review: Exchange Server 2010 Backup and Recovery Training

Paul Cunningham — Sun, 21 Aug 2011 12:23:35 +0000

How many times have you seen a forums thread that goes something like this:

System administrator has recently started at a new job
Something goes wrong – an Exchange server crashes or somebody deletes some emails they didn’t mean to
System administrator looks into it, and discovers that the Exchange server is not being backed up
They don’t know what to do next

I’ve seen that a lot. I’ve also had my fair share of phone calls with customers dealing with Exchange server problems and asked them when was their last successful backup. Sadly the answer is often “I don’t know”.

Exchange Server Backups – Serious Business

Sometimes I really do wonder if people realise how important Exchange Server backups are. Often the people that do realise it are those that have been burned by a serious disaster in the past.

Nobody wants to be the IT guy who has to explain why data has been lost forever. But a stunning number of people I meet really don’t know that much about backup and recovery, particular for Exchange servers.

I suspect that one of the reasons is that there is no Microsoft certification for backup and recovery. With no exam to study for, and no credential to earn, it just isn’t a topic that many people seem to proactively learn.

But take it from me, Exchange Server backup and recovery skills are among the most important ones that you can have as an IT professional.

Exchange Server 2010 Backup and Recovery Training

Because of how important these skills are, and because of how many nights I’ve lost trying to recover data for customers, I’m very glad to see that TrainSignal has come out with an Exchange Server 2010 Backup and Recovery training course.

The instructor for this course is J. Peter Bruzzese, the same guy who teaches the 70-662 and 70-663 training courses from TrainSignal. As an Exchange MVP, MCT, and long time consultant J. Peter is the right person to teach this subject, and he has delivered yet another high quality training program.

The training goes into more depth on the concepts of Exchange Server 2010 backup and recovery than most other training I’ve seen. Backups tend to get shoe horned into a single module of broader product training. This is the first training program I’ve seen that sticks to just the topic of backup and recovery – deep dive training at it’s best.

There is a good mix of theory and practice in the 16 lessons. To give you an example of why the theory is so important here is the overview for Lesson 5: Storage Architecture and Backup/Recovery Basics.

If you truly want to be able to recover your data, you need to understand the storage architecture in Exchange. That may sound deeper than it is, but ultimately we are looking at how Exchange takes an email and puts it in a database, and the steps along the way. Then we will discuss some best practices and theory with those databases, along with an understanding of Exchange provides some built in resiliency to help you avoid having to restore from a backup (because that is the LAST thing you really want to do).

That is the kind of deep understanding that is hard to pick up just from on the job experience.

The practical lessons are also very useful. It would be difficult to create a practical training course that covers everything about backup and recovery because there are just too many different backup products out there.

In this training program you’ll learn how to use Windows Server Backup and Microsoft Data Protection Manager to backup and restore Exchange Server 2010. Those are fine, but probably aren’t going to be the most common backup solutions you encounter in the real world.

However they do teach you the most important side of Exchange Server 2010 backup and recovery – the Exchange server side of it. Those are the most important skills to have, and you can carry them between different backup solutions as you need to.

What You Get with TrainSignal’s Exchange Server 2010 Backup and Recovery Training

When you buy the training package you get instant online access to the lessons, as well as a set of physical DVD’s that are shipped out to you. I’ve always enjoyed how quickly they show up at my door, usually within 7 days and for me they have to be shipped all the way to Australia. I’m used to waiting weeks for stuff to arrive from overseas.

There are nearly 8 hours of video lessons. This is focussed training that you can sit down and work through in a few days or a week and quickly achieve a result.

The DVD’s also include MP3 audio versions of the lessons. I’m a big fan of audio based learning, especially for the more theory-based lessons. Listening to them multiple times while travelling or doing other tasks really helps the information sink in.

Like I said earlier, there is no certification exam for this topic. You won’t earn an MCP or MCTS by completing the training. But you will learn some of the most valuable, and unfortunately the most rare, technical skills for IT professionals working with Microsoft Exchange Server.

Click here to find out more about TrainSignal’s Exchange Server 2010 Backup and Recovery Training.

Get more Exchange Server tips at ExchangeServerPro.com

Exchange Server 2010: Using Database Portability for Disaster Recovery

Paul Cunningham — Wed, 13 Jul 2011 13:23:49 +0000

Exchange Server 2010 has a feature called Database Portability. This is the capability for a mailbox database from one Mailbox server to be mounted on another Mailbox server within the same organization.

In this article I will demonstrate how you can use Database Portability in a disaster recovery situation.

First we need to understand a few key points about Database Portability:

the mailbox database can only be mounted on another Mailbox server within the same Exchange organization
the mailbox database can’t be from an earlier version of Exchange Server
the database can’t be a public folder database

In this example scenario a Mailbox server has failed, and another Mailbox server is available to mount the database from the failed server.

There are different approaches that could be taken instead of using Database Portability, such as:

if the Mailbox server was in a Database Availability Group then the database may be able to be activated on another DAG member that already holds a replica of the database
in some cases a Mailbox server recovery may be more appropriate

The first step is to get the database and log files onto the server where you want to mount them. This could be done by restoring them from backup, or by attaching the same storage that was on the old server to the new server.

Before we proceed further we need to check what shutdown state the database is in. Open the Exchange Management Shell and navigate to the folder containing the database file. Then run the following command to check the database state.

g:\Data\MB-HO-01>eseutil /mh MB-HO-01.edb | findstr "State:"
            State: Dirty Shutdown

With a database in a dirty shutdown state, and all of the transaction log files available, we can perform a soft recovery of the database. This commits any uncommitted transaction logs into the database so that there is no data loss. If you don’t have the transaction logs, or your database is in a clean shutdown state, then you can skip this step.

From the folder that contains the transaction logs run the following command, replacing “E00″ with the log file prefix for your database.

F:\Logs\MB-HO-01>eseutil /r E00 /d G:\Data\MB-HO-01

Extensible Storage Engine Utilities for Microsoft(R) Exchange Server
Version 14.01
Copyright (C) Microsoft Corporation. All Rights Reserved.

Initiating RECOVERY mode...
    Logfile base name: E00
            Log files:
         System files:
   Database Directory: G:\Data\MB-HO-01

Performing soft recovery...
                      Restore Status (% complete)

          0    10   20   30   40   50   60   70   80   90  100
          |----|----|----|----|----|----|----|----|----|----|
          ...................................................

Operation completed successfully in 1.938 seconds.

Note the use of the /d parameter to specify the location of the mailbox database file. This is required if it not in the same path as on the original server.

Create a new mailbox database on the Mailbox server you are using to mount the database. Specify paths for the transaction logs and database file that are different to the current location.

[PS] G:\>New-MailboxDatabase -Server esp-ho-ex2010a -Name MB-HO-01-Recovered -LogFolderPath F:\Logs\MB-HO-01-Recovered -EdbFilePath G:\Data\MB-HO-01-Recovered\MB-HO-01.edb

Name                           Server          Recovery        ReplicationType
----                           ------          --------        ---------------
MB-HO-01-Recovered             ESP-HO-EX2010A  False           None

Copy the database and log files from their current locations into the locations that you specified when creating the new mailbox database.

Mark the database as overwritable.

[PS] G:\>Set-MailboxDatabase MB-HO-01-Recovered -AllowFileRestore:$true

Now mount the mailbox database.

[PS] G:\>Mount-Database MB-HO-01-Recovered

At this stage we’ve got the mailbox successfully mounted, but the users with mailboxes on that database are still pointing to the old server and database name.

[PS] G:\>Get-Mailbox -Database MB-HO-01

Name                      Alias                ServerName       ProhibitSendQuota
----                      -----                ----------       -----------------
Administrator             Administrator        esp-ho-ex2010b   unlimited
Alan.Reid                 Alan.Reid            esp-ho-ex2010b   unlimited
Alex.Heyne                Alex.Heyne           esp-ho-ex2010b   unlimited
Aisha.Bhari               Aisha.Bhari          esp-ho-ex2010b   unlimited
Aleisha.Harrison          Aleisha.Harrison     esp-ho-ex2010b   unlimited

So the final step is to update the attributes for those user accounts to point to the new mailbox database.

[PS] G:\>Get-Mailbox -Database MB-HO-01 | Set-Mailbox -Database MB-HO-01-Recovered

Confirm
Rehoming mailbox "exchangeserverpro.net/Users/Administrator" to database "MB-HO-01-Recovered". This operation will only
 modify the mailbox's Active Directory configuration. Be aware that the current mailbox content will become
inaccessible to the user.
[Y] Yes  [A] Yes to All  [N] No  [L] No to All  [?] Help (default is "Y"): a

Now we can see that the user accounts have been updated with the new attribute.

[PS] G:\>Get-Mailbox -Database MB-HO-01-Recovered

Name                      Alias                ServerName       ProhibitSendQuota
----                      -----                ----------       -----------------
Administrator             Administrator        esp-ho-ex2010a   unlimited
Alan.Reid                 Alan.Reid            esp-ho-ex2010a   unlimited
Alex.Heyne                Alex.Heyne           esp-ho-ex2010a   unlimited
Aisha.Bhari               Aisha.Bhari          esp-ho-ex2010a   unlimited
Aleisha.Harrison          Aleisha.Harrison     esp-ho-ex2010a   unlimited

The users will now be able to connect to their mailboxes again under the following assumptions:

The updated user account attributes may need to replicate through Active Directory
Outlook 2007/2010 and Outlook Web App users will automatically connect to the new database
Outlook 2003 clients will require a manual profile update to connect to the new database

Get more Exchange Server tips at ExchangeServerPro.com

Google Message Continuity Now Supports Exchange Server 2010

Paul Cunningham — Fri, 25 Feb 2011 11:29:38 +0000

In December 2010 Google launched Google Message Continuity for Exchange Server. This service provides synchronization of Exchange Server data into the cloud to allow email to continue operations if a disaster occurs.

Google Message Continuity (Source: http://www.google.com/postini/continuity.html)

At the time of launch Google Message Continuity only supported Exchange 2003 and 2007. Today Google has announced that Google Message Continuity also supports Exchange Server 2010.

At face value I consider Google Message Continuity to be a decent offering for a limited group of customers who can’t or won’t implement an Exchange-based off-site DR capability. However I am not convinced of the accuracy of the statistics that Google uses when they are promoting the merits of the service.

Recent research conducted by the Radicati Group suggests that users of Microsoft Exchange® face on average over five hours of downtime a month, which is a combination of both unplanned outages and regularly scheduled downtime.

I’m sure I’m not the only Exchange specialist who would love to meet some of these customers who are experiencing five hours of downtime a month (or more, considering that is an average) and help them out with what is clearly a serious problem.

Microsoft also shared their thoughts on Google’s claims. You can read more about that here.

Get more Exchange Server tips at ExchangeServerPro.com

Exchange Server 2010 Mailbox Server Backup and Recovery

Paul Cunningham — Mon, 03 Jan 2011 10:10:22 +0000

The Exchange Server 2010 Mailbox server role is responsible for hosting the mailbox and public folder databases in the organization. Mailbox servers can be standalone, or members of a Database Availability Group.

Of all the Exchange 2010 server roles the Mailbox server is the most important to ensure a good backup and recovery strategy is in place.

What Needs to be Backed Up on Mailbox Servers?

To create a strategy for backup and recovery of Exchange 2010 Mailbox servers you should first understand where the server stores data and configuration settings.

Active Directory – the majority of the Mailbox server configuration is stored in Active Directory.

Exchange 2010 Mailbox Server settings in Active Directory

The mailbox and public folder databases also have their configuration settings stored in Active Directory.

Exchange 2010 Mailbox Database storage quota settings in Active Directory

System State – the system state of the Mailbox server stores information such as the SSL certificates that are installed on the server (usually only a self-signed certificate on dedicated Mailbox servers), and service configuration information (eg dependencies and startup options). If there are other applications also installed on the server then they will likely have settings stored in the registry as well.

File System – though the mailbox and public folder databases are stored on the server file system they are not backed up using normal file system backups. Instead they must be backed up with an Exchange application-aware backup product that supports Volume ShadowCopy Service (VSS), for example Windows Server Backup.

Databases – the most critical data on an Exchange 2010 Mailbox server is stored in the mailbox databases and public folder databases. These consist of two basic parts; the database transaction logs, and the database file itself.

Planning the Mailbox Server Backup

When you are planning your Exchange 2010 Mailbox server backup strategy there are different techniques that you can consider depending on your requirements.

Backing up Everything

A full system and database backup of the Exchange 2010 Mailbox server, along with a working Active Directory, will have all of the required data to recover the Mailbox server. Mailbox servers can have very large volumes of data depending on the size of the organization, so full backups will take the longest and consume the most backup storage.

Backing up the Minimum

At a minimum the mailbox databases and public folder databases should be backed up so that the Exchange 2010 Mailbox server can be recovered. Depending on the backup application being used the system state and system volume may also need to be included in the backup.

Exchange 2010 database backups can also be minimized by using incremental or differential backups if the backup application supports them. However a full backup is still required on a regular basis, and the use of incremental and differential backups does increase recovery times.

Backing up Nothing

It may be practical to perform no Mailbox server backups at all if:

An Exchange 2010 Database Availability Group has been deployed with at least:
- 3 servers as members each with a copy of all databases in case of single-host failure
- 2 physical locations to protect from site failure
- 1 lagged database copy in case of application-level corruption
Single Item Recovery is enabled to meet the RPO of the organization
Appropriate Roles-Based Access Control restrictions have been implemented to prevent data loss by administrative error

If all of those conditions are true then Mailbox server backups may not be required.

Backing Up and Restoring Mailbox Servers

Here are two examples of Exchange Server 2010 Mailbox server backup and restore tasks.

Exchange 2010 Mailbox Database Backup/Restore

Windows Server Backup, which is included with Windows Server 2008 and 2008 R2, is capable of performing backups of Exchange 2010 mailbox databases. For a complete step by step guide as well as a video of the process read this article:

Exchange 2010 Mailbox Database Backup and Restore with Windows Server Backup

If the server that will run the backup is a DAG member and you are using Windows Server Backup there is a registry change required first. See Windows Server Backup Completed with Warnings for Exchange 2010 Mailbox Server.

Exchange 2010 Individual Mailbox or Mailbox Item Restore

Exchange Server 2010 also includes functionality called the Recovery Database. This is a feature that allows a mailbox database to be restored to an alternative location, mounted, and have individual mailboxes or mailbox items recovered from them. For a step by step guide to this process read this article:

How to Restore an Individual Exchange 2010 Mailbox with Windows Server Backup

Summary

The Mailbox server backups are the most important of all of the Exchange Server 2010 roles in an organization. At an absolute minimum the mailbox and public folder databases must be backed up regularly to protect from data loss. Although deploying a Database Availablility Group reduces this risk somewhat due to the replication of data between multiple servers, I would always recommend backing up the databases anyway.

Get more Exchange Server tips at ExchangeServerPro.com

Microsoft Responds to Google Message Continuity Launch

Paul Cunningham — Mon, 20 Dec 2010 20:00:17 +0000

A week after Google’s launch of Google Message Continuity Microsoft has responded on two of their company blogs.

Microsoft’s criticism of Google Message Continuity addresses three main points:

Ease of use for end users
Ease of deployment for IT administrators
Support and SLA

In a post on the Unfied Communications Group Team Blog, Julia White writes:

The most important factor is ease of use for end users. In any availability solution to such a critical application as email, the transition process should be seamless. In Exchange 2010, users won’t even know they are accessing a different database. Outlook users can be switched to a backup database when necessary without interruption or notification, and they can continue working in Outlook without switching mailboxes.

This is in contrast to how the Google service delivers email continuity.

Google takes a different approach, which requires a separate Gmail account to access backed up email. This introduces a very different user experience and doesn’t support features like Outlook folders. Additionally, Google requires a different password provided by the Exchange administrator. In the event of a failure, the user will need to switch from Outlook to a new Gmail account and log in with the new password. In looking at the three options, Google’s introduces the most user disruption.

Not just the most user disruption but possibly more IT disruption as well, as Steve Goodman points out:

Disaster strikes, and you have one more problem – a user training issue with a new email system that works in a different way.

Julia White also points out:

The second important factor is ease of deployment and management. In Exchange 2010, Microsoft has made changes in the Exchange architecture that make it much easier to manage a high availability environment, without the deep knowledge of clustering that was previously required. High availability deployment and management tools are built right into Exchange 2010 and there isn’t a need for additional availability solutions. Additionally, Exchange Online provides high availability as a built in part of the service.

Again Google takes a different approach to this with a Google Sync Server required to maintain synchronization between the two email environments, adding more complexity for IT administrators and a new point of failure. In addition to this, Google Message Continuity requires the use of Google’s Postini service, which may be a problem for customers already invested in another email security product.

The final point of concern is with support and SLAs. Here Microsoft urges customers to look closely at Google’s SLA for Google Apps.

(The Google Apps SLA is) not financially backed and allows for 10 minutes of downtime before they start counting. So, the continuity service you pay for could be down more than the Exchange Server you are backing up with the service. . . without anything you can do about it since it’s part of their “SLA”!

The level of support is also questioned. Consider that your organization is already likely to be in a disaster situation if end users are actually using Google Apps for email. In addition to the end user confusion and support that your IT team must deal with, you’re also now in a reduced support mode for a service that is critical to your business (presumably since you’ve invested in email continuity to begin with).

Take this fact: Google offers phone support on weekends and holidays only for “P1″ (Critical Impact – Service Unusable in Production) requests, and only if more than half of users are affected. Google does not respond on weekends or holidays for “P2″ (High Impact – Service Use Severely Impaired) and lower-priority requests. Try explaining to your boss that his/her issue can’t be resolved because it doesn’t rank high enough on Google’s priorities.

So what do you think? Is Google Email Continuity a viable option for businesses of any size?

Get more Exchange Server tips at ExchangeServerPro.com

Google Announces Email Continuity for Exchange Server

Paul Cunningham — Sat, 11 Dec 2010 10:37:04 +0000

Google has announced a new service, Google Message Continuity, that provides customers with a hosted email service that can be used if the on-premises email server suffers an outage.

Google Message Continuity basically replicates all incoming email that is sent through Postini, Googles email security platform, to both the on-premises Exchange server and the backup mailboxes that are hosted in Google Apps (Gmail). An ongoing synchronization process then keeps both servers up to date with internal email and other changes such as deletions.

Google Message Continuity (Source: http://www.google.com/postini/continuity.html)

Google Message Continuity is only compatible with Exchange Server 2003 and Exchange Server 2007 at the moment, which makes some sense since Exchange Server 2010 ships with a very strong continuous availability feature with Database Availability Groups. Although Exchange Server 2007 had cross-site CCR and SCR available, these were somewhat tricky to set up.

I’ve been critical in the past of the Google Apps feature set in comparison to Exchange Server, however as a failover option in a disaster the limitations of Google Apps are less important.

Priced at $25/user per annum Google Message Continuity is potentially a strong option for small to medium sized businesses who are still on Exchange 2003 or 2007, and have a need for a cloud-hosted failover option for their email. The price includes Postini for email antispam and antivirus, making it an option with consideration.

Get more Exchange Server tips at ExchangeServerPro.com

Exchange Recovery: Failed DAG Member in Exchange Server 2010

Paul Cunningham — Fri, 26 Nov 2010 09:00:58 +0000

In this tutorial I will demonstrate a recovery scenario for a failed Mailbox server that is a member of an Exchange 2010 Database Availability Group. In this scenario the DAG has two members, EX1 and EX2. EX2 has suffered a serious hardware failure and needs to be recovered.

With server EX2 down the each mailbox database in the DAG has switched over to EX1 and shows the following status information.

[PS] C:\>Get-MailboxDatabaseCopyStatus -Identity "Mailbox Database 01"

Name                                          Status          CopyQueue
                                                              Length
----                                          ------          ---------
Mailbox Database 01\EX1                       Mounted         0
Mailbox Database 01\EX2                       ServiceDown     0

The Exchange recovery process begins by reinstalling Windows Server 2008 R2 on the new server.

Installing Windows Server 2008 R2

Because this Exchange recovery is for a member of an Exchange 2010 DAG the server must be installed with the Enterprise edition of Windows Server 2008 R2.

Exchange 2010 DAG members require the Enterprise edition of Windows Server

After Windows Server 2008 R2 is finished installing log on to the server and complete the following tasks:

Configure the Timezone settings
Configure the Automatic Update settings
Configure the server with the same TCP/IP configuration as the previous server
Configure the server with the same name as the previous server (in this case EX2)
Join the server to the Active Directory domain

The next step is to install the Exchange 2010 pre-requisites for the Mailbox server role. From an elevated PowerShell prompt run the following commands.

Import-Module ServerManager

Add-WindowsFeature NET-Framework,RSAT-ADDS,Web-Server,Web-Basic-Auth,Web-Windows-Auth,Web-Metabase,Web-Net-Ext,Web-Lgcy-Mgmt-Console,WAS-Process-Model,RSAT-Web-Server -Restart

After the server has restarted we also need to install the Exchange Server 2010 SP1 hotfixes for Windows Server 2008 R2. These updates require another restart of the server.

Before installing Exchange Server 2010 on the server being recovered we first need to remove it from the DAG. On another Exchange 2010 server open the Exchange Management Shell and run the following commands.

First, determine which mailbox databases the server was hosting a copy of, the activation preferences, and any replay lag that was configured. In this example server EX2 hosted copies of Mailbox Database 01 and Mailbox Database 02.

[PS] C:\>Get-MailboxDatabase | fl name, servers, activ*, *lag*

Name                 : Mailbox Database 02
Servers              : {EX2, EX1}
ActivationPreference : {[EX2, 1], [EX1, 2]}
ReplayLagTimes       : {[EX2, 00:00:00], [EX1, 00:00:00]}
TruncationLagTimes   : {[EX2, 00:00:00], [EX1, 00:00:00]}

Name                 : Mailbox Database 01
Servers              : {EX1, EX2}
ActivationPreference : {[EX1, 1], [EX2, 2]}
ReplayLagTimes       : {[EX1, 00:00:00], [EX2, 00:00:00]}
TruncationLagTimes   : {[EX1, 00:00:00], [EX2, 00:00:00]}

Name                 : Archive Mailboxes
Servers              : {EX1}
ActivationPreference : {[EX1, 1]}
ReplayLagTimes       : {[EX1, 00:00:00]}
TruncationLagTimes   : {[EX1, 00:00:00]}

Next, remove the failed server from each of the mailbox databases that it held a copy of.

[PS] C:\>Remove-MailboxDatabaseCopy "Mailbox Database 01\EX2"

[PS] C:\>Remove-MailboxDatabaseCopy "Mailbox Database 02\EX2"

Warnings will appear because the failed Exchange server EX2 can’t be communicated with, however the change can be confirmed by repeating the earlier command.

[PS] C:\>Get-MailboxDatabase | fl name, servers, activ*, *lag*

Name                 : Mailbox Database 02
Servers              : {EX1}
ActivationPreference : {[EX1, 1]}
ReplayLagTimes       : {[EX1, 00:00:00]}
TruncationLagTimes   : {[EX1, 00:00:00]}

Name                 : Mailbox Database 01
Servers              : {EX1}
ActivationPreference : {[EX1, 1]}
ReplayLagTimes       : {[EX1, 00:00:00]}
TruncationLagTimes   : {[EX1, 00:00:00]}

Name                 : Archive Mailboxes
Servers              : {EX1}
ActivationPreference : {[EX1, 1]}
ReplayLagTimes       : {[EX1, 00:00:00]}
TruncationLagTimes   : {[EX1, 00:00:00]}

Next, remove the failed server from the Database Availability Group. Run the following command in the Exchange Management Shell.

[PS] C:\>Remove-DatabaseAvailabilityGroupServer -Identity DAG -MailboxServer EX2

Note: in some DAG topologies this action will fail with an error “A quorum of cluster nodes was not present to form a cluster”. If that error occurs use the solution in this article – Unable to Remove Failed Server from DAG Membership in Exchange Server 2010

When you are ready to proceed with the Exchange 2010 install open a command prompt and run the following command from the directory that has the Exchange setup files located within.

setup /m:recoverserver

When setup has complete and the server has been rebooted, add the recovered server back in to the Database Availability Group.

[PS] C:\>Add-DatabaseAvailabilityGroupServer -Identity DAG -MailboxServer EX2

Then, taking note of any replay or truncation lag times, and activation preferences that were earlier identified, re-add the mailbox database copies to the recovered server. This process can take a long time depending on the size of the mailbox databases that need to be reseeded.

[PS] C:\>Add-MailboxDatabaseCopy -Identity "Mailbox Database 01" -MailboxServer EX2
[PS] C:\>Add-MailboxDatabaseCopy -Identity "Mailbox Database 02" -MailboxServer EX2 -ActivationPreference 1

You can now verify that the databases have the same settings that were identified earlier.

[PS] C:\>Get-MailboxDatabase | fl name, servers, activ*, *lag*

Name                 : Mailbox Database 02
Servers              : {EX2, EX1}
ActivationPreference : {[EX2, 1], [EX1, 2]}
ReplayLagTimes       : {[EX2, 00:00:00], [EX1, 00:00:00]}
TruncationLagTimes   : {[EX2, 00:00:00], [EX1, 00:00:00]}

Name                 : Mailbox Database 01
Servers              : {EX1, EX2}
ActivationPreference : {[EX1, 1], [EX2, 2]}
ReplayLagTimes       : {[EX1, 00:00:00], [EX2, 00:00:00]}
TruncationLagTimes   : {[EX1, 00:00:00], [EX2, 00:00:00]}

Name                 : Archive Mailboxes
Servers              : {EX1}
ActivationPreference : {[EX1, 1]}
ReplayLagTimes       : {[EX1, 00:00:00]}
TruncationLagTimes   : {[EX1, 00:00:00]}

The failed DAG member has now been recovered and the Exchange 2010 Database Availability Group is back to normal operation.

Get more Exchange Server tips at ExchangeServerPro.com

Unable to Remove Failed Server from DAG in Exchange Server 2010

Paul Cunningham — Thu, 25 Nov 2010 09:00:17 +0000

When an Exchange Server 2010 Mailbox server that is a member of a Database Availability Group has failed, part of the recovery process is to remove it from DAG membership.

In some scenarios this process may result in an error “A quorum of cluster nodes was not present to form a cluster“. The full error text is below.

[PS] C:\>Remove-DatabaseAvailabilityGroupServer -Identity DAG -MailboxServer EX2

Confirm
Are you sure you want to perform this action?
Removing Mailbox server "EX2" from database availability group "dag".
[Y] Yes  [A] Yes to All  [N] No  [L] No to All  [?] Help (default is "Y"): y
WARNING: The operation wasn't successful because an error was encountered. You may find more details in log file
"C:\ExchangeSetupLogs\DagTasks\dagtask_2010-11-25_03-48-09.814_remove-databaseavailabiltygroupserver.log".
There was a problem changing the quorum model for database availability group dag. Error: An Active Manager operation f
ailed. Error: An error occurred while attempting a cluster operation. Error: Cluster API '"SetClusterQuorumResource() f
ailed with 0x1725. Error: A quorum of cluster nodes was not present to form a cluster"' failed..
    + CategoryInfo          : InvalidArgument: (:) [Remove-DatabaseAvailabilityGroupServer], DagTaskProblemChangingQuo
   rumException
    + FullyQualifiedErrorId : 80D96894,Microsoft.Exchange.Management.SystemConfigurationTasks.RemoveDatabaseAvailabili
   tyGroupServer

To resolve this issue use the -ConfigurationOnly switch instead to remove the failed Mailbox server from the Exchange 2010 DAG. In this example server EX2 is being removed.

[PS] C:\>Remove-DatabaseAvailabilityGroupServer -Identity DAG -MailboxServer EX2 -ConfigurationOnly

Next, evict the failed node from the Windows Failover Cluster.

Note: you need to import the Failover Cluster module into your Exchange Management Shell session to perform this task.

[PS] C:\>Import-Module FailoverClusters
[PS] C:\>Get-ClusterNode EX2 | Remove-ClusterNode -Force

The failed server has now been removed from the Exchange 2010 DAG and the Windows Failover Cluster.

Get more Exchange Server tips at ExchangeServerPro.com

Exchange Server Pro » Disaster Recovery

Google Discontinues Message Continuity

Related posts:

Exchange 2010: Are Passive Database Copies Included in Backups of DAG Members?

Related posts:

Review: Exchange Server 2010 Backup and Recovery Training

Exchange Server Backups – Serious Business

Exchange Server 2010 Backup and Recovery Training

What You Get with TrainSignal’s Exchange Server 2010 Backup and Recovery Training

Related posts:

Exchange Server 2010: Using Database Portability for Disaster Recovery

Related posts:

Google Message Continuity Now Supports Exchange Server 2010

Related posts:

Exchange Server 2010 Mailbox Server Backup and Recovery

What Needs to be Backed Up on Mailbox Servers?

Planning the Mailbox Server Backup

Backing up Everything

Backing up the Minimum

Backing up Nothing

Backing Up and Restoring Mailbox Servers

Exchange 2010 Mailbox Database Backup/Restore

Exchange 2010 Individual Mailbox or Mailbox Item Restore

Summary

Related posts:

Microsoft Responds to Google Message Continuity Launch

Related posts:

Google Announces Email Continuity for Exchange Server

Related posts:

Exchange Recovery: Failed DAG Member in Exchange Server 2010

Related posts:

Unable to Remove Failed Server from DAG in Exchange Server 2010

Related posts: