Exchange Server Pro » High Availability

How to Configure a File Share Witness for an Exchange 2010 Database Availability Group

Paul Cunningham — Mon, 20 Feb 2012 11:19:11 +0000

In my Exchange Server 2010 lab environment I unwittingly created a problem for the Database Availability Group. In preparing to consolidate all of the server roles onto just two servers and implement a hardware load balancer I went ahead and decommissioned the two CAS/HT servers that previously made up the CAS array in the site.

Naturally one of those CAS/HT servers also happened to be the File Share Witness for my two-member DAG. Whoops!

Now my DAG displays a warning when I check the health of it.

WARNING: Database availability group ‘dag-headoffice’ witness is in a failed state. The database availability group requires the witness server to maintain quorum. Please use the Set-DatabaseAvailabilityGroup cmdlet to re-create the witness server and directory.

In this real world this situation may also arise if the server hosting the File Share Witness was being decommissioned, or if it had failed. Fortunately we can resolve the problem by specifying a new FSW for the DAGm which I will demonstrate here.

I’m going to use another member server within the site as my FSW, which allows me to demonstrate a related problem. The server is named HO-MGT so using the Set-DatabaseAvailabilityGroup cmdlet to configure the FSW would mean I run this command.

[PS] C:\>Set-DatabaseAvailabilityGroup dag-headoffice -WitnessServer ho-mgt -WitnessDirectory C:\DAGFSW

However in this case I get an error.

WARNING: The Exchange Trusted Subsystem is not a member of the local Administrators group on specified witness server
ho-mgt.
WARNING: Insufficient permissions to access file shares on witness server ‘HO-MGT.exchangeserverpro.net’. Until this problem is corrected, the database availability group may be more vulnerable to failures. You can use the Set-DatabaseAvailabilityGroup cmdlet to try the operation again. Error: Access is denied
Unable to change the quorum for database availability group dag-headoffice. Witness server ‘\\HO-MGT.exchangeserverpro.net\dag-headoffice.exchangeserverpro.net’ network name wasn’t found. This may be due to firewall settings.
+ CategoryInfo : InvalidArgument: (:) [Set-DatabaseAvailabilityGroup], DagTaskProblemC…ptionBadNetName
+ FullyQualifiedErrorId : 75321C4E,Microsoft.Exchange.Management.SystemConfigurationTasks.SetDatabaseAvailabilityGroup

If you were running the same command but specifying another Exchange 2010 server to be the FSW you would not receive that error. This is because Exchange servers trust each other to perform this type of administration, thanks to a group called Exchange Trusted Subsystem.

All of the Exchange 2010 servers have this group as a member of their local Administrators group, for example here the local Administrators group of one of my DAG members.

Exchange Trusted Subsystem group in local Administrators

So the solution is to add the Exchange Trusted Subsystem group to the local Administrators group on my HO-MGT server, and then run the Set-DatabaseAvailabilityGroup command again.

After running the command you can see that Exchange has created the folder and shared it on the FSW server, no need to manually create the folder or set any permissions yourself.

Database Availability Group File Share Witness directory

Now when checking the health of the Database Availability Group you should not receive any warnings about missing File Share Witness servers.

Get more Exchange Server tips at ExchangeServerPro.com

TechEd Video: Exchange Server 2010 High Availability Deep Dive

Paul Cunningham — Sun, 25 Sep 2011 16:30:52 +0000

In this video from TechEd Australia 2011 Microsoft’s Scott Schnoll takes us under the hood of Exchange 2010 Database Availability Groups. The best parts are the deep coverage of Active Manager, best copy selection, and the often misunderstood Database Activation Coordination (DAC) mode.

Get more Exchange Server tips at ExchangeServerPro.com

TechEd Video: Exchange Server 2010 High Availability Concepts

Paul Cunningham — Sun, 25 Sep 2011 15:30:16 +0000

This video from TechEd Australia 2011 is Scott Schnoll’s presentation on Exchange Server 2010 high availability concepts. It is entirely slides and speech, no demonstration, but listen closely because Scott takes defines all of the most critical HA concepts for Exchange 2010 in a way that is highly informative yet very easily understood.

Get more Exchange Server tips at ExchangeServerPro.com

How to Move the File Share Witness for Exchange 2007 CCR Clusters

Paul Cunningham — Fri, 29 Jul 2011 14:21:22 +0000

An Exchange Server 2007 CCR cluster uses a cluster quorum configuration called Node and File Share Majority.

This means that the cluster has two servers that are the cluster nodes themselves, and a third server that is the File Share Witness server. The role of the File Share Witness server is to allow a majority of nodes to “vote” on which cluster node should be the active node.

The File Share Witness server is simply a server configured with a file share that the cluster service has access to. During failure scenarios each cluster node (that is still online) attempts to get an SMB lock on the file share to see if it can secure the majority of “votes” in the cluster.

The normal practice is to locate the File Share Witness for an Exchange Server 2007 CCR cluster on a Hub Transport server in the same site or data center as the cluster nodes, or the same data center of at least one of the nodes in a geo-cluster model. It is not mandatory to place it on a Hub Transport server though.

In some situations you may need to move this File Share Witness role to another server. This is a fairly simple process for Exchange Server 2007 CCR clusters running on Windows Server 2008.

In this demonstration I will move the File Share Witness from server HO-EX2007-HT1 to HO-EX2007-CA1 for the clustered mailbox server HO-EX2007-MB1.

Creating the File Share

The first step is to create the file share on the new File Share Witness server. Create a new folder named _FSW.

Open the Properties of the folder and navigate to the Sharing tab, then click Advanced Sharing.

Tick the box to share the folder, and then click Permissions.

Remove the default share permissions, and then add the computer account for the cluster with Full Control permissions.

Click OK, and OK to apply the share permissions, and then click on the Security tab of the folder properties.

Edit the permissions to grant the computer account for the cluster Full Control of the folder.

Click OK and apply all the changes you’ve just made.

Change the File Share Witness for the Failover Cluster

Log on to one of the cluster nodes and open Failover Cluster Management from the Administrative Tools. Right-click the cluster name and choose More Actions -> Configure Cluster Quorum Settings.

Click Next, then select Node and File Share Majority, and click Next to continue.

Enter the UNC path to the new file share that you created earlier, click Next, and then click Next again to configure the cluster with the new File Share Witness.

Click Finish when the operation has completed.

The File Share Witness role has now been moved to the new server.

Testing the New File Share Witness

There are a few ways you can test the new cluster configuration.

You can use the Manage Clustered Mailbox Server wizard to test the failover of the cluster between the two nodes.

You can also create unplanned failover scenarios, although if you do decide to do this I recommend making sure you do it outside of business hours, after a successful backup, and ensure the storage groups are in a healthy replication state with no copy or replay queues.

You can use Failover Cluster Management to stop the cluster service on the active node, which will cause a failover. Make sure you start it again when you’re done with the test.

Or you can simply pull out the network cable from the active node, which will have the same effect.

In these unplanned failover scenarios when one node is unavailable the File Share Witness is used to form a majority so that the other cluster node can bring the cluster resources online. You can verify that the FSW arbitration took place by generating the cluster log and opening it in Notepad.

C:\>cluster log /g
Generating the cluster log(s) ...
The cluster log has been successfully generated on node 'ho-ex2007-node2'...
The cluster log has been successfully generated on node 'ho-ex2007-node1'...

Look at the cluster.log file in C:\Windows\System32\Cluster\Reports and you should see entries for the FSW arbitration.

File Share Witness : Beginning arbitration ...
File Share Witness : Opening file \\ho-ex2007-ca1\ho-ex2007-mb1_fsw\82c44535-d765-4fbe-8ad8-0de1c04a87a6\Witness.log.
File Share Witness : Locking file \\ho-ex2007-ca1\ho-ex2007-mb1_fsw\82c44535-d765-4fbe-8ad8-0de1c04a87a6\Witness.log.

After your testing is complete just be sure to reconnect any network cables, start cluster services if you stopped them, and move the cluster group back to the preferred active node.

Get more Exchange Server tips at ExchangeServerPro.com

How to Move Storage Groups and Databases in an Exchange 2007 CCR Cluster

Paul Cunningham — Tue, 14 Jun 2011 13:19:00 +0000

Exchange Server 2007 makes it fairly easy to move storage group and database paths using a few simple commands. Except if you’re running an Exchange 2007 CCR cluster.

For CCR clusters the technique used for standalone Exchange 2007 mailbox servers has to be modified slightly to account for the fact that the CCR cluster can’t move the transaction logs and mailbox database files itself. Instead the administrator must move the files manually, and issue commands to update the storage group and database configurations to the new paths.

So if the process is different (some would say it is harder) why not take another approach?

For example, why not just delete the storage group and database you don’t want, and create new ones in the correct paths? This might be feasible for the default storage group and database created when the cluster was first installed, but once you have added mailboxes it is no longer an option.

In that case, why not just create the new storage group and database and migrate the mailboxes instead? That might work fine in some cases (certainly that’s how I’ve done it sometimes in the past), but if there is a lot of mailboxes you can run into problems managing the amount of transaction logging that the mailbox moves create. It can also take a lot longer than simply moving the storage group and database.

Preparing to Move a Storage Group in an Exchange 2007 CCR Cluster

Before you move a storage group in an CCR cluster I recommend that you first back up the database in that storage group. By running a full backup you:

Truncate the transaction logs, which reduces the amount of data that needs to be moved
Give yourself a very recent back up to restore from if something goes wrong

Next check the copy and replay queues to make sure they are up to date.

[PS] C:\>Get-StorageGroupCopyStatus

Name                      SummaryCopySt CopyQueueLeng ReplayQueueL
                          atus          th            ength
----                      ------------- ------------- ------------
First Storage Group       Healthy       0             0

You should also make sure the new storage paths are properly configured and ready to be used in production.

Moving a Storage Group and Mailbox Database in an Exchange 2007 CCR Cluster

The first step is to suspend replication for the storage group.

[PS] C:\>Suspend-StorageGroupCopy "First Storage Group"

Confirm
Are you sure you want to perform this action?
Suspending Storage Group Copy "First Storage Group".
[Y] Yes  [A] Yes to All  [N] No  [L] No to All  [S] Suspend  [?] Help

Next, dismount the mailbox database.

[PS] C:\>Dismount-Database "First Storage Group\Mailbox Database"

Confirm
Are you sure you want to perform this action?
Dismounting database "First Storage Group\Mailbox Database".
[Y] Yes  [A] Yes to All  [N] No  [L] No to All  [S] Suspend  [?] Help
(default is "Y"):y

With replication suspended and the database dismounted you can now move the files by simply copying them in Windows Explorer to the new locations. You must do this step on both cluster nodes, and make sure that the path matches on both nodes.

After copying the files the next step is to update the configuration of the storage group with the new folder path. Note the use of the -ConfigurationOnly switch on this command.

[PS] C:\>Move-StorageGroupPath "First Storage Group" -SystemFolderPath E:\SG1 -LogFolderPath E:\SG1 -ConfigurationOnly

Confirm
Are you sure you want to perform this action?
Moving path of storage group "HO-EX2007-MB1.exchangeserverpro.net\First Storage Group".
[Y] Yes  [A] Yes to All  [N] No  [L] No to All  [S] Suspend  [?] Help
(default is "Y"):y

Now the configuration of the mailbox database is also updated with the new folder path. Again note the use of the -ConfigurationOnly switch.

[PS] C:\>Move-DatabasePath "First Storage Group\Mailbox Database" -EdbFilePath '
F:\DB1\Mailbox Database.edb' -ConfigurationOnly

Confirm
This operation will skip the safety check and make the change to the Active
Directory directory service directly. Do you want to continue?
[Y] Yes  [A] Yes to All  [N] No  [L] No to All  [S] Suspend  [?] Help
(default is "Y"):y

Confirm
Are you sure you want to perform this action?
Moving database path "First Storage Group\Mailbox Database".
[Y] Yes  [A] Yes to All  [N] No  [L] No to All  [S] Suspend  [?] Help
(default is "Y"):y

Now we can mount the database again.

[PS] C:\>Mount-Database "First Storage Group\Mailbox Database"

Finally, resume replication for the storage group.

[PS] C:\>Resume-StorageGroupCopy "First Storage Group"

The storage group and mailbox database have now been moved to the new locations and are back online. You can verify the health of the storage group by running Get-StorageGroupCopyStatus.

[PS] C:\>Get-StorageGroupCopyStatus

Name                      SummaryCopySt CopyQueueLeng ReplayQueueL
                          atus          th            ength
----                      ------------- ------------- ------------
First Storage Group       Healthy       0             0

You can also use Test-ReplicationHealth to test the CCR cluster.

[PS] C:\>Test-ReplicationHealth

Server          Check                      Result     Error
------          -----                      ------     -----
HO-EX2007-NODE1 PassiveNodeUp              Passed
HO-EX2007-NODE1 ClusterNetwork             Passed
HO-EX2007-NODE1 QuorumGroup                Passed
HO-EX2007-NODE1 FileShareQuorum            Passed
HO-EX2007-NODE1 CmsGroup                   Passed
HO-EX2007-NODE1 NodePaused                 Passed
HO-EX2007-NODE1 DnsRegistrationStatus      Passed
HO-EX2007-NODE1 ReplayService              Passed
HO-EX2007-NODE1 DBMountedFailover          Passed
HO-EX2007-NODE1 SGCopySuspended            Passed
HO-EX2007-NODE1 SGCopyFailed               Passed
HO-EX2007-NODE1 SGInitializing             Passed
HO-EX2007-NODE1 SGCopyQueueLength          Passed
HO-EX2007-NODE1 SGReplayQueueLength        Passed

As a final test you can also verify that MAPI connectivity is working.

[PS] C:\>Test-MAPIConnectivity

MailboxServer      Database           Result    Latency(MS) Error
-------------      --------           ------    ----------- -----
ho-ex2007-mb1      Mailbox Database   Success             8

Get more Exchange Server tips at ExchangeServerPro.com

Poll: Should CAS Arrays be Deployed in all Exchange 2010 Sites?

Paul Cunningham — Fri, 20 May 2011 01:00:45 +0000

Exchange 2010 CAS Arrays are a great high availability feature, but are organizations making the best use of them?

One of the best practices when deploying Client Access Server Arrays is to establish the CAS Array first before configuring mailbox databases in the same site. Or at the very least, making sure you update each existing mailbox database with the correct “RPCClientAccessServer” attribute before migrating users to them.

The advantage of this is that Outlook profiles get automatically configured with the name of the CAS Array rather than the real name of a Client Access server in that site.

Considering the benefits of this, such as:

making it easy to scale out the CAS Array name to multiple Exchange 2010 Client Access servers
making it simpler to replace a Client Access server with a new one of a different name
migrating the MAPI endpoint to future versions of Exchange Server

…does it make it a sensible best practice to always configure CAS Arrays for AD Sites that contain Exchange 2010 mailbox servers, even when only one Client Access server is deployed?

Update: Microsoft now recommends that you always create a CAS Array in each site, even if you have only one Client Access server in the site.

Please give your opinion by voting in this poll. If you wish to expand on your vote in more detail please leave a comment below.

Note: There is a poll embedded within this post, please visit the site to participate in this post's poll.

Get more Exchange Server tips at ExchangeServerPro.com

What Are The Benefits of Exchange Server 2010 Compared to Exchange 2007

Paul Cunningham — Thu, 31 Mar 2011 10:29:31 +0000

Some businesses who are currently running Exchange Server 2007 wonder whether there are any real benefits to upgrading to Exchange Server 2010. Here are a few key benefits of Exchange Server 2010 compared to Exchange 2007.

Lower Storage Costs

The Exchange Server 2010 database engine is up to 70% more efficient than Exchange Server 2007 in terms of disk read/write performance. This makes it possible to deploy much larger databases, with much larger mailbox quotas, on much larger and less expensive disks.

This improvement came at the cost of single instance storage no longer being a feature of Exchange Server 2010, however the performance gains are well worth it.

Simpler High Availability

Exchange Server 2007 had four different high availability options for the Mailbox server role – Local Continuous Replication, Single Copy Cluster, Cluster Continuous Replication, and Standby Continuous Replication.

Exchange Server 2010 has one high availability option for Mailbox servers, the Database Availability Group. Instead of being an active/passive pair of cluster nodes as with Exchange 2007, a DAG can instead consist of up to 16 Mailbox servers each holding a replica of the mailbox databases. Fail over is also possible at a database-level instead requiring the whole server to fail over.

High availability is also provided at the Client Access server with the new Exchange 2010 CAS Array, which is important because the Client Access server in Exchange 2010 now accepts all MAPI connections from Outlook clients.

Click here for a demonstration video of Exchange 2010 high availability in action.

Some further benefits of Exchange Server 2010 include the following.

Mail Tips

This useful feature of Exchange 2010 when used with Outlook 2010 will warn clients of such email blunders as doing a “reply all” to a very large group, or sending sensitive emails to external parties.

Moderated Transport

Moderated Transport allows distribution lists and mailboxes to be moderated by a non-admin user such as a manager who has to approve messages before they are delivered.

On- and Off-Premises

It is possible to split your Exchange organization between on-premises servers and cloud hosted servers, ideal for customers such as schools who wish to keep staff on-premises but have students in cheaper cloud hosting.

Get more Exchange Server tips at ExchangeServerPro.com

Google Message Continuity Now Supports Exchange Server 2010

Paul Cunningham — Fri, 25 Feb 2011 11:29:38 +0000

In December 2010 Google launched Google Message Continuity for Exchange Server. This service provides synchronization of Exchange Server data into the cloud to allow email to continue operations if a disaster occurs.

Google Message Continuity (Source: http://www.google.com/postini/continuity.html)

At the time of launch Google Message Continuity only supported Exchange 2003 and 2007. Today Google has announced that Google Message Continuity also supports Exchange Server 2010.

At face value I consider Google Message Continuity to be a decent offering for a limited group of customers who can’t or won’t implement an Exchange-based off-site DR capability. However I am not convinced of the accuracy of the statistics that Google uses when they are promoting the merits of the service.

Recent research conducted by the Radicati Group suggests that users of Microsoft Exchange® face on average over five hours of downtime a month, which is a combination of both unplanned outages and regularly scheduled downtime.

I’m sure I’m not the only Exchange specialist who would love to meet some of these customers who are experiencing five hours of downtime a month (or more, considering that is an average) and help them out with what is clearly a serious problem.

Microsoft also shared their thoughts on Google’s claims. You can read more about that here.

Get more Exchange Server tips at ExchangeServerPro.com

Exchange 2010 FAQ: Which Version of Exchange Server 2010 Do I Need?

Paul Cunningham — Fri, 28 Jan 2011 16:37:07 +0000

Question: Do I need to purchase the Standard Edition or the Enterprise Edition of Exchange Server 2010 for my organization?

Exchange Server 2010 is available in two editions; Standard or Enterprise. The only difference between these two product editions is how many databases (of either mailbox or public folder type) it can host.

Exchange Server 2010 Standard Edition can host up to 5 databases
Exchange Server 2010 Enterprise Edition can host up to 100 databases

This makes selecting Exchange Server 2010 licenses fairly simple.

Use Exchange Server 2010 Standard Edition for all Client Access, Hub Transport, Edge Transport, and Unified Messaging servers
Use Exchange Server 2010 Standard Edition for Mailbox servers that require 5 or fewer databases
Use Exchange Server 2010 Enterprise Edition for Mailbox servers that require more than 5 databases

For Mailbox servers that are members of a Database Availability Group each passive copy of a database counts as a database on the server. So if a Mailbox server has 4 active mailbox databases and another 4 passive copies it will need to be running the Enterprise Edition of Exchange Server 2010.

However, Exchange 2010 Enterprise Edition is not required for DAG members that host 5 or fewer active and passive databases. A common misconception is that Enterprise Edition is required for Exchange high availability. This is not true, DAG members can run either Standard or Enterprise edition (even within the same Database Availability Group).

However all DAG members must run Enterprise Edition of Windows Server 2008 or 2008 R2, which is required for the underlying Failover Cluster that the DAG utilizes.

The edition of Exchange Server 2010 is solely determined by the product key that is entered after the server is installed. When the server is running in 120 day trial mode (ie without a product key) it operates under the limits of Standard Edition. A server can be upgraded to Enterprise Edition simply by entering the new product key.

You can read more about Exchange Server 2010 licensing on the Microsoft website:

Get more Exchange Server tips at ExchangeServerPro.com

Microsoft Responds to Google Message Continuity Launch

Paul Cunningham — Mon, 20 Dec 2010 20:00:17 +0000

A week after Google’s launch of Google Message Continuity Microsoft has responded on two of their company blogs.

Microsoft’s criticism of Google Message Continuity addresses three main points:

Ease of use for end users
Ease of deployment for IT administrators
Support and SLA

In a post on the Unfied Communications Group Team Blog, Julia White writes:

The most important factor is ease of use for end users. In any availability solution to such a critical application as email, the transition process should be seamless. In Exchange 2010, users won’t even know they are accessing a different database. Outlook users can be switched to a backup database when necessary without interruption or notification, and they can continue working in Outlook without switching mailboxes.

This is in contrast to how the Google service delivers email continuity.

Google takes a different approach, which requires a separate Gmail account to access backed up email. This introduces a very different user experience and doesn’t support features like Outlook folders. Additionally, Google requires a different password provided by the Exchange administrator. In the event of a failure, the user will need to switch from Outlook to a new Gmail account and log in with the new password. In looking at the three options, Google’s introduces the most user disruption.

Not just the most user disruption but possibly more IT disruption as well, as Steve Goodman points out:

Disaster strikes, and you have one more problem – a user training issue with a new email system that works in a different way.

Julia White also points out:

The second important factor is ease of deployment and management. In Exchange 2010, Microsoft has made changes in the Exchange architecture that make it much easier to manage a high availability environment, without the deep knowledge of clustering that was previously required. High availability deployment and management tools are built right into Exchange 2010 and there isn’t a need for additional availability solutions. Additionally, Exchange Online provides high availability as a built in part of the service.

Again Google takes a different approach to this with a Google Sync Server required to maintain synchronization between the two email environments, adding more complexity for IT administrators and a new point of failure. In addition to this, Google Message Continuity requires the use of Google’s Postini service, which may be a problem for customers already invested in another email security product.

The final point of concern is with support and SLAs. Here Microsoft urges customers to look closely at Google’s SLA for Google Apps.

(The Google Apps SLA is) not financially backed and allows for 10 minutes of downtime before they start counting. So, the continuity service you pay for could be down more than the Exchange Server you are backing up with the service. . . without anything you can do about it since it’s part of their “SLA”!

The level of support is also questioned. Consider that your organization is already likely to be in a disaster situation if end users are actually using Google Apps for email. In addition to the end user confusion and support that your IT team must deal with, you’re also now in a reduced support mode for a service that is critical to your business (presumably since you’ve invested in email continuity to begin with).

Take this fact: Google offers phone support on weekends and holidays only for “P1″ (Critical Impact – Service Unusable in Production) requests, and only if more than half of users are affected. Google does not respond on weekends or holidays for “P2″ (High Impact – Service Use Severely Impaired) and lower-priority requests. Try explaining to your boss that his/her issue can’t be resolved because it doesn’t rank high enough on Google’s priorities.

So what do you think? Is Google Email Continuity a viable option for businesses of any size?

Get more Exchange Server tips at ExchangeServerPro.com

Exchange Server Pro » High Availability

How to Configure a File Share Witness for an Exchange 2010 Database Availability Group

Related posts:

TechEd Video: Exchange Server 2010 High Availability Deep Dive

Related posts:

TechEd Video: Exchange Server 2010 High Availability Concepts

Related posts:

How to Move the File Share Witness for Exchange 2007 CCR Clusters

Creating the File Share

Change the File Share Witness for the Failover Cluster

Testing the New File Share Witness

Related posts:

How to Move Storage Groups and Databases in an Exchange 2007 CCR Cluster

Preparing to Move a Storage Group in an Exchange 2007 CCR Cluster

Moving a Storage Group and Mailbox Database in an Exchange 2007 CCR Cluster

Related posts:

Poll: Should CAS Arrays be Deployed in all Exchange 2010 Sites?

Related posts:

What Are The Benefits of Exchange Server 2010 Compared to Exchange 2007

Lower Storage Costs

Simpler High Availability

Mail Tips

Moderated Transport

On- and Off-Premises

Related posts:

Google Message Continuity Now Supports Exchange Server 2010

Related posts:

Exchange 2010 FAQ: Which Version of Exchange Server 2010 Do I Need?

Related posts:

Microsoft Responds to Google Message Continuity Launch

Related posts: