In my Exchange Server 2010 lab environment I unwittingly created a problem for the Database Availability Group. In preparing to consolidate all of the server roles onto just two servers and implement a hardware load balancer I went ahead and decommissioned the two CAS/HT servers that previously made up the CAS array in the site.
Naturally one of those CAS/HT servers also happened to be the File Share Witness for my two-member DAG. Whoops!
Now my DAG displays a warning when I check the health of it.
WARNING: Database availability group ‘dag-headoffice’ witness is in a failed state. The database availability group requires the witness server to maintain quorum. Please use the Set-DatabaseAvailabilityGroup cmdlet to re-create the witness server and directory.
In this real world this situation may also arise if the server hosting the File Share Witness was being decommissioned, or if it had failed. Fortunately we can resolve the problem by specifying a new FSW for the DAGm which I will demonstrate here.
I’m going to use another member server within the site as my FSW, which allows me to demonstrate a related problem. The server is named HO-MGT so using the Set-DatabaseAvailabilityGroup cmdlet to configure the FSW would mean I run this command.
[PS] C:\>Set-DatabaseAvailabilityGroup dag-headoffice -WitnessServer ho-mgt -WitnessDirectory C:\DAGFSW
However in this case I get an error.
WARNING: The Exchange Trusted Subsystem is not a member of the local Administrators group on specified witness server
WARNING: Insufficient permissions to access file shares on witness server ‘HO-MGT.exchangeserverpro.net’. Until this problem is corrected, the database availability group may be more vulnerable to failures. You can use the Set-DatabaseAvailabilityGroup cmdlet to try the operation again. Error: Access is denied
Unable to change the quorum for database availability group dag-headoffice. Witness server ‘\\HO-MGT.exchangeserverpro.net\dag-headoffice.exchangeserverpro.net’ network name wasn’t found. This may be due to firewall settings.
+ CategoryInfo : InvalidArgument: (:) [Set-DatabaseAvailabilityGroup], DagTaskProblemC…ptionBadNetName
+ FullyQualifiedErrorId : 75321C4E,Microsoft.Exchange.Management.SystemConfigurationTasks.SetDatabaseAvailabilityGroup
If you were running the same command but specifying another Exchange 2010 server to be the FSW you would not receive that error. This is because Exchange servers trust each other to perform this type of administration, thanks to a group called Exchange Trusted Subsystem.
All of the Exchange 2010 servers have this group as a member of their local Administrators group, for example here the local Administrators group of one of my DAG members.
So the solution is to add the Exchange Trusted Subsystem group to the local Administrators group on my HO-MGT server, and then run the Set-DatabaseAvailabilityGroup command again.
After running the command you can see that Exchange has created the folder and shared it on the FSW server, no need to manually create the folder or set any permissions yourself.
Now when checking the health of the Database Availability Group you should not receive any warnings about missing File Share Witness servers.