In my Exchange Server 2010 lab environment I unwittingly created a problem for the Database Availability Group. In preparing to consolidate all of the server roles onto just two servers and implement a hardware load balancer I went ahead and decommissioned the two CAS/HT servers that previously made up the CAS array in the site.
Naturally one of those CAS/HT servers also happened to be the File Share Witness for my two-member DAG. Whoops!
Now my DAG displays a warning when I check the health of it.
WARNING: Database availability group ‘dag-headoffice’ witness is in a failed state. The database availability group requires the witness server to maintain quorum. Please use the Set-DatabaseAvailabilityGroup cmdlet to re-create the witness server and directory.
In this real world this situation may also arise if the server hosting the File Share Witness was being decommissioned, or if it had failed. Fortunately we can resolve the problem by specifying a new FSW for the DAGm which I will demonstrate here.
I’m going to use another member server within the site as my FSW, which allows me to demonstrate a related problem. The server is named HO-MGT so using the Set-DatabaseAvailabilityGroup cmdlet to configure the FSW would mean I run this command.
[PS] C:\>Set-DatabaseAvailabilityGroup dag-headoffice -WitnessServer ho-mgt -WitnessDirectory C:\DAGFSW
However in this case I get an error.
WARNING: The Exchange Trusted Subsystem is not a member of the local Administrators group on specified witness server
ho-mgt.
WARNING: Insufficient permissions to access file shares on witness server ‘HO-MGT.exchangeserverpro.net’. Until this problem is corrected, the database availability group may be more vulnerable to failures. You can use the Set-DatabaseAvailabilityGroup cmdlet to try the operation again. Error: Access is denied
Unable to change the quorum for database availability group dag-headoffice. Witness server ‘\\HO-MGT.exchangeserverpro.net\dag-headoffice.exchangeserverpro.net’ network name wasn’t found. This may be due to firewall settings.
+ CategoryInfo : InvalidArgument: (:) [Set-DatabaseAvailabilityGroup], DagTaskProblemC…ptionBadNetName
+ FullyQualifiedErrorId : 75321C4E,Microsoft.Exchange.Management.SystemConfigurationTasks.SetDatabaseAvailabilityGroup
If you were running the same command but specifying another Exchange 2010 server to be the FSW you would not receive that error. This is because Exchange servers trust each other to perform this type of administration, thanks to a group called Exchange Trusted Subsystem.
All of the Exchange 2010 servers have this group as a member of their local Administrators group, for example here the local Administrators group of one of my DAG members.
Exchange Trusted Subsystem group in local Administrators
So the solution is to add the Exchange Trusted Subsystem group to the local Administrators group on my HO-MGT server, and then run the Set-DatabaseAvailabilityGroup command again.
After running the command you can see that Exchange has created the folder and shared it on the FSW server, no need to manually create the folder or set any permissions yourself.
Database Availability Group File Share Witness directory
Now when checking the health of the Database Availability Group you should not receive any warnings about missing File Share Witness servers.
About Paul Cunningham
Paul,
We have 2 CAS/HTS servers in the same AD Site. When my primary CAS (which also hosts the FSW) goes offline, my outlook clients get prompted to provide credentials and they won’t automatically connect to the secondary CAS until i manually update DNS records. Even then they have to provide credentials to log on to the secondary CAS server.
Do i possibly need to move the FSW role to another member server?
Thanks!
The CAS role (and the RPCClientAccessService) has nothing to do with the File Share Witness for a DAG. They may happen to be on the same server, but they are completely different things.
Your issue is that the RPCClientAccessServer that Outlook users are connecting to is going offline. You would solve that problem with a CAS Array and load balancing solution.
Unfortunately 4 me it doesn’t work. In producton i had to move cluster witness on another server, where I already put exchange subsystem in local admins. The share is created. Inside are two files with 0 kb. The share witness doeasn’t work. In GUI i see success, but in powershell there is an error that exchange subsystem is not local admin (but aparently it is). Just as is stated here:
http://blogs.technet.com/b/scottschnoll/archive/2011/06/08/witness-server-warning-message-when-using-certain-database-availability-group-tasks.aspx
Is this a bug or smth I am missing?
Scott explains in that article that the error is a bug in Exchange and can be disregarded if you have put the ETS group into the local admins group on a non-Exchange FSW.
I have some serious man-love for you. Nobody writes a better guide.
When I read you work, I know what I am doing it and more importantly, why.
Keep up the great work and thanks for all the great articles.
Hi…
Environment:
2 Servers installed with MAILBOX+HUB TRANSPORT+CAS Roles in AD DOMAIN on physical servers
2 Servers installed with EDGE TRANSPORT Roles in a WORKGROUP on Hypervisors
I want to make a DAG for MAILBOX databases for redundancy. Can I make EDGE TRANSPORT Server as a WITNESS SERVER. Please help with that
No your Edge Transports cannot be file share witnesses for the DAG.
Yeah… tried that but didn’t work… I guess I will have to build a HUB+CAS server and that can be made a WITNESS Server…right…?
It doesn’t need to be an Exchange server, though it is generally recommended. But you can also use other non-Exchange servers as long as you first add the Exchange Trusted Subsystem group to the local administrators group on the server.
This is the Exchange 2013/Windows Server 2012 version of how to do it, basically the same for Exchange 2010/Server 2008 but the screenshots will look different of course.
http://exchangeserverpro.com/using-a-non-exchange-server-as-an-exchange-2013-dag-file-share-witness/
OK Great !!! I have one more server for McAfee Antivirus Server running on hypervisor in AD Domain. Shall I use that one as a Witness Server ?
I don’t see a problem with that, as long as the server is reliable and not constantly overloaded already. FSW doesn’t add any significant load to a server but the FSW should generally be reliable.
Thanks a million Mr. Cunningham… Appreciate your help.
Paul,
Another question….
What brand of hardware load balancer have you had good luck with and why? We may be going down the path of purchasing a hardware load balancer for our CAS array. I know Microsoft offers an article on supported hardware load balancers but I’d like to get your opinion on this.
Thanks!
I’m happy with my experience of Kemp and F5 but I recommend you take full advantage of their trial periods and do your own assessment and evaluation.
We have successfully run DR of Exchange 2010 SP2. But we revert back to Primary location. We have observed that file share witness still using DR HT/CAS server. So how to force to use Primary site witness server. Whenever DC-DR link has issue our database get dismounted & once link established database automatically mounted.
Thanks In advance!