How to Configure a File Share Witness for an Exchange 2010 Database Availability Group

In my Exchange Server 2010 lab environment I unwittingly created a problem for the Database Availability Group. In preparing to consolidate all of the server roles onto just two servers and implement a hardware load balancer I went ahead and decommissioned the two CAS/HT servers that previously made up the CAS array in the site.

Naturally one of those CAS/HT servers also happened to be the File Share Witness for my two-member DAG. Whoops!

Now my DAG displays a warning when I check the health of it.

WARNING: Database availability group ‘dag-headoffice’ witness is in a failed state. The database availability group requires the witness server to maintain quorum. Please use the Set-DatabaseAvailabilityGroup cmdlet to re-create the witness server and directory.

In this real world this situation may also arise if the server hosting the File Share Witness was being decommissioned, or if it had failed. Fortunately we can resolve the problem by specifying a new FSW for the DAGm which I will demonstrate here.

I’m going to use another member server within the site as my FSW, which allows me to demonstrate a related problem. The server is named HO-MGT so using the Set-DatabaseAvailabilityGroup cmdlet to configure the FSW would mean I run this command.

[PS] C:\>Set-DatabaseAvailabilityGroup dag-headoffice -WitnessServer ho-mgt -WitnessDirectory C:\DAGFSW

However in this case I get an error.

WARNING: The Exchange Trusted Subsystem is not a member of the local Administrators group on specified witness server
ho-mgt.
WARNING: Insufficient permissions to access file shares on witness server ‘HO-MGT.exchangeserverpro.net’. Until this problem is corrected, the database availability group may be more vulnerable to failures. You can use the Set-DatabaseAvailabilityGroup cmdlet to try the operation again. Error: Access is denied
Unable to change the quorum for database availability group dag-headoffice. Witness server ‘\\HO-MGT.exchangeserverpro.net\dag-headoffice.exchangeserverpro.net’ network name wasn’t found. This may be due to firewall settings.
+ CategoryInfo : InvalidArgument: (:) [Set-DatabaseAvailabilityGroup], DagTaskProblemC…ptionBadNetName
+ FullyQualifiedErrorId : 75321C4E,Microsoft.Exchange.Management.SystemConfigurationTasks.SetDatabaseAvailabilityGroup

If you were running the same command but specifying another Exchange 2010 server to be the FSW you would not receive that error. This is because Exchange servers trust each other to perform this type of administration, thanks to a group called Exchange Trusted Subsystem.

All of the Exchange 2010 servers have this group as a member of their local Administrators group, for example here the local Administrators group of one of my DAG members.

Exchange Trusted Subsystem group in local Administrators

Exchange Trusted Subsystem group in local Administrators

So the solution is to add the Exchange Trusted Subsystem group to the local Administrators group on my HO-MGT server, and then run the Set-DatabaseAvailabilityGroup command again.

After running the command you can see that Exchange has created the folder and shared it on the FSW server, no need to manually create the folder or set any permissions yourself.

Database Availability Group File Share Witness directory

Database Availability Group File Share Witness directory

Now when checking the health of the Database Availability Group you should not receive any warnings about missing File Share Witness servers.

About Paul Cunningham

Paul is a Microsoft Exchange Server MVP and publisher of Exchange Server Pro. He also holds several Microsoft certifications including for Exchange Server 2007, 2010 and 2013. Connect with Paul on Twitter and Google+.

Comments

  1. Jose Parocua says:

    Paul,

    We have 2 CAS/HTS servers in the same AD Site. When my primary CAS (which also hosts the FSW) goes offline, my outlook clients get prompted to provide credentials and they won’t automatically connect to the secondary CAS until i manually update DNS records. Even then they have to provide credentials to log on to the secondary CAS server.

    Do i possibly need to move the FSW role to another member server?

    Thanks!

    • The CAS role (and the RPCClientAccessService) has nothing to do with the File Share Witness for a DAG. They may happen to be on the same server, but they are completely different things.

      Your issue is that the RPCClientAccessServer that Outlook users are connecting to is going offline. You would solve that problem with a CAS Array and load balancing solution.

  2. Unfortunately 4 me it doesn’t work. In producton i had to move cluster witness on another server, where I already put exchange subsystem in local admins. The share is created. Inside are two files with 0 kb. The share witness doeasn’t work. In GUI i see success, but in powershell there is an error that exchange subsystem is not local admin (but aparently it is). Just as is stated here:
    http://blogs.technet.com/b/scottschnoll/archive/2011/06/08/witness-server-warning-message-when-using-certain-database-availability-group-tasks.aspx
    Is this a bug or smth I am missing?

    • Scott explains in that article that the error is a bug in Exchange and can be disregarded if you have put the ETS group into the local admins group on a non-Exchange FSW.

  3. I have some serious man-love for you. Nobody writes a better guide.
    When I read you work, I know what I am doing it and more importantly, why.
    Keep up the great work and thanks for all the great articles.

  4. Gowhar Rashid Zargar says:

    Hi…
    Environment:
    2 Servers installed with MAILBOX+HUB TRANSPORT+CAS Roles in AD DOMAIN on physical servers
    2 Servers installed with EDGE TRANSPORT Roles in a WORKGROUP on Hypervisors

    I want to make a DAG for MAILBOX databases for redundancy. Can I make EDGE TRANSPORT Server as a WITNESS SERVER. Please help with that

  5. Gowhar Rashid Zargar says:

    OK Great !!! I have one more server for McAfee Antivirus Server running on hypervisor in AD Domain. Shall I use that one as a Witness Server ?

    • I don’t see a problem with that, as long as the server is reliable and not constantly overloaded already. FSW doesn’t add any significant load to a server but the FSW should generally be reliable.

  6. Gowhar Rashid Zargar says:

    Thanks a million Mr. Cunningham… Appreciate your help.

  7. Jose Parocua says:

    Paul,

    Another question….

    What brand of hardware load balancer have you had good luck with and why? We may be going down the path of purchasing a hardware load balancer for our CAS array. I know Microsoft offers an article on supported hardware load balancers but I’d like to get your opinion on this.

    Thanks!

  8. Sandeep Sawant says:

    We have successfully run DR of Exchange 2010 SP2. But we revert back to Primary location. We have observed that file share witness still using DR HT/CAS server. So how to force to use Primary site witness server. Whenever DC-DR link has issue our database get dismounted & once link established database automatically mounted.
    Thanks In advance!

  9. I have Windows 2008 R2 server with Exchange as nodes + one another Windows 2008 R2 server to be configured as Witness server.

    All the 3 systems are under “Exchange Trusted Subsystem” security group. However, my DAG configuration constantly fails with the error as below: I have possibly added every place, in all these 3 servers that they are part of “Exchange Trusted Subsystem”

    “Warning:
    Insufficient permissions to access file shares on witness server ‘win-52-239.interopexchange.com’. Until this problem is corrected, the database availability group may be more vulnerable to failures. You can use the Set-DatabaseAvailabilityGroup cmdlet to try the operation again. Error: Access is denied

    Exchange Management Shell command completed:
    New-DatabaseAvailabilityGroup -Name ‘InteropDAG1′ -WitnessServer ‘win-52-239.interopexchange.com’ -WitnessDirectory ‘c:\witness’

    Elapsed Time: 00:00:00″

    Any help possible on this please?

  10. Umang Shah says:

    I have 4 MBX servers, 2 CAS & 2 Hub servers in my environment. The 4 mbx servers are in a DAG and the FSW is on HUB1. When I reboot HUB 1, the FSW goes into a failed state. Even when after the server comes back up and is functional, the FSW stays in a failed state. The only way I can bring the FSW online, is by manually right-clicking the resource and bring it on-line. Any idea how to fix this?

    Thanks!

  11. Hi paul…

    I notice that the “everyone” has full control to my fsw directory……I didn’t set this server up but this seems like a security hole. does the everyone account need full control the the fsw directory.IF i change this to read only will it break the Dag???

    thanks

    Dave

    • Are you referring to NTFS or Share permissions?

      • paul it looks like both share and ntfs…the everyone group have full permission..

        thanks

        Dave

        • Hi Paul,
          I have 2 exchange 2010 Sp3 (MBX/CAS/HT) servers in a DAG installed on windows 2008 R2. I have configured FSW on non exchange server on the same site which is windows 2012 server.The cluster is online and failover/failback happens properly. When we perform test-replicationhealth for any of the members of this DAG from the other exchange server in the same organisation, I get the result as file share quorum failed with error “couldn’t access the file share witness share”.

          Where as exchange trusted subsystem group already has admin rights on the server where we have configured FSW.

  12. Wayne Hutchinson says:

    Paul – Is it possible to use something like a NAS volume to act as a witness server of does it have to be a Windows Server? I use several NFS and CIFS shares hosted off of my NAS for process targets and would have more faith in that as a highly available source to migrate my Witness directory to.

  13. David Tabib says:

    Hi Paul,

    I am running into an issue which is don’t know what to do. Here is my situation. I have 4 member. Two DAG + witness server is on one network and 2 dag server in another network.

    About 2 months ago I moved the Witness server to a new server the old server was called ‘mmwitness’ the new Witness server is called “License”

    I ran this command to verify:
    Get-DatabaseAvailabilityGroup -Identity DAG -Status | FL

    Today we had a network issue and databases moved from one network to another, it really should not have. It really should not have moved because the 2 dag members plus witness resides on the same network. When I looked at the log I saw a surprising error. It said:

    File share witness resource ‘File Share Witness (\\mmwitness.xxxxx.com\dag.xxxxxcom)’ failed to arbitrate for the file share ‘\\licensesrv.xxxxxx.com\dag.xxxxx.com’. Please ensure that file share ‘\\licensesrv.xxxxx.com\dag.xxxxx.com’ exists and is accessible by the cluster.

    I do not understand why there is something in my dag setup that still pointing to my old \\mmwitness server. Can you assist?

Leave a Comment

*

We are an Authorized DigiCert™ SSL Partner.