How to Configure a File Share Witness for an Exchange 2010 Database Availability Group

In my Exchange Server 2010 lab environment I unwittingly created a problem for the Database Availability Group. In preparing to consolidate all of the server roles onto just two servers and implement a hardware load balancer I went ahead and decommissioned the two CAS/HT servers that previously made up the CAS array in the site.

Naturally one of those CAS/HT servers also happened to be the File Share Witness for my two-member DAG. Whoops!

Now my DAG displays a warning when I check the health of it.

WARNING: Database availability group ‘dag-headoffice’ witness is in a failed state. The database availability group requires the witness server to maintain quorum. Please use the Set-DatabaseAvailabilityGroup cmdlet to re-create the witness server and directory.

In this real world this situation may also arise if the server hosting the File Share Witness was being decommissioned, or if it had failed. Fortunately we can resolve the problem by specifying a new FSW for the DAGm which I will demonstrate here.

I’m going to use another member server within the site as my FSW, which allows me to demonstrate a related problem. The server is named HO-MGT so using the Set-DatabaseAvailabilityGroup cmdlet to configure the FSW would mean I run this command.

[PS] C:\>Set-DatabaseAvailabilityGroup dag-headoffice -WitnessServer ho-mgt -WitnessDirectory C:\DAGFSW

However in this case I get an error.

WARNING: The Exchange Trusted Subsystem is not a member of the local Administrators group on specified witness server
ho-mgt.
WARNING: Insufficient permissions to access file shares on witness server ‘HO-MGT.exchangeserverpro.net’. Until this problem is corrected, the database availability group may be more vulnerable to failures. You can use the Set-DatabaseAvailabilityGroup cmdlet to try the operation again. Error: Access is denied
Unable to change the quorum for database availability group dag-headoffice. Witness server ‘\\HO-MGT.exchangeserverpro.net\dag-headoffice.exchangeserverpro.net’ network name wasn’t found. This may be due to firewall settings.
+ CategoryInfo : InvalidArgument: (:) [Set-DatabaseAvailabilityGroup], DagTaskProblemC…ptionBadNetName
+ FullyQualifiedErrorId : 75321C4E,Microsoft.Exchange.Management.SystemConfigurationTasks.SetDatabaseAvailabilityGroup

If you were running the same command but specifying another Exchange 2010 server to be the FSW you would not receive that error. This is because Exchange servers trust each other to perform this type of administration, thanks to a group called Exchange Trusted Subsystem.

All of the Exchange 2010 servers have this group as a member of their local Administrators group, for example here the local Administrators group of one of my DAG members.

Exchange Trusted Subsystem group in local Administrators

Exchange Trusted Subsystem group in local Administrators

So the solution is to add the Exchange Trusted Subsystem group to the local Administrators group on my HO-MGT server, and then run the Set-DatabaseAvailabilityGroup command again.

After running the command you can see that Exchange has created the folder and shared it on the FSW server, no need to manually create the folder or set any permissions yourself.

Database Availability Group File Share Witness directory

Database Availability Group File Share Witness directory

Now when checking the health of the Database Availability Group you should not receive any warnings about missing File Share Witness servers.

About Paul Cunningham

Paul is a Microsoft Exchange Server MVP and publisher of Exchange Server Pro. He also holds several Microsoft certifications including for Exchange Server 2007, 2010 and 2013. Find Paul on Twitter, LinkedIn or Google+, or get in touch for consulting/support engagements.

Comments

  1. Jose Parocua says:

    Paul,

    We have 2 CAS/HTS servers in the same AD Site. When my primary CAS (which also hosts the FSW) goes offline, my outlook clients get prompted to provide credentials and they won’t automatically connect to the secondary CAS until i manually update DNS records. Even then they have to provide credentials to log on to the secondary CAS server.

    Do i possibly need to move the FSW role to another member server?

    Thanks!

    • The CAS role (and the RPCClientAccessService) has nothing to do with the File Share Witness for a DAG. They may happen to be on the same server, but they are completely different things.

      Your issue is that the RPCClientAccessServer that Outlook users are connecting to is going offline. You would solve that problem with a CAS Array and load balancing solution.

  2. Unfortunately 4 me it doesn’t work. In producton i had to move cluster witness on another server, where I already put exchange subsystem in local admins. The share is created. Inside are two files with 0 kb. The share witness doeasn’t work. In GUI i see success, but in powershell there is an error that exchange subsystem is not local admin (but aparently it is). Just as is stated here:
    http://blogs.technet.com/b/scottschnoll/archive/2011/06/08/witness-server-warning-message-when-using-certain-database-availability-group-tasks.aspx
    Is this a bug or smth I am missing?

    • Scott explains in that article that the error is a bug in Exchange and can be disregarded if you have put the ETS group into the local admins group on a non-Exchange FSW.

  3. I have some serious man-love for you. Nobody writes a better guide.
    When I read you work, I know what I am doing it and more importantly, why.
    Keep up the great work and thanks for all the great articles.

  4. Gowhar Rashid Zargar says:

    Hi…
    Environment:
    2 Servers installed with MAILBOX+HUB TRANSPORT+CAS Roles in AD DOMAIN on physical servers
    2 Servers installed with EDGE TRANSPORT Roles in a WORKGROUP on Hypervisors

    I want to make a DAG for MAILBOX databases for redundancy. Can I make EDGE TRANSPORT Server as a WITNESS SERVER. Please help with that

  5. Gowhar Rashid Zargar says:

    OK Great !!! I have one more server for McAfee Antivirus Server running on hypervisor in AD Domain. Shall I use that one as a Witness Server ?

    • I don’t see a problem with that, as long as the server is reliable and not constantly overloaded already. FSW doesn’t add any significant load to a server but the FSW should generally be reliable.

  6. Gowhar Rashid Zargar says:

    Thanks a million Mr. Cunningham… Appreciate your help.

  7. Jose Parocua says:

    Paul,

    Another question….

    What brand of hardware load balancer have you had good luck with and why? We may be going down the path of purchasing a hardware load balancer for our CAS array. I know Microsoft offers an article on supported hardware load balancers but I’d like to get your opinion on this.

    Thanks!

  8. Sandeep Sawant says:

    We have successfully run DR of Exchange 2010 SP2. But we revert back to Primary location. We have observed that file share witness still using DR HT/CAS server. So how to force to use Primary site witness server. Whenever DC-DR link has issue our database get dismounted & once link established database automatically mounted.
    Thanks In advance!

  9. I have Windows 2008 R2 server with Exchange as nodes + one another Windows 2008 R2 server to be configured as Witness server.

    All the 3 systems are under “Exchange Trusted Subsystem” security group. However, my DAG configuration constantly fails with the error as below: I have possibly added every place, in all these 3 servers that they are part of “Exchange Trusted Subsystem”

    “Warning:
    Insufficient permissions to access file shares on witness server ‘win-52-239.interopexchange.com’. Until this problem is corrected, the database availability group may be more vulnerable to failures. You can use the Set-DatabaseAvailabilityGroup cmdlet to try the operation again. Error: Access is denied

    Exchange Management Shell command completed:
    New-DatabaseAvailabilityGroup -Name ‘InteropDAG1′ -WitnessServer ‘win-52-239.interopexchange.com’ -WitnessDirectory ‘c:\witness’

    Elapsed Time: 00:00:00″

    Any help possible on this please?

  10. Umang Shah says:

    I have 4 MBX servers, 2 CAS & 2 Hub servers in my environment. The 4 mbx servers are in a DAG and the FSW is on HUB1. When I reboot HUB 1, the FSW goes into a failed state. Even when after the server comes back up and is functional, the FSW stays in a failed state. The only way I can bring the FSW online, is by manually right-clicking the resource and bring it on-line. Any idea how to fix this?

    Thanks!

  11. Hi paul…

    I notice that the “everyone” has full control to my fsw directory……I didn’t set this server up but this seems like a security hole. does the everyone account need full control the the fsw directory.IF i change this to read only will it break the Dag???

    thanks

    Dave

    • Are you referring to NTFS or Share permissions?

      • paul it looks like both share and ntfs…the everyone group have full permission..

        thanks

        Dave

        • Hi Paul,
          I have 2 exchange 2010 Sp3 (MBX/CAS/HT) servers in a DAG installed on windows 2008 R2. I have configured FSW on non exchange server on the same site which is windows 2012 server.The cluster is online and failover/failback happens properly. When we perform test-replicationhealth for any of the members of this DAG from the other exchange server in the same organisation, I get the result as file share quorum failed with error “couldn’t access the file share witness share”.

          Where as exchange trusted subsystem group already has admin rights on the server where we have configured FSW.

  12. Wayne Hutchinson says:

    Paul – Is it possible to use something like a NAS volume to act as a witness server of does it have to be a Windows Server? I use several NFS and CIFS shares hosted off of my NAS for process targets and would have more faith in that as a highly available source to migrate my Witness directory to.

  13. David Tabib says:

    Hi Paul,

    I am running into an issue which is don’t know what to do. Here is my situation. I have 4 member. Two DAG + witness server is on one network and 2 dag server in another network.

    About 2 months ago I moved the Witness server to a new server the old server was called ‘mmwitness’ the new Witness server is called “License”

    I ran this command to verify:
    Get-DatabaseAvailabilityGroup -Identity DAG -Status | FL

    Today we had a network issue and databases moved from one network to another, it really should not have. It really should not have moved because the 2 dag members plus witness resides on the same network. When I looked at the log I saw a surprising error. It said:

    File share witness resource ‘File Share Witness (\\mmwitness.xxxxx.com\dag.xxxxxcom)’ failed to arbitrate for the file share ‘\\licensesrv.xxxxxx.com\dag.xxxxx.com’. Please ensure that file share ‘\\licensesrv.xxxxx.com\dag.xxxxx.com’ exists and is accessible by the cluster.

    I do not understand why there is something in my dag setup that still pointing to my old \\mmwitness server. Can you assist?

    • Unless you’ve blocked activation on the server in the second site there is nothing to stop it from being the DAG member that Active Manager chooses for a failover.

      For your other question, did you run Set-DatabaseAvailabilityGroup to change the FSW location?

      • David Tabib says:

        I don’t have blocked activation but being the 2 DAG members and Witness servers are in same network, the DAG should not have switched because I have 3 servers on one side and on the other network I have only 2 servers. So, the issue comes in about the Witness server.

        I used the GUI to set the new witness server it shows that my FSW is set to licenserv and NOT the old server that used to be called mmwitness. I also can see in the licenserv a directory is created and that Exchange Trusted Subsystem is the member of local administrator group. But, the error that I got when we had the network issue is wired; somehow Exchange is looking to the old server somewhere. Again the error is:

        “File share witness resource ‘File Share Witness (\\mmwitness.xxxxx.com\dag.xxxxxcom)’ failed to arbitrate for the file share ‘\\licensesrv.xxxxxx.com\dag.xxxxx.com’. Please ensure that file share ‘\\licensesrv.xxxxx.com\dag.xxxxx.com’ exists and is accessible by the cluster.”

        • Where the DAG members are located in the network is not a factor taken into account by Active Manager when it initiates the Best Copy Selection process to determine which database copy to mount in a failover scenario. If you don’t want databases activating on that server you will need to block activation.

          The FSW issue appears to be just how it is. When you create the DAG and specify an FSW the cluster resource is named “File Share Witness (\\servername\sharename)” to match the server name and share name at that time. However, when you specify a different witness server, even though the FSW path updates, the name of the FSW resource in the cluster does not change.

        • Ryan Sastradinata says:

          Hey Paul Great article you have here.

          Unfortunately, i have pretty much the same error, I’m currently having 2 DAG Member across 2 subnet (192.168.6.0/24 and 172.20.7.0/24) and a witness server which a Client Access Server (IMTV-000031) that resides on 192.168.6.0/24 network. My DAG name is “IMTV-000032-DAG”, I’ve setup the DAG to have 2 IP address, each one for each subnet.
          Few days ago we got an electricity problem within the server rack, which cause all of the servers lost its power. When i turn it back on, i was surprised that on the “Server Manager Dashboard” i got an error that say, “refresh failed”, so i do a refresh but it still failed with the error message of

          “IMTV-000032-DAG : Configuration refresh failed with the following error: The metadata failed to be retrieved from the server, due to the following error: WinRM cannot process the request. The following error with errorcode 0×80090322 occurred while using Kerberos authentication: An unknown security error occurred.”

          Then i see on Failover Cluster Manager, I got the server (IMTV-000032-DAG) to be online, but the File Share Witness is in Failed status, it also state that the File Share Witness is not configured. So I dive in to the log to see the problem, it recorded error and critical event

          The critical event I got is
          “File share witness resource ‘File Share Witness (\\imtv-000031.company.com\IMTV-000032-DAG.company.com)’ failed to arbitrate for the file share ”. Please ensure that file share ” exists and is accessible by the cluster.”

          The error event I got is
          “Cluster resource ‘File Share Witness (\\imtv-000031.company.com\IMTV-000032-DAG.company.com)’ of type ‘File Share Witness’ in clustered role ‘Cluster Group’ failed. The error code was ’0xc000006b’.”

          I’ve check the share path, and its still there.
          I’ve check the database copy status, everything is healthy, the database is all mounted correctly.
          Even though everything looks fine, I’m afraid that this will cause a major problem in the future. Could you please help me on this? It will be very much appreciated

        • Server Manager can’t connect to your DAG computer object so I wouldn’t worry too much about that.

          The FSW needs to be online. You can try to manually force it online in Failover Cluster Manager. If that won’t work try setting the FSW again using Set-DatabaseAvailabilityGroup, maybe even try setting it to be a different server.

          Also check clock skew. A simple thing that causes many things to break.

        • Hey Paul, thank you for replying my comment. I’ve try your suggestion about the time skew and running set-databaseAvaibilityGroup for the witness server and path, but yet still no luck. I’ve try the approch suggested by this article (“http://exchangeexpertscommunity.blogspot.com/2013/08/how-to-reconfigure-failed-fileshare.html”), i got pretty much the same error with When i run the Set-DatabaseAvaibilityGroup command from EMS, which say:

          “There was a problem changing the quorum model for database availability group IMTV-000032-DAG. Error: An error
          occurred while attempting a cluster operation. Error: Cluster API failed:
          “ClusterResourceControl(controlcode=CLUSCTL_RESOURCE_SET_PRIVATE_PROPERTIES) failed with 0xc000006b. Error: Unknown
          error (0xc000006b)”
          + CategoryInfo : InvalidArgument: (:) [Set-DatabaseAvailabilityGroup], DagTaskProblemChangingQuorumExcept
          ion
          + FullyQualifiedErrorId : [Server=IMTV-000029,RequestId=84b52516-8e45-4483-9bf0-a6553c81d715,TimeStamp=6/29/2014 9
          :21:21 AM] [FailureCategory=Cmdlet-DagTaskProblemChangingQuorumException] 3B41C8F3,Microsoft.Exchange.Management.S
          ystemConfigurationTasks.SetDatabaseAvailabilityGroup
          + PSComputerName : imtv-000029.company.com”

          When I run test-ReplicationHealth, I got all *Passed* except for “Quorum Group” and “FileShareQuorum” which say that the DAG and the File Share path can’t be reached

          i’m currently waiting for my supervisor to point which server that could be use to replace to current witness server. In the mean time, i was wondering about the do’s and don’ts in my environment (2 node DAG with different subnet and 1 witness server)?
          cause i read some articles saying that i should be very cautious about rebooting server when there’s only 2 node and a witness server in the DAG

        • Yes, if you restart a DAG node in your situation you risk quorum being lost and the entire DAG going offline.

Leave a Comment

*

We are an Authorized DigiCert™ SSL Partner.