It seems there’s a new bug with Apple iOS 6.1 devices (iPhones and iPads) that causes excessive transaction log growth on the Exchange server.
As reported in Windows IT Pro:
Some forums have started to register problems with excessive growth of transaction logs for databases hosting the mailboxes of iOS devices that have been upgraded to iOS 6.1 For example, this note describes a situation where upgraded devices seemed to go into a loop and ended up by generating some 50 GB of transaction logs.
Another interesting report indicating that some form of synchronization loop can happen was posted to the forum used by folks who support the F5 load balancers that are often used to front-end large Microsoft Exchange Server deployments.
If past experience is anything to go by there won’t be a rapid fix available, so Exchange admins should explore their options for blocking iOS 6.1 devices from connecting to their Exchange servers.
Update: reports are that removing the device association and letting it re-establish itself on next connection resolves the issue for that user. More details on the cmdlets you can use to perform this for all iOS 6.1 device users here at Tony Redmond’s blog. Microsoft has included that suggested workaround in their KB 2814847 as well.
Update 2: Apple has released a support article that some are saying is this issue. Given the specific nature of the problem description, the simplistic solution, and the feedback I’m hearing from others that this simple solution doesn’t fix the issue in most cases, it would seem this is not the end of the matter.
For Exchange Server 2010 (and Exchange 2013) customers can use ActiveSync device access rules to block specific device types from connecting to Exchange. In this scenario we have two options available using the New-ActiveSyncDeviceAccessRule cmdlet.
Before we proceed you should be aware that blocking devices that were previously able to connect may not go down very well with your user base. Please consider how best to communicate the change to your key customers and other support staff. Also consider whether a block or quarantine action is more suitable for your environment. For more information on the difference between the two read this.
When a device that was previously connected is suddenly blocked by a device access rule it will present a prompt to the user that the credentials may not be correct, which is possibly going to confuse them.
However they will also receive an email to their inbox advising of the block.
Identifying iOS 6.1 Devices
In my own test environment I have a limited subset of the possible iOS 6.1 device types available to me. So the examples I’m about to give below are specifically for iPhone 4S and the 3rd generation iPad.
In your own environment you can check which ActiveSync devices running iOS 6.1 have connected to your Exchange servers with the following command.
[PS] C:\>Get-ActiveSyncDevice | where {$_.deviceos -match "iOS 6.1"} | select devicetype,deviceos,deviceuseragent
DeviceType DeviceOS DeviceUserAgent
---------- -------- ---------------
iPhone iOS 6.1 10B142 Apple-iPhone4C1/1002.142
iPad iOS 6.1 10B141 Apple-iPad3C3/1002.141
Notice how each of my test devices has different DeviceOS and DeviceUserAgent strings, even though they are all iOS 6.1 devices.
If you choose to implement the below solutions make sure you use the valid strings for your environment.
Blocking iOS 6.1 Devices by Device OS
To create a device access rule to block a specific device OS we would run:
[PS] C:\>New-ActiveSyncDeviceAccessRule -QueryString "iOS 6.1 10B142" -Characteristic DeviceOS -AccessLevel Block
RunspaceId : c2c8c056-446f-44f3-9e94-445f34c070dd
QueryString : iOS 6.1 10B142
Characteristic : DeviceOS
AccessLevel : Block
Name : iOS 6.1 10B142 (DeviceOS)
AdminDisplayName :
ExchangeVersion : 0.10 (14.0.100.0)
DistinguishedName : CN=iOS 6.1 10B142 (DeviceOS),CN=Mobile Mailbox Settings,CN=Exchange Server Pro,CN=Microsoft Exchang
e,CN=Services,CN=Configuration,DC=exchangeserverpro,DC=net
Identity : iOS 6.1 10B142 (DeviceOS)
Guid : 98ddca22-22aa-4bf8-b08a-ff785dc1b911
ObjectCategory : exchangeserverpro.net/Configuration/Schema/ms-Exch-Device-Access-Rule
ObjectClass : {top, msExchDeviceAccessRule}
WhenChanged : 2/11/2013 9:21:52 AM
WhenCreated : 2/11/2013 9:21:52 AM
WhenChangedUTC : 2/10/2013 11:21:52 PM
WhenCreatedUTC : 2/10/2013 11:21:52 PM
OrganizationId :
OriginatingServer : HO-DC.exchangeserverpro.net
IsValid : True
Blocking iOS 6.1 Devices by User Agent
To create a device access rule to block a specific device user agent we would run:
[PS] C:\temp>New-ActiveSyncDeviceAccessRule -QueryString "Apple-iPhone4C1/1002.142" -Characteristic UserAgent -AccessLev
el Block
RunspaceId : c2c8c056-446f-44f3-9e94-445f34c070dd
QueryString : Apple-iPhone4C1/1002.142
Characteristic : UserAgent
AccessLevel : Block
Name : Apple-iPhone4C1/1002.142 (UserAgent)
AdminDisplayName :
ExchangeVersion : 0.10 (14.0.100.0)
DistinguishedName : CN=Apple-iPhone4C1/1002.142 (UserAgent),CN=Mobile Mailbox Settings,CN=Exchange Server Pro,CN=Micros
oft Exchange,CN=Services,CN=Configuration,DC=exchangeserverpro,DC=net
Identity : Apple-iPhone4C1/1002.142 (UserAgent)
Guid : 167bb732-5eef-45b0-aa3a-d421682dabb9
ObjectCategory : exchangeserverpro.net/Configuration/Schema/ms-Exch-Device-Access-Rule
ObjectClass : {top, msExchDeviceAccessRule}
WhenChanged : 2/11/2013 9:31:15 AM
WhenCreated : 2/11/2013 9:31:15 AM
WhenChangedUTC : 2/10/2013 11:31:15 PM
WhenCreatedUTC : 2/10/2013 11:31:15 PM
OrganizationId :
OriginatingServer : HO-DC.exchangeserverpro.net
IsValid : True
Removing Device Access Rules
When the time comes that you want to remove the device access rules you can use the Remove-ActiveSyncDeviceAccessRule cmdlet to remove the specific rule by Identity.
[PS] C:\>Get-ActiveSyncDeviceAccessRule | select identity Identity -------- Apple-iPhone4C1/1002.142 (UserAgent) [PS] C:\>Remove-ActiveSyncDeviceAccessRule "Apple-iPhone4C1/1002.142 (UserAgent)"
Other Options for Blocking ActiveSync Devices
Over at the F5 community some members are sharing iRule customizations for blocking devices by user agent, including some that block just the offending behaviour with meeting requests. Of course that is only helpful if you use F5 load balancers in your environment.
And in a previous article Chris Brown shared how to use IIS request filtering to block specific iPhone models, which is something you could possibly take and adapt to this scenario as well.
Blocking Unknown Devices
I don’t happen to have a list of all possible device OS and user agent strings for blocking every type of iOS 6.1 device that may be out there. Which means that despite configuring device access rules it is possible a different device OS or user agent will still connect to Exchange.
If that is a major concern for you consider using the default access level for ActiveSync devices to manage how previously unknown devices types are treated on first connection. This can give you the opportunity to review and make decisions about which devices can connect to your servers before they are able to do any damage.
More Information:
Customers running their Exchange databases with circular logging will not be impacted by this issue, however you should consider this carefully before enabling it (multi-copy DAGs are less risky than single-copy databases for example).
Exchange 2013, which admittedly few customers will be running in production yet, has some new capabilities around ActiveSync device auto-block when bad behaviour such as this is occurring. I don’t have any real world information or configuration examples to share on that though.
Here are a collection of other articles from fellow Exchange MVPs and bloggers on the matter.
About Paul Cunningham
Thank you for the nice write up. This saved me a lot of time.
Hi Paul,
Wouldn’t be easy, just to disable Active sync for all the users using iOS 6.1 with one script, rather than disabling the device access itself?
That is one approach. But what if they have iOS 6.0 the day you disable all the 6.1 users, then they upgrade? You’d have to check every day to make sure no new 6.1 users appeared (could be scripted/automated of course).
Or what if they switch to an Android phone? You’d need to manually enable them again.
I’d say multiple approaches will work in this scenarios, depending on the organization and the administrators who have to deal with the issue.
Agree, it depends on organization and the situation.
DeviceOS: we have got 10B1441 , 1442 , 1443 and 1444
useragent we have got: iOS/6.1 (10B1441) dataaccessd/1.0 and 1442, 1443 and 144 as well
Thanks Mahmoud.
Is it enough to block by DeviceOS, or we have to block by useragent as well?
HI.
I got an error message when I try to create the “Blocking device by deviceos” access rule. I checked our system and our DeviceOS is iOS 6.1 “10B143″ . Anybody has an idea? Our system has 2 Exchange 2010sp1 in Data Availability Group…
[PS] C:\Windows\system32>New-ActiveSyncDeviceAccessRule -QueryString “iOS 6.1 10B143″ -Characteristic “DeviceOS” -AccessLevel Block -whatif
Cannot process argument transformation on parameter ‘Characteristic’. Cannot convert value “DeviceOS” to type “Microsoft.Exchange.Data.Directory.SystemCo
ation.DeviceAccessCharacteristic” due to invalid enumeration values. Specify one of the following enumeration values and try again. The possible enumerat
lues are “DeviceType, DeviceModel”.
+ CategoryInfo : InvalidData: (:) [New-ActiveSyncDeviceAccessRule], ParameterBindin…mationException
+ FullyQualifiedErrorId : ParameterArgumentTransformationError,New-ActiveSyncDeviceAccessRule
You are using Exchange PS or regular one?
I use the exchange management shell. I think maybe I dont have all the service packs installed on my exchange servers. As we know, the latest SP is Service Pack 2 on exchange 2010. I have SP 1 installed. So I think, Im gonna install all the service packs and try again. But if you have any idea, just please share with me..
I’ve gotten the identical error. I managed to keep the mail server happy by making the users unhappy and blocked every iPhone and iPad, and making the mail addicts (myself included) use POP instead to fetch mail. It’s not ideal but it was either that or the Exchangepocalypse on the server.
I’d rather block only 6.0 and 6.1 though, so if you figure out why you get that message please report here. It’s probably a SP level thing since that’s lacking over here too.
Just for the record here – turns out you can’t block on DeviceOS with Echange 2010 RTM. It was added somewhere along the way to SP3, which is what I upgraded our Exchange to this weekend.
The only way I found to do it in 2010 RTM was to just use a big ban hammer and block all iPhones and iPads.
New-ActiveSyncDeviceAccessRule -Accesslevel Allow -Characteristic DeviceType -Querystring iPhone
With SP3, I could reverse those changes and use the DeviceOS and specific version numbers. In our case (and I would wager, most cases, since users with iOS devices upgrade to the latest version pretty much immediately and routinely) there is no real functional difference – except that when Apple releases a 6.1.2, that will be permitted through with no further intervention.
Oops. “-Accesslevel Allow” should obviously be “-Accesslevel block”
We run Exchange 2007, so we had to use a different approach.
On both our CAS/HUB servers we modified IIS to filter and deny these user-agents.
Also works. This is IIS 7.
See this page, from about halfway down : http://www.flamingkeys.com/2011/10/how-to-block-iphone-4s-from-connecting-to-exchange-2010/
Do you know if this issue is just E2K10 or both E2K10 and Exhcnage 2007?
many thanks
I’ve heard people report the issue on Exchange 2007 as well.
Any way to find out what users have iOS 6.1 and send out a targeted annoucement? Trying some combinations of get-mailbox and piping to your identifying iOS6.1 devices code, but it’s not giving me what I want.
I used PS one-liner: Get-ActiveSyncDevice | where {$_.deviceos -like “iOS 6.1*”} | select identity,deviceOS | ft
It’ll get you AD user and associated device OS – iOS 6.1* in this case.
MS just released a support article http://support.microsoft.com/kb/2814847/en-us
Great write up Paul. Is there a way to associate a user with the iOS device? So I can communicate with them directly?
Via http://blogs.technet.com/b/exchange/archive/2012/01/31/a-script-to-troubleshoot-issues-with-exchange-activesync.aspx
Download Microsoft LogParser.exe (2.2)
Download ActiveSyncReport.ps1
Use this format to query 2 servers for devices syncing more than 1,500 times/day.
The -Date area is set to -1 to pull Yesterday’s Logs.
.\ActiveSyncReport.ps1 -IISLog “\\CASSERVER1\d$\IISLogs\W3SVC1″,”\\CASSERVER2\d$\IISLogs\W3SVC1″ -LogparserExec “C:\Program Files (x86)\Log Parser 2.2\LogParser.exe” -ActiveSyncOutputFolder c:\EASReports -MinimumHits 1500 -Date $(get-date (get-date).Adddays(-1) -Format “MM-dd-yyyy”) -DeviceID ApplC39JG8QHDTTQ -HTMLReport -SendEmailReport -SMTPRecipient XYZ@YourDomain.com -SMTPSender ActiveSyncReport@YourDomain.com -SMTPServer XYZ.YourDomain.com
OOps.
Please remove the “-DeviceID ApplC39JG8QHDTTQ” from the command. That is for SPECIFIC device reporting.
I’ve come across scripts that show which devices and OS versions they are running, but is there one that can tell me specifically which devices have high hits (reads, writes, accesses) to the server?
…that is given a certain time frame to see the hits?
Apple also confirmed this bug and offered a workaround for the iPhone users:
https://support.apple.com/kb/TS4532
Has anyone done the quick “Calendar Toggle” steps by Apple, and have they confirmed it fixed the issue?
And by ‘fixed the issue’ – has anyone noticed the transaction logs and CPU/Memory usage has returned to normal?
How fast are these transaction logs growing anyway? We have around 40 devices on 6.1 and i have free space of 1TB still available. I do see the transaction logs growing, but not sure if its becuase of this issue or of thats normal?
Is this issue related to Exchange 2003 as well or not?
No, it’s not been seen to exchange 2003.
Thank you for these posts! They have been very helpful in getting our Exchange 2010 server working again as it stopped passing email because of 68 GB worth of transaction logs filling up our drive to capacity. I ran a full windows backup to an ext HDD (which removes all the transaction logs that have already been written to the DB). While that was happening I blocked “iOS 6.1 10B142″ which appears to only blocks iPhone 4S users running iOS 6.1 via this command:
New-ActiveSyncDeviceAccessRule -QueryString “iOS 6.1 10B142″ -Characteristic DeviceOS -AccessLevel Block.
Apple released a fix for this (iOS ver 6.1.1), but only released it for iPhone 4S. Maybe it’s not required for the iPad 2, iPad 3, iPhone 4 and iPhone 5s also running 6.1? Here are all the Apple versions running in our environment should you need to explicitly block them:
iOS 6.1 10B141 = iPhone4 (ver 6.1) and iPad2 (ver 6.1)
iOS 6.1 10B142 = iPhone4S (ver 6.1)
iOS 6.1 10B143 = iPhone5 (ver 6.1) and iPad3 (ver 6.1)
iOS 6.1 10B145 = iPhone4s (ver 6.1.1)
I am currently blocking ONLY iOS 6.1 10B142 for the 4S and our transaction logs appear to be growing at a manageable rate.
The 6.1.1 patch is for the 4S, granted, but it is for a 3G connectivity issue. It was already in the works/complete when this Exchange issue sailed up.
I’ve seen rumors of an expected 6.1.2 this upcoming week which will address this issue.
But you need to block all the 6.1 variants and the 6.1.1 variant too in your Exchange, as 6.1.1 will still cause this same issue.
Thanks for the guide!
Now that 6.1.2 is out how do i tell the server to talk to the blocked device again? IE users upgrading still have 6.1 or 6.1.1 listed on the server as thier os as the server has them blocked.
Is there a powershell command where I can view the user and device to clear them? A command to have it recheck block devices?
I too am also wondering this. Will previously blocked devices automatically connect again once running 6.1.2? How often does the policy update?
Or will it be a case of running the above command to identify all 6.1 devices and piping that into a Remove-ActiveSyncDevice?
We noticed that 6.1.2 devices were still being blocked.
We had to remove the blocking rules.
or you can enable the device and leave the rule after it upgrades.
Confirm a user has upgraded then -
Get-ActiveSyncDevice -Mailbox user@domain.com | Fl DeviceId
this will list a users devices by ID. Then –
Set-CASMailbox –Identity user@domain.com –ActiveSyncAllowedDeviceIDs “IDxxxxxxx”
and the previously blocked device works.
I think. Confirmation that this is a good way to do it would be stellar.
After initial testing we have found that previously blocked users who upgrade do not automatically reconnect to their mailbox. Instead they receive a username or password is incorrect error message.
Running the Remove-ActiveSyncDevice against those users lets them connect again automatically.
We will be running the below couple of scripts on a regular basis to slowly remove the 6.1 & 6.1.1 devices.
Get-ActiveSyncDevice | where {$_.deviceos -match “iOS 6.1 10B”} | Remove-ActiveSyncDevice -Confirm:$false
Get-ActiveSyncDevice | where {$_.deviceos -match “iOS 6.1.1 10B”} | Remove-ActiveSyncDevice -Confirm:$false
Graham,
You are saying that there is no need to remove and re-add the account on the device? I saw that we needed to on a device that I tested, did I just not give it enough time? I waited like 5 minutes, still not automatically connecting in.
Thanks,
Ben
Hi Ben,
Yes, in my experience there has been no need to recreate the account on the device. I found that previously blocked devices that had since been upgraded did not automatically sync and received the “username or password is incorrect” error. I then removed the device using the above Cmdlet and verified that it had also been removed from within that user’s device list using the ECP.
At this point, when trying to sync, the user no longer received the “username or password is incorrect” error but instead received a generic connection error. Then after around 5 minutes the device connected and sync’d properly and I could see the devices within the user’s list.
Have you been running the Remove-ActiveSyncDevice for the users in question because that is absolutely necessary?
I had hoped that users would not have to recreate the accounts on their devices but upon establishing that they did not automatically reconnect, that is the first thing we tried – without success. It was only after we ran the above Cmdlet that they would connect again.
Hey Graham,
I fear I was not patient enough. I will test out on a few more users and I’m sure that time was the remedy there. Thank you for that cmdlet, you’ve saved me a lot of headache and carpal tunnel from re-configuring phones.
Thanks,
Ben
I initially saw that the blocked iOS devices didn’t connect, but I tried a few things first like recreating the Exchange account on the phone and simply just waiting a while. Turning push on/off on the phone didn’t do it but eventually I deleted the partnerships from the Exchange server and after a minute or two the blocked devices connected.
I’m not sure which part solved the issue, if it was the recreating, the turning push on/off or just deleting the partnership on the Exchange end, but I didn’t have to touch the blocking rules.
Oops, just read Graham Riley’s comment, apparently it was deleting the device partnerships that did the trick. Good to know.
I can confirm that you must remove the device partnership before a reconnect will occur. Same in my organization.
You can also accomplish this in the Management Console in 2010 via Recipient Config > Mailbox > |user’s mailbox| > “Manage Mobile Phone…” > select which device > Remove button
To run New-ActiveSyncDeviceAccessRule -QueryString “iOS 6.1 10B142″ -Characteristic DeviceOS -AccessLevel command the minimun requirement should be MS Exchange 2010 SP2
I’ve added a link in the post to an article on the difference between block and quarantine actions when it comes to dealing with these types of situations:
http://exchangeserverpro.com/is-quarantine-smarter-than-block-for-activesync-device-access-rules
really good explanation Paul, thanks
The problem with this option alone is that if a user got a new device (say an iPhone 5) and their old one is still on iOS 6.1 or 6.1.1, once you block that iOS version, the user will still get the E-mail stating that their device is blocked. The user may think that they will be blocked when in fact their old device will be but not their new one. You have to include a legacy device cleanup. In addition it would be nice to have a script to disable/remove devices by iOS instead of just blocking them.
You’re saying their new device will trigger a block email notification for their old device?
Sorry it’s taken me so long to get back to you. Yes, say for instance that someone had an old device that had iOS 6.1 and you don’t have a scheduled task to remove stagnant devices. If you choose to block iOS 6.1 but say they have a new phone with a different OS version that isn’t being blocked, that user will still receive a notice stating that their ActiveSync device will be blocked (in reference to the device that will be blocked). But the problem is, the user doesn’t understand this. There are two ways I came up with to prevent this:
Option 1: Run a script to delete specific user agent strings prior to blocking
Remove specific user agent string first:
Get-ActiveSyncDevice -result unlimited | Get-ActiveSyncDeviceStatistics | where {$_.DeviceUserAgent -eq “iOS 6.1 10B141″} | foreach-object {Remove-ActiveSyncDevice ([string]$_.Guid) -confirm:$false}
Then block:
New-ActiveSyncDeviceAccessRule -querystring “iOS 6.1 10B141″ -characteristic DeviceOS -accesslevel block
(Change user agent string and run again to block each iOS 6.1 user agent string)
Option 2: Have a script delete stagnant devices as scheduled task (.bat calling the .ps1)
Add-PSSnapin Microsoft.Exchange.Management.PowerShell.E2010
$DevicesToRemove = Get-ActiveSyncDevice -result unlimited | Get-ActiveSyncDeviceStatistics | where {$_.LastSuccessSync -le (Get-Date).AddDays(“-60″)}
$DevicesToRemove | remove-activesyncdevice -confirm:$false
Remove-PSSnapin Microsoft.Exchange.Management.PowerShell.E2010
To clarify, if they have 2 devices – an old one that has iOS 6.1 that they no longer use but is still recognized by and connected to the Exchange environment and a new device with say iOS 6.1.3 that they currently use. Because the old device falls under the iOS version you plan to block, they will receive the notice you have above stating that their device OS will be blocked. This may confuse the end user, so to prevent that, hence the 2 options above. Hopefully that makes sense.
I always found that using select never gave me the results i liked and i get in the habbit of using the Format-List and then using -Property attrib. I ahve read a few of your blogs mate and they are really awsome and have helped me with power shell and exchange shell heaps.
I use this line
Get-ActiveSyncDevice | where {$_.deviceos -match “ios 6.”} | Format-List -Property Friendlyname, distinguishedname, name, deviceos
Actually on second thoughts couldn’t you just create 2 usiversal groups in AD, 1 BlockIOSDevice, 2 UnBlockiOSDevice and iterate through all users with ios6.1.1 or what ever and make them a member and then just block the group the active directory object/group?
this bug can’t be real… now i know why my logs are getting bigger and bigger!!!