Earlier this week I started getting alerts that our Exchange 2013 server was running low on disk space for our transaction logs. When I started looking at the drive, I noticed that one of the databases was generating a new log file every 1 – 2 seconds and by the time I received the notification about the drive space we had over 460,000 log files for that database.
First step, was to enable circular logging to free up disk space. Within a few minutes we had freed up around 400GB of space.
The second step, was to try and track down the mailbox(es) that were causing the problem. The suggestion from our vendor and Microsoft was to create a new database and start moving mailboxes over to the new database and see which one causes the transaction logs to start growing. They suspected either a message stuck in the users Outbox or an ActiveSync device was causing problems. I did this by moving several mailboxes at a time (I didn’t want to wait to move 60+ mailboxes one at a time). I got it narrowed down to a batch of 6 accounts. I used the Get-MobileDevices -Mailbox username command for each of those 6 accounts and found out that only 3 of them had mobile devices associated.
The third step, was to get the usage statistics for the database and see which of those three users had high usage. We used Get-StoreUsageStatistics -Database “db_name” | Sort-Object LogRecordBytes -desc | Select-Object -First 25 | ft DigestCategory,DisplayName,LogRecordBytes,*time* -auto. We found that two of those three users had high usage.
The fourth step, was to disable Exchange ActiveSync on both of these mailboxes. Once that was done the transaction log growth slowed way down (1 new log every 30 – 60 seconds).
I went to user1 and looked at Outlook and found nothing unusual (no messages stuck in the Outbox). I had user1 log into OWA and clean up the mobile devices associated with their mailbox. This user only needed two devices (iPhone and iPad). I asked what version iOS they were running on the devices. The iPad was running iOS 8.3, but the user said they had just updated their iPhone to iOS 8.4.1 over the weekend (when I started noticing the problem with the logs). So we removed the Exchange mailbox from their iPhone and re-enabled Exchange ActiveSync on their mailbox. Sent a few test emails to and from their iPad. No rapid transaction log growth. We re-added their Exchange mailbox to their iPhone and sent a few more tests and everything looked fine.
I thought maybe the iOS 8.4.1 upgrade hung something on the iPhone and that by removing the mailbox and re-adding it was the solution. So I told user2 to do the same thing on their iPhone. We removed the Exchange mailbox and re-enabled Exchange ActiveSync. All looked good. So I told them to go ahead and add the mailbox back to their iPhone. Almost instantly the problem showed up again. I did a little more digging and discovered that user2 had installed the beta version of iOS9 on their iPhone and that they were using the Microsoft Outlook app from the AppStore (not the built-in Mail, Contacts, Calendar settings on the iPhone). So we removed the Exchange mailbox again and disabled Exchange ActiveSync (just removing the mailbox didn’t stop the log growth). We re-enabled Exchange ActiveSync after the logs slowed down and I had the user add the mailbox back in using Mail, Contacts, and Calendar. Once that was done they were able to send and receive emails and not cause the transaction logs to grow. So there is most likely a bug between the beta of iOS9 and the Microsoft Outlook app.
Once all of this was done I decided to go through and run the usage statistics command against our other databases, and I found one other user that had upgraded to iOS 8.4.1 on their iPhone and had high usage. So we removed the mailbox and re-added it and the problem has been resolved. This got me a little concerned that the upgrade process wasn’t supper clean. Microsoft gave me a script to see what version of iOS we had in our environment so I changed it a little to only show iOS 8.4.x versions, and found out that we have a number of users that have upgraded their devices. So I think this has just been an isolated upgrade problem for a couple of our users only.
Eric
You legend! This helped immensely, in my case it was migrations that had “failed” due to lack of log space on the destination database which then resumed in the background once the log space issue was resolved.
It was showing that mailboxes which had “failed” were generating the majority of the log writes. Once I resumed the migration batch in the ECP everything made sense and it all calmed down when the migrations finally completed properly.
Howdy I am so grateful I found your web site, I really found you by error, while I was researching on Google for something else, Anyways I am here now and would just like to say many thanks for a tremendous post and a all round exciting blog (I also love the theme/design), I don’t have time to browse it all at the moment but I have book-marked it and also included your RSS feeds, so when I have time I will be back to read more, Please do keep up the excellent work.