SCOM – Exchange Client Proxy Error

I was implementing System Center Operations Manager in a project at a customer for monitoring Exchange.
After loading the Exchange Management Pack, a bunch of errors like these were logged:

The Ehlo options for the client proxy target 10.2.38.164 did not match while setting up proxy for user on inbound session 08D2F00F25C773B2. The critical non-matching options were <maxSize>. The non-critical non-matching options were <NONE>. Client proxying will continue.

The Ehlo options for the client proxy target did not match while setting up proxy for user on inbound session 08D2F00F25C773B2. The critical non-matching options were . The non-critical non-matching options were . Client proxying will continue.

This error is logged for servers holding the mailbox role, even though it states Hub Transport in the source (that is no longer a standalone role in Exchange 2013).

 

So, looking at the alert description we can see it somehow relates to max size, and is generated in response to the EHLO options when a test connection is made to the server in question.
To dig into this, we need to determine the what the maximum message size defined in the Exchange organization is. This can be done using an Exchange Management Shell with this command:

Get-TransportConfig | ft maxsendsize, maxreceivesize

This lists the sizes configured for the Exchange organization.

Next, we need to determine which receive connector is configured with the different limit. This can be done by either looking at the receive connectors on the server listed in the error or running a command to list all receive connectors in the organization and correct those who deviate:

Get-ReceiveConnector | ft name, maxmessagesize

Examine the output, find the receive connector in question and correct its message size restriction. Or use this command to find all receive connectors with the deviating limit and correct them at the same time. Change the MB limit to fit your need.

Get-ReceiveConnector | Where-Object {$_.maxmessagesize -ne 35MB} | set-receiveconnector -maxmessagesize 35MB

Restart the Exchange Transport Service and voila, the error will go away.

Excel 2016 crashes after KB3118373

Microsoft has acknowledged after releasing the October 4, 2016 update for Excel 2016 that it may cause Excel to crash, according to customer feedback.

The solution to this problem is simply to remove the update in question, as per this KB article: Error message “Microsoft Excel has stopped working” after installing KB3118373

Fixes for this problem, as well as the other fixes included in the October 4 update will be included in a later update by Microsoft.

DirectAccess Configuration Load Error

I was implementing a DirectAccess solution at a customer using Windows 2012 R2 and Windows 8.1

Everything went fine with installation and configuration and I was looking forward to testing it. But after everything was done I opened the Remote Access Management Console only to be met by an error stating “Configuration Load Error. Settings for server <servername> cannot be retrieved. The system cannot find the file specified.” as seen below:
DirectAccess post install error 2Re-applying the settings from the DirectAccess server GPO did not resolve the issue and since I was unable to find any documentation stating which files are being used in the DirectAccess configuration I decided to start again.

I therefore deleted the GPO’s, and ran a GPUPDATE on the server. The wizard then detected that the GPO settings was no longer present and gave me the option of clearing the configuration on the server.

After this I was able to run the configuration wizard once more and this time it worked…

Un-installing Exchange 2013 server fails with “Setup can’t continue with the uninstall because the eseutil has open files”

So, I was at a customer uninstalling an Exchange 2013 server as the customer had finished their migration to Office 365.

Went through the usual process of decommissioning the server, such as ensuring no mailboxes where left, removing arbitration boxes and so on.
Got to the point where the actual uninstall could take place, and was faced with this:

E2K13 eseutil uninstall error

Thought to myself, that was kinda weird… Looking in Task Manager showed that ESEUTIL has a lot of processes running:

E2K13 eseutil task manager

Well, since all data had been migrated away from the server and all databases had been removed I didn’t see much point in finding out why all these ESEUTIL processes where spawned.

A simple reboot later, and the uninstall procedure continued without any problems.

Virtualizing domain controllers used to be a big no-no…

So, I’ve been away from blogging for a while. Work has kept me busy during the day and my little daughter, which came last year, did the same at night 🙂

But for this post I’d like to tackle the issue of virtualizing a domain controller.

In the past this could cause some rather unforeseen and dramatic consequences if one was not aware of how to do this properly. Most businesses I’ve visited over the years since virtualization began to ramp of have to a large degree virtualized all servers possible and what are better candidates than domain controllers which most of the time doesn’t use the ressources allocated.
Most of these have also adapted their backup solutions to meet these new scenarios where one of the most seen solutions is based on some form of virtual machine snapshotting.
Well, that works okay for a file server, print server or other applications that are crash-consistent. But what happens if you restore a domain controller from a snapshot?

Let’s take some background information on how domain controllers work at first. They, as we all know, run Active Directory and each one of them holds a copy of the Active Directory database. This database is assigned a value known as an InvocationID and each update (that does not require the involvement of a FSMO role master) is done against the local Active Directory database, which is then synced with its replication partners. This is done using Update Sequence Numbers (USN). These two numbers form a unique identifier on the local database and are used to determine whether updates need to be processed.

Now, take the scenario from before where a domain controller is restored from a snapshot. What happens then?
Well, the domain controller that comes up from a restored state doesn’t know this so it will attempt to sync with its partners. But then comes the problems as the USN it tries to use are already acknowledged by the other domain controllers as used and therefor no sync will happen.
And as stated above, as changes happens towards the local Active Directory database then over the course of time they will become more and more out of sync. An example (in the light end is a user changing his/her password on the restored domain controller, which doesn’t sync to the others. You then have one password on some systems and another on others, depending on which DC they validate against).

This is really not a desired scenario, but thankfully Microsoft has done something about this…

In Windows Server 2012 Microsoft made the domain controller virtualization aware (on supported hypervisors that is). This means that the domain controller now knows it is virtual and can take steps to prevent the above described scenario.
So how is this done…

Well, Microsoft introduced a new attribute to Active Directory, more specifically the VmGenerationID which is a number stored on the domain controller object in Active Directory.
This attribute is then monitored by the Windows operating system to ensure it matches the number the server has stored locally. If this is not the case, then the domain controller assumes it has been restored and take actions the prevent the above scenario.
What happens in this case is that the domain controller resets its InvocationID and discards the issued RID pool effectively preventing the re-use of USN. It then marks its local SYSVOL share for a non-authoritive restore.

This picture shows how the proces is done:

Virtualization Safeguards During Normal Boot

The requirements for using this cool new feature is the following:

  • Supported hypervisor platform (Hyper-V 2012 or newer, vSphere 5.02, ESX 5.1)
  • Windows Server 2012 domain controller
  • PDC emulator running Windows Server 2012

 

Debunking the myths of why Vmware is better than Hyper-V – Clustered File System…

Today I’m gonna write about something called Cluster Shared Volumes (CSV) and as usual, the stories that are told by Vmware.

When I talk to Vmware customers these days, one of the stories I hear (out of many) is that Hyper-V does not have a clustered file system that makes it possible to share storage between cluster nodes.
Looking at Vmware’s web site, I can see where this story originates from as they claim Hyper-V has no clustered file system.

Well, that is simply not true. Hyper-V has for many years now (since the release of Windows 2008 R2) had a feature called Cluster Shared Volumes which makes it possible to share the same storage across multiple cluster nodes and thereby also making live migration possible.

But how does CSV work then?

Well, as said CSV makes the same storage available to all nodes in the cluster, as shown in the picture below.

CSV

It does this by using Failover Clustering in Windows Server to make the storage accesible to all nodes. It does this by using something called a CSV coordinator node, which coordinates some types of transactions to the storage to ensure data consistency.
This coordinator role is completely interchangeable within the cluster and can be transitioned from one node to another as you please.

But CSV does offer more than sharing the same storage between nodes, it also offers fault tolerance. Let’s say that in the above scenario, the link to the storage dies for one of the hosts (for example, someone unplugs a fibre cable, the SAN admins makes an error in zoning or so on). Normally, one would assume that this would cause the virtual machines on that hosts to die as well and be failed over to other hosts causing a loss of service (which would be the case in Vmware for example).
Well, if you’re using CSV then it is a bit different as the node will just begin to ship disk I/O over the network to the CSV coordinator node as shown in the picture below.

Failed CSV

This would make it possible for Hyper-V admins to live migrate these machines to other fully working nodes without causing a loss of service the the end users.

Another cool feature of CSV is the ability to allocate a portion of the physical memory on the cluster node to read cache, thereby saving frequently accessed data in RAM (which off course is way way faster than disk) and thereby increasing performance for both the server requesting the data as well as the other servers by sparing the disk subsystem for the many read I/O’s.

For more reading, see this technet article: http://technet.microsoft.com/en-us/library/jj612868.aspx

Debunking the myths of why Vmware is better than Hyper-V – Performance

With this post, I’m continuing my series on the myths surrounding Hyper-V that I encounter in my workdays. Previously, I’ve written about Transparent Page Sharing, Memory overcommit and disk footprint.

Today I’m gonna write about another major issue I hear from customers, namely performance, and one that is completely understandable. Everyone can relate to wanting the most bang for your buck.
When looking at the Vmware sales pitch on their website (and one that is repeated time and again from their sales reps) it sounds like Vmware is the best option to buy:

VMware vSphere—the industry’s first x86 “bare-metal” hypervisor—is the most reliable and robust hypervisor. Launched in 2001 and now in its fifth generation, VMware vSphere has been production-proven in tens of thousands of customer deployments all over the world.

Other hypervisors are less mature, unproven in a wide cross-section of production datacenters, and lacking core capabilities needed to deliver the reliability, scalability, and performance that customers require.

You can clearly see from this text that Vmware believes their hypervisor to be superior in any way, including performance. But is this true?

Well, I can’t answer that and neither can most of the other people out there. Why is that?

Let’s snip a bit more from Vmware’s material, this time their EULA:

2.4 Benchmarking.You may use the Software to conduct internal performance testing and benchmarking studies. You may only publish or otherwise distribute the results of such studies to third parties as follows: (a) if with respect to VMware’s Workstation or Fusion products, only if You provide a copy of Your study to benchmark@vmware.comprior to distribution; (b) if with respect to any other Software, only if VMware has reviewed and approved of the methodology, assumptions and other parameters of the study (please contact VMware at benchmark@vmware.comto request such review and approval) prior to such publication and distribution.
As you can see, this clearly states that if you wish to do a benchmark pitching Hyper-V against Vmware, then you need Vmware’s approval for both doing the test, how you do it and afterwards for the results.
Wanna make a guess that a bad test for Vmware’s isn’t going to be approved?
So can you trust a vendor that claims superior performance, but is unwilling to allow tests where the results are approved by the said vendor… That’s up to you to decide…