Cluster Manager, Server Manager & Hyper-V Console not starting

This week I had a very strange issue with a Hyper-V Cluster managed by Virtual Machine Manager.

Completely randomly different cluster nodes failed and I weren’t able to start failover cluster manager on one of the cluster nodes.  On the infected node it self, I wasn’t able to open the hyper-v manager or server manager.

After a lot of research I found a solution from the windows server core team which pointed me to the solution.

Unable to launch Cluster Failover Manager on any node of a 2012/2012R2 Cluster

When Failover Cluster Manager is opened to manage a Cluster, it will contact all the nodes and retrieve Cluster configuration information using WMI calls. If any one of the nodes in the Cluster does not have the cluster namespace “root\mscluster” in WMI, Failover Cluster Manager will fail and give one of the below errors:

clip_image002

Or,

Unfortunately, it does not give any indication of which node is missing the WMI namespace.  One of the ways you can check to see which one has it missing is to run the below command on each node of the Cluster.

It can be a bit tedious and time consuming if you have quite a few nodes, say like 64 of them.  The below script can be run on one of the nodes that will connect to all the other nodes and check to see if the namespace is present.  If it is, it will succeed.  If the namespace does not exist, it will fail.

—————–

 
—————–

In the below example, you can see that one of the nodes failed.

To correct the problem, you would need to run the below from an administrative command prompt on the “failed” node(s).

cd c:\windows\system32\wbem
mofcomp.exe cluswmi.mof

Once the Cluster WMI has been added back, you can successfully open Failover Cluster Management.  There is no restart of the machine or the Cluster Service needed.

Quote: Microsoft Ask the Core Team Blog

In my case I wasn’t able to fix it so easy because the server vendor implemented the WMI Provider directly in his BMC via Agent (for the interested ones Fujitsu). during the process of recompiling the WMI for the Cluster the whole Server Network interfaces and BMC fail.

so my fix:

  1. shutdown the server
  2. make it powerless
  3. start it
  4. check cluster (everything fine)
  5. uninstall the (fucking) agent

Since than it worked.

How to fix VMM error 20553

Hi,

today I want you to provide you some GPO Templates, which could help you to fix following error in VMM:

Error (20553)
The Windows Remote Management (WinRM) client on the VMM server cannot process the request. A computer policy does not allow the delegation of the user credentials to the target computer **.

WinRM: URL: ** , Verb: [ENUMERATE], Resource: [http://schemas.microsoft.com/wbem/wsman/1/wmi/root/cimv2/Win32_ComputerSystemProduct], Filter: []

Unknown error (0x803381a3)

Recommended Action
Use gpedit.msc and look at the following policy: Computer Configuration -> Administrative Templates -> System -> Credentials Delegation -> Allow Delegating Fresh Credentials. Verify that it is enabled and configured with an SPN appropriate for the target computer. For example, for a target computer name myserver.domain.com, the SPN can be one of the following: WSMAN/myserver.domain.com OR WSMAN/*.domain.com OR WSMAN/*

Thanks to Radhika Gupta for his blog on TechNet which gave me the final solution 🙂

In my case I needed to create two GPOs.

The first deployed on the Hyper-V Hosts to enable WinRM with CreedSSP

Computer Configuration\Administrative template\Windows Components\Windows Remote Management (WinRM)\WinRM Service\[Allow CredSSP authentication] = true

The first deployed on the VMM Hosts to enable WinRM with CreedSSP and Credentials Delegation

Computer Configuration\Administrative template\Windows Components\Windows Remote Management (WinRM)\WinRM Service\[Allow CredSSP authentication] = true

Computer Configuration\Administrative Templates\System\Credentials Delegation\[AllowFreshCredentials ] = “WSMAN/*”

How to get notified and react when a cluster NIC fails

Hi everybody,

today a short day to day helper regarding cluster networks.

Last weekend I had a question from my coworker, if it is possible to react on failed networkconnections within a cluster even if the cluster is still running.

As you know Hyper-V is running as long as the cluster heartbeat is available and the storage is reachable. Only if these connections fail, Hyper-V Cluster forces a node to failover his roles.

My answer, yes it is possible with some tricks. First thing you need to know is, that if a network connection in the cluster failes, we get an error event.

Event ID 1129 — Cluster Network Connectivity Partitioned https://technet.microsoft.com/en-us/library/dd353962%28v=ws.10%29.aspx?f=255&MSPPError=-214721739

This event is also noticed in the event viewer of your Hyper-V Host. Now you can attach a task to that event and for example run a script which forces a failover.

02-06-_2015_09-17-51 02-06-_2015_09-18-35 02-06-_2015_09-19-07

 

The script to set the node in maintenance  mode and force a failover could look like:

You can achieve the same effect when you use System Center Operations Manager by configuring an Agent Task for your Hyper-V Host.

Find more info here: https://technet.microsoft.com/en-us/library/hh563486.aspx

 

Error /Faile Hyper-V Host has an unsupported VMM Agent installed

When you update your System Center Virtual Machine Environment, you maybe face the issue that your Agent version is outdated and you are not able to update it.

VMM01

The easiest way is to logon to the host and uninstall the old agent.

VMM02

After you uninstalled it, run the task in VMM again.

VMM03

 

Afterwards it should be fine.

 

Error “Could not udpate managed code add-in pipeline” when starting system center virtual machine manager

Some of you maybe face the issue below when installing the VMM management console on a client.

VMM-Log-Error01

The reason for that issue is, that you user has no read/write permission on the folder.

First step to fix the issue navigate to the folder. In my case C:\Program Files\Microsoft System Center 2012 R2\Virtual Machine Manager\bin but it depends on where you installed the VMM.

VMM-Log-Error02

Than you need to change the permission to full controll for your your user or user group (I use my VMM Admins Group). That fixes the issue.

VMM-Log-Error03 VMM-Log-Error04