Posted in: VMware

Cluster Tasks fails with “namespace” error

You may experience issues where cluster task specially like putting the host in maintenance mode, deleting a cluster etc will fail with below error:

2020-12-21T12:43:56.848-08:00 info vpxd[10034] [Originator@6876 sub=MoHost opID=opId-18b14-105289-d9] WCP exitMaintenanceMode vAPI returns error: Error:
-->    com.vmware.vapi.std.errors.unauthenticated
--> Messages:
-->    vapi.security.authentication.invalid<Unable to authenticate user>
-->  
2020-12-21T12:43:56.851-08:00 error vpxd[10034] [Originator@6876 sub=MoHost opID=opId-18b14-105289-d9] [Delete] Failed to delete vAPI session. Error:
--> Error:
-->    com.vmware.vapi.std.errors.unauthenticated
--> Messages:
-->    vapi.security.authentication.invalid<Unable to authenticate user>
..
..
..
2020-12-21T12:43:56.860-08:00 info vpxd[10034] [Originator@6876 sub=Default opID=opId-18b14-105289-d9] [VpxLRO] -- ERROR task-6215 -- host-9421 -- vim.HostSystem.enterMaintenanceMode: vim.fault.InvalidState:
--> Result:
--> (vim.fault.InvalidState) {
-->    faultCause = (vmodl.MethodFault) null, 
-->    faultMessage = (vmodl.LocalizableMessage) [
-->       (vmodl.LocalizableMessage) {
-->          key = "com.vmware.cdrs.maintenancemode.wcp.entermaintenancemode", 
-->          arg = <unset>, 
-->          messa

Step 1: SSH to the vCenter in question where the WCP service needs to be repaired.

Step 2: Get the unique Machine ID and hostname:

/usr/lib/vmware-vmafd/bin/vmafd-cli get-machine-id --server-name localhost

Hostname

hostname -f

Step 3: Create a solution user configuration file for the WCP service:

vi /tmp/wcp.cfg

Step 4: Enter the following contents into the file:

Country = US
Name = wcp- <machine_id>
Organization = VMware Engineering
OrgUnit = mID- <machine_id>
State = CA
Locality = Palo Alto
IPAddress = 127.0.0.1
Email = acme@vmware.com
Hostname = <VC_FQDN>

Step 5: Generate WCP solution user key:

/usr/lib/vmware-vmca/bin/certool --server localhost --genkey --privkey=/tmp/wcp.key --pubkey=/tmp/wcp.pub

Step 6: Generate WCP solution user certificate:

/usr/lib/vmware-vmca/bin/certool --server=localhost  --gencert --privkey=/tmp/wcp.key --config=/tmp/wcp.cfg --cert=/tmp/wcp.crt

Step 7: Get WCP service name using dir-cli:

[default name - wcp-<machine id>]/usr/lib/vmware-vmafd/bin/dir-cli service list

Step 8: Update the WCP service with the new WCP certificate:

/usr/lib/vmware-vmafd/bin/dir-cli service update --name <insert wcp service name from the service list> --cert /tmp/wcp.crt

Step 9: Delete the WCP solution user entry from VECS store:

/usr/lib/vmware-vmafd/bin/vecs-cli entry delete --store wcp --alias wcp -y

Do a refresh

/usr/lib/vmware-vmafd/bin/vecs-cli force-refresh

Step 10: Update the new WCP solution user certificate to VECS store:

/usr/lib/vmware-vmafd/bin/vecs-cli entry create --store wcp --alias wcp --cert  /tmp/wcp.crt --key /tmp/wcp.key

Step 11: Verify that the WCP certificate is updated – The Subject should contain unique CN as updated in wcp.cfg, as well as a new Issue and Expiration date:

/usr/lib/vmware-vmafd/bin/vecs-cli entry getcert --store wcp --alias wcp --text

Step 12: Restart services on the vCenter:

service-control --stop --all && service-control --start --all

Finally: Re-try whatever previous workflow was failing due to WCP errors.

Back to Top