An issue at our NSX vSAN storage resulted in a corrupted state. Here are some steps you can take to attempt to revive the NSX-T Manager appliance back to life. BTW these steps also work for Edge Nodes.
The issue starts with the appliance having file system in read only mode. After reboot you will see a message:UNEXPECTED INCONSISTENCY: RUN fsck Manually
The first step is to go into appliance GRUB menu that appears briefly after start up, hit e key, enter root/VMware1 GRUB credentials (these are different from the regular credentials) and edit the line with starting with linux and replace ro with rw and delete the rest of the line.
Continue with the boot process by pressing Ctrl+x. Hopefully now you are able to get into BusyBox shell and run fsck /dev/sda2 or similar to fix the corrupted partition. Reboot.
What can happen now is that the appliance will boot but again will find LVM corruption and will go into emergency mode and you can see repeated Login incorrect messages.
Repeat the process with the GRUB edit. This time you will be asked to enter root password to go into maintenance mode.
Type the root password
- Run the fsck /dev/mapper/nsx-tmp command.
- Press “Yes” to fix all errors encountered by the file system.
- Run the “Reboot” command. The NSX-T Edge should now boot into default mode.