ESXi 6.5 corrupted/full store

Hi, recently I have met situation, that I couldn’t update VMware tools, becuase there was no space on disk where the /store data are.

I have first check disk sizes

[root@host1:~] df -h

Filesystem Size Used Available Use% Mounted on
VMFS-5 2.0T 877.0G 1.1T 43% /vmfs/volumes/VMDATA01
vfat 249.7M 157.9M 91.9M 63% /vmfs/volumes/9d8fb6ea-f57794d7-5503-3f393eed9f13
vfat 249.7M 160.6M 89.2M 64% /vmfs/volumes/022aaeee-7333de8d-e822-257b899196cc
vfat 285.8M 285.8M 0.0M 100% /vmfs/volumes/5a3fa82d-b078b8da-e3aa-28f10ed2bc11

We can see, that last disk has 0MB free space. So I checked content of it.

cd /store
ls -la

From picture you can see, that folder containing a lot of „corrupted“ files and folders. I have tried to remove some of the files or folders, but I always fail. So I have decided to format the entire volume. Be aware, that if you format entire volume, you will have to copy content of packages folder from another host

First of all, you need to locate the volume which is pointing to /store. This part was quite tricky, but I have finally found, that it is this one: /vmfs/devices/disks/mpx.vmhba32:C0:T0:L0:8

You can use this KB article from Vmware to help you identify volume – https://kb.vmware.com/s/article/1014953

Attaching also picture of default disk layout.

Now you need to put host to maintenance mode (I have tried it without maitenance mode, and the command fail). Once the host will be in maintenance, run following command:

vmkfstools -C vfat -S <new label> /vmfs/devices/disks/<your disk>

root@host1:~] vmkfstools -C vfat -S BasicData /vmfs/devices/disks/mpx.vmhba32:C0:T0:L0:8
create fs deviceName:’/vmfs/devices/disks/mpx.vmhba32:C0:T0:L0:8′, fsShortName:‘ vfat‘, fsName:’BasicData‘ deviceFullPath:/dev/disks/mpx.vmhba32:C0:T0:L0:8 deviceFile:mpx.vmhba32:C0:T0:L0 :8
Checking if remote hosts are using this device as a valid file system. This may take a few seconds…
Creating vfat file system on „mpx.vmhba32:C0:T0:L0:8“ with blockSize 1048576 and volume label „BasicData“.
Filesystem was created but mount failed on device „mpx.vmhba32:C0:T0:L0:8“.: Not found.
Successfully created new volume: 5c617aee-fcbcf6fd-c08a-28f10ed2bc11

Rememebr the volume id

Now we have to recreate link to /store. First we remove the old link and then we create new link.

[root@host1:~] rm store
[root@host1:~] ln -s /vmfs/volumes/5c617aee-fcbcf6fd-c08a-28f10ed2bc11 store

Finally we have to copy content of packages folder to /store from working host. For that you can use WinScp if you are not familiar with shell.

Napsat komentář

Vaše emailová adresa nebude zveřejněna. Vyžadované informace jsou označeny *