{"id":158,"date":"2019-02-11T16:48:32","date_gmt":"2019-02-11T16:48:32","guid":{"rendered":"http:\/\/www.djblond.cz\/?p=158"},"modified":"2019-02-12T14:37:12","modified_gmt":"2019-02-12T14:37:12","slug":"esxi-6-5-corrupted-full-store","status":"publish","type":"post","link":"https:\/\/www.djblond.cz\/index.php\/2019\/02\/11\/esxi-6-5-corrupted-full-store\/","title":{"rendered":"ESXi 6.5 corrupted\/full store"},"content":{"rendered":"<p>Hi, recently I have met situation, that I couldn&#8217;t update VMware tools, becuase there was no space on disk where the \/store data are.<\/p>\n<p>I have first check disk sizes<\/p>\n<pre class=\"brush: plain; title: ; notranslate\" title=\"\">[root@host1:~] df -h<\/pre>\n<p>Filesystem Size Used Available Use% Mounted on<br \/>\nVMFS-5&nbsp;2.0T 877.0G 1.1T 43% \/vmfs\/volumes\/VMDATA01<br \/>\nvfat 249.7M 157.9M 91.9M 63% \/vmfs\/volumes\/9d8fb6ea-f57794d7-5503-3f393eed9f13<br \/>\nvfat 249.7M 160.6M 89.2M 64% \/vmfs\/volumes\/022aaeee-7333de8d-e822-257b899196cc<br \/>\nvfat 285.8M 285.8M 0.0M 100% <strong><span style=\"color: #ff0000;\">\/vmfs\/volumes\/5a3fa82d-b078b8da-e3aa-28f10ed2bc11<\/span><\/strong><\/p>\n<p><span style=\"color: #000000;\">We can see, that last disk has 0MB free space. So I checked content of it.<\/span><\/p>\n<pre class=\"brush: plain; title: ; notranslate\" title=\"\">cd \/store\r\nls -la<\/pre>\n<p><img loading=\"lazy\" class=\"alignnone wp-image-159\" src=\"http:\/\/www.djblond.cz\/wp-content\/uploads\/2019\/02\/corupted_store-277x300.jpg\" alt=\"\" width=\"433\" height=\"469\" srcset=\"https:\/\/www.djblond.cz\/wp-content\/uploads\/2019\/02\/corupted_store-277x300.jpg 277w, https:\/\/www.djblond.cz\/wp-content\/uploads\/2019\/02\/corupted_store.jpg 572w\" sizes=\"(max-width: 433px) 100vw, 433px\" \/><\/p>\n<p>From picture you can see, that folder containing a lot of &#8222;corrupted&#8220; files and folders. I have tried to remove some of the files or folders, but I always fail. So I have decided to format the entire volume. <span style=\"color: #ff0000;\"><strong>Be aware, that if you format entire volume, you will have to copy content of packages folder from another host<\/strong><\/span><\/p>\n<p>First of all, you need to locate the volume which is pointing to \/store. This part was quite tricky, but I have finally found, that it is this one:\u00a0<strong>\/vmfs\/devices\/disks\/mpx.vmhba32:C0:T0:L0:8<\/strong><\/p>\n<p>You can use this KB article from Vmware to help you identify volume &#8211;\u00a0<a href=\"https:\/\/kb.vmware.com\/s\/article\/1014953\">https:\/\/kb.vmware.com\/s\/article\/1014953<\/a><\/p>\n<p>Attaching also picture of default disk layout.<\/p>\n<p><img loading=\"lazy\" class=\"alignnone wp-image-161\" src=\"http:\/\/www.djblond.cz\/wp-content\/uploads\/2019\/02\/Partition_diagram-300x153.png\" alt=\"\" width=\"437\" height=\"223\" srcset=\"https:\/\/www.djblond.cz\/wp-content\/uploads\/2019\/02\/Partition_diagram-300x153.png 300w, https:\/\/www.djblond.cz\/wp-content\/uploads\/2019\/02\/Partition_diagram.png 535w\" sizes=\"(max-width: 437px) 100vw, 437px\" \/><\/p>\n<p>Now you need to put host to maintenance mode (I have tried it without maitenance mode, and the command fail). Once the host will be in maintenance, run following command:<\/p>\n<pre class=\"brush: plain; title: ; notranslate\" title=\"\">vmkfstools -C vfat -S &lt;new label&gt; \/vmfs\/devices\/disks\/&lt;your disk&gt;<\/pre>\n<p>root@host1:~] vmkfstools -C vfat -S BasicData \/vmfs\/devices\/disks\/mpx.vmhba32:C0:T0:L0:8<br \/>\ncreate fs deviceName:&#8217;\/vmfs\/devices\/disks\/mpx.vmhba32:C0:T0:L0:8&#8242;, fsShortName:&#8216; vfat&#8216;, fsName:&#8217;BasicData&#8216; deviceFullPath:\/dev\/disks\/mpx.vmhba32:C0:T0:L0:8 deviceFile:mpx.vmhba32:C0:T0:L0 :8<br \/>\nChecking if remote hosts are using this device as a valid file system. This may take a few seconds&#8230;<br \/>\nCreating vfat file system on &#8222;mpx.vmhba32:C0:T0:L0:8&#8220; with blockSize 1048576 and volume label &#8222;BasicData&#8220;.<br \/>\nFilesystem was created but mount failed on device &#8222;mpx.vmhba32:C0:T0:L0:8&#8220;.: Not found.<br \/>\n<strong>Successfully created new volume: 5c617aee-fcbcf6fd-c08a-28f10ed2bc11<\/strong><\/p>\n<p><span style=\"color: #ff0000;\">Rememebr the volume id<\/span><\/p>\n<p>Now we have to recreate link to \/store. First we remove the old link and then we create new link.<\/p>\n<pre class=\"brush: plain; title: ; notranslate\" title=\"\">[root@host1:~] rm store\r\n[root@host1:~] ln -s \/vmfs\/volumes\/5c617aee-fcbcf6fd-c08a-28f10ed2bc11 store<\/pre>\n<p>Finally we have to copy content of packages folder to \/store from working host. For that you can use WinScp if you are not familiar with shell.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>Hi, recently I have met situation, that I couldn&#8217;t update VMware tools, becuase there was no space on disk where the \/store data are. I have first check disk sizes&#8230; <a href=\"https:\/\/www.djblond.cz\/index.php\/2019\/02\/11\/esxi-6-5-corrupted-full-store\/\">Read more &raquo;<\/a><\/p>\n","protected":false},"author":1,"featured_media":174,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":[],"categories":[12],"tags":[],"_links":{"self":[{"href":"https:\/\/www.djblond.cz\/index.php\/wp-json\/wp\/v2\/posts\/158"}],"collection":[{"href":"https:\/\/www.djblond.cz\/index.php\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.djblond.cz\/index.php\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.djblond.cz\/index.php\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.djblond.cz\/index.php\/wp-json\/wp\/v2\/comments?post=158"}],"version-history":[{"count":16,"href":"https:\/\/www.djblond.cz\/index.php\/wp-json\/wp\/v2\/posts\/158\/revisions"}],"predecessor-version":[{"id":192,"href":"https:\/\/www.djblond.cz\/index.php\/wp-json\/wp\/v2\/posts\/158\/revisions\/192"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.djblond.cz\/index.php\/wp-json\/wp\/v2\/media\/174"}],"wp:attachment":[{"href":"https:\/\/www.djblond.cz\/index.php\/wp-json\/wp\/v2\/media?parent=158"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.djblond.cz\/index.php\/wp-json\/wp\/v2\/categories?post=158"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.djblond.cz\/index.php\/wp-json\/wp\/v2\/tags?post=158"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}