vSAN Troubleshooting

vSAN iSCSI targets configuration deep dive

Hareesh K G

— 9 min read
Share
vSAN iSCSI targets configuration deep dive

I have been asked about vSAN iSCSI target configuration and  how the objects looks within vSAN ? Here is a small guide that gets you going with vSAN iSCSI configuration and how you get to see the actual luns within vSAN as vSAN objects  .

The iSCSI lun creation and presentation is very simple ,  I have put together screenshots from my lab to create and export vSAN iSCSI luns to a Windows server 2012 box . Please follow the series of screenshots to create and setup iSCSI exports from vSAN to external physical machines . In the below example we do not see creation of initiator group , usually initiator groups are created to allow a set of Luns to a certain set of IQN to which the luns must be visible .

Note: iSCSI Lun exports to virtual machines is unsupported officially , strictly refrain from presenting an iSCSI vSAN export to a VM in production environment .*

Looking at the objects for the iSCSI Lun export thru vSAN.

VIT (virtual iSCSI Target) is the service which runs on all the host is responsible to manage and facilitate the vSAN iSCSI target on all the hosts , I suggest you to see VMwareKB to troubleshoot “iSCSI target service”

After creating first iSCSI target when we look at Target owner host we see below messages , this indicates that the object for the target iSCSI lun was created successfully , if there were any issues while creating the iSCSI target from vcenter server , please see the vmkernel.log on the owner host for more details

Vmkerne.log : “less /var/log/vmkernel.log | grep -i vit” : this the logs seen during the time of creating a target .

2018-01-14T03:38:39.296Z cpu16:9176868 opID=a09e5a8d)World: 12235: VC opID VSAN-iSCSI-OP-52e79e76-7274-e36b-daa3-c5a03437e5b2 maps to vmkernel opID a09e5a8d 2018-01-14T03:38:44.440Z cpu17:9176879)Loading module vit ... 2018-01-14T03:38:44.441Z cpu17:9176879)Elf: 2043: module vit has license VMware 2018-01-14T03:38:44.450Z cpu17:9176879)VIT: Resetting config options 2018-01-14T03:38:44.450Z cpu17:9176879)vit: HeapLogSizes:497: Size of global heap 0x4307450f5000 : 8681472 bytes 2018-01-14T03:38:44.450Z cpu17:9176879)vit: HeapLogSizes:498: Size of target heap: 952320 bytes 2018-01-14T03:38:44.450Z cpu17:9176879)vit: HeapLogSizes:499: Size of LUN heap: 4601856 bytes 2018-01-14T03:38:44.450Z cpu17:9176879)vit: HeapLogSizes:500: Size of ctl_softc 12256 bytes. 2018-01-14T03:38:44.450Z cpu17:9176879)vit: HeapLogSizes:501: Size of cfiscsi_target 928 bytes. 2018-01-14T03:38:44.450Z cpu17:9176879)vit: HeapLogSizes:502: Size of ctl_lun 118584 bytes. 2018-01-14T03:38:44.453Z cpu17:9176879)vit: VitFssBeInit:543: Number of threads for IO task queue: 32 2018-01-14T03:38:44.457Z cpu17:9176879)Mod: 4968: Initialization of vit succeeded with module ID 119. 2018-01-14T03:38:44.457Z cpu17:9176879)vit loaded successfully. 2018-01-14T03:40:36.410Z cpu18:9177736 opID=40c602bf)World: 12235: VC opID VSAN-iSCSI-OP-52b07c6f-192f-031e-1c15-011af0ab1edf maps to vmkernel opID 40c602bf 2018-01-14T03:40:38.580Z cpu1:9177019)vit: cfiscsi_target_find_or_create:2542: Created heap RDP-win for target RDP-win. Heap pointer: 0x4305e6412000. 2018-01-14T03:40:42.078Z cpu22:9177018)vit: VitFSSBEIoctl_Create:2207: LUN heap cfss0 created (heap ID: 0x430750e13000). 2018-01-14T03:40:42.078Z cpu22:9177018)vit: VitFSSBE_GetInfo:1927: GetInfo size 10737418240 bs=512 size_blocks 20971520 2018-01-14T03:42:53.006Z cpu13:68816)vit: VITVsiGetIoStats:14820: Throttled: Perf stats isn't initialized for ctl LUN 0 2018-01-14T03:42:53.006Z cpu13:68816)vit: VITVsiGetBwStats:14711: Throttled: Perf stats isn't initialized for ctl LUN 0 2018-01-14T03:42:53.006Z cpu13:68816)vit: VITVsiGetLatencyStats:14939: Throttled: Perf stats isn't initialized for ctl LUN 0

Using certain esxcli vsan commands to get more information about the iSCSI targets

***localcli vsan iscsi status get
****Status: Enabled: true****

localcli vsan iscsi target list***
Alias       iSCSI Qualified Name (IQN)                          Interface Port Authentication type LUNs Is Compliant UUID I/O Owner UUID
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
**<span style="color: #0000ff;">RDP-win</span>**<span style="color: #0000ff;">*iqn.1998-01.com.vmware:22455fef-6b44-e32b-eb2a-f5535af9a5de*</span> vmk1 3260 No-Authentication 1 true **<span style="color: #0000ff;">34d15a5a-7ff7-924b-776c-ecf4bbec91a8</span>** 5937c663-8cb8-3d48-d3ad-ecf4bbec91a8

**<span style="color: #993300;">Win-iSCSI-Lun2</span>***<span style="color: #993300;">iqn.1998-01.com.vmware:0f4751c1-2a44-bb90-83e4-f567cfc711d4</span>* vmk1 3260 No-Authentication 1 true **<span style="color: #993300;">62d65a5a-48dd-477d-8c86-ecf4bbec65d8</span>** 5938de9a-e35b-d745-c9ff-ecf4bbec65d8

***localcli vsan iscsi homeobject get
***
Namespace Information:
 UUID:**<span style="color: #008000;"> bfd05a5a-0dc2-53da-444a-ecf4bbec91a8</span>**
 Policy: (("stripeWidth" i1) ("cacheReservation" i0) ("proportionalCapacity" (i0 i100)) ("hostFailuresToTolerate" i1) ("forceProvisioning" i0) ("spbmProfileId" "aa6d5a82-1c88-45da-85d3-3d74b91a5bad") ("spbmProfileGenerationNumber" l+2) ("CSN" l1) ("spbmProfileName" "vSAN Default Storage Policy"))
 Is Compliant: true

***localcli vsan iscsi target get --alias*****<span style="color: #0000ff;">RDP-win</span>**
Target Information:
 IQN: iqn.1998-01.com.vmware:22455fef-6b44-e32b-eb2a-f5535af9a5de
 Alias: RDP-win
 UUID: **<span style="color: #0000ff;">34d15a5a-7ff7-924b-776c-ecf4bbec91a8</span>**
 Is Compliant: true
 Policy: (("stripeWidth" i1) ("cacheReservation" i0) ("proportionalCapacity" (i0 i100)) ("hostFailuresToTolerate" i1) ("forceProvisioning" i0) ("spbmProfileId" "aa6d5a82-1c88-45da-85d3-3d74b91a5bad") ("spbmProfileGenerationNumber" l+2) ("CSN" l1) ("spbmProfileName" "vSAN Default Storage Policy"))
 Interface: vmk1
 Port: 3260
 Authentication type: No-Authentication
 Authentication userid:
 Authentication mutual-userid:
 LUNs: 1
 I/O owner UUID: 5937c663-8cb8-3d48-d3ad-ecf4bbec91a8
 I/O owner: IP-Address of the owning host .
 Allowed initiators: iqn.1991-05.com.microsoft:rdp
 Allowed initiator groups:

vit.conf :: cat /vmfs/volumes/vsan:523d5e5605a4d751-0c3304ae7a42599b/.iSCSI-CONFIG/etc/vit.conf

generation 8
auth-group default { auth-type none}
auth-group 34d15a5a-7ff7-924b-776c-ecf4bbec91a8 { auth-type none}
auth-group 62d65a5a-48dd-477d-8c86-ecf4bbec65d8 { auth-type none}
portal-group default { discovery-auth-group no-authentication listen vmk1:3260}
portal-group pg-vmk1-3260 { discovery-auth-group no-authentication listen vmk1:3260}
target **<span style="color: #0000ff;">iqn.1998-01.com.vmware:22455fef-6b44-e32b-eb2a-f5535af9a5de</span>**
{ alias "***<span style="color: #0000ff;">RDP-win</span>***" portal-group pg-vmk1-3260
 auth-group 34d15a5a-7ff7-924b-776c-ecf4bbec91a8
 option uuid 34d15a5a-7ff7-924b-776c-ecf4bbec91a8
 option owner-id 34d15a5a-7ff7-924b-776c-ecf4bbec91a8
 lun 1 { backend vmdk path 34d15a5a-7ff7-924b-776c-ecf4bbec91a8/<span style="color: #0000ff;">***39d15a5a-c635-9e1b-0c92-ecf4bbec91a8.vmdk***</span>
 size **20971520**
 option lun-alias "<span style="color: #0000ff;">***RDP-win***</span>"
 }
}
target **<span style="color: #993300;">iqn.1998-01.com.vmware:0f4751c1-2a44-bb90-83e4-f567cfc711d4</span>** {
 alias "Win-iSCSI-Lun2"
 portal-group pg-vmk1-3260
 auth-group 62d65a5a-48dd-477d-8c86-ecf4bbec65d8
 option uuid 62d65a5a-48dd-477d-8c86-ecf4bbec65d8
 option owner-id 62d65a5a-48dd-477d-8c86-ecf4bbec65d8
 lun 3 { backend vmdk path 62d65a5a-48dd-477d-8c86-ecf4bbec65d8/**<span style="color: #993300;">*67d65a5a-b1e9-fc41-a4d2-ecf4bbec91a8.vmdk*</span>**
 size **20971520**
 option lun-alias "<span style="color: #993300;">***Win-iSCSI-Lun2***</span>"
 }
}

**Looking at the Actual objects for the Luns which are created as objects in vSAN **

change directory to : /vmfs/volumes/vsan:523d5e5605a4d751-0c3304ae7a42599b/.iSCSI-CONFIG/targets :

/vmfs/volumes/vsan:523d5e5605a4d751-0c3304ae7a42599b/bfd05a5a-0dc2-53da-444a-ecf4bbec91a8/targets] ls -l total 16 lrwxrwxrwx 1 root root 42 Jan 14 03:40 34d15a5a-7ff7-924b-776c-ecf4bbec91a8 -> ../../34d15a5a-7ff7-924b-776c-ecf4bbec91a8 lrwxrwxrwx 1 root root 42 Jan 14 04:02 62d65a5a-48dd-477d-8c86-ecf4bbec65d8 -> ../../62d65a5a-48dd-477d-8c86-ecf4bbec65d8

cd to **<span style="color: #0000ff;">34d15a5a-7ff7-924b-776c-ecf4bbec91a8</span>** (Absolute Path : */vmfs/volumes/vsan:523d5e5605a4d751-0c3304ae7a42599b/34d15a5a-7ff7-924b-776c-ecf4bbec91a8*)

<span style="color: #0000ff;">/vmfs/volumes/vsan:523d5e5605a4d751-0c3304ae7a42599b/34d15a5a-7ff7-924b-776c-ecf4bbec91a8</span>] ls -l
total 0
-rw------- 1 root root 521 Jan 14 03:40 **<span style="color: #0000ff;">39d15a5a-c635-9e1b-0c92-ecf4bbec91a8.vmdk
</span>**cat <span style="color: #0000ff;">**39d15a5a-c635-9e1b-0c92-ecf4bbec91a8.vmdk**</span> | grep vsan
RW 20971520 VMFS "vsan://<span style="color: #0000ff;">**39d15a5a-c635-9e1b-0c92-ecf4bbec91a8**</span>" <--- This is the destination object where the data is stored for the Lun "***<span style="color: #0000ff;">RDP-win</span>***" .

[root@is-tse-d156:/vmfs/volumes/vsan:523d5e5605a4d751-0c3304ae7a42599b/bfd05a5a-0dc2-53da-444a-ecf4bbec91a8/targets] ls -l
total 16
lrwxrwxrwx 1 root root 42 Jan 14 03:40 34d15a5a-7ff7-924b-776c-ecf4bbec91a8 -> ../../34d15a5a-7ff7-924b-776c-ecf4bbec91a8
lrwxrwxrwx 1 root root 42 Jan 14 04:02 62d65a5a-48dd-477d-8c86-ecf4bbec65d8 -> ../../62d65a5a-48dd-477d-8c86-ecf4bbec65d8

cd : <span style="color: #993300;">**62d65a5a-48dd-477d-8c86-ecf4bbec65d8**</span> (Absolute Path : */vmfs/volumes/vsan:523d5e5605a4d751-0c3304ae7a42599b/62d65a5a-48dd-477d-8c86-ecf4bbec65d8*)

<span style="color: #993300;">/vmfs/volumes/vsan:523d5e5605a4d751-0c3304ae7a42599b/62d65a5a-48dd-477d-8c86-ecf4bbec65d8</span>] ls -l
total 0
-rw------- 1 root root 521 Jan 14 04:02 **<span style="color: #993300;">67d65a5a-b1e9-fc41-a4d2-ecf4bbec91a8</span><span style="color: #993300;">.vmdk</span>**

cat ***<span style="color: #993300;">67d65a5a-b1e9-fc41-a4d2-ecf4bbec91a8.vmdk</span>*** | grep vsan
RW 20971520 VMFS "vsan://***<span style="color: #993300;">67d65a5a-b1e9-fc41-a4d2-ecf4bbec91a8</span>***" <--- This is the destination object where the data is stored for the Lun "***<span style="color: #993300;">Win-iSCSI-Lun2</span>***"

**Using esxcli vsan debug objects list to dump all object attributes to a file , we can look at the placement of the components for each Lun objects and component health .**

esxcli vsan debug object list  > /tmp/list_all_object.txt

less the output file “less /tmp/list_all_object.txt” , search for “39d15a5a-c635-9e1b-0c92-ecf4bbec91a8“

Object UUID: 39d15a5a-c635-9e1b-0c92-ecf4bbec91a8 Version: 5 Health: healthy Owner: X.X.X.X (Host IP/FQDN) Policy: spbmProfileName: vSAN Default Storage Policy forceProvisioning: 0 spbmProfileGenerationNumber: 2 stripeWidth: 1 cacheReservation: 0 CSN: 2 spbmProfileId: aa6d5a82-1c88-45da-85d3-3d74b91a5bad proportionalCapacity: 0 hostFailuresToTolerate: 1

Configuration:

RAID_1 Component: 39d15a5a-1794-5d1c-c279-ecf4bbec91a8 Component State: ACTIVE, Address Space(B): 10737418240 (10.00GB), Disk UUID: 52f3a35d-2937-9811-81b0-026815670d32, Disk Name: naa.5002538c4044d87f:2 Votes: 1, Capacity Used(B): 12582912 (0.01GB), Physical Capacity Used(B): 4194304 (0.00GB), Host Name: is-tse-d157.isl.vmware.com

Component: 39d15a5a-afc7-5e1c-d3d4-ecf4bbec91a8 Component State: ACTIVE, Address Space(B): 10737418240 (10.00GB), Disk UUID: 52bf76f2-4863-2447-b919-1d7923a822e3, Disk Name: naa.5002538c4044d6ad:2 Votes: 1, Capacity Used(B): 12582912 (0.01GB), Physical Capacity Used(B): 4194304 (0.00GB), Host Name: is-tse-d155.isl.vmware.com

Witness: 39d15a5a-048a-5f1c-5381-ecf4bbec91a8 Component State: ACTIVE, Address Space(B): 0 (0.00GB), Disk UUID: 525b1f66-2d22-7897-d936-6eefd92c7019, Disk Name: naa.5002538c4044d6a3:2 Votes: 1, Capacity Used(B): 12582912 (0.01GB), Physical Capacity Used(B): 4194304 (0.00GB), Host Name: is-tse-d156.isl.vmware.com

Type: vdisk Path: /vmfs/volumes/vsan:523d5e5605a4d751-0c3304ae7a42599b/.iSCSI-CONFIG/targets/34d15a5a-7ff7-924b-776c-ecf4bbec91a8/52dc6cdc-d19d-ac5c-1b32-c5c94383310c.vmdk (Missing) Group UUID: 34d15a5a-7ff7-924b-776c-ecf4bbec91a8 Directory Name: (null)

less the previous output file “less /tmp/list_all_object.txt” , search for “62d65a5a-48dd-477d-8c86-ecf4bbec65d8“

Object UUID: 67d65a5a-b1e9-fc41-a4d2-ecf4bbec91a8
 Version: 5
**Health: healthy**
 Owner: X.X.X.X (Host IP/FQDN)
 Policy:
** spbmProfileName: vSAN Default Storage Policy**
 forceProvisioning: 0
 spbmProfileGenerationNumber: 2
 stripeWidth: 1
 cacheReservation: 0
 CSN: 4
 spbmProfileId: aa6d5a82-1c88-45da-85d3-3d74b91a5bad
 proportionalCapacity: 0
 hostFailuresToTolerate: 1

Configuration:

**RAID_1**
 Component: 67d65a5a-a21d-5c42-d279-ecf4bbec91a8
**Component State: ACTIVE**, Address Space(B): 10737418240 (10.00GB), Disk UUID: 5270b13f-a6a0-50cf-0cb3-88d86b7d323e, Disk Name: naa.5002538c4044d6ab:2
 Votes: 1, Capacity Used(B): 12582912 (0.01GB), Physical Capacity Used(B): 4194304 (0.00GB), Host Name: is-tse-d155.isl.vmware.com

 Component: 67d65a5a-9967-5d42-3ad0-ecf4bbec91a8
**Component State: ACTIVE**, Address Space(B): 10737418240 (10.00GB), Disk UUID: 525b1f66-2d22-7897-d936-6eefd92c7019, Disk Name: naa.5002538c4044d6a3:2
 Votes: 1, Capacity Used(B): 12582912 (0.01GB), Physical Capacity Used(B): 4194304 (0.00GB), Host Name: is-tse-d156.isl.vmware.com

 Witness: 67d65a5a-7227-5e42-31ab-ecf4bbec91a8
**Component State: ACTIVE**, Address Space(B): 0 (0.00GB), Disk UUID: 529e9a5d-a5d8-6a18-3933-ed69eca58c36, Disk Name: naa.5002538c4044d6ae:2
 Votes: 1, Capacity Used(B): 12582912 (0.01GB), Physical Capacity Used(B): 4194304 (0.00GB), Host Name: is-tse-d157.isl.vmware.com

**Type: vdisk**
 Path: /vmfs/volumes/vsan:523d5e5605a4d751-0c3304ae7a42599b/.iSCSI-CONFIG/targets/62d65a5a-48dd-477d-8c86-ecf4bbec65d8/5239144d-34bc-439c-1907-2c541a56a141.vmdk (Missing)
 Group UUID: 62d65a5a-48dd-477d-8c86-ecf4bbec65d8
 Directory Name: (null)

RVC command to look at the objects from vCenter server RVC session

***Note*: All iSCSI target will be listed under unassociated objects , be extra careful while deleting associated objects as you may accidentally delete vSAN iSCSI home folder and the target VMDKs and they cannot be recovered once deleted .***

/localhost/vsan66-AF-DELL/computers/vsan66-ALL-FLASH-DELL> vsan.obj_status_report . -t

Unassociated objects 
1a39c759-70d6-ea0a-29f4-ecf4bbec65d8 
d9e59b59-9031-180b-6000-ecf4bbec65d8 
79ca385a-f785-a610-5038-ecf4bbec65d8 
**<span style="color: #0000ff;">39d15a5a-c635-9e1b-0c92-ecf4bbec91a8</span>** <---------------iSCSI target Lun 1 object for **<span style="color: #0000ff;">RDP-win</span>** vmdk 
**<span style="color: #993300;">67d65a5a-b1e9-fc41-a4d2-ecf4bbec91a8</span>** <---------------iSCSI target Lun 3 object <span style="color: #993300;">**Win-iSCSI-Lun2**</span> vmdk 
<span style="color: #339966;">**bfd05a5a-0dc2-53da-444a-ecf4bbec91a8**</span> <---------------iSCSI home folder object

Read more