Test Procedure with KVM guests + iPXE
=====================================

- 2 guests: iSCSI target/server and iSCSI initiator/client.
- 1 bridge for iSCSI traffic (virbr-iscsi, new), static ip.
- 1 bridge for internet access (virbr0, exists), dhcp ip. 


Host:
----

Configure the iSCSI bridge and QEMU access in the host:

    $ sudo ip link add dev virbr-iscsi type bridge
    $ sudo ip link set dev virbr-iscsi up
    $ echo 'allow virbr-iscsi' | sudo tee -a /etc/qemu/bridge.conf
    $ sudo chmod +s /usr/lib/qemu/qemu-bridge-helper

iSCSI target:
------------

This guest serves an iSCSI target with one LUN
in iSCSI NIC with IP 10.0.0.1 for IP 10.0.0.2.

Install/boot this guest:

	$ qemu-img create -f qcow2 guest-iscsi-target.qcow2 16g

    $ qemu-system-x86_64 \
      -nodefaults \
      -enable-kvm \
      -smp 2 -m 4096 \
      -serial stdio \
      -vga virtio \
      -display vnc=0.0.0.0:2 \
      -netdev bridge,id=bridge-world,br=virbr0 \
      -netdev bridge,id=bridge-iscsi,br=virbr-iscsi \
      -device virtio-net-pci,netdev=bridge-world,id=nic-world,mac=52:54:00:00:00:11 \
      -device virtio-net-pci,netdev=bridge-iscsi,id=nic-iscsi,mac=52:54:00:00:00:22 \
      -drive file=guest-iscsi-target.qcow2,if=virtio \
      -drive file=$RELEASE-server-amd64.iso,media=cdrom,read-only,if=scsi \
      -boot once=d

Configure iSCSI NIC:

$ cat <<EOF | sudo tee /etc/netplan/ens5.yaml && sudo netplan apply
network:
    version: 2
    ethernets:
        ens5:
            addresses: [10.0.0.1/24]
EOF


    # ip addr list ens5
    3: ens5: <...>
        link/ether 52:54:00:00:00:22 brd ff:ff:ff:ff:ff:ff
        inet 10.0.0.1/24 brd 10.0.0.255 scope global ens4
    ...

Configure iSCSI target/lun:

    # apt-get install -y tgt

    # mkdir /var/lib/iscsi
    # dd if=/dev/zero of=/var/lib/iscsi/disk bs=1 count=0 seek=8G

    # tgtadm --lld iscsi --op new --mode target --tid 1 -T iqn.2019-03.com.example:target1
    # tgtadm --lld iscsi --op new --mode logicalunit --tid 1 --lun 1 -b /var/lib/iscsi/disk

    # tgtadm --lld iscsi --op bind --mode target --tid 1 -I 10.0.0.2
    # tgt-admin --dump >/etc/tgt/conf.d/target1.conf



iSCSI initiator:
---------------

This guest first boots iPXE to configure iBFT,
and then boots/chainloads to debian-installer.

The netboot initrd does not contain all patched udebs,
so download and install disk-detect and partman-iscsi
from the PPA during the install.

    $ wget http://boot.ipxe.org/ipxe.lkrn
    $ wget http://ppa.launchpad.net/mfo/lp1817321v3/ubuntu/dists/$RELEASE/main/installer-amd64/current/images/netboot/ubuntu-installer/amd64/{linux,initrd.gz}

    $ python3 -m http.server &
    Serving HTTP on 0.0.0.0 port 8000 ...

    $ qemu-system-x86_64 \
      -nodefaults \
      -enable-kvm \
      -smp 2 -m 4096 \
      -serial stdio \
      -vga virtio \
      -display vnc=0.0.0.0:1 \
      -netdev bridge,id=bridge-world,br=virbr0 \
      -netdev bridge,id=bridge-iscsi,br=virbr-iscsi \
      -device virtio-net-pci,netdev=bridge-world,id=nic-world,mac=52:54:00:00:00:01 \
      -device virtio-net-pci,netdev=bridge-iscsi,id=nic-iscsi,mac=52:54:00:00:00:02 \
      -kernel ipxe.lkrn

Connect to VNC for iPXE shell:

    $ vncviewer <address>:1
    iPXE <...>
    <wait for NIC address detection>

    Press Ctrl-B for iPXE command line.
    ^B

    iPXE>

Configure iSCSI NIC:

    iPXE> ifopen net1
    iPXE> set net1/ip 10.0.0.2
    iPXE> set net1/netmask 255.255.255.0

Configure iBFT: (iSCSI portal 10.0.0.1, LUN 1 on target iqn.<...>:target1)

    iPXE> sanhook iscsi:10.0.0.1:::1:iqn.2019-03.com.example:target1
    Registered SAN device 0x80

Boot the installer 
(add option 'disk-detect/ibft/enable=true' for installer to detect iBFT iSCSI disks
 and option 'partman-iscsi/iscsi_auto=true' to set the system to boot from iBFT):

    iPXE> ifopen net0
    iPXE> kernel http://192.168.122.1:8000/linux initrd=initrd.gz disk-detect/ibft/enable=true partman-iscsi/iscsi_auto=true --- console=ttyS0
    iPXE> initrd http://192.168.122.1:8000/initrd.gz
    iPXE> boot 

Back to serial console.
Proceed with the installer.

In 'Users and passwords' dialog, select 'Go back', and 'Execute a shell', and 'Continue'.

Check kernel version and iscsi_ibft.ko module is present.

	~ # uname -rv
	5.0.0-8-generic #9-Ubuntu SMP Tue Mar 12 21:58:11 UTC 2019

~ # depmod -a
~ # modinfo --filename iscsi_ibft
/lib/modules/5.0.0-8-generic/kernel/drivers/firmware/iscsi_ibft.ko

	~ # wget http://ppa.launchpad.net/mfo/lp1817321v3/ubuntu/pool/main/h/hw-detect/disk-detect_1.117ubuntu7.$VERSION_amd64.udeb
	~ # wget http://ppa.launchpad.net/mfo/lp1817321v3/ubuntu/pool/main/p/partman-iscsi/partman-iscsi_40ubuntu4.$VERSION_all.udeb

    ~ # udpkg --unpack *.udeb

~ # debconf-get disk-detect/ibft/enable 
true
~ # debconf-get partman-iscsi/iscsi_auto
true

    (Use this if you need it; e.g., forgot kernel cmdline options)
    ~ # debconf-set disk-detect/ibft/enable true                                    

Start another installer menu with the new debconf templates/question:

    ~ # debconf -o d-i /usr/bin/main-menu

Proceed with the installer.

In the 'Partition disks' dialog, the iSCSI LUN should be present:

    SCSI3 (0,0,1) (sda) - 8.6 GB IET VIRTUAL-DISK 

Proceed with the installer.

In the 'Finish the Installation' dialog, 'Go Back', 'Execute a shell', 'Continue'.

Check ISCSI_AUTO is set to true in the installation target's /etc/iscsi/iscsi.initramfs.

	~ # cat /target/etc/iscsi/iscsi.initramfs 
	ISCSI_AUTO=true

Resume the installation with 'exit' and finish it.

System reboots.

Back to VNC console, go again to iPXE shell:

    $ vncviewer <address>:1
    iPXE <...>
    <wait for NIC address detection>

    Press Ctrl-B for iPXE command line.
    ^B

    iPXE>

Configure iSCSI NIC:

    iPXE> ifopen net1
    iPXE> set net1/ip 10.0.0.2
    iPXE> set net1/netmask 255.255.255.0

Boot from iSCSI (iSCSI portal 10.0.0.1, LUN 1 on target iqn.<...>:target1)

    iPXE> sanboot iscsi:10.0.0.1:::1:iqn.2019-03.com.example:target1
    Registered SAN device 0x80
    Booting from SAN device 0x80

Back to serial console.
Watch the system boot:

	...
	Setting up software interface ens4
	iscsistart: Logging into iqn.2019-03.com.example:target1 10.0.0.1:3260,1
	iscsistart: version 2.0-874
	iscsistart: Connection1:0 to [target: iqn.2019-03.com.example:target1, portal: 10.0.0.1,3260] through [iface: default] is operational now
	/scripts/local-top/iscsi: line 397: : Permission denied
	/dev/sda1: recovering journal
	/dev/sda1: clean, 60800/524288 files, 519693/2096640 blocks
	...

	ubuntu login: ubuntu
	Password: 

	ubuntu@ubuntu:~$ mount | grep -w /
	/dev/sda1 on / type ext4 (rw,relatime,errors=remount-ro) # plus 'data=ordered' in bionic

	ubuntu@ubuntu:~$ ls -ld /sys/block/sda
	lrwxrwxrwx 1 root root 0 Apr  3 21:38 /sys/block/sda -> ../devices/platform/host2/session1/target2:0:0/2:0:0:1/block/sda

	ubuntu@ubuntu:~$ cat /sys/devices/platform/host2/session1/iscsi_session/session1/targetname
	iqn.2019-03.com.example:target1





syslog example
==============

	$ sed -n '/iBFT found/p ; /iBFT detected/,/iBFT disk detection finished/p' syslog 
	Apr  3 20:03:53 kernel: [    0.006887] iBFT found at 0x9e520.
	Apr  3 20:12:59 kernel: [  546.682048] iBFT detected.
	Apr  3 20:12:59 disk-detect: insmod /lib/modules/5.0.0-8-generic/kernel/drivers/scsi/iscsi_boot_sysfs.ko 
	Apr  3 20:12:59 disk-detect: insmod /lib/modules/5.0.0-8-generic/kernel/drivers/firmware/iscsi_ibft.ko 
	Apr  3 20:12:59 disk-detect: # BEGIN RECORD 2.0-874
	Apr  3 20:12:59 disk-detect: iface.initiatorname = iqn.2010-04.org.ipxe:00000000-0000-0000-0000-000000000000
	Apr  3 20:12:59 disk-detect: iface.hwaddress = 52:54:00:00:00:02
	Apr  3 20:12:59 disk-detect: iface.bootproto = STATIC
	Apr  3 20:12:59 disk-detect: iface.ipaddress = 10.0.0.2
	Apr  3 20:12:59 disk-detect: iface.subnet_mask = 255.255.255.0
	Apr  3 20:12:59 disk-detect: iface.primary_dns = 192.168.122.1
	Apr  3 20:12:59 disk-detect: iface.vlan_id = 0
	Apr  3 20:12:59 disk-detect: iface.net_ifacename = ens4
	Apr  3 20:12:59 disk-detect: node.name = iqn.2019-03.com.example:target1
	Apr  3 20:12:59 disk-detect: node.conn[0].address = 10.0.0.1
	Apr  3 20:12:59 disk-detect: node.conn[0].port = 3260
	Apr  3 20:12:59 disk-detect: node.boot_lun = 01000000
	Apr  3 20:12:59 disk-detect: # END RECORD
	Apr  3 20:12:59 disk-detect: Setting up software interface ens4
	Apr  3 20:12:59 disk-detect: iscsistart: 
	Apr  3 20:12:59 disk-detect: can not connect to iSCSI daemon (111)!
	Apr  3 20:12:59 disk-detect: 
	Apr  3 20:12:59 disk-detect: iscsistart: version 2.0-874
	Apr  3 20:12:59 kernel: [  546.699057] Loading iSCSI transport class v2.0-870.
	Apr  3 20:13:00 kernel: [  547.707438] iscsi: registered transport (tcp)
	Apr  3 20:13:00 kernel: [  547.709836] scsi host2: iSCSI Initiator over TCP/IP
	Apr  3 20:13:00 disk-detect: iscsistart: Connection1:0 to [target: iqn.2019-03.com.example:target1, portal: 10.0.0.1,3260] through [iface: default] is operational now
	Apr  3 20:13:00 kernel: [  547.716901] scsi 2:0:0:0: RAID              IET      Controller       0001 PQ: 0 ANSI: 5
	Apr  3 20:13:00 kernel: [  547.719266] scsi 2:0:0:0: Attached scsi generic sg0 type 12
	Apr  3 20:13:00 kernel: [  547.722572] scsi 2:0:0:1: Direct-Access     IET      VIRTUAL-DISK     0001 PQ: 0 ANSI: 5
	Apr  3 20:13:00 kernel: [  547.724360] sd 2:0:0:1: Attached scsi generic sg1 type 0
	Apr  3 20:13:00 kernel: [  547.724668] sd 2:0:0:1: Power-on or device reset occurred
	Apr  3 20:13:00 disk-detect: iscsistart: Logging into iqn.2019-03.com.example:target1 10.0.0.1:3260,1
	Apr  3 20:13:00 kernel: [  547.727114] sd 2:0:0:1: [sda] 16777216 512-byte logical blocks: (8.59 GB/8.00 GiB)
	Apr  3 20:13:00 kernel: [  547.727117] sd 2:0:0:1: [sda] 4096-byte physical blocks
	Apr  3 20:13:00 kernel: [  547.727463] sd 2:0:0:1: [sda] Write Protect is off
	Apr  3 20:13:00 kernel: [  547.727466] sd 2:0:0:1: [sda] Mode Sense: 69 00 10 08
	Apr  3 20:13:00 kernel: [  547.728047] sd 2:0:0:1: [sda] Write cache: enabled, read cache: enabled, supports DPO and FUA
	Apr  3 20:13:00 kernel: [  547.742488] sd 2:0:0:1: [sda] Attached SCSI disk
	Apr  3 20:13:00 disk-detect: iBFT disk detection finished.