SCST KVM UEK 3.8

This guide explains how to build SCST SAN on an Oracle Enterprise Linux 6.5 KVM guest VM based on the excellent instructions here at spiderbird. That guide also has a wealth of reference URLs that can be accessed as well.

This is the revised version that uses the Oracle Enterprise Linux UEK version 3 kernel for the build. This was motivated for consistency (so that OEL 6.5 is running it's own proprietary kernel, and not a CentOS kernel) and so that DTrace tool which Oracle has ported over to it's UEK kernel could be utilized (although issues were encountered on the DTrace part). The first build utilizing the CentOS kernel was done due to time constraints and the kernel source for Oracle UEK kernel version 3 was not handy already downloaded at that time, and since it has to be downloaded as part of the DVD from edelivery.oracle.com a POC was done quickly using CentOS 6 kernel. Now in the present blog that is remedied so that the SCST SAN is a 100% OEL UEK kernel custom-modified for SCST iSCSI Linux SAN.

I intend to use this KVM guest SCST SAN to present 4K logcial/4K physical virtual LUNs to Oracle running in other KVM guests, including Oracle Real Application Clusters running in KVM guests. Because TGT cannot handle LUNs which are 4K logical / 4K physical it was necessary to get up to speed on SCST which can. This sourceforge SAN feature comparison sheet shows the full power of SCST compared to other linux SAN options.

Ensure the virtual ethernet interface, typically "eth0" in the OEL 6.5 KVM guest is set to "connect automatically". If not, configure it as shown in the screenshots below. This can be configured from the server desktop, if using OEL 6.5 with a desktop, or simply edited in the ifcfg-eth0 file as shown below. As shown in bold below, be sure that "ONBOOT=yes" has been set in the ifcfg-ethX file where in this case my interface is ifcfg-eth0.

[root@oracle651 network-scripts]# cat ifcfg-eth0

DEVICE=eth0

TYPE=Ethernet

UUID=824a1d97-1dc7-410b-89a5-49b353be826a

ONBOOT=yes

NM_CONTROLLED=yes

BOOTPROTO=dhcp

DNS1=127.0.0.1

DOMAIN=vmem.org

DEFROUTE=yes

IPV4_FAILURE_FATAL=yes

IPV6INIT=no

NAME="System eth0"

HWADDR=52:54:00:66:26:A0

DNS2=192.168.1.1

PEERDNS=no

PEERROUTES=yes

LAST_CONNECT=1407733083

[root@oracle651 network-scripts]#

Download the required Oracle Linux 6.5 Source ISO files. The Oracle Enterprise Linux ISO files are only available as downloads from edelivery.oracle.com so therefore an Oracle Network login will be needed. (Both an Oracle Network Login and the ISO's are free, but registration is required). The ISOs needed are as shown below.

gstanden@vmem1:~/Downloads$ ls -lrt V4*.iso

-rw-rw-r-- 1 libvirt-qemu kvm 3885117440 Aug 7 15:50 V41362-01.iso <- Install ISO

-rw-rw-r-- 1 gstanden gstanden 3221997568 Aug 12 17:55 V41365-01.iso <- Source Code SRPMS ISO #1

-rw-rw-r-- 1 gstanden gstanden 2116456448 Aug 12 18:03 V41366-01.iso <- Source Code SRPMS ISO #2

gstanden@vmem1:~/Downloads$

Do a yum update to update the kernel before beginning this work. Once this work is done the kernel will be locked so that there will be no more kernel updated, because this procedure builds a custom kernel for SCST which will not be updated any further normally (without going through this entire procedure again!).

Once the yum update is complete, reboot the KVM guest so that the just-updated kernel is running in the KVM guest.

Install the following packages using yum as shown below. Do this as root. Note that further on in this blog there was a missing dependency on "xmlto" which I am retroactively addressing here in this blog so there won't be that missing dependency.

[root@oracle651 ~]# yum install newt-devel

[root@oracle651 ~]# yum install asciidoc

[root@oracle651 ~]# yum install xmlto

[root@oracle651 ~]# yum install libdtrace-ctf-devel

As root, add the "mockbuild" user as shown below.

[root@oracle651 ~]# useradd mockbuild

Now su to the "mockbuild" user as shown below. A password is not needed for the mockbuild user unless you explicitly want one. It's just an account that will be used for this build.

[root@oracle651 ~]# su - mockbuild

[mockbuild@oracle651 ~]$ pwd

/home/mockbuild

[mockbuild@oracle651 ~]$ id

uid=501(mockbuild) gid=501(mockbuild) groups=501(mockbuild) context=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023

[mockbuild@oracle651 ~]$

Run the following commands as the mockbuild user from the /home/mockbuild directory as shown below. It will be necessary to obtain the OEL UEK version 3 source rpm's from edelivery.oracle.com which are on the source DVD's downloadable from there, and then stage them as shown in /home/mockbuild with correct ownership of mockbuild:mockbuild as shown below.

[mockbuild@oracle651 ~]$ pwd

/home/mockbuild

[mockbuild@oracle651 ~]$ ls -lrt

total 171080

-rw-r--r-- 1 mockbuild mockbuild 85551190 Aug 23 11:58 kernel-uek-2.6.39-400.211.1.el6uek.src.rpm

-rw-r--r-- 1 mockbuild mockbuild 89621592 Aug 23 11:58 kernel-uek-3.8.13-16.2.1.el6uek.src.rpm

drwxrwxr-x 8 mockbuild mockbuild 4096 Aug 23 13:49 rpmbuild

[mockbuild@oracle651 ~]$

Next run the steps as shown below from the /home/mockbuild directory while connected as the linux "mockbuild" user.

[mockbuild@oracle651 ~]$ mkdir -p ~/rpmbuild/{BUILD,BUILDROOT,RPMS,SOURCES,SPECS,SRPMS}

[mockbuild@oracle651 ~]$ echo '%_topdir %(echo $HOME)/rpmbuild' > ~/.rpmmacros

[mockbuild@oracle651 ~]$ rpm -i kernel-uek-3.8.13-16.2.1.el6uek.src.rpm 2>&1 | grep -v mock

Before running the below "rpmbuild" command in step after the next one, open another window as root and run the entropy generator as shown here. If the run of rpmbuild in the next step bogs down, run this command again to refill the entropy pool. It will make the rpmbuild complete faster.

[root@oracle651 ~]# rngd -r /dev/urandom

[root@oracle651 ~]#

Now you can run the command. If it bogs down towards the end run the above rngd command again. Note: be sure when running the command below, that if you cut and paste from this blog that the word "--target" has "two dashes" i.e "--" in front of it, and not just one big dash. Sometimes characterset conversion can cause two dashed cut and pasted to become an uninterpretable single large dash.

Also, note the change for kernel-uek.spec as shown in bold below (compared to say CentOS 6 or RHEL 6, for example).

[root@oracle651 ~]# su - mockbuild

[mockbuild@oracle651 ~]$ rpmbuild -bp --target=$(uname -m) ./rpmbuild/SPECS/kernel-uek.spec

Building target platforms: x86_64

Building for target x86_64

Executing(%prep): /bin/sh -e /var/tmp/rpm-tmp.1l1BDW

+ umask 022

+ cd /home/mockbuild/rpmbuild/BUILD

+ LANG=C

+ export LANG

+ unset DISPLAY

+ patch_command='patch -p1 -F1 -s'

+ '[' -f .shared-srctree ']'

+ '[' '!' -d kernel-3.8.13/vanilla-2.6.13 ']'

+ '[' -d kernel-3.8.13/vanilla-3.8.13 ']'

+ rm -f pax_global_header

+ [[ ! -z '' ]]

+ cd /home/mockbuild/rpmbuild/BUILD

+ rm -rf kernel-3.8.13

+ /bin/mkdir -p kernel-3.8.13

+ cd kernel-3.8.13

+ /usr/bin/bzip2 -dc /home/mockbuild/rpmbuild/SOURCES/linux-3.8.13.tar.bz2

+ /bin/tar -xf -

+ STATUS=0

+ '[' 0 -ne 0 ']'

+ /bin/chmod -Rf a+rX,u+w,g-w,o-w .

+ mv linux-3.8.13 vanilla-3.8.13

+ [[ ! -z '' ]]

+ cp -rl vanilla-3.8.13 vanilla-2.6.13

+ cd vanilla-2.6.13

+ cd ..

+ '[' -d linux-3.8.13.x86_64 ']'

+ cp -rl vanilla-2.6.13 linux-3.8.13-16.2.1.el6uek

+ cd linux-3.8.13-16.2.1.el6uek

+ cp /usr/lib/rpm/find-debuginfo.sh /home/mockbuild/rpmbuild/BUILD

+ patch /home/mockbuild/rpmbuild/BUILD/find-debuginfo.sh /home/mockbuild/rpmbuild/SOURCES/debuginfo-g1.diff

patching file /home/mockbuild/rpmbuild/BUILD/find-debuginfo.sh

+ chmod +x /home/mockbuild/rpmbuild/BUILD/find-debuginfo.sh

+ mkdir -p configs

+ cp /home/mockbuild/rpmbuild/SOURCES/config-x86_64-debug configs/config-debug

+ cp /home/mockbuild/rpmbuild/SOURCES/config-x86_64 configs/config

+ find . '(' -name '*.orig' -o -name '*~' ')' -exec rm -f '{}' ';'

+ exit 0

Now execute the following command as shown below. The full correct expected output of the following command is attached to this blog as "scst.log" at the end of this blog page.

[mockbuild@oracle651 ~]$ svn co https://svn.code.sf.net/p/scst/svn/branches/2.2.x scst

The scst code is installed under the /home/mockbuild directory. Now Change directory as shown below and check availability of the required patches for the 3.8 kernel as shown below.

[mockbuild@oracle651 ~]$ uname -a

Linux oracle651.vmem.org 3.8.13-35.3.4.el6uek.x86_64 #2 SMP Wed Jul 30 00:59:13 PDT 2014 x86_64 x86_64 x86_64 GNU/Linux

[mockbuild@oracle651 ~]$ id

uid=501(mockbuild) gid=501(mockbuild) groups=501(mockbuild)

[mockbuild@oracle651 ~]$ cd ~/rpmbuild/BUILD/kernel-3.8.13/linux-3.8.13-16.2.1.el6uek/

[mockbuild@oracle651 linux-3.8.13-16.2.1.el6uek]$ ls -l ~/scst/scst/kernel/scst_exec_req_fifo-3.8.patch

-rw-rw-r-- 1 mockbuild mockbuild 12833 Aug 23 14:21 /home/mockbuild/scst/scst/kernel/scst_exec_req_fifo-3.8.patch

[mockbuild@oracle651 linux-3.8.13-16.2.1.el6uek]$

[mockbuild@oracle651 linux-3.8.13-16.2.1.el6uek]$ ls -l ~/scst/iscsi-scst/kernel/patches/put_page_callback-3.8.patch

-rw-rw-r-- 1 mockbuild mockbuild 12499 Aug 23 14:21 /home/mockbuild/scst/iscsi-scst/kernel/patches/put_page_callback-3.8.patch

[mockbuild@oracle651 linux-3.8.13-16.2.1.el6uek]$

Apply the patches as shown below. Note, if you get the following error when you try to run the command it is because your "p1" is typed as a "pl" (p-ell). Be sure to use "p1" (p-one).

[mockbuild@oracle651 ~]$ cd ~/rpmbuild/BUILD/kernel-3.8.13/linux-3.8.13-16.2.1.el6uek/

[mockbuild@oracle651 linux-3.8.13-16.2.1.el6uek]$ patch -p1 < ~/scst/scst/kernel/scst_exec_req_fifo-3.8.patch | tee /home/mockbuild/logs/scst_exec_req_fifo-3.8.patch.log

patching file block/blk-map.c

patching file include/linux/blkdev.h

patching file include/linux/scatterlist.h

patching file lib/scatterlist.c

[mockbuild@oracle651 linux-3.8.13-16.2.1.el6uek]$

[mockbuild@oracle651 linux-3.8.13-16.2.1.el6uek]$ patch -p1 < ~/scst/iscsi-scst/kernel/patches/put_page_callback-3.8.patch | tee /home/mockbuild/logs/put_page_callback-3.8.patch.log

patching file drivers/net/macvtap.c

patching file drivers/net/tun.c

Hunk #1 succeeded at 1014 (offset 2 lines).

patching file drivers/net/vmxnet3/vmxnet3_drv.c

patching file drivers/net/xen-netback/netback.c

Hunk #1 succeeded at 1329 (offset 239 lines).

Hunk #2 succeeded at 1867 with fuzz 1 (offset 318 lines).

patching file include/linux/mm_types.h

patching file include/linux/net.h

patching file include/linux/skbuff.h

Hunk #1 succeeded at 1949 (offset 5 lines).

Hunk #2 succeeded at 1972 (offset 5 lines).

patching file net/Kconfig

patching file net/core/skbuff.c

patching file net/core/sock.c

patching file net/ipv4/Makefile

patching file net/ipv4/ip_output.c

patching file net/ipv4/tcp.c

patching file net/ipv4/tcp_zero_copy.c

patching file net/ipv6/ip6_output.c

Hunk #1 succeeded at 1461 (offset 10 lines).

[mockbuild@oracle651 linux-3.8.13-16.2.1.el6uek]$

Check to make sure the config files are about the same. On my build, there was no .config at this point. I proceeded on to the "make menuconfig" step next, and made the required changes to configuration, and then saved, and then there was a ".config" file in the directory.

One difference here is that I made the SCSI Disk Support option a built-in instead of a module. It was ticked with <M> and I changed it to <*>.

I ran the "make oldconfig" and then did a diff on the config files as shown below. Most of the differences looked like the changes that were make with menuconfig so I went ahead and proceeded to kernel compile step.

[mockbuild@oracle651 linux-3.8.13-16.2.1.el6uek]$ cp -p /boot/config-3.8.13-16.2.1.el6uek.x86_64 .config

[mockbuild@oracle651 linux-3.8.13-16.2.1.el6uek]$

The menuconfig GUI opens. Navigate through the GUI and choose the options as quoted and described at the spiderweb reference here. For completeness, those changes that need to be made and verified in menuconfig are as follows from the reference but restated here for convenience.

Now run the command as shown below:

[mockbuild@oracle651 linux-3.8.13-16.2.1.el6uek]$ make menuconfig

and answer change the following as shown below

“Network support”->”Networking options”

“TCP/IP networking”

“TCP/IP zero-copy transfer completion notification”

“Device Drivers”->”SCSI device support”

”SCSI disk support”

“Enable the block layer”->”IO Schedulers”->”CFQ I/O scheduler”->“Default I/O scheduler”

"No-op"

“Processor type an features”->”Preemption Model”

“No Forced Preemption - Server"

Edit the Makefile as shown with the EXTRAVERSION information. In this example, the kernel version will have no special identifier and will just overwrite the existing kernel with a kernel of the same name but with the added SCST features. It is also possible to append a custom tag to mark the kernel as modified such as "-16.2.1.el6uek.scst.x86_64".

[mockbuild@oracle651 linux-3.8.13-16.2.1.el6uek]$ vi Makefile

[mockbuild@oracle651 linux-3.8.13-16.2.1.el6uek]$ grep EXTRAVERSION Makefile | grep -v KERNELVERSION

EXTRAVERSION =-16.2.1.el6uek.x86_64

[mockbuild@oracle651 linux-3.8.13-16.2.1.el6uek]$

Now run make as shown below.

[mockbuild@oracle651 linux-3.8.13-16.2.1.el6uek]$ time make -j8 all | tee /home/mockbuild/make.log

Now make modules install as shown below, as the "root" user.

[mockbuild@oracle651 linux-2.6.32-431.23.3.el6.x86_64]$ su - root

[root@oracle651 linux-2.6.32-431.23.3.el6.x86_64]# cd /home/mockbuild/rpmbuild/BUILD/kernel-2.6.32-431.23.3.el6/linux-2.6.32-431.23.3.el6.x86_64

[root@oracle651 linux-2.6.32-431.23.3.el6.x86_64]# make modules_install

cp -pR /lib/modules/3.8.13-16.2.1.el6uek.x86_64/ /lib/modules/3.8.13-16.2.1.el6uek.scst.x86_64/

make modules install

cd /home/mockbuild/rpmbuild/BUILD/kernel-3.8.13/linux-3.8.13-16.2.1.el6uek/

Next, again as root, do a "make install" as shown below. The errors shown are OK. These modules will be added after reboot.

[root@oracle651 linux-3.8.13-16.2.1.el6uek]# make install | tee /home/mockbuild/install.log

sh /home/mockbuild/rpmbuild/BUILD/kernel-3.8.13/linux-3.8.13-16.2.1.el6uek/arch/x86/boot/install.sh 3.8.13-16.2.1.el6uek.x86_64 arch/x86/boot/bzImage \

System.map "/boot"

ERROR: modinfo: could not find module iscsi_scst

ERROR: modinfo: could not find module scst_vdisk

ERROR: modinfo: could not find module scst

[root@oracle651 linux-3.8.13-16.2.1.el6uek]#

Now edit /boot/grub/menu.lst to update the server to use the new kernel at boot as shown below. Change the value of "default=" to use the new kernel. The first kernel listed is always the "0" (zero) kernel so in this example set "detault=0". Also, while editing this file, remove the "rhgb quiet" from that kernel line so that all boot output messages will be buffered to console, and also add at the end of the kernel line "console=ttyS0" so that KVM console is configured.

[root@oracle651 linux-3.8.13-16.2.1.el6uek]# cat /boot/grub/menu.lst

# grub.conf generated by anaconda

#

# Note that you do not have to rerun grub after making changes to this file

# NOTICE: You have a /boot partition. This means that

# all kernel and initrd paths are relative to /boot/, eg.

# root (hd0,0)

# kernel /vmlinuz-version ro root=/dev/mapper/vg_oracle651-lv_root

# initrd /initrd-[generic-]version.img

#boot=/dev/vda

default=0

timeout=5

splashimage=(hd0,0)/grub/splash.xpm.gz

hiddenmenu

title Oracle Linux Server (3.8.13-16.2.1.el6uek.x86_64)

root (hd0,0)

kernel /vmlinuz-3.8.13-16.2.1.el6uek.x86_64 ro root=/dev/mapper/vg_oracle651-lv_root rd_NO_LUKS LANG=en_US.UTF-8 rd_LVM_LV=vg_oracle651/lv_swap rd_NO_MD rd_LVM_LV=vg_oracle651/lv_root SYSFONT=latarcyrheb-sun16 KEYBOARDTYPE=pc KEYTABLE=us rd_NO_DM crashkernel=auto console=ttyS0

initrd /initramfs-3.8.13-16.2.1.el6uek.x86_64.img

title Oracle Linux Server Red Hat Compatible Kernel (2.6.32-scst)

root (hd0,0)

kernel /vmlinuz-2.6.32-scst ro root=/dev/mapper/vg_oracle651-lv_root rd_NO_LUKS LANG=en_US.UTF-8 rd_LVM_LV=vg_oracle651/lv_swap rd_NO_MD rd_LVM_LV=vg_oracle651/lv_root SYSFONT=latarcyrheb-sun16 KEYBOARDTYPE=pc KEYTABLE=us rd_NO_DM crashkernel=auto console=ttyS0

initrd /initramfs-2.6.32-scst.img

title Oracle Linux Server (3.8.13-35.3.4.el6uek.x86_64)

root (hd0,0)

kernel /vmlinuz-3.8.13-35.3.4.el6uek.x86_64 ro root=/dev/mapper/vg_oracle651-lv_root rd_NO_LUKS LANG=en_US.UTF-8 rd_LVM_LV=vg_oracle651/lv_swap rd_NO_MD rd_LVM_LV=vg_oracle651/lv_root SYSFONT=latarcyrheb-sun16 KEYBOARDTYPE=pc KEYTABLE=us rd_NO_DM rhgb quiet

initrd /initramfs-3.8.13-35.3.4.el6uek.x86_64.img

title Oracle Linux Server Red Hat Compatible Kernel (2.6.32-431.23.3.el6.x86_64)

root (hd0,0)

kernel /vmlinuz-2.6.32-431.23.3.el6.x86_64 ro root=/dev/mapper/vg_oracle651-lv_root rd_NO_LUKS LANG=en_US.UTF-8 rd_LVM_LV=vg_oracle651/lv_swap rd_NO_MD rd_LVM_LV=vg_oracle651/lv_root SYSFONT=latarcyrheb-sun16 KEYBOARDTYPE=pc KEYTABLE=us rd_NO_DM rhgb quiet crashkernel=auto

initrd /initramfs-2.6.32-431.23.3.el6.x86_64.img

title Oracle Linux Server Unbreakable Enterprise Kernel (3.8.13-16.2.1.el6uek.x86_64)

root (hd0,0)

kernel /vmlinuz-3.8.13-16.2.1.el6uek.x86_64 ro root=/dev/mapper/vg_oracle651-lv_root rd_NO_LUKS LANG=en_US.UTF-8 rd_LVM_LV=vg_oracle651/lv_swap rd_NO_MD rd_LVM_LV=vg_oracle651/lv_root SYSFONT=latarcyrheb-sun16 KEYBOARDTYPE=pc KEYTABLE=us rd_NO_DM rhgb quiet

initrd /initramfs-3.8.13-16.2.1.el6uek.x86_64.img

title Oracle Linux Server Red Hat Compatible Kernel (2.6.32-431.el6.x86_64)

root (hd0,0)

kernel /vmlinuz-2.6.32-431.el6.x86_64 ro root=/dev/mapper/vg_oracle651-lv_root rd_NO_LUKS LANG=en_US.UTF-8 rd_LVM_LV=vg_oracle651/lv_swap rd_NO_MD rd_LVM_LV=vg_oracle651/lv_root SYSFONT=latarcyrheb-sun16 crashkernel=auto KEYBOARDTYPE=pc KEYTABLE=us rd_NO_DM rhgb quiet

initrd /initramfs-2.6.32-431.el6.x86_64.img

[root@oracle651 linux-3.8.13-16.2.1.el6uek]#

Now copy the .config file over to the "/boot" directory as shown below. Expected output and commands are shown below.

[root@oracle651 linux-3.8.13-16.2.1.el6uek]# cp .config /boot/config-3.8.13-16.2.1.el6uek.x86_64

cp: overwrite `/boot/config-3.8.13-16.2.1.el6uek.x86_64'? y

[root@oracle651 linux-3.8.13-16.2.1.el6uek]#

Reboot the KVM guest. Be sure the config file has been named exactly as shown above before rebooting, or if your kernel version was slightly different, be sure the kernel version and format of the name of the config file is of the format "config-ver.s.ion-scst.el6.x86_64".

Next change the various Makefiles to release version by issuing “make debug2release” in the main scst dir. This will remove the debugging stuff. Do this as root.

Linux oracle651.vmem.org 3.8.13-16.2.1.el6uek.x86_64 #1 SMP Sun Aug 24 11:05:09 CDT 2014 x86_64 x86_64 x86_64 GNU/Linux

[root@oracle651 ~]# cd /home/mockbuild/scst

[root@oracle651 scst]# make debug2release | tee /home/mockbuild/debug2release.log

rm -f TAGS

cd scst && make extraclean

make[1]: Entering directory `/home/mockbuild/scst/scst'

cd src && make extraclean

make[2]: Entering directory `/home/mockbuild/scst/scst/src'

rm -f *.o *.ko .*.cmd *.mod.c .*.d .depend Modules.symvers \

Module.symvers Module.markers modules.order

rm -rf .tmp_versions

cd dev_handlers && make clean

make[3]: Entering directory `/home/mockbuild/scst/scst/src/dev_handlers'

rm -f *.o *.ko .*.cmd *.mod.c .*.d .depend Modules.symvers \

Module.symvers Module.markers modules.order

rm -rf .tmp_versions

make[3]: Leaving directory `/home/mockbuild/scst/scst/src/dev_handlers'

cd dev_handlers && make extraclean

make[3]: Entering directory `/home/mockbuild/scst/scst/src/dev_handlers'

rm -f *.o *.ko .*.cmd *.mod.c .*.d .depend Modules.symvers \

Module.symvers Module.markers modules.order

rm -rf .tmp_versions

rm -f *.orig *.rej

make[3]: Leaving directory `/home/mockbuild/scst/scst/src/dev_handlers'

rm -f *.orig *.rej

make[2]: Leaving directory `/home/mockbuild/scst/scst/src'

make[1]: Leaving directory `/home/mockbuild/scst/scst'

make[1]: Entering directory `/home/mockbuild/scst/doc'

rm -f *.txt *.html *.tex *.dvi *.ps *.pdf *.info *.lyx *.rtf

rm -f *.orig *.rej

make[1]: Leaving directory `/home/mockbuild/scst/doc'

make[1]: Entering directory `/home/mockbuild/scst/qla2x00t'

rm -f *.o *.ko .*.cmd *.mod.c .*.d .depend *~ Modules.symvers \

Module.symvers Module.markers modules.order

rm -rf .tmp_versions

rm -f *.orig *.rej

make[1]: Leaving directory `/home/mockbuild/scst/qla2x00t'

make[1]: Entering directory `/home/mockbuild/scst/qla2x00t/qla2x00-target'

rm -f *.o *.ko .*.cmd *.mod.c .*.d .depend *~ Modules.symvers \

Module.symvers Module.markers modules.order

rm -rf .tmp_versions

rm -f *.orig *.rej

make[1]: Leaving directory `/home/mockbuild/scst/qla2x00t/qla2x00-target'

make[1]: Entering directory `/home/mockbuild/scst/srpt'

make -C /lib/modules/3.8.13-16.2.1.el6uek.x86_64/build SUBDIRS=/home/mockbuild/scst/srpt/src clean

make[2]: Entering directory `/home/mockbuild/rpmbuild/BUILD/kernel-3.8.13/linux-3.8.13-16.2.1.el6uek'

make[2]: Leaving directory `/home/mockbuild/rpmbuild/BUILD/kernel-3.8.13/linux-3.8.13-16.2.1.el6uek'

rm -f src/Modules.symvers src/Module.symvers src/Module.markers \

src/modules.order

rm -f *.orig *.rej

make[1]: Leaving directory `/home/mockbuild/scst/srpt'

make[1]: Entering directory `/home/mockbuild/scst/iscsi-scst'

make -C usr extraclean

make[2]: Entering directory `/home/mockbuild/scst/iscsi-scst/usr'

rm -f *.o iscsi-scstd iscsi-scst-adm .depend*

rm -f *.orig *.rej

make[2]: Leaving directory `/home/mockbuild/scst/iscsi-scst/usr'

make -C /lib/modules/3.8.13-16.2.1.el6uek.x86_64/build SUBDIRS=/home/mockbuild/scst/iscsi-scst/kernel clean

make[2]: Entering directory `/home/mockbuild/rpmbuild/BUILD/kernel-3.8.13/linux-3.8.13-16.2.1.el6uek'

make[2]: Leaving directory `/home/mockbuild/rpmbuild/BUILD/kernel-3.8.13/linux-3.8.13-16.2.1.el6uek'

rm -f kernel/Modules.symvers kernel/Module.symvers \

kernel/Module.markers kernel/modules.order \

include/iscsi_scst_itf_ver.h \

kernel/*.orig kernel/*.rej

make[1]: Leaving directory `/home/mockbuild/scst/iscsi-scst'

make[1]: Entering directory `/home/mockbuild/scst/usr/fileio'

rm -f *.o fileio_tgt .depend*

rm -f *.orig *.rej

make[1]: Leaving directory `/home/mockbuild/scst/usr/fileio'

make[1]: Entering directory `/home/mockbuild/scst/scst_local'

make[2]: Entering directory `/home/mockbuild/rpmbuild/BUILD/kernel-3.8.13/linux-3.8.13-16.2.1.el6uek'

make[2]: Leaving directory `/home/mockbuild/rpmbuild/BUILD/kernel-3.8.13/linux-3.8.13-16.2.1.el6uek'

rm -f *.orig *.rej

make[1]: Leaving directory `/home/mockbuild/scst/scst_local'

cd scst && make debug2release

make[1]: Entering directory `/home/mockbuild/scst/scst'

cd src && make debug2release

make[2]: Entering directory `/home/mockbuild/scst/scst/src'

sed -i.aa s/"^EXTRA_CFLAGS += \-DCONFIG_SCST_EXTRACHECKS"/"#EXTRA_CFLAGS += \-DCONFIG_SCST_EXTRACHECKS"/ Makefile

sed -i.aa s/"#EXTRA_CFLAGS += \-DCONFIG_SCST_TRACING"/"EXTRA_CFLAGS += \-DCONFIG_SCST_TRACING"/ Makefile

sed -i.aa s/"^EXTRA_CFLAGS += \-DCONFIG_SCST_DEBUG -g -fno-inline -fno-inline-functions"/"#EXTRA_CFLAGS += \-DCONFIG_SCST_DEBUG -g -fno-inline -fno-inline-functions"/ Makefile

rm Makefile.aa

cd dev_handlers && make debug2release

make[3]: Entering directory `/home/mockbuild/scst/scst/src/dev_handlers'

sed -i.aa s/"^EXTRA_CFLAGS += \-DCONFIG_SCST_EXTRACHECKS"/"#EXTRA_CFLAGS += \-DCONFIG_SCST_EXTRACHECKS"/ Makefile

sed -i.aa s/"#EXTRA_CFLAGS += \-DCONFIG_SCST_TRACING"/"EXTRA_CFLAGS += \-DCONFIG_SCST_TRACING"/ Makefile

sed -i.aa s/"^EXTRA_CFLAGS += \-DCONFIG_SCST_DEBUG -g -fno-inline -fno-inline-functions"/"#EXTRA_CFLAGS += \-DCONFIG_SCST_DEBUG -g -fno-inline -fno-inline-functions"/ Makefile

rm Makefile.aa

make[3]: Leaving directory `/home/mockbuild/scst/scst/src/dev_handlers'

make[2]: Leaving directory `/home/mockbuild/scst/scst/src'

make[1]: Leaving directory `/home/mockbuild/scst/scst'

make[1]: Entering directory `/home/mockbuild/scst/qla2x00t/qla2x00-target'

sed -i.aa s/"^EXTRA_CFLAGS += \-DCONFIG_SCST_EXTRACHECKS"/"#EXTRA_CFLAGS += \-DCONFIG_SCST_EXTRACHECKS"/ Makefile

sed -i.aa s/"#EXTRA_CFLAGS += \-DCONFIG_SCST_TRACING"/"EXTRA_CFLAGS += \-DCONFIG_SCST_TRACING"/ Makefile

sed -i.aa s/"^EXTRA_CFLAGS += \-DCONFIG_SCST_DEBUG -g -fno-inline -fno-inline-functions"/"#EXTRA_CFLAGS += \-DCONFIG_SCST_DEBUG -g -fno-inline -fno-inline-functions"/ Makefile

rm Makefile.aa

make[1]: Leaving directory `/home/mockbuild/scst/qla2x00t/qla2x00-target'

make[1]: Entering directory `/home/mockbuild/scst/srpt'

make clean

make[2]: Entering directory `/home/mockbuild/scst/srpt'

make -C /lib/modules/3.8.13-16.2.1.el6uek.x86_64/build SUBDIRS=/home/mockbuild/scst/srpt/src clean

make[3]: Entering directory `/home/mockbuild/rpmbuild/BUILD/kernel-3.8.13/linux-3.8.13-16.2.1.el6uek'

make[3]: Leaving directory `/home/mockbuild/rpmbuild/BUILD/kernel-3.8.13/linux-3.8.13-16.2.1.el6uek'

rm -f src/Modules.symvers src/Module.symvers src/Module.markers \

src/modules.order

make[2]: Leaving directory `/home/mockbuild/scst/srpt'

make[1]: Leaving directory `/home/mockbuild/scst/srpt'

make[1]: Entering directory `/home/mockbuild/scst/iscsi-scst'

sed -i.aa s/"^EXTRA_CFLAGS += \-DCONFIG_SCST_EXTRACHECKS"/"#EXTRA_CFLAGS += \-DCONFIG_SCST_EXTRACHECKS"/ /home/mockbuild/scst/iscsi-scst/kernel/Makefile

sed -i.aa s/"#EXTRA_CFLAGS += \-DCONFIG_SCST_TRACING"/"EXTRA_CFLAGS += \-DCONFIG_SCST_TRACING"/ /home/mockbuild/scst/iscsi-scst/kernel/Makefile

sed -i.aa s/"^EXTRA_CFLAGS += \-DCONFIG_SCST_DEBUG -g -fno-inline -fno-inline-functions"/"#EXTRA_CFLAGS += \-DCONFIG_SCST_DEBUG -g -fno-inline -fno-inline-functions"/ /home/mockbuild/scst/iscsi-scst/kernel/Makefile

rm /home/mockbuild/scst/iscsi-scst/kernel/Makefile.aa

make[1]: Leaving directory `/home/mockbuild/scst/iscsi-scst'

make[1]: Entering directory `/home/mockbuild/scst/usr/fileio'

sed -i.aa s/"^CFLAGS += \-DEXTRACHECKS"/"#CFLAGS += \-DEXTRACHECKS"/ Makefile

sed -i.aa s/"#CFLAGS += \-DTRACING"/"CFLAGS += \-DTRACING"/ Makefile

sed -i.aa s/"^CFLAGS += \-DDEBUG -g -fno-inline -fno-inline-functions"/"#CFLAGS += \-DDEBUG -g -fno-inline -fno-inline-functions"/ Makefile

rm Makefile.aa

make[1]: Leaving directory `/home/mockbuild/scst/usr/fileio'

make[1]: Entering directory `/home/mockbuild/scst/scst_local'

sed -i.aa s/"^EXTRA_CFLAGS += \-DCONFIG_SCST_EXTRACHECKS"/"#EXTRA_CFLAGS += \-DCONFIG_SCST_EXTRACHECKS"/ Makefile

sed -i.aa s/"#EXTRA_CFLAGS += \-DCONFIG_SCST_TRACING"/"EXTRA_CFLAGS += \-DCONFIG_SCST_TRACING"/ Makefile

sed -i.aa s/"^EXTRA_CFLAGS += \-DCONFIG_SCST_DEBUG -g -fno-inline -fno-inline-functions"/"#EXTRA_CFLAGS += \-DCONFIG_SCST_DEBUG -g -fno-inline -fno-inline-functions"/ Makefile

rm Makefile.aa

make[1]: Leaving directory `/home/mockbuild/scst/scst_local'

[root@oracle651 scst]#

Now go ahead and issue “make scst scst_install” as root in the mockbuild directory’s scst folder. Commands and expected output are shown below. Do this as "root" user.

[root@oracle651 scst]# make scst scst_install | tee /home/mockbuild/scst_install.log

cd scst && make all

make[1]: Entering directory `/home/mockbuild/scst/scst'

cd src && make all

make[2]: Entering directory `/home/mockbuild/scst/scst/src'

make -C /lib/modules/3.8.13-16.2.1.el6uek.x86_64/build SUBDIRS=/home/mockbuild/scst/scst/src BUILD_DEV=m

make[3]: Entering directory `/home/mockbuild/rpmbuild/BUILD/kernel-3.8.13/linux-3.8.13-16.2.1.el6uek'

LD /home/mockbuild/scst/scst/src/built-in.o

CC [M] /home/mockbuild/scst/scst/src/scst_main.o

CC [M] /home/mockbuild/scst/scst/src/scst_targ.o

CC [M] /home/mockbuild/scst/scst/src/scst_lib.o

CC [M] /home/mockbuild/scst/scst/src/scst_sysfs.o

CC [M] /home/mockbuild/scst/scst/src/scst_mem.o

CC [M] /home/mockbuild/scst/scst/src/scst_debug.o

CC [M] /home/mockbuild/scst/scst/src/scst_pres.o

CC [M] /home/mockbuild/scst/scst/src/scst_tg.o

LD [M] /home/mockbuild/scst/scst/src/scst.o

LD /home/mockbuild/scst/scst/src/dev_handlers/built-in.o

CC [M] /home/mockbuild/scst/scst/src/dev_handlers/scst_cdrom.o

CC [M] /home/mockbuild/scst/scst/src/dev_handlers/scst_changer.o

CC [M] /home/mockbuild/scst/scst/src/dev_handlers/scst_disk.o

CC [M] /home/mockbuild/scst/scst/src/dev_handlers/scst_modisk.o

CC [M] /home/mockbuild/scst/scst/src/dev_handlers/scst_tape.o

CC [M] /home/mockbuild/scst/scst/src/dev_handlers/scst_vdisk.o

CC [M] /home/mockbuild/scst/scst/src/dev_handlers/scst_raid.o

CC [M] /home/mockbuild/scst/scst/src/dev_handlers/scst_processor.o

CC [M] /home/mockbuild/scst/scst/src/dev_handlers/scst_user.o

Building modules, stage 2.

MODPOST 10 modules

CC /home/mockbuild/scst/scst/src/dev_handlers/scst_cdrom.mod.o

CTF

LD [M] /home/mockbuild/scst/scst/src/dev_handlers/scst_cdrom.ko

CC /home/mockbuild/scst/scst/src/dev_handlers/scst_changer.mod.o

LD [M] /home/mockbuild/scst/scst/src/dev_handlers/scst_changer.ko

CC /home/mockbuild/scst/scst/src/dev_handlers/scst_disk.mod.o

LD [M] /home/mockbuild/scst/scst/src/dev_handlers/scst_disk.ko

CC /home/mockbuild/scst/scst/src/dev_handlers/scst_modisk.mod.o

LD [M] /home/mockbuild/scst/scst/src/dev_handlers/scst_modisk.ko

CC /home/mockbuild/scst/scst/src/dev_handlers/scst_processor.mod.o

LD [M] /home/mockbuild/scst/scst/src/dev_handlers/scst_processor.ko

CC /home/mockbuild/scst/scst/src/dev_handlers/scst_raid.mod.o

LD [M] /home/mockbuild/scst/scst/src/dev_handlers/scst_raid.ko

CC /home/mockbuild/scst/scst/src/dev_handlers/scst_tape.mod.o

LD [M] /home/mockbuild/scst/scst/src/dev_handlers/scst_tape.ko

CC /home/mockbuild/scst/scst/src/dev_handlers/scst_user.mod.o

LD [M] /home/mockbuild/scst/scst/src/dev_handlers/scst_user.ko

CC /home/mockbuild/scst/scst/src/dev_handlers/scst_vdisk.mod.o

LD [M] /home/mockbuild/scst/scst/src/dev_handlers/scst_vdisk.ko

CC /home/mockbuild/scst/scst/src/scst.mod.o

LD [M] /home/mockbuild/scst/scst/src/scst.ko

make[3]: Leaving directory `/home/mockbuild/rpmbuild/BUILD/kernel-3.8.13/linux-3.8.13-16.2.1.el6uek'

make[2]: Leaving directory `/home/mockbuild/scst/scst/src'

make[1]: Leaving directory `/home/mockbuild/scst/scst'

cd scst && make install

make[1]: Entering directory `/home/mockbuild/scst/scst'

cd src && make install

make[2]: Entering directory `/home/mockbuild/scst/scst/src'

make -C /lib/modules/3.8.13-16.2.1.el6uek.x86_64/build SUBDIRS=/home/mockbuild/scst/scst/src BUILD_DEV=m

make[3]: Entering directory `/home/mockbuild/rpmbuild/BUILD/kernel-3.8.13/linux-3.8.13-16.2.1.el6uek'

Building modules, stage 2.

MODPOST 10 modules

make[3]: Leaving directory `/home/mockbuild/rpmbuild/BUILD/kernel-3.8.13/linux-3.8.13-16.2.1.el6uek'

rm -f /lib/modules/3.8.13-16.2.1.el6uek.x86_64/extra/scsi_tgt.ko

install -d /lib/modules/3.8.13-16.2.1.el6uek.x86_64/extra

install -d /lib/modules/3.8.13-16.2.1.el6uek.x86_64/extra/dev_handlers

install -m 644 dev_handlers/*.ko /lib/modules/3.8.13-16.2.1.el6uek.x86_64/extra/dev_handlers

install -m 644 scst.ko /lib/modules/3.8.13-16.2.1.el6uek.x86_64/extra

install -d /usr/local/include/scst

install -m 644 ../include/scst.h /usr/local/include/scst

install -m 644 ../include/scst_sgv.h /usr/local/include/scst

install -m 644 ../include/scst_debug.h /usr/local/include/scst

install -m 644 ../include/scst_user.h /usr/local/include/scst

install -m 644 ../include/scst_const.h /usr/local/include/scst

rm -f /usr/local/include/scst/Modules.symvers

install -m 644 Module.symvers /usr/local/include/scst

/sbin/depmod -a 3.8.13-16.2.1.el6uek.x86_64

mkdir -p /var/lib/scst/pr

****************************************************************

*!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!*

*!! !!*

*!! Now don't forget to rebuild and reinstall all your !!*

*!! target drivers, custom dev handlers and necessary user !!*

*!! space applications. Otherwise, because of the versions !!*

*!! mismatch, you could have many problems and crashes. !!*

*!! See IMPORTANT note in the "Installation" section of !!*

*!! SCST's README file for more info. !!*

*!! !!*

*!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!*

****************************************************************

make[2]: Leaving directory `/home/mockbuild/scst/scst/src'

make[1]: Leaving directory `/home/mockbuild/scst/scst'

[root@oracle651 scst]#

Issue “make iscsi iscsi_install” as root in the /home/mockbuild folder. Commands and expected output are shown below.

[root@oracle651 scst]# make iscsi iscsi_install | tee /home/mockbuild/iscsi_install.log

cd iscsi-scst && make all

make[1]: Entering directory `/home/mockbuild/scst/iscsi-scst'

echo "/* Autogenerated, don't edit */" >include/iscsi_scst_itf_ver.h

echo "" >>include/iscsi_scst_itf_ver.h

echo -n "#define ISCSI_SCST_INTERFACE_VERSION " >>include/iscsi_scst_itf_ver.h

echo -n "ISCSI_VERSION_STRING \"_\" " >>include/iscsi_scst_itf_ver.h

echo "\"`sha1sum include/iscsi_scst.h|awk '{printf $1}'`\"" >>include/iscsi_scst_itf_ver.h

make -C usr SCST_INC_DIR=/home/mockbuild/scst/iscsi-scst/../scst/include

make[2]: Entering directory `/home/mockbuild/scst/iscsi-scst/usr'

cc -M -O2 -fno-inline -Wall -Wextra -Wstrict-prototypes -Wno-sign-compare -Wimplicit-function-declaration -Wno-unused-parameter -Wno-missing-field-initializers -g -I../include -I/home/mockbuild/scst/iscsi-scst/../scst/include -D_GNU_SOURCE iscsid.c iscsi_scstd.c conn.c session.c target.c message.c ctldev.c log.c chap.c event.c param.c config.c isns.c md5.c sha1.c >.depend_d

cc -c -o iscsid.o -O2 -fno-inline -Wall -Wextra -Wstrict-prototypes -Wno-sign-compare -Wimplicit-function-declaration -Wno-unused-parameter -Wno-missing-field-initializers -g -I../include -I/home/mockbuild/scst/iscsi-scst/../scst/include -D_GNU_SOURCE iscsid.c

cc -c -o iscsi_scstd.o -O2 -fno-inline -Wall -Wextra -Wstrict-prototypes -Wno-sign-compare -Wimplicit-function-declaration -Wno-unused-parameter -Wno-missing-field-initializers -g -I../include -I/home/mockbuild/scst/iscsi-scst/../scst/include -D_GNU_SOURCE iscsi_scstd.c

cc -c -o conn.o -O2 -fno-inline -Wall -Wextra -Wstrict-prototypes -Wno-sign-compare -Wimplicit-function-declaration -Wno-unused-parameter -Wno-missing-field-initializers -g -I../include -I/home/mockbuild/scst/iscsi-scst/../scst/include -D_GNU_SOURCE conn.c

cc -c -o session.o -O2 -fno-inline -Wall -Wextra -Wstrict-prototypes -Wno-sign-compare -Wimplicit-function-declaration -Wno-unused-parameter -Wno-missing-field-initializers -g -I../include -I/home/mockbuild/scst/iscsi-scst/../scst/include -D_GNU_SOURCE session.c

cc -c -o target.o -O2 -fno-inline -Wall -Wextra -Wstrict-prototypes -Wno-sign-compare -Wimplicit-function-declaration -Wno-unused-parameter -Wno-missing-field-initializers -g -I../include -I/home/mockbuild/scst/iscsi-scst/../scst/include -D_GNU_SOURCE target.c

cc -c -o message.o -O2 -fno-inline -Wall -Wextra -Wstrict-prototypes -Wno-sign-compare -Wimplicit-function-declaration -Wno-unused-parameter -Wno-missing-field-initializers -g -I../include -I/home/mockbuild/scst/iscsi-scst/../scst/include -D_GNU_SOURCE message.c

cc -c -o ctldev.o -O2 -fno-inline -Wall -Wextra -Wstrict-prototypes -Wno-sign-compare -Wimplicit-function-declaration -Wno-unused-parameter -Wno-missing-field-initializers -g -I../include -I/home/mockbuild/scst/iscsi-scst/../scst/include -D_GNU_SOURCE ctldev.c

cc -c -o log.o -O2 -fno-inline -Wall -Wextra -Wstrict-prototypes -Wno-sign-compare -Wimplicit-function-declaration -Wno-unused-parameter -Wno-missing-field-initializers -g -I../include -I/home/mockbuild/scst/iscsi-scst/../scst/include -D_GNU_SOURCE log.c

cc -c -o chap.o -O2 -fno-inline -Wall -Wextra -Wstrict-prototypes -Wno-sign-compare -Wimplicit-function-declaration -Wno-unused-parameter -Wno-missing-field-initializers -g -I../include -I/home/mockbuild/scst/iscsi-scst/../scst/include -D_GNU_SOURCE chap.c

cc -c -o event.o -O2 -fno-inline -Wall -Wextra -Wstrict-prototypes -Wno-sign-compare -Wimplicit-function-declaration -Wno-unused-parameter -Wno-missing-field-initializers -g -I../include -I/home/mockbuild/scst/iscsi-scst/../scst/include -D_GNU_SOURCE event.c

cc -c -o param.o -O2 -fno-inline -Wall -Wextra -Wstrict-prototypes -Wno-sign-compare -Wimplicit-function-declaration -Wno-unused-parameter -Wno-missing-field-initializers -g -I../include -I/home/mockbuild/scst/iscsi-scst/../scst/include -D_GNU_SOURCE param.c

cc -c -o config.o -O2 -fno-inline -Wall -Wextra -Wstrict-prototypes -Wno-sign-compare -Wimplicit-function-declaration -Wno-unused-parameter -Wno-missing-field-initializers -g -I../include -I/home/mockbuild/scst/iscsi-scst/../scst/include -D_GNU_SOURCE config.c

cc -c -o isns.o -O2 -fno-inline -Wall -Wextra -Wstrict-prototypes -Wno-sign-compare -Wimplicit-function-declaration -Wno-unused-parameter -Wno-missing-field-initializers -g -I../include -I/home/mockbuild/scst/iscsi-scst/../scst/include -D_GNU_SOURCE isns.c

cc -c -o md5.o -O2 -fno-inline -Wall -Wextra -Wstrict-prototypes -Wno-sign-compare -Wimplicit-function-declaration -Wno-unused-parameter -Wno-missing-field-initializers -g -I../include -I/home/mockbuild/scst/iscsi-scst/../scst/include -D_GNU_SOURCE md5.c

cc -c -o sha1.o -O2 -fno-inline -Wall -Wextra -Wstrict-prototypes -Wno-sign-compare -Wimplicit-function-declaration -Wno-unused-parameter -Wno-missing-field-initializers -g -I../include -I/home/mockbuild/scst/iscsi-scst/../scst/include -D_GNU_SOURCE sha1.c

cc iscsid.o iscsi_scstd.o conn.o session.o target.o message.o ctldev.o log.o chap.o event.o param.o config.o isns.o md5.o sha1.o -o iscsi-scstd

cc -M -O2 -fno-inline -Wall -Wextra -Wstrict-prototypes -Wno-sign-compare -Wimplicit-function-declaration -Wno-unused-parameter -Wno-missing-field-initializers -g -I../include -I/home/mockbuild/scst/iscsi-scst/../scst/include -D_GNU_SOURCE iscsi_adm.c param.c >.depend_adm

cc -c -o iscsi_adm.o -O2 -fno-inline -Wall -Wextra -Wstrict-prototypes -Wno-sign-compare -Wimplicit-function-declaration -Wno-unused-parameter -Wno-missing-field-initializers -g -I../include -I/home/mockbuild/scst/iscsi-scst/../scst/include -D_GNU_SOURCE iscsi_adm.c

cc iscsi_adm.o param.o -o iscsi-scst-adm

make[2]: Leaving directory `/home/mockbuild/scst/iscsi-scst/usr'

cp /home/mockbuild/scst/iscsi-scst/../scst/src/Module.symvers kernel/

make -C /lib/modules/3.8.13-16.2.1.el6uek.x86_64/build SCST_INC_DIR=/home/mockbuild/scst/iscsi-scst/../scst/include SUBDIRS=/home/mockbuild/scst/iscsi-scst/kernel modules

make[2]: Entering directory `/home/mockbuild/rpmbuild/BUILD/kernel-3.8.13/linux-3.8.13-16.2.1.el6uek'

CC [M] /home/mockbuild/scst/iscsi-scst/kernel/iscsi.o

CC [M] /home/mockbuild/scst/iscsi-scst/kernel/nthread.o

CC [M] /home/mockbuild/scst/iscsi-scst/kernel/config.o

CC [M] /home/mockbuild/scst/iscsi-scst/kernel/digest.o

CC [M] /home/mockbuild/scst/iscsi-scst/kernel/conn.o

CC [M] /home/mockbuild/scst/iscsi-scst/kernel/session.o

CC [M] /home/mockbuild/scst/iscsi-scst/kernel/target.o

CC [M] /home/mockbuild/scst/iscsi-scst/kernel/event.o

CC [M] /home/mockbuild/scst/iscsi-scst/kernel/param.o

LD [M] /home/mockbuild/scst/iscsi-scst/kernel/iscsi-scst.o

Building modules, stage 2.

MODPOST 1 modules

CC /home/mockbuild/scst/iscsi-scst/kernel/iscsi-scst.mod.o

CTF

LD [M] /home/mockbuild/scst/iscsi-scst/kernel/iscsi-scst.ko

make[2]: Leaving directory `/home/mockbuild/rpmbuild/BUILD/kernel-3.8.13/linux-3.8.13-16.2.1.el6uek'

make[1]: Leaving directory `/home/mockbuild/scst/iscsi-scst'

cd iscsi-scst && make DESTDIR= install

make[1]: Entering directory `/home/mockbuild/scst/iscsi-scst'

make -C usr SCST_INC_DIR=/home/mockbuild/scst/iscsi-scst/../scst/include

make[2]: Entering directory `/home/mockbuild/scst/iscsi-scst/usr'

make[2]: Nothing to be done for `all'.

make[2]: Leaving directory `/home/mockbuild/scst/iscsi-scst/usr'

cp /home/mockbuild/scst/iscsi-scst/../scst/src/Module.symvers kernel/

make -C /lib/modules/3.8.13-16.2.1.el6uek.x86_64/build SCST_INC_DIR=/home/mockbuild/scst/iscsi-scst/../scst/include SUBDIRS=/home/mockbuild/scst/iscsi-scst/kernel modules

make[2]: Entering directory `/home/mockbuild/rpmbuild/BUILD/kernel-3.8.13/linux-3.8.13-16.2.1.el6uek'

Building modules, stage 2.

MODPOST 1 modules

make[2]: Leaving directory `/home/mockbuild/rpmbuild/BUILD/kernel-3.8.13/linux-3.8.13-16.2.1.el6uek'

removed `/usr/local/sbin/iscsi-scstd'

`usr/iscsi-scstd' -> `/usr/local/sbin/iscsi-scstd'

removed `/usr/local/man/man5/iscsi-scstd.conf.5'

`doc/manpages/iscsi-scstd.conf.5' -> `/usr/local/man/man5/iscsi-scstd.conf.5'

removed `/usr/local/man/man8/iscsi-scstd.8'

`doc/manpages/iscsi-scstd.8' -> `/usr/local/man/man8/iscsi-scstd.8'

removed `/usr/local/sbin/iscsi-scst-adm'

`usr/iscsi-scst-adm' -> `/usr/local/sbin/iscsi-scst-adm'

removed `/usr/local/man/man8/iscsi-scst-adm.8'

`doc/manpages/iscsi-scst-adm.8' -> `/usr/local/man/man8/iscsi-scst-adm.8'

`kernel/iscsi-scst.ko' -> `/lib/modules/3.8.13-16.2.1.el6uek.x86_64/extra/iscsi-scst.ko'

/sbin/depmod -aq 3.8.13-16.2.1.el6uek.x86_64

make[1]: Leaving directory `/home/mockbuild/scst/iscsi-scst'

[root@oracle651 scst]#

Finally, issue “make scstadm scstadm_install” as well. This finalizes the compiling part. Do this as root. Commands and expected output are shown below.

[root@oracle651 scst]# make scstadm scstadm_install | tee /home/mockbuild/scstadm.log

cd scstadmin && make all

make[1]: Entering directory `/home/mockbuild/scst/scstadmin'

cd scstadmin && make all

make[2]: Entering directory `/home/mockbuild/scst/scstadmin/scstadmin.sysfs'

Checking if your kit is complete...

Looks good

Writing Makefile for SCST-SCST

make -C scst-0.9.10

make[3]: Entering directory `/home/mockbuild/scst/scstadmin/scstadmin.sysfs/scst-0.9.10'

cp lib/SCST/SCST.pm blib/lib/SCST/SCST.pm

Manifying blib/man3/SCST::SCST.3pm

make[3]: Leaving directory `/home/mockbuild/scst/scstadmin/scstadmin.sysfs/scst-0.9.10'

make[2]: Leaving directory `/home/mockbuild/scst/scstadmin/scstadmin.sysfs'

make[1]: Leaving directory `/home/mockbuild/scst/scstadmin'

cd scstadmin && make install

make[1]: Entering directory `/home/mockbuild/scst/scstadmin'

cd scstadmin && make install

make[2]: Entering directory `/home/mockbuild/scst/scstadmin/scstadmin.sysfs'

Writing Makefile for SCST-SCST

make -C scst-0.9.10

make[3]: Entering directory `/home/mockbuild/scst/scstadmin/scstadmin.sysfs/scst-0.9.10'

Skip blib/lib/SCST/SCST.pm (unchanged)

Manifying blib/man3/SCST::SCST.3pm

make[3]: Leaving directory `/home/mockbuild/scst/scstadmin/scstadmin.sysfs/scst-0.9.10'

make -C scst-0.9.10 install DESTDIR=

make[3]: Entering directory `/home/mockbuild/scst/scstadmin/scstadmin.sysfs/scst-0.9.10'

Manifying blib/man3/SCST::SCST.3pm

Installing /usr/local/share/man/man3/SCST::SCST.3pm

Appending installation info to /usr/lib64/perl5/perllocal.pod

make[3]: Leaving directory `/home/mockbuild/scst/scstadmin/scstadmin.sysfs/scst-0.9.10'

install -d /usr/local/sbin

install -m 755 scstadmin /usr/local/sbin

make[2]: Leaving directory `/home/mockbuild/scst/scstadmin/scstadmin.sysfs'

for m in scstadmin/man*/*.[1-9]; do \

if [ -e $m ]; then \

d=/usr/local/man/$(basename $(dirname $m)); \

if install -d $d; then \

install -m 644 $m $d; \

fi \

done

install -d /etc/init.d

tmp=/tmp/scst-initd-$$ && init.d/scst >$tmp && \

install -m 755 $tmp /etc/init.d/scst && \

rm -f $tmp

if [ ! -e /etc/default/scst ]; then \

mkdir -p /etc/default; \

install -m 755 default/scst /etc/default; \

fi

for s in iscsi-scst qla2x00t; do \

{ if [ -n "" ]; then chr="chroot "; else chr=""; fi; if type systemctl >/dev/null 2>&1; then $chr systemctl disable "$s.service"; elif type chkconfig >/dev/null 2>&1; then $chr chkconfig --del "$s"; elif type update-rc.d >/dev/null 2>&1; then $chr update-rc.d -f "$s" remove; elif type rc-update >/dev/null 2>&1; then $chr rc-update del "$s"; elif type /usr/lib/lsb/remove_initd >/dev/null 2>&1; then $chr /usr/lib/lsb/remove_initd "$s"; fi; rm -f /etc/init.d/$s; } >/dev/null 2>&1; \

done

echo

echo 'If you want SCST to start automatically at boot time, run' \

'the following command:'

If you want SCST to start automatically at boot time, run the following command:

if [ -n "" ]; then chr="chroot "; else chr=""; fi; if type systemctl >/dev/null 2>&1; then echo $chr systemctl enable "scst.service"; elif type chkconfig >/dev/null 2>&1; then echo $chr chkconfig --add "scst"; elif type update-rc.d >/dev/null 2>&1; then echo $chr update-rc.d "scst" defaults; elif type rc-update >/dev/null 2>&1; then echo $chr rc-update add "scst" default; elif type /usr/lib/lsb/install_initd >/dev/null 2>&1; then echo $chr /usr/lib/lsb/install_initd "scst"; fi

chkconfig --add scst

echo

make[1]: Leaving directory `/home/mockbuild/scst/scstadmin'

[root@oracle651 scst]#

Now set SCST to start on boot. Do this as root. Commands and expected output are shown below.

[root@oracle651 scst]# /usr/lib/lsb/install_initd scst

[root@oracle651 scst]#

Now test start SCST manually. There is no configuration file yet for SCST, so it will give a warning as shown below.

[root@oracle651 scst]# service scst start

Loading and configuring SCSTSCST configuration file /etc/scst.conf missing

[ OK ]

[root@oracle651 scst]#

Now lock down the kernel so that the custom kernel just built will not be updated. This requires updated the /etc/yum.conf file as shown below. The line that needs to be added is shown in bold below "exclude=kernel*". This prevents the kernel from being updated.

[root@oracle651 yum.repos.d]# cat /etc/yum.conf

[main]

cachedir=/var/cache/yum/$basearch/$releasever

keepcache=0

debuglevel=2

logfile=/var/log/yum.log

exactarch=1

obsoletes=1

gpgcheck=1

plugins=1

installonly_limit=3

exclude=kernel*

# This is the default, if you make this bigger yum won't see if the metadata

# is newer on the remote and so you'll "gain" the bandwidth of not having to

# download the new metadata and "pay" for it by yum not having correct

# information.

# It is esp. important, to have correct metadata, for distributions like

# Fedora which don't keep old packages around. If you don't like this checking

# interupting your command line usage, it's much better to have something

# manually check the metadata once an hour (yum-updatesd will do this).

# metadata_expire=90m

# PUT YOUR REPOS HERE OR IN separate files named file.repo

# in /etc/yum.repos.d

[root@oracle651 yum.repos.d]#

Reboot and verify that SCST starts up automatically and that all targets, groups, and LUNs are active as shown below.

[root@oracle651 ~]# uname -a

Linux oracle651.vmem.org 3.8.13-16.2.1.el6uek.x86_64 #1 SMP Sun Aug 24 11:05:09 CDT 2014 x86_64 x86_64 x86_64 GNU/Linux

[root@oracle651 ~]# service scst status

SCST status: OK

[root@oracle651 ~]# ps -ef | grep scst

root 1811 2 1 11:35 ? 00:00:00 [scst_uid]

root 1814 2 0 11:35 ? 00:00:00 [scstd0]

root 1815 2 0 11:35 ? 00:00:00 [scstd1]

root 1816 2 0 11:35 ? 00:00:00 [scstd2]

root 1817 2 0 11:35 ? 00:00:00 [scstd3]

root 1818 2 0 11:35 ? 00:00:00 [scstd4]

root 1819 2 0 11:35 ? 00:00:00 [scstd5]

root 1820 2 0 11:35 ? 00:00:00 [scstd6]

root 1821 2 0 11:35 ? 00:00:00 [scstd7]

root 1822 2 0 11:35 ? 00:00:00 [scst_initd]

root 1824 2 0 11:35 ? 00:00:00 [scst_mgmtd]

root 1873 1 0 11:35 ? 00:00:00 /usr/local/sbin/iscsi-scstd

root 2532 2489 0 11:36 pts/0 00:00:00 grep scst

[root@oracle651 ~]# scstadmin -list_group

Collecting current configuration: done.

Driver: iscsi

Target: iqn.2014-08.org.vmem:oracle651.san.asm.luns

Driver/target 'iscsi/iqn.2014-08.org.vmem:oracle651.san.asm.luns' has no associated LUNs.

Group: oracle631

Assigned LUNs:

LUN Device

-----------------

0 AsmSys631_01

1 AsmDat631_01

10 AsmRec631_02

11 AsmRec631_03

12 AsmRec631_04

13 AsmRec631_05

14 AsmRec631_06

15 AsmRec631_07

16 AsmRec631_08

2 AsmDat631_02

20 AsmSlb631_01

21 AsmSlb631_02

22 AsmSlb631_03

23 AsmSlb631_04

24 AsmSlb631_05

25 AsmSlb631_06

26 AsmSlb631_07

27 AsmSlb631_08

3 AsmDat631_03

4 AsmDat631_04

5 AsmDat631_05

6 AsmDat631_06

7 AsmDat631_07

8 AsmDat631_08

9 AsmRec631_01

Assigned Initiators:

Initiator

----------------------------------

iqn.2014-08.org.vmem:oracle631.asm

Group: oracle632

Assigned LUNs:

LUN Device

-----------------

0 AsmSys632_01

1 AsmDat632_01

10 AsmRec632_02

11 AsmRec632_03

12 AsmRec632_04

13 AsmRec632_05

14 AsmRec632_06

15 AsmRec632_07

16 AsmRec632_08

17 AsmTmp632_01

18 AsmTmp632_02

19 AsmTmp632_03

2 AsmDat632_02

20 AsmTmp632_04

21 AsmTmp632_05

22 AsmTmp632_06

23 AsmTmp632_07

24 AsmTmp632_08

25 AsmSlb632_01

26 AsmSlb632_02

27 AsmSlb632_03

28 AsmSlb632_04

29 AsmSlb632_05

3 AsmDat632_03

30 AsmSlb632_06

31 AsmSlb632_07

32 AsmSlb632_08

4 AsmDat632_04

5 AsmDat632_05

6 AsmDat632_06

7 AsmDat632_07

8 AsmDat632_08

9 AsmRec632_01

Assigned Initiators:

Initiator

----------------------------------

iqn.2014-08.org.vmem:oracle632.asm

All done.

[root@oracle651 ~]#

Configure DTrace

In ULN, enable the required repos as shown in the screenshot below. Save the ULN subscriptions.

Perform a yum update as shown below.

gstanden@vmem1:~$ ssh root@10.207.39.74

root@10.207.39.74's password:

Last login: Sat Aug 23 17:47:51 2014

[root@oracle651 ~]# clear

[root@oracle651 ~]# yum update

Loaded plugins: refresh-packagekit, rhnplugin, security

This system is receiving updates from ULN.

ol6_x86_64_Dtrace_BETA | 1.2 kB 00:00

ol6_x86_64_Dtrace_BETA/primary | 1.1 MB 00:01

ol6_x86_64_Dtrace_BETA 45/45

ol6_x86_64_Dtrace_latest | 1.2 kB 00:00

ol6_x86_64_Dtrace_userspace_latest | 1.2 kB 00:00

ol6_x86_64_Dtrace_userspace_latest/primary | 2.0 kB 00:00

ol6_x86_64_Dtrace_userspace_latest 8/8

ol6_x86_64_UEKR3_latest | 1.2 kB 00:00

ol6_x86_64_ksplice | 1.2 kB 00:00

ol6_x86_64_latest | 1.4 kB 00:00

Setting up Update Process

Resolving Dependencies

--> Running transaction check

---> Package dtrace-utils.x86_64 0:0.3.2-1.el6 will be updated

---> Package dtrace-utils.x86_64 0:0.4.4-2.el6 will be an update

--> Processing Dependency: dtrace-modules-headers for package: dtrace-utils-0.4.4-2.el6.x86_64

---> Package glibc.i686 0:2.12-1.132.el6_5.2 will be updated

---> Package glibc.x86_64 0:2.12-1.132.el6_5.2 will be updated

---> Package glibc.i686 0:2.12-1.132.el6_5.3 will be an update

---> Package glibc.x86_64 0:2.12-1.132.el6_5.3 will be an update

---> Package glibc-common.x86_64 0:2.12-1.132.el6_5.2 will be updated

---> Package glibc-common.x86_64 0:2.12-1.132.el6_5.3 will be an update

---> Package glibc-devel.x86_64 0:2.12-1.132.el6_5.2 will be updated

---> Package glibc-devel.x86_64 0:2.12-1.132.el6_5.3 will be an update

---> Package glibc-headers.x86_64 0:2.12-1.132.el6_5.2 will be updated

---> Package glibc-headers.x86_64 0:2.12-1.132.el6_5.3 will be an update

---> Package grubby.x86_64 0:7.0.15-5.0.2.el6 will be updated

---> Package grubby.x86_64 0:7.0.15-5.0.4.el6 will be an update

---> Package hwloc.x86_64 0:1.5-1.el6 will be updated

---> Package hwloc.x86_64 0:1.5-2.el6_5 will be an update

---> Package openssl.x86_64 0:1.0.1e-16.el6_5.14 will be updated

---> Package openssl.x86_64 0:1.0.1e-16.el6_5.15 will be an update

---> Package openssl-devel.x86_64 0:1.0.1e-16.el6_5.14 will be updated

---> Package openssl-devel.x86_64 0:1.0.1e-16.el6_5.15 will be an update

--> Running transaction check

---> Package dtrace-modules-3.8.13-16.2.1.el6uek-headers.x86_64 0:0.4.1-3.el6 will be installed

--> Finished Dependency Resolution

Dependencies Resolved

==========================================================================================================================================================================

Package Arch Version Repository Size

==========================================================================================================================================================================

Updating:

dtrace-utils x86_64 0.4.4-2.el6 ol6_x86_64_Dtrace_userspace_latest 262 k

glibc i686 2.12-1.132.el6_5.3 ol6_x86_64_latest 4.3 M

glibc x86_64 2.12-1.132.el6_5.3 ol6_x86_64_latest 3.8 M

glibc-common x86_64 2.12-1.132.el6_5.3 ol6_x86_64_latest 14 M

glibc-devel x86_64 2.12-1.132.el6_5.3 ol6_x86_64_latest 977 k

glibc-headers x86_64 2.12-1.132.el6_5.3 ol6_x86_64_latest 608 k

grubby x86_64 7.0.15-5.0.4.el6 ol6_x86_64_latest 43 k

hwloc x86_64 1.5-2.el6_5 ol6_x86_64_latest 1.4 M

openssl x86_64 1.0.1e-16.el6_5.15 ol6_x86_64_latest 1.5 M

openssl-devel x86_64 1.0.1e-16.el6_5.15 ol6_x86_64_latest 1.2 M

Installing for dependencies:

dtrace-modules-3.8.13-16.2.1.el6uek-headers x86_64 0.4.1-3.el6 ol6_x86_64_UEKR3_latest 27 k

Transaction Summary

==========================================================================================================================================================================

Install 1 Package(s)

Upgrade 10 Package(s)

Total download size: 28 M

Is this ok [y/N]:

Confirm yes and install the updates. The successful completion message is shown below.

Dependency Installed:

dtrace-modules-3.8.13-16.2.1.el6uek-headers.x86_64 0:0.4.1-3.el6

Updated:

dtrace-utils.x86_64 0:0.4.4-2.el6 glibc.i686 0:2.12-1.132.el6_5.3 glibc.x86_64 0:2.12-1.132.el6_5.3 glibc-common.x86_64 0:2.12-1.132.el6_5.3

glibc-devel.x86_64 0:2.12-1.132.el6_5.3 glibc-headers.x86_64 0:2.12-1.132.el6_5.3 grubby.x86_64 0:7.0.15-5.0.4.el6 hwloc.x86_64 0:1.5-2.el6_5

openssl.x86_64 0:1.0.1e-16.el6_5.15 openssl-devel.x86_64 0:1.0.1e-16.el6_5.15

Complete!

[root@oracle651 ~]#

All that remains now is to reboot and test that it starts. Also, a configuration file must be created, which will be the next step. There was not a lot of documentation on the web about how to create a iSCSI SAN with SCST but I found a couple of very helpful posts here and here.

Configuration of a simple 4K flash-aware SCST iSCSI SAN as shown below on the oracle651 KVM guest.

[root@oracle651 ~]# modprobe scst

[root@oracle651 ~]# modprobe scst_vdisk

[root@oracle651 ~]# modprobe iscsi-scst

[root@oracle651 ~]# scstadmin -add_target iqn.2014-08.oracle.asm.storage:luns -driver iscsi

[root@oracle651 ~]# scstadmin -open_dev disk0 -handler vdisk_blockio -attributes filename=/dev/vdc,blocksize=4096

[root@oracle651 ~]# scstadmin -open_dev disk1 -handler vdisk_blockio -attributes filename=/dev/vdb

[root@oracle651 ~]# scstadmin -add_lun 0 -driver iscsi -target iqn.2014-08.oracle.asm.storage:luns -device disk0

[root@oracle651 ~]# scstadmin -add_lun 1 -driver iscsi -target iqn.2014-08.oracle.asm.storage:luns -device disk1

[root@oracle651 ~]# scstadmin -enable_target iqn.2014-08.oracle.asm.storage:luns -driver iscs

[root@oracle651 ~]# scstadmin -set_drv_attr iscsi -attributes enabled=1

[root@oracle651 ~]# scstadmin -write_config /etc/scst.conf

The /etc/scst.conf file generated by the above command is shown below. Notice the "blocksize 4096" parameter which can be used.

[root@oracle651 ~]# cat /etc/scst.conf

# Automatically generated by SCST Configurator v2.2.1.

HANDLER vdisk_blockio {

DEVICE disk0 {

blocksize 4096

filename /dev/vdc

}

DEVICE disk1 {

filename /dev/vdb

}

TARGET_DRIVER iscsi {

enabled 1

TARGET iqn.2014-08.oracle.asm.storage:luns {

cpu_mask 3

enabled 1

rel_tgt_id 1

LUN 0 disk0

LUN 1 disk1

}

[root@oracle651 ~]# cat /etc/sysconfig/selinux

In order to get the initiator over on the iSCSI client on oracle631 KVM guest to see the portal, stop iptables and disable it on reboot as shown below and also disable selinux as shown below and turn off iptables on the iSCSI SAN host.

[root@oracle651 ~]# cat /etc/sysconfig/selinux

# This file controls the state of SELinux on the system.

# SELINUX= can take one of these three values:

# enforcing - SELinux security policy is enforced.

# permissive - SELinux prints warnings instead of enforcing.

# disabled - No SELinux policy is loaded.

SELINUX=disabled

# SELINUXTYPE= can take one of these two values:

# targeted - Targeted processes are protected,

# mls - Multi Level Security protection.

SELINUXTYPE=targeted

[root@oracle651 ~]# service iptables stop

[root@oracle651 ~]# chkconfig iptables off

[root@oracle651 ~]# reboot

Finally, use iscsiadm on the oracle631 KVM guest to discover and configure the access to the SCST SAN on oracle651 KVM guest. The iscsi-initiator utils are installed as shown below, and then the connection to the SAN is configured as shown.

[root@oracle631 ~]# yum install iscsi-initiator-utils

[root@oracle631 ~]# iscsiadm --mode discovery --type sendtargets --portal 10.207.39.74

10.207.39.74:3260,1 iqn.2014-08.oracle.asm.storage:luns

[root@oracle631 ~]# iscsiadm -m node -T iqn.2014-08.oracle.asm.storage:luns -p 10.207.39.74 --op update -n node.startup -v automatic

[root@oracle631 ~]# iscsiadm --mode node --targetname iqn.2014-08.oracle.asm.storage:luns --portal 10.207.39.74 --login

Logging in to [iface: default, target: iqn.2014-08.oracle.asm.storage:luns, portal: 10.207.39.74,3260] (multiple)

Login to [iface: default, target: iqn.2014-08.oracle.asm.storage:luns, portal: 10.207.39.74,3260] successful.

[root@oracle631 ~]#

Here are the LUNs over on the client side and show of their characteristics on oracle631 host as shown below. One very interesting and intriguing feature is that SCST sees the 512 emulated LUN on oracle631 KVM guest (the client) for what it really is i.e (512 logical / 4096 physical) even though the virtual disk on the oracle651 KVM guest (the SAN) sees that LUN as (512 logical / 512 physical), as shown below.

[root@oracle631 ~]# ls -lrt /dev/sd*

brw-rw---- 1 root disk 8, 0 Aug 11 20:11 /dev/sda

brw-rw---- 1 root disk 8, 16 Aug 11 20:11 /dev/sdb

[root@oracle631 ~]# scsi_id -g -u -d /dev/sda

26162356632353435

[root@oracle631 ~]# scsi_id -g -u -d /dev/sdb

26238666462643332

[root@oracle631 ~]# fdisk -l /dev/sda

Note: sector size is 4096 (not 512)

Disk /dev/sda: 10.7 GB, 10737418240 bytes

64 heads, 32 sectors/track, 1280 cylinders

Units = cylinders of 2048 * 4096 = 8388608 bytes

Sector size (logical/physical): 4096 bytes / 4096 bytes

I/O size (minimum/optimal): 4096 bytes / 524288 bytes

Disk identifier: 0x00000000

[root@oracle631 ~]# fdisk -l /dev/sdb

Disk /dev/sdb: 10.7 GB, 10737418240 bytes

64 heads, 32 sectors/track, 10240 cylinders

Units = cylinders of 2048 * 512 = 1048576 bytes

Sector size (logical/physical): 512 bytes / 4096 bytes <-- Notice

I/O size (minimum/optimal): 4096 bytes / 524288 bytes

Disk identifier: 0x00000000

[root@oracle631 ~]#

Here are the properties of the same LUNs over on the SAN oracle651 as shown below.

[root@oracle651 ~]# cat /etc/scst.conf

# Automatically generated by SCST Configurator v2.2.1.

HANDLER vdisk_blockio {

DEVICE disk0 {

blocksize 4096

filename /dev/vdc

}

DEVICE disk1 {

filename /dev/vdb

}

TARGET_DRIVER iscsi {

enabled 1

TARGET iqn.2014-08.oracle.asm.storage:luns {

cpu_mask 3

enabled 1

rel_tgt_id 1

LUN 0 disk0

LUN 1 disk1

}

[root@oracle651 ~]# fdisk -l /dev/vdc

Note: sector size is 4096 (not 512)

Disk /dev/vdc: 10.7 GB, 10737418240 bytes

16 heads, 56 sectors/track, 2925 cylinders

Units = cylinders of 896 * 4096 = 3670016 bytes

Sector size (logical/physical): 4096 bytes / 4096 bytes

I/O size (minimum/optimal): 4096 bytes / 4096 bytes

Disk identifier: 0x00000000

[root@oracle651 ~]# fdisk -l /dev/vdb

Disk /dev/vdb: 10.7 GB, 10737418240 bytes

16 heads, 63 sectors/track, 20805 cylinders

Units = cylinders of 1008 * 512 = 516096 bytes

Sector size (logical/physical): 512 bytes / 512 bytes <-- Notice

I/O size (minimum/optimal): 512 bytes / 512 bytes

Disk identifier: 0x00000000

[root@oracle651 ~]#

Recall that the blocksize defaults in KVM to (logical/physical) 512 bytes / 512 bytes unless it is set otherwise in the XML file of the KVM guest, as shown below i the XML file that defines oracle651 the KVM guest which is the SAN provider. Looking at the end of the XML file at bottom it can be seen that only disk2 has been set to specified blocksizes. Therefore, disk1 is presented to the KVM guest oracle651 as a 512 bytes / 512 bytes LUN by default. Thus it is surprising that SCST sees it for what it is 'physically' which is a LUN sitting on a consumer-grade SSD (Crucial M550) and flash always uses 4096 byte physical sectors, and 512 byte logical emulation. The LUN as seen by oracle631 the iscsi client sees it as what it 'really' is, 512 bytes / 4096 bytes (logical/physical). More investigation is warranted to understand this better, but it would seem that SCST is ignoring the KVM setting and reading the storage itself, and that the blocksize setting in the SCST configuration file refers to the logical block size, and thus set the logical block size of the other LUN to 4096 bytes (the LUN that was presented by KVM on oracle631 as (logical/physical) 4096 bytes / 4096 bytes.

gstanden@vmem1:~$ virsh dumpxml oracle651

<name>oracle651</name>

<partition>/machine</partition>

</resource>

<os>

</os>

<acpi/>

<apic/>

<pae/>

</features>

<on_poweroff>destroy</on_poweroff>

<on_reboot>restart</on_reboot>

<on_crash>restart</on_crash>

<emulator>/usr/bin/kvm-spice</emulator>

</disk>

</disk>

</disk>

</disk>

</controller>

</controller>

</controller>

</virtualport>

</interface>

</serial>

</console>

</graphics>

</sound>

<video>

</video>

</memballoon>

</devices>

<label>libvirt-9a017de6-d362-4269-90fe-09331460aca0</label>

<imagelabel>libvirt-9a017de6-d362-4269-90fe-09331460aca0</imagelabel>

</seclabel>

<qemu:commandline>

<qemu:arg value='-set'/>

<qemu:arg value='device.virtio-disk0.scsi=off'/>

<qemu:arg value='-set'/>

<qemu:arg value='device.virtio-disk0.config-wce=off'/>

<qemu:arg value='-set'/>

<qemu:arg value='device.virtio-disk0.x-data-plane=on'/>

<qemu:arg value='-set'/>

<qemu:arg value='device.virtio-disk1.scsi=off'/>

<qemu:arg value='-set'/>

<qemu:arg value='device.virtio-disk1.config-wce=off'/>

<qemu:arg value='-set'/>

<qemu:arg value='device.virtio-disk1.x-data-plane=on'/>

<qemu:arg value='-set'/>

<qemu:arg value='device.virtio-disk2.scsi=off'/>

<qemu:arg value='-set'/>

<qemu:arg value='device.virtio-disk2.config-wce=off'/>

<qemu:arg value='-set'/>

<qemu:arg value='device.virtio-disk2.x-data-plane=on'/>

<qemu:arg value='-set'/>

<qemu:arg value='device.virtio-disk2.logical_block_size=4096'/>

<qemu:arg value='-set'/>

<qemu:arg value='device.virtio-disk2.physical_block_size=4096'/>

</qemu:commandline>

</domain>

gstanden@vmem1:~$

Some useful additonal info about the LUNs over on the oracle651 client.

[root@oracle631 disk]# ls -lrt by-path | grep asm

lrwxrwxrwx 1 root root 9 Aug 11 20:11 ip-10.207.39.74:3260-iscsi-iqn.2014-08.oracle.asm.storage:luns-lun-0 -> ../../sda

lrwxrwxrwx 1 root root 9 Aug 11 20:11 ip-10.207.39.74:3260-iscsi-iqn.2014-08.oracle.asm.storage:luns-lun-1 -> ../../sdb

[root@oracle631 disk]# ls -lrt by-id | grep scsi

lrwxrwxrwx 1 root root 9 Aug 11 20:11 scsi-26162356632353435 -> ../../sda

lrwxrwxrwx 1 root root 9 Aug 11 20:11 scsi-26238666462643332 -> ../../sdb

[root@oracle631 disk]#

More useful information, including the vendor name (for use in the multipath.conf file) as shown below.

[root@oracle631 disk]# cat /proc/scsi/scsi

Attached devices:

Host: scsi1 Channel: 00 Id: 00 Lun: 00

Vendor: QEMU Model: QEMU DVD-ROM Rev: 2.0.

Type: CD-ROM ANSI SCSI revision: 05

Host: scsi2 Channel: 00 Id: 00 Lun: 00

Vendor: SCST_BIO Model: disk0 Rev: 221

Type: Direct-Access ANSI SCSI revision: 05

Host: scsi2 Channel: 00 Id: 00 Lun: 01

Vendor: SCST_BIO Model: disk1 Rev: 221

Type: Direct-Access ANSI SCSI revision: 05

[root@oracle631 disk]#

And yet more information about the SCST iSCSI LUN /dev/sda shown below using udevadm command.

[root@oracle631 disk]# udevadm info -a -p $(udevadm info -q path -n /dev/sda)

Udevadm info starts with the device specified by the devpath and then

walks up the chain of parent devices. It prints for every device

found, all possible attributes in the udev rules key format.

A rule to match, can be composed by the attributes of the device

and the attributes from one single parent device.

looking at device '/devices/platform/host2/session1/target2:0:0/2:0:0:0/block/sda':

KERNEL=="sda"

SUBSYSTEM=="block"

DRIVER==""

ATTR{ro}=="0"

ATTR{size}=="20971520"

ATTR{stat}==" 455 0 3648 329 0 0 0 0 0 328 328"

ATTR{range}=="16"

ATTR{discard_alignment}=="0"

ATTR{events}==""

ATTR{ext_range}=="256"

ATTR{events_poll_msecs}=="-1"

ATTR{alignment_offset}=="0"

ATTR{inflight}==" 0 0"

ATTR{removable}=="0"

ATTR{capability}=="50"

ATTR{events_async}==""

looking at parent device '/devices/platform/host2/session1/target2:0:0/2:0:0:0':

KERNELS=="2:0:0:0"

SUBSYSTEMS=="scsi"

DRIVERS=="sd"

ATTRS{rev}==" 221"

ATTRS{type}=="0"

ATTRS{scsi_level}=="6"

ATTRS{model}=="disk0 "

ATTRS{state}=="running"

ATTRS{queue_type}=="none"

ATTRS{modalias}=="scsi:t-0x00"

ATTRS{iodone_cnt}=="0x212"

ATTRS{iorequest_cnt}=="0x212"

ATTRS{queue_ramp_up_period}=="120000"

ATTRS{timeout}=="30"

ATTRS{evt_media_change}=="0"

ATTRS{ioerr_cnt}=="0x2"

ATTRS{queue_depth}=="32"

ATTRS{vendor}=="SCST_BIO"

ATTRS{device_blocked}=="0"

ATTRS{dh_state}=="detached"

ATTRS{iocounterbits}=="32"

ATTRS{eh_timeout}=="10"

looking at parent device '/devices/platform/host2/session1/target2:0:0':

KERNELS=="target2:0:0"

SUBSYSTEMS=="scsi"

DRIVERS==""

looking at parent device '/devices/platform/host2/session1':

KERNELS=="session1"

SUBSYSTEMS==""

DRIVERS==""

looking at parent device '/devices/platform/host2':

KERNELS=="host2"

SUBSYSTEMS=="scsi"

DRIVERS==""

looking at parent device '/devices/platform':

KERNELS=="platform"

SUBSYSTEMS==""

DRIVERS==""

And now for /dev/sdb shown below using udevadm command.

[root@oracle631 disk]# udevadm info -a -p $(udevadm info -q path -n /dev/sdb)

Udevadm info starts with the device specified by the devpath and then

walks up the chain of parent devices. It prints for every device

found, all possible attributes in the udev rules key format.

A rule to match, can be composed by the attributes of the device

and the attributes from one single parent device.

looking at device '/devices/platform/host2/session1/target2:0:0/2:0:0:1/block/sdb':

KERNEL=="sdb"

SUBSYSTEM=="block"

DRIVER==""

ATTR{ro}=="0"

ATTR{size}=="20971520"

ATTR{stat}==" 455 0 3664 337 0 0 0 0 0 337 337"

ATTR{range}=="16"

ATTR{discard_alignment}=="0"

ATTR{events}==""

ATTR{ext_range}=="256"

ATTR{events_poll_msecs}=="-1"

ATTR{alignment_offset}=="0"

ATTR{inflight}==" 0 0"

ATTR{removable}=="0"

ATTR{capability}=="50"

ATTR{events_async}==""

looking at parent device '/devices/platform/host2/session1/target2:0:0/2:0:0:1':

KERNELS=="2:0:0:1"

SUBSYSTEMS=="scsi"

DRIVERS=="sd"

ATTRS{rev}==" 221"

ATTRS{type}=="0"

ATTRS{scsi_level}=="6"

ATTRS{model}=="disk1 "

ATTRS{state}=="running"

ATTRS{queue_type}=="none"

ATTRS{modalias}=="scsi:t-0x00"

ATTRS{iodone_cnt}=="0x211"

ATTRS{iorequest_cnt}=="0x211"

ATTRS{queue_ramp_up_period}=="120000"

ATTRS{timeout}=="30"

ATTRS{evt_media_change}=="0"

ATTRS{ioerr_cnt}=="0x2"

ATTRS{queue_depth}=="32"

ATTRS{vendor}=="SCST_BIO"

ATTRS{device_blocked}=="0"

ATTRS{dh_state}=="detached"

ATTRS{iocounterbits}=="32"

ATTRS{eh_timeout}=="10"

looking at parent device '/devices/platform/host2/session1/target2:0:0':

KERNELS=="target2:0:0"

SUBSYSTEMS=="scsi"

DRIVERS==""

looking at parent device '/devices/platform/host2/session1':

KERNELS=="session1"

SUBSYSTEMS==""

DRIVERS==""

looking at parent device '/devices/platform/host2':

KERNELS=="host2"

SUBSYSTEMS=="scsi"

DRIVERS==""

looking at parent device '/devices/platform':

KERNELS=="platform"

SUBSYSTEMS==""

DRIVERS==""

Additional information after the configuration of the SCST LUNs on oracle651 using device-mapper-multipath and /etc/udev/rules.d/12-dm-permissions.rules as shown below. Friendly names set from the /etc/multipath.conf file correctly.

[root@oracle631 ~]# ls -lrt /dev/mapper

total 0

crw-rw---- 1 root root 10, 236 Aug 11 21:30 control

lrwxrwxrwx 1 root root 7 Aug 11 21:30 vg_oracle631-lv_swap -> ../dm-1

lrwxrwxrwx 1 root root 7 Aug 11 21:30 vg_oracle631-lv_root -> ../dm-0

lrwxrwxrwx 1 root root 7 Aug 11 21:30 vg_u00-lv_oracle -> ../dm-2

lrwxrwxrwx 1 root root 7 Aug 11 21:30 ASM_4096B_1 -> ../dm-3

lrwxrwxrwx 1 root root 7 Aug 11 21:30 ASM_0512B_1 -> ../dm-4

Friendly names accurately indicate logical block sizes as shown below.

[root@oracle631 ~]# fdisk -l /dev/dm-3 <--ASM_4096B_1

Note: sector size is 4096 (not 512)

Disk /dev/dm-3: 10.7 GB, 10737418240 bytes

255 heads, 63 sectors/track, 163 cylinders

Units = cylinders of 16065 * 4096 = 65802240 bytes

Sector size (logical/physical): 4096 bytes / 4096 bytes <-- 4096 logical sector size

I/O size (minimum/optimal): 4096 bytes / 524288 bytes

Disk identifier: 0x00000000

[root@oracle631 ~]# fdisk -l /dev/dm-4

Disk /dev/dm-4: 10.7 GB, 10737418240 bytes <--ASM_0512B_1

255 heads, 63 sectors/track, 1305 cylinders

Units = cylinders of 16065 * 512 = 8225280 bytes

Sector size (logical/physical): 512 bytes / 4096 bytes <-- 512 logical sector size

I/O size (minimum/optimal): 4096 bytes / 524288 bytes

Disk identifier: 0x00000000

[root@oracle631 ~]#

Permissions set correctly from the /etc/udev/rules.d/12-dm-permissions.rules file.

[root@oracle631 ~]# ls -lrt /dev/dm-3

brw-rw---- 1 grid asmadmin 252, 3 Aug 11 21:30 /dev/dm-3

[root@oracle631 ~]# ls -lrt /dev/dm-4

brw-rw---- 1 grid asmadmin 252, 4 Aug 11 21:30 /dev/dm-4

The /etc/multipath.conf file for the SCST LUNs on oracle631 as shown below.

[root@oracle631 ~]# cat /etc/multipath.conf

# Configuration file for device mapper multipath.

# For a complete list of the default configuration values:

# /usr/share/doc/device-mapper-multipath-0.4.9/multipath.conf.defaults

# For a list of configuration options with descriptions:

# /usr/share/doc/device-mapper-multipath-0.4.9/multipath.conf.annotated

defaults {

user_friendly_names yes

}

blacklist {

devnode "^(ram|raw|loop|fd|md|dm-|sr|scd|st)[0-9]*"

devnode "^hd[a-z]"

}

devices {

device {

vendor "SCST_BIO"

product "Direct-Access"

path_grouping_policy multibus

path_grouping_policy group_by_prio

path_selector "round-robin 0"

failback immediate

rr_min_io 1

rr_weight priorities

}

multipaths {

multipath {

wwid 26162356632353435

alias ASM_4096B_1

}

multipath {

wwid 26238666462643332

alias ASM_0512B_1

}

[root@oracle631 ~]#

Listing of the /etc/udev/rules.d/12-dm-permissions.rules file on oracle631 the scsi client as shown below with section that handles the above LUNs bolded (near bottom of listing).

[root@oracle631 ~]# cat /etc/udev/rules.d/12-dm-permissions.rules

#

# This file is part of LVM2.

# Udev rules for device-mapper devices.

#

# These rules set permissions for DM devices.

#

# This file is considered to be a template where users can put their

# own entries and then put a copy of it manually to a usual place with

# user-edited udev rules (usually /etc/udev/rules.d).

#

# There are some environment variables set that can be used:

# DM_UDEV_RULES_VSN - DM udev rules version

# DM_NAME - actual DM device's name

# DM_UUID - UUID set for DM device (blank if not specified)

# DM_SUSPENDED - suspended state of DM device (0 or 1)

# DM_LV_NAME - logical volume name (not set if LVM device not present)

# DM_VG_NAME - volume group name (not set if LVM device not present)

# DM_LV_LAYER - logical volume layer (not set if LVM device not present)

# "add" event is processed on coldplug only!

ACTION!="add|change", GOTO="dm_end"

ENV{DM_UDEV_RULES_VSN}!="?*", GOTO="dm_end"

# A few demonstrational examples...

# PLAIN DM DEVICES

#

# Set permissions for a DM device named 'my_device' exactly

# ENV{DM_NAME}=="my_device", OWNER:="root", GROUP:="root", MODE:="660"

# Set permissions for all DM devices having 'MY_UUID-' UUID prefix

# ENV{DM_UUID}=="MY_UUID-?*", OWNER:="root", GROUP:="root", MODE:="660"

# LVM DEVICES

#

# Set permissions for all LVM devices

# ENV{DM_UUID}=="LVM-?*", OWNER:="root", GROUP:="root", MODE:="660"

# Set permissions for all devices that belong to one LVM VG

# ENV{DM_VG_NAME}=="VolGroup00", OWNER:="root", GROUP:="root", MODE:="660"

# Set permissions for an LVM device with VG named VolGroup00 and LV named LogVol00 exactly

# ENV{DM_VG_NAME}=="VolGroup00", ENV{DM_LV_NAME}=="LogVol00", OWNER:="root", GROUP:="root", MODE:="660"

# Set permissions for all LVM devices that does not belong to a VG named VolGroup00

# ENV{DM_VG_NAME}!="VolGroup00", OWNER:="root", GROUP:="root", MODE:="660"

# ENCRYPTED DEVICES (using cryptsetup >= 1.1)

#

# Set permissions for all encrypted devices created by cryptsetup (plain devices)

# ENV{DM_UUID}=="CRYPT-PLAIN-?*", OWNER:="root", GROUP:="root", MODE:="660"

# Set permissions for all encrypted devices created by cryptsetup (LUKS extension)

# ENV{DM_UUID}=="CRYPT-LUKS1-?*", OWNER:="root", GROUP:="root", MODE:="660"

# Set permissions for an encrypted device created by cryptsetup and having an exact luks UUID

# ENV{DM_UUID}=="CRYPT-LUKS1-22fce5c8313c43c68d84b50a3b0fee78-?*", OWNER:="root", GROUP:="root", MODE:="660"

# MULTIPATH DEVICES

#

# Set permissions for all multipath devices

# ENV{DM_UUID}=="mpath-?*", OWNER:="root", GROUP:="root", MODE:="660"

ENV{DM_UUID}=="mpath-26162356632353435", OWNER:="grid", GROUP:="asmadmin", MODE:="660" # <-- LUN modes are set here

ENV{DM_UUID}=="mpath-26238666462643332", OWNER:="grid", GROUP:="asmadmin", MODE:="660" # <-- LUN modes are set here

# Set permissions for first two partitions created on a multipath device (and detected by kpartx)

# ENV{DM_UUID}=="part[1-2]-mpath-?*", OWNER:="root", GROUP:="root", MODE:="660"

# ...you can use any combination of the comparisons with the environment variables

# listed at the beginning of this file (udev provides simple pattern matching by

# using *, ? and [] that you can use, see 'man udev' for more information).

# Set default permissions for all DM devices if not set before.

# OWNER:="root", GROUP:="root", MODE:="660"

LABEL="dm_end"

[root@oracle631 ~]#

If the console terminal is running on the KVM guest SCST SAN host oracle651, then when a SAN client oracle631 is shutdown, the following message is buffered to console and written to dmesg as shown below.

[root@oracle651 ~]# iscsi-scst: Logout received from initiator iqn.1988-12.com.oracle:49c39d4d389d

iscsi-scst: Closing connection at initiator's iqn.1988-12.com.oracle:49c39d4d389d request

scst: TM fn 6 (ffff88003dfd50b8)

scst: TM fn 6 (ffff88003dfd50b8) finished, status 0

scst: Processing thread disk00_0 (PID 1783) finished

scst: Processing thread disk10_0 (PID 1789) finished

[root@oracle651 ~]# exit

logout

Oracle Linux Server release 6.5

Kernel 2.6.32-scst on an x86_64

oracle651.vmem.org login:

Here is an example of the commands for adding a file-backed LUN rather than a block device as shown below.

[root@oracle651 ~] dd if=/dev/zero of=/root/scst-scripts/asm_disk1.img bs=512 count=0 seek=1M

[root@oracle651 ~] scstadmin -open_dev disk2 -handler vdisk_fileio -attributes filename=/root/scst-scripts/asm_disk1.img,blocksize=4096

[root@oracle651 ~] scstadmin -add_lun 2 -driver iscsi -target iqn.2014-08.oracle.asm.storage:luns -device disk2

[root@oracle651 ~] scstadmin -write_config /etc/scst.conf

The last command saves it to the /etc/scst.conf configuration file as shown below. The "vdisk_fileio" handler has been added, as well as the LUN.

[root@oracle651 scst-scripts]# cat /etc/scst.conf

# Automatically generated by SCST Configurator v2.2.1.

HANDLER vdisk_blockio {

DEVICE disk0 {

blocksize 4096

filename /dev/vdc

}

DEVICE disk1 {

filename /dev/vdb

}

HANDLER vdisk_fileio {

DEVICE disk2 {

blocksize 4096

filename /root/scst-scripts/asm_disk1.img

}

TARGET_DRIVER iscsi {

enabled 1

TARGET iqn.2014-08.oracle.asm.storage:luns {

cpu_mask 3

enabled 1

rel_tgt_id 1

LUN 0 disk0

LUN 1 disk1

LUN 2 disk2

}

[root@oracle651 scst-scripts]#

Here is an example of removing the LUN that was just added. Notice that the HANDLER information for the deleted LUN remains, but the TARGET DRIVER LUN list is updated and the removed LUN has been removed from the list in /etc/scst.conf as shown below.

[root@oracle651 scst-scripts]# scstadmin -rem_lun 2 -driver iscsi -target iqn.2014-08.oracle.asm.storage:luns -device disk2

Collecting current configuration: done.

Performing this action may result in lost or corrupt data, are you sure you wish to continue (y/[n]) ? y

-> Making requested changes.

-> Removing LUN 2 from driver/target 'iscsi/iqn.2014-08.oracle.asm.storage:luns': done.

Writing current configuration to file '/etc/scst.conf'

All done.

[root@oracle651 scst-scripts]# scstadmin -write_config /etc/scst.conf

Collecting current configuration: done.

Writing current configuration to file '/etc/scst.conf'..

All done.

[root@oracle651 scst-scripts]# cat /etc/scst.conf

# Automatically generated by SCST Configurator v2.2.1.

HANDLER vdisk_blockio {

DEVICE disk0 {

blocksize 4096

filename /dev/vdc

}

DEVICE disk1 {

filename /dev/vdb

}

HANDLER vdisk_fileio { <-- disk2 handler is still defined after LUN 2 was dropped from target.

DEVICE disk2 {

blocksize 4096

filename /root/scst-scripts/asm_disk1.img

}

TARGET_DRIVER iscsi {

enabled 1

TARGET iqn.2014-08.oracle.asm.storage:luns {

cpu_mask 3

enabled 1

rel_tgt_id 1

LUN 0 disk0

LUN 1 disk1

} <-- LUN 2 disk2 is removed here.

}

[root@oracle651 scst-scripts]# history

Thus, the LUN can be re-added back with just the one command below if it is needed back in the target as shown below.

[root@oracle651 ~]# scstadmin -add_lun 2 -driver iscsi -target iqn.2014-08.oracle.asm.storage:luns -device disk2

Collecting current configuration: done.

-> Making requested changes.

-> Adding device 'disk2' at LUN 2 to driver/target 'iscsi/iqn.2014-08.oracle.asm.storage:luns': done.

-> Driver/target is not a fibre channel target, ignoring.

-> Done.

All done.

[root@oracle651 ~]# scstadmin -write_config /etc/scst.conf

Collecting current configuration: done.

Writing current configuration to file '/etc/scst.conf'..

All done.

[root@oracle651 ~]#

View /etc/scst.conf again and see that the LUN has been added back to the TARGET as shown below.

[root@oracle651 ~]# cat /etc/scst.conf

# Automatically generated by SCST Configurator v2.2.1.

HANDLER vdisk_blockio {

DEVICE disk0 {

blocksize 4096

filename /dev/vdc

}

DEVICE disk1 {

filename /dev/vdb

}

HANDLER vdisk_fileio {

DEVICE disk2 {

blocksize 4096

filename /root/scst-scripts/asm_disk1.img

}

TARGET_DRIVER iscsi {

enabled 1

TARGET iqn.2014-08.oracle.asm.storage:luns {

cpu_mask 3

enabled 1

rel_tgt_id 1

LUN 0 disk0

LUN 1 disk1

LUN 2 disk2

}

[root@oracle651 ~]#

Examples of adding file-backed SCST LUNs to the OS of the SCST OEL host are shown below. The example below shows adding the 8 LUNs per the Violin Memory Oracle Best Practice Document dated April 2014 to the SCST SAN configuration.

[root@oracle651 ~]# scstadmin -add_lun 11 -driver iscsi -target iqn.2014-08.oracle.asm.storage:luns -device asm_reco1

Collecting current configuration: done.

-> Making requested changes.

-> Adding device 'asm_reco1' at LUN 11 to driver/target 'iscsi/iqn.2014-08.oracle.asm.storage:luns': done.

-> Driver/target is not a fibre channel target, ignoring.

-> Done.

All done.

[root@oracle651 ~]# scstadmin -add_lun 12 -driver iscsi -target iqn.2014-08.oracle.asm.storage:luns -device asm_reco2

Collecting current configuration: done.

-> Making requested changes.

-> Adding device 'asm_reco2' at LUN 12 to driver/target 'iscsi/iqn.2014-08.oracle.asm.storage:luns': done.

-> Driver/target is not a fibre channel target, ignoring.

-> Done.

All done.

[root@oracle651 ~]# scstadmin -add_lun 13 -driver iscsi -target iqn.2014-08.oracle.asm.storage:luns -device asm_reco3

Collecting current configuration: done.

-> Making requested changes.

-> Adding device 'asm_reco3' at LUN 13 to driver/target 'iscsi/iqn.2014-08.oracle.asm.storage:luns': done.

-> Driver/target is not a fibre channel target, ignoring.

-> Done.

All done.

[root@oracle651 ~]# scstadmin -add_lun 14 -driver iscsi -target iqn.2014-08.oracle.asm.storage:luns -device asm_reco4

Collecting current configuration: done.

-> Making requested changes.

-> Adding device 'asm_reco4' at LUN 14 to driver/target 'iscsi/iqn.2014-08.oracle.asm.storage:luns': done.

-> Driver/target is not a fibre channel target, ignoring.

-> Done.

All done.

[root@oracle651 ~]# scstadmin -add_lun 15 -driver iscsi -target iqn.2014-08.oracle.asm.storage:luns -device asm_reco5

Collecting current configuration: done.

-> Making requested changes.

-> Adding device 'asm_reco5' at LUN 15 to driver/target 'iscsi/iqn.2014-08.oracle.asm.storage:luns': done.

-> Driver/target is not a fibre channel target, ignoring.

-> Done.

All done.

[root@oracle651 ~]# scstadmin -add_lun 16 -driver iscsi -target iqn.2014-08.oracle.asm.storage:luns -device asm_reco6

Collecting current configuration: done.

-> Making requested changes.

-> Adding device 'asm_reco6' at LUN 16 to driver/target 'iscsi/iqn.2014-08.oracle.asm.storage:luns': done.

-> Driver/target is not a fibre channel target, ignoring.

-> Done.

All done.

[root@oracle651 ~]# scstadmin -add_lun 17 -driver iscsi -target iqn.2014-08.oracle.asm.storage:luns -device asm_reco7

Collecting current configuration: done.

-> Making requested changes.

-> Adding device 'asm_reco7' at LUN 17 to driver/target 'iscsi/iqn.2014-08.oracle.asm.storage:luns': done.

-> Driver/target is not a fibre channel target, ignoring.

-> Done.

All done.

[root@oracle651 ~]# scstadmin -add_lun 18 -driver iscsi -target iqn.2014-08.oracle.asm.storage:luns -device asm_reco8

Collecting current configuration: done.

-> Making requested changes.

-> Adding device 'asm_reco8' at LUN 18 to driver/target 'iscsi/iqn.2014-08.oracle.asm.storage:luns': done.

-> Driver/target is not a fibre channel target, ignoring.

-> Done.

All done.

[root@oracle651 ~]#

Now write the configuration changes to the /etc/scst.conf configuration file as shown below.

[root@oracle651 ~]# scstadmin -write_config /etc/scst.conf

Collecting current configuration: done.

Writing current configuration to file '/etc/scst.conf'..

All done.

[root@oracle651 ~]# cat /etc/scst.conf

# Automatically generated by SCST Configurator v2.2.1.

HANDLER vdisk_blockio {

DEVICE disk0 {

blocksize 4096

filename /dev/vdc

}

DEVICE disk1 {

filename /dev/vdb

}

HANDLER vdisk_fileio {

DEVICE asm_data1 {

filename /scst/asm_data1.img

}

DEVICE asm_data2 {

filename /scst/asm_data2.img

}

DEVICE asm_data3 {

filename /scst/asm_data3.img

}

DEVICE asm_data4 {

filename /scst/asm_data4.img

}

DEVICE asm_data5 {

filename /scst/asm_data5.img

}

DEVICE asm_data6 {

filename /scst/asm_data6.img

}

DEVICE asm_data7 {

filename /scst/asm_data7.img

}

DEVICE asm_data8 {

filename /scst/asm_data8.img

}

DEVICE asm_reco1 {

filename /scst/asm_reco1.img

}

DEVICE asm_reco2 {

filename /scst/asm_reco2.img

}

DEVICE asm_reco3 {

filename /scst/asm_reco3.img

}

DEVICE asm_reco4 {

filename /scst/asm_reco4.img

}

DEVICE asm_reco5 {

filename /scst/asm_reco5.img

}

DEVICE asm_reco6 {

filename /scst/asm_reco6.img

}

DEVICE asm_reco7 {

filename /scst/asm_reco7.img

}

DEVICE asm_reco8 {

filename /scst/asm_reco8.img

}

DEVICE systemdg1 {

filename /scst/systemdg1.img

}

TARGET_DRIVER iscsi {

enabled 1

TARGET iqn.2014-08.oracle.asm.storage:luns {

cpu_mask 3

enabled 1

rel_tgt_id 1

LUN 0 disk0

LUN 1 disk1

LUN 2 systemdg1

LUN 3 asm_data1

LUN 4 asm_data2

LUN 5 asm_data3

LUN 6 asm_data4

LUN 7 asm_data5

LUN 8 asm_data6

LUN 9 asm_data7

LUN 10 asm_data8

LUN 11 asm_reco1

LUN 12 asm_reco2

LUN 13 asm_reco3

LUN 14 asm_reco4

LUN 15 asm_reco5

LUN 16 asm_reco6

LUN 17 asm_reco7

LUN 18 asm_reco8

}

[root@oracle651 ~]#

Adding the new LUNs to the Oracle database server is shown below. As soon as the above command to add the LUNs to the target is completed, the LUNs will immediately appear on the the SCSI client machine - the database server - (as long as it is logged in to the SAN) and will be visible as shown below using the multipath command.

[root@oracle631 ~]# multipath -ll -v2 | grep reco

mpathr (23734383163616238) dm-17 SCST_FIO,asm_reco1

mpathq (23239613933333133) dm-16 SCST_FIO,asm_reco4

mpathp (23533633466613536) dm-14 SCST_FIO,asm_reco3

mpatho (23430363636323231) dm-13 SCST_FIO,asm_reco2

mpathv (26661333739313737) dm-21 SCST_FIO,asm_reco8

mpathu (23164346539623861) dm-20 SCST_FIO,asm_reco7

mpatht (2656563303366642d) dm-19 SCST_FIO,asm_reco6

mpaths (23361306261623634) dm-18 SCST_FIO,asm_reco5

[root@oracle631 ~]#

Use the output above the /etc/multipath.conf can be edited.

[root@oracle631 ~]# vi /etc/multipath.conf

[root@oracle631 ~]# cat /etc/multipath.conf

# Configuration file for device mapper multipath.

# For a complete list of the default configuration values:

# /usr/share/doc/device-mapper-multipath-0.4.9/multipath.conf.defaults

# For a list of configuration options with descriptions:

# /usr/share/doc/device-mapper-multipath-0.4.9/multipath.conf.annotated

defaults {

user_friendly_names yes

}

blacklist {

devnode "^(ram|raw|loop|fd|md|dm-|sr|scd|st)[0-9]*"

devnode "^hd[a-z]"

}

devices {

device {

vendor "SCST_BIO"

product "Direct-Access"

path_grouping_policy multibus

path_selector "round-robin 0"

failback immediate

rr_min_io 1

rr_weight priorities

}

multipaths {

multipath {

wwid 26162356632353435

alias ASM_4096B_1

}

multipath {

wwid 26238666462643332

alias ASM_0512B_1

}

multipath {

wwid 23863316131356162

alias ASM_4096F_1

}

multipath {

wwid 23233626538613166

alias ASM_0512F_1

}

multipath {

wwid 26161383334393638

alias ASM_DATA8

}

multipath {

wwid 23464666134333935

alias ASM_DATA7

}

multipath {

wwid 23565353864626532

alias ASM_DATA6

}

multipath {

wwid 23661626637333762

alias ASM_DATA5

}

multipath {

wwid 23739316465623063

alias ASM_DATA4

}

multipath {

wwid 2333730323234392d

alias ASM_DATA3

}

multipath {

wwid 23130643262613365

alias ASM_DATA2

}

multipath {

wwid 23234333531326137

alias ASM_DATA1

}

multipath {

wwid 23734383163616238

alias ASM_RECO1

}

multipath {

wwid 23430363636323231

alias ASM_RECO2

}

multipath {

wwid 23533633466613536

alias ASM_RECO3

}

multipath {

wwid 23239613933333133

alias ASM_RECO4

}

multipath {

wwid 23361306261623634

alias ASM_RECO5

}

multipath {

wwid 2656563303366642d

alias ASM_RECO6

}

multipath {

wwid 23164346539623861

alias ASM_RECO7

}

multipath {

wwid 26661333739313737

alias ASM_RECO8

}

[root@oracle631 ~]#

Check the permissions, ownership and group of the LUNs to be sure they are correct. With these iSCSI LUNs from SCST normally the add instantly to the client machine with correct permissions and modes and no reboot is needed. The only step required is to edit the /etc/multipath.conf file and restart multipath as shown below.

[root@oracle631 ~]# ls -lrt /dev/mapper

total 0

crw-rw---- 1 root root 10, 236 Aug 13 10:17 control

lrwxrwxrwx 1 root root 7 Aug 13 10:17 vg_oracle631-lv_swap -> ../dm-1

lrwxrwxrwx 1 root root 7 Aug 13 10:17 vg_oracle631-lv_root -> ../dm-0

lrwxrwxrwx 1 root root 7 Aug 13 10:17 vg_u00-lv_oracle -> ../dm-2

lrwxrwxrwx 1 root root 7 Aug 13 10:38 ASM_0512B_1 -> ../dm-7

lrwxrwxrwx 1 root root 7 Aug 13 10:38 ASM_DATA1 -> ../dm-5

lrwxrwxrwx 1 root root 7 Aug 13 10:38 ASM_0512F_1 -> ../dm-4

lrwxrwxrwx 1 root root 8 Aug 13 10:38 ASM_DATA6 -> ../dm-10

lrwxrwxrwx 1 root root 7 Aug 13 10:38 ASM_DATA3 -> ../dm-8

lrwxrwxrwx 1 root root 7 Aug 13 10:38 ASM_DATA4 -> ../dm-6

lrwxrwxrwx 1 root root 7 Aug 13 10:38 ASM_DATA5 -> ../dm-9

lrwxrwxrwx 1 root root 8 Aug 13 10:38 ASM_DATA8 -> ../dm-11

lrwxrwxrwx 1 root root 7 Aug 13 10:38 ASM_4096B_1 -> ../dm-3

lrwxrwxrwx 1 root root 8 Aug 13 10:38 ASM_RECO1 -> ../dm-17

lrwxrwxrwx 1 root root 8 Aug 13 10:38 ASM_DATA7 -> ../dm-12

lrwxrwxrwx 1 root root 8 Aug 13 10:38 ASM_DATA2 -> ../dm-15

lrwxrwxrwx 1 root root 8 Aug 13 10:38 ASM_RECO4 -> ../dm-16

lrwxrwxrwx 1 root root 8 Aug 13 10:38 ASM_RECO2 -> ../dm-13

lrwxrwxrwx 1 root root 8 Aug 13 10:38 ASM_RECO3 -> ../dm-14

lrwxrwxrwx 1 root root 8 Aug 13 10:38 ASM_RECO6 -> ../dm-19

lrwxrwxrwx 1 root root 8 Aug 13 10:38 ASM_RECO8 -> ../dm-21

lrwxrwxrwx 1 root root 8 Aug 13 10:38 ASM_RECO5 -> ../dm-18

lrwxrwxrwx 1 root root 8 Aug 13 10:38 ASM_RECO7 -> ../dm-20

[root@oracle631 ~]# ls -lrt /dev/dm-*

brw-rw---- 1 root disk 252, 1 Aug 13 10:17 /dev/dm-1

brw-rw---- 1 root disk 252, 0 Aug 13 10:17 /dev/dm-0

brw-rw---- 1 root disk 252, 2 Aug 13 10:17 /dev/dm-2

brw-rw---- 1 grid asmadmin 252, 7 Aug 13 10:38 /dev/dm-7

brw-rw---- 1 grid asmadmin 252, 5 Aug 13 10:38 /dev/dm-5

brw-rw---- 1 grid asmadmin 252, 4 Aug 13 10:38 /dev/dm-4

brw-rw---- 1 grid asmadmin 252, 10 Aug 13 10:38 /dev/dm-10

brw-rw---- 1 grid asmadmin 252, 8 Aug 13 10:38 /dev/dm-8

brw-rw---- 1 grid asmadmin 252, 6 Aug 13 10:38 /dev/dm-6

brw-rw---- 1 grid asmadmin 252, 9 Aug 13 10:38 /dev/dm-9

brw-rw---- 1 grid asmadmin 252, 3 Aug 13 10:38 /dev/dm-3

brw-rw---- 1 grid asmadmin 252, 11 Aug 13 10:38 /dev/dm-11

brw-rw---- 1 grid asmadmin 252, 17 Aug 13 10:38 /dev/dm-17

brw-rw---- 1 grid asmadmin 252, 12 Aug 13 10:38 /dev/dm-12

brw-rw---- 1 grid asmadmin 252, 15 Aug 13 10:38 /dev/dm-15

brw-rw---- 1 grid asmadmin 252, 16 Aug 13 10:38 /dev/dm-16

brw-rw---- 1 grid asmadmin 252, 13 Aug 13 10:38 /dev/dm-13

brw-rw---- 1 grid asmadmin 252, 14 Aug 13 10:38 /dev/dm-14

brw-rw---- 1 grid asmadmin 252, 19 Aug 13 10:38 /dev/dm-19

brw-rw---- 1 grid asmadmin 252, 21 Aug 13 10:38 /dev/dm-21

brw-rw---- 1 grid asmadmin 252, 18 Aug 13 10:38 /dev/dm-18

brw-rw---- 1 grid asmadmin 252, 20 Aug 13 10:38 /dev/dm-20

[root@oracle631 ~]#

Configure the LUNs with ASMLib as shown below. Alternatively, for a large number of LUNs scripted methods can be used as shown in the Violin Memory Oracle Database Best Practice Document April 2014.

[root@oracle631 ~]# oracleasm createdisk ASM_RECO1 /dev/mapper/ASM_RECO1

Writing disk header: done

Instantiating disk: done

[root@oracle631 ~]# oracleasm createdisk ASM_RECO2 /dev/mapper/ASM_RECO2

Writing disk header: done

Instantiating disk: done

[root@oracle631 ~]# oracleasm createdisk ASM_RECO3 /dev/mapper/ASM_RECO3

Writing disk header: done

Instantiating disk: done

[root@oracle631 ~]# oracleasm createdisk ASM_RECO4 /dev/mapper/ASM_RECO4

Writing disk header: done

Instantiating disk: done

[root@oracle631 ~]# oracleasm createdisk ASM_RECO5 /dev/mapper/ASM_RECO5

Writing disk header: done

Instantiating disk: done

[root@oracle631 ~]# oracleasm createdisk ASM_RECO6 /dev/mapper/ASM_RECO6

Writing disk header: done

Instantiating disk: done

[root@oracle631 ~]# oracleasm createdisk ASM_RECO7 /dev/mapper/ASM_RECO7

Writing disk header: done

Instantiating disk: done

[root@oracle631 ~]# oracleasm createdisk ASM_RECO8 /dev/mapper/ASM_RECO8

Writing disk header: done

Instantiating disk: done

[root@oracle631 ~]#

Connect to the server as grid user as shown below and run "asmca" to build the RECO asm diskgroup. Note that asmca can be used here because these are 512 emulated LUNs so no special attributes are needed. If these were 4096 native LUNs command line would have to be used to build the diskgroup.

gstanden@vmem1:~$ ssh -Y -C grid@10.207.39.72

grid@10.207.39.72's password:

Last login: Wed Aug 13 10:45:42 2014 from 10.207.39.1

[grid@oracle631 ~]$ asmca

The screenshots of asmca are shown below.

The final configuration of disks for the Oracle database install.

SQL> l

1* select name, label, path, total_mb, header_status, mount_status, mode_status, sector_size from v$asm_disk

SQL> /

NAME LABEL PATH TOTAL_MB HEADER_STATU MOUNT_S MODE_ST SECTOR_SIZE

------------------------------ ------------------------------- -------------------- ---------- ------------ ------- ------- -----------

ASM_DATA1 ASM_DATA1 ORCL:ASM_DATA1 1024 MEMBER CACHED ONLINE 512

ASM_DATA2 ASM_DATA2 ORCL:ASM_DATA2 1024 MEMBER CACHED ONLINE 512

ASM_DATA3 ASM_DATA3 ORCL:ASM_DATA3 1024 MEMBER CACHED ONLINE 512

ASM_DATA4 ASM_DATA4 ORCL:ASM_DATA4 1024 MEMBER CACHED ONLINE 512

ASM_DATA5 ASM_DATA5 ORCL:ASM_DATA5 1024 MEMBER CACHED ONLINE 512

ASM_DATA6 ASM_DATA6 ORCL:ASM_DATA6 1024 MEMBER CACHED ONLINE 512

ASM_DATA7 ASM_DATA7 ORCL:ASM_DATA7 1024 MEMBER CACHED ONLINE 512

ASM_DATA8 ASM_DATA8 ORCL:ASM_DATA8 1024 MEMBER CACHED ONLINE 512

SYSTEMDG SYSTEMDG ORCL:SYSTEMDG 1024 MEMBER CACHED ONLINE 512

ASM_RECO1 ASM_RECO1 ORCL:ASM_RECO1 1024 MEMBER CACHED ONLINE 512

ASM_RECO2 ASM_RECO2 ORCL:ASM_RECO2 1024 MEMBER CACHED ONLINE 512

ASM_RECO3 ASM_RECO3 ORCL:ASM_RECO3 1024 MEMBER CACHED ONLINE 512

ASM_RECO4 ASM_RECO4 ORCL:ASM_RECO4 1024 MEMBER CACHED ONLINE 512

ASM_RECO5 ASM_RECO5 ORCL:ASM_RECO5 1024 MEMBER CACHED ONLINE 512

ASM_RECO6 ASM_RECO6 ORCL:ASM_RECO6 1024 MEMBER CACHED ONLINE 512

ASM_RECO7 ASM_RECO7 ORCL:ASM_RECO7 1024 MEMBER CACHED ONLINE 512

ASM_RECO8 ASM_RECO8 ORCL:ASM_RECO8 1024 MEMBER CACHED ONLINE 512

17 rows selected.

SQL>

Obtain the Oracle binaries and install for your chosen version of database. Begin the install. Here the oracle install media is unpacked under the /home/oracle directory as shown below.

[oracle@oracle631 database]$ pwd

/home/oracle/database

[oracle@oracle631 database]$ ./runInstaller

Starting Oracle Universal Installer...

Checking Temp space: must be greater than 120 MB. Actual 2527 MB Passed

Checking swap space: must be greater than 150 MB. Actual 5087 MB Passed

Checking monitor: must be configured to display at least 256 colors. Actual 16777216 Passed

Preparing to launch Oracle Universal Installer from /tmp/OraInstall2014-08-13_11-36-50AM. Please wait ...

[oracle@oracle631 database]$