SCST KVM OEL 6.5
This guide explains how to build SCST SAN on an Oracle Enterprise Linux 6.5 KVM guest VM based on the instructions here at spiderbird. This blog is based on that excellent guide. That guide also has a wealth of reference URLs that can be accessed as well. I intend to use this KVM guest SCST SAN to present 4K logcial/4K physical virtual LUNs to Oracle running in other KVM guests, including Oracle Real Application Clusters running in KVM guests. Because TGT cannot handle LUNs which are 4K logical / 4K physical it was necessary to get up to speed on SCST which can. This sourceforge SAN feature comparison sheet shows the full power of SCST compared to other linux SAN options.
Ensure the virtual ethernet interface, typically "eth0" in the OEL 6.5 KVM guest is set to "connect automatically". If not, configure it as shown in the screenshots below. This can be configured from the server desktop, if using OEL 6.5 with a desktop, or simply edited in the ifcfg-eth0 file as shown below. As shown in bold below, be sure that "ONBOOT=yes" has been set in the ifcfg-ethX file where in this case my interface is ifcfg-eth0.
[root@oracle651 network-scripts]# cat ifcfg-eth0
DEVICE=eth0
TYPE=Ethernet
UUID=824a1d97-1dc7-410b-89a5-49b353be826a
ONBOOT=yes
NM_CONTROLLED=yes
BOOTPROTO=dhcp
DNS1=127.0.0.1
DOMAIN=vmem.org
DEFROUTE=yes
IPV4_FAILURE_FATAL=yes
IPV6INIT=no
NAME="System eth0"
HWADDR=52:54:00:66:26:A0
DNS2=192.168.1.1
PEERDNS=no
PEERROUTES=yes
LAST_CONNECT=1407733083
[root@oracle651 network-scripts]#
If using any virtualiztion options, remove them now (does not apply for this blog because this is being installed into a KVM guest). But for completeness, this is included from the spiderbird blog. These are options below which could be selected duiring OEL 6.5 OS install.
Select the “Software Development Workstation” option then unselect groups virtualization, virtualization-client and virtualization-platform or let the installer finish then use “yum groupremove virtualization virtualization-client virtualization-platform” once you’ve booted into the OS the first time. By selecting the “Software Development Workstation” as your install type, it will pretty much install every package you’ll need to do the kernel compile.
Do a yum update to update the kernel before beginning this work. Once this work is done the kernel will be locked so that there will be no more kernel updated, because this procedure builds a custom kernel for SCST which will not be updated any further normally (without going through this entire procedure again!).
Once the yum update is complete, reboot the KVM guest so that the just-updated kernel is running in the KVM guest.
Install the following packages using yum as shown below. Do this as root. Note that further on in this blog there was a missing dependency on "xmlto" which I am retroactively addressing here in this blog so there won't be that missing dependency.
[root@oracle651 ~]# yum install newt-devel
[root@oracle651 ~]# yum install asciidoc
[root@oracle651 ~]# yum install xmlto
As root, add the "mockbuild" user as shown below.
[root@oracle651 ~]# useradd mockbuild
Now su to the "mockbuild" user as shown below. A password is not needed for the mockbuild user unless you explicitly want one. It's just an account that will be used for this build.
[root@oracle651 ~]# su - mockbuild
[mockbuild@oracle651 ~]$ pwd
/home/mockbuild
[mockbuild@oracle651 ~]$ id
uid=501(mockbuild) gid=501(mockbuild) groups=501(mockbuild) context=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023
[mockbuild@oracle651 ~]$
Run the following commands as the mockbuild user from the /home/mockbuild directory as shown below.
[mockbuild@oracle651 ~]$ mkdir -p ~/rpmbuild/{BUILD,BUILDROOT,RPMS,SOURCES,SPECS,SRPMS}
[mockbuild@oracle651 ~]$ echo '%_topdir %(echo $HOME)/rpmbuild' > ~/.rpmmacros
[mockbuild@oracle651 ~]$ rpm -i http://vault.centos.org/6.5/updates/Source/SPackages/kernel-2.6.32-431.23.3.el6.src.rpm 2>&1 | grep -v mock
Note that even though this is the Oracle Red Hat Compatible kernel, I downloaded the CentOS kernel source from vault.centos.org, which should be binary compatible with RedHat kernel source. However, it should be possible to obtain the actual Oracle Red Hat Compatible kernel source which *should* also work.
Before running the below "rpmbuild" command in step after the next one, open another window as root and run the entropy generator as shown here. If the run of rpmbuild in the next step bogs down, run this command again to refill the entropy pool. It will make the rpmbuild complete faster.
[root@oracle651 ~]# rngd -r /dev/urandom
[root@oracle651 ~]#
Now you can run the command. If it bogs down towards the end run the above rngd command again. Note: be sure when running the command below, that if you cut and paste from this blog that the word "--target" has "two dashes" i.e "--" in front of it, and not just one big dash. Sometimes characterset conversion can cause two dashed cut and pasted to become an uninterpretable single large dash.
[root@oracle651 ~]# su - mockbuild
[mockbuild@oracle651 ~]$ rpmbuild -bp --target=$(uname -m) ./rpmbuild/SPECS/kernel.spec
Building target platforms: x86_64
Building for target x86_64
Executing(%prep): /bin/sh -e /var/tmp/rpm-tmp.rlmm5g
+ umask 022
+ cd /home/mockbuild/rpmbuild/BUILD
+ LANG=C
+ export LANG
+ unset DISPLAY
+ patch_command='patch -p1 -F1 -s'
+ '[' '!' -d kernel-2.6.32-431.23.3.el6/vanilla-2.6.32-431.23.3.el6/ ']'
+ rm -f pax_global_header
+ cd /home/mockbuild/rpmbuild/BUILD
+ rm -rf kernel-2.6.32-431.23.3.el6
+ /bin/mkdir -p kernel-2.6.32-431.23.3.el6
+ cd kernel-2.6.32-431.23.3.el6
+ /usr/bin/bzip2 -dc /home/mockbuild/rpmbuild/SOURCES/linux-2.6.32-431.23.3.el6.tar.bz2
+ /bin/tar -xf -
+ STATUS=0
+ '[' 0 -ne 0 ']'
+ /bin/chmod -Rf a+rX,u+w,g-w,o-w .
+ mv linux-2.6.32-431.23.3.el6 vanilla-2.6.32-431.23.3.el6
+ '[' -d linux-2.6.32-431.23.3.el6.x86_64 ']'
+ cp -rl vanilla-2.6.32-431.23.3.el6 linux-2.6.32-431.23.3.el6.x86_64
+ cd linux-2.6.32-431.23.3.el6.x86_64
+ cp /home/mockbuild/rpmbuild/SOURCES/config-debug /home/mockbuild/rpmbuild/SOURCES/config-debug-rhel /home/mockbuild/rpmbuild/SOURCES/config-framepointer /home/mockbuild/rpmbuild/SOURCES/config-generic /home/mockbuild/rpmbuild/SOURCES/config-generic-rhel /home/mockbuild/rpmbuild/SOURCES/config-i686 /home/mockbuild/rpmbuild/SOURCES/config-i686-debug /home/mockbuild/rpmbuild/SOURCES/config-i686-debug-rhel /home/mockbuild/rpmbuild/SOURCES/config-i686-nodebug /home/mockbuild/rpmbuild/SOURCES/config-i686-nodebug-rhel /home/mockbuild/rpmbuild/SOURCES/config-i686-rhel /home/mockbuild/rpmbuild/SOURCES/config-ia64-generic-rhel /home/mockbuild/rpmbuild/SOURCES/config-nodebug /home/mockbuild/rpmbuild/SOURCES/config-nodebug-rhel /home/mockbuild/rpmbuild/SOURCES/config-powerpc-generic /home/mockbuild/rpmbuild/SOURCES/config-powerpc-generic-rhel /home/mockbuild/rpmbuild/SOURCES/config-powerpc64 /home/mockbuild/rpmbuild/SOURCES/config-powerpc64-debug /home/mockbuild/rpmbuild/SOURCES/config-powerpc64-debug-rhel /home/mockbuild/rpmbuild/SOURCES/config-powerpc64-kdump /home/mockbuild/rpmbuild/SOURCES/config-powerpc64-kdump-rhel /home/mockbuild/rpmbuild/SOURCES/config-powerpc64-rhel /home/mockbuild/rpmbuild/SOURCES/config-s390x /home/mockbuild/rpmbuild/SOURCES/config-s390x-debug /home/mockbuild/rpmbuild/SOURCES/config-s390x-debug-rhel /home/mockbuild/rpmbuild/SOURCES/config-s390x-generic-rhel /home/mockbuild/rpmbuild/SOURCES/config-s390x-kdump /home/mockbuild/rpmbuild/SOURCES/config-s390x-kdump-rhel /home/mockbuild/rpmbuild/SOURCES/config-s390x-rhel /home/mockbuild/rpmbuild/SOURCES/config-x86-generic /home/mockbuild/rpmbuild/SOURCES/config-x86-generic-rhel /home/mockbuild/rpmbuild/SOURCES/config-x86_64-debug /home/mockbuild/rpmbuild/SOURCES/config-x86_64-debug-rhel /home/mockbuild/rpmbuild/SOURCES/config-x86_64-generic /home/mockbuild/rpmbuild/SOURCES/config-x86_64-generic-rhel /home/mockbuild/rpmbuild/SOURCES/config-x86_64-nodebug /home/mockbuild/rpmbuild/SOURCES/config-x86_64-nodebug-rhel .
+ cp /home/mockbuild/rpmbuild/SOURCES/merge.pl /home/mockbuild/rpmbuild/SOURCES/Makefile.common /home/mockbuild/rpmbuild/SOURCES/perf /home/mockbuild/rpmbuild/SOURCES/kabitool /home/mockbuild/rpmbuild/SOURCES/check-kabi .
+ make -f /home/mockbuild/rpmbuild/SOURCES/Makefile.config VERSION=2.6.32 configs
perl merge.pl config-nodebug-merged config-generic-merged > temp-generic
perl merge.pl config-x86_64-generic-merged temp-generic > temp-x86_64-generic
perl merge.pl config-x86_64-nodebug-merged temp-x86_64-generic x86_64 > kernel-2.6.32-x86_64.config
perl merge.pl config-debug-merged config-generic-merged > temp-debug-generic
perl merge.pl config-x86_64-generic-merged temp-debug-generic > temp-x86_64-debug-generic
perl merge.pl config-x86_64-debug-merged temp-x86_64-debug-generic x86_64 > kernel-2.6.32-x86_64-debug.config
perl merge.pl config-x86-generic-merged temp-generic > temp-x86-generic
perl merge.pl config-i686-merged temp-x86-generic > temp-i686-final
perl merge.pl config-i686-nodebug-merged temp-i686-final i386 > kernel-2.6.32-i686.config
perl merge.pl config-x86-generic-merged temp-debug-generic > temp-x86-debug-generic
perl merge.pl config-i686-merged temp-x86-debug-generic > temp-i686-debug-final
perl merge.pl config-i686-debug-merged temp-i686-debug-final i386 > kernel-2.6.32-i686-debug.config
perl merge.pl config-s390x-merged temp-generic > temp-s390x-generic
perl merge.pl config-s390x-merged temp-s390x-generic s390 > kernel-2.6.32-s390x.config
perl merge.pl config-s390x-merged temp-debug-generic > temp-s390x-debug-generic
perl merge.pl config-s390x-debug-merged temp-s390x-debug-generic s390 > kernel-2.6.32-s390x-debug.config
perl merge.pl config-s390x-kdump-merged kernel-2.6.32-s390x.config s390 > kernel-2.6.32-s390x-kdump.config
perl merge.pl config-powerpc-generic-merged temp-generic > temp-powerpc-generic
perl merge.pl config-powerpc64-merged temp-powerpc-generic powerpc > kernel-2.6.32-ppc64.config
perl merge.pl config-powerpc64-kdump-merged kernel-2.6.32-ppc64.config powerpc > kernel-2.6.32-ppc64-kdump.config
perl merge.pl config-powerpc-generic-merged temp-debug-generic > temp-powerpc-debug-generic
perl merge.pl config-powerpc64-merged temp-powerpc-debug-generic powerpc > temp-powerpc64-debug-generic
perl merge.pl config-powerpc64-debug-merged temp-powerpc64-debug-generic powerpc > kernel-2.6.32-ppc64-debug.config
+ ApplyOptionalPatch linux-kernel-test.patch
+ local patch=linux-kernel-test.patch
+ shift
+ '[' '!' -f /home/mockbuild/rpmbuild/SOURCES/linux-kernel-test.patch ']'
++ wc -l /home/mockbuild/rpmbuild/SOURCES/linux-kernel-test.patch
++ awk '{print $1}'
+ local C=1
+ '[' 1 -gt 9 ']'
+ chmod +x scripts/checkpatch.pl
+ mkdir configs
+ for cfg in 'kernel-2.6.32-*.config'
++ grep -c kernel-2.6.32-i686-debug.config
++ echo kernel-2.6.32-x86_64-debug.config kernel-2.6.32-x86_64.config
+ '[' 0 -eq 0 ']'
+ rm -f kernel-2.6.32-i686-debug.config
+ for cfg in 'kernel-2.6.32-*.config'
++ grep -c kernel-2.6.32-i686.config
++ echo kernel-2.6.32-x86_64-debug.config kernel-2.6.32-x86_64.config
+ '[' 0 -eq 0 ']'
+ rm -f kernel-2.6.32-i686.config
+ for cfg in 'kernel-2.6.32-*.config'
++ echo kernel-2.6.32-x86_64-debug.config kernel-2.6.32-x86_64.config
++ grep -c kernel-2.6.32-ppc64-debug.config
+ '[' 0 -eq 0 ']'
+ rm -f kernel-2.6.32-ppc64-debug.config
+ for cfg in 'kernel-2.6.32-*.config'
++ echo kernel-2.6.32-x86_64-debug.config kernel-2.6.32-x86_64.config
++ grep -c kernel-2.6.32-ppc64-kdump.config
+ '[' 0 -eq 0 ']'
+ rm -f kernel-2.6.32-ppc64-kdump.config
+ for cfg in 'kernel-2.6.32-*.config'
++ grep -c kernel-2.6.32-ppc64.config
++ echo kernel-2.6.32-x86_64-debug.config kernel-2.6.32-x86_64.config
+ '[' 0 -eq 0 ']'
+ rm -f kernel-2.6.32-ppc64.config
+ for cfg in 'kernel-2.6.32-*.config'
++ echo kernel-2.6.32-x86_64-debug.config kernel-2.6.32-x86_64.config
++ grep -c kernel-2.6.32-s390x-debug.config
+ '[' 0 -eq 0 ']'
+ rm -f kernel-2.6.32-s390x-debug.config
+ for cfg in 'kernel-2.6.32-*.config'
++ echo kernel-2.6.32-x86_64-debug.config kernel-2.6.32-x86_64.config
++ grep -c kernel-2.6.32-s390x-kdump.config
+ '[' 0 -eq 0 ']'
+ rm -f kernel-2.6.32-s390x-kdump.config
+ for cfg in 'kernel-2.6.32-*.config'
++ echo kernel-2.6.32-x86_64-debug.config kernel-2.6.32-x86_64.config
++ grep -c kernel-2.6.32-s390x.config
+ '[' 0 -eq 0 ']'
+ rm -f kernel-2.6.32-s390x.config
+ for cfg in 'kernel-2.6.32-*.config'
++ grep -c kernel-2.6.32-x86_64-debug.config
++ echo kernel-2.6.32-x86_64-debug.config kernel-2.6.32-x86_64.config
+ '[' 1 -eq 0 ']'
+ for cfg in 'kernel-2.6.32-*.config'
++ echo kernel-2.6.32-x86_64-debug.config kernel-2.6.32-x86_64.config
++ grep -c kernel-2.6.32-x86_64.config
+ '[' 1 -eq 0 ']'
+ for i in '*.config'
+ mv kernel-2.6.32-x86_64-debug.config .config
++ head -1 .config
++ cut -b 3-
+ Arch=x86_64
+ make ARCH=x86_64 nonint_oldconfig
+ echo '# x86_64'
+ cat .config
+ rm -f include/generated/kernel.arch
+ rm -f include/generated/kernel.cross
+ for i in '*.config'
+ mv kernel-2.6.32-x86_64.config .config
++ head -1 .config
++ cut -b 3-
+ Arch=x86_64
+ make ARCH=x86_64 nonint_oldconfig
+ echo '# x86_64'
+ cat .config
+ rm -f include/generated/kernel.arch
+ rm -f include/generated/kernel.cross
+ find . '(' -name '*.orig' -o -name '*~' ')' -exec rm -f '{}' ';'
+ cp /home/mockbuild/rpmbuild/SOURCES/extrakeys.pub .
+ cat
###
### Now generating a PGP key pair to be used for signing modules.
###
### If this takes a long time, you might wish to run rngd in the background to
### keep the supply of entropy topped up. It needs to be run as root, and
### should use a hardware random number generator if one is available, eg:
###
### rngd -r /dev/hwrandom
###
### If one isn't available, the pseudo-random number generator can be used:
###
### rngd -r /dev/urandom
###
+ gpg --homedir . --batch --gen-key /home/mockbuild/rpmbuild/SOURCES/genkey
gpg: WARNING: unsafe permissions on homedir `.'
gpg: keyring `./secring.gpg' created
gpg: keyring `./pubring.gpg' created
+ cat
###
### Key pair generated.
###
+ '[' -s /home/mockbuild/rpmbuild/SOURCES/extrakeys.pub ']'
+ gpg --homedir . --no-default-keyring --keyring kernel.pub --import /home/mockbuild/rpmbuild/SOURCES/extrakeys.pub
gpg: WARNING: unsafe permissions on homedir `.'
gpg: ./trustdb.gpg: trustdb created
gpg: key CD09BEDA: public key "Red Hat Enterprise Linux Driver Update Program <secalert@redhat.com>" imported
gpg: Total number processed: 1
gpg: imported: 1
+ gpg --homedir . --export --keyring ./kernel.pub CentOS
gpg: WARNING: unsafe permissions on homedir `.'
+ gcc -o scripts/bin2c scripts/bin2c.c
+ scripts/bin2c ksign_def_public_key __initdata
+ cd ..
+ exit 0
[mockbuild@oracle651 ~]
Now execute the following command as shown below.
[mockbuild@oracle651 ~]$ svn co https://svn.code.sf.net/p/scst/svn/branches/2.2.x scst
The scst code is installed under the /home/mockbuild directory. Now Change directory as shown below.
[mockbuild@oracle651 ~]$ cd ~/rpmbuild/BUILD/kernel-2.6.32-431.23.3.el6/linux-2.6.32-431.23.3.el6.x86_64/
Now apply the two required patches for SCST as shown below.
[mockbuild@oracle651 ~]$ cd ~/rpmbuild/BUILD/kernel-2.6.32-431.23.3.el6/linux-2.6.32-431.23.3.el6.x86_64/
[mockbuild@oracle651 linux-2.6.32-431.23.3.el6.x86_64]$ patch -p1 < ~/scst/scst/kernel/rhel/scst_exec_req_fifo-2.6.32.patch
patching file block/blk-map.c
Hunk #2 succeeded at 275 (offset 3 lines).
patching file include/linux/blkdev.h
Hunk #1 succeeded at 697 (offset -11 lines).
Hunk #2 succeeded at 811 (offset -3 lines).
patching file include/linux/scatterlist.h
patching file lib/scatterlist.c
[mockbuild@oracle651 linux-2.6.32-431.23.3.el6.x86_64]$ patch -p1 < ~/scst/iscsi-scst/kernel/patches/rhel/put_page_callback-2.6.32-220.patch
patching file include/linux/Kbuild
patching file include/linux/mm_types.h
patching file include/linux/net.h
Hunk #2 succeeded at 390 (offset 23 lines).
patching file net/core/dev.c
Hunk #1 succeeded at 3622 (offset 625 lines).
patching file net/core/skbuff.c
Hunk #2 succeeded at 394 (offset 50 lines).
Hunk #3 succeeded at 699 (offset 51 lines).
Hunk #4 succeeded at 714 (offset 51 lines).
Hunk #5 succeeded at 880 (offset 40 lines).
Hunk #6 succeeded at 961 with fuzz 2 (offset 35 lines).
Hunk #7 succeeded at 1240 with fuzz 2 (offset 38 lines).
Hunk #8 succeeded at 1409 (offset 38 lines).
Hunk #9 succeeded at 1511 (offset 38 lines).
Hunk #10 succeeded at 1535 (offset 38 lines).
Hunk #11 succeeded at 1545 (offset 38 lines).
Hunk #12 succeeded at 1566 (offset 38 lines).
Hunk #13 succeeded at 2196 (offset 38 lines).
Hunk #14 succeeded at 2319 (offset 39 lines).
Hunk #15 succeeded at 2341 (offset 39 lines).
Hunk #16 succeeded at 2753 (offset 56 lines).
patching file net/ipv4/ip_output.c
Hunk #1 succeeded at 998 (offset 17 lines).
Hunk #2 succeeded at 1235 (offset 22 lines).
patching file net/ipv4/Makefile
Hunk #1 succeeded at 51 (offset 2 lines).
patching file net/ipv4/tcp.c
Hunk #1 succeeded at 820 with fuzz 1 (offset 21 lines).
Hunk #2 succeeded at 1029 (offset 22 lines).
Hunk #3 succeeded at 1070 (offset 22 lines).
patching file net/ipv4/tcp_output.c
Hunk #1 succeeded at 1061 (offset 150 lines).
patching file net/ipv4/tcp_zero_copy.c
patching file net/ipv6/ip6_output.c
Hunk #1 succeeded at 1361 (offset -39 lines).
patching file net/Kconfig
[mockbuild@oracle651 linux-2.6.32-431.23.3.el6.x86_64]$
Check to make sure the config files are about the same. Here they only differ by comment lines so that is acceptable.
[mockbuild@oracle651 linux-2.6.32-431.23.3.el6.x86_64]$ diff .config /boot/config-2.6.32-431.23.3.el6.x86_64
3,4c3,4
< # Linux kernel version: 2.6.32
< # Mon Aug 11 14:27:09 2014
---
> # Linux kernel version: 2.6.32-431.23.3.el6.x86_64
> # Tue Jul 29 09:15:22 2014
[mockbuild@oracle651 linux-2.6.32-431.23.3.el6.x86_64]$
Now do the menuconfig step as shown below. The menuconfig GUI open. Navigate through the GUI and choose the options as quoted and described at the spiderweb reference here. For completeness, those changes that need to be made and verified in menuconfig are as follows from the reference but restated here for convenience.
Select “Network support”->”Networking options” make sure “TCP/IP networking” is checked (should be) and check “TCP/IP zero-copy transfer completion notification” underneath so SCST will be notified when a transfer of data across the nic has finished. This way, it can hurry up & send the next block. Should speed things up.
Next, verify that “Device Drivers”->”SCSI device support”->”SCSI disk support” is already checked. Again, for RHEL 6.4 it should be.
Verify that “Enable the block layer”->”IO Schedulers”->”CFQ I/O scheduler” and “Default I/O scheduler” set to CFQ are both set.
Lastly, set “Processor type an features”->”Preemption Model” is set to “No Forced Preemption” versus Desktop mode.
[mockbuild@oracle651 linux-2.6.32-431.23.3.el6.x86_64]$ make menuconfig
HOSTCC scripts/kconfig/lxdialog/checklist.o
HOSTCC scripts/kconfig/lxdialog/inputbox.o
HOSTCC scripts/kconfig/lxdialog/menubox.o
HOSTCC scripts/kconfig/lxdialog/textbox.o
HOSTCC scripts/kconfig/lxdialog/util.o
HOSTCC scripts/kconfig/lxdialog/yesno.o
HOSTCC scripts/kconfig/mconf.o
HOSTLD scripts/kconfig/mconf
scripts/kconfig/mconf arch/x86/Kconfig
#
# configuration written to .config
#
*** End of Linux kernel configuration.
*** Execute 'make' to build the kernel or try 'make help'.
[mockbuild@oracle651 linux-2.6.32-431.23.3.el6.x86_64]$
Now run make as shown below.
[mockbuild@oracle651 linux-2.6.32-431.23.3.el6.x86_64]$ time make -j8 all | tee makescst.log
Now make modules install as shown below, as the "root" user.
[mockbuild@oracle651 linux-2.6.32-431.23.3.el6.x86_64]$ su - root
[root@oracle651 linux-2.6.32-431.23.3.el6.x86_64]# cd /home/mockbuild/rpmbuild/BUILD/kernel-2.6.32-431.23.3.el6/linux-2.6.32-431.23.3.el6.x86_64
[root@oracle651 linux-2.6.32-431.23.3.el6.x86_64]# make modules_install
Next, again as root, do a "make install" as shown below.
[root@oracle651 linux-2.6.32-431.23.3.el6.x86_64]# make install | tee make-install.log
sh /home/mockbuild/rpmbuild/BUILD/kernel-2.6.32-431.23.3.el6/linux-2.6.32-431.23.3.el6.x86_64/arch/x86/boot/install.sh 2.6.32-scst arch/x86/boot/bzImage \
System.map "/boot"
[root@oracle651 linux-2.6.32-431.23.3.el6.x86_64]#
Now edit /boot/grub/menu.lst to update the server to use the new kernel at boot as shown below. Change the value of "default=" to use the new kernel. The first kernel listed is always the "0" (zero) kernel so in this example set "detault=0". Also, while editing this file, remove the "rhgb quiet" from that kernel line so that all boot output messages will be buffered to console, and also add at the end of the kernel line "console=ttyS0" so that KVM console is configured.
# grub.conf generated by anaconda
#
# Note that you do not have to rerun grub after making changes to this file
# NOTICE: You have a /boot partition. This means that
# all kernel and initrd paths are relative to /boot/, eg.
# root (hd0,0)
# kernel /vmlinuz-version ro root=/dev/mapper/vg_oracle651-lv_root
# initrd /initrd-[generic-]version.img
#boot=/dev/vda
default=0
timeout=5
splashimage=(hd0,0)/grub/splash.xpm.gz
hiddenmenu
title Oracle Linux Server Red Hat Compatible Kernel (2.6.32-scst)
root (hd0,0)
kernel /vmlinuz-2.6.32-scst ro root=/dev/mapper/vg_oracle651-lv_root rd_NO_LUKS LANG=en_US.UTF-8 rd_LVM_LV=vg_oracle651/lv_swap rd_NO_MD rd_LVM_LV=vg_oracle651/lv_root SYSFONT=latarcyrheb-sun16 KEYBOARDTYPE=pc KEYTABLE=us rd_NO_DM crashkernel=auto console=ttyS0
initrd /initramfs-2.6.32-scst.img
title Oracle Linux Server (3.8.13-35.3.4.el6uek.x86_64)
root (hd0,0)
kernel /vmlinuz-3.8.13-35.3.4.el6uek.x86_64 ro root=/dev/mapper/vg_oracle651-lv_root rd_NO_LUKS LANG=en_US.UTF-8 rd_LVM_LV=vg_oracle651/lv_swap rd_NO_MD rd_LVM_LV=vg_oracle651/lv_root SYSFONT=latarcyrheb-sun16 KEYBOARDTYPE=pc KEYTABLE=us rd_NO_DM rhgb quiet
initrd /initramfs-3.8.13-35.3.4.el6uek.x86_64.img
title Oracle Linux Server Red Hat Compatible Kernel (2.6.32-431.23.3.el6.x86_64)
root (hd0,0)
kernel /vmlinuz-2.6.32-431.23.3.el6.x86_64 ro root=/dev/mapper/vg_oracle651-lv_root rd_NO_LUKS LANG=en_US.UTF-8 rd_LVM_LV=vg_oracle651/lv_swap rd_NO_MD rd_LVM_LV=vg_oracle651/lv_root SYSFONT=latarcyrheb-sun16 KEYBOARDTYPE=pc KEYTABLE=us rd_NO_DM rhgb quiet crashkernel=auto
initrd /initramfs-2.6.32-431.23.3.el6.x86_64.img
title Oracle Linux Server Unbreakable Enterprise Kernel (3.8.13-16.2.1.el6uek.x86_64)
root (hd0,0)
kernel /vmlinuz-3.8.13-16.2.1.el6uek.x86_64 ro root=/dev/mapper/vg_oracle651-lv_root rd_NO_LUKS LANG=en_US.UTF-8 rd_LVM_LV=vg_oracle651/lv_swap rd_NO_MD rd_LVM_LV=vg_oracle651/lv_root SYSFONT=latarcyrheb-sun16 KEYBOARDTYPE=pc KEYTABLE=us rd_NO_DM rhgb quiet
initrd /initramfs-3.8.13-16.2.1.el6uek.x86_64.img
title Oracle Linux Server Red Hat Compatible Kernel (2.6.32-431.el6.x86_64)
root (hd0,0)
kernel /vmlinuz-2.6.32-431.el6.x86_64 ro root=/dev/mapper/vg_oracle651-lv_root rd_NO_LUKS LANG=en_US.UTF-8 rd_LVM_LV=vg_oracle651/lv_swap rd_NO_MD rd_LVM_LV=vg_oracle651/lv_root SYSFONT=latarcyrheb-sun16 crashkernel=auto KEYBOARDTYPE=pc KEYTABLE=us rd_NO_DM rhgb quiet
initrd /initramfs-2.6.32-431.el6.x86_64.img
Now copy the .config file over to the "/boot" directory as shown below. Expected output and commands are shown below.
[root@oracle651 linux-2.6.32-431.23.3.el6.x86_64]# pwd
/home/mockbuild/rpmbuild/BUILD/kernel-2.6.32-431.23.3.el6/linux-2.6.32-431.23.3.el6.x86_64
[root@oracle651 linux-2.6.32-431.23.3.el6.x86_64]# cp -p .config /boot/config-2.6.32-431.23.3-scst.el6.x86_64
[root@oracle651 linux-2.6.32-431.23.3.el6.x86_64]# ls -lrt /boot/config-2.6* | sort
-rw-r--r--. 1 root root 105195 Nov 21 2013 /boot/config-2.6.32-431.el6.x86_64
-rw-r--r--. 1 root root 105200 Jul 29 11:42 /boot/config-2.6.32-431.23.3.el6.x86_64
-rw-rw-r--. 1 mockbuild mockbuild 105242 Aug 11 15:13 /boot/config-2.6.32-431.23.3-scst.el6.x86_64
[root@oracle651 linux-2.6.32-431.23.3.el6.x86_64]#
Reboot the KVM guest. Be sure the config file has been named exactly as shown above before rebooting, or if your kernel version was slightly different, be sure the kernel version and format of the name of the config file is of the format "config-ver.s.ion-scst.el6.x86_64".
Next change the various Makefiles to release version by issuing “make debug2release” in the main scst dir. This will remove the debugging stuff. Do this as root.
[root@oracle651 scst]# pwd
/home/mockbuild/scst
[root@oracle651 scst]# id
uid=0(root) gid=0(root) groups=0(root) context=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023
[root@oracle651 scst]# make debug2release | tee make_debug2release.log
rm -f TAGS
cd scst && make extraclean
make[1]: Entering directory `/home/mockbuild/scst/scst'
cd src && make extraclean
make[2]: Entering directory `/home/mockbuild/scst/scst/src'
rm -f *.o *.ko .*.cmd *.mod.c .*.d .depend Modules.symvers \
Module.symvers Module.markers modules.order
rm -rf .tmp_versions
cd dev_handlers && make clean
make[3]: Entering directory `/home/mockbuild/scst/scst/src/dev_handlers'
rm -f *.o *.ko .*.cmd *.mod.c .*.d .depend Modules.symvers \
Module.symvers Module.markers modules.order
rm -rf .tmp_versions
make[3]: Leaving directory `/home/mockbuild/scst/scst/src/dev_handlers'
cd dev_handlers && make extraclean
make[3]: Entering directory `/home/mockbuild/scst/scst/src/dev_handlers'
rm -f *.o *.ko .*.cmd *.mod.c .*.d .depend Modules.symvers \
Module.symvers Module.markers modules.order
rm -rf .tmp_versions
rm -f *.orig *.rej
make[3]: Leaving directory `/home/mockbuild/scst/scst/src/dev_handlers'
rm -f *.orig *.rej
make[2]: Leaving directory `/home/mockbuild/scst/scst/src'
make[1]: Leaving directory `/home/mockbuild/scst/scst'
make[1]: Entering directory `/home/mockbuild/scst/doc'
rm -f *.txt *.html *.tex *.dvi *.ps *.pdf *.info *.lyx *.rtf
rm -f *.orig *.rej
make[1]: Leaving directory `/home/mockbuild/scst/doc'
make[1]: Entering directory `/home/mockbuild/scst/qla2x00t'
rm -f *.o *.ko .*.cmd *.mod.c .*.d .depend *~ Modules.symvers \
Module.symvers Module.markers modules.order
rm -rf .tmp_versions
rm -f *.orig *.rej
make[1]: Leaving directory `/home/mockbuild/scst/qla2x00t'
make[1]: Entering directory `/home/mockbuild/scst/qla2x00t/qla2x00-target'
rm -f *.o *.ko .*.cmd *.mod.c .*.d .depend *~ Modules.symvers \
Module.symvers Module.markers modules.order
rm -rf .tmp_versions
rm -f *.orig *.rej
make[1]: Leaving directory `/home/mockbuild/scst/qla2x00t/qla2x00-target'
make[1]: Entering directory `/home/mockbuild/scst/srpt'
make -C /lib/modules/2.6.32-scst/build SUBDIRS=/home/mockbuild/scst/srpt/src clean
make[2]: Entering directory `/home/mockbuild/rpmbuild/BUILD/kernel-2.6.32-431.23.3.el6/linux-2.6.32-431.23.3.el6.x86_64'
make[2]: Leaving directory `/home/mockbuild/rpmbuild/BUILD/kernel-2.6.32-431.23.3.el6/linux-2.6.32-431.23.3.el6.x86_64'
rm -f src/Modules.symvers src/Module.symvers src/Module.markers \
src/modules.order
rm -f *.orig *.rej
make[1]: Leaving directory `/home/mockbuild/scst/srpt'
make[1]: Entering directory `/home/mockbuild/scst/iscsi-scst'
make -C usr extraclean
make[2]: Entering directory `/home/mockbuild/scst/iscsi-scst/usr'
rm -f *.o iscsi-scstd iscsi-scst-adm .depend*
rm -f *.orig *.rej
make[2]: Leaving directory `/home/mockbuild/scst/iscsi-scst/usr'
make -C /lib/modules/2.6.32-scst/build SUBDIRS=/home/mockbuild/scst/iscsi-scst/kernel clean
make[2]: Entering directory `/home/mockbuild/rpmbuild/BUILD/kernel-2.6.32-431.23.3.el6/linux-2.6.32-431.23.3.el6.x86_64'
make[2]: Leaving directory `/home/mockbuild/rpmbuild/BUILD/kernel-2.6.32-431.23.3.el6/linux-2.6.32-431.23.3.el6.x86_64'
rm -f kernel/Modules.symvers kernel/Module.symvers \
kernel/Module.markers kernel/modules.order \
include/iscsi_scst_itf_ver.h \
kernel/*.orig kernel/*.rej
make[1]: Leaving directory `/home/mockbuild/scst/iscsi-scst'
make[1]: Entering directory `/home/mockbuild/scst/usr/fileio'
rm -f *.o fileio_tgt .depend*
rm -f *.orig *.rej
make[1]: Leaving directory `/home/mockbuild/scst/usr/fileio'
make[1]: Entering directory `/home/mockbuild/scst/scst_local'
make[2]: Entering directory `/home/mockbuild/rpmbuild/BUILD/kernel-2.6.32-431.23.3.el6/linux-2.6.32-431.23.3.el6.x86_64'
make[2]: Leaving directory `/home/mockbuild/rpmbuild/BUILD/kernel-2.6.32-431.23.3.el6/linux-2.6.32-431.23.3.el6.x86_64'
rm -f *.orig *.rej
make[1]: Leaving directory `/home/mockbuild/scst/scst_local'
cd scst && make debug2release
make[1]: Entering directory `/home/mockbuild/scst/scst'
cd src && make debug2release
make[2]: Entering directory `/home/mockbuild/scst/scst/src'
sed -i.aa s/"^EXTRA_CFLAGS += \-DCONFIG_SCST_EXTRACHECKS"/"#EXTRA_CFLAGS += \-DCONFIG_SCST_EXTRACHECKS"/ Makefile
sed -i.aa s/"#EXTRA_CFLAGS += \-DCONFIG_SCST_TRACING"/"EXTRA_CFLAGS += \-DCONFIG_SCST_TRACING"/ Makefile
sed -i.aa s/"^EXTRA_CFLAGS += \-DCONFIG_SCST_DEBUG -g -fno-inline -fno-inline-functions"/"#EXTRA_CFLAGS += \-DCONFIG_SCST_DEBUG -g -fno-inline -fno-inline-functions"/ Makefile
rm Makefile.aa
cd dev_handlers && make debug2release
make[3]: Entering directory `/home/mockbuild/scst/scst/src/dev_handlers'
sed -i.aa s/"^EXTRA_CFLAGS += \-DCONFIG_SCST_EXTRACHECKS"/"#EXTRA_CFLAGS += \-DCONFIG_SCST_EXTRACHECKS"/ Makefile
sed -i.aa s/"#EXTRA_CFLAGS += \-DCONFIG_SCST_TRACING"/"EXTRA_CFLAGS += \-DCONFIG_SCST_TRACING"/ Makefile
sed -i.aa s/"^EXTRA_CFLAGS += \-DCONFIG_SCST_DEBUG -g -fno-inline -fno-inline-functions"/"#EXTRA_CFLAGS += \-DCONFIG_SCST_DEBUG -g -fno-inline -fno-inline-functions"/ Makefile
rm Makefile.aa
make[3]: Leaving directory `/home/mockbuild/scst/scst/src/dev_handlers'
make[2]: Leaving directory `/home/mockbuild/scst/scst/src'
make[1]: Leaving directory `/home/mockbuild/scst/scst'
make[1]: Entering directory `/home/mockbuild/scst/qla2x00t/qla2x00-target'
sed -i.aa s/"^EXTRA_CFLAGS += \-DCONFIG_SCST_EXTRACHECKS"/"#EXTRA_CFLAGS += \-DCONFIG_SCST_EXTRACHECKS"/ Makefile
sed -i.aa s/"#EXTRA_CFLAGS += \-DCONFIG_SCST_TRACING"/"EXTRA_CFLAGS += \-DCONFIG_SCST_TRACING"/ Makefile
sed -i.aa s/"^EXTRA_CFLAGS += \-DCONFIG_SCST_DEBUG -g -fno-inline -fno-inline-functions"/"#EXTRA_CFLAGS += \-DCONFIG_SCST_DEBUG -g -fno-inline -fno-inline-functions"/ Makefile
rm Makefile.aa
make[1]: Leaving directory `/home/mockbuild/scst/qla2x00t/qla2x00-target'
make[1]: Entering directory `/home/mockbuild/scst/srpt'
make clean
make[2]: Entering directory `/home/mockbuild/scst/srpt'
make -C /lib/modules/2.6.32-scst/build SUBDIRS=/home/mockbuild/scst/srpt/src clean
make[3]: Entering directory `/home/mockbuild/rpmbuild/BUILD/kernel-2.6.32-431.23.3.el6/linux-2.6.32-431.23.3.el6.x86_64'
make[3]: Leaving directory `/home/mockbuild/rpmbuild/BUILD/kernel-2.6.32-431.23.3.el6/linux-2.6.32-431.23.3.el6.x86_64'
rm -f src/Modules.symvers src/Module.symvers src/Module.markers \
src/modules.order
make[2]: Leaving directory `/home/mockbuild/scst/srpt'
make[1]: Leaving directory `/home/mockbuild/scst/srpt'
make[1]: Entering directory `/home/mockbuild/scst/iscsi-scst'
sed -i.aa s/"^EXTRA_CFLAGS += \-DCONFIG_SCST_EXTRACHECKS"/"#EXTRA_CFLAGS += \-DCONFIG_SCST_EXTRACHECKS"/ /home/mockbuild/scst/iscsi-scst/kernel/Makefile
sed -i.aa s/"#EXTRA_CFLAGS += \-DCONFIG_SCST_TRACING"/"EXTRA_CFLAGS += \-DCONFIG_SCST_TRACING"/ /home/mockbuild/scst/iscsi-scst/kernel/Makefile
sed -i.aa s/"^EXTRA_CFLAGS += \-DCONFIG_SCST_DEBUG -g -fno-inline -fno-inline-functions"/"#EXTRA_CFLAGS += \-DCONFIG_SCST_DEBUG -g -fno-inline -fno-inline-functions"/ /home/mockbuild/scst/iscsi-scst/kernel/Makefile
rm /home/mockbuild/scst/iscsi-scst/kernel/Makefile.aa
make[1]: Leaving directory `/home/mockbuild/scst/iscsi-scst'
make[1]: Entering directory `/home/mockbuild/scst/usr/fileio'
sed -i.aa s/"^CFLAGS += \-DEXTRACHECKS"/"#CFLAGS += \-DEXTRACHECKS"/ Makefile
sed -i.aa s/"#CFLAGS += \-DTRACING"/"CFLAGS += \-DTRACING"/ Makefile
sed -i.aa s/"^CFLAGS += \-DDEBUG -g -fno-inline -fno-inline-functions"/"#CFLAGS += \-DDEBUG -g -fno-inline -fno-inline-functions"/ Makefile
rm Makefile.aa
make[1]: Leaving directory `/home/mockbuild/scst/usr/fileio'
make[1]: Entering directory `/home/mockbuild/scst/scst_local'
sed -i.aa s/"^EXTRA_CFLAGS += \-DCONFIG_SCST_EXTRACHECKS"/"#EXTRA_CFLAGS += \-DCONFIG_SCST_EXTRACHECKS"/ Makefile
sed -i.aa s/"#EXTRA_CFLAGS += \-DCONFIG_SCST_TRACING"/"EXTRA_CFLAGS += \-DCONFIG_SCST_TRACING"/ Makefile
sed -i.aa s/"^EXTRA_CFLAGS += \-DCONFIG_SCST_DEBUG -g -fno-inline -fno-inline-functions"/"#EXTRA_CFLAGS += \-DCONFIG_SCST_DEBUG -g -fno-inline -fno-inline-functions"/ Makefile
rm Makefile.aa
make[1]: Leaving directory `/home/mockbuild/scst/scst_local'
[root@oracle651 scst]#
Now go ahead and issue “make scst scst_install” as root in the mockbuild directory’s scst folder. Commands and expected output are shown below. Do this as "root" user.
[root@oracle651 scst]# pwd
/home/mockbuild/scst
[root@oracle651 scst]# id
uid=0(root) gid=0(root) groups=0(root) context=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023
[root@oracle651 scst]# make scst scst_install | tee make_scst_scst_install.log
cd scst && make all
make[1]: Entering directory `/home/mockbuild/scst/scst'
cd src && make all
make[2]: Entering directory `/home/mockbuild/scst/scst/src'
make -C /lib/modules/2.6.32-scst/build SUBDIRS=/home/mockbuild/scst/scst/src BUILD_DEV=m
make[3]: Entering directory `/home/mockbuild/rpmbuild/BUILD/kernel-2.6.32-431.23.3.el6/linux-2.6.32-431.23.3.el6.x86_64'
LD /home/mockbuild/scst/scst/src/built-in.o
CC [M] /home/mockbuild/scst/scst/src/scst_main.o
CC [M] /home/mockbuild/scst/scst/src/scst_targ.o
CC [M] /home/mockbuild/scst/scst/src/scst_lib.o
CC [M] /home/mockbuild/scst/scst/src/scst_sysfs.o
CC [M] /home/mockbuild/scst/scst/src/scst_mem.o
CC [M] /home/mockbuild/scst/scst/src/scst_debug.o
CC [M] /home/mockbuild/scst/scst/src/scst_pres.o
CC [M] /home/mockbuild/scst/scst/src/scst_tg.o
LD [M] /home/mockbuild/scst/scst/src/scst.o
LD /home/mockbuild/scst/scst/src/dev_handlers/built-in.o
CC [M] /home/mockbuild/scst/scst/src/dev_handlers/scst_cdrom.o
CC [M] /home/mockbuild/scst/scst/src/dev_handlers/scst_changer.o
CC [M] /home/mockbuild/scst/scst/src/dev_handlers/scst_disk.o
CC [M] /home/mockbuild/scst/scst/src/dev_handlers/scst_modisk.o
CC [M] /home/mockbuild/scst/scst/src/dev_handlers/scst_tape.o
CC [M] /home/mockbuild/scst/scst/src/dev_handlers/scst_vdisk.o
CC [M] /home/mockbuild/scst/scst/src/dev_handlers/scst_raid.o
CC [M] /home/mockbuild/scst/scst/src/dev_handlers/scst_processor.o
CC [M] /home/mockbuild/scst/scst/src/dev_handlers/scst_user.o
Building modules, stage 2.
MODPOST 10 modules
CC /home/mockbuild/scst/scst/src/dev_handlers/scst_cdrom.mod.o
LD [M] /home/mockbuild/scst/scst/src/dev_handlers/scst_cdrom.ko.unsigned
NO SIGN [M] /home/mockbuild/scst/scst/src/dev_handlers/scst_cdrom.ko
CC /home/mockbuild/scst/scst/src/dev_handlers/scst_changer.mod.o
LD [M] /home/mockbuild/scst/scst/src/dev_handlers/scst_changer.ko.unsigned
NO SIGN [M] /home/mockbuild/scst/scst/src/dev_handlers/scst_changer.ko
CC /home/mockbuild/scst/scst/src/dev_handlers/scst_disk.mod.o
LD [M] /home/mockbuild/scst/scst/src/dev_handlers/scst_disk.ko.unsigned
NO SIGN [M] /home/mockbuild/scst/scst/src/dev_handlers/scst_disk.ko
CC /home/mockbuild/scst/scst/src/dev_handlers/scst_modisk.mod.o
LD [M] /home/mockbuild/scst/scst/src/dev_handlers/scst_modisk.ko.unsigned
NO SIGN [M] /home/mockbuild/scst/scst/src/dev_handlers/scst_modisk.ko
CC /home/mockbuild/scst/scst/src/dev_handlers/scst_processor.mod.o
LD [M] /home/mockbuild/scst/scst/src/dev_handlers/scst_processor.ko.unsigned
NO SIGN [M] /home/mockbuild/scst/scst/src/dev_handlers/scst_processor.ko
CC /home/mockbuild/scst/scst/src/dev_handlers/scst_raid.mod.o
LD [M] /home/mockbuild/scst/scst/src/dev_handlers/scst_raid.ko.unsigned
NO SIGN [M] /home/mockbuild/scst/scst/src/dev_handlers/scst_raid.ko
CC /home/mockbuild/scst/scst/src/dev_handlers/scst_tape.mod.o
LD [M] /home/mockbuild/scst/scst/src/dev_handlers/scst_tape.ko.unsigned
NO SIGN [M] /home/mockbuild/scst/scst/src/dev_handlers/scst_tape.ko
CC /home/mockbuild/scst/scst/src/dev_handlers/scst_user.mod.o
LD [M] /home/mockbuild/scst/scst/src/dev_handlers/scst_user.ko.unsigned
NO SIGN [M] /home/mockbuild/scst/scst/src/dev_handlers/scst_user.ko
CC /home/mockbuild/scst/scst/src/dev_handlers/scst_vdisk.mod.o
LD [M] /home/mockbuild/scst/scst/src/dev_handlers/scst_vdisk.ko.unsigned
NO SIGN [M] /home/mockbuild/scst/scst/src/dev_handlers/scst_vdisk.ko
CC /home/mockbuild/scst/scst/src/scst.mod.o
LD [M] /home/mockbuild/scst/scst/src/scst.ko.unsigned
NO SIGN [M] /home/mockbuild/scst/scst/src/scst.ko
make[3]: Leaving directory `/home/mockbuild/rpmbuild/BUILD/kernel-2.6.32-431.23.3.el6/linux-2.6.32-431.23.3.el6.x86_64'
make[2]: Leaving directory `/home/mockbuild/scst/scst/src'
make[1]: Leaving directory `/home/mockbuild/scst/scst'
cd scst && make install
make[1]: Entering directory `/home/mockbuild/scst/scst'
cd src && make install
make[2]: Entering directory `/home/mockbuild/scst/scst/src'
make -C /lib/modules/2.6.32-scst/build SUBDIRS=/home/mockbuild/scst/scst/src BUILD_DEV=m
make[3]: Entering directory `/home/mockbuild/rpmbuild/BUILD/kernel-2.6.32-431.23.3.el6/linux-2.6.32-431.23.3.el6.x86_64'
Building modules, stage 2.
MODPOST 10 modules
make[3]: Leaving directory `/home/mockbuild/rpmbuild/BUILD/kernel-2.6.32-431.23.3.el6/linux-2.6.32-431.23.3.el6.x86_64'
rm -f /lib/modules/2.6.32-scst/extra/scsi_tgt.ko
install -d /lib/modules/2.6.32-scst/extra
install -d /lib/modules/2.6.32-scst/extra/dev_handlers
install -m 644 dev_handlers/*.ko /lib/modules/2.6.32-scst/extra/dev_handlers
install -m 644 scst.ko /lib/modules/2.6.32-scst/extra
install -d /usr/local/include/scst
install -m 644 ../include/scst.h /usr/local/include/scst
install -m 644 ../include/scst_sgv.h /usr/local/include/scst
install -m 644 ../include/scst_debug.h /usr/local/include/scst
install -m 644 ../include/scst_user.h /usr/local/include/scst
install -m 644 ../include/scst_const.h /usr/local/include/scst
rm -f /usr/local/include/scst/Modules.symvers
install -m 644 Module.symvers /usr/local/include/scst
/sbin/depmod -a 2.6.32-scst
mkdir -p /var/lib/scst/pr
****************************************************************
*!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!*
*!! !!*
*!! Now don't forget to rebuild and reinstall all your !!*
*!! target drivers, custom dev handlers and necessary user !!*
*!! space applications. Otherwise, because of the versions !!*
*!! mismatch, you could have many problems and crashes. !!*
*!! See IMPORTANT note in the "Installation" section of !!*
*!! SCST's README file for more info. !!*
*!! !!*
*!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!*
****************************************************************
make[2]: Leaving directory `/home/mockbuild/scst/scst/src'
make[1]: Leaving directory `/home/mockbuild/scst/scst'
[root@oracle651 scst]#
Issue “make iscsi iscsi_install” as root in the /home/mockbuild folder. Commands and expected output are shown below.
[root@oracle651 scst]# pwd
/home/mockbuild/scst
[root@oracle651 scst]# id
uid=0(root) gid=0(root) groups=0(root) context=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023
[root@oracle651 scst]# make iscsi iscsi_install | tee make_iscsi_install
cd iscsi-scst && make all
make[1]: Entering directory `/home/mockbuild/scst/iscsi-scst'
echo "/* Autogenerated, don't edit */" >include/iscsi_scst_itf_ver.h
echo "" >>include/iscsi_scst_itf_ver.h
echo -n "#define ISCSI_SCST_INTERFACE_VERSION " >>include/iscsi_scst_itf_ver.h
echo -n "ISCSI_VERSION_STRING \"_\" " >>include/iscsi_scst_itf_ver.h
echo "\"`sha1sum include/iscsi_scst.h|awk '{printf $1}'`\"" >>include/iscsi_scst_itf_ver.h
make -C usr SCST_INC_DIR=/home/mockbuild/scst/iscsi-scst/../scst/include
make[2]: Entering directory `/home/mockbuild/scst/iscsi-scst/usr'
cc -M -O2 -fno-inline -Wall -Wextra -Wstrict-prototypes -Wno-sign-compare -Wimplicit-function-declaration -Wno-unused-parameter -Wno-missing-field-initializers -g -I../include -I/home/mockbuild/scst/iscsi-scst/../scst/include -D_GNU_SOURCE iscsid.c iscsi_scstd.c conn.c session.c target.c message.c ctldev.c log.c chap.c event.c param.c config.c isns.c md5.c sha1.c >.depend_d
cc -c -o iscsid.o -O2 -fno-inline -Wall -Wextra -Wstrict-prototypes -Wno-sign-compare -Wimplicit-function-declaration -Wno-unused-parameter -Wno-missing-field-initializers -g -I../include -I/home/mockbuild/scst/iscsi-scst/../scst/include -D_GNU_SOURCE iscsid.c
cc -c -o iscsi_scstd.o -O2 -fno-inline -Wall -Wextra -Wstrict-prototypes -Wno-sign-compare -Wimplicit-function-declaration -Wno-unused-parameter -Wno-missing-field-initializers -g -I../include -I/home/mockbuild/scst/iscsi-scst/../scst/include -D_GNU_SOURCE iscsi_scstd.c
cc -c -o conn.o -O2 -fno-inline -Wall -Wextra -Wstrict-prototypes -Wno-sign-compare -Wimplicit-function-declaration -Wno-unused-parameter -Wno-missing-field-initializers -g -I../include -I/home/mockbuild/scst/iscsi-scst/../scst/include -D_GNU_SOURCE conn.c
cc -c -o session.o -O2 -fno-inline -Wall -Wextra -Wstrict-prototypes -Wno-sign-compare -Wimplicit-function-declaration -Wno-unused-parameter -Wno-missing-field-initializers -g -I../include -I/home/mockbuild/scst/iscsi-scst/../scst/include -D_GNU_SOURCE session.c
cc -c -o target.o -O2 -fno-inline -Wall -Wextra -Wstrict-prototypes -Wno-sign-compare -Wimplicit-function-declaration -Wno-unused-parameter -Wno-missing-field-initializers -g -I../include -I/home/mockbuild/scst/iscsi-scst/../scst/include -D_GNU_SOURCE target.c
cc -c -o message.o -O2 -fno-inline -Wall -Wextra -Wstrict-prototypes -Wno-sign-compare -Wimplicit-function-declaration -Wno-unused-parameter -Wno-missing-field-initializers -g -I../include -I/home/mockbuild/scst/iscsi-scst/../scst/include -D_GNU_SOURCE message.c
cc -c -o ctldev.o -O2 -fno-inline -Wall -Wextra -Wstrict-prototypes -Wno-sign-compare -Wimplicit-function-declaration -Wno-unused-parameter -Wno-missing-field-initializers -g -I../include -I/home/mockbuild/scst/iscsi-scst/../scst/include -D_GNU_SOURCE ctldev.c
cc -c -o log.o -O2 -fno-inline -Wall -Wextra -Wstrict-prototypes -Wno-sign-compare -Wimplicit-function-declaration -Wno-unused-parameter -Wno-missing-field-initializers -g -I../include -I/home/mockbuild/scst/iscsi-scst/../scst/include -D_GNU_SOURCE log.c
cc -c -o chap.o -O2 -fno-inline -Wall -Wextra -Wstrict-prototypes -Wno-sign-compare -Wimplicit-function-declaration -Wno-unused-parameter -Wno-missing-field-initializers -g -I../include -I/home/mockbuild/scst/iscsi-scst/../scst/include -D_GNU_SOURCE chap.c
cc -c -o event.o -O2 -fno-inline -Wall -Wextra -Wstrict-prototypes -Wno-sign-compare -Wimplicit-function-declaration -Wno-unused-parameter -Wno-missing-field-initializers -g -I../include -I/home/mockbuild/scst/iscsi-scst/../scst/include -D_GNU_SOURCE event.c
cc -c -o param.o -O2 -fno-inline -Wall -Wextra -Wstrict-prototypes -Wno-sign-compare -Wimplicit-function-declaration -Wno-unused-parameter -Wno-missing-field-initializers -g -I../include -I/home/mockbuild/scst/iscsi-scst/../scst/include -D_GNU_SOURCE param.c
cc -c -o config.o -O2 -fno-inline -Wall -Wextra -Wstrict-prototypes -Wno-sign-compare -Wimplicit-function-declaration -Wno-unused-parameter -Wno-missing-field-initializers -g -I../include -I/home/mockbuild/scst/iscsi-scst/../scst/include -D_GNU_SOURCE config.c
cc -c -o isns.o -O2 -fno-inline -Wall -Wextra -Wstrict-prototypes -Wno-sign-compare -Wimplicit-function-declaration -Wno-unused-parameter -Wno-missing-field-initializers -g -I../include -I/home/mockbuild/scst/iscsi-scst/../scst/include -D_GNU_SOURCE isns.c
cc -c -o md5.o -O2 -fno-inline -Wall -Wextra -Wstrict-prototypes -Wno-sign-compare -Wimplicit-function-declaration -Wno-unused-parameter -Wno-missing-field-initializers -g -I../include -I/home/mockbuild/scst/iscsi-scst/../scst/include -D_GNU_SOURCE md5.c
cc -c -o sha1.o -O2 -fno-inline -Wall -Wextra -Wstrict-prototypes -Wno-sign-compare -Wimplicit-function-declaration -Wno-unused-parameter -Wno-missing-field-initializers -g -I../include -I/home/mockbuild/scst/iscsi-scst/../scst/include -D_GNU_SOURCE sha1.c
cc iscsid.o iscsi_scstd.o conn.o session.o target.o message.o ctldev.o log.o chap.o event.o param.o config.o isns.o md5.o sha1.o -o iscsi-scstd
cc -M -O2 -fno-inline -Wall -Wextra -Wstrict-prototypes -Wno-sign-compare -Wimplicit-function-declaration -Wno-unused-parameter -Wno-missing-field-initializers -g -I../include -I/home/mockbuild/scst/iscsi-scst/../scst/include -D_GNU_SOURCE iscsi_adm.c param.c >.depend_adm
cc -c -o iscsi_adm.o -O2 -fno-inline -Wall -Wextra -Wstrict-prototypes -Wno-sign-compare -Wimplicit-function-declaration -Wno-unused-parameter -Wno-missing-field-initializers -g -I../include -I/home/mockbuild/scst/iscsi-scst/../scst/include -D_GNU_SOURCE iscsi_adm.c
cc iscsi_adm.o param.o -o iscsi-scst-adm
make[2]: Leaving directory `/home/mockbuild/scst/iscsi-scst/usr'
cp /home/mockbuild/scst/iscsi-scst/../scst/src/Module.symvers kernel/
make -C /lib/modules/2.6.32-scst/build SCST_INC_DIR=/home/mockbuild/scst/iscsi-scst/../scst/include SUBDIRS=/home/mockbuild/scst/iscsi-scst/kernel modules
make[2]: Entering directory `/home/mockbuild/rpmbuild/BUILD/kernel-2.6.32-431.23.3.el6/linux-2.6.32-431.23.3.el6.x86_64'
CC [M] /home/mockbuild/scst/iscsi-scst/kernel/iscsi.o
CC [M] /home/mockbuild/scst/iscsi-scst/kernel/nthread.o
CC [M] /home/mockbuild/scst/iscsi-scst/kernel/config.o
CC [M] /home/mockbuild/scst/iscsi-scst/kernel/digest.o
CC [M] /home/mockbuild/scst/iscsi-scst/kernel/conn.o
CC [M] /home/mockbuild/scst/iscsi-scst/kernel/session.o
CC [M] /home/mockbuild/scst/iscsi-scst/kernel/target.o
CC [M] /home/mockbuild/scst/iscsi-scst/kernel/event.o
CC [M] /home/mockbuild/scst/iscsi-scst/kernel/param.o
LD [M] /home/mockbuild/scst/iscsi-scst/kernel/iscsi-scst.o
Building modules, stage 2.
MODPOST 1 modules
CC /home/mockbuild/scst/iscsi-scst/kernel/iscsi-scst.mod.o
LD [M] /home/mockbuild/scst/iscsi-scst/kernel/iscsi-scst.ko.unsigned
NO SIGN [M] /home/mockbuild/scst/iscsi-scst/kernel/iscsi-scst.ko
make[2]: Leaving directory `/home/mockbuild/rpmbuild/BUILD/kernel-2.6.32-431.23.3.el6/linux-2.6.32-431.23.3.el6.x86_64'
make[1]: Leaving directory `/home/mockbuild/scst/iscsi-scst'
cd iscsi-scst && make DESTDIR= install
make[1]: Entering directory `/home/mockbuild/scst/iscsi-scst'
make -C usr SCST_INC_DIR=/home/mockbuild/scst/iscsi-scst/../scst/include
make[2]: Entering directory `/home/mockbuild/scst/iscsi-scst/usr'
make[2]: Nothing to be done for `all'.
make[2]: Leaving directory `/home/mockbuild/scst/iscsi-scst/usr'
cp /home/mockbuild/scst/iscsi-scst/../scst/src/Module.symvers kernel/
make -C /lib/modules/2.6.32-scst/build SCST_INC_DIR=/home/mockbuild/scst/iscsi-scst/../scst/include SUBDIRS=/home/mockbuild/scst/iscsi-scst/kernel modules
make[2]: Entering directory `/home/mockbuild/rpmbuild/BUILD/kernel-2.6.32-431.23.3.el6/linux-2.6.32-431.23.3.el6.x86_64'
Building modules, stage 2.
MODPOST 1 modules
make[2]: Leaving directory `/home/mockbuild/rpmbuild/BUILD/kernel-2.6.32-431.23.3.el6/linux-2.6.32-431.23.3.el6.x86_64'
`usr/iscsi-scstd' -> `/usr/local/sbin/iscsi-scstd'
install: creating directory `/usr/local/man'
install: creating directory `/usr/local/man/man5'
`doc/manpages/iscsi-scstd.conf.5' -> `/usr/local/man/man5/iscsi-scstd.conf.5'
install: creating directory `/usr/local/man/man8'
`doc/manpages/iscsi-scstd.8' -> `/usr/local/man/man8/iscsi-scstd.8'
`usr/iscsi-scst-adm' -> `/usr/local/sbin/iscsi-scst-adm'
`doc/manpages/iscsi-scst-adm.8' -> `/usr/local/man/man8/iscsi-scst-adm.8'
`kernel/iscsi-scst.ko' -> `/lib/modules/2.6.32-scst/extra/iscsi-scst.ko'
/sbin/depmod -aq 2.6.32-scst
make[1]: Leaving directory `/home/mockbuild/scst/iscsi-scst'
[root@oracle651 scst]#
Finally, issue “make scstadm scstadm_install” as well. This finalizes the compiling part. Do this as root. Commands and expected output are shown below.
[root@oracle651 scst]# make scstadm scstadm_install | tee make_scst_install.log
cd scstadmin && make all
make[1]: Entering directory `/home/mockbuild/scst/scstadmin'
cd scstadmin && make all
make[2]: Entering directory `/home/mockbuild/scst/scstadmin/scstadmin.sysfs'
Checking if your kit is complete...
Looks good
Writing Makefile for SCST-SCST
make -C scst-0.9.10
make[3]: Entering directory `/home/mockbuild/scst/scstadmin/scstadmin.sysfs/scst-0.9.10'
cp lib/SCST/SCST.pm blib/lib/SCST/SCST.pm
Manifying blib/man3/SCST::SCST.3pm
make[3]: Leaving directory `/home/mockbuild/scst/scstadmin/scstadmin.sysfs/scst-0.9.10'
make[2]: Leaving directory `/home/mockbuild/scst/scstadmin/scstadmin.sysfs'
make[1]: Leaving directory `/home/mockbuild/scst/scstadmin'
cd scstadmin && make install
make[1]: Entering directory `/home/mockbuild/scst/scstadmin'
cd scstadmin && make install
make[2]: Entering directory `/home/mockbuild/scst/scstadmin/scstadmin.sysfs'
Writing Makefile for SCST-SCST
make -C scst-0.9.10
make[3]: Entering directory `/home/mockbuild/scst/scstadmin/scstadmin.sysfs/scst-0.9.10'
Skip blib/lib/SCST/SCST.pm (unchanged)
Manifying blib/man3/SCST::SCST.3pm
make[3]: Leaving directory `/home/mockbuild/scst/scstadmin/scstadmin.sysfs/scst-0.9.10'
make -C scst-0.9.10 install DESTDIR=
make[3]: Entering directory `/home/mockbuild/scst/scstadmin/scstadmin.sysfs/scst-0.9.10'
Manifying blib/man3/SCST::SCST.3pm
Installing /usr/local/share/perl5/SCST/SCST.pm
Installing /usr/local/share/man/man3/SCST::SCST.3pm
Appending installation info to /usr/lib64/perl5/perllocal.pod
make[3]: Leaving directory `/home/mockbuild/scst/scstadmin/scstadmin.sysfs/scst-0.9.10'
install -d /usr/local/sbin
install -m 755 scstadmin /usr/local/sbin
make[2]: Leaving directory `/home/mockbuild/scst/scstadmin/scstadmin.sysfs'
for m in scstadmin/man*/*.[1-9]; do \
if [ -e $m ]; then \
d=/usr/local/man/$(basename $(dirname $m)); \
if install -d $d; then \
install -m 644 $m $d; \
fi \
fi \
done
install -d /etc/init.d
tmp=/tmp/scst-initd-$$ && init.d/scst >$tmp && \
install -m 755 $tmp /etc/init.d/scst && \
rm -f $tmp
if [ ! -e /etc/default/scst ]; then \
mkdir -p /etc/default; \
install -m 755 default/scst /etc/default; \
fi
for s in iscsi-scst qla2x00t; do \
{ if [ -n "" ]; then chr="chroot "; else chr=""; fi; if type systemctl >/dev/null 2>&1; then $chr systemctl disable "$s.service"; elif type chkconfig >/dev/null 2>&1; then $chr chkconfig --del "$s"; elif type update-rc.d >/dev/null 2>&1; then $chr update-rc.d -f "$s" remove; elif type rc-update >/dev/null 2>&1; then $chr rc-update del "$s"; elif type /usr/lib/lsb/remove_initd >/dev/null 2>&1; then $chr /usr/lib/lsb/remove_initd "$s"; fi; rm -f /etc/init.d/$s; } >/dev/null 2>&1; \
done
echo
echo 'If you want SCST to start automatically at boot time, run' \
'the following command:'
If you want SCST to start automatically at boot time, run the following command:
if [ -n "" ]; then chr="chroot "; else chr=""; fi; if type systemctl >/dev/null 2>&1; then echo $chr systemctl enable "scst.service"; elif type chkconfig >/dev/null 2>&1; then echo $chr chkconfig --add "scst"; elif type update-rc.d >/dev/null 2>&1; then echo $chr update-rc.d "scst" defaults; elif type rc-update >/dev/null 2>&1; then echo $chr rc-update add "scst" default; elif type /usr/lib/lsb/install_initd >/dev/null 2>&1; then echo $chr /usr/lib/lsb/install_initd "scst"; fi
chkconfig --add scst
echo
make[1]: Leaving directory `/home/mockbuild/scst/scstadmin'
[root@oracle651 scst]#
Now set SCST to start on boot. Do this as root. Commands and expected output are shown below.
[root@oracle651 scst]# /usr/lib/lsb/install_initd scst
[root@oracle651 scst]#
Now test start SCST manually. There is no configuration file yet for SCST, so it will give a warning as shown below.
[root@oracle651 scst]# service scst start
Loading and configuring SCSTSCST configuration file /etc/scst.conf missing
[ OK ]
[root@oracle651 scst]#
Now lock down the kernel so that the custom kernel just built will not be updated. This requires updated the /etc/yum.conf file as shown below. The line that needs to be added is shown in bold below "exclude=kernel*". This prevents the kernel from being updated.
[root@oracle651 yum.repos.d]# cat /etc/yum.conf
[main]
cachedir=/var/cache/yum/$basearch/$releasever
keepcache=0
debuglevel=2
logfile=/var/log/yum.log
exactarch=1
obsoletes=1
gpgcheck=1
plugins=1
installonly_limit=3
exclude=kernel*
# This is the default, if you make this bigger yum won't see if the metadata
# is newer on the remote and so you'll "gain" the bandwidth of not having to
# download the new metadata and "pay" for it by yum not having correct
# information.
# It is esp. important, to have correct metadata, for distributions like
# Fedora which don't keep old packages around. If you don't like this checking
# interupting your command line usage, it's much better to have something
# manually check the metadata once an hour (yum-updatesd will do this).
# metadata_expire=90m
# PUT YOUR REPOS HERE OR IN separate files named file.repo
# in /etc/yum.repos.d
[root@oracle651 yum.repos.d]#
All that remains now is to reboot and test that it starts. Also, a configuration file must be created, which will be the next step. There was not a lot of documentation on the web about how to create a iSCSI SAN with SCST but I found a couple of very helpful posts here and here.
Configuration of a simple 4K flash-aware SCST iSCSI SAN as shown below on the oracle651 KVM guest.
[root@oracle651 ~]# modprobe scst
[root@oracle651 ~]# modprobe scst_vdisk
[root@oracle651 ~]# modprobe iscsi-scst
[root@oracle651 ~]# scstadmin -add_target iqn.2014-08.oracle.asm.storage:luns -driver iscsi
[root@oracle651 ~]# scstadmin -open_dev disk0 -handler vdisk_blockio -attributes filename=/dev/vdc,blocksize=4096
[root@oracle651 ~]# scstadmin -open_dev disk1 -handler vdisk_blockio -attributes filename=/dev/vdb
[root@oracle651 ~]# scstadmin -add_lun 0 -driver iscsi -target iqn.2014-08.oracle.asm.storage:luns -device disk0
[root@oracle651 ~]# scstadmin -add_lun 1 -driver iscsi -target iqn.2014-08.oracle.asm.storage:luns -device disk1
[root@oracle651 ~]# scstadmin -enable_target iqn.2014-08.oracle.asm.storage:luns -driver iscs
[root@oracle651 ~]# scstadmin -set_drv_attr iscsi -attributes enabled=1
[root@oracle651 ~]# scstadmin -write_config /etc/scst.conf
The /etc/scst.conf file generated by the above command is shown below. Notice the "blocksize 4096" parameter which can be used.
[root@oracle651 ~]# cat /etc/scst.conf
# Automatically generated by SCST Configurator v2.2.1.
HANDLER vdisk_blockio {
DEVICE disk0 {
blocksize 4096
filename /dev/vdc
}
DEVICE disk1 {
filename /dev/vdb
}
}
TARGET_DRIVER iscsi {
enabled 1
TARGET iqn.2014-08.oracle.asm.storage:luns {
cpu_mask 3
enabled 1
rel_tgt_id 1
LUN 0 disk0
LUN 1 disk1
}
}
[root@oracle651 ~]# cat /etc/sysconfig/selinux
In order to get the initiator over on the iSCSI client on oracle631 KVM guest to see the portal, stop iptables and disable it on reboot as shown below and also disable selinux as shown below and turn off iptables on the iSCSI SAN host.
[root@oracle651 ~]# cat /etc/sysconfig/selinux
# This file controls the state of SELinux on the system.
# SELINUX= can take one of these three values:
# enforcing - SELinux security policy is enforced.
# permissive - SELinux prints warnings instead of enforcing.
# disabled - No SELinux policy is loaded.
SELINUX=disabled
# SELINUXTYPE= can take one of these two values:
# targeted - Targeted processes are protected,
# mls - Multi Level Security protection.
SELINUXTYPE=targeted
[root@oracle651 ~]# service iptables stop
[root@oracle651 ~]# chkconfig iptables off
[root@oracle651 ~]# reboot
Finally, use iscsiadm on the oracle631 KVM guest to discover and configure the access to the SCST SAN on oracle651 KVM guest. The iscsi-initiator utils are installed as shown below, and then the connection to the SAN is configured as shown.
[root@oracle631 ~]# yum install iscsi-initiator-utils
[root@oracle631 ~]# iscsiadm --mode discovery --type sendtargets --portal 10.207.39.74
10.207.39.74:3260,1 iqn.2014-08.oracle.asm.storage:luns
[root@oracle631 ~]# iscsiadm -m node -T iqn.2014-08.oracle.asm.storage:luns -p 10.207.39.74 --op update -n node.startup -v automatic
[root@oracle631 ~]# iscsiadm --mode node --targetname iqn.2014-08.oracle.asm.storage:luns --portal 10.207.39.74 --login
Logging in to [iface: default, target: iqn.2014-08.oracle.asm.storage:luns, portal: 10.207.39.74,3260] (multiple)
Login to [iface: default, target: iqn.2014-08.oracle.asm.storage:luns, portal: 10.207.39.74,3260] successful.
[root@oracle631 ~]#
Here are the LUNs over on the client side and show of their characteristics on oracle631 host as shown below. One very interesting and intriguing feature is that SCST sees the 512 emulated LUN on oracle631 KVM guest (the client) for what it really is i.e (512 logical / 4096 physical) even though the virtual disk on the oracle651 KVM guest (the SAN) sees that LUN as (512 logical / 512 physical), as shown below.
[root@oracle631 ~]# ls -lrt /dev/sd*
brw-rw---- 1 root disk 8, 0 Aug 11 20:11 /dev/sda
brw-rw---- 1 root disk 8, 16 Aug 11 20:11 /dev/sdb
[root@oracle631 ~]# scsi_id -g -u -d /dev/sda
26162356632353435
[root@oracle631 ~]# scsi_id -g -u -d /dev/sdb
26238666462643332
[root@oracle631 ~]# fdisk -l /dev/sda
Note: sector size is 4096 (not 512)
Disk /dev/sda: 10.7 GB, 10737418240 bytes
64 heads, 32 sectors/track, 1280 cylinders
Units = cylinders of 2048 * 4096 = 8388608 bytes
Sector size (logical/physical): 4096 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 524288 bytes
Disk identifier: 0x00000000
[root@oracle631 ~]# fdisk -l /dev/sdb
Disk /dev/sdb: 10.7 GB, 10737418240 bytes
64 heads, 32 sectors/track, 10240 cylinders
Units = cylinders of 2048 * 512 = 1048576 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes <-- Notice
I/O size (minimum/optimal): 4096 bytes / 524288 bytes
Disk identifier: 0x00000000
[root@oracle631 ~]#
Here are the properties of the same LUNs over on the SAN oracle651 as shown below.
[root@oracle651 ~]# cat /etc/scst.conf
# Automatically generated by SCST Configurator v2.2.1.
HANDLER vdisk_blockio {
DEVICE disk0 {
blocksize 4096
filename /dev/vdc
}
DEVICE disk1 {
filename /dev/vdb
}
}
TARGET_DRIVER iscsi {
enabled 1
TARGET iqn.2014-08.oracle.asm.storage:luns {
cpu_mask 3
enabled 1
rel_tgt_id 1
LUN 0 disk0
LUN 1 disk1
}
}
[root@oracle651 ~]# fdisk -l /dev/vdc
Note: sector size is 4096 (not 512)
Disk /dev/vdc: 10.7 GB, 10737418240 bytes
16 heads, 56 sectors/track, 2925 cylinders
Units = cylinders of 896 * 4096 = 3670016 bytes
Sector size (logical/physical): 4096 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disk identifier: 0x00000000
[root@oracle651 ~]# fdisk -l /dev/vdb
Disk /dev/vdb: 10.7 GB, 10737418240 bytes
16 heads, 63 sectors/track, 20805 cylinders
Units = cylinders of 1008 * 512 = 516096 bytes
Sector size (logical/physical): 512 bytes / 512 bytes <-- Notice
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000
[root@oracle651 ~]#
Recall that the blocksize defaults in KVM to (logical/physical) 512 bytes / 512 bytes unless it is set otherwise in the XML file of the KVM guest, as shown below i the XML file that defines oracle651 the KVM guest which is the SAN provider. Looking at the end of the XML file at bottom it can be seen that only disk2 has been set to specified blocksizes. Therefore, disk1 is presented to the KVM guest oracle651 as a 512 bytes / 512 bytes LUN by default. Thus it is surprising that SCST sees it for what it is 'physically' which is a LUN sitting on a consumer-grade SSD (Crucial M550) and flash always uses 4096 byte physical sectors, and 512 byte logical emulation. The LUN as seen by oracle631 the iscsi client sees it as what it 'really' is, 512 bytes / 4096 bytes (logical/physical). More investigation is warranted to understand this better, but it would seem that SCST is ignoring the KVM setting and reading the storage itself, and that the blocksize setting in the SCST configuration file refers to the logical block size, and thus set the logical block size of the other LUN to 4096 bytes (the LUN that was presented by KVM on oracle631 as (logical/physical) 4096 bytes / 4096 bytes.
gstanden@vmem1:~$ virsh dumpxml oracle651
<domain type='kvm' id='6' xmlns:qemu='http://libvirt.org/schemas/domain/qemu/1.0'>
<name>oracle651</name>
<uuid>9a017de6-d362-4269-90fe-09331460aca0</uuid>
<memory unit='KiB'>1048576</memory>
<currentMemory unit='KiB'>1048576</currentMemory>
<vcpu placement='static'>2</vcpu>
<resource>
<partition>/machine</partition>
</resource>
<os>
<type arch='x86_64' machine='pc-i440fx-trusty'>hvm</type>
<boot dev='hd'/>
<boot dev='cdrom'/>
<bootmenu enable='yes'/>
</os>
<features>
<acpi/>
<apic/>
<pae/>
</features>
<clock offset='utc'/>
<on_poweroff>destroy</on_poweroff>
<on_reboot>restart</on_reboot>
<on_crash>restart</on_crash>
<devices>
<emulator>/usr/bin/kvm-spice</emulator>
<disk type='file' device='disk'>
<driver name='qemu' type='raw' cache='none' io='native'/>
<source file='/var/lib/libvirt/images/oracle651-0.img'/>
<target dev='vda' bus='virtio'/>
<alias name='virtio-disk0'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/>
</disk>
<disk type='file' device='disk'>
<driver name='qemu' type='raw' cache='none' io='native'/>
<source file='/var/lib/libvirt/images/oracle651-1.img'/>
<target dev='vdb' bus='virtio'/>
<alias name='virtio-disk1'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/>
</disk>
<disk type='file' device='disk'>
<driver name='qemu' type='raw' cache='none' io='native'/>
<source file='/var/lib/libvirt/images/oracle651-2.img'/>
<target dev='vdc' bus='virtio'/>
<alias name='virtio-disk2'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x08' function='0x0'/>
</disk>
<disk type='block' device='cdrom'>
<driver name='qemu' type='raw'/>
<target dev='hdc' bus='ide' tray='open'/>
<readonly/>
<alias name='ide0-1-0'/>
<address type='drive' controller='0' bus='1' target='0' unit='0'/>
</disk>
<controller type='usb' index='0'>
<alias name='usb0'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x2'/>
</controller>
<controller type='pci' index='0' model='pci-root'>
<alias name='pci.0'/>
</controller>
<controller type='ide' index='0'>
<alias name='ide0'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x1'/>
</controller>
<interface type='bridge'>
<mac address='52:54:00:66:26:a0'/>
<source bridge='sw1'/>
<virtualport type='openvswitch'>
<parameters interfaceid='0efa5f2a-cbf2-46df-9ff2-66bddb09218d'/>
</virtualport>
<target dev='s4'/>
<model type='virtio'/>
<alias name='net0'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
</interface>
<serial type='pty'>
<source path='/dev/pts/12'/>
<target port='0'/>
<alias name='serial0'/>
</serial>
<console type='pty' tty='/dev/pts/12'>
<source path='/dev/pts/12'/>
<target type='serial' port='0'/>
<alias name='serial0'/>
</console>
<input type='mouse' bus='ps2'/>
<input type='keyboard' bus='ps2'/>
<graphics type='vnc' port='5900' autoport='yes' listen='127.0.0.1'>
<listen type='address' address='127.0.0.1'/>
</graphics>
<sound model='ich6'>
<alias name='sound0'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
</sound>
<video>
<model type='cirrus' vram='9216' heads='1'/>
<alias name='video0'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>
</video>
<memballoon model='virtio'>
<alias name='balloon0'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/>
</memballoon>
</devices>
<seclabel type='dynamic' model='apparmor' relabel='yes'>
<label>libvirt-9a017de6-d362-4269-90fe-09331460aca0</label>
<imagelabel>libvirt-9a017de6-d362-4269-90fe-09331460aca0</imagelabel>
</seclabel>
<qemu:commandline>
<qemu:arg value='-set'/>
<qemu:arg value='device.virtio-disk0.scsi=off'/>
<qemu:arg value='-set'/>
<qemu:arg value='device.virtio-disk0.config-wce=off'/>
<qemu:arg value='-set'/>
<qemu:arg value='device.virtio-disk0.x-data-plane=on'/>
<qemu:arg value='-set'/>
<qemu:arg value='device.virtio-disk1.scsi=off'/>
<qemu:arg value='-set'/>
<qemu:arg value='device.virtio-disk1.config-wce=off'/>
<qemu:arg value='-set'/>
<qemu:arg value='device.virtio-disk1.x-data-plane=on'/>
<qemu:arg value='-set'/>
<qemu:arg value='device.virtio-disk2.scsi=off'/>
<qemu:arg value='-set'/>
<qemu:arg value='device.virtio-disk2.config-wce=off'/>
<qemu:arg value='-set'/>
<qemu:arg value='device.virtio-disk2.x-data-plane=on'/>
<qemu:arg value='-set'/>
<qemu:arg value='device.virtio-disk2.logical_block_size=4096'/>
<qemu:arg value='-set'/>
<qemu:arg value='device.virtio-disk2.physical_block_size=4096'/>
</qemu:commandline>
</domain>
gstanden@vmem1:~$
Some useful additonal info about the LUNs over on the oracle651 client.
[root@oracle631 disk]# ls -lrt by-path | grep asm
lrwxrwxrwx 1 root root 9 Aug 11 20:11 ip-10.207.39.74:3260-iscsi-iqn.2014-08.oracle.asm.storage:luns-lun-0 -> ../../sda
lrwxrwxrwx 1 root root 9 Aug 11 20:11 ip-10.207.39.74:3260-iscsi-iqn.2014-08.oracle.asm.storage:luns-lun-1 -> ../../sdb
[root@oracle631 disk]# ls -lrt by-id | grep scsi
lrwxrwxrwx 1 root root 9 Aug 11 20:11 scsi-26162356632353435 -> ../../sda
lrwxrwxrwx 1 root root 9 Aug 11 20:11 scsi-26238666462643332 -> ../../sdb
[root@oracle631 disk]#
More useful information, including the vendor name (for use in the multipath.conf file) as shown below.
[root@oracle631 disk]# cat /proc/scsi/scsi
Attached devices:
Host: scsi1 Channel: 00 Id: 00 Lun: 00
Vendor: QEMU Model: QEMU DVD-ROM Rev: 2.0.
Type: CD-ROM ANSI SCSI revision: 05
Host: scsi2 Channel: 00 Id: 00 Lun: 00
Vendor: SCST_BIO Model: disk0 Rev: 221
Type: Direct-Access ANSI SCSI revision: 05
Host: scsi2 Channel: 00 Id: 00 Lun: 01
Vendor: SCST_BIO Model: disk1 Rev: 221
Type: Direct-Access ANSI SCSI revision: 05
[root@oracle631 disk]#
And yet more information about the SCST iSCSI LUN /dev/sda shown below using udevadm command.
[root@oracle631 disk]# udevadm info -a -p $(udevadm info -q path -n /dev/sda)
Udevadm info starts with the device specified by the devpath and then
walks up the chain of parent devices. It prints for every device
found, all possible attributes in the udev rules key format.
A rule to match, can be composed by the attributes of the device
and the attributes from one single parent device.
looking at device '/devices/platform/host2/session1/target2:0:0/2:0:0:0/block/sda':
KERNEL=="sda"
SUBSYSTEM=="block"
DRIVER==""
ATTR{ro}=="0"
ATTR{size}=="20971520"
ATTR{stat}==" 455 0 3648 329 0 0 0 0 0 328 328"
ATTR{range}=="16"
ATTR{discard_alignment}=="0"
ATTR{events}==""
ATTR{ext_range}=="256"
ATTR{events_poll_msecs}=="-1"
ATTR{alignment_offset}=="0"
ATTR{inflight}==" 0 0"
ATTR{removable}=="0"
ATTR{capability}=="50"
ATTR{events_async}==""
looking at parent device '/devices/platform/host2/session1/target2:0:0/2:0:0:0':
KERNELS=="2:0:0:0"
SUBSYSTEMS=="scsi"
DRIVERS=="sd"
ATTRS{rev}==" 221"
ATTRS{type}=="0"
ATTRS{scsi_level}=="6"
ATTRS{model}=="disk0 "
ATTRS{state}=="running"
ATTRS{queue_type}=="none"
ATTRS{modalias}=="scsi:t-0x00"
ATTRS{iodone_cnt}=="0x212"
ATTRS{iorequest_cnt}=="0x212"
ATTRS{queue_ramp_up_period}=="120000"
ATTRS{timeout}=="30"
ATTRS{evt_media_change}=="0"
ATTRS{ioerr_cnt}=="0x2"
ATTRS{queue_depth}=="32"
ATTRS{vendor}=="SCST_BIO"
ATTRS{device_blocked}=="0"
ATTRS{dh_state}=="detached"
ATTRS{iocounterbits}=="32"
ATTRS{eh_timeout}=="10"
looking at parent device '/devices/platform/host2/session1/target2:0:0':
KERNELS=="target2:0:0"
SUBSYSTEMS=="scsi"
DRIVERS==""
looking at parent device '/devices/platform/host2/session1':
KERNELS=="session1"
SUBSYSTEMS==""
DRIVERS==""
looking at parent device '/devices/platform/host2':
KERNELS=="host2"
SUBSYSTEMS=="scsi"
DRIVERS==""
looking at parent device '/devices/platform':
KERNELS=="platform"
SUBSYSTEMS==""
DRIVERS==""
And now for /dev/sdb shown below using udevadm command.
[root@oracle631 disk]# udevadm info -a -p $(udevadm info -q path -n /dev/sdb)
Udevadm info starts with the device specified by the devpath and then
walks up the chain of parent devices. It prints for every device
found, all possible attributes in the udev rules key format.
A rule to match, can be composed by the attributes of the device
and the attributes from one single parent device.
looking at device '/devices/platform/host2/session1/target2:0:0/2:0:0:1/block/sdb':
KERNEL=="sdb"
SUBSYSTEM=="block"
DRIVER==""
ATTR{ro}=="0"
ATTR{size}=="20971520"
ATTR{stat}==" 455 0 3664 337 0 0 0 0 0 337 337"
ATTR{range}=="16"
ATTR{discard_alignment}=="0"
ATTR{events}==""
ATTR{ext_range}=="256"
ATTR{events_poll_msecs}=="-1"
ATTR{alignment_offset}=="0"
ATTR{inflight}==" 0 0"
ATTR{removable}=="0"
ATTR{capability}=="50"
ATTR{events_async}==""
looking at parent device '/devices/platform/host2/session1/target2:0:0/2:0:0:1':
KERNELS=="2:0:0:1"
SUBSYSTEMS=="scsi"
DRIVERS=="sd"
ATTRS{rev}==" 221"
ATTRS{type}=="0"
ATTRS{scsi_level}=="6"
ATTRS{model}=="disk1 "
ATTRS{state}=="running"
ATTRS{queue_type}=="none"
ATTRS{modalias}=="scsi:t-0x00"
ATTRS{iodone_cnt}=="0x211"
ATTRS{iorequest_cnt}=="0x211"
ATTRS{queue_ramp_up_period}=="120000"
ATTRS{timeout}=="30"
ATTRS{evt_media_change}=="0"
ATTRS{ioerr_cnt}=="0x2"
ATTRS{queue_depth}=="32"
ATTRS{vendor}=="SCST_BIO"
ATTRS{device_blocked}=="0"
ATTRS{dh_state}=="detached"
ATTRS{iocounterbits}=="32"
ATTRS{eh_timeout}=="10"
looking at parent device '/devices/platform/host2/session1/target2:0:0':
KERNELS=="target2:0:0"
SUBSYSTEMS=="scsi"
DRIVERS==""
looking at parent device '/devices/platform/host2/session1':
KERNELS=="session1"
SUBSYSTEMS==""
DRIVERS==""
looking at parent device '/devices/platform/host2':
KERNELS=="host2"
SUBSYSTEMS=="scsi"
DRIVERS==""
looking at parent device '/devices/platform':
KERNELS=="platform"
SUBSYSTEMS==""
DRIVERS==""
Additional information after the configuration of the SCST LUNs on oracle651 using device-mapper-multipath and /etc/udev/rules.d/12-dm-permissions.rules as shown below. Friendly names set from the /etc/multipath.conf file correctly.
[root@oracle631 ~]# ls -lrt /dev/mapper
total 0
crw-rw---- 1 root root 10, 236 Aug 11 21:30 control
lrwxrwxrwx 1 root root 7 Aug 11 21:30 vg_oracle631-lv_swap -> ../dm-1
lrwxrwxrwx 1 root root 7 Aug 11 21:30 vg_oracle631-lv_root -> ../dm-0
lrwxrwxrwx 1 root root 7 Aug 11 21:30 vg_u00-lv_oracle -> ../dm-2
lrwxrwxrwx 1 root root 7 Aug 11 21:30 ASM_4096B_1 -> ../dm-3
lrwxrwxrwx 1 root root 7 Aug 11 21:30 ASM_0512B_1 -> ../dm-4
Friendly names accurately indicate logical block sizes as shown below.
[root@oracle631 ~]# fdisk -l /dev/dm-3 <--ASM_4096B_1
Note: sector size is 4096 (not 512)
Disk /dev/dm-3: 10.7 GB, 10737418240 bytes
255 heads, 63 sectors/track, 163 cylinders
Units = cylinders of 16065 * 4096 = 65802240 bytes
Sector size (logical/physical): 4096 bytes / 4096 bytes <-- 4096 logical sector size
I/O size (minimum/optimal): 4096 bytes / 524288 bytes
Disk identifier: 0x00000000
[root@oracle631 ~]# fdisk -l /dev/dm-4
Disk /dev/dm-4: 10.7 GB, 10737418240 bytes <--ASM_0512B_1
255 heads, 63 sectors/track, 1305 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes <-- 512 logical sector size
I/O size (minimum/optimal): 4096 bytes / 524288 bytes
Disk identifier: 0x00000000
[root@oracle631 ~]#
Permissions set correctly from the /etc/udev/rules.d/12-dm-permissions.rules file.
[root@oracle631 ~]# ls -lrt /dev/dm-3
brw-rw---- 1 grid asmadmin 252, 3 Aug 11 21:30 /dev/dm-3
[root@oracle631 ~]# ls -lrt /dev/dm-4
brw-rw---- 1 grid asmadmin 252, 4 Aug 11 21:30 /dev/dm-4
The /etc/multipath.conf file for the SCST LUNs on oracle631 as shown below.
[root@oracle631 ~]# cat /etc/multipath.conf
# Configuration file for device mapper multipath.
# For a complete list of the default configuration values:
# /usr/share/doc/device-mapper-multipath-0.4.9/multipath.conf.defaults
# For a list of configuration options with descriptions:
# /usr/share/doc/device-mapper-multipath-0.4.9/multipath.conf.annotated
defaults {
user_friendly_names yes
}
blacklist {
devnode "^(ram|raw|loop|fd|md|dm-|sr|scd|st)[0-9]*"
devnode "^hd[a-z]"
}
devices {
device {
vendor "SCST_BIO"
product "Direct-Access"
path_grouping_policy multibus
path_grouping_policy group_by_prio
path_selector "round-robin 0"
failback immediate
rr_min_io 1
rr_weight priorities
}
}
multipaths {
multipath {
wwid 26162356632353435
alias ASM_4096B_1
}
multipath {
wwid 26238666462643332
alias ASM_0512B_1
}
}
[root@oracle631 ~]#
Listing of the /etc/udev/rules.d/12-dm-permissions.rules file on oracle631 the scsi client as shown below with section that handles the above LUNs bolded (near bottom of listing).
[root@oracle631 ~]# cat /etc/udev/rules.d/12-dm-permissions.rules
# Copyright (C) 2009 Red Hat, Inc. All rights reserved.
#
# This file is part of LVM2.
# Udev rules for device-mapper devices.
#
# These rules set permissions for DM devices.
#
# This file is considered to be a template where users can put their
# own entries and then put a copy of it manually to a usual place with
# user-edited udev rules (usually /etc/udev/rules.d).
#
# There are some environment variables set that can be used:
# DM_UDEV_RULES_VSN - DM udev rules version
# DM_NAME - actual DM device's name
# DM_UUID - UUID set for DM device (blank if not specified)
# DM_SUSPENDED - suspended state of DM device (0 or 1)
# DM_LV_NAME - logical volume name (not set if LVM device not present)
# DM_VG_NAME - volume group name (not set if LVM device not present)
# DM_LV_LAYER - logical volume layer (not set if LVM device not present)
# "add" event is processed on coldplug only!
ACTION!="add|change", GOTO="dm_end"
ENV{DM_UDEV_RULES_VSN}!="?*", GOTO="dm_end"
# A few demonstrational examples...
# PLAIN DM DEVICES
#
# Set permissions for a DM device named 'my_device' exactly
# ENV{DM_NAME}=="my_device", OWNER:="root", GROUP:="root", MODE:="660"
# Set permissions for all DM devices having 'MY_UUID-' UUID prefix
# ENV{DM_UUID}=="MY_UUID-?*", OWNER:="root", GROUP:="root", MODE:="660"
# LVM DEVICES
#
# Set permissions for all LVM devices
# ENV{DM_UUID}=="LVM-?*", OWNER:="root", GROUP:="root", MODE:="660"
# Set permissions for all devices that belong to one LVM VG
# ENV{DM_VG_NAME}=="VolGroup00", OWNER:="root", GROUP:="root", MODE:="660"
# Set permissions for an LVM device with VG named VolGroup00 and LV named LogVol00 exactly
# ENV{DM_VG_NAME}=="VolGroup00", ENV{DM_LV_NAME}=="LogVol00", OWNER:="root", GROUP:="root", MODE:="660"
# Set permissions for all LVM devices that does not belong to a VG named VolGroup00
# ENV{DM_VG_NAME}!="VolGroup00", OWNER:="root", GROUP:="root", MODE:="660"
# ENCRYPTED DEVICES (using cryptsetup >= 1.1)
#
# Set permissions for all encrypted devices created by cryptsetup (plain devices)
# ENV{DM_UUID}=="CRYPT-PLAIN-?*", OWNER:="root", GROUP:="root", MODE:="660"
# Set permissions for all encrypted devices created by cryptsetup (LUKS extension)
# ENV{DM_UUID}=="CRYPT-LUKS1-?*", OWNER:="root", GROUP:="root", MODE:="660"
# Set permissions for an encrypted device created by cryptsetup and having an exact luks UUID
# ENV{DM_UUID}=="CRYPT-LUKS1-22fce5c8313c43c68d84b50a3b0fee78-?*", OWNER:="root", GROUP:="root", MODE:="660"
# MULTIPATH DEVICES
#
# Set permissions for all multipath devices
# ENV{DM_UUID}=="mpath-?*", OWNER:="root", GROUP:="root", MODE:="660"
ENV{DM_UUID}=="mpath-26162356632353435", OWNER:="grid", GROUP:="asmadmin", MODE:="660" # <-- LUN modes are set here
ENV{DM_UUID}=="mpath-26238666462643332", OWNER:="grid", GROUP:="asmadmin", MODE:="660" # <-- LUN modes are set here
# Set permissions for first two partitions created on a multipath device (and detected by kpartx)
# ENV{DM_UUID}=="part[1-2]-mpath-?*", OWNER:="root", GROUP:="root", MODE:="660"
# ...you can use any combination of the comparisons with the environment variables
# listed at the beginning of this file (udev provides simple pattern matching by
# using *, ? and [] that you can use, see 'man udev' for more information).
# Set default permissions for all DM devices if not set before.
# OWNER:="root", GROUP:="root", MODE:="660"
LABEL="dm_end"
[root@oracle631 ~]#
If the console terminal is running on the KVM guest SCST SAN host oracle651, then when a SAN client oracle631 is shutdown, the following message is buffered to console and written to dmesg as shown below.
[root@oracle651 ~]# iscsi-scst: Logout received from initiator iqn.1988-12.com.oracle:49c39d4d389d
iscsi-scst: Closing connection at initiator's iqn.1988-12.com.oracle:49c39d4d389d request
scst: TM fn 6 (ffff88003dfd50b8)
scst: TM fn 6 (ffff88003dfd50b8) finished, status 0
scst: Processing thread disk00_0 (PID 1783) finished
scst: Processing thread disk10_0 (PID 1789) finished
[root@oracle651 ~]# exit
logout
Oracle Linux Server release 6.5
Kernel 2.6.32-scst on an x86_64
oracle651.vmem.org login:
Here is an example of the commands for adding a file-backed LUN rather than a block device as shown below.
[root@oracle651 ~] dd if=/dev/zero of=/root/scst-scripts/asm_disk1.img bs=512 count=0 seek=1M
[root@oracle651 ~] scstadmin -open_dev disk2 -handler vdisk_fileio -attributes filename=/root/scst-scripts/asm_disk1.img,blocksize=4096
[root@oracle651 ~] scstadmin -add_lun 2 -driver iscsi -target iqn.2014-08.oracle.asm.storage:luns -device disk2
[root@oracle651 ~] scstadmin -write_config /etc/scst.conf
The last command saves it to the /etc/scst.conf configuration file as shown below. The "vdisk_fileio" handler has been added, as well as the LUN.
[root@oracle651 scst-scripts]# cat /etc/scst.conf
# Automatically generated by SCST Configurator v2.2.1.
HANDLER vdisk_blockio {
DEVICE disk0 {
blocksize 4096
filename /dev/vdc
}
DEVICE disk1 {
filename /dev/vdb
}
}
HANDLER vdisk_fileio {
DEVICE disk2 {
blocksize 4096
filename /root/scst-scripts/asm_disk1.img
}
}
TARGET_DRIVER iscsi {
enabled 1
TARGET iqn.2014-08.oracle.asm.storage:luns {
cpu_mask 3
enabled 1
rel_tgt_id 1
LUN 0 disk0
LUN 1 disk1
LUN 2 disk2
}
}
[root@oracle651 scst-scripts]#
Here is an example of removing the LUN that was just added. Notice that the HANDLER information for the deleted LUN remains, but the TARGET DRIVER LUN list is updated and the removed LUN has been removed from the list in /etc/scst.conf as shown below.
[root@oracle651 scst-scripts]# scstadmin -rem_lun 2 -driver iscsi -target iqn.2014-08.oracle.asm.storage:luns -device disk2
Collecting current configuration: done.
Performing this action may result in lost or corrupt data, are you sure you wish to continue (y/[n]) ? y
-> Making requested changes.
-> Removing LUN 2 from driver/target 'iscsi/iqn.2014-08.oracle.asm.storage:luns': done.
Writing current configuration to file '/etc/scst.conf'
All done.
[root@oracle651 scst-scripts]# scstadmin -write_config /etc/scst.conf
Collecting current configuration: done.
Writing current configuration to file '/etc/scst.conf'..
All done.
[root@oracle651 scst-scripts]# cat /etc/scst.conf
# Automatically generated by SCST Configurator v2.2.1.
HANDLER vdisk_blockio {
DEVICE disk0 {
blocksize 4096
filename /dev/vdc
}
DEVICE disk1 {
filename /dev/vdb
}
}
HANDLER vdisk_fileio { <-- disk2 handler is still defined after LUN 2 was dropped from target.
DEVICE disk2 {
blocksize 4096
filename /root/scst-scripts/asm_disk1.img
}
}
TARGET_DRIVER iscsi {
enabled 1
TARGET iqn.2014-08.oracle.asm.storage:luns {
cpu_mask 3
enabled 1
rel_tgt_id 1
LUN 0 disk0
LUN 1 disk1
} <-- LUN 2 disk2 is removed here.
}
[root@oracle651 scst-scripts]# history
Thus, the LUN can be re-added back with just the one command below if it is needed back in the target as shown below.
[root@oracle651 ~]# scstadmin -add_lun 2 -driver iscsi -target iqn.2014-08.oracle.asm.storage:luns -device disk2
Collecting current configuration: done.
-> Making requested changes.
-> Adding device 'disk2' at LUN 2 to driver/target 'iscsi/iqn.2014-08.oracle.asm.storage:luns': done.
-> Driver/target is not a fibre channel target, ignoring.
-> Done.
All done.
[root@oracle651 ~]# scstadmin -write_config /etc/scst.conf
Collecting current configuration: done.
Writing current configuration to file '/etc/scst.conf'..
All done.
[root@oracle651 ~]#
View /etc/scst.conf again and see that the LUN has been added back to the TARGET as shown below.
[root@oracle651 ~]# cat /etc/scst.conf
# Automatically generated by SCST Configurator v2.2.1.
HANDLER vdisk_blockio {
DEVICE disk0 {
blocksize 4096
filename /dev/vdc
}
DEVICE disk1 {
filename /dev/vdb
}
}
HANDLER vdisk_fileio {
DEVICE disk2 {
blocksize 4096
filename /root/scst-scripts/asm_disk1.img
}
}
TARGET_DRIVER iscsi {
enabled 1
TARGET iqn.2014-08.oracle.asm.storage:luns {
cpu_mask 3
enabled 1
rel_tgt_id 1
LUN 0 disk0
LUN 1 disk1
LUN 2 disk2
}
}
[root@oracle651 ~]#
Examples of adding file-backed SCST LUNs to the OS of the SCST OEL host are shown below. The example below shows adding the 8 LUNs per the Violin Memory Oracle Best Practice Document dated April 2014 to the SCST SAN configuration.
[root@oracle651 ~]# scstadmin -add_lun 11 -driver iscsi -target iqn.2014-08.oracle.asm.storage:luns -device asm_reco1
Collecting current configuration: done.
-> Making requested changes.
-> Adding device 'asm_reco1' at LUN 11 to driver/target 'iscsi/iqn.2014-08.oracle.asm.storage:luns': done.
-> Driver/target is not a fibre channel target, ignoring.
-> Done.
All done.
[root@oracle651 ~]# scstadmin -add_lun 12 -driver iscsi -target iqn.2014-08.oracle.asm.storage:luns -device asm_reco2
Collecting current configuration: done.
-> Making requested changes.
-> Adding device 'asm_reco2' at LUN 12 to driver/target 'iscsi/iqn.2014-08.oracle.asm.storage:luns': done.
-> Driver/target is not a fibre channel target, ignoring.
-> Done.
All done.
[root@oracle651 ~]# scstadmin -add_lun 13 -driver iscsi -target iqn.2014-08.oracle.asm.storage:luns -device asm_reco3
Collecting current configuration: done.
-> Making requested changes.
-> Adding device 'asm_reco3' at LUN 13 to driver/target 'iscsi/iqn.2014-08.oracle.asm.storage:luns': done.
-> Driver/target is not a fibre channel target, ignoring.
-> Done.
All done.
[root@oracle651 ~]# scstadmin -add_lun 14 -driver iscsi -target iqn.2014-08.oracle.asm.storage:luns -device asm_reco4
Collecting current configuration: done.
-> Making requested changes.
-> Adding device 'asm_reco4' at LUN 14 to driver/target 'iscsi/iqn.2014-08.oracle.asm.storage:luns': done.
-> Driver/target is not a fibre channel target, ignoring.
-> Done.
All done.
[root@oracle651 ~]# scstadmin -add_lun 15 -driver iscsi -target iqn.2014-08.oracle.asm.storage:luns -device asm_reco5
Collecting current configuration: done.
-> Making requested changes.
-> Adding device 'asm_reco5' at LUN 15 to driver/target 'iscsi/iqn.2014-08.oracle.asm.storage:luns': done.
-> Driver/target is not a fibre channel target, ignoring.
-> Done.
All done.
[root@oracle651 ~]# scstadmin -add_lun 16 -driver iscsi -target iqn.2014-08.oracle.asm.storage:luns -device asm_reco6
Collecting current configuration: done.
-> Making requested changes.
-> Adding device 'asm_reco6' at LUN 16 to driver/target 'iscsi/iqn.2014-08.oracle.asm.storage:luns': done.
-> Driver/target is not a fibre channel target, ignoring.
-> Done.
All done.
[root@oracle651 ~]# scstadmin -add_lun 17 -driver iscsi -target iqn.2014-08.oracle.asm.storage:luns -device asm_reco7
Collecting current configuration: done.
-> Making requested changes.
-> Adding device 'asm_reco7' at LUN 17 to driver/target 'iscsi/iqn.2014-08.oracle.asm.storage:luns': done.
-> Driver/target is not a fibre channel target, ignoring.
-> Done.
All done.
[root@oracle651 ~]# scstadmin -add_lun 18 -driver iscsi -target iqn.2014-08.oracle.asm.storage:luns -device asm_reco8
Collecting current configuration: done.
-> Making requested changes.
-> Adding device 'asm_reco8' at LUN 18 to driver/target 'iscsi/iqn.2014-08.oracle.asm.storage:luns': done.
-> Driver/target is not a fibre channel target, ignoring.
-> Done.
All done.
[root@oracle651 ~]#
Now write the configuration changes to the /etc/scst.conf configuration file as shown below.
[root@oracle651 ~]# scstadmin -write_config /etc/scst.conf
Collecting current configuration: done.
Writing current configuration to file '/etc/scst.conf'..
All done.
[root@oracle651 ~]# cat /etc/scst.conf
# Automatically generated by SCST Configurator v2.2.1.
HANDLER vdisk_blockio {
DEVICE disk0 {
blocksize 4096
filename /dev/vdc
}
DEVICE disk1 {
filename /dev/vdb
}
}
HANDLER vdisk_fileio {
DEVICE asm_data1 {
filename /scst/asm_data1.img
}
DEVICE asm_data2 {
filename /scst/asm_data2.img
}
DEVICE asm_data3 {
filename /scst/asm_data3.img
}
DEVICE asm_data4 {
filename /scst/asm_data4.img
}
DEVICE asm_data5 {
filename /scst/asm_data5.img
}
DEVICE asm_data6 {
filename /scst/asm_data6.img
}
DEVICE asm_data7 {
filename /scst/asm_data7.img
}
DEVICE asm_data8 {
filename /scst/asm_data8.img
}
DEVICE asm_reco1 {
filename /scst/asm_reco1.img
}
DEVICE asm_reco2 {
filename /scst/asm_reco2.img
}
DEVICE asm_reco3 {
filename /scst/asm_reco3.img
}
DEVICE asm_reco4 {
filename /scst/asm_reco4.img
}
DEVICE asm_reco5 {
filename /scst/asm_reco5.img
}
DEVICE asm_reco6 {
filename /scst/asm_reco6.img
}
DEVICE asm_reco7 {
filename /scst/asm_reco7.img
}
DEVICE asm_reco8 {
filename /scst/asm_reco8.img
}
DEVICE systemdg1 {
filename /scst/systemdg1.img
}
}
TARGET_DRIVER iscsi {
enabled 1
TARGET iqn.2014-08.oracle.asm.storage:luns {
cpu_mask 3
enabled 1
rel_tgt_id 1
LUN 0 disk0
LUN 1 disk1
LUN 2 systemdg1
LUN 3 asm_data1
LUN 4 asm_data2
LUN 5 asm_data3
LUN 6 asm_data4
LUN 7 asm_data5
LUN 8 asm_data6
LUN 9 asm_data7
LUN 10 asm_data8
LUN 11 asm_reco1
LUN 12 asm_reco2
LUN 13 asm_reco3
LUN 14 asm_reco4
LUN 15 asm_reco5
LUN 16 asm_reco6
LUN 17 asm_reco7
LUN 18 asm_reco8
}
}
[root@oracle651 ~]#
Adding the new LUNs to the Oracle database server is shown below. As soon as the above command to add the LUNs to the target is completed, the LUNs will immediately appear on the the SCSI client machine - the database server - (as long as it is logged in to the SAN) and will be visible as shown below using the multipath command.
[root@oracle631 ~]# multipath -ll -v2 | grep reco
mpathr (23734383163616238) dm-17 SCST_FIO,asm_reco1
mpathq (23239613933333133) dm-16 SCST_FIO,asm_reco4
mpathp (23533633466613536) dm-14 SCST_FIO,asm_reco3
mpatho (23430363636323231) dm-13 SCST_FIO,asm_reco2
mpathv (26661333739313737) dm-21 SCST_FIO,asm_reco8
mpathu (23164346539623861) dm-20 SCST_FIO,asm_reco7
mpatht (2656563303366642d) dm-19 SCST_FIO,asm_reco6
mpaths (23361306261623634) dm-18 SCST_FIO,asm_reco5
[root@oracle631 ~]#
Use the output above the /etc/multipath.conf can be edited.
[root@oracle631 ~]# vi /etc/multipath.conf
[root@oracle631 ~]# cat /etc/multipath.conf
# Configuration file for device mapper multipath.
# For a complete list of the default configuration values:
# /usr/share/doc/device-mapper-multipath-0.4.9/multipath.conf.defaults
# For a list of configuration options with descriptions:
# /usr/share/doc/device-mapper-multipath-0.4.9/multipath.conf.annotated
defaults {
user_friendly_names yes
}
blacklist {
devnode "^(ram|raw|loop|fd|md|dm-|sr|scd|st)[0-9]*"
devnode "^hd[a-z]"
}
devices {
device {
vendor "SCST_BIO"
product "Direct-Access"
path_grouping_policy multibus
path_selector "round-robin 0"
failback immediate
rr_min_io 1
rr_weight priorities
}
}
multipaths {
multipath {
wwid 26162356632353435
alias ASM_4096B_1
}
multipath {
wwid 26238666462643332
alias ASM_0512B_1
}
multipath {
wwid 23863316131356162
alias ASM_4096F_1
}
multipath {
wwid 23233626538613166
alias ASM_0512F_1
}
multipath {
wwid 26161383334393638
alias ASM_DATA8
}
multipath {
wwid 23464666134333935
alias ASM_DATA7
}
multipath {
wwid 23565353864626532
alias ASM_DATA6
}
multipath {
wwid 23661626637333762
alias ASM_DATA5
}
multipath {
wwid 23739316465623063
alias ASM_DATA4
}
multipath {
wwid 2333730323234392d
alias ASM_DATA3
}
multipath {
wwid 23130643262613365
alias ASM_DATA2
}
multipath {
wwid 23234333531326137
alias ASM_DATA1
}
multipath {
wwid 23734383163616238
alias ASM_RECO1
}
multipath {
wwid 23430363636323231
alias ASM_RECO2
}
multipath {
wwid 23533633466613536
alias ASM_RECO3
}
multipath {
wwid 23239613933333133
alias ASM_RECO4
}
multipath {
wwid 23361306261623634
alias ASM_RECO5
}
multipath {
wwid 2656563303366642d
alias ASM_RECO6
}
multipath {
wwid 23164346539623861
alias ASM_RECO7
}
multipath {
wwid 26661333739313737
alias ASM_RECO8
}
}
[root@oracle631 ~]#
Check the permissions, ownership and group of the LUNs to be sure they are correct. With these iSCSI LUNs from SCST normally the add instantly to the client machine with correct permissions and modes and no reboot is needed. The only step required is to edit the /etc/multipath.conf file and restart multipath as shown below.
[root@oracle631 ~]# ls -lrt /dev/mapper
total 0
crw-rw---- 1 root root 10, 236 Aug 13 10:17 control
lrwxrwxrwx 1 root root 7 Aug 13 10:17 vg_oracle631-lv_swap -> ../dm-1
lrwxrwxrwx 1 root root 7 Aug 13 10:17 vg_oracle631-lv_root -> ../dm-0
lrwxrwxrwx 1 root root 7 Aug 13 10:17 vg_u00-lv_oracle -> ../dm-2
lrwxrwxrwx 1 root root 7 Aug 13 10:38 ASM_0512B_1 -> ../dm-7
lrwxrwxrwx 1 root root 7 Aug 13 10:38 ASM_DATA1 -> ../dm-5
lrwxrwxrwx 1 root root 7 Aug 13 10:38 ASM_0512F_1 -> ../dm-4
lrwxrwxrwx 1 root root 8 Aug 13 10:38 ASM_DATA6 -> ../dm-10
lrwxrwxrwx 1 root root 7 Aug 13 10:38 ASM_DATA3 -> ../dm-8
lrwxrwxrwx 1 root root 7 Aug 13 10:38 ASM_DATA4 -> ../dm-6
lrwxrwxrwx 1 root root 7 Aug 13 10:38 ASM_DATA5 -> ../dm-9
lrwxrwxrwx 1 root root 8 Aug 13 10:38 ASM_DATA8 -> ../dm-11
lrwxrwxrwx 1 root root 7 Aug 13 10:38 ASM_4096B_1 -> ../dm-3
lrwxrwxrwx 1 root root 8 Aug 13 10:38 ASM_RECO1 -> ../dm-17
lrwxrwxrwx 1 root root 8 Aug 13 10:38 ASM_DATA7 -> ../dm-12
lrwxrwxrwx 1 root root 8 Aug 13 10:38 ASM_DATA2 -> ../dm-15
lrwxrwxrwx 1 root root 8 Aug 13 10:38 ASM_RECO4 -> ../dm-16
lrwxrwxrwx 1 root root 8 Aug 13 10:38 ASM_RECO2 -> ../dm-13
lrwxrwxrwx 1 root root 8 Aug 13 10:38 ASM_RECO3 -> ../dm-14
lrwxrwxrwx 1 root root 8 Aug 13 10:38 ASM_RECO6 -> ../dm-19
lrwxrwxrwx 1 root root 8 Aug 13 10:38 ASM_RECO8 -> ../dm-21
lrwxrwxrwx 1 root root 8 Aug 13 10:38 ASM_RECO5 -> ../dm-18
lrwxrwxrwx 1 root root 8 Aug 13 10:38 ASM_RECO7 -> ../dm-20
[root@oracle631 ~]# ls -lrt /dev/dm-*
brw-rw---- 1 root disk 252, 1 Aug 13 10:17 /dev/dm-1
brw-rw---- 1 root disk 252, 0 Aug 13 10:17 /dev/dm-0
brw-rw---- 1 root disk 252, 2 Aug 13 10:17 /dev/dm-2
brw-rw---- 1 grid asmadmin 252, 7 Aug 13 10:38 /dev/dm-7
brw-rw---- 1 grid asmadmin 252, 5 Aug 13 10:38 /dev/dm-5
brw-rw---- 1 grid asmadmin 252, 4 Aug 13 10:38 /dev/dm-4
brw-rw---- 1 grid asmadmin 252, 10 Aug 13 10:38 /dev/dm-10
brw-rw---- 1 grid asmadmin 252, 8 Aug 13 10:38 /dev/dm-8
brw-rw---- 1 grid asmadmin 252, 6 Aug 13 10:38 /dev/dm-6
brw-rw---- 1 grid asmadmin 252, 9 Aug 13 10:38 /dev/dm-9
brw-rw---- 1 grid asmadmin 252, 3 Aug 13 10:38 /dev/dm-3
brw-rw---- 1 grid asmadmin 252, 11 Aug 13 10:38 /dev/dm-11
brw-rw---- 1 grid asmadmin 252, 17 Aug 13 10:38 /dev/dm-17
brw-rw---- 1 grid asmadmin 252, 12 Aug 13 10:38 /dev/dm-12
brw-rw---- 1 grid asmadmin 252, 15 Aug 13 10:38 /dev/dm-15
brw-rw---- 1 grid asmadmin 252, 16 Aug 13 10:38 /dev/dm-16
brw-rw---- 1 grid asmadmin 252, 13 Aug 13 10:38 /dev/dm-13
brw-rw---- 1 grid asmadmin 252, 14 Aug 13 10:38 /dev/dm-14
brw-rw---- 1 grid asmadmin 252, 19 Aug 13 10:38 /dev/dm-19
brw-rw---- 1 grid asmadmin 252, 21 Aug 13 10:38 /dev/dm-21
brw-rw---- 1 grid asmadmin 252, 18 Aug 13 10:38 /dev/dm-18
brw-rw---- 1 grid asmadmin 252, 20 Aug 13 10:38 /dev/dm-20
[root@oracle631 ~]#
Configure the LUNs with ASMLib as shown below. Alternatively, for a large number of LUNs scripted methods can be used as shown in the Violin Memory Oracle Database Best Practice Document April 2014.
[root@oracle631 ~]# oracleasm createdisk ASM_RECO1 /dev/mapper/ASM_RECO1
Writing disk header: done
Instantiating disk: done
[root@oracle631 ~]# oracleasm createdisk ASM_RECO2 /dev/mapper/ASM_RECO2
Writing disk header: done
Instantiating disk: done
[root@oracle631 ~]# oracleasm createdisk ASM_RECO3 /dev/mapper/ASM_RECO3
Writing disk header: done
Instantiating disk: done
[root@oracle631 ~]# oracleasm createdisk ASM_RECO4 /dev/mapper/ASM_RECO4
Writing disk header: done
Instantiating disk: done
[root@oracle631 ~]# oracleasm createdisk ASM_RECO5 /dev/mapper/ASM_RECO5
Writing disk header: done
Instantiating disk: done
[root@oracle631 ~]# oracleasm createdisk ASM_RECO6 /dev/mapper/ASM_RECO6
Writing disk header: done
Instantiating disk: done
[root@oracle631 ~]# oracleasm createdisk ASM_RECO7 /dev/mapper/ASM_RECO7
Writing disk header: done
Instantiating disk: done
[root@oracle631 ~]# oracleasm createdisk ASM_RECO8 /dev/mapper/ASM_RECO8
Writing disk header: done
Instantiating disk: done
[root@oracle631 ~]#
Connect to the server as grid user as shown below and run "asmca" to build the RECO asm diskgroup. Note that asmca can be used here because these are 512 emulated LUNs so no special attributes are needed. If these were 4096 native LUNs command line would have to be used to build the diskgroup.
gstanden@vmem1:~$ ssh -Y -C grid@10.207.39.72
grid@10.207.39.72's password:
Last login: Wed Aug 13 10:45:42 2014 from 10.207.39.1
[grid@oracle631 ~]$ asmca
The screenshots of asmca are shown below.
The final configuration of disks for the Oracle database install.
SQL> l
1* select name, label, path, total_mb, header_status, mount_status, mode_status, sector_size from v$asm_disk
SQL> /
NAME LABEL PATH TOTAL_MB HEADER_STATU MOUNT_S MODE_ST SECTOR_SIZE
------------------------------ ------------------------------- -------------------- ---------- ------------ ------- ------- -----------
ASM_DATA1 ASM_DATA1 ORCL:ASM_DATA1 1024 MEMBER CACHED ONLINE 512
ASM_DATA2 ASM_DATA2 ORCL:ASM_DATA2 1024 MEMBER CACHED ONLINE 512
ASM_DATA3 ASM_DATA3 ORCL:ASM_DATA3 1024 MEMBER CACHED ONLINE 512
ASM_DATA4 ASM_DATA4 ORCL:ASM_DATA4 1024 MEMBER CACHED ONLINE 512
ASM_DATA5 ASM_DATA5 ORCL:ASM_DATA5 1024 MEMBER CACHED ONLINE 512
ASM_DATA6 ASM_DATA6 ORCL:ASM_DATA6 1024 MEMBER CACHED ONLINE 512
ASM_DATA7 ASM_DATA7 ORCL:ASM_DATA7 1024 MEMBER CACHED ONLINE 512
ASM_DATA8 ASM_DATA8 ORCL:ASM_DATA8 1024 MEMBER CACHED ONLINE 512
SYSTEMDG SYSTEMDG ORCL:SYSTEMDG 1024 MEMBER CACHED ONLINE 512
ASM_RECO1 ASM_RECO1 ORCL:ASM_RECO1 1024 MEMBER CACHED ONLINE 512
ASM_RECO2 ASM_RECO2 ORCL:ASM_RECO2 1024 MEMBER CACHED ONLINE 512
ASM_RECO3 ASM_RECO3 ORCL:ASM_RECO3 1024 MEMBER CACHED ONLINE 512
ASM_RECO4 ASM_RECO4 ORCL:ASM_RECO4 1024 MEMBER CACHED ONLINE 512
ASM_RECO5 ASM_RECO5 ORCL:ASM_RECO5 1024 MEMBER CACHED ONLINE 512
ASM_RECO6 ASM_RECO6 ORCL:ASM_RECO6 1024 MEMBER CACHED ONLINE 512
ASM_RECO7 ASM_RECO7 ORCL:ASM_RECO7 1024 MEMBER CACHED ONLINE 512
ASM_RECO8 ASM_RECO8 ORCL:ASM_RECO8 1024 MEMBER CACHED ONLINE 512
17 rows selected.
SQL>
Obtain the Oracle binaries and install for your chosen version of database. Begin the install. Here the oracle install media is unpacked under the /home/oracle directory as shown below.
[oracle@oracle631 database]$ pwd
/home/oracle/database
[oracle@oracle631 database]$ ./runInstaller
Starting Oracle Universal Installer...
Checking Temp space: must be greater than 120 MB. Actual 2527 MB Passed
Checking swap space: must be greater than 150 MB. Actual 5087 MB Passed
Checking monitor: must be configured to display at least 256 colors. Actual 16777216 Passed
Preparing to launch Oracle Universal Installer from /tmp/OraInstall2014-08-13_11-36-50AM. Please wait ...
[oracle@oracle631 database]$