Oracle RAC 6-node 12c GNS ASM Flex Cluster Ubuntu 15.04 Install
GENERAL NOTES
What is this page? It is a guide that provides a set of scripts so that you can prepare an Ubuntu 15.04 64-bit desktop to run Oracle Enterprise software products (such as Oracle Enterprise Edition 12c RAC GNS ASM Flex Cluster) using LXC Linux Containers on an OpenvSwitch network.
Why would you want to build this? To have an elastic and dense deployment of Oracle Enterprise Edtion 12c RAC nodes on Ubuntu 15.04 64-bit desktop edition that runs at bare metal speed with NO hypervisor. This project is very useful for RAC training schools. The scripts can be rerun to destroy and rebuild the cluster in the cycle typical of an Oracle RAC training program, all on Ubuntu Linux, the premier free linux desktop OS, ideal for students.
Note that currently running Oracle Enteprise Edition software in LXC containers ontop of Ubuntu Linux is not supported by Oracle, so you would have to run this configuration and support it on your own or simply just use it for educational and training purposes. It should not be used for production in cases where you need fully-supported deployment.
The setup uses the following main components as shown below
bind9
isc-dhcp-server
openvswitch
lxc
[ Note: that this LXC containerized approach to Oracle Enteprise Software product deployment on Linux(es) can theoretically be used on any Linux distribution, such as trivially RedHat, but theoretically also Gentoo and really any distribution that supports LXC Linux Containers. Of course the details could be wildly different as to how to duplicate this Ubuntu work, because on Gentoo for example, you'd have to rewrite all the package commands as "emerge" commands. ]
I was able to create a 12-node LXC-container RAC on my Lenovo W520 Mobile Workstation:
Intel® Core™ i7-2720QM CPU @ 2.20GHz × 8
32Gb of RAM
ST1000LM014-1EJ164 (SM28) (1Tb Hybrid Disk with SSD Cache by Seagate)
The load averages begin to increase non-linearly after about 8-node RAC, and at 12-node RAC the load averages as measured by top are close to 100. Thus, the 6-lxc container RAC is about the upper limit for a high-end desktop or laptop. Some specs from the running 12-node RAC are shown below. Note that one interesting feature of LXC containerized RAC is that each LXC node can access ALL memory available on the Ubuntu host. The 12-node containerized RAC is at the very edge of the upper-limit for this workstation.
Memory Usage: 15 Gb
Note: September 18, 2015: The software for this project is now maintained at github. The bundles attached at this webpage should not be used as the primary source for the software. Go to https://github.com/gstanden/orabuntu-lxc to download the latest zip bundle of the software. This page can be used as a supplementary guide for configuring the software that you download from github.
NOTE: The install bundle of files are now a github project at "orabuntu-lxc" at github.
Gilbert Standen
Yonkers, NY
September 13, 2015
INSTALL INSTRUCTIONS START HERE
DO A FRESH INSTALL OF UBUNTU 15.04 64-bit DESKTOP SOFTWARE. ONLY RUN THE SCRIPT BUNDLE ON A FRESH INSTALL.
NOTE: When you install the Ubuntu 15.04 be sure to take the "set time automatically over the internet" options which is to say be sure NTP is configured and running on your Ubuntu 15.04 install. This is needed because this guide uses the Ubuntu host server as a local NTP source for the LXC containers.
Do not do an install on a customized and been-running-for-awhile Ubuntu 15.04 - such install is at your own risk!! These scripts are not yet sufficiently bulletproofed for that type of install and could overwrite your existing customizations on an Ubuntu 15.04 that has been running for some time and has customizations.
UPDATE YOUR FRESH INSTALL OF UBUNTU 15.04 64-bit DESKTOP SOFTWARE BEFORE DOING THIS INSTALL.
IF YOUR UBUNTU 15.04 IS UP-TO-DATE YOU WILL GET THE FOLLOWING DIALOG WHEN YOU RUN THE UBUNTU SOFTWARE UPDATER UTILITY, AS SHOWN BELOW.
Get this software from github now, do not use the early development bundles at the bottom of this blog.
https://github.com/gstanden/orabuntu-lxc
Don't use the bundles at the end of this blog anymore, the software has moved to github. This google sites page is just the guide now.
Note also that the IP address assignments are designed to have the IP 4th triplet match the container name, i.e. "lxcora10" has a public IP address of "10.207.39.10", and lxcora11 has a public IP address of "10.207.39.11", and so on. and all the ASM and private networks are the same, so that the IP can be easily remembered for the container to which the IP is assigned.
Oracle Support Document # 1918426.1
12.1.0.2 root script fails to start ora.ctssd if nodes name length are not the same
The root.sh script will error out on any nodes that have a different number of characters in the length of the node name compared to the first node with the following error as shown below. The latest version at github ensures that all lxc container nodes have names of the form { lxcora00, lxcora01, ..., lxcora99 }. Since Oracle 12c allows a maximum of 64 RAC nodes, this fix should allow for any number of nodes up to the maximum the software allows.
CRS-4406: Oracle High Availability Services synchronous start failed.
CRS-4000: Command Start failed, or completed with errors.
2015/09/13 13:10:16 CLSRSC-117: Failed to start Oracle Clusterware stack
Died at /u00/app/grid/product/12.1.0/grid/crs/install/crsinstall.pm line 914.
The command '/u00/app/grid/product/12.1.0/grid/perl/bin/perl -I/u00/app/grid/product/12.1.0/grid/perl/lib -I/u00/app/grid/product/12.1.0/grid/crs/install /u00/app/grid/product/12.1.0/grid/crs/install/rootcrs.pl ' execution failed
Now open a terminal session on the Ubuntu desktop and go the the ~/Downloads subdirectory of your user account. For example, on my desktop this directory will be:
/home/oracle/Downloads
because I installed this Ubuntu 15.04 using "oracle" as the default username.
Now gunzip and untar the bundle as shown below.
oracle@W521:~$ cd Downloads
oracle@W521:~/Downloads$ ls -lrt
total 52
-rw-rw-r-- 1 oracle oracle 51614 Sep 3 21:54 ubuntu-lxc-oracle.v9.tar.gz
oracle@W521:~/Downloads$ gunzip ubuntu-lxc-oracle.v9.tar.gz
oracle@W521:~/Downloads$ tar -xvf ubuntu-lxc-oracle.v9.tar
ubuntu-host.lst
ubuntu-host.tar
lxc-config.lst
lxc-config.tar
lxc-lxcora01.lst
lxc-lxcora01.tar
lxc-lxcora0x.lst
lxc-lxcora0x.tar
scst-files.tar
ubuntu-services-1.sh
ubuntu-services-2a.sh
ubuntu-services-2b.sh
ubuntu-services-3a.sh
ubuntu-services-3b.sh
ubuntu-services-3c.sh
ubuntu-services-3d.sh
ubuntu-host-backup.sh
rc.local
dhclient.conf
lxc-services.sh
install_grid.sh
edit_bashrc
crt_links_v2.sh
ubuntu-lxc-oracle.lst.copy
README
oracle@W521:~/Downloads$
Run the scripts in this order as shown below. Some scripts will reboot the host when finished.
oracle@W521:~/Downloads$ ls ubuntu-services* | sort
ubuntu-services-1.sh
ubuntu-services-2a.sh
ubuntu-services-2b.sh
ubuntu-services-3a.sh
ubuntu-services-3b.sh
ubuntu-services-3c.sh
ubuntu-services-3d.sh
oracle@W521:~/Downloads$
If you have not previously run any sudo commands, the ubuntu-services-1.sh script will stop to prompt for your user password so that sudo commands can be run as shown below. You probably won't need to enter your sudo password again, this will be the only time.
oracle@W521:~/Downloads$ ./ubuntu-services-1.sh
============================================
Verify network up....
============================================
PING google.com (74.125.196.113) 56(84) bytes of data.
64 bytes from yk-in-f113.1e100.net (74.125.196.113): icmp_seq=1 ttl=41 time=31.8 ms
--- google.com ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 31.882/31.882/31.882/0.000 ms
[sudo] password for oracle:
In this example, the default user account created at install of the OS was "oracle". Yours might be /home/jsmith/Downloads if your name was Jim Smith and you selected "jsmith" as your username when you installed the Ubuntu base OS on your laptop or desktop.
The ubuntu-services-3c.sh script script needs to have a parameter passed in (the number of RAC nodes you want). I have a Lenovo W520 top-of-the-line mobile workstation with a quad-core processor and 32 Gb of RAM, so I choose "6" for "six RAC node LXC containers". If you give no parameter at all, it defaults to "2" so if you have an "ordinary" desktop with 4Gb of RAM or maybe 8Gb of RAM, just pass in "2" as the parameter or take the default which is also "2".
================================================
Now run ubuntu-services-3c.sh X
Note that ubuntu-services-3c.sh takes an input
variable X which is the number of LXC RAC nodes
you wish to create. If X is not entered, the
build defaults to a 2-node RAC cluster. If X is
set to 6 it will create a 6-node RAC
================================================
oracle@W521:~/Downloads$ ./ubuntu-services-3c.sh 6
When you get this message below, in most cases you will answer "N" because you should be installing on a fresh copy of Ubuntu 15.04 64-bit desktop and there should be no previously-created containers. However, this code exists in the script because I'm working on dealing with the case that containers of the same name already exist and in general working towards making this script bundle safe for a customized been-running-for-awhile deployment of Ubuntu 15.04 64-bit desktop. Usually, just answer "N" to this, especially if as shown below, no containers are found matching the naming criteria output shows empty set { } as shown below.
!!! WARNING !!!
===========================================
Destruction of cloned containers ( Y / N )
===========================================
ClonedContainersExist =
Existing containers in the set { } have been found. <-- the "{ }" means no containers were found, so just answer "N".
These containers match the names of containers that are about to be created.
Please answer Y to destroy the existing containers or N to keep them
!!! WARNING: ANSWERING Y WILL DESTROY EXISTING CONTAINERS !!!
Destroy existing containers? [ Y | N ]:
N
Final output shows all Oracle-ready lxcora0x containers are running and ready for Oracle RAC Grid Infrastructure install as shown below.
================================================
Starting LXC clone containers for Oracle
================================================
starting a container...
lxcora01
next command will be: sudo lxc-start -n lxcora01
sleeping 20 seconds...
starting a container...
lxcora02
next command will be: sudo lxc-start -n lxcora02
sleeping 20 seconds...
starting a container...
lxcora03
next command will be: sudo lxc-start -n lxcora03
sleeping 20 seconds...
starting a container...
lxcora04
next command will be: sudo lxc-start -n lxcora04
sleeping 20 seconds...
starting a container...
lxcora05
next command will be: sudo lxc-start -n lxcora05
sleeping 20 seconds...
starting a container...
lxcora06
next command will be: sudo lxc-start -n lxcora06
sleeping 20 seconds...
================================================
Waiting for final container initialization...
================================================
================================================
LXC containers for Oracle started.
================================================
NAME STATE IPV4 IPV6 GROUPS AUTOSTART
--------------------------------------------------------------------------------------------------------------------------------------------------
lxcora00 STOPPED - - - NO
lxcora01 RUNNING 10.207.39.10, 172.220.40.10, 172.221.40.10, 192.210.39.10, 192.211.39.10, 192.212.39.10, 192.213.39.10 - - NO
lxcora02 RUNNING 10.207.39.11, 172.220.40.11, 172.221.40.11, 192.210.39.11, 192.211.39.11, 192.212.39.11, 192.213.39.11 - - NO
lxcora03 RUNNING 10.207.39.12, 172.220.40.12, 172.221.40.12, 192.210.39.12, 192.211.39.12, 192.212.39.12, 192.213.39.12 - - NO
lxcora04 RUNNING 10.207.39.13, 172.220.40.13, 172.221.40.13, 192.210.39.13, 192.211.39.13, 192.212.39.13, 192.213.39.13 - - NO
lxcora05 RUNNING 10.207.39.14, 172.220.40.14, 172.221.40.14, 192.210.39.14, 192.211.39.14, 192.212.39.14, 192.213.39.14 - - NO
lxcora06 RUNNING 10.207.39.15, 172.220.40.15, 172.221.40.15, 192.210.39.15, 192.211.39.15, 192.212.39.15, 192.213.39.15 - - NO
oracle@W521:~/Downloads$
SCST SAN SETUP
[ A side note about running SCST (which is covered next): This guide uses SCST, which requires a custom kernel, because SCST is one of the few Linux SAN open source offerings that support 4K logical / 4K physical LUN presentation. This blog grew out of some ideas I had while working at Violin Memory to create a desktop "workshop" where customers could experiment with 4K native format storage with Oracle software products, so a Linux SAN that fully supports 4K as well as the traditional 512-byte format was needed. Also, SCST is behind-the-scenes a component of VMOS5 Violin Memory arrays, and also some generations of Kaminario all-flash arrays.
However, if you will not need 4K support for your Linux SAN, you could just as easily use Linux TGT which does not require the kernel build steps, and can just be configured and used without a custom SCST kernel build. So, if you prefer, you can build out your Linux SAN storage for this containerized RAC using TGT although I'm not providing those instructions because they are more trivial and guides for TGT setup are readily available on the net for example here. Note that if you go the TGT route, you won't be able to experiment with Oracle and 4K native storage interactions, which are interesting and never dull to say the least. ]
Next untar the scst-files.tar file and cd to scst-files to create the SCST Linux SAN which will provide the storage LUNs for the Oracle 12c Grid Infrastructure install as shown below. Run the scripts in the order shown below.
oracle@W521:~/Downloads$ ls -lrt scst*
-rw-rw-r-- 1 oracle oracle 40960 Sep 2 21:57 scst-files.tar
oracle@W521:~/Downloads$ tar -xf scst-files.tar
oracle@W521:~/Downloads$ cd scst-files
oracle@W521:~/Downloads/scst-files$ ls create-scst* | sort
create-scst-1a.sh <-- Run the scripts in this order !
create-scst-1b.sh
create-scst-1c.sh
create-scst-1d.sh
create-scst-2a.sh
create-scst-2b.sh
create-scst-3.sh
create-scst-4a.sh
create-scst-4b.sh
create-scst-5a.sh
create-scst-5b.sh
create-scst.sh
oracle@W521:~/Downloads/scst-files$
When you run the create-scst-1d.sh script, you will get these prompts of the form "(NEW)" and just hit enter for each one (accepts default value) as shown below. See bold section at end of following blockcode quote where the first prompt for "(NEW)" appears.
oracle@W521:~/Downloads/scst-files$ ./create-scst-1d.sh
for i in debian.master/d-i/kernel-versions.in debian.master/control.stub.in; do \
new=`echo $i | sed 's/\.in$//'`; \
cat $i | sed -e 's/PKGVER/3.19.0/g' \
-e 's/ABINUM/26/g' \
-e 's/SRCPKGNAME/linux/g' \
-e 's/=HUMAN=/64 bit x86/g' \
> $new; \
done
flavours="debian.master/control.d/vars.generic debian.master/control.d/vars.generic-lpae debian.master/control.d/vars.lowlatency debian.master/control.d/vars.powerpc-e500mc debian.master/control.d/vars.powerpc-smp debian.master/control.d/vars.powerpc64-emb debian.master/control.d/vars.powerpc64-smp debian.master/control.d/vars.scst";\
for i in $flavours; do \
/bin/bash -e debian/scripts/control-create $i | \
sed -e 's/PKGVER/3.19.0/g' \
-e 's/ABINUM/26/g' \
-e 's/SRCPKGNAME/linux/g' \
-e 's/=HUMAN=/64 bit x86/g' \
>> debian.master/control.stub; \
done
cp debian.master/control.stub debian.master/control
rm -rf /home/oracle/Downloads/linux-3.19.0/debian/build/modules /home/oracle/Downloads/linux-3.19.0/debian/build/firmware \
/home/oracle/Downloads/linux-3.19.0/debian/build/kernel-versions /home/oracle/Downloads/linux-3.19.0/debian/build/package-list \
/home/oracle/Downloads/linux-3.19.0/debian/build/debian.master
mkdir -p /home/oracle/Downloads/linux-3.19.0/debian/build/modules/amd64/
cp debian.master/d-i/modules/* /home/oracle/Downloads/linux-3.19.0/debian/build/modules/amd64/
mkdir -p /home/oracle/Downloads/linux-3.19.0/debian/build/firmware/amd64/
cp debian.master/d-i/firmware/* /home/oracle/Downloads/linux-3.19.0/debian/build/firmware/amd64/
cp debian.master/d-i/package-list debian.master/d-i/kernel-versions /home/oracle/Downloads/linux-3.19.0/debian/build/
touch /home/oracle/Downloads/linux-3.19.0/debian/build/modules/amd64/kernel-image
# kernel-wedge needs to poke around in debian.master/
ln -nsf /home/oracle/Downloads/linux-3.19.0/debian /home/oracle/Downloads/linux-3.19.0/debian/build/debian
# Some files may need to differ between architectures
if [ -d debian.master/d-i/modules-amd64 ]; then \
cp debian.master/d-i/modules-amd64/* \
/home/oracle/Downloads/linux-3.19.0/debian/build/modules/amd64/; \
fi
if [ -d debian.master/d-i/firmware-amd64 ]; then \
cp debian.master/d-i/firmware-amd64/* \
/home/oracle/Downloads/linux-3.19.0/debian/build/firmware/amd64/; \
fi
# Remove unwanted stuff for this architecture
if [ -r "debian.master/d-i/exclude-modules.amd64" ]; then \
(cat debian.master/d-i/exclude-modules.amd64; \
ls /home/oracle/Downloads/linux-3.19.0/debian/build/modules/amd64/) | sort | uniq -d | \
(cd /home/oracle/Downloads/linux-3.19.0/debian/build/modules/amd64/; xargs rm -f); \
fi
if [ -r "debian.master/d-i/exclude-firmware.amd64" ]; then \
(cat debian.master/d-i/exclude-firmware.amd64; \
ls /home/oracle/Downloads/linux-3.19.0/debian/build/firmware/amd64/) | sort | uniq -d | \
(cd /home/oracle/Downloads/linux-3.19.0/debian/build/firmware/amd64/; xargs rm -f); \
fi
# Per flavour module lists
flavour_modules=`ls debian.master/d-i/modules.amd64-* 2>/dev/null` \
|| true; \
if [ "$flavour_modules" != "" ]; then \
for flav in $flavour_modules; do \
name=`echo $flav | sed 's/.*\/modules.amd64-//'`; \
mkdir /home/oracle/Downloads/linux-3.19.0/debian/build/modules/amd64-$name; \
(cd /home/oracle/Downloads/linux-3.19.0/debian/build/modules/; tar cf - `cat ../$flav`) | \
(cd /home/oracle/Downloads/linux-3.19.0/debian/build/modules/amd64-$name/; tar xf -); \
touch /home/oracle/Downloads/linux-3.19.0/debian/build/modules/amd64-$name/kernel-image; \
done; \
fi
flavour_firmware=`ls debian.master/d-i/firmware.amd64-* 2>/dev/null` \
|| true; \
if [ "$flavour_firmware" != "" ]; then \
for flav in $flavour_firmware; do \
name=`echo $flav | sed 's/.*\/firmware.amd64-//'`; \
mkdir /home/oracle/Downloads/linux-3.19.0/debian/build/firmware/amd64-$name; \
(cd /home/oracle/Downloads/linux-3.19.0/debian/build/firmware/; tar cf - `cat ../$flav`) | \
(cd /home/oracle/Downloads/linux-3.19.0/debian/build/firmware/amd64-$name/; tar xf -);\
touch /home/oracle/Downloads/linux-3.19.0/debian/build/firmware/amd64-$name/kernel-image; \
done; \
fi
# Some files may need to differ between flavours
flavour_module_dirs=`ls -d debian.master/d-i/modules-amd64-* 2>/dev/null`\
|| true; \
if [ "$flavour_module_dirs" ]; then \
for flav in $flavour_module_dirs; do \
name=`echo $flav | sed 's/.*\/modules-amd64-//'`; \
[ -d /home/oracle/Downloads/linux-3.19.0/debian/build/modules/amd64-$name ] || \
cp -a /home/oracle/Downloads/linux-3.19.0/debian/build/modules/amd64 \
modules/amd64-$name; \
cp $flav/* /home/oracle/Downloads/linux-3.19.0/debian/build/modules/amd64-$name/; \
done; \
fi
flavour_firmware_dirs=`ls -d debian.master/d-i/firmware-amd64-* 2>/dev/null`\
|| true; \
if [ "$flavour_firmware_dirs" ]; then \
for flav in $flavour_firmware_dirs; do \
name=`echo $flav | sed 's/.*\/firmware-amd64-//'`; \
[ -d /home/oracle/Downloads/linux-3.19.0/debian/build/firmware/amd64-$name ] || \
cp -a /home/oracle/Downloads/linux-3.19.0/debian/build/firmware/amd64 \
firmware/amd64-$name; \
cp $flav/* /home/oracle/Downloads/linux-3.19.0/debian/build/firmware/amd64-$name/; \
done; \
fi
# Remove unwanted stuff for each flavour
flavour_exclude=`ls debian.master/d-i/exclude-modules.amd64-* 2>/dev/null`\
|| true; \
if [ "$flavour_exclude" ]; then \
for flav in $flavour_exclude; do \
name=`echo $flav | sed 's/.*\/exclude-modules.amd64-//'`;\
[ -d /home/oracle/Downloads/linux-3.19.0/debian/build/modules/amd64-$name ] || \
cp -a /home/oracle/Downloads/linux-3.19.0/debian/build/modules/amd64 \
/home/oracle/Downloads/linux-3.19.0/debian/build/modules/amd64-$name; \
(cat $flav; \
ls /home/oracle/Downloads/linux-3.19.0/debian/build/modules/amd64-$name) | \
sort | uniq -d | \
(cd /home/oracle/Downloads/linux-3.19.0/debian/build/modules/amd64-$name/; \
xargs rm -f); \
done; \
fi
flavour_exclude=`ls debian.master/d-i/exclude-firmware.amd64-* 2>/dev/null`\
|| true; \
if [ "$flavour_exclude" ]; then \
for flav in $flavour_exclude; do \
name=`echo $flav | sed 's/.*\/exclude-firmware.amd64-//'`;\
[ -d /home/oracle/Downloads/linux-3.19.0/debian/build/firmware/amd64-$name ] || \
cp -a /home/oracle/Downloads/linux-3.19.0/debian/build/firmware/amd64 \
/home/oracle/Downloads/linux-3.19.0/debian/build/firmware/amd64-$name; \
(cat $flav; \
ls /home/oracle/Downloads/linux-3.19.0/debian/build/firmware/amd64-$name) | \
sort | uniq -d | \
(cd /home/oracle/Downloads/linux-3.19.0/debian/build/firmware/amd64-$name/; \
xargs rm -f); \
done; \
fi
if [ ! -d /home/oracle/Downloads/linux-3.19.0/debian/build/modules/x86_64 ]; then \
mkdir -p /home/oracle/Downloads/linux-3.19.0/debian/build/modules/x86_64; \
cp /home/oracle/Downloads/linux-3.19.0/debian/build/modules/amd64/* \
/home/oracle/Downloads/linux-3.19.0/debian/build/modules/x86_64; \
fi
if [ ! -d /home/oracle/Downloads/linux-3.19.0/debian/build/firmware/x86_64 ]; then \
mkdir -p /home/oracle/Downloads/linux-3.19.0/debian/build/firmware/x86_64; \
cp /home/oracle/Downloads/linux-3.19.0/debian/build/firmware/amd64/* \
/home/oracle/Downloads/linux-3.19.0/debian/build/firmware/x86_64; \
fi
cp debian.master/control.stub debian/control.stub
cd /home/oracle/Downloads/linux-3.19.0/debian/build && LANG=C kernel-wedge gen-control > /home/oracle/Downloads/linux-3.19.0/debian/control
Use of uninitialized value $builddep in string ne at /usr/share/kernel-wedge/commands/gen-control line 43, <KVERS> line 6.
Use of uninitialized value $builddep in split at /usr/share/kernel-wedge/commands/gen-control line 44, <KVERS> line 6.
Use of uninitialized value $builddep in string ne at /usr/share/kernel-wedge/commands/gen-control line 43, <KVERS> line 8.
Use of uninitialized value $builddep in split at /usr/share/kernel-wedge/commands/gen-control line 44, <KVERS> line 8.
Use of uninitialized value $builddep in string ne at /usr/share/kernel-wedge/commands/gen-control line 43, <KVERS> line 10.
Use of uninitialized value $builddep in split at /usr/share/kernel-wedge/commands/gen-control line 44, <KVERS> line 10.
Use of uninitialized value $builddep in string ne at /usr/share/kernel-wedge/commands/gen-control line 43, <KVERS> line 12.
Use of uninitialized value $builddep in split at /usr/share/kernel-wedge/commands/gen-control line 44, <KVERS> line 12.
dh_testdir
dh_testroot
dh_clean
# d-i stuff
rm -rf debian.master/d-i-amd64
# Generated on the fly.
rm -f debian.master/d-i/firmware/kernel-image
# normal build junk
rm -rf debian.master/abi/3.19.0-26.28
rm -rf /home/oracle/Downloads/linux-3.19.0/debian/build
rm -f /home/oracle/Downloads/linux-3.19.0/debian/stamps/stamp-*
rm -rf debian.master/linux-*
# This gets rid of the d-i packages in control
cp -f debian.master/control.stub debian.master/control
cp debian.master/changelog debian/changelog
# Install the copyright information.
cp debian.master/copyright debian/copyright
# If we have a reconstruct script use it.
[ -f debian.master/reconstruct ] && bash -x debian.master/reconstruct
+ '[' '!' -L ubuntu/vbox/vboxguest/include ']'
+ ln -sf ../include ubuntu/vbox/vboxguest/include
+ '[' '!' -L ubuntu/vbox/vboxguest/r0drv ']'
+ ln -sf ../r0drv ubuntu/vbox/vboxguest/r0drv
+ '[' '!' -L ubuntu/vbox/vboxsf/include ']'
+ ln -sf ../include ubuntu/vbox/vboxsf/include
+ '[' '!' -L ubuntu/vbox/vboxsf/r0drv ']'
+ ln -sf ../r0drv ubuntu/vbox/vboxsf/r0drv
+ '[' '!' -L ubuntu/vbox/vboxvideo/include ']'
+ ln -sf ../include ubuntu/vbox/vboxvideo/include
+ exit 0
dh_testdir;
/bin/bash -e debian/scripts/misc/kernelconfig updateconfigs
* Run silentoldconfig (yes=0) on amd64/config.flavour.generic ...
make[1]: Entering directory '/home/oracle/Downloads/linux-3.19.0'
make[2]: Entering directory '/home/oracle/Downloads/linux-3.19.0/build'
HOSTCC scripts/basic/fixdep
GEN ./Makefile
HOSTCC scripts/kconfig/conf.o
SHIPPED scripts/kconfig/zconf.tab.c
SHIPPED scripts/kconfig/zconf.lex.c
SHIPPED scripts/kconfig/zconf.hash.c
HOSTCC scripts/kconfig/zconf.tab.o
HOSTLD scripts/kconfig/conf
scripts/kconfig/conf --silentoldconfig Kconfig
.config:3896:warning: override: M686 changes choice state
.config:7457:warning: override: TREE_RCU changes choice state
.config:8637:warning: override: TICK_CPU_ACCOUNTING changes choice state
.config:8640:warning: override: USB_DWC2_HOST changes choice state
*
* Restart config...
*
*
* Networking options
*
Packet socket (PACKET) [Y/n/m/?] y
Packet: sockets monitoring interface (PACKET_DIAG) [M/n/y/?] m
Unix domain sockets (UNIX) [Y/n/m/?] y
UNIX: socket monitoring interface (UNIX_DIAG) [M/n/y/?] m
Transformation user configuration interface (XFRM_USER) [M/n/y/?] m
Transformation sub policy support (XFRM_SUB_POLICY) [N/y/?] n
Transformation migrate database (XFRM_MIGRATE) [N/y/?] n
Transformation statistics (XFRM_STATISTICS) [Y/n/?] y
PF_KEY sockets (NET_KEY) [M/n/y/?] m
PF_KEY MIGRATE (NET_KEY_MIGRATE) [N/y/?] n
TCP/IP networking (INET) [Y/n/?] y
TCP/IP zero-copy transfer completion notification (TCP_ZERO_COPY_TRANSFER_COMPLETION_NOTIFICATION) [N/y/?] (NEW)
After answering many of these "(NEW)" prompts and accepting default value "(NEW)" you will eventually get the following prompt about "config.flavour.scst" as shown below.
Notice that all prompts should be answered with "n" in this phase EXCEPT the line for "amd64/config.flavour.scst? [Y/n] Y" which as you can see should be answered with "Y" as shown below (bolded line). Be sure that ONLY this config is answered with "Y" . All other config line prompts should be answered with "n" as shown below.
check-config: 43/43 checks passed -- exit 0
rm -rf build
dh_testdir;
/bin/bash -e debian/scripts/misc/kernelconfig editconfigs
Do you want to edit config: amd64/config.flavour.generic? [Y/n] n
Do you want to edit config: amd64/config.flavour.lowlatency? [Y/n] n
Do you want to edit config: amd64/config.flavour.scst? [Y/n] Y
Answer "n" to the rest of the "config" prompts. There will be an error message about some builds as shown below, but these are expected and can be ignored because we only care about the build on amd64 architecture, which is successful.
*** ERROR: 12 config-check failures detected <-- This error is ok, ignore
When you choose the "Y" for config.flavour.scst you will be presented with a linux GUI screen that will popup automatically. This utility is so that you can configure the new SCST options. Screenshots are shown below to guide you through choosing the required SCST options for the kernel.
This is the screen you first get as shown below.
Use the "down arrow" key on your keyboard to move down to "Networking support --->" as shown below.
Use the down arrow key to move down to "Networking options --->" as shown below.
Use the down arrow key to move down to "TCP/IP zero-copy transfer completion notification" and press the spacebar on the keyboard to put a " * " symbol in the brackets, i.e. [ * ] as shown below (selects the feature).
Now use the right-arrow key on the keyboard to move the lower highlighted value from "<Select>" to "<Exit>", and then hit <Enter> to exit this screen, as shown below.
Repeat the same sequence of steps to "<Exit>" the next screen too, as shown below.
Now use the down-arrow key on the keyboard to select "Device Drivers --->" as shown below and hit <Enter> to select.
Now use the down-arrow key on the keyboard to move down to "SCSI target (SCST) support --->" and hit <Enter> to select option.
The "< > SCSI target (SCST) support" option is displayed. Press the spacebar to enable the large number of options associated with this and to put a " *M" in the < M> to select all options as shown in the next screenshot following.
Once the "<M>" is in the option, all the additional options appear. No need to make anymore changes, this is it for this screen. Now use the right arrow key as before to exit from this screen back up to the main screen as shown in the next screen following.
Next screenshot shows using right-arrow key to exit back up to the previous screen, as shown below.
Again, use the right-arrow key to select "<Exit>" to exit up to the next higher screen as shown below.
Again, use right-arrow screen to select "<Exit>" to move back up to the next higher screen as shown below.
This is the final screen. Just accept the default "<Yes>" to accept all the change you have made as shown below.
Now the kernel build will continue on and will run for a relatively long time such as 1 hour or more. It can take quite a long time for the kernel to build. You can go to sleep if it's late and check it in the morning, or go have a few beers and some tacos and come back in 90 minutes or so to see if the kernel build has finished.
When the kernel build finishes, these are the last few lines where you get back to the prompt as shown below.
flock -w 60 /home/oracle/Downloads/linux-3.19.0/debian/.LOCK dh_gencontrol -plinux-cloud-tools-3.19.0-26-scst
dh_md5sums -plinux-cloud-tools-3.19.0-26-scst
dh_builddeb -plinux-cloud-tools-3.19.0-26-scst
dpkg-deb: building package `linux-cloud-tools-3.19.0-26-scst' in `../linux-cloud-tools-3.19.0-26-scst_3.19.0-26.28_amd64.deb'.
oracle@W521:~/Downloads/scst-files$
Once the kernel build has finished as shown above, you need to continue with the next script to be run as shown below. The first few output lines of the next script create-scst-2a.sh is shown below.
oracle@W521:~/Downloads/scst-files$ ./create-scst-2a.sh
-rw-r--r-- 1 root root 9499358 Sep 3 23:09 linux-headers-3.19.0-26_3.19.0-26.28_all.deb
-rw-r--r-- 1 root root 873666 Sep 4 00:03 linux-headers-3.19.0-26-scst_3.19.0-26.28_amd64.deb
-rw-r--r-- 1 root root 55869032 Sep 4 00:03 linux-image-3.19.0-26-scst_3.19.0-26.28_amd64.deb
[sudo] password for oracle:
Next run create-scst-2b.sh script. This builds scst modules.
Now it's time to run create-scst-3.sh script.
Now run create-scst-4a.sh. During create-scst-4a.sh you will get the prompt shown below, and you should answer "y" to this prompt.
Performing this action may result in lost or corrupt data, are you sure you wish to continue (y/[n]) ? y
When the create-scst-4a.sh script has finished, it will wait for 30 seconds, and will present the following output (this is the expected output which shows your SCST SAN has successfully been configured and is ready with 3 LUNs for Oracle Grid Infrastructure install.
======================================================
Verify that SCST SAN is fully configured and ready
Sleeping for 30 seconds...
======================================================
Collecting current configuration: done.
Driver: iscsi
Target: iqn.2015-08.org.vmem:w520.san.asm.luns
Driver/target 'iscsi/iqn.2015-08.org.vmem:w520.san.asm.luns' has no associated LUNs.
Group: lxc1
Assigned LUNs:
LUN Device
--------------------
0 asm_systemdg_00
1 asm_systemdg_01
2 asm_systemdg_02
Assigned Initiators:
Initiator
-------------------------------------
iqn.2014-09.org.vmem1:oracle.asm.luns
All done.
Next run create-scst-4b.sh script. The expected output is shown below. Notice that the LUNs are discovered on many networks, but that we only login to SCST LUNs on the two OpenvSwitch networks, sw2 and sw3, that are dedicated as LUN storage networks. Notice also that the last step does an "ls -l /dev/mapper" and shows that we now have 3 multipath LUNs that are intended for the SYSTEMDG Oracle ASM diskgroup of the 12c ASM Flex Cluster Grid Infrastructure install.
oracle@W521:~/Downloads/scst-files$ ./create-scst-4b.sh
10.207.40.1:3260,1 iqn.2015-08.org.vmem:w520.san.asm.luns
192.168.1.7:3260,1 iqn.2015-08.org.vmem:w520.san.asm.luns
192.213.39.1:3260,1 iqn.2015-08.org.vmem:w520.san.asm.luns
10.0.3.1:3260,1 iqn.2015-08.org.vmem:w520.san.asm.luns
10.207.39.1:3260,1 iqn.2015-08.org.vmem:w520.san.asm.luns
192.210.39.1:3260,1 iqn.2015-08.org.vmem:w520.san.asm.luns
172.221.40.1:3260,1 iqn.2015-08.org.vmem:w520.san.asm.luns
10.207.41.1:3260,1 iqn.2015-08.org.vmem:w520.san.asm.luns
172.220.40.1:3260,1 iqn.2015-08.org.vmem:w520.san.asm.luns
10.207.29.1:3260,1 iqn.2015-08.org.vmem:w520.san.asm.luns
192.211.39.1:3260,1 iqn.2015-08.org.vmem:w520.san.asm.luns
192.212.39.1:3260,1 iqn.2015-08.org.vmem:w520.san.asm.luns
192.168.122.1:3260,1 iqn.2015-08.org.vmem:w520.san.asm.luns
10.207.41.1:3260,1 iqn.2015-08.org.vmem:w520.san.asm.luns
192.168.1.7:3260,1 iqn.2015-08.org.vmem:w520.san.asm.luns
192.213.39.1:3260,1 iqn.2015-08.org.vmem:w520.san.asm.luns
10.0.3.1:3260,1 iqn.2015-08.org.vmem:w520.san.asm.luns
10.207.39.1:3260,1 iqn.2015-08.org.vmem:w520.san.asm.luns
192.210.39.1:3260,1 iqn.2015-08.org.vmem:w520.san.asm.luns
10.207.40.1:3260,1 iqn.2015-08.org.vmem:w520.san.asm.luns
172.221.40.1:3260,1 iqn.2015-08.org.vmem:w520.san.asm.luns
172.220.40.1:3260,1 iqn.2015-08.org.vmem:w520.san.asm.luns
10.207.29.1:3260,1 iqn.2015-08.org.vmem:w520.san.asm.luns
192.211.39.1:3260,1 iqn.2015-08.org.vmem:w520.san.asm.luns
192.212.39.1:3260,1 iqn.2015-08.org.vmem:w520.san.asm.luns
192.168.122.1:3260,1 iqn.2015-08.org.vmem:w520.san.asm.luns
Logging in to [iface: default, target: iqn.2015-08.org.vmem:w520.san.asm.luns, portal: 10.207.41.1,3260] (multiple)
Login to [iface: default, target: iqn.2015-08.org.vmem:w520.san.asm.luns, portal: 10.207.41.1,3260] successful.
Logging in to [iface: default, target: iqn.2015-08.org.vmem:w520.san.asm.luns, portal: 10.207.40.1,3260] (multiple)
Login to [iface: default, target: iqn.2015-08.org.vmem:w520.san.asm.luns, portal: 10.207.40.1,3260] successful.
total 0
crw------- 1 root root 10, 236 Sep 4 00:12 control
brw-rw---- 1 root disk 252, 2 Sep 4 00:24 mpath2
brw-rw---- 1 root disk 252, 0 Sep 4 00:24 mpath1
brw-rw---- 1 root disk 252, 1 Sep 4 00:24 mpath0
oracle@W521:~/Downloads/scst-files$
,
Now run create-scst-5a.sh script. This script if it works as designed should auto-create the /etc/multipath.conf file you will need to configure your multipath LUNs in /dev/mapper with friendly names and with other settings designed for the Oracle deployment. Some details of what create-scst-5a.sh does is shown below. The multipath.conf file that is autocreated for my Lenovo W520 mobile workstation is shown below. You will likely be on different laptop or desktop and different disk and your multipath.conf will be different in the "blacklist" and "devices" stanzas, but the script has been designed to configure the multipath.conf for your system. If you have issues, please email me at "gilstanden@hotmail.com" so that I can improve the script to work for all types of combos of machine and disks.
oracle@W521:~/Downloads/scst-files$ ./create-scst-5a.sh
oracle@W521:~/Downloads/scst-files$ more multipath.conf
blacklist {
devnode "sd[a]$"
wwid "350004cf20fba5cfd"
device {
vendor "ATA"
product "ST2000LM003HN-M"
revision "0001"
}
}
defaults {
user_friendly_names yes
}
devices {
device {
vendor "SCST_FIO"
product "asm*"
revision "310"
path_grouping_policy group_by_serial
getuid_callout "/lib/udev/scsi_id --whitelisted --device=/dev/%n"
hardware_handler "0"
features "1 queue_if_no_path"
fast_io_fail_tmo 5
dev_loss_tmo 30
failback immediate
rr_weight uniform
no_path_retry fail
path_checker tur
rr_min_io 4
path_selector "round-robin 0"
}
}
multipaths {
multipath {
wwid 26537623737636234
alias asm_systemdg_00
}
multipath {
wwid 26634313565346333
alias asm_systemdg_01
}
multipath {
wwid 26330663234633561
alias asm_systemdg_02
}
}
oracle@W521:~/Downloads/scst-files$
Now run create-scst-5b.sh script. See the output of this script below for details. Note that it has assigned the user-friendly names to the mpathX lunx in /dev/mapper. If the multipath.conf was created correctly for your system, you will also see the LUN "friendly names" in /dev/mapper also. Notice highlighed in bold the friendly names and the true multiapath (2 paths) that have been achieved for this SCST SAN for Oracle.
oracle@W521:~/Downloads/scst-files$ ./create-scst-5b.sh
Logging out of session [sid: 1, target: iqn.2015-08.org.vmem:w520.san.asm.luns, portal: 10.207.41.1,3260]
Logout of [sid: 1, target: iqn.2015-08.org.vmem:w520.san.asm.luns, portal: 10.207.41.1,3260] successful.
Logging out of session [sid: 2, target: iqn.2015-08.org.vmem:w520.san.asm.luns, portal: 10.207.40.1,3260]
Logout of [sid: 2, target: iqn.2015-08.org.vmem:w520.san.asm.luns, portal: 10.207.40.1,3260] successful.
===========================================================================
Login to SCST target...
Verify that login to SCST target is successful...
===========================================================================
Logging in to [iface: default, target: iqn.2015-08.org.vmem:w520.san.asm.luns, portal: 10.207.41.1,3260] (multiple)
Login to [iface: default, target: iqn.2015-08.org.vmem:w520.san.asm.luns, portal: 10.207.41.1,3260] successful.
Logging in to [iface: default, target: iqn.2015-08.org.vmem:w520.san.asm.luns, portal: 10.207.40.1,3260] (multiple)
Login to [iface: default, target: iqn.2015-08.org.vmem:w520.san.asm.luns, portal: 10.207.40.1,3260] successful.
===========================================================================
Verify /dev/mapper output...should be using multipath friendly names...
Sleeping 5 seconds...
===========================================================================
total 0
crw------- 1 root root 10, 236 Sep 4 00:12 control
brw-rw---- 1 root disk 252, 1 Sep 4 00:32 asm_systemdg_02
brw-rw---- 1 root disk 252, 0 Sep 4 00:32 asm_systemdg_01
brw-rw---- 1 root disk 252, 2 Sep 4 00:32 asm_systemdg_00
===========================================================================
Verify multipath -ll -v2 output ...
Sleeping 5 seconds...
===========================================================================
asm_systemdg_02 (26330663234633561) dm-1 SCST_FIO,asm_systemdg_02
size=10G features='0' hwhandler='0' wp=rw
`-+- policy='round-robin 0' prio=1 status=active
|- 9:0:0:2 sdg 8:96 active ready running
`- 8:0:0:2 sdd 8:48 active ready running
asm_systemdg_01 (26634313565346333) dm-0 SCST_FIO,asm_systemdg_01
size=10G features='0' hwhandler='0' wp=rw
`-+- policy='round-robin 0' prio=1 status=active
|- 9:0:0:1 sdf 8:80 active ready running
`- 8:0:0:1 sdc 8:32 active ready running
asm_systemdg_00 (26537623737636234) dm-2 SCST_FIO,asm_systemdg_00
size=10G features='0' hwhandler='0' wp=rw
`-+- policy='round-robin 0' prio=1 status=active
|- 9:0:0:0 sde 8:64 active ready running
`- 8:0:0:0 sdb 8:16 active ready running
oracle@W521:~/Downloads/scst-files$
This concludes the series of "create-scst-*.sh" scripts. If all has run as intended, the SCST SAN is up and running and ready. You can run some checks on both the LXC Containers and the SCST SAN as shown below.
oracle@W521:~/Downloads/scst-files$ sudo lxc-ls -f
NAME STATE IPV4 IPV6 GROUPS AUTOSTART
------------------------------------------------
lxcora00 STOPPED - - - NO
lxcora01 STOPPED - - - NO
lxcora02 STOPPED - - - NO
lxcora03 STOPPED - - - NO
lxcora04 STOPPED - - - NO
lxcora05 STOPPED - - - NO
lxcora06 STOPPED - - - NO
oracle@W521:~/Downloads/scst-files$ sudo scstadmin -list_group
Collecting current configuration: done.
Driver: iscsi
Target: iqn.2015-08.org.vmem:w520.san.asm.luns
Driver/target 'iscsi/iqn.2015-08.org.vmem:w520.san.asm.luns' has no associated LUNs.
Group: lxc1
Assigned LUNs:
LUN Device
--------------------
0 asm_systemdg_00
1 asm_systemdg_01
2 asm_systemdg_02
Assigned Initiators:
Initiator
-------------------------------------
iqn.2014-09.org.vmem1:oracle.asm.luns
All done.
oracle@W521:~/Downloads/scst-files$
Notice that when the Ubuntu host was last rebooted it booted into the "-scst" kernel that we built as shown below.
oracle@W521:~/Downloads/scst-files$ uname -a
Linux W521 3.19.0-26-scst #28 SMP Thu Sep 3 23:08:42 EDT 2015 x86_64 x86_64 x86_64 GNU/Linux
oracle@W521:~/Downloads/scst-files$
At this point, you might as well kick off the download of the Oracle 12c Grid Infrastructure software from http://www.oracle.com/downloads and then also while it is downloading you can start up the LXC RAC node containers as shown below.
Notice that it's important to be sure that on all containers, that the "10.207.39.x" public interface comes up and is the first interface listed. It sometimes takes a minute or so for the public interface IP to show up on the output of "sudo lxc-ls -f" so give it at least 30 seconds or so to show up.
Make sure that all containers have got their public interfaces up before starting any Oracle software installation by using "sudo lxc-ls -f" to check.
oracle@W521:~/Downloads/scst-files$ sudo lxc-start -n lxcora01
oracle@W521:~/Downloads/scst-files$ sudo lxc-ls -f
NAME STATE IPV4 IPV6 GROUPS AUTOSTART
------------------------------------------------------------------------------------------------------------------------------------
lxcora00 STOPPED - - - NO
lxcora01 RUNNING 172.220.40.10, 172.221.40.10, 192.210.39.10, 192.211.39.10, 192.212.39.10, 192.213.39.10 - - NO
lxcora02 STOPPED - - - NO
lxcora03 STOPPED - - - NO
lxcora04 STOPPED - - - NO
lxcora05 STOPPED - - - NO
lxcora06 STOPPED - - - NO
oracle@W521:~/Downloads/scst-files$ sudo lxc-ls -f
NAME STATE IPV4 IPV6 GROUPS AUTOSTART
--------------------------------------------------------------------------------------------------------------------------------------------------
lxcora00 STOPPED - - - NO
lxcora01 RUNNING 10.207.39.10, 172.220.40.10, 172.221.40.10, 192.210.39.10, 192.211.39.10, 192.212.39.10, 192.213.39.10 - - NO
lxcora02 STOPPED - - - NO
lxcora03 STOPPED - - - NO
lxcora04 STOPPED - - - NO
lxcora05 STOPPED - - - NO
lxcora06 STOPPED - - - NO
oracle@W521:~/Downloads/scst-files$ sudo lxc-start -n lxcora02
oracle@W521:~/Downloads/scst-files$ sudo lxc-start -n lxcora03
oracle@W521:~/Downloads/scst-files$ sudo lxc-start -n lxcora04
oracle@W521:~/Downloads/scst-files$ sudo lxc-start -n lxcora05
oracle@W521:~/Downloads/scst-files$ sudo lxc-start -n lxcora06
oracle@W521:~/Downloads/scst-files$ sudo lxc-ls -f
NAME STATE IPV4 IPV6 GROUPS AUTOSTART
--------------------------------------------------------------------------------------------------------------------------------------------------
lxcora00 STOPPED - - - NO
lxcora01 RUNNING 10.207.39.10, 172.220.40.10, 172.221.40.10, 192.210.39.10, 192.211.39.10, 192.212.39.10, 192.213.39.10 - - NO
lxcora02 RUNNING 10.207.39.11, 172.220.40.11, 172.221.40.11, 192.210.39.11, 192.211.39.11, 192.212.39.11, 192.213.39.11 - - NO
lxcora03 RUNNING 10.207.39.12, 172.220.40.12, 172.221.40.12, 192.210.39.12, 192.211.39.12, 192.212.39.12, 192.213.39.12 - - NO
lxcora04 RUNNING 10.207.39.13, 172.220.40.13, 172.221.40.13, 192.210.39.13, 192.211.39.13, 192.212.39.13, 192.213.39.13 - - NO
lxcora05 RUNNING 10.207.39.14, 172.220.40.14, 172.221.40.14, 192.210.39.14, 192.211.39.14, 192.212.39.14, 192.213.39.14 - - NO
lxcora06 RUNNING 172.220.40.15, 172.221.40.15, 192.210.39.15, 192.211.39.15, 192.212.39.15, 192.213.39.15 - - NO
oracle@W521:~/Downloads/scst-files$ sudo lxc-ls -f
NAME STATE IPV4 IPV6 GROUPS AUTOSTART
--------------------------------------------------------------------------------------------------------------------------------------------------
lxcora00 STOPPED - - - NO
lxcora01 RUNNING 10.207.39.10, 172.220.40.10, 172.221.40.10, 192.210.39.10, 192.211.39.10, 192.212.39.10, 192.213.39.10 - - NO
lxcora02 RUNNING 10.207.39.11, 172.220.40.11, 172.221.40.11, 192.210.39.11, 192.211.39.11, 192.212.39.11, 192.213.39.11 - - NO
lxcora03 RUNNING 10.207.39.12, 172.220.40.12, 172.221.40.12, 192.210.39.12, 192.211.39.12, 192.212.39.12, 192.213.39.12 - - NO
lxcora04 RUNNING 10.207.39.13, 172.220.40.13, 172.221.40.13, 192.210.39.13, 192.211.39.13, 192.212.39.13, 192.213.39.13 - - NO
lxcora05 RUNNING 10.207.39.14, 172.220.40.14, 172.221.40.14, 192.210.39.14, 192.211.39.14, 192.212.39.14, 192.213.39.14 - - NO
lxcora06 RUNNING 172.220.40.15, 172.221.40.15, 192.210.39.15, 192.211.39.15, 192.212.39.15, 192.213.39.15 - - NO
oracle@W521:~/Downloads/scst-files$ sudo lxc-ls -f
NAME STATE IPV4 IPV6 GROUPS AUTOSTART
--------------------------------------------------------------------------------------------------------------------------------------------------
lxcora00 STOPPED - - - NO
lxcora01 RUNNING 10.207.39.10, 172.220.40.10, 172.221.40.10, 192.210.39.10, 192.211.39.10, 192.212.39.10, 192.213.39.10 - - NO
lxcora02 RUNNING 10.207.39.11, 172.220.40.11, 172.221.40.11, 192.210.39.11, 192.211.39.11, 192.212.39.11, 192.213.39.11 - - NO
lxcora03 RUNNING 10.207.39.12, 172.220.40.12, 172.221.40.12, 192.210.39.12, 192.211.39.12, 192.212.39.12, 192.213.39.12 - - NO
lxcora04 RUNNING 10.207.39.13, 172.220.40.13, 172.221.40.13, 192.210.39.13, 192.211.39.13, 192.212.39.13, 192.213.39.13 - - NO
lxcora05 RUNNING 10.207.39.14, 172.220.40.14, 172.221.40.14, 192.210.39.14, 192.211.39.14, 192.212.39.14, 192.213.39.14 - - NO
lxcora06 RUNNING 172.220.40.15, 172.221.40.15, 192.210.39.15, 192.211.39.15, 192.212.39.15, 192.213.39.15 - - NO
oracle@W521:~/Downloads/scst-files$ sudo lxc-ls -f
NAME STATE IPV4 IPV6 GROUPS AUTOSTART
--------------------------------------------------------------------------------------------------------------------------------------------------
lxcora00 STOPPED - - - NO
lxcora01 RUNNING 10.207.39.10, 172.220.40.10, 172.221.40.10, 192.210.39.10, 192.211.39.10, 192.212.39.10, 192.213.39.10 - - NO
lxcora02 RUNNING 10.207.39.11, 172.220.40.11, 172.221.40.11, 192.210.39.11, 192.211.39.11, 192.212.39.11, 192.213.39.11 - - NO
lxcora03 RUNNING 10.207.39.12, 172.220.40.12, 172.221.40.12, 192.210.39.12, 192.211.39.12, 192.212.39.12, 192.213.39.12 - - NO
lxcora04 RUNNING 10.207.39.13, 172.220.40.13, 172.221.40.13, 192.210.39.13, 192.211.39.13, 192.212.39.13, 192.213.39.13 - - NO
lxcora05 RUNNING 10.207.39.14, 172.220.40.14, 172.221.40.14, 192.210.39.14, 192.211.39.14, 192.212.39.14, 192.213.39.14 - - NO
lxcora06 RUNNING 10.207.39.15, 172.220.40.15, 172.221.40.15, 192.210.39.15, 192.211.39.15, 192.212.39.15, 192.213.39.15 - - NO
oracle@W521:~/Downloads/scst-files$
Once the Oracle Grid Infrastructure 12c (12.1.0.2.0) install media is downloaded, you can scp it to the lxcora01 LXC containers as the "grid" user as shown below. After scp'g the install media into the lxcora01 LXC containers, ssh into the lxcora01 container as "grid" user using the "-Y" and "-C" switches to enable x-windows for the Oracle installer step, make sure the install media is owned by "grid:oinstall" and unzip, first the "1of2" file and then the "2of2" file.
You can run "xclock" from the command prompt to verify that X is working.
Logged into container as "grid" linux user, unzip the install media as shown in the next following section.
oracle@W521:~/Downloads$ ls -lrt linuxam*
-rw-rw-r-- 1 oracle oracle 646972897 Sep 4 00:58 linuxamd64_12102_grid_2of2.zip
-rw-rw-r-- 1 oracle oracle 1747043545 Sep 4 01:04 linuxamd64_12102_grid_1of2.zip
oracle@W521:~/Downloads$ scp linuxamd64_12102_grid_*.zip grid@lxcora01:~/.
grid@lxcora01's password:
linuxamd64_12102_grid_1of2.zip 100% 1666MB 333.2MB/s 00:05
linuxamd64_12102_grid_2of2.zip 100% 617MB 308.5MB/s 00:02
oracle@W521:~/Downloads$ ssh -Y -C grid@lxcora01
grid@lxcora01's password:
/usr/bin/xauth: creating new authority file /home/grid/.Xauthority
[grid@lxcora01 ~]$ ls -lrt
total 2337920
drwxr-xr-x 3 grid oinstall 4096 Sep 3 22:27 grid
-rwxr-xr-x 1 grid oinstall 28 Sep 3 22:27 edit_bashrc
-rw-r--r-- 1 grid oinstall 1747043545 Sep 4 01:05 linuxamd64_12102_grid_1of2.zip
-rw-r--r-- 1 grid oinstall 646972897 Sep 4 01:05 linuxamd64_12102_grid_2of2.zip
[grid@lxcora01 ~]$
When you unzip the first Oracle install media file, you can say "y" to overwrite the cvuqdisk rpm file as shown below. This occurs because the cvuqdisk rpm was installed much earlier as a one-off step so that the cloned containers would already have this package installed at clone time.
[grid@lxcora01 ~]$ unzip linuxamd64_12102_grid_1of2.zip
Archive: linuxamd64_12102_grid_1of2.zip
replace grid/rpm/cvuqdisk-1.0.9-1.rpm? [y]es, [n]o, [A]ll, [N]one, [r]ename: y
Once both of the Oracle Grid install media files have been unzipped, you can cd into the grid directory (/home/grid/grid) to begin the GRID install with "./runInstaller" as shown below.
[grid@lxcora01 ~]$ ls -lrt
total 2337928
-rwxr-xr-x 1 grid oinstall 28 Sep 3 22:27 edit_bashrc
-rw-r--r-- 1 grid oinstall 1747043545 Sep 4 01:05 linuxamd64_12102_grid_1of2.zip
-rw-r--r-- 1 grid oinstall 646972897 Sep 4 01:05 linuxamd64_12102_grid_2of2.zip
drwxr-xr-x 7 grid oinstall 4096 Sep 4 01:11 grid
[grid@lxcora01 ~]$ cd grid
[grid@lxcora01 grid]$ ./runInstaller
The ntp check must PASS in the installer. If it fails, you can open a terminal and check to be sure all ntpd processes are running and verify that the command:
ntpq -p
has correct output.
When evaluating the detail of the error, this part of the error details screen helps to point out which nodes have issue.
PRVF-5408 : NTP Time Server "198.55.111.5" is common only to the following nodes "lxcora06,lxcora05,lxcora03,lxcora02,lxcora01"
The above indicates to correct the NTP issues on node lxcora04.
oracle0@W520:~$ ssh root@lxcora4
The authenticity of host 'lxcora4 (10.207.39.14)' can't be established.
RSA key fingerprint is 22:8d:f3:73:49:4c:82:a8:29:a0:00:e1:63:c7:64:89.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'lxcora4,10.207.39.14' (RSA) to the list of known hosts.
[root@lxcora4 ~]# ntpq -p
ntpq: read: Connection refused <-- Incorrect output
[root@lxcora4 ~]# ntpd <-- This is how you start NTP on the containers (note: "service ntpd start" will NOT work)
[root@lxcora4 ~]# ntpq -p <-- Expected correct output returned this time
remote refid st t when poll reach delay offset jitter
==============================================================================
vmem1.vmem.org 198.55.111.5 3 u 3 64 1 0.555 0.076 0.000
[root@lxcora4 ~]
has valid output, and then retry the checks again for the installer.
oracle@W521:~$ ssh root@lxcora01
Last login: Fri Sep 4 01:16:09 2015 from 10.207.39.1
[root@lxcora01 ~]# ntpq -p
remote refid st t when poll reach delay offset jitter
==============================================================================
*vmem1.vmem.org 132.163.4.102 2 u 49 64 377 0.077 0.012 0.053
[root@lxcora01 ~]# ssh lxcora02
The authenticity of host 'lxcora02 (10.207.39.11)' can't be established.
RSA key fingerprint is a9:85:d2:96:d8:38:56:fa:ba:0b:64:be:f0:a7:59:c4.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'lxcora02,10.207.39.11' (RSA) to the list of known hosts.
root@lxcora02's password:
[root@lxcora02 ~]# ntpq -p
ntpq: read: Connection refused
[root@lxcora02 ~]# ntpd
[root@lxcora02 ~]# ntpq -p
remote refid st t when poll reach delay offset jitter
==============================================================================
vmem1.vmem.org 132.163.4.102 2 u 2 64 1 0.693 0.097 0.000
[root@lxcora02 ~]# ssh lxcora03
The authenticity of host 'lxcora03 (10.207.39.12)' can't be established.
RSA key fingerprint is a9:85:d2:96:d8:38:56:fa:ba:0b:64:be:f0:a7:59:c4.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'lxcora03,10.207.39.12' (RSA) to the list of known hosts.
root@lxcora03's password:
[root@lxcora03 ~]# ntpq -p
remote refid st t when poll reach delay offset jitter
==============================================================================
*vmem1.vmem.org 132.163.4.102 2 u 38 64 377 0.209 0.046 0.054
[root@lxcora03 ~]# ssh lxcora04
The authenticity of host 'lxcora04 (10.207.39.13)' can't be established.
RSA key fingerprint is a9:85:d2:96:d8:38:56:fa:ba:0b:64:be:f0:a7:59:c4.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'lxcora04,10.207.39.13' (RSA) to the list of known hosts.
root@lxcora04's password:
[root@lxcora04 ~]# ntpq -p
remote refid st t when poll reach delay offset jitter
==============================================================================
*vmem1.vmem.org 132.163.4.102 2 u 60 64 377 0.179 0.029 0.074
[root@lxcora04 ~]# ssh lxcora05
The authenticity of host 'lxcora05 (10.207.39.14)' can't be established.
RSA key fingerprint is a9:85:d2:96:d8:38:56:fa:ba:0b:64:be:f0:a7:59:c4.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'lxcora05,10.207.39.14' (RSA) to the list of known hosts.
root@lxcora05's password:
[root@lxcora05 ~]# ntpq -p
remote refid st t when poll reach delay offset jitter
==============================================================================
*vmem1.vmem.org 132.163.4.102 2 u 62 128 377 0.389 0.052 0.045
[root@lxcora05 ~]# ssh lxcora06
The authenticity of host 'lxcora06 (10.207.39.15)' can't be established.
RSA key fingerprint is a9:85:d2:96:d8:38:56:fa:ba:0b:64:be:f0:a7:59:c4.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'lxcora06,10.207.39.15' (RSA) to the list of known hosts.
root@lxcora06's password:
[root@lxcora06 ~]# ntpq -p
remote refid st t when poll reach delay offset jitter
==============================================================================
*vmem1.vmem.org 132.163.4.102 2 u 56 64 377 0.463 0.001 0.090
[root@lxcora06 ~]#
Once the Oracle Grid Infrastructure (GI) installer kicks off, you can monitor the installer actions using "tail -f" on the installer log as shown below. The exact name of your install log file will differ of course, but it will be similar name in format.
[root@lxcora01 logs]# pwd
/u00/app/12.1.0/oraInventory/logs
[root@lxcora01 logs]# ls -lrt
total 1356
-rw------- 1 grid oinstall 124 Sep 4 01:43 oraInstall2015-09-04_01-15-18AM.out
-rw------- 1 grid oinstall 0 Sep 4 01:43 oraInstall2015-09-04_01-15-18AM.err
-rw-r----- 1 grid oinstall 314 Sep 4 01:43 silentInstall2015-09-04_01-15-18AM.log
-rw-r--r-- 1 grid oinstall 1378661 Sep 4 01:43 installActions2015-09-04_01-15-18AM.log
[root@lxcora01 logs]# tail -f installActions2015-09-04_01-15-18AM.log
Note that some of the screenshots near the end of the GI install show the Ubuntu OS screen and the output of the above "tail -f" on the install log. Below are the install screens and options to choose for this installation.
The values for the screen below, such as "lxc1" and "lxc1-scan.gns1.vmem.org" should be typed in exactly as shonw below. Be sure to include "." (period symbols) where shown (they are small here and hard to see, examine the screenshot carefully!). All the values shown must be entered exactly as shown because the install bundle has configured bind9 (aka "named") to resolve these values, so simply just use the exact value below.
Below the addtional lxcora0x nodes need to be added to the list of nodes as shown below.
NOTE: The v13 bundles starts container numbering at "lxcora10" so the below screenshot is OUT OF DATE and needs update. The naming of containers starts at lxcora10 to workaround an Oracle bug related to the number of characters in the node names.
Pressing the "Add" button will bring up this dialog as shown below. Add each of the additional nodes in the same way. The Node Role is always "HUB" and the "Virtual Hostname" is always "AUTO" so just accept the defaults for each pop-up add cluster node information dialog box.
The final list of nodes on the screen is shown below (for 6 nodes). If you chose fewer nodes then you will of course add fewer nodes here.
The below screen looks almost identical to the previous one, but note that the SSH connectivity lower part of screen has been activiated by clicking on the "SSH connectivity" button. Enter "grid" as the password and then click on "Setup" to setup no-password ssh for "grid" user across all LXC nodes.
Note, if for any reason passowrdless SSH setup fails, just try the tool again. Click "Next" to go to the next screen.
Now that passwordless SSH is setup, the next screen is the interface setup. Use the dropdowns to allocate the interfaces exactly as shown below. The Private is a 4-bond HAIP for the interconnect, and the two ASM interfaces are for the ASM Flex Cluster network. These are all OpenvSwitch networks. All the networking for this RAC is OpenvSwitch.
The below screen will not show the "Disk Path" values until you change the "Disk Discovery Path" as shown next screen following below by clicking on "Change Discovery Path" and changing it to "/dev/mapper/asm*" as shown below.
Set a password for the accounts as shown below.
Select "Do not use IPMI" option.
Do not register with EM Cloud Control unless you are already running it.
Accept the defaults for the groups as shown below and click next.
Set the "Oracle base" and the "Software location" exactly as shown below.
Accept the default Inventory Directory exactly as shown below.
Type in the password which is "root" for the root user so that the installation software can automatically run the root installation scripts automatically. I have tested this option and it works very reliably. If you monitor the install log with the "tail -f" that was shown before the screenshots section, you can monitor the progress of the root.sh scripts running.
The Oracle pre-install Cluster Verification Utility runs as shown below.
The "Network Time Protocol" check should SUCCEED (should be no warnings for NTP). If you get an NTP warning as shown below, it should be addressed and the cluster verification step rerun.
You can click on the "(more details)" link and get the popup dialog with more details on the Cluster Verification check details for that check, as shown below.
After checking all the LXC containers, one container was found to have NTP not running, NTP was started on that LXC containers, the "ntpq -p" command was run on that LXC container, and then the verification check was rerun using the "Check Again" button. This time there was no error for NTP, which is the desired result.
The check warnings and failures below are the ONLY warnings and failures that should occur. The "OS kernel parameter" warnings for {rmem_default, rmem_max, wmem_default, and wmem_max} can be safely ignored because we have set these values at the host level (in the Ubuntu sysctl.conf file) and since all LXC containers share the same kernel with the host, these are actually set where they need to be. The only reason that check fails is because the check itself does not know how to check the host kernel values.
The Device Checks for ASM is a known error about not being able to validate the path on all nodes and can be safely ignored. The same is true of the resolv.conf integrity check, the DHCP check, and the /boot mount check, all of which can be safely ignored.
Therefore, next check the "Ignore All" box as long as only the below warnings and failed messages looks just like the screenshot below with the exact same warnings, and click on "Next" after checking the "Ignore All" box.
Check the "Ignore All" button in the upper-right-corner of the screen as shown below. Then click "Next" to continue.
Click on "Yes" to continue with the install and confirm that the warnings are being ignored.
The install summary screen is displayed as shown below.
Save the response file so that a "silent" install could be done on another subsequent install. Click on "Save" to save the response file.
The install begins and the progress bar indicates progress of the install as shown below.
When the installer gets to running the root scripts, it asks for confirmation to run them. click on "Yes' to run the root scripts.
The root scripts can be monitored using the "tail -f" as previously described and as shown below.
Sometimes the Oracle Cluster Verification may fail. The Cluster Verification Utillity can be re-run by clicking on "Retry". In this case as seen in the remaining screenshots, the second time Oracle Cluster Veriication Utility ran, it was successful. However, even if CVU is not successful, it usually does not affect the install, and you would find the install was totally successful. So if after a couple of retries of the CVU it still shows some checks failed, then just ignore and move on to the next screen. Note the CVU check can take a few minutes.
The install is successful. Close the installer.
Here is the output below of how the initially installed Oracle 12c ASM Flex Cluster will look immediately after the runInstaller session completes. Notice how only 3 ASM instances and associated services are up. This is because the default cardinality of ASM in a Flex Cluster is "3". This can be changed so that ALL services are up using the command:
[root@lxcora01 ] srvctl modify asm -count 6
for example if as in this case the ASM Flex Cluster has 6 nodes.
So, as shown below, running the command to modify ASM and re-run "crsctl stat res -t" gives the following output as shown below. Now as can be seen below, ALL installed and configured services have status ONLINE ONLINE including ALL of the 6 ASM instances and associated ASMNET1LSNR services.
[root@lxcora01 logs]# which srvctl
/u00/app/grid/product/12.1.0/grid/bin/srvctl
[root@lxcora01 logs]# srvctl modify asm -count 6
[root@lxcora01 logs]# crsctl stat res -t
--------------------------------------------------------------------------------
Name Target State Server State details
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.ASMNET1LSNR_ASM.lsnr
ONLINE ONLINE lxcora01 STABLE
ONLINE ONLINE lxcora02 STABLE
ONLINE ONLINE lxcora03 STABLE
ONLINE ONLINE lxcora04 STABLE
ONLINE ONLINE lxcora05 STABLE
ONLINE ONLINE lxcora06 STABLE
ora.LISTENER.lsnr
ONLINE ONLINE lxcora01 STABLE
ONLINE ONLINE lxcora02 STABLE
ONLINE ONLINE lxcora03 STABLE
ONLINE ONLINE lxcora04 STABLE
ONLINE ONLINE lxcora05 STABLE
ONLINE ONLINE lxcora06 STABLE
ora.SYSTEMDG.dg
ONLINE ONLINE lxcora01 STABLE
ONLINE ONLINE lxcora02 STABLE
ONLINE ONLINE lxcora03 STABLE
ONLINE ONLINE lxcora04 STABLE
ONLINE ONLINE lxcora05 STABLE
ONLINE ONLINE lxcora06 STABLE
ora.net1.network
ONLINE ONLINE lxcora01 STABLE
ONLINE ONLINE lxcora02 STABLE
ONLINE ONLINE lxcora03 STABLE
ONLINE ONLINE lxcora04 STABLE
ONLINE ONLINE lxcora05 STABLE
ONLINE ONLINE lxcora06 STABLE
ora.ons
ONLINE ONLINE lxcora01 STABLE
ONLINE ONLINE lxcora02 STABLE
ONLINE ONLINE lxcora03 STABLE
ONLINE ONLINE lxcora04 STABLE
ONLINE ONLINE lxcora05 STABLE
ONLINE ONLINE lxcora06 STABLE
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.LISTENER_SCAN1.lsnr
1 ONLINE ONLINE lxcora06 STABLE
ora.LISTENER_SCAN2.lsnr
1 ONLINE ONLINE lxcora02 STABLE
ora.LISTENER_SCAN3.lsnr
1 ONLINE ONLINE lxcora03 STABLE
ora.MGMTLSNR
1 ONLINE ONLINE lxcora01 169.254.39.117 192.2
10.39.10 192.211.39.
10,STABLE
ora.asm
1 ONLINE ONLINE lxcora01 Started,STABLE
2 ONLINE ONLINE lxcora06 Started,STABLE
3 ONLINE ONLINE lxcora02 Started,STABLE
4 ONLINE ONLINE lxcora04 Started,STABLE
5 ONLINE ONLINE lxcora05 Started,STABLE
6 ONLINE ONLINE lxcora03 Started,STABLE
ora.cvu
1 ONLINE ONLINE lxcora01 STABLE
ora.gns
1 ONLINE ONLINE lxcora02 STABLE
ora.gns.vip
1 ONLINE ONLINE lxcora02 STABLE
ora.lxcora01.vip
1 ONLINE ONLINE lxcora01 STABLE
ora.lxcora02.vip
1 ONLINE ONLINE lxcora02 STABLE
ora.lxcora03.vip
1 ONLINE ONLINE lxcora03 STABLE
ora.lxcora04.vip
1 ONLINE ONLINE lxcora04 STABLE
ora.lxcora05.vip
1 ONLINE ONLINE lxcora05 STABLE
ora.lxcora06.vip
1 ONLINE ONLINE lxcora06 STABLE
ora.mgmtdb
1 ONLINE ONLINE lxcora01 Open,STABLE
ora.oc4j
1 ONLINE ONLINE lxcora01 STABLE
ora.scan1.vip
1 ONLINE ONLINE lxcora06 STABLE
ora.scan2.vip
1 ONLINE ONLINE lxcora02 STABLE
ora.scan3.vip
1 ONLINE ONLINE lxcora03 STABLE
--------------------------------------------------------------------------------
[root@lxcora01 logs]#
The "iotop" package can be installed for monitoring IO across all the instances as shown below. Once installed run "sudo iotop" to start the monitor.
oracle@W521:~/Downloads$ sudo iotop
[sudo] password for oracle:
sudo: iotop: command not found
oracle@W521:~/Downloads$ sudo apt-get install iotop
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following packages were automatically installed and are no longer required:
linux-headers-3.19.0-15 linux-headers-3.19.0-15-generic linux-image-3.19.0-15-generic linux-image-extra-3.19.0-15-generic
Use 'apt-get autoremove' to remove them.
The following NEW packages will be installed:
iotop
0 upgraded, 1 newly installed, 0 to remove and 8 not upgraded.
Need to get 23.8 kB of archives.
After this operation, 127 kB of additional disk space will be used.
Get:1 http://us.archive.ubuntu.com/ubuntu/ vivid/universe iotop amd64 0.6-1 [23.8 kB]
Fetched 23.8 kB in 1s (17.2 kB/s)
Selecting previously unselected package iotop.
(Reading database ... 236524 files and directories currently installed.)
Preparing to unpack .../archives/iotop_0.6-1_amd64.deb ...
Unpacking iotop (0.6-1) ...
Processing triggers for man-db (2.7.0.2-5) ...
Setting up iotop (0.6-1) ...
oracle@W521:~/Downloads$
You can view the OpenvSwitch at the Ubuntu host level using the command "sudo ovs-vsctl show".
To login from the Ubuntu host to the Oracle in the containers, I recommend installing Oracle Instant Client as described here. The Oracle RAC is readily accessible from the Ubuntu host over sqlplus.
To ssh into the containers, just use:
ssh root@lxcora01 (or whichever container you want to ssh to).
To stop the container:
sudo lxc-stop -n lxcora01
To start the container
sudo lxc-start -n lxcora01
To list all your containers
sudo lxc-ls -f
Note that when you reboot the desktop, you'll need to login manually to SCST on 10.207.40.1 and 10.207.41.1, the storage networks using the appropriate iscsiadm command (which can be found in the create-scst-*.sh scripts) to get the LUNs active in /dev/mapper on the Ubuntu host. You'll then need to start the LXC containers,which will take care of mapping the storage LUNs into themselves and starting up Oracle GI automatically in the usual way.
Also note that it can take up to 15 seconds of so for the DHCP provided public IP for each RAC node (the 10.207.39.x network) to come up. Be sure that IP comes up. It's best to start the containers one by one and verify each one has started up it's public IP ok using sudo lxc-ls -f before starting the next containers.
10-lxc-node containerized 12.1.0.2.0 Oracle RAC ASM Flex Cluster with GNS running on Ubuntu 15.04 64-bit desktop shown below.
oracle0@W520:~/orabuntu-lxc$ crsstat
--------------------------------------------------------------------------------
Name Target State Server State details
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.ASMNET1LSNR_ASM.lsnr
ONLINE ONLINE lxcora10 STABLE
ONLINE ONLINE lxcora11 STABLE
ONLINE ONLINE lxcora12 STABLE
ONLINE ONLINE lxcora13 STABLE
ONLINE ONLINE lxcora14 STABLE
ONLINE ONLINE lxcora15 STABLE
ONLINE ONLINE lxcora16 STABLE
ONLINE ONLINE lxcora17 STABLE
ONLINE ONLINE lxcora18 STABLE
ONLINE ONLINE lxcora19 STABLE
ora.LISTENER.lsnr
ONLINE ONLINE lxcora10 STABLE
ONLINE ONLINE lxcora11 STABLE
ONLINE ONLINE lxcora12 STABLE
ONLINE ONLINE lxcora13 STABLE
ONLINE ONLINE lxcora14 STABLE
ONLINE ONLINE lxcora15 STABLE
ONLINE ONLINE lxcora16 STABLE
ONLINE ONLINE lxcora17 STABLE
ONLINE ONLINE lxcora18 STABLE
ONLINE ONLINE lxcora19 STABLE
ora.SYSTEMDG.dg
ONLINE ONLINE lxcora10 STABLE
ONLINE ONLINE lxcora11 STABLE
OFFLINE OFFLINE lxcora12 STABLE
ONLINE ONLINE lxcora13 STABLE
OFFLINE OFFLINE lxcora14 STABLE
OFFLINE OFFLINE lxcora15 STABLE
OFFLINE OFFLINE lxcora16 STABLE
OFFLINE OFFLINE lxcora17 STABLE
OFFLINE OFFLINE lxcora18 STABLE
OFFLINE OFFLINE lxcora19 STABLE
ora.net1.network
ONLINE ONLINE lxcora10 STABLE
ONLINE ONLINE lxcora11 STABLE
ONLINE ONLINE lxcora12 STABLE
ONLINE ONLINE lxcora13 STABLE
ONLINE ONLINE lxcora14 STABLE
ONLINE ONLINE lxcora15 STABLE
ONLINE ONLINE lxcora16 STABLE
ONLINE ONLINE lxcora17 STABLE
ONLINE ONLINE lxcora18 STABLE
ONLINE ONLINE lxcora19 STABLE
ora.ons
ONLINE ONLINE lxcora10 STABLE
ONLINE ONLINE lxcora11 STABLE
ONLINE ONLINE lxcora12 STABLE
ONLINE ONLINE lxcora13 STABLE
ONLINE ONLINE lxcora14 STABLE
ONLINE ONLINE lxcora15 STABLE
ONLINE ONLINE lxcora16 STABLE
ONLINE ONLINE lxcora17 STABLE
ONLINE ONLINE lxcora18 STABLE
ONLINE ONLINE lxcora19 STABLE
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.LISTENER_SCAN1.lsnr
1 ONLINE ONLINE lxcora12 STABLE
ora.LISTENER_SCAN2.lsnr
1 ONLINE ONLINE lxcora10 STABLE
ora.LISTENER_SCAN3.lsnr
1 ONLINE ONLINE lxcora11 STABLE
ora.MGMTLSNR
1 ONLINE ONLINE lxcora10 169.254.28.57 192.21
0.39.10 192.211.39.1
0,STABLE
ora.asm
1 ONLINE ONLINE lxcora10 Started,STABLE
2 ONLINE ONLINE lxcora11 Started,STABLE
3 ONLINE ONLINE lxcora13 Started,STABLE
ora.cvu
1 ONLINE ONLINE lxcora14 STABLE
ora.gns
1 ONLINE ONLINE lxcora10 STABLE
ora.gns.vip
1 ONLINE ONLINE lxcora10 STABLE
ora.lxcora10.vip
1 ONLINE ONLINE lxcora10 STABLE
ora.lxcora11.vip
1 ONLINE ONLINE lxcora11 STABLE
ora.lxcora12.vip
1 ONLINE ONLINE lxcora12 STABLE
ora.lxcora13.vip
1 ONLINE ONLINE lxcora13 STABLE
ora.lxcora14.vip
1 ONLINE ONLINE lxcora14 STABLE
ora.lxcora15.vip
1 ONLINE ONLINE lxcora15 STABLE
ora.lxcora16.vip
1 ONLINE ONLINE lxcora16 STABLE
ora.lxcora17.vip
1 ONLINE ONLINE lxcora17 STABLE
ora.lxcora18.vip
1 ONLINE ONLINE lxcora18 STABLE
ora.lxcora19.vip
1 ONLINE ONLINE lxcora19 STABLE
ora.lxcora20.vip
1 ONLINE INTERMEDIATE lxcora12 FAILED OVER,STABLE
ora.lxcora21.vip
1 ONLINE INTERMEDIATE lxcora11 FAILED OVER,STABLE
ora.lxcora22.vip
1 ONLINE INTERMEDIATE lxcora10 FAILED OVER,STABLE
ora.mgmtdb
1 ONLINE ONLINE lxcora10 Open,STABLE
ora.oc4j
1 ONLINE ONLINE lxcora18 STABLE
ora.scan1.vip
1 ONLINE ONLINE lxcora12 STABLE
ora.scan2.vip
1 ONLINE ONLINE lxcora10 STABLE
ora.scan3.vip
1 ONLINE ONLINE lxcora11 STABLE
--------------------------------------------------------------------------------
oracle0@W520:~/orabuntu-lxc$