Ubuntu LXC OEL OVS ASM GNS RAC (12c)
Hi sportsfans well today I'm working on building a 2-node Oracle 12c Real Application Cluster (RAC) using Ubuntu 14.04.1 as my base operating system, and Oracle Enterprise Linux 6.5 LXC Containers for my two RAC nodes running ontop of the Ubuntu. You might be one of those people who would ask "Why would you do that?" There are many answers. One answer is "Because it seems like LXC for the first time I can remember makes building Oracle RAC on Ubuntu a very interesting and reachable goal." Or I could go with the Sir Edmund Hillary answer "Because it was there" (Mt. Everest). I think the main reason is that I am absolutely thrilled about the possibility of running Oracle RAC on Ubuntu Linux in Oracle Enterprise Linux 6.5 LXC containers.
Note that this site is under construction, however, the final result is already achieved - the 2-node 12c RAC on LXC containers on Ubuntu 14.04.1 base OS is built and running. Because therefore this is a work-in-progress (WIP) and is primarily a place at this point where I can record my own notes on building this, plus improvements and tweaks, etc., therefore I make no assurances as to the flow or readability of the content: there may be sudden breaks in flow where I add an update noting how something was redesigned or improved (or fixed).
I will be adding the details about how storage was presented and many many other details, so this blog post will grow for some time. Check back for additional information if the topic of building Oracle Grid Infrastructure on Ubuntu Linux in Oracle Enteprise Linux LXC containers is of interest to you.
Important Please Note as usual that this is not a supported configuration for a production deployment, neither from a Linux Containers perspective in general, nor from a supported Operating System perspective (Ubuntu Linux is not officially supported by Oracle Corporation for database or grid infrastructure) and, as usual Jim, if you or any of your IMF team are killed or captured, the State Department will disavow all knowledge of your mission. Good Luck, Jim. This tape will self-destruct in 5 seconds....
This build uses the following technologies:
LXC Linux Containers for the 2 RAC Nodes
Ubuntu 14.04.1 Desktop Edition 64-bit with all latest updates from the following repositores shown below for the Base OS
Oracle Enterprise Linux 6.5 for the two RAC-node LXC containers
OpenvSwitch for the networking
Oracle 12c Grid Infrastructure (GI) version 12.1.0.2.0
Oracle 12c Database (RDBMS) version 12.1.0.2.0
Oracle Grid Naming Service (GNS)
DHCP isc-dhcp-server
Named (bind9) DNS server
InMon sflow for performance monitoring
KVM (single guest is used for providing SCST Linux iSCSI SAN LUNs for the RAC)
Note : DHCP an BIND9 are deployed so as to update DNS automatically when new LXC containers are added
Note : All LXC Linux containers in this setup use OpenvSwitch for networking. They DO NOT use the lxcbr0 default LXC out-of-the-box networking solution.
Note : InMon sflow is designed to automatically add hosts to sflow monitoring, also through DNS
Note : Regarding Ubuntu repositores, below, note that I get the "ubuntu-lxc" trusty-tahr daily build updates, and I apply them when notified they are availalble
These are my currently active Ubuntu repositories. I always install all updates as soon as I am aware the are available. My Ubuntu base OS is not a "locked-down" deployment because it is on my laptop which I keep updated with all latest Ubuntu mainline updates (as well as the LXC daily build updates, as noted below).
Update 26 October 2014
Opted to use block storage and not NFS after all as originally planned for this 2-node RAC and went with the option of just mounting the entire /dev/mapper directory of the Ubuntu 14.04.1 host inside the two LXC containers. The configuration passed cluster verification for storage with respect the ASM LUNs, and I knew from building the previous standalone 12c GI that it works that way. This change was made because after checking the Oracle documentation found that whereas in 11gR2 GI clustering NFS was supported for OCR and VOTE, now in 12c, NFS is no longer supported for OCR and VOTE, and I wanted to build a supported version first time around, so went with the ASM option after all.
Important Note: I'm leaving all this NFS work I did with LXC in here because I am going to build a 2-node Oracle 11gR2 RAC using NFS, so this work will come in handy for that later, but for the purposes of how this 2-node ASM 12c RAC on LXC Linux containers was built, the NFS material herein can be skipped over.
End Update
Here are some notes for me. I'm going for a build that uses NFS as the shared storage for the two LXC RAC nodes. Why? Because presenting block storage that ASM could use to LXC containers has proved challenging. So far I had good luck building a single instance of 12c EE ASM using the technique of just mounting all of /dev/mapper (host) inside the LXC ASM guest. However, this won't work in general because mounting the entire /dev/mapper if you think about it gets kind of problematic when you think about doing that with multiple LXC containers. It lacks a certain elegance on top of which idk if it's even possible.
I found that LXC on Oracle Enterprise Linux 6.5 base operating system also likes NFS mounts inside the LXC container (running on a btrfs FS) so I'm going to make this my defacto standard for my first go at this (building RAC on Ubuntu Linux 14.04.1 base OS using Oracle Enterprise Linux 6.5 containers).
Ubuntu LXC Guide is found here.
The first thing I found after mounting some test NFS mounts was that upon reboot my LXC container would not boot. I got this message:
gstanden@vmem1:~$ sudo lxc-start -n lxcora01 -F
lxc-start: lsm/apparmor.c: apparmor_process_label_set: 200 No such file or directory - failed to change apparmor profile to lxc-container-default
lxc-start: sync.c: __sync_wait: 51 invalid sequence number 1. expected 4
lxc-start: start.c: __lxc_start: 1087 failed to spawn 'lxcora01'
lxc-start: cgmanager.c: cgm_remove_cgroup: 503 call to cgmanager_remove_sync failed: invalid request
lxc-start: cgmanager.c: cgm_remove_cgroup: 505 Error removing name=systemd:lxc/lxcora01-2
lxc-start: cgmanager.c: cgm_remove_cgroup: 503 call to cgmanager_remove_sync failed: invalid request
lxc-start: cgmanager.c: cgm_remove_cgroup: 505 Error removing perf_event:lxc/lxcora01-2
lxc-start: cgmanager.c: cgm_remove_cgroup: 503 call to cgmanager_remove_sync failed: invalid request
lxc-start: cgmanager.c: cgm_remove_cgroup: 505 Error removing memory:lxc/lxcora01-2
lxc-start: cgmanager.c: cgm_remove_cgroup: 503 call to cgmanager_remove_sync failed: invalid request
lxc-start: cgmanager.c: cgm_remove_cgroup: 505 Error removing hugetlb:lxc/lxcora01-2
lxc-start: cgmanager.c: cgm_remove_cgroup: 503 call to cgmanager_remove_sync failed: invalid request
lxc-start: cgmanager.c: cgm_remove_cgroup: 505 Error removing freezer:lxc/lxcora01-2
lxc-start: cgmanager.c: cgm_remove_cgroup: 503 call to cgmanager_remove_sync failed: invalid request
lxc-start: cgmanager.c: cgm_remove_cgroup: 505 Error removing devices:lxc/lxcora01-2
lxc-start: cgmanager.c: cgm_remove_cgroup: 503 call to cgmanager_remove_sync failed: invalid request
lxc-start: cgmanager.c: cgm_remove_cgroup: 505 Error removing blkio:lxc/lxcora01-2
lxc-start: cgmanager.c: cgm_remove_cgroup: 503 call to cgmanager_remove_sync failed: invalid request
lxc-start: cgmanager.c: cgm_remove_cgroup: 505 Error removing cpuacct:lxc/lxcora01-2
lxc-start: cgmanager.c: cgm_remove_cgroup: 503 call to cgmanager_remove_sync failed: invalid request
lxc-start: cgmanager.c: cgm_remove_cgroup: 505 Error removing cpu:lxc/lxcora01-2
lxc-start: cgmanager.c: cgm_remove_cgroup: 503 call to cgmanager_remove_sync failed: invalid request
lxc-start: cgmanager.c: cgm_remove_cgroup: 505 Error removing cpuset:lxc/lxcora01-2
lxc-start: lxc_start.c: main: 341 The container failed to start.
lxc-start: lxc_start.c: main: 345 Additional information can be obtained by setting the --logfile and --logpriority options.
gstanden@vmem1:~$
I was at Starbucks with my mom who is 80 and who I take out for coffee quite a bit, and then I keep working at the Starbucks. Well at Starbucks my LXC container with the new NFS mounts would not start. So there are several steps that thank God I found on the internet which got me through this latest fire trial walk on coals.
First, I found these links here at Technology Talk website and here at TQuerci's Weblog. These sites indicated that you had to make some changes to the /etc/apparmor.d/lxc/lxc-default file, so I made the recommended changes and that file now looks like this as shown below. Thanks very much to that website and weblog for keeping this project moving forward with that very helpful information about setup for NFS in LXC containers.
gstanden@vmem1:~$ cat /etc/apparmor.d/lxc/lxc-default
# Do not load this file. Rather, load /etc/apparmor.d/lxc-containers, which
# will source all profiles under /etc/apparmor.d/lxc
profile lxc-container-default flags=(attach_disconnected,mediate_deleted) {
#include <abstractions/lxc/container-base>
# the container may never be allowed to mount devpts. If it does, it
# will remount the host's devpts. We could allow it to do it with
# the newinstance option (but, right now, we don't).
deny mount fstype=devpts,
# allow nfs mount everywhere
mount fstype=nfs,
mount fstype=nfs4,
mount fstype=rpc_pipefs,
}
gstanden@vmem1:~$
Well unfortunately, I still got some error that was simlar to the first one (but different). Unfortunately it's not available on my screen anymore and I can't cut and paste it because I stupidly did a clear screen, but you get the idea, this still did not fix it completely.
Well, Ubuntu LXC documentation in this case came to the rescue. I followed the steps here and ran this command as shown below which supposedly would fix this problem possibly (and together with another step, it did). Now as to what this command actually means, does, etc. idk. I am just trying to get Oracle RAC running in some LXC containers on Ubuntu why only God knows and since I'm still an intermediate level LXC newb, I don't entirely know what these commands did, but one thing they did actually do is get my LXC container to start normally again (with a couple of additional steps which I will also share).
sudo apparmor_parser -R /etc/apparmor.d/usr.bin.lxc-start
sudo ln -s /etc/apparmor.d/usr.bin.lxc-start /etc/apparmor.d/disabled
So after these steps, still had issue. But that turned out to be from having mucked about trying to add a second network interface to my LXC container and now knowing how, which resulted in a port hanging off my sw1 OpenvSwitch and so I had to delete that manually as shown below (you won't need to do this for sure).
sudo ovs-vsctl del-port sw1 lxcora01b
Ok so after that was done, there was one more step that was required, I had to add some single line add-in to my LXC config file as shown below (bold line at end is the one I added).
root@vmem1:~# cd /var/lib/lxc/lxcora01
root@vmem1:/var/lib/lxc/lxcora01# cat config
# Template used to create this container: /usr/share/lxc/templates/lxc-oracle
# Parameters passed to the template:
# For additional config options, please look at lxc.container.conf(5)
lxc.network.type = veth
lxc.network.flags = up
# lxc.network.link = lxcbr0
lxc.network.script.up = /etc/network/if-up.d/lxc-ifup
lxc.network.script.down = /etc/network/if-down.d/lxc-ifdown
lxc.network.veth.pair = lxcora01
# GLS 2014-10-22 Above 3 lines added for OpenvSwitch networking.
lxc.network.hwaddr = 00:16:3e:5d:ae:70
lxc.rootfs = /var/lib/lxc/lxcora01/rootfs
# Common configuration
lxc.include = /usr/share/lxc/config/oracle.common.conf
# Container configuration for Oracle Linux 6.5
lxc.arch = x86_64
lxc.utsname = lxcora01
lxc.cap.drop = sys_resource
lxc.cap.drop = setfcap setpcap
# Networking
lxc.network.name = eth0
lxc.network.mtu = 1500
lxc.network.hwaddr = fe:cb:23:4e:a9:22
lxc.cgroup.devices.allow = a # Allow access to all devices
lxc.cgroup.devices.allow = a 10:236 rwm
lxc.cgroup.devices.allow = a 252:* rwm
lxc.cgroup.devices.allow = a 8:* rwm
lxc.mount.entry = /dev/mapper /var/lib/lxc/lxcora01/rootfs/dev/mapper none defaults,bind,create=dir 0 0
lxc.mount.auto = proc sys cgroup
lxc.mount.auto = proc:rw sys:rw cgroup-full:rw
lxc.aa_allow_incomplete = 1
lxc.aa_profile = unconfined
root@vmem1:/var/lib/lxc/lxcora01#
Now the LXC container lxcora01 would start and it would also get it's DHCP ip address that is on my OpenvSwitch sw1 and it mounted it's NFS shares. Halleluluah! The NFS shares are going to be the shared storage. LXC container lxcora01 is just a sort of test bed where I am trying to confirm that I can create all the things that will be needed. Here are the NFS shares that are setup for the RAC on NFS in the LXC container as shown below. Shown below are both the output of "df -TH" and also how the /etc/fstab is setup for NFS.
[root@lxcora01 ~]# df -TH
Filesystem Type Size Used Avail Use% Mounted on
tmpfs tmpfs 4.0G 662M 3.3G 17% /dev/shm
vmem1:/shared_config nfs 964G 465G 450G 51% /shared_config
vmem1:/shared_grid nfs 964G 465G 450G 51% /shared_grid
vmem1:/shared_home nfs 964G 465G 450G 51% /shared_home
vmem1:/shared_data nfs 964G 465G 450G 51% /shared_data
[root@lxcora01 ~]# cat /etc/fstab
tmpfs /dev/shm tmpfs rw,exec,size=3500m 0 0
vmem1:/shared_config /shared_config nfs rw,bg,hard,nointr,tcp,vers=3,timeo=600,rsize=32768,wsize=32768,actimeo=0 0 0
vmem1:/shared_grid /shared_grid nfs rw,bg,hard,nointr,tcp,vers=3,timeo=600,rsize=32768,wsize=32768,actimeo=0 0 0
vmem1:/shared_home /shared_home nfs rw,bg,hard,nointr,tcp,vers=3,timeo=600,rsize=32768,wsize=32768,actimeo=0 0 0
vmem1:/shared_data /shared_data nfs rw,bg,hard,nointr,tcp,vers=3,timeo=600,rsize=32768,wsize=32768,actimeo=0 0 0
[root@lxcora01 ~]#
So I have storage. Presumably, subject to verification, I should be able to setup two new LXC containers and mount this NFS share to both of them for my shared RAC storage.
Update 26 October 2014
As previously noted in another update, I opted to use ASM block storage after all, not NFS. However, also as previously noted, I've kept all the NFS information in this blog post because at some point an 11gR2 RAC on LXC containers will be built using NFS.
This note is that to get NFS to mount at boot time, I had to add the resolution information for the NFS server (vmem1.vmem.org) to /etc/hosts, as it does not seem to be getting the information successfully at boot time from DNS. Therefore, below is my configuration for the system with respect to all these various needs. The setup involves edits to all of the following files:
/etc/hosts
/etc/nsswitch.conf
/etc/dhcp/dhclient.conf
/etc/resolv.conf
and these files in their current configuration are shown, with some relevant sections bolder where applicable, as shown below.
[root@lxcora02 ~]# cat /etc/dhcp/dhclient.conf <-- This puts the GNS information into /etc/resolv.conf on boot
prepend domain-name-servers 10.207.39.3;
append domain-name " lxcrac-gns.vmem.org";
[root@lxcora02 ~]# cat /etc/resolv.conf
options attempts:2 timeout:1 <-- This is the resolv.conf format that is required for GNS
; generated by /sbin/dhclient-script
search vmem.org lxcrac-gns.vmem.org
nameserver 10.207.39.3
nameserver 10.207.39.1
[root@lxcora02 ~]# cat /etc/hosts <-- This is required so that the NFS mounts will mount at boot
127.0.0.1 localhost lxcora02
10.207.39.88 lxcora03.vmem.org lxcora03
10.207.39.87 lxcora02.vmem.org lxcora02
10.207.39.1 vmem1.vmem.org vmem1
[root@lxcora02 ~]# cat /etc/nsswitch.conf
#
# /etc/nsswitch.conf
#
# An example Name Service Switch config file. This file should be
# sorted with the most-used services at the beginning.
#
# The entry '[NOTFOUND=return]' means that the search for an
# entry should stop if the search in the previous entry turned
# up nothing. Note that if the search failed due to some other reason
# (like no NIS server responding) then the search continues with the
# next entry.
#
# Valid entries include:
#
# nisplus Use NIS+ (NIS version 3)
# nis Use NIS (NIS version 2), also called YP
# dns Use DNS (Domain Name Service)
# files Use the local files
# db Use the local database (.db) files
# compat Use NIS on compat mode
# hesiod Use Hesiod for user lookups
# [NOTFOUND=return] Stop searching if not found so far
#
# To use db, put the "db" in front of "files" for entries you want to be
# looked up first in the databases
#
# Example:
#passwd: db files nisplus nis
#shadow: db files nisplus nis
#group: db files nisplus nis
passwd: files
shadow: files
group: files
#hosts: db files nisplus nis dns
hosts: dns files nisplus nis db <-- Want DNS to be the preferential name resolution method, so it goes first
# Example - obey only what nisplus tells us...
#services: nisplus [NOTFOUND=return] files
#networks: nisplus [NOTFOUND=return] files
#protocols: nisplus [NOTFOUND=return] files
#rpc: nisplus [NOTFOUND=return] files
#ethers: nisplus [NOTFOUND=return] files
#netmasks: nisplus [NOTFOUND=return] files
bootparams: nisplus [NOTFOUND=return] files
ethers: files
netmasks: files
networks: files
protocols: files
rpc: files
services: files
netgroup: nisplus
publickey: nisplus
automount: files nisplus
aliases: files nisplus
[root@lxcora02 ~]#
Additional update to the update (a short while later). The below shows what would seem to be the ideal settings which are changed a bit from above for /etc/resolv.conf and /etc/dhcp/dhclient.conf so that (a) all external resolutions will work and (b) resolution of RAC VIPs will also work via GNS.
[root@lxcora03 ~]# nslookup lxcora03-vip.lxcrac-gns.vmem.org
Server: 10.207.39.3
Address: 10.207.39.3#53
Name: lxcora03-vip.lxcrac-gns.vmem.org
Address: 10.207.39.93
[root@lxcora03 ~]# nslookup lxcora03-vip
;; Got recursion not available from 10.207.39.3, trying next server
Server: 10.207.39.3
Address: 10.207.39.3#53
Name: lxcora03-vip.lxcrac-gns.vmem.org
Address: 10.207.39.93
[root@lxcora03 ~]# nslookup oracle652
;; Got recursion not available from 10.207.39.3, trying next server
Server: 10.207.39.1
Address: 10.207.39.1#53
Name: oracle652.vmem.org
Address: 10.207.39.74
[root@lxcora03 ~]# cat /etc/dhcp/dhclient.conf
prepend domain-name-servers 10.207.39.3;
append domain-name " lxcrac-gns.vmem.org";
[root@lxcora03 ~]# cat /etc/resolv.conf
options attempts:2 timeout:1 rotate
; generated by /sbin/dhclient-script
search vmem.org lxcrac-gns.vmem.org
nameserver 10.207.39.3
nameserver 10.207.39.1
[root@lxcora03 ~]#
A Note about ntp
The ntp was installed according to some notes at oracle-base. It doesn't seem to run however in the LXC containers. So I uninstalled it using yum. And guess what? The Oracle has failed to startup! Even though ntp was not running, it seems has wanted it there. So I reinstalled ntp, and voila! has stack and all Oracle services started. Need to figure out what that was all about.
Adding a Second OpenvSwitch Ethernet Interface to LXC Container
Update 26 October 2014
I ended up using OpenvSwitch for all the networking and decommissioned the "lxcbr0" that I was using mostly as a convenience and out of sheer bold laziness. First, the lxcbr0 was having some issue where I could not ping or ssh between the LXC containers (which I believe occurred after I had setup OpenvSwitch for the public interface which may have somehow broken inter-container ssh), even though both were on the same 10.0.3.1 default LXC network, so opted to just create an additional OpenvSwitch for the private RAC interconnect network and completely decommission the out-of-the-box lxcbr0 default networking solution and go with an all-OpenvSwitch solution. I made some desultory efforts to solve this, but with no success on that front, decided that a better use of hair-pulling time would be to just setup the private RAC interconnect on OpenvSwitch since that is of much more interest anyway, and since OpenvSwitch is a fully-functional professional-grade layer 2 switch. Also, using OpenvSwitch makes the upcoming exercise to completely re-ip a GNS RAC a much more palatable and enjoyable exercise.
End Update
The LXC container config file used to obtain a second ethernet interface (i.e. the config file for having two ethernet interfaces in the LXC container) is shown below. The single most important reference used for getting a second network interface setup for the RAC private interconnect in the LXC container was here at Ta-Ching Chen's blog and the Network Administrator Blog here, and also some good info at box.matto.nl here, was also very helpful (this was actually the first reference that I found that got things rolling). This blog also had another page that gave the best information I had found on how to use LXC with OpenvSwitch networking. The solution for this RAC on LXC uses interface eth0 (on the OpenvSwitch network) as the public interface, and eth1, on the default LXC "lxcbr0" bridge for the private RAC interconnect network. The solution could of course use a second network defined in bind9 (named) for the private network, but it was convenient to use lxcbr0 default LXC networking for the private interconnect for this first attempt at building a 2-node Oracle 12c RAC on LXC. Below is shown the config file that is used. The two stanzas in the "Networking" section are the ethernet interfaces. Some entries of interest are bolded as shown below.
root@vmem1:~# cd /var/lib/lxc/lxcora01
root@vmem1:/var/lib/lxc/lxcora01# cat config
# Template used to create this container: /usr/share/lxc/templates/lxc-oracle
# Parameters passed to the template:
# For additional config options, please look at lxc.container.conf(5)
# Networking
lxc.network.type = veth
lxc.network.flags = up
lxc.network.script.up = /etc/network/if-up.d/lxc-ifup-sw1
lxc.network.script.down = /etc/network/if-down.d/lxc-ifdown-sw1
lxc.network.veth.pair = lxcora01-pub
lxc.network.name = eth0
lxc.network.mtu = 1500
lxc.network.hwaddr = fe:cb:23:4e:a9:22
lxc.network.type = veth
lxc.network.flags = up
lxc.network.link = lxcbr0
lxc.network.veth.pair = lxcora01-pri
lxc.network.name = eth1
lxc.network.mtu = 1500
#lxc.network.hwaddr = fe:cb:23:4e:a9:23
lxc.network.hwaddr = 00:16:3e:5d:ae:70
# Common configuration
lxc.rootfs = /var/lib/lxc/lxcora01/rootfs
lxc.include = /usr/share/lxc/config/oracle.common.conf
# Container configuration for Oracle Linux 6.5
lxc.arch = x86_64
lxc.utsname = lxcora01
lxc.cap.drop = sys_resource
lxc.cap.drop = setfcap setpcap
# Devices
lxc.cgroup.devices.allow = a # Allow access to all devices
lxc.cgroup.devices.allow = a 10:236 rwm
lxc.cgroup.devices.allow = a 252:* rwm
lxc.cgroup.devices.allow = a 8:* rwm
lxc.mount.entry = /dev/mapper /var/lib/lxc/lxcora01/rootfs/dev/mapper none defaults,bind,create=dir 0 0
lxc.mount.auto = proc sys cgroup
lxc.mount.auto = proc:rw sys:rw cgroup-full:rw
lxc.aa_allow_incomplete = 1
# GLS 2014-10-23 NFS
lxc.aa_profile = unconfined
root@vmem1:/var/lib/lxc/lxcora01#
A show of the OpenvSwitch configuration helps to show how the networking is setup for the public interface for the RAC as shown below. The lxcora01-pub port off of sw1 is shown in bold.
root@vmem1:/var/lib/lxc/lxcora01# ovs-vsctl show
2fc24710-34b5-4aa2-a32d-4e7bcb1afa1a
Bridge "sw2"
Port "sw2"
Interface "sw2"
type: internal
Port "t5"
Interface "t5"
Port "t1"
Interface "t1"
Port "t6"
Interface "t6"
Port "t3"
Interface "t3"
Port "t2"
Interface "t2"
Port "t4"
Interface "t4"
Bridge "sw1"
Port "s2"
Interface "s2"
Port "s4"
Interface "s4"
Port "s1"
Interface "s1"
Port "s6"
Interface "s6"
Port "sw1"
Interface "sw1"
type: internal
Port "s5"
Interface "s5"
Port "lxcora01-pub"
Interface "lxcora01-pub"
Port "s3"
Interface "s3"
Bridge "sw3"
Port "w4"
Interface "w4"
Port "w2"
Interface "w2"
Port "w6"
Interface "w6"
Port "w3"
Interface "w3"
Port "w5"
Interface "w5"
Port "w1"
Interface "w1"
Port "sw3"
Interface "sw3"
type: internal
ovs_version: "2.0.2"
root@vmem1:/var/lib/lxc/lxcora01#
The two scripts refernced in the config file are shown below. These scripts dynamically create the port on the OpenvSwitch and then destroy the port when the LXC container is shut down.
root@vmem1:/etc/network/if-up.d# cat lxc-ifup-sw1
#!/bin/bash
ovsBr='sw1'
ovs-vsctl --may-exist add-br ${ovsBr}
ovs-vsctl --may-exist add-port ${ovsBr} $5
root@vmem1:/etc/network/if-up.d# pwd
/etc/network/if-up.d
root@vmem1:/etc/network/if-up.d#
The similar script for shutdown is shown below.
root@vmem1:/etc/network/if-down.d# cat lxc-ifdown-sw1
#!/bin/bash
ovsBr='sw1'
ovs-vsctl --if-exists del-port ${ovsBr} $5
root@vmem1:/etc/network/if-down.d# pwd
/etc/network/if-down.d
root@vmem1:/etc/network/if-down.d# ls -lrt lxc-ifdown-sw1
-rwxr-xr-x 1 root root 67 Sep 6 20:03 lxc-ifdown-sw1
root@vmem1:/etc/network/if-down.d#
Update 26 October 2014
Couple of updates here. First. when shutting down the LXC container RAC nodes lxcora02 or lxcora03 it is best to use "shutdown -h now" and not "reboot" because the reboot sequence does not execute the "lxc-ifdown-sw*" script, and so when the LXC node tries to restart, it fails because the port on the OpenvSwitch still exists (was not destroyed at shutdown). Thus, shutdown should be used, not reboot.
Found that the "if-exists" clause on the "del-port" command above was causing the "del-port" command to fail so took it out. Here is the form of the commands which is working and which is reproducibly and reliably removing the port at node shutdown, as shown below. This format should be used (without the "if-exists" clause).
Another update, a short while later, also found that even with the "if-exists" clause removed, this command was still sometimes failing, not sure why. So in addition, I also put in an if-then to re-run the command if the first run of the command was unsuccessful. Here is the current scritping being used, so far successfully, for destroying the port on shutdown.
gstanden@vmem1:/etc/network/if-down.d$ cat lxc-ifdown-sw1
#!/bin/bash
ovsBr='sw1'
sudo ovs-vsctl del-port ${ovsBr} $5
gstanden@vmem1:/etc/network/if-down.d$ cat lxc-ifdown-sw4
#!/bin/bash
ovsBr='sw4'
sudo ovs-vsctl del-port ${ovsBr} $5
gstanden@vmem1:/etc/network/if-down.d$
So here is the version of the script for down which seems to be working every time as shown below. You can see the function has been added to re-run the del-port command if it still exists.
gstanden@vmem1:~$ cat /etc/network/if-down.d/lxc-ifdown-sw1
#!/bin/bash
ovsBr='sw1'
sudo ovs-vsctl del-port ${ovsBr} $5
function CheckPortExists
{
sudo ovs-vsctl show | grep \"$5\" | grep Port | sed 's/^[ \t]*//;s/[ \t]*$//' | cut -f2 -d' ' | sed 's/"//g'
}
PortExists=$(CheckPortExists)
if [ "$PortExists" = "$5" ]
then
sudo ovs-vsctl del-port ${ovsBr} $5
fi
gstanden@vmem1:~$
End Update
Update 27 October 2014
Every now and then the above lxc-ifdown-sw1 script is still failing to clear out the ports from the OpenvSwitch. I'm going to give each LXC container it's own file because the only thing I can think of is that the instances may be concurrently trying to access the files and this is causing them to occassionally fail.
So even larger changes were made today later in the day and so far they seem to be working perfectly. First, I added 3 more private networks so that on this install I will add HAIP 4-network to the build. Second, I have given each interface up/down it's own file so that there will be no chance of interfaces trying to access the same up/down files. Here are those files as shown below.
gstanden@vmem1:/etc/network/if-up.d$ ls -l lxcora*
-rwxr-xr-x 1 root root 117 Oct 27 20:09 lxcora02-ifup-sw1
-rwxr-xr-x 1 root root 117 Oct 27 20:09 lxcora02-priv1-ifup-sw4
-rwxr-xr-x 1 root root 117 Oct 27 20:31 lxcora02-priv2-ifup-sw5
-rwxr-xr-x 1 root root 117 Oct 27 20:31 lxcora02-priv3-ifup-sw6
-rwxr-xr-x 1 root root 117 Oct 27 20:31 lxcora02-priv4-ifup-sw7
-rwxr-xr-x 1 root root 117 Oct 27 20:09 lxcora03-ifup-sw1
-rwxr-xr-x 1 root root 117 Oct 27 20:09 lxcora03-priv1-ifup-sw4
-rwxr-xr-x 1 root root 117 Oct 27 20:32 lxcora03-priv2-ifup-sw5
-rwxr-xr-x 1 root root 117 Oct 27 20:32 lxcora03-priv3-ifup-sw6
-rwxr-xr-x 1 root root 117 Oct 27 20:33 lxcora03-priv4-ifup-sw7
gstanden@vmem1:/etc/network/if-up.d$ cd ..
gstanden@vmem1:/etc/network$ cd if-down.d
gstanden@vmem1:/etc/network/if-down.d$ ls -l lxcora*
-rwxr-xr-x 1 root root 300 Oct 27 20:08 lxcora02-ifdown-sw1
-rwxr-xr-x 1 root root 301 Oct 27 20:20 lxcora02-priv1-ifdown-sw4
-rwxr-xr-x 1 root root 301 Oct 27 20:21 lxcora02-priv2-ifdown-sw5
-rwxr-xr-x 1 root root 301 Oct 27 20:21 lxcora02-priv3-ifdown-sw6
-rwxr-xr-x 1 root root 301 Oct 27 20:21 lxcora02-priv4-ifdown-sw7
-rwxr-xr-x 1 root root 300 Oct 27 20:09 lxcora03-ifdown-sw1
-rwxr-xr-x 1 root root 301 Oct 27 20:20 lxcora03-priv1-ifdown-sw4
-rwxr-xr-x 1 root root 301 Oct 27 20:21 lxcora03-priv2-ifdown-sw5
-rwxr-xr-x 1 root root 301 Oct 27 20:21 lxcora03-priv3-ifdown-sw6
-rwxr-xr-x 1 root root 301 Oct 27 20:21 lxcora03-priv4-ifdown-sw7
gstanden@vmem1:/etc/network/if-down.d$
And here is an example of how those files look as shown below. The files are all the same except for the name of the particular OpenvSwitch as shown below.
gstanden@vmem1:/etc/network/if-down.d$ cat lxc-ifdown-sw1
#!/bin/bash
ovsBr='sw1'
sudo ovs-vsctl del-port ${ovsBr} $5
function CheckPortExists
{
sudo ovs-vsctl show | grep \"$5\" | grep Port | sed 's/^[ \t]*//;s/[ \t]*$//' | cut -f2 -d' ' | sed 's/"//g'
}
PortExists=$(CheckPortExists)
if [ "$PortExists" = "$5" ]
then
sudo ovs-vsctl del-port ${ovsBr} $5
fi
gstanden@vmem1:/etc/network/if-down.d$
gstanden@vmem1:/etc/network/if-down.d$ diff lxc-ifdown-sw4 lxc-ifdown-sw1
2c2
< ovsBr='sw4'
---
> ovsBr='sw1'
End Update
Edit /etc/dhcp/dhcpclient.conf
To get the /etc/resolv.conf to generate correctly at startup of the LXC container RAC node, it was necessary to edit the /etc/dhcp/dhcpclient.conf file (inside the LXC container) as shown below. Note that the "nameserver 10.0.3.1" is added automatically without need of an entry in dhcpclient.conf and it was only the need to get the LXC container to be able to resolve DNS names on the vmem.org 10.207.39.1 network which drove the need to add these lines to the dhcpclient.conf file.
gstanden@vmem1:~$ ssh root@lxcora01
root@lxcora01's password:
Last login: Fri Oct 24 10:59:59 2014 from vmem1.vmem.org
[root@lxcora01 ~]# cat /etc/dhcp/dhclient.conf
prepend domain-name-servers 10.207.39.1;
supersede domain-search "vmem.org";
[root@lxcora01 ~]#
The /etc/resolv.conf file which results from the lines added to dhcpclient.conf is shown below. This /etc/resolv.conf will resolve named and addresses on both networks.
[root@lxcora01 ~]# cat /etc/resolv.conf
; generated by /sbin/dhclient-script
search vmem.org.
nameserver 10.207.39.1
nameserver 10.0.3.1
[root@lxcora01 ~]#
For example, here is a lookup of the Grid Naming Service (GNS) DNS entry for this RAC.
[root@lxcora01 ~]# nslookup lxcrac-gns
Server: 10.207.39.1
Address: 10.207.39.1#53
Name: lxcrac-gns.vmem.org
Address: 10.207.39.3
[root@lxcora01 ~]# nslookup 10.207.39.3
Server: 10.207.39.1
Address: 10.207.39.1#53
3.39.207.10.in-addr.arpa name = lxcrac-gns.vmem.org.
[root@lxcora01 ~]#
Here's a lookup of the RAC node itself.
[root@lxcora01 ~]# nslookup lxcora01
Server: 10.207.39.1
Address: 10.207.39.1#53
Name: lxcora01.vmem.org
Address: 10.207.39.85
[root@lxcora01 ~]# nslookup 10.207.39.85
Server: 10.207.39.1
Address: 10.207.39.1#53
85.39.207.10.in-addr.arpa name = lxcora01.vmem.org.
[root@lxcora01 ~]#
Notice that if the vmem.org and associated nameserver information were not in /etc/resolv.conf it would resolve the LXC RAC node on the default LXC lxcbr0 network (private RAC interconnect network) instead of the desired public network resolution which is why the additional entries in /etc/resolv.conf are needed.
[root@lxcora01 ~]# cat /etc/resolv.conf
; generated by /sbin/dhclient-script
#search vmem.org.
#nameserver 10.207.39.1
nameserver 10.0.3.1
[root@lxcora01 ~]# nslookup lxcora01
Server: 10.0.3.1
Address: 10.0.3.1#53
Name: lxcora01
Address: 10.0.3.2
[root@lxcora01 ~]#
Setup Oracle GNS (Grid Naming Service) on LXC Container
Martin Carstenbach has a great post on how to test to be sure the GNS setup is correct here at Simplified GNS setup for RAC 11.2.0.2 and newer. That's what is about to be done. Last night the GNS was setup with help from these references. This is my first time using GNS - I've been an Oracle DBA over 20 years but kept putting this off and using static DNS entries for RAC. However , with this setup, so much work has been done setting up this laptop system to use Dynamic DNS for everything, that it was very appealing to stay consistent with that motif so here now finally using GNS which is long overdue. Some references that were found very helpful were hereand also the Oracle official documentation here and Martin Carstenbach has some very useful-looking blogs that I have not used (yet) but which look like they will eventually be consulted since they are chock full of great information here and here.
Shown below are the forward zone lookup file and the reverse zone lookup file with the GNS additions in bold which will be used on this 2-node LXC RAC. This is on the Ubuntu Host OS and included is the directory location of the zone files as well as the checkzone test. The DNS environment on this laptop uses a dynamic DNS and DHCP setup which was built from help from Lee Hutchinson's Big Dinosaur blog here.
root@vmem1:/var/lib/bind# pwd
/var/lib/bind
root@vmem1:/var/lib/bind# cat fwd.vmem.org
$ORIGIN .
$TTL 86400 ; 1 day
vmem.org IN SOA vmem1.vmem.org. postmaster.vmem.org. (
1410240112 ; serial
60 ; refresh (1 minute)
1800 ; retry (30 minutes)
604800 ; expire (1 week)
86400 ; minimum (1 day)
)
NS vmem1.vmem.org.
$ORIGIN vmem.org.
_sflow._udp TXT "txtvers=1" "polling=20" "sampling=512"
SRV 0 0 6343 vmem1
$TTL 3600 ; 1 hour
lxcora01 A 10.207.39.85
TXT "001060b0cef034d0d8b3ec569ea6615637"
$TTL 86400 ; 1 day
lxcora1 A 10.207.39.79
$TTL 3600 ; 1 hour
lxcora3 A 10.207.39.83
TXT "007e7554cced9024a1bac46cc4370293c6"
lxcora5 A 10.207.39.84
TXT "0092ff209913ebc9ca832c943e48eff86f"
$TTL 86400 ; 1 day
oracle631 A 10.207.39.72
oracle632 A 10.207.39.76
oracle652 A 10.207.39.74
vmem1 A 10.207.39.1
lxcrac-gns A 10.207.39.3
$TTL 3600 ; 1 hour
vmem2 A 10.207.39.81
TXT "3118a6c1688c312b822454df8baa10bfff"
; subdomain (lxcrac1.vmem.org) for Oracle RAC GNS
$ORIGIN lxcrac.vmem.org.
@ IN NS lxcrac-gns.vmem.org.
root@vmem1:/var/lib/bind# named-checkzone vmem.org fwd.vmem.org
zone vmem.org/IN: loaded serial 1410240112
OK
root@vmem1:/var/lib/bind#
Update 27 October 2014
So because I had forgotten to change the name of my ASM system diskgroup from the default "DATA" during the GI install to "SYSTEMDG" I tried to move the DATA over to SYSTEMDG but this has become significantly more complicated. Levi Pereira had a very comprehensive procedure, but there were some issues, and I decided to instead uninstall the entire GI and redo it. I had some other reason for a redo too. I wanted to simplify the naming convention for the GNS setup as shown below this is the revised fwd.vmem.org and rev.vmem.org files on the Ubuntu host as shown below. The GNS related parts revised are shown in bold. On this install the delegated domain will simply be "gns1.vmem.org". The VIP was changed to simply be "lxc1". Also, I missed some screenshots during the GI install that pertained to some non-fatal errors, so I'm going to be sure I get those on the next go of installing GI.
root@vmem1:/var/lib/bind# cat fwd.vmem.org
$ORIGIN .
$TTL 86400 ; 1 day
vmem.org IN SOA vmem1.vmem.org. postmaster.vmem.org. (
1410271922 ; serial
60 ; refresh (1 minute)
1800 ; retry (30 minutes)
604800 ; expire (1 week)
86400 ; minimum (1 day)
)
NS vmem1.vmem.org.
$ORIGIN vmem.org.
_sflow._udp TXT "txtvers=1" "polling=20" "sampling=512"
SRV 0 0 6343 vmem1
$TTL 3600 ; 1 hour
lxcora01 A 10.207.39.85
TXT "001060b0cef034d0d8b3ec569ea6615637"
lxcora02 A 10.207.39.87
TXT "001d6eb17bf9f9081897f98548b528bfee"
lxcora03 A 10.207.39.88
TXT "0084a3923f9bbe6559acbb98034af0c10a"
$TTL 86400 ; 1 day
lxcora1 A 10.207.39.79
$TTL 3600 ; 1 hour
lxcora3 A 10.207.39.83
TXT "007e7554cced9024a1bac46cc4370293c6"
lxcora5 A 10.207.39.84
TXT "0092ff209913ebc9ca832c943e48eff86f"
$TTL 86400 ; 1 day
oracle631 A 10.207.39.72
oracle632 A 10.207.39.76
oracle652 A 10.207.39.74
lxc1 A 10.207.39.3
vmem1 A 10.207.39.1
$TTL 3600 ; 1 hour
vmem2 A 10.207.39.81
TXT "3118a6c1688c312b822454df8baa10bfff"
$ORIGIN gns1.vmem.org
@ IN NS lxc1.vmem.org.
root@vmem1:/var/lib/bind#
And this is the reverse lookup zone file information as shown below with the added reverse lookup for lxc1 in bold.
gstanden@vmem1:~$ cat /var/lib/bind/rev.vmem.org
$ORIGIN .
$TTL 86400 ; 1 day
39.207.10.in-addr.arpa IN SOA vmem1.vmem.org. postmaster.vmem.org. (
1410271922 ; serial
3600 ; refresh (1 hour)
1800 ; retry (30 minutes)
604800 ; expire (1 week)
86400 ; minimum (1 day)
)
NS vmem1.vmem.org.
$ORIGIN 39.207.10.in-addr.arpa.
1 PTR vmem1.vmem.org.
3 PTR lxc1.vmem.org.
72 PTR oracle631.vmem.org.
76 PTR oracle632.vmem.org.
$TTL 3600 ; 1 hour
81 PTR vmem2.vmem.org.
85 PTR lxcora01.vmem.org.
87 PTR lxcora02.vmem.org.
88 PTR lxcora03.vmem.org.
$TTL 86400 ; 1 day
90 PTR oracle635.vmem.org.
39.207.10.in-addr.arpa NS vmem1.vmem.org.
vmem1 A 10.207.39.1
gstanden@vmem1:~$
Next just finished getting the /etc/resolv.conf setup for GNS. There was quite a bit of discovery and work there. Here is what the /etc/resolv.conf looks like for this system as correctly configured shown below.
[root@lxcora01 ~]# cat /etc/resolv.conf
options attempts:2 timeout:1
; generated by /sbin/dhclient-script
search vmem.org gns.vmem.org
nameserver 10.207.39.3
nameserver 10.207.39.1
nameserver 10.0.3.1
[root@lxcora01 ~]#
As many know, /etc/resolv.conf gets overwritten at each reboot (i.e. each restart of networking) so some other files have to be configured to get the desired result. For this system, configuration of /etc/dhcp/dhclient.conf was sufficient to setup the required "search" and "nameserver" options (and order of nameservers matters in this case). Shown below is the required /etc/dhcp/dhclient.conf file.
[root@lxcora01 ~]# cat /etc/dhcp/dhclient.conf
prepend domain-name-servers 10.207.39.3,10.207.39.1;
append domain-name " gns.vmem.org";
[root@lxcora01 ~]#
As shown above, the file sets the FIRST nameserver as the GNS nameserver, then sets the vmem.org BIND9 namerserver second. The third nameserver entry is probably irrelevant and is set automatically by other settings (unknown) and does not harm being there afaik, and is part of the default LXC network "lxcbr0" setup, so no effort was made to remove it.
The "options" line at the top of the /etc/resolv.conf was simply typed into the resolv.conf and it was found that it persists across reboots, so apparently the "options" directives are not overwritten when networking restarts. So, the resolv.conf is believed to be complete. There were several references that were very helpful working out how to get the required options in the /etc/resolv.conf at the pages here at good old Cyberciti, here at good old AskUbuntu (for the "append" directive), colleague Edmund White at ServerFault.
End Update
Speed Up SSH
After setting up GNS in the /etc/resolv.conf ssh became markedly slow. The following fixes were found here at the GeekStuff and also here at In Just Five Minutes and worked very well. Here is the /etc/ssh/sshd_config that is working very well with the GNS setup as shown below with the changed entries in bold. The two options that should be used are:
GSSAPIAuthentication no
UseDNS no
[root@lxcora02 ~]# cat /etc/ssh/sshd_config
# $OpenBSD: sshd_config,v 1.80 2008/07/02 02:24:18 djm Exp $
# This is the sshd server system-wide configuration file. See
# sshd_config(5) for more information.
# This sshd was compiled with PATH=/usr/local/bin:/bin:/usr/bin
# The strategy used for options in the default sshd_config shipped with
# OpenSSH is to specify options with their default value where
# possible, but leave them commented. Uncommented options change a
# default value.
#Port 22
#AddressFamily any
#ListenAddress 0.0.0.0
#ListenAddress ::
# Disable legacy (protocol version 1) support in the server for new
# installations. In future the default will change to require explicit
# activation of protocol 1
Protocol 2
# HostKey for protocol version 1
#HostKey /etc/ssh/ssh_host_key
# HostKeys for protocol version 2
#HostKey /etc/ssh/ssh_host_rsa_key
#HostKey /etc/ssh/ssh_host_dsa_key
# Lifetime and size of ephemeral version 1 server key
#KeyRegenerationInterval 1h
#ServerKeyBits 1024
# Logging
# obsoletes QuietMode and FascistLogging
#SyslogFacility AUTH
SyslogFacility AUTHPRIV
#LogLevel INFO
# Authentication:
#LoginGraceTime 2m
#PermitRootLogin yes
#StrictModes yes
#MaxAuthTries 6
#MaxSessions 10
#RSAAuthentication yes
#PubkeyAuthentication yes
#AuthorizedKeysFile .ssh/authorized_keys
#AuthorizedKeysCommand none
#AuthorizedKeysCommandRunAs nobody
# For this to work you will also need host keys in /etc/ssh/ssh_known_hosts
#RhostsRSAAuthentication no
# similar for protocol version 2
#HostbasedAuthentication no
# Change to yes if you don't trust ~/.ssh/known_hosts for
# RhostsRSAAuthentication and HostbasedAuthentication
#IgnoreUserKnownHosts no
# Don't read the user's ~/.rhosts and ~/.shosts files
#IgnoreRhosts yes
# To disable tunneled clear text passwords, change to no here!
#PasswordAuthentication yes
#PermitEmptyPasswords no
PasswordAuthentication yes
# Change to no to disable s/key passwords
#ChallengeResponseAuthentication yes
ChallengeResponseAuthentication no
# Kerberos options
#KerberosAuthentication no
#KerberosOrLocalPasswd yes
#KerberosTicketCleanup yes
#KerberosGetAFSToken no
#KerberosUseKuserok yes
# GSSAPI options
#GSSAPIAuthentication yes
GSSAPIAuthentication no
#GSSAPICleanupCredentials yes
GSSAPICleanupCredentials yes
#GSSAPIStrictAcceptorCheck yes
#GSSAPIKeyExchange no
# Set this to 'yes' to enable PAM authentication, account processing,
# and session processing. If this is enabled, PAM authentication will
# be allowed through the ChallengeResponseAuthentication and
# PasswordAuthentication. Depending on your PAM configuration,
# PAM authentication via ChallengeResponseAuthentication may bypass
# the setting of "PermitRootLogin without-password".
# If you just want the PAM account and session checks to run without
# PAM authentication, then enable this but set PasswordAuthentication
# and ChallengeResponseAuthentication to 'no'.
#UsePAM no
UsePAM yes
# Accept locale-related environment variables
AcceptEnv LANG LC_CTYPE LC_NUMERIC LC_TIME LC_COLLATE LC_MONETARY LC_MESSAGES
AcceptEnv LC_PAPER LC_NAME LC_ADDRESS LC_TELEPHONE LC_MEASUREMENT
AcceptEnv LC_IDENTIFICATION LC_ALL LANGUAGE
AcceptEnv XMODIFIERS
#AllowAgentForwarding yes
#AllowTcpForwarding yes
#GatewayPorts no
#X11Forwarding no
X11Forwarding yes
#X11DisplayOffset 10
#X11UseLocalhost yes
#PrintMotd yes
#PrintLastLog yes
#TCPKeepAlive yes
#UseLogin no
#UsePrivilegeSeparation yes
#PermitUserEnvironment no
#Compression delayed
#ClientAliveInterval 0
#ClientAliveCountMax 3
#ShowPatchLevel no
UseDNS no
#PidFile /var/run/sshd.pid
#MaxStartups 10:30:100
#PermitTunnel no
#ChrootDirectory none
# no default banner path
#Banner none
# override default of no subsystems
Subsystem sftp /usr/libexec/openssh/sftp-server
# Example of overriding settings on a per-user basis
#Match User anoncvs
# X11Forwarding no
# AllowTcpForwarding no
# ForceCommand cvs server
[root@lxcora02 ~]#
Test GNS Configuration Using cluvfy
The configuration of the GNS can be tested as described by Martin Carstenbach here. The results of my test are shown below.
[root@lxcora01 ~]# su - grid
[grid@lxcora01 ~]$ cd grid
[grid@lxcora01 grid]$ ./runcluvfy.sh comp gns -precrsinst -domain vmem.org -vip 10.207.39.3 -verbose -n lxcora01
Verifying GNS integrity
Checking GNS integrity...
Checking if the GNS subdomain name is valid...
The GNS subdomain name "vmem.org" is a valid domain name
Checking if the GNS VIP is a valid address...
GNS VIP "10.207.39.3" resolves to a valid IP address
Checking the status of GNS VIP...
GNS integrity check passed
Verification of GNS integrity was successful.
[grid@lxcora01 grid]$
Update 27 October 2014
Here is the GNS configuration precrsinst check for the reinstall as shown below.
[grid@lxcora02 grid]$ ./runcluvfy.sh comp gns -precrsinst -domain vmem.org -vip 10.207.39.3 -verbose -n lxcora02,lxcora03
Verifying GNS integrity
Checking GNS integrity...
Checking if the GNS subdomain name is valid...
The GNS subdomain name "vmem.org" is a valid domain name
Checking if the GNS VIP is a valid address...
GNS VIP "10.207.39.3" resolves to a valid IP address
Checking the status of GNS VIP...
GNS integrity check passed
Verification of GNS integrity was successful.
[grid@lxcora02 grid]$
End Update
Clone the LXC Container to Create Second RAC Node
When the container is fully configured the quickest way to create the additional RAC node(s) is to just use the lxc-clone to clone the LXC container. The clone command is shown below.
gstanden@vmem1:~$ sudo lxc-clone -o lxcora01 -n lxcora02
Created container lxcora02 as copy of lxcora01
gstanden@vmem1:~$
The lxcora02 container needs just a couple of small manual edits since the clone utility apparently doesn't know about handling OpenvSwitch networking since the default networking is lxcbr0 and also in the original container non-default rac-meaningful names for the veth.pair were used.. The lxc-clone utility did update the MAC addresses just fine, so no issue there. However, the ports on the OpenvSwitch need to be manually edited to have the matching name for the container as shown below (manual edits in bold).
# Template used to create this container: /usr/share/lxc/templates/lxc-oracle
# Parameters passed to the template:
# For additional config options, please look at lxc.container.conf(5)
# Networking
lxc.network.type = veth
lxc.network.flags = up
lxc.network.script.up = /etc/network/if-up.d/lxc-ifup-sw1
lxc.network.script.down = /etc/network/if-down.d/lxc-ifdown-sw1
lxc.network.veth.pair = lxcora02-pub
lxc.network.name = eth0
lxc.network.mtu = 1500
lxc.network.hwaddr = 00:16:3e:16:14:93 <-- Note: lxc-clone updates MAC address automatically as part of the clone process, no manual edit needed nere.
lxc.network.type = veth
lxc.network.flags = up
lxc.network.link = lxcbr0
lxc.network.veth.pair = lxcora02-pri
lxc.network.name = eth1
lxc.network.mtu = 9000
lxc.network.hwaddr = 00:16:3e:3e:a6:b4 <-- Note: lxc-clone updates MAC address automatically as part of the clone process, no manual edit needed here.
# Common configuration
lxc.include = /usr/share/lxc/config/oracle.common.conf
# Container configuration for Oracle Linux 6.5
lxc.arch = x86_64
lxc.cap.drop = sys_resource
lxc.cap.drop = setfcap setpcap
# Devices
lxc.cgroup.devices.allow = a # Allow access to all devices
lxc.cgroup.devices.allow = a 10:236 rwm
lxc.cgroup.devices.allow = a 252:* rwm
lxc.cgroup.devices.allow = a 8:* rwm
lxc.mount.entry = /dev/mapper /var/lib/lxc/lxcora01/rootfs/dev/mapper none defaults,bind,create=dir 0 0
lxc.mount.auto = proc sys cgroup
lxc.mount.auto = proc:rw sys:rw cgroup-full:rw
lxc.aa_allow_incomplete = 1
# GLS 2014-10-23 NFS
lxc.aa_profile = unconfined
lxc.rootfs = /var/lib/lxc/lxcora02/rootfs
lxc.utsname = lxcora02 <-- Note: lxc-clone updates name automatically, no manual edit needed here.
Update 28 October 2014
Now using 4 interfaces for Oracle GI HAIP. The config file is changed for both lxcora02 and lxcora03 as shown below.
root@vmem1:/var/lib/lxc/lxcora02# cat config
# Template used to create this container: /usr/share/lxc/templates/lxc-oracle
# Parameters passed to the template:
# For additional config options, please look at lxc.container.conf(5)
# Networking
lxc.network.type = veth
lxc.network.flags = up
lxc.network.script.up = /etc/network/if-up.d/lxcora02-ifup-sw1
lxc.network.script.down = /etc/network/if-down.d/lxcora02-ifdown-sw1
lxc.network.veth.pair = lxcora02
lxc.network.name = eth0
lxc.network.mtu = 1500
lxc.network.hwaddr = 00:16:3e:16:14:93
lxc.network.type = veth
lxc.network.flags = up
lxc.network.script.up = /etc/network/if-up.d/lxcora02-priv1-ifup-sw4
lxc.network.script.down = /etc/network/if-down.d/lxcora02-priv1-ifdown-sw4
lxc.network.veth.pair = lxcora02-priv1
lxc.network.name = eth1
lxc.network.mtu = 9000
lxc.network.ipv4 = 192.210.39.10
lxc.network.type = veth
lxc.network.flags = up
lxc.network.script.up = /etc/network/if-up.d/lxcora02-priv2-ifup-sw5
lxc.network.script.down = /etc/network/if-down.d/lxcora02-priv2-ifdown-sw5
lxc.network.veth.pair = lxcora02-priv2
lxc.network.name = eth2
lxc.network.mtu = 9000
lxc.network.ipv4 = 192.211.39.10
lxc.network.type = veth
lxc.network.flags = up
lxc.network.script.up = /etc/network/if-up.d/lxcora02-priv3-ifup-sw6
lxc.network.script.down = /etc/network/if-down.d/lxcora02-priv3-ifdown-sw6
lxc.network.veth.pair = lxcora02-priv3
lxc.network.name = eth3
lxc.network.mtu = 9000
lxc.network.ipv4 = 192.212.39.10
lxc.network.type = veth
lxc.network.flags = up
lxc.network.script.up = /etc/network/if-up.d/lxcora02-priv4-ifup-sw7
lxc.network.script.down = /etc/network/if-down.d/lxcora02-priv4-ifdown-sw7
lxc.network.veth.pair = lxcora02-priv4
lxc.network.name = eth4
lxc.network.mtu = 9000
lxc.network.ipv4 = 192.213.39.10
#lxc.network.type = veth
#lxc.network.flags = up
#lxc.network.link = lxcbr0
#lxc.network.name = eth1
#lxc.network.mtu = 1500
lxc.network.hwaddr = 00:16:3e:3e:a6:b4
# Common configuration
lxc.include = /usr/share/lxc/config/oracle.common.conf
# Container configuration for Oracle Linux 6.5
lxc.arch = x86_64
lxc.cap.drop = sys_resource
lxc.cap.drop = setfcap setpcap
# Devices
lxc.cgroup.devices.allow = a # Allow access to all devices
lxc.cgroup.devices.allow = a 10:236 rwm
lxc.cgroup.devices.allow = a 252:* rwm
lxc.cgroup.devices.allow = a 8:* rwm
lxc.mount.entry = /dev/mapper /var/lib/lxc/lxcora02/rootfs/dev/mapper none defaults,bind,create=dir 0 0
lxc.mount.auto = proc sys cgroup
lxc.mount.auto = proc:rw sys:rw cgroup-full:rw
lxc.aa_allow_incomplete = 1
# GLS 2014-10-23 NFS
lxc.aa_profile = unconfined
lxc.rootfs = /var/lib/lxc/lxcora02/rootfs
lxc.utsname = lxcora02
root@vmem1:/var/lib/lxc/lxcora02#
And here is the config file for lxcora03 as shown below.
root@vmem1:/var/lib/lxc/lxcora03# cat config
# Template used to create this container: /usr/share/lxc/templates/lxc-oracle
# Parameters passed to the template:
# For additional config options, please look at lxc.container.conf(5)
# Networking
lxc.network.type = veth
lxc.network.flags = up
lxc.network.script.up = /etc/network/if-up.d/lxcora03-ifup-sw1
lxc.network.script.down = /etc/network/if-down.d/lxcora03-ifdown-sw1
lxc.network.veth.pair = lxcora03
lxc.network.name = eth0
lxc.network.mtu = 1500
lxc.network.hwaddr = 00:16:3e:88:bc:f6
lxc.network.type = veth
lxc.network.flags = up
lxc.network.script.up = /etc/network/if-up.d/lxcora03-priv1-ifup-sw4
lxc.network.script.down = /etc/network/if-down.d/lxcora03-priv1-ifdown-sw4
lxc.network.veth.pair = lxcora03-priv1
lxc.network.name = eth1
lxc.network.mtu = 9000
lxc.network.ipv4 = 192.210.39.11
lxc.network.type = veth
lxc.network.flags = up
lxc.network.script.up = /etc/network/if-up.d/lxcora03-priv2-ifup-sw5
lxc.network.script.down = /etc/network/if-down.d/lxcora03-priv2-ifdown-sw5
lxc.network.veth.pair = lxcora03-priv2
lxc.network.name = eth2
lxc.network.mtu = 9000
lxc.network.ipv4 = 192.211.39.11
lxc.network.type = veth
lxc.network.flags = up
lxc.network.script.up = /etc/network/if-up.d/lxcora03-priv3-ifup-sw6
lxc.network.script.down = /etc/network/if-down.d/lxcora03-priv3-ifdown-sw6
lxc.network.veth.pair = lxcora03-priv3
lxc.network.name = eth3
lxc.network.mtu = 9000
lxc.network.ipv4 = 192.212.39.11
lxc.network.type = veth
lxc.network.flags = up
lxc.network.script.up = /etc/network/if-up.d/lxcora03-priv4-ifup-sw7
lxc.network.script.down = /etc/network/if-down.d/lxcora03-priv4-ifdown-sw7
lxc.network.veth.pair = lxcora03-priv4
lxc.network.name = eth4
lxc.network.mtu = 9000
lxc.network.ipv4 = 192.213.39.11
#lxc.network.type = veth
#lxc.network.flags = up
#lxc.network.link = lxcbr0
#lxc.network.name = eth1
#lxc.network.mtu = 1500
lxc.network.hwaddr = 00:16:3e:11:f8:13
# Common configuration
lxc.include = /usr/share/lxc/config/oracle.common.conf
# Container configuration for Oracle Linux 6.5
lxc.arch = x86_64
lxc.cap.drop = sys_resource
lxc.cap.drop = setfcap setpcap
# Devices
lxc.cgroup.devices.allow = a # Allow access to all devices
lxc.cgroup.devices.allow = a 10:236 rwm
lxc.cgroup.devices.allow = a 252:* rwm
lxc.cgroup.devices.allow = a 8:* rwm
lxc.mount.entry = /dev/mapper /var/lib/lxc/lxcora03/rootfs/dev/mapper none defaults,bind,create=dir 0 0
lxc.mount.auto = proc sys cgroup
lxc.mount.auto = proc:rw sys:rw cgroup-full:rw
lxc.aa_allow_incomplete = 1
# GLS 2014-10-23 NFS
lxc.aa_profile = unconfined
lxc.rootfs = /var/lib/lxc/lxcora03/rootfs
lxc.utsname = lxcora03
root@vmem1:/var/lib/lxc/lxcora03#
Now start up the container and see what starts up leaving the original container lxcora01 down until verification has been done that the actual hostname and networking etc are correct (non-duplicate). Since this is a first for the author too, all these steps will be carefully documented.
Check the MTU 9000 is Working on the Private Interfaces
This link here at Myles Gray shows how to do this.
Check to see if the new container is visible to lxc-ls as shown below.
gstanden@vmem1:~$ sudo lxc-ls -f
[sudo] password for gstanden:
NAME STATE IPV4 IPV6 GROUPS AUTOSTART
------------------------------------------------
lxcora01 STOPPED - - - NO
lxcora02 STOPPED - - - NO
Here's the startup (with "-F" option used to get console output). The startup is normal and exactly the same as the source LXC container as shown below. The messages about apparmor and ulimit were present in the source LXC container lxcora01 as well, and those I am still looking for an answer/solution about, but they dont' affect the functioning of the LXC container for the Oracle install purposes so for now they are not an issue. The startup of the cloned container is shown below.
gstanden@vmem1:~$ sudo lxc-start -n lxcora02 -F
lxc-start: lsm/apparmor.c: mount_feature_enabled: 61 Permission denied - Error mounting securityfs
Welcome to Oracle Linux Server
Setting hostname lxcora02: [ OK ]
Checking filesystems
[ OK ]
Mounting local filesystems: [ OK ]
Enabling /etc/fstab swaps: [ OK ]
Entering non-interactive startup
Calling the system activity data collector (sadc)...
Bringing up loopback interface: [ OK ]
Bringing up interface eth0:
Determining IP information for eth0... done.
[ OK ]
Bringing up interface eth1:
Determining IP information for eth1... done.
[ OK ]
Starting system logger: [ OK ]
Starting rpcbind: [ OK ]
Starting NFS statd: [ OK ]
NFS filesystems queued to be mounted
Mounting filesystems: [ OK ]
Starting sshd: [ OK ]
hsflowd start: OK
Starting crond: [ OK ]
/etc/rc3.d/S96ohasd: line 153: ulimit: max locked memory: cannot modify limit: Operation not permitted
/etc/rc3.d/S96ohasd: line 155: ulimit: %CRS_LIMIT_OPENFILE%: invalid number
Starting ohasd:
chown: cannot access `/dev/mapper/asm_disk*': No such file or directory
chmod: cannot access `/dev/mapper/asm_disk*': No such file or directory
Oracle Linux Server release 6.5
Kernel 3.13.11.6 on an x86_64
lxcora02 login:
Hostname info looks good as shown below. The hostname has been updated automatically by lxc-clone and correctly shows "lxcora02" as shown below.
[root@lxcora02 ~]# uname -a
Linux lxcora02 3.13.11.6 #1 SMP Mon Sep 15 11:54:55 CDT 2014 x86_64 x86_64 x86_64 GNU/Linux
[root@lxcora02 ~]# hostname -a
lxcora02
[root@lxcora02 ~]# hostname -f
localhost
[root@lxcora02 ~]#
The ethernet interfaces also look fine. As expected they have new MAC addresses, and also have new IP addresses as shown below.
[root@lxcora02 ~]# ifconfig
eth0 Link encap:Ethernet HWaddr 00:16:3E:16:14:93
inet addr:10.207.39.87 Bcast:10.207.39.255 Mask:255.255.255.0
inet6 addr: fe80::216:3eff:fe16:1493/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:386 errors:0 dropped:0 overruns:0 frame:0
TX packets:283 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:61965 (60.5 KiB) TX bytes:31126 (30.3 KiB)
eth1 Link encap:Ethernet HWaddr 00:16:3E:3E:A6:B4
inet addr:10.0.3.236 Bcast:10.0.3.255 Mask:255.255.255.0
inet6 addr: fe80::216:3eff:fe3e:a6b4/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:9000 Metric:1
RX packets:200 errors:0 dropped:0 overruns:0 frame:0
TX packets:15 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:36470 (35.6 KiB) TX bytes:1638 (1.5 KiB)
lo Link encap:Local Loopback
inet addr:127.0.0.1 Mask:255.0.0.0
inet6 addr: ::1/128 Scope:Host
UP LOOPBACK RUNNING MTU:65536 Metric:1
RX packets:44 errors:0 dropped:0 overruns:0 frame:0
TX packets:44 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:3512 (3.4 KiB) TX bytes:3512 (3.4 KiB)
[root@lxcora02 ~]#
However, as might be expected, trying to start cluster and ASM fails. That is shown below.
[root@lxcora02 ~]# crsctl stat res -t
Oracle Clusterware infrastructure error in CRSCTL (OS PID 1227): ADR home path /u00/app/grid/diag/crs/lxcora02/crs does not exist; ADR initialization will try to create it
CRS-4639: Could not contact Oracle High Availability Services
CRS-4000: Command Status failed, or completed with errors.
[root@lxcora02 ~]#
This is ok because for this example first build, I will be ripping out the Oracle RDBMS and ASM installation. The LXC container lxcora01 was more or less the product of learning how to build Oracle ASM 12c EE on LXC containers, so rather than build a fresh container for RAC and then clone it, I've just cloned this one since it was used to work out and do all the RAC GNS networking setups,, but also has all the prerequisites for ASM installation.
For anyone who might be planning on using this guide, the way this would normally be done would be to just build a fresh Oracle Enterprise Linux container on Ubuntu 14.04.1, then do all the Oracle RAC GNS steps, install all the ASM and RDBMS prerequisites, and then do the clone. So in this guide, at least for now, there will be these extra steps documenting the rip out of the existing ASM and RDBMS from the cloned LXC container lxcora02.
I'm going to startup the original lxcora01 container to verify it still runs normally with lxcora02 running at the same time. Connected to lxcora01 and testing DNS lookup shows that the DHCP and DNS combo has added the new cloned container to the bind9 DNS automatically (thanks to the configuration skills learned from BigDino!) as shown below displaying the correct IP address lookup information for the cloned lxcora02 LXC container (note: this new container was NOT added to DNS manually. It was automatically added to the DNS configuration).
[root@lxcora01 ~]# nslookup lxcora02
Server: 10.207.39.1
Address: 10.207.39.1#53
Name: lxcora02.vmem.org
Address: 10.207.39.87
Showing how this looks below in the DNS configuration. The bolded lines show how DNS automatically added the new LXC container to /var/lib/bind/fwd.vmem.org file. Also, note to self, see how the fwd.vmem.org.jnl file is not created until a new DNS entry is automatically added. Last night the static entries for GNS were added to the fwd.vmem.org (and also to the rev.vmem.org) and the serial updated, and at that time, all that was done was that bind9 was restarted to put the GNS changes into DNS. At that time, no fwd.vmem.org.jnl file was created. I had been wondering what triggered the creation of the *.jnl zone files, and now it's clear that when a new LXC container (or KVM guest) i.e. any "new server" appears on the network, then a *.jnl file is created for the related zone files, in this case fwd.vmem.org.jnl and rev.vmem.org.jnl were both created just now when lxcora02 was automatically added.
root@vmem1:/var/lib/bind# cat fwd.vmem.org
$ORIGIN .
$TTL 86400 ; 1 day
vmem.org IN SOA vmem1.vmem.org. postmaster.vmem.org. (
1410240113 ; serial
60 ; refresh (1 minute)
1800 ; retry (30 minutes)
604800 ; expire (1 week)
86400 ; minimum (1 day)
)
NS vmem1.vmem.org.
$ORIGIN vmem.org.
_sflow._udp TXT "txtvers=1" "polling=20" "sampling=512"
SRV 0 0 6343 vmem1
$TTL 3600 ; 1 hour
lxcora01 A 10.207.39.85
TXT "001060b0cef034d0d8b3ec569ea6615637"
lxcora02 A 10.207.39.87
TXT "001d6eb17bf9f9081897f98548b528bfee"
$TTL 86400 ; 1 day
lxcora1 A 10.207.39.79
$TTL 3600 ; 1 hour
lxcora3 A 10.207.39.83
TXT "007e7554cced9024a1bac46cc4370293c6"
lxcora5 A 10.207.39.84
TXT "0092ff209913ebc9ca832c943e48eff86f"
lxcrac NS lxcrac-gns
$TTL 86400 ; 1 day
lxcrac-gns A 10.207.39.3
oracle631 A 10.207.39.72
oracle632 A 10.207.39.76
oracle652 A 10.207.39.74
vmem1 A 10.207.39.1
$TTL 3600 ; 1 hour
vmem2 A 10.207.39.81
TXT "3118a6c1688c312b822454df8baa10bfff"
root@vmem1:/var/lib/bind# pwd
/var/lib/bind
root@vmem1:/var/lib/bind# ls -lrt *.jnl
-rw-r--r-- 1 bind bind 791 Oct 24 16:26 fwd.vmem.org.jnl <-- *.jnl files were created when lxcora02 was automatically added to the zone files
-rw-r--r-- 1 bind bind 774 Oct 24 16:26 rev.vmem.org.jnl <--
root@vmem1:/var/lib/bind#
For completeness, here's the rev.vmem.org zone file showing how lxcora02 was automatically added here too.
root@vmem1:/var/lib/bind# cat rev.vmem.org
$ORIGIN .
$TTL 86400 ; 1 day
39.207.10.in-addr.arpa IN SOA vmem1.vmem.org. postmaster.vmem.org. (
1410232201 ; serial
3600 ; refresh (1 hour)
1800 ; retry (30 minutes)
604800 ; expire (1 week)
86400 ; minimum (1 day)
)
NS vmem1.vmem.org.
$ORIGIN 39.207.10.in-addr.arpa.
1 PTR vmem1.vmem.org.
3 PTR lxcrac-gns.vmem.org.
72 PTR oracle631.vmem.org.
76 PTR oracle632.vmem.org.
$TTL 3600 ; 1 hour
81 PTR vmem2.vmem.org.
85 PTR lxcora01.vmem.org.
87 PTR lxcora02.vmem.org.
$TTL 86400 ; 1 day
90 PTR oracle635.vmem.org.
39.207.10.in-addr.arpa NS vmem1.vmem.org.
vmem1 A 10.207.39.1
root@vmem1:/var/lib/bind#
Some ping and resolution and ssh tests are done as shown below. Note: the ssh from one node to the other took a few seconds - maybe 10 seconds or so - it's not as fast as I would like. Probably there are some tweaks that a network guru would know to speed the ssh and name resolution up a bit so that connection was much faster. It's not terribly slow, but slower than it should be. Here are the ping and resolution tests as shown below which all work as expected and required.
gstanden@vmem1:~$ ssh root@lxcora01
root@lxcora01's password:
Last login: Fri Oct 24 17:49:59 2014
[root@lxcora01 ~]# nslookup lxcora02
Server: 10.207.39.1
Address: 10.207.39.1#53
Name: lxcora02.vmem.org
Address: 10.207.39.87
[root@lxcora01 ~]# ping -c 3 lxcora02
PING lxcora02.vmem.org (10.207.39.87) 56(84) bytes of data.
64 bytes from lxcora02.vmem.org (10.207.39.87): icmp_seq=1 ttl=64 time=0.868 ms
64 bytes from lxcora02.vmem.org (10.207.39.87): icmp_seq=2 ttl=64 time=0.075 ms
64 bytes from lxcora02.vmem.org (10.207.39.87): icmp_seq=3 ttl=64 time=0.124 ms
--- lxcora02.vmem.org ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2003ms
rtt min/avg/max/mdev = 0.075/0.355/0.868/0.363 ms
[root@lxcora01 ~]# ssh root@lxcora02
The authenticity of host 'lxcora02 (10.207.39.87)' can't be established.
RSA key fingerprint is 4f:62:a6:23:e4:58:2e:e2:1b:a4:1c:39:28:a0:ff:df.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'lxcora02,10.207.39.87' (RSA) to the list of known hosts.
root@lxcora02's password:
Last login: Fri Oct 24 17:48:37 2014
[root@lxcora02 ~]# nslookup lxcora01
Server: 10.207.39.1
Address: 10.207.39.1#53
Name: lxcora01.vmem.org
Address: 10.207.39.85
[root@lxcora02 ~]# ping -c 3 lxcora01
PING lxcora01.vmem.org (10.207.39.85) 56(84) bytes of data.
64 bytes from lxcora01.vmem.org (10.207.39.85): icmp_seq=1 ttl=64 time=0.481 ms
64 bytes from lxcora01.vmem.org (10.207.39.85): icmp_seq=2 ttl=64 time=0.073 ms
64 bytes from lxcora01.vmem.org (10.207.39.85): icmp_seq=3 ttl=64 time=0.092 ms
--- lxcora01.vmem.org ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2001ms
rtt min/avg/max/mdev = 0.073/0.215/0.481/0.188 ms
[root@lxcora02 ~]# exit
logout
Connection to lxcora02 closed.
[root@lxcora01 ~]#
It can be seen that lxcora01 Oracle ASM and RDBMS are running fine with both the lxcora01 and it's clone lxcora02 up and running at the same time as shown below.
[root@lxcora01 ~]# crsctl stat res -t
--------------------------------------------------------------------------------
Name Target State Server State details
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.DATA.dg
ONLINE ONLINE lxcora01 STABLE
ora.FRA.dg
ONLINE ONLINE lxcora01 STABLE
ora.IOPS.dg
ONLINE ONLINE lxcora01 STABLE
ora.LISTENER.lsnr
ONLINE ONLINE lxcora01 STABLE
ora.SYSTEMDG.dg
ONLINE ONLINE lxcora01 STABLE
ora.asm
ONLINE ONLINE lxcora01 Started,STABLE
ora.ons
OFFLINE OFFLINE lxcora01 STABLE
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.cssd
1 ONLINE ONLINE lxcora01 STABLE
ora.diskmon
1 OFFLINE OFFLINE STABLE
ora.evmd
1 ONLINE ONLINE lxcora01 STABLE
ora.vmem1.db
1 ONLINE ONLINE lxcora01 Open,STABLE
--------------------------------------------------------------------------------
[root@lxcora01 ~]#
The next steps which I will do after driving mom home from her coffee outing will be to (a) rip out the ASM and RDBMS on the lxcora02; and then (b) do an lxc-clone of lxcora02 to lxcora03 which will give a nice clean RAC node so that there is no need to rip out the ASM and RDBMS twice.
Here is the way that the OpenvSwitches look with both LXC containers running. Note that sw1 is my main OpenvSwitch for public (management) network, while sw2 and sw3 are basically created for iscsi SAN storage networks (not being used with LXC because I don't really have a good handle on how to present iSCSI block devices to LXC containers since iscsi packages don't seem to work in the containers). This RAC is going to use NFS as the storage, side note. Here's the OpenvSwitch detail as shown below. The ports to sw1 for lxcora01 and lxcora02 are shown in bold below.
gstanden@vmem1:~$ sudo ovs-vsctl show
[sudo] password for gstanden:
2fc24710-34b5-4aa2-a32d-4e7bcb1afa1a
Bridge "sw2"
Port "sw2"
Interface "sw2"
type: internal
Port "t5"
Interface "t5"
Port "t1"
Interface "t1"
Port "t6"
Interface "t6"
Port "t3"
Interface "t3"
Port "t2"
Interface "t2"
Port "t4"
Interface "t4"
Bridge "sw1"
Port "lxcora02-pub"
Interface "lxcora02-pub"
Port "s2"
Interface "s2"
Port "s4"
Interface "s4"
Port "s1"
Interface "s1"
Port "sw1"
Interface "sw1"
type: internal
Port "s6"
Interface "s6"
Port "s5"
Interface "s5"
Port "lxcora01-pub"
Interface "lxcora01-pub"
Port "s3"
Interface "s3"
Bridge "sw3"
Port "w4"
Interface "w4"
Port "w2"
Interface "w2"
Port "w3"
Interface "w3"
Port "w6"
Interface "w6"
Port "w5"
Interface "w5"
Port "w1"
Interface "w1"
Port "sw3"
Interface "sw3"
type: internal
ovs_version: "2.0.2"
gstanden@vmem1:~$
So now I have taken my mother home to her condo in my motorcar (an Eclipse Special Edition (SE) 2012) and I have watched with my wife the movie Out of Africa (1985) with Meryl Streep and Robert Redford on Turner Classic Movies (TCM) and we have shared a bottle of Trader Joe "3 Buck Chuck" white wine, and now I am going to rip out Oracle RDBMS and Oracle ASM from lxcora02 in preparation for making a clone of lxcora02 to lxcora03 and then hopefully building a 2-node RAC on the lxc nodes lxcora02 and lxcora03. Here is the first step in that process as shown below, starting DBCA so I can delete the database and database binaries. I will not show those screenshots, I already know how to do that :-) ! )(smiling man smoking a cigar!).
gstanden@vmem1:~$ ssh -Y -C oracle@lxcora02
The authenticity of host 'lxcora02 (10.207.39.87)' can't be established.
RSA key fingerprint is 4f:62:a6:23:e4:58:2e:e2:1b:a4:1c:39:28:a0:ff:df.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'lxcora02,10.207.39.87' (RSA) to the list of known hosts.
oracle@lxcora02's password:
Permission denied, please try again.
oracle@lxcora02's password:
Last login: Sun Sep 21 23:45:02 2014 from 10.0.3.1
[oracle@lxcora02 ~]$ xclock
[oracle@lxcora02 ~]$ dbca
The database had been deleted using DBCA. Next the RDBMS home and binaries are deconfigured using 12c deinstall utility as shown below. For this I refreshed my memory on how to do this at Thinking Out Loud weblog here.
gstanden@vmem1:~$ ssh -Y -C oracle@lxcora02
oracle@lxcora02's password:
Last login: Fri Oct 24 22:55:08 2014 from vmem1.vmem.org
[oracle@lxcora02 ~]$ $ORACLE_HOME/deinstall/deinstall
Checking for required files and bootstrapping ...
Please wait ...
Location of logs /u00/app/oraInventory/logs/
############ ORACLE DECONFIG TOOL START ############
######################### DECONFIG CHECK OPERATION START #########################
## [START] Install check configuration ##
Checking for existence of the Oracle home location /u00/app/oracle/product/12.1.0/dbhome_1
Oracle Home type selected for deinstall is: Oracle Single Instance Database
Oracle Base selected for deinstall is: /u00/app/oracle
Checking for existence of central inventory location /u00/app/oraInventory
## [END] Install check configuration ##
Network Configuration check config START
Network de-configuration trace file location: /u00/app/oraInventory/logs/netdc_check2014-10-24_11-15-21-PM.log
Network Configuration check config END
Database Check Configuration START
Database de-configuration trace file location: /u00/app/oraInventory/logs/databasedc_check2014-10-24_11-15-27-PM.log
Use comma as separator when specifying list of values as input
Specify the list of database names that are configured in this Oracle home []: VMEM1
###### For Database 'VMEM1' ######
Specify the type of this database (1.Single Instance Database|2.Oracle Restart Enabled Database) [1]: 2
Specify the list of nodes on which this database has instances [lxcora02]:
Specify the diagnostic destination location of the database [/u00/app/oracle/diag/rdbms/vmem1]:
Specify the storage type used by the Database ASM|FS []: ASM
Database Check Configuration END
Oracle Configuration Manager check START
OCM check log file location : /u00/app/oraInventory/logs//ocm_check9034.log
Oracle Configuration Manager check END
######################### DECONFIG CHECK OPERATION END #########################
####################### DECONFIG CHECK OPERATION SUMMARY #######################
Oracle Home selected for deinstall is: /u00/app/oracle/product/12.1.0/dbhome_1
Inventory Location where the Oracle home registered is: /u00/app/oraInventory
The following databases were selected for de-configuration : VMEM1
Database unique name : VMEM1
Storage used : ASM
Checking the config status for CCR
Oracle Home exists and CCR is configured
CCR check is finished
Do you want to continue (y - yes, n - no)? [n]: y
A log of this session will be written to: '/u00/app/oraInventory/logs/deinstall_deconfig2014-10-24_11-15-21-PM.out'
Any error messages from this session will be written to: '/u00/app/oraInventory/logs/deinstall_deconfig2014-10-24_11-15-21-PM.err'
######################## DECONFIG CLEAN OPERATION START ########################
Database de-configuration trace file location: /u00/app/oraInventory/logs/databasedc_clean2014-10-24_11-16-02-PM.log
Database Clean Configuration START VMEM1
This operation may take few minutes.
Database Clean Configuration END VMEM1
Network Configuration clean config START
Network de-configuration trace file location: /u00/app/oraInventory/logs/netdc_clean2014-10-24_11-16-35-PM.log
De-configuring Naming Methods configuration file...
Naming Methods configuration file de-configured successfully.
De-configuring Local Net Service Names configuration file...
Local Net Service Names configuration file de-configured successfully.
De-configuring backup files...
Backup files de-configured successfully.
The network configuration has been cleaned up successfully.
Network Configuration clean config END
Oracle Configuration Manager clean START
OCM clean log file location : /u00/app/oraInventory/logs//ocm_clean9034.log
Oracle Configuration Manager clean END
######################### DECONFIG CLEAN OPERATION END #########################
####################### DECONFIG CLEAN OPERATION SUMMARY #######################
Successfully de-configured the following database instances : VMEM1
Cleaning the config for CCR
Cleaning the CCR configuration by executing its binaries
CCR clean is finished
#######################################################################
############# ORACLE DECONFIG TOOL END #############
Using properties file /tmp/deinstall2014-10-24_11-15-03PM/response/deinstall_2014-10-24_11-15-21-PM.rsp
Location of logs /u00/app/oraInventory/logs/
############ ORACLE DEINSTALL TOOL START ############
####################### DEINSTALL CHECK OPERATION SUMMARY #######################
A log of this session will be written to: '/u00/app/oraInventory/logs/deinstall_deconfig2014-10-24_11-15-21-PM.out'
Any error messages from this session will be written to: '/u00/app/oraInventory/logs/deinstall_deconfig2014-10-24_11-15-21-PM.err'
######################## DEINSTALL CLEAN OPERATION START ########################
## [START] Preparing for Deinstall ##
Setting LOCAL_NODE to localhost
Setting CRS_HOME to false
Setting oracle.installer.invPtrLoc to /tmp/deinstall2014-10-24_11-15-03PM/oraInst.loc
Setting oracle.installer.local to false
## [END] Preparing for Deinstall ##
Setting the force flag to false
Setting the force flag to cleanup the Oracle Base
Oracle Universal Installer clean START
Detach Oracle home '/u00/app/oracle/product/12.1.0/dbhome_1' from the central inventory on the local node : Done
Delete directory '/u00/app/oracle/product/12.1.0/dbhome_1' on the local node : Done
The Oracle Base directory '/u00/app/oracle' will not be removed on local node. The directory is not empty.
Oracle Universal Installer cleanup was successful.
Oracle Universal Installer clean END
## [START] Oracle install clean ##
Clean install operation removing temporary directory '/tmp/deinstall2014-10-24_11-15-03PM' on node 'localhost'
## [END] Oracle install clean ##
######################### DEINSTALL CLEAN OPERATION END #########################
####################### DEINSTALL CLEAN OPERATION SUMMARY #######################
Successfully detached Oracle home '/u00/app/oracle/product/12.1.0/dbhome_1' from the central inventory on the local node.
Successfully deleted directory '/u00/app/oracle/product/12.1.0/dbhome_1' on the local node.
Oracle Universal Installer cleanup was successful.
Oracle deinstall tool successfully cleaned up temporary directories.
#######################################################################
############# ORACLE DEINSTALL TOOL END #############
[oracle@lxcora02 ~]$
Next the Grid Infrastructure is deconfigured, again refreshed my memory from Thinking Out Loud.
gstanden@vmem1:~$ ssh grid@lxcora02
grid@lxcora02's password:
Last login: Sun Sep 21 23:28:24 2014 from 10.0.3.1
[grid@lxcora02 ~]$ clear
[grid@lxcora02 ~]$ echo $GRID_HOME
/u00/app/grid/product/12.1.0/grid
[grid@lxcora02 ~]$ $GRID_HOME/deinstall/deinstall
Checking for required files and bootstrapping ...
Please wait ...
Location of logs /tmp/deinstall2014-10-24_11-21-17PM/logs/
############ ORACLE DECONFIG TOOL START ############
######################### DECONFIG CHECK OPERATION START #########################
## [START] Install check configuration ##
Checking for existence of the Oracle home location /u00/app/grid/product/12.1.0/grid
Oracle Home type selected for deinstall is: Oracle Grid Infrastructure for a Standalone Server
Oracle Base selected for deinstall is: /u00/app/grid
Checking for existence of central inventory location /u00/app/oraInventory
Checking for existence of the Oracle Grid Infrastructure home /u00/app/grid/product/12.1.0/grid
## [END] Install check configuration ##
Traces log file: /tmp/deinstall2014-10-24_11-21-17PM/logs//crsdc_2014-10-24_11-21-36PM.log
Network Configuration check config START
Network de-configuration trace file location: /tmp/deinstall2014-10-24_11-21-17PM/logs/netdc_check2014-10-24_11-21-37-PM.log
Specify all Oracle Restart enabled listeners that are to be de-configured. Enter .(dot) to deselect all. [LISTENER]:
Network Configuration check config END
Asm Check Configuration START
ASM de-configuration trace file location: /tmp/deinstall2014-10-24_11-21-17PM/logs/asmcadc_check2014-10-24_11-21-43-PM.log
Automatic Storage Management (ASM) instance is detected in this Oracle home /u00/app/grid/product/12.1.0/grid.
ASM Diagnostic Destination : /u00/app/grid
ASM Diskgroups :
ASM diskstring : <Default>
Diskgroups will not be dropped
If you want to retain the existing diskgroups or if any of the information detected is incorrect, you can modify by entering 'y'. Do you want to modify above information (y|n) [n]:
Database Check Configuration START
Database de-configuration trace file location: /tmp/deinstall2014-10-24_11-21-17PM/logs/databasedc_check2014-10-24_11-22-03-PM.log
Database Check Configuration END
######################### DECONFIG CHECK OPERATION END #########################
####################### DECONFIG CHECK OPERATION SUMMARY #######################
Oracle Grid Infrastructure Home is: /u00/app/grid/product/12.1.0/grid
The following nodes are part of this cluster: null
The cluster node(s) on which the Oracle home deinstallation will be performed are:null
Oracle Home selected for deinstall is: /u00/app/grid/product/12.1.0/grid
Inventory Location where the Oracle home registered is: /u00/app/oraInventory
Following Oracle Restart enabled listener(s) will be de-configured: LISTENER
ASM instance will be de-configured from this Oracle home
Do you want to continue (y - yes, n - no)? [n]: y
A log of this session will be written to: '/tmp/deinstall2014-10-24_11-21-17PM/logs/deinstall_deconfig2014-10-24_11-21-36-PM.out'
Any error messages from this session will be written to: '/tmp/deinstall2014-10-24_11-21-17PM/logs/deinstall_deconfig2014-10-24_11-21-36-PM.err'
######################## DECONFIG CLEAN OPERATION START ########################
Database de-configuration trace file location: /tmp/deinstall2014-10-24_11-21-17PM/logs/databasedc_clean2014-10-24_11-22-19-PM.log
ASM de-configuration trace file location: /tmp/deinstall2014-10-24_11-21-17PM/logs/asmcadc_clean2014-10-24_11-22-19-PM.log
ASM Clean Configuration START
ASM Clean Configuration END
Network Configuration clean config START
Network de-configuration trace file location: /tmp/deinstall2014-10-24_11-21-17PM/logs/netdc_clean2014-10-24_11-22-22-PM.log
De-configuring Oracle Restart enabled listener(s): LISTENER
De-configuring listener: LISTENER
Stopping listener: LISTENER
Warning: Failed to stop listener. Listener may not be running.
Deleting listener: LISTENER
Listener deleted successfully.
Listener de-configured successfully.
De-configuring Listener configuration file...
Listener configuration file de-configured successfully.
De-configuring Naming Methods configuration file...
Naming Methods configuration file de-configured successfully.
De-configuring Local Net Service Names configuration file...
Local Net Service Names configuration file de-configured successfully.
De-configuring backup files...
Backup files de-configured successfully.
The network configuration has been cleaned up successfully.
Network Configuration clean config END
---------------------------------------->
Run the following command as the root user or the administrator on node "localhost".
/tmp/deinstall2014-10-24_11-21-17PM/perl/bin/perl -I/tmp/deinstall2014-10-24_11-21-17PM/perl/lib -I/tmp/deinstall2014-10-24_11-21-17PM/crs/install /tmp/deinstall2014-10-24_11-21-17PM/crs/install/roothas.pl -force -deconfig -paramfile "/tmp/deinstall2014-10-24_11-21-17PM/response/deinstall_OraGI12Home1.rsp"
Press Enter after you finish running the above commands
<----------------------------------------
######################### DECONFIG CLEAN OPERATION END #########################
####################### DECONFIG CLEAN OPERATION SUMMARY #######################
ASM instance was de-configured successfully from the Oracle home
Following Oracle Restart enabled listener(s) were de-configured successfully: LISTENER
Oracle Restart was already stopped and de-configured on node "localhost"
Oracle Restart is stopped and de-configured successfully.
#######################################################################
############# ORACLE DECONFIG TOOL END #############
Using properties file /tmp/deinstall2014-10-24_11-21-17PM/response/deinstall_2014-10-24_11-21-36-PM.rsp
Location of logs /tmp/deinstall2014-10-24_11-21-17PM/logs/
############ ORACLE DEINSTALL TOOL START ############
####################### DEINSTALL CHECK OPERATION SUMMARY #######################
A log of this session will be written to: '/tmp/deinstall2014-10-24_11-21-17PM/logs/deinstall_deconfig2014-10-24_11-21-36-PM.out'
Any error messages from this session will be written to: '/tmp/deinstall2014-10-24_11-21-17PM/logs/deinstall_deconfig2014-10-24_11-21-36-PM.err'
######################## DEINSTALL CLEAN OPERATION START ########################
## [START] Preparing for Deinstall ##
Setting LOCAL_NODE to localhost
Setting CRS_HOME to true
Setting oracle.installer.invPtrLoc to /tmp/deinstall2014-10-24_11-21-17PM/oraInst.loc
Setting oracle.installer.local to false
## [END] Preparing for Deinstall ##
Setting the force flag to false
Setting the force flag to cleanup the Oracle Base
Oracle Universal Installer clean START
Detach Oracle home '/u00/app/grid/product/12.1.0/grid' from the central inventory on the local node : Done
Delete directory '/u00/app/grid/product/12.1.0/grid' on the local node : Done
Delete directory '/u00/app/oraInventory' on the local node : Done
The Oracle Base directory '/u00/app/grid' will not be removed on local node. The directory is not empty.
Oracle Universal Installer cleanup was successful.
Oracle Universal Installer clean END
## [START] Oracle install clean ##
Clean install operation removing temporary directory '/tmp/deinstall2014-10-24_11-21-17PM' on node 'localhost'
## [END] Oracle install clean ##
######################### DEINSTALL CLEAN OPERATION END #########################
####################### DEINSTALL CLEAN OPERATION SUMMARY #######################
Successfully detached Oracle home '/u00/app/grid/product/12.1.0/grid' from the central inventory on the local node.
Successfully deleted directory '/u00/app/grid/product/12.1.0/grid' on the local node.
Successfully deleted directory '/u00/app/oraInventory' on the local node.
Oracle Universal Installer cleanup was successful.
Run 'rm -r /etc/oraInst.loc' as root on node(s) 'localhost' at the end of the session.
Run 'rm -r /opt/ORCLfmap' as root on node(s) 'localhost' at the end of the session.
Run 'rm -r /etc/oratab' as root on node(s) 'localhost' at the end of the session.
Oracle deinstall tool successfully cleaned up temporary directories.
#######################################################################
############# ORACLE DEINSTALL TOOL END #############
[grid@lxcora02 ~]$
And here are the results of running the root portion of the GI deinstall as shown below.
gstanden@vmem1:~$ ssh root@lxcora02
root@lxcora02's password:
Last login: Fri Oct 24 23:10:13 2014
[root@lxcora02 ~]# /tmp/deinstall2014-10-24_11-21-17PM/perl/bin/perl -I/tmp/deinstall2014-10-24_11-21-17PM/perl/lib -I/tmp/deinstall2014-10-24_11-21-17PM/crs/install /tmp/deinstall2014-10-24_11-21-17PM/crs/install/roothas.pl -force -deconfig -paramfile "/tmp/deinstall2014-10-24_11-21-17PM/response/deinstall_OraGI12Home1.rsp"
Using configuration parameter file: /tmp/deinstall2014-10-24_11-21-17PM/response/deinstall_OraGI12Home1.rsp
CRS-4639: Could not contact Oracle High Availability Services
CRS-4000: Command Stop failed, or completed with errors.
CRS-4639: Could not contact Oracle High Availability Services
CRS-4000: Command Delete failed, or completed with errors.
CLSU-00100: operating system function: opendir failed with error data: 2
CLSU-00101: operating system error message: No such file or directory
CLSU-00103: error location: scrsearch1
CLSU-00104: additional error information: cant open scr home dir scls_scr_getval
CRS-4639: Could not contact Oracle High Availability Services
CRS-4000: Command Stop failed, or completed with errors.
2014/10/24 23:24:13 CLSRSC-337: Successfully deconfigured Oracle Restart stack
[root@lxcora02 ~]#
Remove the entire $ORACLE_HOME and $GRID_HOME top level directory, u00 as shown below.
[root@lxcora02 /]# rm -Rf u00
[root@lxcora02 /]#
Configure and Test Jumbo Frames on Private Interconnect Interfaces
[root@lxcora02 ~]# ping -c 3 -M do -s 8972 10.207.39.11
PING 10.207.39.11 (10.207.39.11) 8972(9000) bytes of data.
ping: local error: Message too long, mtu=1500
ping: local error: Message too long, mtu=1500
ping: local error: Message too long, mtu=1500
--- 10.207.39.11 ping statistics ---
3 packets transmitted, 0 received, +3 errors, 100% packet loss, time 3012ms
[root@lxcora02 ~]# ping -c 3 -M do -s 8972 192.210.39.11
PING 192.210.39.11 (192.210.39.11) 8972(9000) bytes of data.
8980 bytes from 192.210.39.11: icmp_seq=1 ttl=64 time=0.189 ms
8980 bytes from 192.210.39.11: icmp_seq=2 ttl=64 time=0.062 ms
8980 bytes from 192.210.39.11: icmp_seq=3 ttl=64 time=0.047 ms
--- 192.210.39.11 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 1999ms
rtt min/avg/max/mdev = 0.047/0.099/0.189/0.064 ms
[root@lxcora02 ~]# ping -c 3 -M do -s 8972 192.211.39.11
PING 192.211.39.11 (192.211.39.11) 8972(9000) bytes of data.
8980 bytes from 192.211.39.11: icmp_seq=1 ttl=64 time=0.143 ms
8980 bytes from 192.211.39.11: icmp_seq=2 ttl=64 time=0.046 ms
8980 bytes from 192.211.39.11: icmp_seq=3 ttl=64 time=0.062 ms
--- 192.211.39.11 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 1999ms
rtt min/avg/max/mdev = 0.046/0.083/0.143/0.043 ms
[root@lxcora02 ~]# ping -c 3 -M do -s 8972 192.212.39.11
PING 192.212.39.11 (192.212.39.11) 8972(9000) bytes of data.
8980 bytes from 192.212.39.11: icmp_seq=1 ttl=64 time=0.156 ms
8980 bytes from 192.212.39.11: icmp_seq=2 ttl=64 time=0.096 ms
8980 bytes from 192.212.39.11: icmp_seq=3 ttl=64 time=0.068 ms
--- 192.212.39.11 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 1998ms
rtt min/avg/max/mdev = 0.068/0.106/0.156/0.038 ms
[root@lxcora02 ~]# ping -c 3 -M do -s 8972 192.213.39.11
PING 192.213.39.11 (192.213.39.11) 8972(9000) bytes of data.
8980 bytes from 192.213.39.11: icmp_seq=1 ttl=64 time=0.082 ms
8980 bytes from 192.213.39.11: icmp_seq=2 ttl=64 time=0.066 ms
8980 bytes from 192.213.39.11: icmp_seq=3 ttl=64 time=0.094 ms
--- 192.213.39.11 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 1998ms
rtt min/avg/max/mdev = 0.066/0.080/0.094/0.015 ms
[root@lxcora02 ~]#
Now test from the other node as shown below.
[root@lxcora03 ~]# ping -c 3 -M do -s 8972 10.207.39.10
PING 10.207.39.10 (10.207.39.10) 8972(9000) bytes of data.
ping: local error: Message too long, mtu=1500
ping: local error: Message too long, mtu=1500
ping: local error: Message too long, mtu=1500
--- 10.207.39.10 ping statistics ---
3 packets transmitted, 0 received, +3 errors, 100% packet loss, time 2094ms
[root@lxcora03 ~]# ping -c 3 -M do -s 8972 192.210.39.10
PING 192.210.39.10 (192.210.39.10) 8972(9000) bytes of data.
8980 bytes from 192.210.39.10: icmp_seq=1 ttl=64 time=0.360 ms
8980 bytes from 192.210.39.10: icmp_seq=2 ttl=64 time=0.072 ms
8980 bytes from 192.210.39.10: icmp_seq=3 ttl=64 time=0.065 ms
--- 192.210.39.10 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 1998ms
rtt min/avg/max/mdev = 0.065/0.165/0.360/0.138 ms
[root@lxcora03 ~]# ping -c 3 -M do -s 8972 192.211.39.10
PING 192.211.39.10 (192.211.39.10) 8972(9000) bytes of data.
8980 bytes from 192.211.39.10: icmp_seq=1 ttl=64 time=0.033 ms
8980 bytes from 192.211.39.10: icmp_seq=2 ttl=64 time=0.048 ms
8980 bytes from 192.211.39.10: icmp_seq=3 ttl=64 time=0.041 ms
--- 192.211.39.10 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 1998ms
rtt min/avg/max/mdev = 0.033/0.040/0.048/0.009 ms
[root@lxcora03 ~]# ping -c 3 -M do -s 8972 192.212.39.10
PING 192.212.39.10 (192.212.39.10) 8972(9000) bytes of data.
8980 bytes from 192.212.39.10: icmp_seq=1 ttl=64 time=0.136 ms
8980 bytes from 192.212.39.10: icmp_seq=2 ttl=64 time=0.112 ms
8980 bytes from 192.212.39.10: icmp_seq=3 ttl=64 time=0.040 ms
--- 192.212.39.10 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2000ms
rtt min/avg/max/mdev = 0.040/0.096/0.136/0.040 ms
[root@lxcora03 ~]# ping -c 3 -M do -s 8972 192.213.39.10
PING 192.213.39.10 (192.213.39.10) 8972(9000) bytes of data.
8980 bytes from 192.213.39.10: icmp_seq=1 ttl=64 time=0.037 ms
8980 bytes from 192.213.39.10: icmp_seq=2 ttl=64 time=0.061 ms
8980 bytes from 192.213.39.10: icmp_seq=3 ttl=64 time=0.045 ms
--- 192.213.39.10 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 1998ms
rtt min/avg/max/mdev = 0.037/0.047/0.061/0.012 ms
[root@lxcora03 ~]#
Install cvuqdisk rpm Package on Both RAC nodes
gstanden@vmem1:~$ ssh root@lxcora02
root@lxcora02's password:
Last login: Tue Oct 28 00:11:19 2014
[root@lxcora02 ~]# cd /home/grid
[root@lxcora02 grid]# cd grid
[root@lxcora02 grid]# cd rpm
[root@lxcora02 rpm]# rpm -Uvh cvuqdisk-1.0.9-1.rpm
Preparing... ########################################### [100%]
Using default group oinstall to install package
1:cvuqdisk ########################################### [100%]
[root@lxcora02 rpm]# ssh lxcora03
root@lxcora03's password:
Last login: Tue Oct 28 00:10:46 2014
[root@lxcora03 ~]# cd /home/grid/grid/rpm
[root@lxcora03 rpm]# rpm -Uvh cvuqdisk-1.0.9-1.rpm
Preparing... ########################################### [100%]
Using default group oinstall to install package
1:cvuqdisk ########################################### [100%]
[root@lxcora03 rpm]#
Establish No-password ssh between RAC nodes
The following steps are done to establish no-password ssh between nodes. The GUI installer can do this automatically in 12c, but still I prefer to do it ahead of time and verify it is set prior to install. The first step was that since the lxcora02 was cloned, it still had the public key in the .ssh directory for lxcora01 so need to delete key and create new for lxcora02 as shown below.
[grid@lxcora02 .ssh]$ ls -lrt
total 16
-rw-r--r-- 1 grid oinstall 395 Sep 18 22:06 id_rsa.pub
-rw------- 1 grid oinstall 1675 Sep 18 22:06 id_rsa
-rw-r--r-- 1 grid oinstall 395 Sep 18 22:06 authorized_keys
-rw-r--r-- 1 grid oinstall 1958 Oct 24 23:34 known_hosts
[grid@lxcora02 .ssh]$ cd ..
[grid@lxcora02 ~]$ rm .ssh/id_rsa*
[grid@lxcora02 ~]$
[grid@lxcora02 ~]$ ssh-keygen -t rsa
Generating public/private rsa key pair.
Enter file in which to save the key (/home/grid/.ssh/id_rsa):
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /home/grid/.ssh/id_rsa.
Your public key has been saved in /home/grid/.ssh/id_rsa.pub.
The key fingerprint is:
60:41:ce:f4:10:a0:1b:3e:5e:5e:77:dd:8d:b6:da:93 grid@lxcora02
The key's randomart image is:
+--[ RSA 2048]----+
| .o*. |
| . + + |
| o = . |
| . o . . . . o |
| + . . S . . + .|
| . + . . . . . |
| . . .. |
| oE |
| . .. |
+-----------------+
[grid@lxcora02 ~]$ cd .ssh
[grid@lxcora02 .ssh]$ ls -lrt
total 16
-rw-r--r-- 1 grid oinstall 395 Sep 18 22:06 authorized_keys
-rw-r--r-- 1 grid oinstall 1958 Oct 24 23:34 known_hosts
-rw-r--r-- 1 grid oinstall 395 Oct 24 23:37 id_rsa.pub
-rw------- 1 grid oinstall 1675 Oct 24 23:37 id_rsa
[grid@lxcora02 .ssh]$ cat authorized_keys id_rsa.pub > new
[grid@lxcora02 .ssh]$ more new
ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAx3VNFs0b7Qr4rOjP3myGVY+F6dm1VpBxcUDm1tdcX/c77SZ/a9l1gkiAX0OcgpyWZII1gUJfm/qVi7U94lI+KTBCq3bzhxA78OnSHnw/eGpk/EuzAZ9+3T9RtJoqv44KBdLOUBU+Q7NxCQAhiaD21qNUp9CWYpHQdt1z4xG9u
fGsQPjK5+6mxFjsk2eJZwXKD2KhaKqgBsULkf2wJq6Hg5S+Gta3uKLz4mSrtx7zeblapL4VJWlX7mqLI77M+QEEt2qienDBtoLet1MEy8UGe4EN3aS8rU2MDyin+FrT2OAYv5DHKuYrm2bFb7loddAf+lANwVWaKzfOl2Yy7i8BSw== grid@lxcora01
ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAqIVZGpnGL5WjAUaQLPCK5sjBdXXoBs7kL827cMySCzR7KeOSTxk4OLDDhVK349B1hJn1eiyTDR7NQ7jd0bu2+IfEn3ICchIg1LMdsLMb3xj93hNOSbcNhjVeqVWqF4GVXKY2lcCLaJQ6dELuhfvdkuZ18uwjhebnnCt0ESf7b
cqwAm7AxQbrqHf2A/7W9iICGdqL2KtYXch5DVYITA2+wmb5Ww1DTd3KNs1aBOCC8N80DZRaOX4DTeZ7zh+LK7lleKB3aTTmiK5tT/3d/HD2BoCdbZbtOd2ro/QDlZLs0Zh9Av6xiGtyaufoeS0ycCIoZoEESqZrSfAu8vkyqMREMw== grid@lxcora02
[grid@lxcora02 .ssh]$ mv new authorized_keys
[grid@lxcora02 .ssh]$ cat authorized_keys
ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAx3VNFs0b7Qr4rOjP3myGVY+F6dm1VpBxcUDm1tdcX/c77SZ/a9l1gkiAX0OcgpyWZII1gUJfm/qVi7U94lI+KTBCq3bzhxA78OnSHnw/eGpk/EuzAZ9+3T9RtJoqv44KBdLOUBU+Q7NxCQAhiaD21qNUp9CWYpHQdt1z4xG9ufGsQPjK5+6mxFjsk2eJZwXKD2KhaKqgBsULkf2wJq6Hg5S+Gta3uKLz4mSrtx7zeblapL4VJWlX7mqLI77M+QEEt2qienDBtoLet1MEy8UGe4EN3aS8rU2MDyin+FrT2OAYv5DHKuYrm2bFb7loddAf+lANwVWaKzfOl2Yy7i8BSw== grid@lxcora01
ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAqIVZGpnGL5WjAUaQLPCK5sjBdXXoBs7kL827cMySCzR7KeOSTxk4OLDDhVK349B1hJn1eiyTDR7NQ7jd0bu2+IfEn3ICchIg1LMdsLMb3xj93hNOSbcNhjVeqVWqF4GVXKY2lcCLaJQ6dELuhfvdkuZ18uwjhebnnCt0ESf7bcqwAm7AxQbrqHf2A/7W9iICGdqL2KtYXch5DVYITA2+wmb5Ww1DTd3KNs1aBOCC8N80DZRaOX4DTeZ7zh+LK7lleKB3aTTmiK5tT/3d/HD2BoCdbZbtOd2ro/QDlZLs0Zh9Av6xiGtyaufoeS0ycCIoZoEESqZrSfAu8vkyqMREMw== grid@lxcora02
[grid@lxcora02 .ssh]$
Next copy the authorized_keys file to lxcora01 as shown below.
[grid@lxcora02 .ssh]$ scp authorized_keys grid@lxcora01:/home/grid/.ssh/.
grid@lxcora01's password:
authorized_keys 100% 790 0.8KB/s 00:00
[grid@lxcora02 .ssh]$
Next test that no-password ssh works and login to the nodes as variations of the hostname. Run the test_login.sh script on both nodes until the only output is the date as shown below.
[grid@lxcora02 .ssh]$ chmod 775 test_login.sh
[grid@lxcora02 .ssh]$ ./test_login.sh
Fri Oct 24 23:44:43 EDT 2014
Fri Oct 24 23:44:43 EDT 2014
Fri Oct 24 23:44:43 EDT 2014
Fri Oct 24 23:45:07 EDT 2014
Fri Oct 24 23:45:21 EDT 2014
Fri Oct 24 23:45:35 EDT 2014
The authenticity of host 'lxcora01.vmem.org (10.207.39.85)' can't be established.
RSA key fingerprint is 4f:62:a6:23:e4:58:2e:e2:1b:a4:1c:39:28:a0:ff:df.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'lxcora01.vmem.org' (RSA) to the list of known hosts.
Fri Oct 24 23:46:13 EDT 2014
The authenticity of host 'lxcora02.vmem.org (10.207.39.87)' can't be established.
RSA key fingerprint is 4f:62:a6:23:e4:58:2e:e2:1b:a4:1c:39:28:a0:ff:df.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'lxcora02.vmem.org' (RSA) to the list of known hosts.
Fri Oct 24 23:46:34 EDT 2014
[grid@lxcora02 .ssh]$ ./test_login.sh
Fri Oct 24 23:46:49 EDT 2014
Fri Oct 24 23:46:50 EDT 2014
Fri Oct 24 23:46:50 EDT 2014
Fri Oct 24 23:47:14 EDT 2014
Fri Oct 24 23:47:28 EDT 2014
Fri Oct 24 23:47:42 EDT 2014
Fri Oct 24 23:48:01 EDT 2014
Fri Oct 24 23:48:20 EDT 2014
[grid@lxcora02 .ssh]$ cat test_login.sh
ssh grid@localhost date
ssh grid@127.0.0.1 date
ssh grid@lxcora02 date
ssh grid@lxcora01 date
ssh grid@10.207.39.87 date
ssh grid@10.207.39.85 date
ssh grid@lxcora01.vmem.org date
ssh grid@lxcora02.vmem.org date
[grid@lxcora02 .ssh]$
Now scp the script to the other node and run until output is only date and time on the other node (lxcora01) as shown below.
[grid@lxcora02 .ssh]$ scp test_login.sh grid@lxcora01:/home/grid/.
test_login.sh 100% 212 0.2KB/s 00:00
[grid@lxcora02 .ssh]$
Then login to lxcora01 as grid and run test_login.sh script until output is only date and time as shown below.
[grid@lxcora01 ~]$ ./test_login.sh
Fri Oct 24 23:52:18 EDT 2014
Fri Oct 24 23:52:18 EDT 2014
The authenticity of host 'lxcora02 (10.207.39.87)' can't be established.
RSA key fingerprint is 4f:62:a6:23:e4:58:2e:e2:1b:a4:1c:39:28:a0:ff:df.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'lxcora02,10.207.39.87' (RSA) to the list of known hosts.
Fri Oct 24 23:52:45 EDT 2014
Fri Oct 24 23:52:45 EDT 2014
Fri Oct 24 23:52:59 EDT 2014
The authenticity of host '10.207.39.85 (10.207.39.85)' can't be established.
RSA key fingerprint is 4f:62:a6:23:e4:58:2e:e2:1b:a4:1c:39:28:a0:ff:df.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added '10.207.39.85' (RSA) to the list of known hosts.
Fri Oct 24 23:53:14 EDT 2014
The authenticity of host 'lxcora01.vmem.org (10.207.39.85)' can't be established.
RSA key fingerprint is 4f:62:a6:23:e4:58:2e:e2:1b:a4:1c:39:28:a0:ff:df.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'lxcora01.vmem.org' (RSA) to the list of known hosts.
Fri Oct 24 23:53:35 EDT 2014
The authenticity of host 'lxcora02.vmem.org (10.207.39.87)' can't be established.
RSA key fingerprint is 4f:62:a6:23:e4:58:2e:e2:1b:a4:1c:39:28:a0:ff:df.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'lxcora02.vmem.org' (RSA) to the list of known hosts.
Fri Oct 24 23:53:55 EDT 2014
[grid@lxcora01 ~]$ ./test_login.sh
Fri Oct 24 23:54:00 EDT 2014
Fri Oct 24 23:54:00 EDT 2014
Fri Oct 24 23:54:24 EDT 2014
Fri Oct 24 23:54:24 EDT 2014
Fri Oct 24 23:54:38 EDT 2014
Fri Oct 24 23:54:52 EDT 2014
Fri Oct 24 23:55:12 EDT 2014
Fri Oct 24 23:55:31 EDT 2014
[grid@lxcora01 ~]$
Now it is necessary to setup the final NFS mount configurations and also setup for Normal Redundancy (mirrored). So change in plans. After looking at the Oracle 12c ASM guide find that NFS is no longer supported for OCR and VOTE as it was in 11gR2. So, will use lxc-device command to add the required non-multipath disk onto the lxc RAC nodes. Actually, this was already done. These are LUNS delivered from a KVM guest (oracle652.vmem.org) running SCST Linux SAN (used because it has a switch for 4K / 512 logical setting and 4K / 512 physical setting of sector sizes, so that LUNs can be delivered for any format combination. The Ubuntu 14.04.1 laptop OS is the iSCSI client. The LUNs are then added to the lxc nodes using the lxc-device add command which is persistend across lxc reboots. It would be preferable to use some sort of UDEV rule in the lxc nodes to ensure that the lun is persistent across reboots with respect to actual identity.
At the moment I need to resize the iSCSI LUN so I use the method at my blog documented here. For completeness of this blog, the steps are shown below.
[root@oracle651 scst_lxcora01]# dd if=/dev/zero of=/scst_lxcora01/AsmTst_02.img bs=512 count=0 seek=10M
0+0 records in
0+0 records out
0 bytes (0 B) copied, 0.000411897 s, 0.0 kB/s
[root@oracle651 scst_lxcora01]# ls -lrt AsmTst_02.img
total 377860
-rw-r--r-- 1 root root 5368709120 Oct 25 00:01 AsmTst_02.img
[root@oracle651 scst_lxcora01]# scstadmin -resync_dev AsmTst_02
Collecting current configuration: done.
-> Making requested changes.
-> Setting device attribute 'resync_size' to value '1' for device 'AsmTst_02': done.
-> Done.
All done.
[root@oracle651 scst_lxcora01]# scstadmin -write_config /etc/scst.conf
Collecting current configuration: done.
Writing current configuration to file '/etc/scst.conf'..
All done.
[root@oracle651 scst_lxcora01]# scstadmin -list_group
Collecting current configuration: done.
Driver: iscsi
Target: iqn.2014-08.org.vmem:oracle651.san.asm.luns
Driver/target 'iscsi/iqn.2014-08.org.vmem:oracle651.san.asm.luns' has no associated LUNs.
Group: oracle631
Assigned LUNs:
LUN Device
-----------------
1 AsmDat631_01
2 AsmDat631_02
3 AsmDat631_03
4 AsmDat631_04
5 AsmDat631_05
6 AsmDat631_06
7 AsmDat631_07
8 AsmDat631_08
Assigned Initiators:
Initiator
----------------------------------
iqn.2014-08.org.vmem:oracle631.asm
iqn.2014-09.org.vmem:lxcora3.asm
Group: lxcora01
Assigned LUNs:
LUN Device
--------------
1 AsmTst_01
2 AsmTst_02
3 AsmTst_03
Assigned Initiators:
Initiator
-------------------------------------
iqn.2014-09.org.vmem1:oracle.asm.luns
All done.
[root@oracle651 scst_lxcora01]# exit
Next on the Ubuntu laptop host, use iscsiadm to update with the new size as shown below.
gstanden@vmem1:~$ sudo iscsiadm -m node -R
[sudo] password for gstanden:
Rescanning session [sid: 1, target: iqn.2014-08.org.vmem:oracle651.san.asm.luns, portal: 10.207.40.74,3260]
Rescanning session [sid: 2, target: iqn.2014-08.org.vmem:oracle651.san.asm.luns, portal: 10.207.41.74,3260]
gstanden@vmem1:~$
Check that the updated size has been detected as shown below.
Here are some notes from the install of RAC on the LXC containers. This is the root.sh output from lxcora02 (first node) as shown below. The root.sh execution successful should look something like as shown below.
[root@lxcora02 ~]# /u00/app/grid/product/12.1.0/grid/root.sh
Performing root user operation.
The following environment variables are set as:
ORACLE_OWNER= grid
ORACLE_HOME= /u00/app/grid/product/12.1.0/grid
Enter the full pathname of the local bin directory: [/usr/local/bin]:
The contents of "dbhome" have not changed. No need to overwrite.
The contents of "oraenv" have not changed. No need to overwrite.
The contents of "coraenv" have not changed. No need to overwrite.
Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be performed.
Using configuration parameter file: /u00/app/grid/product/12.1.0/grid/crs/install/crsconfig_params
2014/10/25 18:15:01 CLSRSC-4001: Installing Oracle Trace File Analyzer (TFA) Collector.
TFA-00012: Oracle Trace File Analyzer (TFA) requires BASH shell. Please install bash and try again.
2014/10/25 18:15:10 CLSRSC-4004: Failed to install Oracle Trace File Analyzer (TFA) Collector. Grid Infrastructure operations will continue.
2014/10/25 18:15:11 CLSRSC-363: User ignored prerequisites during installation
OLR initialization - successful
root wallet
root wallet cert
root cert export
peer wallet
profile reader wallet
pa wallet
peer wallet keys
pa wallet keys
peer cert request
pa cert request
peer cert
pa cert
peer root cert TP
profile reader root cert TP
pa root cert TP
peer pa cert TP
pa peer cert TP
profile reader pa cert TP
profile reader peer cert TP
peer user cert
pa user cert
2014/10/25 18:15:42 CLSRSC-330: Adding Clusterware entries to file 'oracle-ohasd.conf'
CRS-4133: Oracle High Availability Services has been stopped.
CRS-4123: Oracle High Availability Services has been started.
CRS-4133: Oracle High Availability Services has been stopped.
CRS-4123: Oracle High Availability Services has been started.
CRS-2672: Attempting to start 'ora.evmd' on 'lxcora02'
CRS-2672: Attempting to start 'ora.mdnsd' on 'lxcora02'
CRS-2676: Start of 'ora.mdnsd' on 'lxcora02' succeeded
CRS-2676: Start of 'ora.evmd' on 'lxcora02' succeeded
CRS-2672: Attempting to start 'ora.gpnpd' on 'lxcora02'
CRS-2676: Start of 'ora.gpnpd' on 'lxcora02' succeeded
CRS-2672: Attempting to start 'ora.cssdmonitor' on 'lxcora02'
CRS-2672: Attempting to start 'ora.gipcd' on 'lxcora02'
CRS-2676: Start of 'ora.cssdmonitor' on 'lxcora02' succeeded
CRS-2676: Start of 'ora.gipcd' on 'lxcora02' succeeded
CRS-2672: Attempting to start 'ora.cssd' on 'lxcora02'
CRS-2672: Attempting to start 'ora.diskmon' on 'lxcora02'
CRS-2676: Start of 'ora.diskmon' on 'lxcora02' succeeded
CRS-2676: Start of 'ora.cssd' on 'lxcora02' succeeded
ASM created and started successfully.
Disk Group DATA created successfully.
CRS-2672: Attempting to start 'ora.crf' on 'lxcora02'
CRS-2672: Attempting to start 'ora.storage' on 'lxcora02'
CRS-2676: Start of 'ora.storage' on 'lxcora02' succeeded
CRS-2676: Start of 'ora.crf' on 'lxcora02' succeeded
CRS-2672: Attempting to start 'ora.crsd' on 'lxcora02'
CRS-2676: Start of 'ora.crsd' on 'lxcora02' succeeded
CRS-4256: Updating the profile
Successful addition of voting disk b5c728b6fd2c4f5dbf311ac955d78133.
Successful addition of voting disk 2a797f2ddb824feabfd38214d88f30b7.
Successful addition of voting disk 21cdfa0bb8c04f26bf07f6901e789466.
Successfully replaced voting disk group with +DATA.
CRS-4256: Updating the profile
CRS-4266: Voting file(s) successfully replaced
## STATE File Universal Id File Name Disk group
-- ----- ----------------- --------- ---------
1. ONLINE b5c728b6fd2c4f5dbf311ac955d78133 (/dev/mapper/asm_disk26) [DATA]
2. ONLINE 2a797f2ddb824feabfd38214d88f30b7 (/dev/mapper/asm_disk27) [DATA]
3. ONLINE 21cdfa0bb8c04f26bf07f6901e789466 (/dev/mapper/asm_disk28) [DATA]
Located 3 voting disk(s).
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'lxcora02'
CRS-2673: Attempting to stop 'ora.crsd' on 'lxcora02'
CRS-2677: Stop of 'ora.crsd' on 'lxcora02' succeeded
CRS-2673: Attempting to stop 'ora.storage' on 'lxcora02'
CRS-2673: Attempting to stop 'ora.mdnsd' on 'lxcora02'
CRS-2673: Attempting to stop 'ora.gpnpd' on 'lxcora02'
CRS-2677: Stop of 'ora.storage' on 'lxcora02' succeeded
CRS-2673: Attempting to stop 'ora.crf' on 'lxcora02'
CRS-2673: Attempting to stop 'ora.ctssd' on 'lxcora02'
CRS-2673: Attempting to stop 'ora.evmd' on 'lxcora02'
CRS-2673: Attempting to stop 'ora.asm' on 'lxcora02'
CRS-2677: Stop of 'ora.mdnsd' on 'lxcora02' succeeded
CRS-2677: Stop of 'ora.gpnpd' on 'lxcora02' succeeded
CRS-2677: Stop of 'ora.ctssd' on 'lxcora02' succeeded
CRS-2677: Stop of 'ora.crf' on 'lxcora02' succeeded
CRS-2677: Stop of 'ora.evmd' on 'lxcora02' succeeded
CRS-2677: Stop of 'ora.asm' on 'lxcora02' succeeded
CRS-2673: Attempting to stop 'ora.cluster_interconnect.haip' on 'lxcora02'
CRS-2677: Stop of 'ora.cluster_interconnect.haip' on 'lxcora02' succeeded
CRS-2673: Attempting to stop 'ora.cssd' on 'lxcora02'
CRS-2677: Stop of 'ora.cssd' on 'lxcora02' succeeded
CRS-2673: Attempting to stop 'ora.gipcd' on 'lxcora02'
CRS-2677: Stop of 'ora.gipcd' on 'lxcora02' succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'lxcora02' has completed
CRS-4133: Oracle High Availability Services has been stopped.
CRS-4123: Starting Oracle High Availability Services-managed resources
CRS-2672: Attempting to start 'ora.mdnsd' on 'lxcora02'
CRS-2672: Attempting to start 'ora.evmd' on 'lxcora02'
CRS-2676: Start of 'ora.mdnsd' on 'lxcora02' succeeded
CRS-2676: Start of 'ora.evmd' on 'lxcora02' succeeded
CRS-2672: Attempting to start 'ora.gpnpd' on 'lxcora02'
CRS-2676: Start of 'ora.gpnpd' on 'lxcora02' succeeded
CRS-2672: Attempting to start 'ora.gipcd' on 'lxcora02'
CRS-2676: Start of 'ora.gipcd' on 'lxcora02' succeeded
CRS-2672: Attempting to start 'ora.cssdmonitor' on 'lxcora02'
CRS-2676: Start of 'ora.cssdmonitor' on 'lxcora02' succeeded
CRS-2672: Attempting to start 'ora.cssd' on 'lxcora02'
CRS-2672: Attempting to start 'ora.diskmon' on 'lxcora02'
CRS-2676: Start of 'ora.diskmon' on 'lxcora02' succeeded
CRS-2676: Start of 'ora.cssd' on 'lxcora02' succeeded
CRS-2672: Attempting to start 'ora.cluster_interconnect.haip' on 'lxcora02'
CRS-2672: Attempting to start 'ora.ctssd' on 'lxcora02'
CRS-2676: Start of 'ora.ctssd' on 'lxcora02' succeeded
CRS-2676: Start of 'ora.cluster_interconnect.haip' on 'lxcora02' succeeded
CRS-2672: Attempting to start 'ora.asm' on 'lxcora02'
CRS-2676: Start of 'ora.asm' on 'lxcora02' succeeded
CRS-2672: Attempting to start 'ora.storage' on 'lxcora02'
CRS-2676: Start of 'ora.storage' on 'lxcora02' succeeded
CRS-2672: Attempting to start 'ora.crf' on 'lxcora02'
CRS-2676: Start of 'ora.crf' on 'lxcora02' succeeded
CRS-2672: Attempting to start 'ora.crsd' on 'lxcora02'
CRS-2676: Start of 'ora.crsd' on 'lxcora02' succeeded
CRS-6023: Starting Oracle Cluster Ready Services-managed resources
CRS-6017: Processing resource auto-start for servers: lxcora02
CRS-6016: Resource auto-start has completed for server lxcora02
CRS-6024: Completed start of Oracle Cluster Ready Services-managed resources
CRS-4123: Oracle High Availability Services has been started.
2014/10/25 18:31:31 CLSRSC-343: Successfully started Oracle Clusterware stack
CRS-2672: Attempting to start 'ora.asm' on 'lxcora02'
CRS-2676: Start of 'ora.asm' on 'lxcora02' succeeded
CRS-2672: Attempting to start 'ora.DATA.dg' on 'lxcora02'
CRS-2676: Start of 'ora.DATA.dg' on 'lxcora02' succeeded
2014/10/25 18:33:02 CLSRSC-325: Configure Oracle Grid Infrastructure for a Cluster ... succeeded
[root@lxcora02 ~]#
Results of root.sh on the second node (lxcora03) are shown below.
[root@lxcora03 ~]# /u00/app/grid/product/12.1.0/grid/root.sh
Performing root user operation.
The following environment variables are set as:
ORACLE_OWNER= grid
ORACLE_HOME= /u00/app/grid/product/12.1.0/grid
Enter the full pathname of the local bin directory: [/usr/local/bin]:
The contents of "dbhome" have not changed. No need to overwrite.
The contents of "oraenv" have not changed. No need to overwrite.
The contents of "coraenv" have not changed. No need to overwrite.
Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be performed.
Using configuration parameter file: /u00/app/grid/product/12.1.0/grid/crs/install/crsconfig_params
2014/10/25 18:35:38 CLSRSC-4001: Installing Oracle Trace File Analyzer (TFA) Collector.
TFA-00012: Oracle Trace File Analyzer (TFA) requires BASH shell. Please install bash and try again.
2014/10/25 18:35:47 CLSRSC-4004: Failed to install Oracle Trace File Analyzer (TFA) Collector. Grid Infrastructure operations will continue.
2014/10/25 18:35:48 CLSRSC-363: User ignored prerequisites during installation
OLR initialization - successful
2014/10/25 18:37:15 CLSRSC-330: Adding Clusterware entries to file 'oracle-ohasd.conf'
CRS-4133: Oracle High Availability Services has been stopped.
CRS-4123: Oracle High Availability Services has been started.
CRS-4133: Oracle High Availability Services has been stopped.
CRS-4123: Oracle High Availability Services has been started.
CRS-4133: Oracle High Availability Services has been stopped.
CRS-4123: Starting Oracle High Availability Services-managed resources
CRS-2672: Attempting to start 'ora.mdnsd' on 'lxcora03'
CRS-2672: Attempting to start 'ora.evmd' on 'lxcora03'
CRS-2676: Start of 'ora.mdnsd' on 'lxcora03' succeeded
CRS-2676: Start of 'ora.evmd' on 'lxcora03' succeeded
CRS-2672: Attempting to start 'ora.gpnpd' on 'lxcora03'
CRS-2676: Start of 'ora.gpnpd' on 'lxcora03' succeeded
CRS-2672: Attempting to start 'ora.gipcd' on 'lxcora03'
CRS-2676: Start of 'ora.gipcd' on 'lxcora03' succeeded
CRS-2672: Attempting to start 'ora.cssdmonitor' on 'lxcora03'
CRS-2676: Start of 'ora.cssdmonitor' on 'lxcora03' succeeded
CRS-2672: Attempting to start 'ora.cssd' on 'lxcora03'
CRS-2672: Attempting to start 'ora.diskmon' on 'lxcora03'
CRS-2676: Start of 'ora.diskmon' on 'lxcora03' succeeded
CRS-2676: Start of 'ora.cssd' on 'lxcora03' succeeded
CRS-2672: Attempting to start 'ora.cluster_interconnect.haip' on 'lxcora03'
CRS-2672: Attempting to start 'ora.ctssd' on 'lxcora03'
CRS-2676: Start of 'ora.ctssd' on 'lxcora03' succeeded
CRS-2676: Start of 'ora.cluster_interconnect.haip' on 'lxcora03' succeeded
CRS-2672: Attempting to start 'ora.asm' on 'lxcora03'
CRS-2676: Start of 'ora.asm' on 'lxcora03' succeeded
CRS-2672: Attempting to start 'ora.storage' on 'lxcora03'
CRS-2676: Start of 'ora.storage' on 'lxcora03' succeeded
CRS-2672: Attempting to start 'ora.crf' on 'lxcora03'
CRS-2676: Start of 'ora.crf' on 'lxcora03' succeeded
CRS-2672: Attempting to start 'ora.crsd' on 'lxcora03'
CRS-2676: Start of 'ora.crsd' on 'lxcora03' succeeded
CRS-6017: Processing resource auto-start for servers: lxcora03
CRS-2673: Attempting to stop 'ora.LISTENER_SCAN1.lsnr' on 'lxcora02'
CRS-2672: Attempting to start 'ora.net1.network' on 'lxcora03'
CRS-2676: Start of 'ora.net1.network' on 'lxcora03' succeeded
CRS-2672: Attempting to start 'ora.ons' on 'lxcora03'
CRS-2677: Stop of 'ora.LISTENER_SCAN1.lsnr' on 'lxcora02' succeeded
CRS-2673: Attempting to stop 'ora.scan1.vip' on 'lxcora02'
CRS-2677: Stop of 'ora.scan1.vip' on 'lxcora02' succeeded
CRS-2672: Attempting to start 'ora.scan1.vip' on 'lxcora03'
CRS-2676: Start of 'ora.ons' on 'lxcora03' succeeded
CRS-2676: Start of 'ora.scan1.vip' on 'lxcora03' succeeded
CRS-2672: Attempting to start 'ora.LISTENER_SCAN1.lsnr' on 'lxcora03'
CRS-2676: Start of 'ora.LISTENER_SCAN1.lsnr' on 'lxcora03' succeeded
CRS-6016: Resource auto-start has completed for server lxcora03
CRS-6024: Completed start of Oracle Cluster Ready Services-managed resources
CRS-4123: Oracle High Availability Services has been started.
2014/10/25 18:41:14 CLSRSC-343: Successfully started Oracle Clusterware stack
2014/10/25 18:41:26 CLSRSC-325: Configure Oracle Grid Infrastructure for a Cluster ... succeeded
[root@lxcora03 ~]#
Ok so I'm going to paste in all the screenshots that I have of the GUI grid install hopefully today, but I wanted to upload the shots of the running system (crsctl output) of the 12c 2-node RAC GI running perfectly on Ubuntu 14.04.1 inside of Oracle Enteprise Linux 6.5 conntainers running on the Ubuntu so that I could share this result even while I'm still finishing up the documentation at this blog. I'm stoked! Finally! Oracle RAC on Ubuntu without endless tweaks!! All of the taste, none of the calories!! The background is a webpage at the QNX website.
And here is the GNS post-install validation step from cluvy.
Install RAC Database on LXC Containers GI
Setup ssh user equivalence as shown below.
gstanden@vmem1:~$ ssh oracle@lxcora02
oracle@lxcora02's password:
Last login: Sun Oct 26 22:17:26 2014 from vmem1.vmem.org
[oracle@lxcora02 ~]$ cd .ssh
[oracle@lxcora02 .ssh]$ ls
[oracle@lxcora02 .ssh]$ cd ..
[oracle@lxcora02 ~]$ ssh-keygen -t rsa
Generating public/private rsa key pair.
Enter file in which to save the key (/home/oracle/.ssh/id_rsa):
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /home/oracle/.ssh/id_rsa.
Your public key has been saved in /home/oracle/.ssh/id_rsa.pub.
The key fingerprint is:
4a:15:3d:26:86:ce:7a:47:71:0e:3e:0f:b8:42:8c:ec oracle@lxcora02
The key's randomart image is:
+--[ RSA 2048]----+
| ... |
| . =.= |
| . o o +.B . |
| o o +.= . |
| . . ..oS+ |
| E o.o.. . |
| o.. |
| |
| |
+-----------------+
[oracle@lxcora02 ~]$ cd .ssh
[oracle@lxcora02 .ssh]$ cat id_rsa.pub > authorized_keys
[oracle@lxcora02 .ssh]$ scp authorized_keys oracle@lxcora03
[oracle@lxcora02 .ssh]$ scp authorized_keys oracle@lxcora03:/home/oracle/.
The authenticity of host 'lxcora03 (10.207.39.88)' can't be established.
RSA key fingerprint is 4f:62:a6:23:e4:58:2e:e2:1b:a4:1c:39:28:a0:ff:df.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'lxcora03,10.207.39.88' (RSA) to the list of known hosts.
oracle@lxcora03's password:
authorized_keys 100% 397 0.4KB/s 00:00
[oracle@lxcora02 .ssh]$ ssh lxcora03
oracle@lxcora03's password:
Last login: Fri Oct 24 23:14:34 2014 from vmem1.vmem.org
[oracle@lxcora03 ~]$ ssh-keygen -t rsa
Generating public/private rsa key pair.
Enter file in which to save the key (/home/oracle/.ssh/id_rsa):
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /home/oracle/.ssh/id_rsa.
Your public key has been saved in /home/oracle/.ssh/id_rsa.pub.
The key fingerprint is:
3f:47:5f:6a:2a:c0:c2:e2:b2:96:99:a6:62:22:08:4b oracle@lxcora03
The key's randomart image is:
+--[ RSA 2048]----+
| |
| |
| |
| |
| . .S . .|
|.E . o o. . . o |
|+. = . . .o . + |
|*.B . .o o |
|*+.o .. |
+-----------------+
[oracle@lxcora03 ~]$ mv authorized_keys ./.ssh/.
[oracle@lxcora03 ~]$ cd .ssh
[oracle@lxcora03 .ssh]$ cat id_rsa.pub >> authorized_keys
[oracle@lxcora03 .ssh]$ scp authorized_keys oracle@lxcora02:/home/oracle/.ssh/.
The authenticity of host 'lxcora02 (10.207.39.87)' can't be established.
RSA key fingerprint is 4f:62:a6:23:e4:58:2e:e2:1b:a4:1c:39:28:a0:ff:df.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'lxcora02,10.207.39.87' (RSA) to the list of known hosts.
oracle@lxcora02's password:
authorized_keys 100% 794 0.8KB/s 00:00
[oracle@lxcora03 .ssh]$ ssh lxcora02
Last login: Sun Oct 26 22:22:15 2014 from vmem1.vmem.org
[oracle@lxcora02 ~]$ ssh lxcora03
Last login: Sun Oct 26 22:24:22 2014 from lxcora02.vmem.org
[oracle@lxcora03 ~]$ exit
logout
Connection to lxcora03 closed.
[oracle@lxcora02 ~]$ exit
logout
Connection to lxcora02 closed.
[oracle@lxcora03 .ssh]$
Create and run the test_login.sh script until output is as shown below to establish user equivalency for linux user "oracle" on both lxcora02 and lxcora03 nodes.
[oracle@lxcora03 .ssh]$ ./test_login.sh
Sun Oct 26 22:36:09 EDT 2014
Sun Oct 26 22:36:09 EDT 2014
Sun Oct 26 22:36:09 EDT 2014
Sun Oct 26 22:36:09 EDT 2014
Sun Oct 26 22:36:09 EDT 2014
Sun Oct 26 22:36:09 EDT 2014
Sun Oct 26 22:36:09 EDT 2014
Sun Oct 26 22:36:09 EDT 2014
[oracle@lxcora03 .ssh]$ ssh oracle@lxcora02
Last login: Sun Oct 26 22:33:04 2014 from vmem1.vmem.org
[oracle@lxcora02 ~]$ cd .ssh
[oracle@lxcora02 .ssh]$ ./test_login.sh
Sun Oct 26 22:36:24 EDT 2014
Sun Oct 26 22:36:24 EDT 2014
Sun Oct 26 22:36:24 EDT 2014
Sun Oct 26 22:36:24 EDT 2014
Sun Oct 26 22:36:24 EDT 2014
Sun Oct 26 22:36:25 EDT 2014
Sun Oct 26 22:36:25 EDT 2014
Sun Oct 26 22:36:25 EDT 2014
[oracle@lxcora02 .ssh]$ cat test_login.sh
ssh oracle@localhost date
ssh oracle@127.0.0.1 date
ssh oracle@lxcora03 date
ssh oracle@lxcora02 date
ssh oracle@10.207.39.88 date
ssh oracle@10.207.39.87 date
ssh oracle@lxcora02.vmem.org date
ssh oracle@lxcora03.vmem.org date
[oracle@lxcora02 .ssh]$