Ubuntu 14.04.1 Oracle GI 12c ASM Flex Cluster on LXC Containers
This blog page documents building an Oracle ASM 12c (12.1.0.2.0) Flex Cluster on Ubuntu 14.04.1 Desktop (64-bit) using two Oracle Enterprise Linux 6.5 LXC Linux Containers. The build is complete and the screenshots of the completed installed system running on Ubuntu can be seen at the end of the post. I will be filling in details of the build such as the LXC config files for lxcora02 and lxcora03, as well as other key configuration files and details about how storage was presented to the nodes, etc. so if interested in this topic check back often or contact me directly. If one wants to run Oracle Enterprise software products on Ubuntu Linux, LXC Containers are probably the best way. Note that running Oracle database on Ubuntu is not officially supported by Oracle, nor is running Oracle database in Linux Containers officially supported either, so strictly speaking this must be considered for personal and test use only.
My To Do List for This Build
Work out details of 11gR2 integration/compatibility with 12c ASM Flex Clustering
Redistribute the LUNs across the multiple targets so the array outage can be simulated (one LUN from each DG on a target)
Document the work creating the reboot, start, and shutdown aliases for managing the LXC containers
Document the work using "chkconfig" to move "scst" in the KVM SAN to run AFTER "ssh" so that the KVM guest can ssh to the vmem1 host and create the LUNs at laptop bootup.
Related to this is to document the files that were created in the '/etc/sudoers.d" directory to enable nopassword sudo from the KVM SAN to vmem1 at bootup to create the LUNs
External name resolution in LXC containers not working (can only resolve by IP), i.e. "nslookup" and "dig" work but "ping" and "ssh" cannot resolve by name, only by IP.
This was added to /etc/init.d/scst as shown below.
Configure SCST to Start after SSH
On the KVM guest which has the SCST iSCSI Linux SAN, the scst services are configured to start after ssh so that the SCST SAN can be autostarted at bootup, and as soon as SCST has started, it executes a remote command on the Ubuntu host to login to the iscsi initiators and obtain the LUNs.
[root@oracle651 ~]# grep -A2 -B5 gstanden /etc/init.d/scst
case "$1" in
start)
## Start the service.
echo -n "Loading and configuring SCST"
start_scst
ssh -t gstanden@vmem1 "/home/gstanden/create_asm_luns.sh"
rc=$?
;;
[root@oracle651 ~]#
And chkconfig was used to move /etc/init.d/scst on oracle652 KVM guest to run AFTER ssh so that the above ssh would work. The point of all this work was so that oracle652 KVM guest could be set to "autostart" on boot, and that once ssh and scst were up, it would send an ssh command to the Ubuntu host to login to the SCST Linux SAN on oracle652 KVM guest and load up the iSCSI oracle ASM LUNs.
[root@oracle651 ~]# grep chkconfig /etc/init.d/scst
# chkconfig: 2345 56 87 <-- Set start value to "56" so that scst would start after ssh was already up7
[root@oracle651 ~]# grep chkconfig /etc/init.d/ssh*
# chkconfig: 2345 55 25 <-- The ssh service starts at priorty "55" so now it starts before scst
[root@oracle651 ~]# grep chkconfig /etc/init.d/scst.2014.11.01.1552.bak
# chkconfig: 2345 13 87 <-- Here are the original values for scst where it was starting very early at priority "13"
[root@oracle651 ~]#
The remote script on the Ubuntu host that is executed by scst init script on the KVM guest is shown below. The initiators are login/logout/login because of a workaround. It was found that most of the LUNs were created in Linux 5 style as multipath device nodes under /dev/mapper, but sometimes one or two would be created in Linux 6 style as soft links pointing to /dev/dm-* and for the purposes of this work with Linux LXC containers, what is needed is for all the LUNs to be created in Linux 5 style (no soft links) because so far there has not been found a way to preserve those links when delivering the storage to the LXC containers. In other words, /dev/mapper can be presented to each LXC container, but when there are links involved, it is problematic to preserve those links and somehow "promote" /dev/dm-* devices to the LXC container. Furthermore, the "lxc-device" command was found not to work with Oracle for storage. One can present a dm-* device to be sure, and Oracle will even accept the device as an ASM disk, but when it attempts to do IO to that device, there are errors. It was found that mounting /dev/mapper in the LXC container provides storage for the LXC nodes that seems to work in all ways with no problems.
gstanden@vmem1:~$ cat create_asm_luns.sh
sudo iscsiadm --mode node --targetname iqn.2014-08.org.vmem:oracle651.san.asm.luns --portal 10.207.40.74 --login
sudo iscsiadm --mode node --targetname iqn.2014-08.org.vmem:oracle651.san.asm.luns --portal 10.207.41.74 --login
sudo iscsiadm --mode node --targetname iqn.2014-10.org.vmem1:oracle651.san.asm.luns --portal 10.207.40.74 --login
sudo iscsiadm --mode node --targetname iqn.2014-10.org.vmem1:oracle651.san.asm.luns --portal 10.207.41.74 --login
sudo iscsiadm --mode node --targetname iqn.2014-10.org.vmem1:oracle651.san.asm.luns2 --portal 10.207.40.74 --login
sudo iscsiadm --mode node --targetname iqn.2014-10.org.vmem1:oracle651.san.asm.luns2 --portal 10.207.41.74 --login
ls -lrt /dev/mapper
sudo iscsiadm --mode node --targetname iqn.2014-08.org.vmem:oracle651.san.asm.luns --portal 10.207.40.74 --logout
sudo iscsiadm --mode node --targetname iqn.2014-08.org.vmem:oracle651.san.asm.luns --portal 10.207.41.74 --logout
sudo iscsiadm --mode node --targetname iqn.2014-10.org.vmem1:oracle651.san.asm.luns --portal 10.207.40.74 --logout
sudo iscsiadm --mode node --targetname iqn.2014-10.org.vmem1:oracle651.san.asm.luns --portal 10.207.41.74 --logout
sudo iscsiadm --mode node --targetname iqn.2014-10.org.vmem1:oracle651.san.asm.luns2 --portal 10.207.40.74 --logout
sudo iscsiadm --mode node --targetname iqn.2014-10.org.vmem1:oracle651.san.asm.luns2 --portal 10.207.41.74 --logout
sudo service multipath-tools restart
sudo iscsiadm --mode node --targetname iqn.2014-08.org.vmem:oracle651.san.asm.luns --portal 10.207.40.74 --login
sudo iscsiadm --mode node --targetname iqn.2014-08.org.vmem:oracle651.san.asm.luns --portal 10.207.41.74 --login
sudo iscsiadm --mode node --targetname iqn.2014-10.org.vmem1:oracle651.san.asm.luns --portal 10.207.40.74 --login
sudo iscsiadm --mode node --targetname iqn.2014-10.org.vmem1:oracle651.san.asm.luns --portal 10.207.41.74 --login
sudo iscsiadm --mode node --targetname iqn.2014-10.org.vmem1:oracle651.san.asm.luns2 --portal 10.207.40.74 --login
sudo iscsiadm --mode node --targetname iqn.2014-10.org.vmem1:oracle651.san.asm.luns2 --portal 10.207.41.74 --login
ls -lrt /dev/mapper
gstanden@vmem1:~$
My To Do List for The Next Build - Insights
Use enumerated usernames on each node, such as "oracle1" and "dba1" on lxcora01 | "oracle2" and "dba2" on lxcora02 and so on.
This will help manageability since all accounts exist on the node.
The users could also be created on the Ubuntu host so that the names would display in "ps -ef" output rather than just UID and GID numbers.
Longer-term project would be to package up the whole setup in deb file(s) so that it was deb installable.
Oracle software would have to installed separately due to license and export restrictions, but the whole testbed could be deb-ized would think
[grid@lxcora02 grid]$ ./runcluvfy.sh comp gns -precrsinst -domain vmem.org -vip 10.207.39.3 -verbose -n lxcora02,lxcora03
Verifying GNS integrity
Checking GNS integrity...
Checking if the GNS subdomain name is valid...
The GNS subdomain name "vmem.org" is a valid domain name
Checking if the GNS VIP is a valid address...
GNS VIP "10.207.39.3" resolves to a valid IP address
Checking the status of GNS VIP...
GNS integrity check passed
Verification of GNS integrity was successful.
[grid@lxcora02 grid]$
Edit /etc/security/limits.conf in LXC Nodes
Most of the values are set in the /etc/security/limits.conf of the Ubuntu host, but a couple values are set locally in the container as shown below. Note that the Oracle Grid Infrastructure values are commented out. I have found that if I set those, I lose the ability to login to the "grid" user (i.e. with those set, "su - grid" fails with "could not open session"). I have not figured out the fix for this so am not setting them for now. If you have understanding of this problem and how to address, please leave a comment on the blog for the benefit of all, thanks.
[root@lxcora02 ~]# cat /etc/security/limits.conf
# /etc/security/limits.conf
#
#Each line describes a limit for a user in the form:
#
#<domain> <type> <item> <value>
#
#Where:
#<domain> can be:
# - an user name
# - a group name, with @group syntax
# - the wildcard *, for default entry
# - the wildcard %, can be also used with %group syntax,
# for maxlogin limit
#
#<type> can have the two values:
# - "soft" for enforcing the soft limits
# - "hard" for enforcing hard limits
#
#<item> can be one of the following:
# - core - limits the core file size (KB)
# - data - max data size (KB)
# - fsize - maximum filesize (KB)
# - memlock - max locked-in-memory address space (KB)
# - nofile - max number of open files
# - rss - max resident set size (KB)
# - stack - max stack size (KB)
# - cpu - max CPU time (MIN)
# - nproc - max number of processes
# - as - address space limit (KB)
# - maxlogins - max number of logins for this user
# - maxsyslogins - max number of logins on the system
# - priority - the priority to run user process with
# - locks - max number of file locks the user can hold
# - sigpending - max number of pending signals
# - msgqueue - max memory used by POSIX message queues (bytes)
# - nice - max nice priority allowed to raise to values: [-20, 19]
# - rtprio - max realtime priority
#
#<domain> <type> <item> <value>
#
#* soft core 0
#* hard rss 10000
#@student hard nproc 20
#@faculty soft nproc 20
#@faculty hard nproc 50
#ftp hard nproc 0
#@student - maxlogins 4
# End of file
# Oracle Grid Infrastructure
# grid hard nofile 65536
# grid soft nproc 2047
# Oracle Database
* soft memlock 9873408
* hard memlock 9873408
[root@lxcora02 ~]#
Add Entries to /etc/sysctl.conf on LXC Nodes for rp_filter
Add entries as shown in bold below. Note that this is required for the multicast as well.
[root@lxcora03 ~]# cat /etc/sysctl.conf
# Kernel sysctl configuration file for Red Hat Linux
#
# For binary values, 0 is disabled, 1 is enabled. See sysctl(8) and
# sysctl.conf(5) for more details.
# Controls IP packet forwarding
net.ipv4.ip_forward = 0
# Controls source route verification
net.ipv4.conf.default.rp_filter = 1
# Do not accept source routing
net.ipv4.conf.default.accept_source_route = 0
# Controls the System Request debugging functionality of the kernel
kernel.sysrq = 0
# Controls whether core dumps will append the PID to the core filename.
# Useful for debugging multi-threaded applications.
kernel.core_uses_pid = 1
# Controls the use of TCP syncookies
net.ipv4.tcp_syncookies = 1
# Disable netfilter on bridges.
net.bridge.bridge-nf-call-ip6tables = 0
net.bridge.bridge-nf-call-iptables = 0
net.bridge.bridge-nf-call-arptables = 0
# Controls the default maxmimum size of a mesage queue
kernel.msgmnb = 65536
# Controls the maximum size of a message, in bytes
kernel.msgmax = 65536
# Controls the maximum shared segment size, in bytes
kernel.shmmax = 68719476736
# Controls the maximum number of shared memory segments, in pages
kernel.shmall = 4294967296
# Oracle
kernel.shmmni = 4096
kernel.sem = 250 32000 100 128
fs.file-max = 6815744
fs.aio-max-nr = 1048576
net.ipv4.ip_local_port_range = 9000 65500
net.core.rmem_default = 262144
net.core.rmem_max = 4194304
net.core.wmem_default = 262144
net.core.wmem_max = 1048576
kernel.panic_on_oops = 1
net.ipv4.conf.eth1.rp_filter = 2
net.ipv4.conf.eth2.rp_filter = 2
net.ipv4.conf.eth3.rp_filter = 2
net.ipv4.conf.eth4.rp_filter = 2
net.ipv4.conf.eth5.rp_filter = 2
net.ipv4.conf.eth6.rp_filter = 2
[root@lxcora03 ~]#
Then ran "sysctl -p" to have new settings take effect, and it worked in the LXC containers.
[root@lxcora03 ~]# sysctl -p
net.ipv4.ip_forward = 0
net.ipv4.conf.default.rp_filter = 1
net.ipv4.conf.default.accept_source_route = 0
kernel.sysrq = 0
kernel.core_uses_pid = 1
error: "net.ipv4.tcp_syncookies" is an unknown key
error: "net.bridge.bridge-nf-call-ip6tables" is an unknown key
error: "net.bridge.bridge-nf-call-iptables" is an unknown key
error: "net.bridge.bridge-nf-call-arptables" is an unknown key
kernel.msgmnb = 65536
kernel.msgmax = 65536
kernel.shmmax = 68719476736
kernel.shmall = 4294967296
kernel.shmmni = 4096
kernel.sem = 250 32000 100 128
fs.file-max = 6815744
fs.aio-max-nr = 1048576
net.ipv4.ip_local_port_range = 9000 65500
error: "net.core.rmem_default" is an unknown key
error: "net.core.rmem_max" is an unknown key
error: "net.core.wmem_default" is an unknown key
error: "net.core.wmem_max" is an unknown key
kernel.panic_on_oops = 1
net.ipv4.conf.eth1.rp_filter = 2
net.ipv4.conf.eth2.rp_filter = 2
net.ipv4.conf.eth3.rp_filter = 2
net.ipv4.conf.eth4.rp_filter = 2
net.ipv4.conf.eth5.rp_filter = 2
net.ipv4.conf.eth6.rp_filter = 2
[root@lxcora03 ~]#
Install Oracle 12.1.0.2.0 ASM Flex Clusterware
Putting in the screenshots now.
Note on the next installer page, the pre-requisite checks, the shown errors are acceptable for the LXC container install.
Execute root.sh Scripts on Each LXC Node
Execute first script as shown below on lxcora02 and then lxcora03.
[root@lxcora03 ~]# /u00/app/12.1.0/oraInventory/orainstRoot.sh
Changing permissions of /u00/app/12.1.0/oraInventory.
Adding read,write permissions for group.
Removing read,write,execute permissions for world.
Changing groupname of /u00/app/12.1.0/oraInventory to oinstall.
The execution of the script is complete.
[root@lxcora03 ~]#
Run the second root script as shown below on lxcora02.
[root@lxcora02 ~]# /u00/app/grid/product/12.1.0/grid/root.sh
Performing root user operation.
The following environment variables are set as:
ORACLE_OWNER= grid
ORACLE_HOME= /u00/app/grid/product/12.1.0/grid
Enter the full pathname of the local bin directory: [/usr/local/bin]:
The contents of "dbhome" have not changed. No need to overwrite.
The contents of "oraenv" have not changed. No need to overwrite.
The contents of "coraenv" have not changed. No need to overwrite.
Creating /etc/oratab file...
Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be performed.
Using configuration parameter file: /u00/app/grid/product/12.1.0/grid/crs/install/crsconfig_params
2014/10/29 19:46:56 CLSRSC-4001: Installing Oracle Trace File Analyzer (TFA) Collector.
TFA-00012: Oracle Trace File Analyzer (TFA) requires BASH shell. Please install bash and try again.
2014/10/29 19:47:05 CLSRSC-4004: Failed to install Oracle Trace File Analyzer (TFA) Collector. Grid Infrastructure operations will continue.
2014/10/29 19:47:12 CLSRSC-363: User ignored prerequisites during installation
OLR initialization - successful
root wallet
root wallet cert
root cert export
peer wallet
profile reader wallet
pa wallet
peer wallet keys
pa wallet keys
peer cert request
pa cert request
peer cert
pa cert
peer root cert TP
profile reader root cert TP
pa root cert TP
peer pa cert TP
pa peer cert TP
profile reader pa cert TP
profile reader peer cert TP
peer user cert
pa user cert
2014/10/29 19:49:29 CLSRSC-330: Adding Clusterware entries to file 'oracle-ohasd.conf'
CRS-4133: Oracle High Availability Services has been stopped.
CRS-4123: Oracle High Availability Services has been started.
CRS-4133: Oracle High Availability Services has been stopped.
CRS-4123: Oracle High Availability Services has been started.
CRS-2672: Attempting to start 'ora.evmd' on 'lxcora02'
CRS-2672: Attempting to start 'ora.mdnsd' on 'lxcora02'
CRS-2676: Start of 'ora.mdnsd' on 'lxcora02' succeeded
CRS-2676: Start of 'ora.evmd' on 'lxcora02' succeeded
CRS-2672: Attempting to start 'ora.gpnpd' on 'lxcora02'
CRS-2676: Start of 'ora.gpnpd' on 'lxcora02' succeeded
CRS-2672: Attempting to start 'ora.cssdmonitor' on 'lxcora02'
CRS-2672: Attempting to start 'ora.gipcd' on 'lxcora02'
CRS-2676: Start of 'ora.cssdmonitor' on 'lxcora02' succeeded
CRS-2676: Start of 'ora.gipcd' on 'lxcora02' succeeded
CRS-2672: Attempting to start 'ora.cssd' on 'lxcora02'
CRS-2672: Attempting to start 'ora.diskmon' on 'lxcora02'
CRS-2676: Start of 'ora.diskmon' on 'lxcora02' succeeded
CRS-2676: Start of 'ora.cssd' on 'lxcora02' succeeded
ASM created and started successfully.
Disk Group SYSTEMDG created successfully.
CRS-2672: Attempting to start 'ora.crf' on 'lxcora02'
CRS-2672: Attempting to start 'ora.storage' on 'lxcora02'
CRS-2676: Start of 'ora.crf' on 'lxcora02' succeeded
CRS-2676: Start of 'ora.storage' on 'lxcora02' succeeded
CRS-2672: Attempting to start 'ora.crsd' on 'lxcora02'
CRS-2676: Start of 'ora.crsd' on 'lxcora02' succeeded
CRS-4256: Updating the profile
Successful addition of voting disk 4cf21b2b34044fcabfdfedfe2aea7644.
Successful addition of voting disk 36febb54b8bc4fa2bfee0509b3f4ab1e.
Successful addition of voting disk 1a7d8c36835a4fa4bf27dd1bf0edc23d.
Successfully replaced voting disk group with +SYSTEMDG.
CRS-4256: Updating the profile
CRS-4266: Voting file(s) successfully replaced
## STATE File Universal Id File Name Disk group
-- ----- ----------------- --------- ---------
1. ONLINE 4cf21b2b34044fcabfdfedfe2aea7644 (/dev/mapper/asm_disk_lxc1_systemdg_01) [SYSTEMDG]
2. ONLINE 36febb54b8bc4fa2bfee0509b3f4ab1e (/dev/mapper/asm_disk_lxc1_systemdg_02) [SYSTEMDG]
3. ONLINE 1a7d8c36835a4fa4bf27dd1bf0edc23d (/dev/mapper/asm_disk_lxc1_systemdg_03) [SYSTEMDG]
Located 3 voting disk(s).
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'lxcora02'
CRS-2673: Attempting to stop 'ora.crsd' on 'lxcora02'
CRS-2677: Stop of 'ora.crsd' on 'lxcora02' succeeded
CRS-2673: Attempting to stop 'ora.ctssd' on 'lxcora02'
CRS-2673: Attempting to stop 'ora.mdnsd' on 'lxcora02'
CRS-2673: Attempting to stop 'ora.gpnpd' on 'lxcora02'
CRS-2677: Stop of 'ora.ctssd' on 'lxcora02' succeeded
CRS-2673: Attempting to stop 'ora.evmd' on 'lxcora02'
CRS-2673: Attempting to stop 'ora.storage' on 'lxcora02'
CRS-2677: Stop of 'ora.gpnpd' on 'lxcora02' succeeded
CRS-2677: Stop of 'ora.mdnsd' on 'lxcora02' succeeded
CRS-2677: Stop of 'ora.storage' on 'lxcora02' succeeded
CRS-2673: Attempting to stop 'ora.asm' on 'lxcora02'
CRS-2677: Stop of 'ora.evmd' on 'lxcora02' succeeded
CRS-2677: Stop of 'ora.asm' on 'lxcora02' succeeded
CRS-2673: Attempting to stop 'ora.cluster_interconnect.haip' on 'lxcora02'
CRS-2677: Stop of 'ora.cluster_interconnect.haip' on 'lxcora02' succeeded
CRS-2673: Attempting to stop 'ora.cssd' on 'lxcora02'
CRS-2677: Stop of 'ora.cssd' on 'lxcora02' succeeded
CRS-2673: Attempting to stop 'ora.crf' on 'lxcora02'
CRS-2677: Stop of 'ora.crf' on 'lxcora02' succeeded
CRS-2673: Attempting to stop 'ora.gipcd' on 'lxcora02'
CRS-2677: Stop of 'ora.gipcd' on 'lxcora02' succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'lxcora02' has completed
CRS-4133: Oracle High Availability Services has been stopped.
CRS-4123: Starting Oracle High Availability Services-managed resources
CRS-2672: Attempting to start 'ora.mdnsd' on 'lxcora02'
CRS-2672: Attempting to start 'ora.evmd' on 'lxcora02'
CRS-2676: Start of 'ora.mdnsd' on 'lxcora02' succeeded
CRS-2676: Start of 'ora.evmd' on 'lxcora02' succeeded
CRS-2672: Attempting to start 'ora.gpnpd' on 'lxcora02'
CRS-2676: Start of 'ora.gpnpd' on 'lxcora02' succeeded
CRS-2672: Attempting to start 'ora.gipcd' on 'lxcora02'
CRS-2676: Start of 'ora.gipcd' on 'lxcora02' succeeded
CRS-2672: Attempting to start 'ora.cssdmonitor' on 'lxcora02'
CRS-2676: Start of 'ora.cssdmonitor' on 'lxcora02' succeeded
CRS-2672: Attempting to start 'ora.cssd' on 'lxcora02'
CRS-2672: Attempting to start 'ora.diskmon' on 'lxcora02'
CRS-2676: Start of 'ora.diskmon' on 'lxcora02' succeeded
CRS-2676: Start of 'ora.cssd' on 'lxcora02' succeeded
CRS-2672: Attempting to start 'ora.cluster_interconnect.haip' on 'lxcora02'
CRS-2672: Attempting to start 'ora.ctssd' on 'lxcora02'
CRS-2676: Start of 'ora.ctssd' on 'lxcora02' succeeded
CRS-2676: Start of 'ora.cluster_interconnect.haip' on 'lxcora02' succeeded
CRS-2672: Attempting to start 'ora.asm' on 'lxcora02'
CRS-2676: Start of 'ora.asm' on 'lxcora02' succeeded
CRS-2672: Attempting to start 'ora.storage' on 'lxcora02'
CRS-2676: Start of 'ora.storage' on 'lxcora02' succeeded
CRS-2672: Attempting to start 'ora.crf' on 'lxcora02'
CRS-2676: Start of 'ora.crf' on 'lxcora02' succeeded
CRS-2672: Attempting to start 'ora.crsd' on 'lxcora02'
CRS-2676: Start of 'ora.crsd' on 'lxcora02' succeeded
CRS-6023: Starting Oracle Cluster Ready Services-managed resources
CRS-6017: Processing resource auto-start for servers: lxcora02
CRS-6016: Resource auto-start has completed for server lxcora02
CRS-6024: Completed start of Oracle Cluster Ready Services-managed resources
CRS-4123: Oracle High Availability Services has been started.
2014/10/29 20:00:45 CLSRSC-343: Successfully started Oracle Clusterware stack
CRS-2672: Attempting to start 'ora.net1.network' on 'lxcora02'
CRS-2676: Start of 'ora.net1.network' on 'lxcora02' succeeded
CRS-2672: Attempting to start 'ora.gns.vip' on 'lxcora02'
CRS-2676: Start of 'ora.gns.vip' on 'lxcora02' succeeded
CRS-2672: Attempting to start 'ora.gns' on 'lxcora02'
CRS-2676: Start of 'ora.gns' on 'lxcora02' succeeded
CRS-2672: Attempting to start 'ora.ASMNET1LSNR_ASM.lsnr' on 'lxcora02'
CRS-2676: Start of 'ora.ASMNET1LSNR_ASM.lsnr' on 'lxcora02' succeeded
CRS-2672: Attempting to start 'ora.asm' on 'lxcora02'
CRS-2676: Start of 'ora.asm' on 'lxcora02' succeeded
CRS-2672: Attempting to start 'ora.SYSTEMDG.dg' on 'lxcora02'
CRS-2676: Start of 'ora.SYSTEMDG.dg' on 'lxcora02' succeeded
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'lxcora02'
CRS-2673: Attempting to stop 'ora.crsd' on 'lxcora02'
CRS-2790: Starting shutdown of Cluster Ready Services-managed resources on 'lxcora02'
CRS-2673: Attempting to stop 'ora.SYSTEMDG.dg' on 'lxcora02'
CRS-2673: Attempting to stop 'ora.oc4j' on 'lxcora02'
CRS-2673: Attempting to stop 'ora.LISTENER_SCAN3.lsnr' on 'lxcora02'
CRS-2673: Attempting to stop 'ora.LISTENER_SCAN1.lsnr' on 'lxcora02'
CRS-2673: Attempting to stop 'ora.LISTENER_SCAN2.lsnr' on 'lxcora02'
CRS-2673: Attempting to stop 'ora.lxcora02.vip' on 'lxcora02'
CRS-2673: Attempting to stop 'ora.cvu' on 'lxcora02'
CRS-2673: Attempting to stop 'ora.gns' on 'lxcora02'
CRS-2677: Stop of 'ora.cvu' on 'lxcora02' succeeded
CRS-2677: Stop of 'ora.SYSTEMDG.dg' on 'lxcora02' succeeded
CRS-2673: Attempting to stop 'ora.asm' on 'lxcora02'
CRS-2677: Stop of 'ora.LISTENER_SCAN2.lsnr' on 'lxcora02' succeeded
CRS-2673: Attempting to stop 'ora.scan2.vip' on 'lxcora02'
CRS-2677: Stop of 'ora.LISTENER_SCAN1.lsnr' on 'lxcora02' succeeded
CRS-2677: Stop of 'ora.LISTENER_SCAN3.lsnr' on 'lxcora02' succeeded
CRS-2673: Attempting to stop 'ora.scan3.vip' on 'lxcora02'
CRS-2673: Attempting to stop 'ora.scan1.vip' on 'lxcora02'
CRS-2677: Stop of 'ora.asm' on 'lxcora02' succeeded
CRS-2673: Attempting to stop 'ora.ASMNET1LSNR_ASM.lsnr' on 'lxcora02'
CRS-2677: Stop of 'ora.lxcora02.vip' on 'lxcora02' succeeded
CRS-2677: Stop of 'ora.ASMNET1LSNR_ASM.lsnr' on 'lxcora02' succeeded
CRS-2677: Stop of 'ora.scan3.vip' on 'lxcora02' succeeded
CRS-2677: Stop of 'ora.scan2.vip' on 'lxcora02' succeeded
CRS-2677: Stop of 'ora.gns' on 'lxcora02' succeeded
CRS-2673: Attempting to stop 'ora.gns.vip' on 'lxcora02'
CRS-2677: Stop of 'ora.oc4j' on 'lxcora02' succeeded
CRS-2677: Stop of 'ora.scan1.vip' on 'lxcora02' succeeded
CRS-2677: Stop of 'ora.gns.vip' on 'lxcora02' succeeded
CRS-2673: Attempting to stop 'ora.ons' on 'lxcora02'
CRS-2677: Stop of 'ora.ons' on 'lxcora02' succeeded
CRS-2673: Attempting to stop 'ora.net1.network' on 'lxcora02'
CRS-2677: Stop of 'ora.net1.network' on 'lxcora02' succeeded
CRS-2792: Shutdown of Cluster Ready Services-managed resources on 'lxcora02' has completed
CRS-2677: Stop of 'ora.crsd' on 'lxcora02' succeeded
CRS-2673: Attempting to stop 'ora.storage' on 'lxcora02'
CRS-2673: Attempting to stop 'ora.mdnsd' on 'lxcora02'
CRS-2673: Attempting to stop 'ora.gpnpd' on 'lxcora02'
CRS-2677: Stop of 'ora.storage' on 'lxcora02' succeeded
CRS-2673: Attempting to stop 'ora.asm' on 'lxcora02'
CRS-2677: Stop of 'ora.gpnpd' on 'lxcora02' succeeded
CRS-2677: Stop of 'ora.mdnsd' on 'lxcora02' succeeded
CRS-2677: Stop of 'ora.asm' on 'lxcora02' succeeded
CRS-2673: Attempting to stop 'ora.cluster_interconnect.haip' on 'lxcora02'
CRS-2677: Stop of 'ora.cluster_interconnect.haip' on 'lxcora02' succeeded
CRS-2673: Attempting to stop 'ora.ctssd' on 'lxcora02'
CRS-2673: Attempting to stop 'ora.evmd' on 'lxcora02'
CRS-2677: Stop of 'ora.ctssd' on 'lxcora02' succeeded
CRS-2677: Stop of 'ora.evmd' on 'lxcora02' succeeded
CRS-2673: Attempting to stop 'ora.cssd' on 'lxcora02'
CRS-2677: Stop of 'ora.cssd' on 'lxcora02' succeeded
CRS-2673: Attempting to stop 'ora.crf' on 'lxcora02'
CRS-2677: Stop of 'ora.crf' on 'lxcora02' succeeded
CRS-2673: Attempting to stop 'ora.gipcd' on 'lxcora02'
CRS-2677: Stop of 'ora.gipcd' on 'lxcora02' succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'lxcora02' has completed
CRS-4133: Oracle High Availability Services has been stopped.
CRS-4123: Starting Oracle High Availability Services-managed resources
CRS-2672: Attempting to start 'ora.mdnsd' on 'lxcora02'
CRS-2672: Attempting to start 'ora.evmd' on 'lxcora02'
CRS-2676: Start of 'ora.mdnsd' on 'lxcora02' succeeded
CRS-2676: Start of 'ora.evmd' on 'lxcora02' succeeded
CRS-2672: Attempting to start 'ora.gpnpd' on 'lxcora02'
CRS-2676: Start of 'ora.gpnpd' on 'lxcora02' succeeded
CRS-2672: Attempting to start 'ora.gipcd' on 'lxcora02'
CRS-2676: Start of 'ora.gipcd' on 'lxcora02' succeeded
CRS-2672: Attempting to start 'ora.cssdmonitor' on 'lxcora02'
CRS-2676: Start of 'ora.cssdmonitor' on 'lxcora02' succeeded
CRS-2672: Attempting to start 'ora.cssd' on 'lxcora02'
CRS-2672: Attempting to start 'ora.diskmon' on 'lxcora02'
CRS-2676: Start of 'ora.diskmon' on 'lxcora02' succeeded
CRS-2676: Start of 'ora.cssd' on 'lxcora02' succeeded
CRS-2672: Attempting to start 'ora.cluster_interconnect.haip' on 'lxcora02'
CRS-2672: Attempting to start 'ora.ctssd' on 'lxcora02'
CRS-2676: Start of 'ora.ctssd' on 'lxcora02' succeeded
CRS-2676: Start of 'ora.cluster_interconnect.haip' on 'lxcora02' succeeded
CRS-2672: Attempting to start 'ora.asm' on 'lxcora02'
CRS-2676: Start of 'ora.asm' on 'lxcora02' succeeded
CRS-2672: Attempting to start 'ora.storage' on 'lxcora02'
CRS-2676: Start of 'ora.storage' on 'lxcora02' succeeded
CRS-2672: Attempting to start 'ora.crf' on 'lxcora02'
CRS-2676: Start of 'ora.crf' on 'lxcora02' succeeded
CRS-2672: Attempting to start 'ora.crsd' on 'lxcora02'
CRS-2676: Start of 'ora.crsd' on 'lxcora02' succeeded
CRS-6023: Starting Oracle Cluster Ready Services-managed resources
CRS-2664: Resource 'ora.SYSTEMDG.dg' is already running on 'lxcora02'
CRS-6017: Processing resource auto-start for servers: lxcora02
CRS-2672: Attempting to start 'ora.oc4j' on 'lxcora02'
CRS-2672: Attempting to start 'ora.LISTENER_SCAN1.lsnr' on 'lxcora02'
CRS-2672: Attempting to start 'ora.LISTENER_SCAN2.lsnr' on 'lxcora02'
CRS-2676: Start of 'ora.LISTENER_SCAN1.lsnr' on 'lxcora02' succeeded
CRS-2676: Start of 'ora.LISTENER_SCAN2.lsnr' on 'lxcora02' succeeded
CRS-2676: Start of 'ora.oc4j' on 'lxcora02' succeeded
CRS-6016: Resource auto-start has completed for server lxcora02
CRS-6024: Completed start of Oracle Cluster Ready Services-managed resources
CRS-4123: Oracle High Availability Services has been started.
2014/10/29 20:06:31 CLSRSC-325: Configure Oracle Grid Infrastructure for a Cluster ... succeeded
[root@lxcora02 ~]#
Run the second root script on the second LXC node lxcora02 as shown below.
[root@lxcora03 ~]# /u00/app/grid/product/12.1.0/grid/root.sh
Performing root user operation.
The following environment variables are set as:
ORACLE_OWNER= grid
ORACLE_HOME= /u00/app/grid/product/12.1.0/grid
Enter the full pathname of the local bin directory: [/usr/local/bin]:
The contents of "dbhome" have not changed. No need to overwrite.
The contents of "oraenv" have not changed. No need to overwrite.
The contents of "coraenv" have not changed. No need to overwrite.
Creating /etc/oratab file...
Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be performed.
Using configuration parameter file: /u00/app/grid/product/12.1.0/grid/crs/install/crsconfig_params
2014/10/29 20:09:22 CLSRSC-4001: Installing Oracle Trace File Analyzer (TFA) Collector.
TFA-00012: Oracle Trace File Analyzer (TFA) requires BASH shell. Please install bash and try again.
2014/10/29 20:09:31 CLSRSC-4004: Failed to install Oracle Trace File Analyzer (TFA) Collector. Grid Infrastructure operations will continue.
2014/10/29 20:09:32 CLSRSC-363: User ignored prerequisites during installation
OLR initialization - successful
2014/10/29 20:10:41 CLSRSC-330: Adding Clusterware entries to file 'oracle-ohasd.conf'
CRS-4133: Oracle High Availability Services has been stopped.
CRS-4123: Oracle High Availability Services has been started.
CRS-4133: Oracle High Availability Services has been stopped.
CRS-4123: Oracle High Availability Services has been started.
CRS-4133: Oracle High Availability Services has been stopped.
CRS-4123: Starting Oracle High Availability Services-managed resources
CRS-2672: Attempting to start 'ora.mdnsd' on 'lxcora03'
CRS-2672: Attempting to start 'ora.evmd' on 'lxcora03'
CRS-2676: Start of 'ora.mdnsd' on 'lxcora03' succeeded
CRS-2676: Start of 'ora.evmd' on 'lxcora03' succeeded
CRS-2672: Attempting to start 'ora.gpnpd' on 'lxcora03'
CRS-2676: Start of 'ora.gpnpd' on 'lxcora03' succeeded
CRS-2672: Attempting to start 'ora.gipcd' on 'lxcora03'
CRS-2676: Start of 'ora.gipcd' on 'lxcora03' succeeded
CRS-2672: Attempting to start 'ora.cssdmonitor' on 'lxcora03'
CRS-2676: Start of 'ora.cssdmonitor' on 'lxcora03' succeeded
CRS-2672: Attempting to start 'ora.cssd' on 'lxcora03'
CRS-2672: Attempting to start 'ora.diskmon' on 'lxcora03'
CRS-2676: Start of 'ora.diskmon' on 'lxcora03' succeeded
CRS-2676: Start of 'ora.cssd' on 'lxcora03' succeeded
CRS-2672: Attempting to start 'ora.cluster_interconnect.haip' on 'lxcora03'
CRS-2672: Attempting to start 'ora.ctssd' on 'lxcora03'
CRS-2676: Start of 'ora.ctssd' on 'lxcora03' succeeded
CRS-2676: Start of 'ora.cluster_interconnect.haip' on 'lxcora03' succeeded
CRS-2672: Attempting to start 'ora.asm' on 'lxcora03'
CRS-2676: Start of 'ora.asm' on 'lxcora03' succeeded
CRS-2672: Attempting to start 'ora.storage' on 'lxcora03'
CRS-2676: Start of 'ora.storage' on 'lxcora03' succeeded
CRS-2672: Attempting to start 'ora.crf' on 'lxcora03'
CRS-2676: Start of 'ora.crf' on 'lxcora03' succeeded
CRS-2672: Attempting to start 'ora.crsd' on 'lxcora03'
CRS-2676: Start of 'ora.crsd' on 'lxcora03' succeeded
CRS-6017: Processing resource auto-start for servers: lxcora03
CRS-2672: Attempting to start 'ora.net1.network' on 'lxcora03'
CRS-2676: Start of 'ora.net1.network' on 'lxcora03' succeeded
CRS-2672: Attempting to start 'ora.ons' on 'lxcora03'
CRS-2676: Start of 'ora.ons' on 'lxcora03' succeeded
CRS-2673: Attempting to stop 'ora.LISTENER_SCAN1.lsnr' on 'lxcora02'
CRS-2677: Stop of 'ora.LISTENER_SCAN1.lsnr' on 'lxcora02' succeeded
CRS-2673: Attempting to stop 'ora.scan1.vip' on 'lxcora02'
CRS-2677: Stop of 'ora.scan1.vip' on 'lxcora02' succeeded
CRS-2672: Attempting to start 'ora.scan1.vip' on 'lxcora03'
CRS-2676: Start of 'ora.scan1.vip' on 'lxcora03' succeeded
CRS-2672: Attempting to start 'ora.LISTENER_SCAN1.lsnr' on 'lxcora03'
CRS-2676: Start of 'ora.LISTENER_SCAN1.lsnr' on 'lxcora03' succeeded
CRS-6016: Resource auto-start has completed for server lxcora03
CRS-6024: Completed start of Oracle Cluster Ready Services-managed resources
CRS-4123: Oracle High Availability Services has been started.
2014/10/29 20:14:58 CLSRSC-343: Successfully started Oracle Clusterware stack
2014/10/29 20:15:10 CLSRSC-325: Configure Oracle Grid Infrastructure for a Cluster ... succeeded
[root@lxcora03 ~]#
Some details about installed ASM Flex Cluster on LXC Nodes
Some details about the system as shown below.
[grid@lxcora02 ~]$ srvctl config scan
SCAN name: lxc1-scan.gns1.vmem.org, Network: 1
Subnet IPv4: 10.207.39.0/255.255.255.0/eth0, dhcp
Subnet IPv6:
SCAN 0 IPv4 VIP: -/scan1-vip/10.207.39.100
SCAN VIP is enabled.
SCAN VIP is individually enabled on nodes:
SCAN VIP is individually disabled on nodes:
SCAN 1 IPv4 VIP: -/scan2-vip/10.207.39.101
SCAN VIP is enabled.
SCAN VIP is individually enabled on nodes:
SCAN VIP is individually disabled on nodes:
SCAN 2 IPv4 VIP: -/scan3-vip/10.207.39.102
SCAN VIP is enabled.
SCAN VIP is individually enabled on nodes:
SCAN VIP is individually disabled on nodes:
[grid@lxcora02 ~]$ nslookup lxc1-scan
Server: 10.207.39.3
Address: 10.207.39.3#53
Name: lxc1-scan.gns1.vmem.org
Address: 10.207.39.100
Name: lxc1-scan.gns1.vmem.org
Address: 10.207.39.101
Name: lxc1-scan.gns1.vmem.org
Address: 10.207.39.102
[grid@lxcora02 ~]$ srvctl config gns
GNS is enabled.
GNS VIP addresses: 10.207.39.3
Domain served by GNS: gns1.vmem.org
[grid@lxcora02 ~]$ ps -ef | grep pmon
grid 8652 8465 0 20:40 lxc/console 00:00:00 grep pmon
grid 21404 1 0 20:05 ? 00:00:00 asm_pmon_+ASM1
grid 29026 1 0 20:22 ? 00:00:00 mdb_pmon_-MGMTDB
[grid@lxcora02 ~]$ ssh root@lxcora03
root@lxcora03's password:
Last login: Wed Oct 29 20:36:59 2014
[root@lxcora03 ~]# ps -ef | grep pmon
grid 15383 1 0 20:14 ? 00:00:00 asm_pmon_+ASM2
root 25811 25799 0 20:40 pts/0 00:00:00 grep pmon
[root@lxcora03 ~]# crsctl stat res -t
--------------------------------------------------------------------------------
Name Target State Server State details
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.ASMNET1LSNR_ASM.lsnr
ONLINE ONLINE lxcora02 STABLE
OFFLINE OFFLINE lxcora03 STABLE
ora.LISTENER.lsnr
ONLINE ONLINE lxcora02 STABLE
ONLINE ONLINE lxcora03 STABLE
ora.SYSTEMDG.dg
ONLINE ONLINE lxcora02 STABLE
ONLINE ONLINE lxcora03 STABLE
ora.net1.network
ONLINE ONLINE lxcora02 STABLE
ONLINE ONLINE lxcora03 STABLE
ora.ons
ONLINE ONLINE lxcora02 STABLE
ONLINE ONLINE lxcora03 STABLE
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.LISTENER_SCAN1.lsnr
1 ONLINE ONLINE lxcora03 STABLE
ora.LISTENER_SCAN2.lsnr
1 ONLINE ONLINE lxcora02 STABLE
ora.LISTENER_SCAN3.lsnr
1 ONLINE ONLINE lxcora02 STABLE
ora.MGMTLSNR
1 ONLINE ONLINE lxcora02 169.254.40.120 192.2
10.39.10 192.211.39.
10,STABLE
ora.asm
1 ONLINE ONLINE lxcora02 Started,STABLE
2 ONLINE ONLINE lxcora03 Started,STABLE
3 OFFLINE OFFLINE STABLE
ora.cvu
1 ONLINE ONLINE lxcora02 STABLE
ora.gns
1 ONLINE ONLINE lxcora02 STABLE
ora.gns.vip
1 ONLINE ONLINE lxcora02 STABLE
ora.lxcora02.vip
1 ONLINE ONLINE lxcora02 STABLE
ora.lxcora03.vip
1 ONLINE ONLINE lxcora03 STABLE
ora.mgmtdb
1 ONLINE ONLINE lxcora02 Open,STABLE
ora.oc4j
1 ONLINE ONLINE lxcora02 STABLE
ora.scan1.vip
1 ONLINE ONLINE lxcora03 STABLE
ora.scan2.vip
1 ONLINE ONLINE lxcora02 STABLE
ora.scan3.vip
1 ONLINE ONLINE lxcora02 STABLE
--------------------------------------------------------------------------------
[root@lxcora03 ~]#
Completed System
Picture of completed system running on Ubuntu 14.04.1 inside Oracle Enterprise Linux 6.5 LXC Linux Containers
Some details of the system from the Ubuntu 14.04.1. Notice that the Oracle Grid Infrastructure processes are visible from the Base OS even though running in the LXC containers.
gstanden@vmem1:~$ uname -a
Linux vmem1.vmem.org 3.13.11.6 #1 SMP Mon Sep 15 11:54:55 CDT 2014 x86_64 x86_64 x86_64 GNU/Linux
gstanden@vmem1:~$
gstanden@vmem1:~$ cat /etc/lsb-release
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=14.04
DISTRIB_CODENAME=trusty
DISTRIB_DESCRIPTION="Ubuntu 14.04.1 LTS"
gstanden@vmem1:~$
gstanden@vmem1:~$ ps -ef | grep pmon
1098 14706 12394 0 22:47 ? 00:00:00 asm_pmon_+ASM1
1098 15499 12394 0 22:48 ? 00:00:00 mdb_pmon_-MGMTDB
1098 30057 28706 0 22:53 ? 00:00:00 asm_pmon_+ASM2
gstanden 31975 30291 0 23:04 pts/36 00:00:00 grep --color=auto pmon
gstanden@vmem1:~$ sudo lxc-ls -f
[sudo] password for gstanden:
NAME STATE IPV4 IPV6 GROUPS AUTOSTART
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
lxcora01 STOPPED - - - NO
lxcora02 RUNNING 10.207.39.101, 10.207.39.102, 10.207.39.3, 10.207.39.87, 10.207.39.97, 169.254.133.43, 169.254.254.234, 169.254.56.148, 169.254.88.208, 172.220.40.10, 172.221.40.10, 192.210.39.10, 192.211.39.10, 192.212.39.10, 192.213.39.10 - - NO
lxcora03 RUNNING 10.207.39.100, 10.207.39.88, 10.207.39.93, 169.254.122.71, 169.254.141.10, 169.254.223.146, 169.254.28.2, 172.220.40.11, 172.221.40.11, 192.210.39.11, 192.211.39.11, 192.212.39.11, 192.213.39.11 - - NO
gstanden@vmem1:~$
Install RAC Database
Disable Transparent Huge Pages
On the Ubuntu host it is necessary to disable Transparent Huge Pages when running OEL 6 on Ubuntu 3 kernels as described here at Oracle-Base Configuring HugePages for Oracle on Linux (x86-64)
There is a tool on Ubuntu 14.04 that can be used to manage hugepages as shown below.
gstanden@vmem1:~$ hugeadm
The program 'hugeadm' is currently not installed. You can install it by typing:
sudo apt-get install hugepages
gstanden@vmem1:~$ sudo apt-get install hugepages
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following package was automatically installed and is no longer required:
linux-image-extra-3.13.0-32-generic
Use 'apt-get autoremove' to remove it.
The following extra packages will be installed:
libhugetlbfs0
Suggested packages:
libhugetlbfs-tests
The following NEW packages will be installed:
hugepages libhugetlbfs0
0 upgraded, 2 newly installed, 0 to remove and 6 not upgraded.
Need to get 104 kB of archives.
After this operation, 348 kB of additional disk space will be used.
Do you want to continue? [Y/n] y
Get:1 http://us.archive.ubuntu.com/ubuntu/ trusty/universe libhugetlbfs0 amd64 2.17-0ubuntu2 [55.4 kB]
Get:2 http://us.archive.ubuntu.com/ubuntu/ trusty/universe hugepages amd64 2.17-0ubuntu2 [48.4 kB]
Fetched 104 kB in 0s (223 kB/s)
Selecting previously unselected package libhugetlbfs0.
(Reading database ... 374320 files and directories currently installed.)
Preparing to unpack .../libhugetlbfs0_2.17-0ubuntu2_amd64.deb ...
Unpacking libhugetlbfs0 (2.17-0ubuntu2) ...
Selecting previously unselected package hugepages.
Preparing to unpack .../hugepages_2.17-0ubuntu2_amd64.deb ...
Unpacking hugepages (2.17-0ubuntu2) ...
Processing triggers for man-db (2.6.7.1-1ubuntu1) ...
Setting up libhugetlbfs0 (2.17-0ubuntu2) ...
Setting up hugepages (2.17-0ubuntu2) ...
Processing triggers for libc-bin (2.19-0ubuntu6.3) ...
gstanden@vmem1:~$
Here are some details about usage of the "hugeadm" tool. Documentation of manpage for this tool is here.
gstanden@vmem1:~$ hugeadm
hugeadm [options]
options:
--list-all-mounts List all current hugetlbfs mount points
--pool-list List all pools
--hard specified with --pool-pages-min to make
multiple attempts at adjusting the pool size to the
specified count on failure
--pool-pages-min <size|DEFAULT>:[+|-]<pagecount|memsize<G|M|K>>
Adjust pool 'size' lower bound
--obey-mempolicy Obey the NUMA memory policy when
adjusting the pool 'size' lower bound
--thp-always Enable transparent huge pages always
--thp-madvise Enable transparent huge pages with madvise
--thp-never Disable transparent huge pages
--thp-khugepaged-pages <pages to scan> Number of pages that khugepaged
should scan on each pass
--thp-khugepaged-scan-sleep <milliseconds> Time in ms to sleep between
khugepaged passes
--thp-khugepages-alloc-sleep <milliseconds> Time in ms for khugepaged
to wait if there was a huge page allocation failure
--pool-pages-max <size|DEFAULT>:[+|-]<pagecount|memsize<G|M|K>>
Adjust pool 'size' upper bound
--set-recommended-min_free_kbytes
Sets min_free_kbytes to a recommended value to improve availability of
huge pages at runtime
--set-recommended-shmmax Sets shmmax to a recommended value to
maximise the size possible for shared memory pools
--set-shm-group <gid|groupname> Sets hugetlb_shm_group to the
specified group, which has permission to use hugetlb shared memory pools
--add-temp-swap[=count] Specified with --pool-pages-min to create
temporary swap space for the duration of the pool resize. Default swap
size is 5 huge pages. Optional arg sets size to 'count' huge pages
--add-ramdisk-swap Specified with --pool-pages-min to create
swap space on ramdisks. By default, swap is removed after the resize.
--persist Specified with --add-temp-swap or --add-ramdisk-swap
options to make swap space persist after the resize.
--enable-zone-movable Use ZONE_MOVABLE for huge pages
--disable-zone-movable Do not use ZONE_MOVABLE for huge pages
--create-mounts Creates a mount point for each available
huge page size on this system under /var/lib/hugetlbfs
--create-user-mounts <user>
Creates a mount point for each available huge
page size under /var/lib/hugetlbfs/<user>
usable by user <user>
--create-group-mounts <group>
Creates a mount point for each available huge
page size under /var/lib/hugetlbfs/<group>
usable by group <group>
--create-global-mounts
Creates a mount point for each available huge
page size under /var/lib/hugetlbfs/global
usable by anyone
--max-size <size<G|M|K>> Limit the filesystem size of a new mount point
--max-inodes <number> Limit the number of inodes on a new mount point
--page-sizes Display page sizes that a configured pool
--page-sizes-all Display page sizes support by the hardware
--dry-run Print the equivalent shell commands for what
the specified options would have done without
taking any action
--explain Gives a overview of the status of the system
with respect to huge page availability
--verbose <level>, -v Increases/sets tracing levels
--help, -h Prints this message
gstanden@vmem1:~$
Disable of transparent pages is accomplished using this tool as shown below.
gstanden@vmem1:~$ cat /sys/kernel/mm/transparent_hugepage/enabled
[always] madvise never
gstanden@vmem1:~$ sudo hugeadm --thp-never
gstanden@vmem1:~$ cat /sys/kernel/mm/transparent_hugepage/enabled
always madvise [never]
gstanden@vmem1:~$
Ensure SCST Linux iSCSI SAN Space Sufficient
Ran into an issue where the KVM SCST Linux SAN ran out of space on the LVM where the file-backed LUNs for SCST for this database are stored. This resulted in one of the DATA LUNs having a status of CLOSED ONLINE MEMBER but showing 0 size and other oddities and diskgroup was basically unusable although ti would mount.
Turned out there was space on the VG that has somehow not been allocated, so that was fixed. Then it was necessary to add the LUN back to the diskgroup. This referenced here at the website IT with Coffee was very helpful getting the LUN added back into the Normal Redundancy DATA ASM diskgroup. The command was simply as shown below.
alter diskgroup DATA add disk '/dev/mapper/asm_disk_lxc1_data_01' force;
Also, the work getting the space on the vg_scst added to lv_vmem1 is worth noting for future reference. The vgs showed 40.00 GB available on physical volume /dev/vdg (which is 100Gb size, but only 60Gb was in use, with 40Gb still free). These were the commands used to get the remaining entire space added to lv_vmem1 as shown below. I had to do it stepwise for some reason but in the end got all available free space on the PV added to the lv_vmem1. After that, the RAC database was created again, and used "Admin" management instead of "Pools" this time.
lvextend -L+39G /dev/mapper/vg_scst-lv_vmem1
resize2fs /dev/mapper/vg_scst-lv_vmem1
lvextend -l+100%FREE /dev/mapper/vg_scst-lv_vmem1
resize2fs /dev/mapper/vg_scst-lv_vmem1
gstanden@vmem1:~$ ssh -Y -C oracle@lxcora02
oracle@lxcora02's password:
Last login: Thu Oct 30 00:03:20 2014 from 10.207.39.1
[oracle@lxcora02 ~]$ cd database
[oracle@lxcora02 database]$ ./runInstaller
Starting Oracle Universal Installer...
Checking Temp space: must be greater than 500 MB. Actual 324014 MB Passed
Checking swap space: must be greater than 150 MB. Actual 29991 MB Passed
Checking monitor: must be configured to display at least 256 colors. Actual 16777216 Passed
Preparing to launch Oracle Universal Installer from /tmp/OraInstall2014-10-30_11-04-43AM. Please wait ...[oracle@lxcora02 database]$
During install of the RAC DB, found that there was insufficient space for the RAC database in the DATA diskgroup. Here are the steps for resizing the LUNs in this particular case. The general procedure is documented here at this blog.
Edit the original file used to create the LUNs to use a new "seek" value. In this case just a bit over 2X the space is desired, so change the value to "45" as shown below.
[root@oracle651 scripts]# cat dd-create-DATA-luns.sh
dd if=/dev/zero of=/scst_vmem1/AsmDat_01.img bs=512 count=0 seek=20M
dd if=/dev/zero of=/scst_vmem1/AsmDat_02.img bs=512 count=0 seek=20M
dd if=/dev/zero of=/scst_vmem1/AsmDat_03.img bs=512 count=0 seek=20M
[root@oracle651 scripts]# cp -p dd-create-DATA-luns.sh dd-extend-DATA-luns.sh
[root@oracle651 scripts]# vi dd-extend-DATA-luns.sh
[root@oracle651 scripts]# cat dd-extend-DATA-luns.sh
dd if=/dev/zero of=/scst_vmem1/AsmDat_01.img bs=512 count=0 seek=45M
dd if=/dev/zero of=/scst_vmem1/AsmDat_02.img bs=512 count=0 seek=45M
dd if=/dev/zero of=/scst_vmem1/AsmDat_03.img bs=512 count=0 seek=45M
Now extend the LUNs using the script as shown below.
[root@oracle651 scripts]# ./dd-extend-DATA-luns.sh
0+0 records in
0+0 records out
0 bytes (0 B) copied, 0.00144059 s, 0.0 kB/s
0+0 records in
0+0 records out
0 bytes (0 B) copied, 0.000682248 s, 0.0 kB/s
0+0 records in
0+0 records out
0 bytes (0 B) copied, 0.00102967 s, 0.0 kB/s
Verify that the LUNs have been resized as shown below.
[root@oracle651 scst_vmem1]# ls -lrt /scst_vmem1/AsmDat*
-rw-r--r-- 1 root root 24159191040 Oct 30 13:50 /scst_vmem1/AsmDat_03.img
-rw-r--r-- 1 root root 24159191040 Oct 30 13:50 /scst_vmem1/AsmDat_02.img
-rw-r--r-- 1 root root 24159191040 Oct 30 13:50 /scst_vmem1/AsmDat_01.img
[root@oracle651 scst_vmem1]#
Resync the new sizes with scstadmin as shown below, and verify status of group and target after resync operation as shown below.
[root@oracle651 scst_vmem1]# scstadmin -resync_dev AsmDat_01
Collecting current configuration: done.
-> Making requested changes.
-> Setting device attribute 'resync_size' to value '1' for device 'AsmDat_01': done.
-> Done.
All done.
[root@oracle651 scst_vmem1]# scstadmin -resync_dev AsmDat_02
Collecting current configuration: done.
-> Making requested changes.
-> Setting device attribute 'resync_size' to value '1' for device 'AsmDat_02': done.
-> Done.
All done.
[root@oracle651 scst_vmem1]# scstadmin -resync_dev AsmDat_03
Collecting current configuration: done.
-> Making requested changes.
-> Setting device attribute 'resync_size' to value '1' for device 'AsmDat_03': done.
-> Done.
All done.
[root@oracle651 scst_vmem1]# scstadmin -list_group vmem1 -driver iscsi -target iqn.2014-10.org.vmem1:oracle651.san.asm.luns2
Collecting current configuration: done.
Assigned LUNs:
LUN Device
--------------
1 AsmFra_01
2 AsmFra_02
3 AsmFra_03
Assigned Initiators:
Initiator
-------------------------------------
iqn.2014-09.org.vmem1:oracle.asm.luns
All done.
[root@oracle651 scst_vmem1]#
Update the iSCSI client (vmem1) with the new sizes using the multipathd command: multipathd -k"resize map asm_disk_lxc1_data_01" (example).
gstanden@vmem1:~$ sudo multipathd -k"resize map asm_disk_lxc1_data_01"
fail
gstanden@vmem1:~$
Trying again actually logged in as root also fails as shown below (stubbornly tried a couple of times with same result of course!)
root@vmem1:~# multipathd -k
multipathd> resize map asm_disk_lxc1_data_01
fail
multipathd> resize map "asm_disk_lxc1_data_01"
fail
multipathd> exit
root@vmem1:~#
Try different method as described here at sig-io as shown below.
root@vmem1:~# blockdev --rereadpt /dev/sdt
root@vmem1:~# blockdev --rereadpt /dev/sdn
root@vmem1:~# blockdev --getsz /dev/sdn
47185920
root@vmem1:~# blockdev --getsz /dev/sdt
47185920
root@vmem1:
Dump and edit the mapping table as shown below.
root@vmem1:~# dmsetup table asm_disk_lxc1_data_01 | tee asm_disk_lxc1_data_01_mapping.bak asm_disk_lxc1_data_01_mapping.cur
0 20971520 multipath 0 0 2 1 round-robin 0 1 1 65:48 1000 round-robin 0 1 1 8:208 1000
root@vmem1:~# ls -lrt
total 24
drwxr-xr-x 2 root root 4096 Aug 16 18:11 Desktop
drwxr-xr-x 2 root root 4096 Aug 16 18:21 R-Studio
-rw-r--r-- 1 root root 2125 Sep 4 10:27 xamarin.gpg
drwxr-xr-x 19 root root 4096 Sep 15 11:14 scst
-rw-r--r-- 1 root root 88 Oct 30 14:27 asm_disk_lxc1_data_01_mapping.cur
-rw-r--r-- 1 root root 88 Oct 30 14:27 asm_disk_lxc1_data_01_mapping.bak
root@vmem1:~# more asm*
::::::::::::::
asm_disk_lxc1_data_01_mapping.bak
::::::::::::::
0 20971520 multipath 0 0 2 1 round-robin 0 1 1 65:48 1000 round-robin 0 1 1 8:208 1000
::::::::::::::
asm_disk_lxc1_data_01_mapping.cur
::::::::::::::
0 20971520 multipath 0 0 2 1 round-robin 0 1 1 65:48 1000 round-robin 0 1 1 8:208 1000
root@vmem1:~# vi asm_disk_lxc1_data_01_mapping.cur
root@vmem1:~# cat asm_disk_lxc1_data_01_mapping.cur
0 47185920 multipath 0 0 2 1 round-robin 0 1 1 65:48 1000 round-robin 0 1 1 8:208 1000
root@vmem1:~#
Apply the changes by reloading the edited map as shown below.
root@vmem1:~# dmsetup suspend asm_disk_lxc1_data_01; dmsetup reload asm_disk_lxc1_data_01 asm_disk_lxc1_data_01_mapping.cur; dmsetup resume asm_disk_lxc1_data_01
Verify procedure has worked. It has!
root@vmem1:~# multipath -ll -v2 asm_disk_lxc1_data_01
asm_disk_lxc1_data_01 (2626237633731622d) dm-11 SCST_FIO,AsmDat_01
size=22G features='0' hwhandler='0' wp=rw
|-+- policy='round-robin 0' prio=1 status=active
| `- 15:0:0:1 sdt 65:48 active ready running
`-+- policy='round-robin 0' prio=1 status=enabled
`- 14:0:0:1 sdn 8:208 active ready running
root@vmem1:~# multipath -ll -v2 asm_disk_lxc1_data_02
asm_disk_lxc1_data_02 (23366353036663832) dm-6 SCST_FIO,AsmDat_02
size=10G features='0' hwhandler='0' wp=rw
|-+- policy='round-robin 0' prio=1 status=active
| `- 15:0:0:2 sdu 65:64 active ready running
`-+- policy='round-robin 0' prio=1 status=enabled
`- 14:0:0:2 sdo 8:224 active ready running
root@vmem1:~#
Size all other LUNs as needed using same procedure. Final results of steps are shown below.
root@vmem1:~# multipath -ll -v2 asm_disk_lxc1_data_01
asm_disk_lxc1_data_01 (2626237633731622d) dm-11 SCST_FIO,AsmDat_01
size=22G features='0' hwhandler='0' wp=rw
|-+- policy='round-robin 0' prio=1 status=active
| `- 15:0:0:1 sdt 65:48 active ready running
`-+- policy='round-robin 0' prio=1 status=enabled
`- 14:0:0:1 sdn 8:208 active ready running
root@vmem1:~# multipath -ll -v2 asm_disk_lxc1_data_02
asm_disk_lxc1_data_02 (23366353036663832) dm-6 SCST_FIO,AsmDat_02
size=22G features='0' hwhandler='0' wp=rw
|-+- policy='round-robin 0' prio=1 status=active
| `- 15:0:0:2 sdu 65:64 active ready running
`-+- policy='round-robin 0' prio=1 status=enabled
`- 14:0:0:2 sdo 8:224 active ready running
root@vmem1:~# multipath -ll -v2 asm_disk_lxc1_data_03
asm_disk_lxc1_data_03 (23263663266376635) dm-8 SCST_FIO,AsmDat_03
size=22G features='0' hwhandler='0' wp=rw
|-+- policy='round-robin 0' prio=1 status=active
| `- 15:0:0:3 sdv 65:80 active ready running
`-+- policy='round-robin 0' prio=1 status=enabled
`- 14:0:0:3 sdp 8:240 active ready running
root@vmem1:~#
Finally, resize the LUN in ASM as shown below.
[grid@lxcora02 ~]$ sqlplus "/ as sysasm"
SQL*Plus: Release 12.1.0.2.0 Production on Thu Oct 30 15:59:03 2014
Copyright (c) 1982, 2014, Oracle. All rights reserved.
Connected to:
Oracle Database 12c Enterprise Edition Release 12.1.0.2.0 - 64bit Production
With the Real Application Clusters and Automatic Storage Management options
SQL> set linesize 200
SQL> column path format a40
SQL> select name, label, path, total_mb, header_status, mount_status, mode_status from v$asm_disk where name like '%DATAT%';
NAME LABEL PATH TOTAL_MB HEADER_STATU MOUNT_S MODE_ST
------------------------------ ------------------------------- ---------------------------------------- ---------- ------------ ------- -------
DATA_0000 /dev/mapper/asm_disk_lxc1_data_01 10240 MEMBER CACHED ONLINE
DATA_0002 /dev/mapper/asm_disk_lxc1_data_03 10240 MEMBER CACHED ONLINE
DATA_0001 /dev/mapper/asm_disk_lxc1_data_02 10240 MEMBER CACHED ONLINE
3 rows selected.
SQL> alter diskgroup DATA resize all;
Diskgroup altered.
SQL> select name, label, path, total_mb, header_status, mount_status, mode_status from v$asm_disk where name like '%DATA%';
NAME LABEL PATH TOTAL_MB HEADER_STATU MOUNT_S MODE_ST
------------------------------ ------------------------------- ---------------------------------------- ---------- ------------ ------- -------
DATA_0000 /dev/mapper/asm_disk_lxc1_data_01 23040 MEMBER CACHED ONLINE
DATA_0002 /dev/mapper/asm_disk_lxc1_data_03 23040 MEMBER CACHED ONLINE
DATA_0001 /dev/mapper/asm_disk_lxc1_data_02 23040 MEMBER CACHED ONLINE
SQL>
Done.
Continue with install...using DBCA this time as this is a re-do after fixes.
[grid@lxcora03 grid]$ ./runcluvfy.sh comp scan -verbose
Verifying SCAN
Checking Single Client Access Name (SCAN)...
SCAN Name Node Running? ListenerName Port Running?
---------------- ------------ ------------ ------------ ------------ ------------
lxc1-scan.gns1.vmem.org lxcora03 true LISTENER_SCAN1 1521 true
lxc1-scan.gns1.vmem.org lxcora02 true LISTENER_SCAN2 1521 true
lxc1-scan.gns1.vmem.org lxcora02 true LISTENER_SCAN3 1521 true
Checking TCP connectivity to SCAN listeners...
Node ListenerName TCP connectivity?
------------ ------------------------ ------------------------
lxcora02 LISTENER_SCAN1 yes
lxcora02 LISTENER_SCAN2 yes
lxcora02 LISTENER_SCAN3 yes
TCP connectivity to SCAN listeners exists on all cluster nodes
Checking name resolution setup for "lxc1-scan.gns1.vmem.org"...
Checking integrity of name service switch configuration file "/etc/nsswitch.conf" ...
Checking if "hosts" entry in file "/etc/nsswitch.conf" is consistent across nodes...
Checking file "/etc/nsswitch.conf" to make sure that only one "hosts" entry is defined
More than one "hosts" entry does not exist in any "/etc/nsswitch.conf" file
PRVG-2058 : "hosts" entry in the existing "/etc/nsswitch.conf" files is inconsistent
"hosts" entry was found as "hosts: dns files" on nodes: lxcora03
"hosts" entry was found as "hosts: dns files nisplus nis db" on nodes: lxcora02
Check for integrity of name service switch configuration file "/etc/nsswitch.conf" failed
Checking SCAN IP addresses...
Check of SCAN IP addresses passed
Verification of SCAN VIP and listener setup failed
Verification of SCAN was unsuccessful on all the specified nodes.
[grid@lxcora03 grid]$
Cleaned up the /etc/nsswitch.conf file (removed #commented out #hosts entries) and re-run SCAN verification as shown below which is successful this time.
[grid@lxcora03 grid]$ ./runcluvfy.sh comp scan -verbose
Verifying SCAN
Checking Single Client Access Name (SCAN)...
SCAN Name Node Running? ListenerName Port Running?
---------------- ------------ ------------ ------------ ------------ ------------
lxc1-scan.gns1.vmem.org lxcora03 true LISTENER_SCAN1 1521 true
lxc1-scan.gns1.vmem.org lxcora02 true LISTENER_SCAN2 1521 true
lxc1-scan.gns1.vmem.org lxcora02 true LISTENER_SCAN3 1521 true
Checking TCP connectivity to SCAN listeners...
Node ListenerName TCP connectivity?
------------ ------------------------ ------------------------
lxcora02 LISTENER_SCAN1 yes
lxcora02 LISTENER_SCAN2 yes
lxcora02 LISTENER_SCAN3 yes
TCP connectivity to SCAN listeners exists on all cluster nodes
Checking name resolution setup for "lxc1-scan.gns1.vmem.org"...
Checking integrity of name service switch configuration file "/etc/nsswitch.conf" ...
Checking if "hosts" entry in file "/etc/nsswitch.conf" is consistent across nodes...
Checking file "/etc/nsswitch.conf" to make sure that only one "hosts" entry is defined
More than one "hosts" entry does not exist in any "/etc/nsswitch.conf" file
All nodes have same "hosts" entry defined in file "/etc/nsswitch.conf"
Check for integrity of name service switch configuration file "/etc/nsswitch.conf" passed
SCAN Name IP Address Status Comment
------------ ------------------------ ------------------------ ----------
lxc1-scan.gns1.vmem.org 10.207.39.102 passed
lxc1-scan.gns1.vmem.org 10.207.39.101 passed
lxc1-scan.gns1.vmem.org 10.207.39.100 passed
Checking SCAN IP addresses...
Check of SCAN IP addresses passed
Verification of SCAN VIP and listener setup passed
Verification of SCAN was successful.
[grid@lxcora03 grid]$
Now re-run DBCA installer verification checks.
Run root.sh scripts at appropriate time as shown below.
[root@lxcora02 ~]# /u00/app/oracle/product/12.1.0/dbhome_1/root.sh
Performing root user operation.
The following environment variables are set as:
ORACLE_OWNER= oracle
ORACLE_HOME= /u00/app/oracle/product/12.1.0/dbhome_1
Enter the full pathname of the local bin directory: [/usr/local/bin]:
The contents of "dbhome" have not changed. No need to overwrite.
The contents of "oraenv" have not changed. No need to overwrite.
The contents of "coraenv" have not changed. No need to overwrite.
Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be performed.
[root@lxcora02 ~]# ssh root@lxcora03
root@lxcora03's password:
Last login: Thu Oct 30 16:47:36 2014 from 10.207.39.87
[root@lxcora03 ~]# /u00/app/oracle/product/12.1.0/dbhome_1/root.sh
Performing root user operation.
The following environment variables are set as:
ORACLE_OWNER= oracle
ORACLE_HOME= /u00/app/oracle/product/12.1.0/dbhome_1
Enter the full pathname of the local bin directory: [/usr/local/bin]:
The contents of "dbhome" have not changed. No need to overwrite.
The contents of "oraenv" have not changed. No need to overwrite.
The contents of "coraenv" have not changed. No need to overwrite.
Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be performed.
[root@lxcora03 ~]#
Cluster Startup Failure on LXC Node 2 (lxcora03)
Ran into an issue with the always one LXC node failing to startup HAS and cluster intermittently failing to startup cluster services. This turned out to be caused by CSS (Cluster Synchronization Service) so switched to NTP as discussed below. However, for reference, the document I looked at was Oracle Support document: Troubleshoot Grid Infrastructure Startup Issues (Doc ID 1050908.1) which however turned out NOT to be the issue. I did NOT need to use any of the material in that document, but it's a potentially useful document, so it' referenced here. The main issue is to configure NTP for the LXC nodes as described in the next section.
Configure NTP on Ubuntu Host
Install NTP as described here at eHowStuff (skip the "Malaysia" part unless you happen to live in Malaysia). For my system, these were the commands used as shown below.
gstanden@vmem1:~$ sudo apt-get install ntp -y
Then be sure that the Ubuntu host is set to use "Internet" for time sychnronization by opening up "System Settings" GUI and checking "Automatically from the internet" as shown below. If using Ubuntu Server instead of desktop, there will be a flat file to configure instead.
Finally, reboot the Ubuntu host and verify that NTP service starts automatically at boot as shown below.
gstanden@vmem1:~$ ps -ef | grep ntp
ntp 7995 1 0 15:27 ? 00:00:00 /usr/sbin/ntpd -p /var/run/ntpd.pid -g -u 128:142
gstanden 8422 7914 0 15:36 pts/9 00:00:00 grep --color=auto ntp
gstanden@vmem1:~$ ntpq -p
remote refid st t when poll reach delay offset jitter
==============================================================================
50.7.0.147 .INIT. 16 u - 64 0 0.000 0.000 0.000
host2.kingrst.c .INIT. 16 u - 64 0 0.000 0.000 0.000
bindcat.fhsu.ed .INIT. 16 u - 64 0 0.000 0.000 0.000
70.35.113.43 .INIT. 16 u - 64 0 0.000 0.000 0.000
golem.canonical .INIT. 16 u - 64 0 0.000 0.000 0.000
10.207.39.255 .BCST. 16 u - 64 0 0.000 0.000 0.000
gstanden@vmem1:~$
Configure NTP on LXC Nodes
As noted below, ntp service is needed for time synchronization on the LXC RAC nodes as it has been found that using Cluster Synchronization Service (CSS) results in a problem where only one of the RAC nodes will startup HAS successfully (if CSS is in use). The lxcora02 node will start, but lxcora02 HAS will not, or vice versa, but not both.
However, when NTP is used, both nodes start normally. Clearly, one solution would be to debug why CSS causes this problem, but here NTP is simply used to workaround this issue.
The next problem that occurs is that if NTP is installed using YUM on the LXC RAC nodes, then run "chkconfig ntpd on" one finds that the NTP service immediately dies. There is an error message that appears in /var/log/messages associated with this problem, as shown below.
March 23 10:44:56 ntpd [10401]: cap_set_proc () failed to drop root privileges: Operation not permitted
After searching on this issue, found one hit (in French language) here which was used to solve this issue. At that referenced link, poster found that simply starting the ntpd manually works. Therefore, here is the /etc/rc.local that is being used on the LXC RAC nodes to startup ntpd at boot (successfully).
gstanden@vmem1:~$ ssh root@lxcora02
root@lxcora02's password:
Last login: Sat Nov 1 15:48:28 2014 from 10.207.39.88
[root@lxcora02 ~]# cat /etc/rc.local
#!/bin/sh
#
# This script will be executed *after* all the other init scripts.
# You can put your own initialization stuff in here if you don't
# want to do the full Sys V style init stuff.
touch /var/lock/subsys/local
# Oracle ASM Storage
chown grid:asmadmin /dev/mapper/asm_disk*
chmod 0660 /dev/mapper/asm_disk*
# NTP
ntpd
[root@lxcora02 ~]#
Also, the Ubuntu host runs NTP so the Ubuntu host has been configured in the LXC RAC nodes as a local time source, and remote NTP servers have been commented out, so that the LXC nodes will get time from the Ubuntu laptop, and all 3, laptop, and both LXC RAC nodes will get the same time. The bolded entries in in the /etc/ntp.conf file on each LXC RAC node have been modified as shown below.
[root@lxcora02 ~]# cat /etc/ntp.conf
# For more information about this file, see the man pages
# ntp.conf(5), ntp_acc(5), ntp_auth(5), ntp_clock(5), ntp_misc(5), ntp_mon(5).
driftfile /var/lib/ntp/drift
# Permit time synchronization with our time source, but do not
# permit the source to query or modify the service on this system.
restrict default kod nomodify notrap nopeer noquery
restrict -6 default kod nomodify notrap nopeer noquery
# Permit all access over the loopback interface. This could
# be tightened as well, but to do so would effect some of
# the administrative functions.
restrict 127.0.0.1
restrict -6 ::1
# Hosts on local network are less restricted.
#restrict 192.168.1.0 mask 255.255.255.0 nomodify notrap
# Use public servers from the pool.ntp.org project.
# Please consider joining the pool (http://www.pool.ntp.org/join.html).
server 10.207.39.1 prefer
# server 0.rhel.pool.ntp.org iburst
# server 1.rhel.pool.ntp.org iburst
# server 2.rhel.pool.ntp.org iburst
# server 3.rhel.pool.ntp.org iburst
#broadcast 192.168.1.255 autokey # broadcast server
#broadcastclient # broadcast client
#broadcast 224.0.1.1 autokey # multicast server
#multicastclient 224.0.1.1 # multicast client
#manycastserver 239.255.254.254 # manycast server
#manycastclient 239.255.254.254 autokey # manycast client
# Enable public key cryptography.
#crypto
includefile /etc/ntp/crypto/pw
# Key file containing the keys and key identifiers used when operating
# with symmetric key cryptography.
keys /etc/ntp/keys
# Specify the key identifiers which are trusted.
#trustedkey 4 8 42
# Specify the key identifier to use with the ntpdc utility.
#requestkey 8
# Specify the key identifier to use with the ntpq utility.
#controlkey 8
# Enable writing of statistics records.
#statistics clockstats cryptostats loopstats peerstats
[root@lxcora02 ~]#
Once these changes are made test the ntp functionality by rebooting and testing as shown below.
[root@lxcora02 ~]# ntpq -p
remote refid st t when poll reach delay offset jitter
==============================================================================
vmem1.vmem.org .INIT. 16 u - 64 0 0.000 0.000 0.000
[root@lxcora02 ~]#
Optionally, one could simply configure NTP on each of the LXC nodes, rather than synchronizing to the Ubuntu host.
Automate Creation of SCST LUNs on Ubuntu LXC Host
The KVM Guest "oracle652" can be autostarted using the "virsh autostart oracle652" command, but it is also required to create the storage LUNs on the Ubuntu host server. That is what's being worked on now. Some code however was put into KVM guest oracle652 which caused it to stop booting (waiting for a sudo password). To fix this "guestfish" was used. Since this is an amazingly useful program for fixing these kinds of programs, it's documented here as shown below. (The operation done below was to edit /etc/init.d/scst file).
gstanden@vmem1:~$ sudo guestfish -d oracle652
Welcome to guestfish, the guest filesystem shell for
editing virtual machine filesystems and disk images.
Type: 'help' for help on commands
'man' to read the manual
'quit' to quit the shell
><fs> run
><fs> list-filesystems
/dev/sda1: ext4
/dev/sdh1: iso9660
/dev/sdh2: vfat
/dev/vg_oracle651/lv_root: ext4
/dev/vg_oracle651/lv_swap: swap
/dev/vg_scst/lv_oracle631: ext4
/dev/vg_scst/lv_oracle632: ext4
/dev/vg_scst/lv_vmem1: ext4
><fs> mount /dev/vg_oracle651/lv_root /
><fs> vi /etc/init.d/scst
><fs>
GNS Configuration Notes
Just finished working all day and pulling hair out plus cursing a bit (no one but me at home) and finally have it working. These are just some rough notes to capture the configuration files that are working for this so I have them. There were some very useful posts out on the net which I used, although in the end, the Oracle Install Guide turned out to be farily accurate. The page here at Oracle RAC with GNS (the only caveat Ihave to this reference is that author named the VIP "oralab-scan" which could be a bit misleading because this VIP has nothing to do per se with the scan name except to point to GNS which resolves it. Still, the author of this page put in a wealth of details which was absolutely critical to helping get this done. Another good reference is the page at Martin Bach blog here . Martin Bach also has another good post here.
The Oracle Official Documentation was quite useful as well, both the Oracle® Grid Infrastructure Installation Guide12c Release 1 (12.1) for LinuxE48914-12 and the related manual on Oracle® Clusterware Administration and Deployment Guide12c Release 1 (12.1)E48819-07 is also helpful.
Here is /etc/resolv.conf as shown below for the Ubuntu laptop host.
gstanden@vmem1:~$ cat /etc/resolv.conf
# Dynamic resolv.conf(5) file for glibc resolver(3) generated by resolvconf(8)
# DO NOT EDIT THIS FILE BY HAND -- YOUR CHANGES WILL BE OVERWRITTEN
options attempts: 1
options timeout: 1
search vmem.org gns1.vmem.org
nameserver 10.207.39.1
nameserver 10.207.39.3
gstanden@vmem1:~$
Begin Update #1 2014 November 4
Found that the above /etc/resolv.conf does not work as well for my purposes as the following resolv.conf, so, the following /etc/resolv.conf is now the one in use as shown below. Also shown is tne /etc/network/interfaces Ubuntu configuration file which generates this /etc/resolv.conf file at bootup. The bolded entries are the ones which are responsible for generating the required /etc/resolv.conf file.
gstanden@vmem1:~$ cat /etc/resolv.conf
# Dynamic resolv.conf(5) file for glibc resolver(3) generated by resolvconf(8)
# DO NOT EDIT THIS FILE BY HAND -- YOUR CHANGES WILL BE OVERWRITTEN
nameserver 127.0.0.1
search vmem.org gns1.vmem.org
gstanden@vmem1:~$ cat /etc/network/interfaces
# interfaces(5) file used by ifup(8) and ifdown(8)
auto lo
iface lo inet loopback
dns-domain vmem.org
dns-search gns1.vmem.org
dns-nameserver 127.0.0.1
gstanden@vmem1:~$
Why is this better? Because it results in several resolution improvements as listed below. As far as why from an absolute DNS engineering perspective, still working to understand that one. The empirical results however are excellent with this modification, and it meets all resolution requirements, so it is in use now.
Both "lxc1-scan" and "lxc1-scan.gns1.vmem.org" resolve equally well and instantly
With the previous /etc/resolv.conf, "lxc1-scan" did not resolve, and the FQDN version was slightly delayed in resolution by a few seconds
Internet firefox page loads on the Ubuntu host are very fast with this new /etc/resolv.conf.
End Update #1 2014 November 4
Here is /etc/nsswitch.conf for the Ubuntu laptop host
gstanden@vmem1:~$ cat /etc/nsswitch.conf
# /etc/nsswitch.conf
#
# Example configuration of GNU Name Service Switch functionality.
# If you have the `glibc-doc-reference' and `info' packages installed, try:
# `info libc "Name Service Switch"' for information about this file.
passwd: compat
group: compat
shadow: compat
#hosts: dns files mdns4_minimal [NOTFOUND=return]
hosts: files dns nis
networks: files
protocols: db files
services: db files
ethers: db files
rpc: db files
netgroup: nis
gstanden@vmem1:~$
Here is the /etc/resolv.conf for lxcora02 as shown below.
[root@lxcora02 ~]# cat /etc/resolv.conf
options attempts:2 timeout:1
; generated by /sbin/dhclient-script
search vmem.org gns1.vmem.org
nameserver 10.207.39.3
nameserver 10.207.39.1
[root@lxcora02 ~]#
And here is /etc/resolv.conf for lxcora03 as shown below.
[root@lxcora03 ~]# cat /etc/resolv.conf
options attempts:2 timeout:1
; generated by /sbin/dhclient-script
search vmem.org gns1.vmem.org
nameserver 10.207.39.3
nameserver 10.207.39.1
[root@lxcora03 ~]#
The forward zone file for the domain of the GNS ASM Flex Cluster as shown below. The lines added for GNS are shown in bold.
root@vmem1:/var/lib/bind# named-checkzone vmem.org fwd.vmem.org
zone vmem.org/IN: loaded serial 1411021420
OK
root@vmem1:/var/lib/bind# pwd
/var/lib/bind
root@vmem1:/var/lib/bind# ls -l fwd.vmem.org
-rw-r--r-- 1 bind bind 1096 Nov 2 14:20 fwd.vmem.org
root@vmem1:/var/lib/bind# named-checkzone vmem.org fwd.vmem.org
zone vmem.org/IN: loaded serial 1411021420
OK
root@vmem1:/var/lib/bind# cat fwd.vmem.org
$ORIGIN .
$TTL 86400 ; 1 day
vmem.org IN SOA vmem1.vmem.org. postmaster.vmem.org. (
1411021420 ; serial
60 ; refresh (1 minute)
1800 ; retry (30 minutes)
604800 ; expire (1 week)
86400 ; minimum (1 day)
)
NS vmem1.vmem.org.
$ORIGIN vmem.org.
_sflow._udp TXT "txtvers=1" "polling=20" "sampling=512"
SRV 0 0 6343 vmem1
$TTL 3600 ; 1 hour
lxcora01 A 10.207.39.85
TXT "001060b0cef034d0d8b3ec569ea6615637"
lxcora02 A 10.207.39.87
TXT "001d6eb17bf9f9081897f98548b528bfee"
lxcora03 A 10.207.39.88
TXT "0084a3923f9bbe6559acbb98034af0c10a"
$TTL 86400 ; 1 day
lxcora1 A 10.207.39.79
$TTL 3600 ; 1 hour
lxcora3 A 10.207.39.83
TXT "007e7554cced9024a1bac46cc4370293c6"
lxcora5 A 10.207.39.84
TXT "0092ff209913ebc9ca832c943e48eff86f"
$TTL 86400 ; 1 day
oracle631 A 10.207.39.72
oracle632 A 10.207.39.76
oracle652 A 10.207.39.74
vmem1 A 10.207.39.1
lxc1-gns-vip.vmem.org. A 10.207.39.3
$TTL 3600 ; 1 hour
vmem2 A 10.207.39.81
TXT "3118a6c1688c312b822454df8baa10bfff"
$ORIGIN gns1.vmem.org.
@ IN NS lxc1-gns-vip.vmem.org.
root@vmem1:/var/lib/bind#
Note that "lxc1-gns-vip" is a COMPLETELY ARBITRARY choice of name (although the domain is not arbitrary). If we look at a table showing the choices that were made at install time for GNS it can be seen that that DNS name does not appear anywhere in the installed configuration. It is just literally a VIP. It could just as easily have been called "lxc1" (i.e. FQDN "lxc1-gns-vip"). For example, here is the table of relevant values for this GNS setup. The first row is this system just built. The second row are the values Martin Bach used, to help see the way the setup goes. I've constructed this table from the references that I used to help figure this out (the last column "Source" gives the link to the referenced webpage work). The table should help to understand how these diffrent authors completed the same task successfully to configure GNS.
There are some differences worth noting and possibly trying to resolve as to how these configurations differ slightly and which, if any, is "better" or if they are all equivalently "good" (i.e all of these configurations work so all roads lead to Rome as they say).
One slight difference between my configuration and Martin Bach is in the DNS forward lookup zone file. Here are my entries to that file as shown below (showing just the GNS lines condensed for clariry).
lxc1-gns-vip.vmem.org. A 10.207.39.3
$ORIGIN gns1.vmem.org.
@ IN NS lxc1-gns-vip.vmem.org.
Here is what Martin Bach used as shown below. Note that Martin's version has an "IN" on the "A" record line. Need to so some additional research. I went with not using the "A" record because the checkzone program did not like the "A" record in there (as shown further below).
$ORIGIN rac.localdomain.
@ IN NS gns.rac.localdomain.
gns.rac.localdomain. IN A 192.168.99.150
Tried checkzone on the fwd.vmem.org both with and without the "IN" on the "A" record line, and both versions passed, so guess that this is options, but note to self need to understand better what "IN" an "A" denote in forward zone files.
root@vmem1:/var/lib/bind# named-checkzone vmem.org fwd.vmem.org
zone vmem.org/IN: loaded serial 1411021420
OK
root@vmem1:/var/lib/bind# vi fwd.vmem.org
root@vmem1:/var/lib/bind# named-checkzone vmem.org fwd.vmem.org
zone vmem.org/IN: loaded serial 1411021420
OK
Recall that column 5 counting from the left is set at install time as shown below. For example, "lxc1-scan" was used here. The non-advanced install will always set the scan name to "clustername-scan" but in the advanced install as shown below, it looks possible to have a scan that does not include the Cluster Name (bit would not recommend doing this).
Similar to "Mystery Author" the dig lookups can be run to verify correct DNS function as shown below.
gstanden@vmem1:~$ dig lxc1-gns-vip.vmem.org
; <<>> DiG 9.9.5-3-Ubuntu <<>> lxc1-gns-vip.vmem.org
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 62496
;; flags: qr aa rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 1, ADDITIONAL: 2
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;lxc1-gns-vip.vmem.org. IN A
;; ANSWER SECTION:
lxc1-gns-vip.vmem.org. 86400 IN A 10.207.39.3
;; AUTHORITY SECTION:
vmem.org. 86400 IN NS vmem1.vmem.org.
;; ADDITIONAL SECTION:
vmem1.vmem.org. 86400 IN A 10.207.39.1
;; Query time: 0 msec
;; SERVER: 10.207.39.1#53(10.207.39.1)
;; WHEN: Sun Nov 02 18:35:57 CST 2014
;; MSG SIZE rcvd: 102
And for the reverse lookup as shown below.
gstanden@vmem1:~$ dig -x 10.207.39.3
; <<>> DiG 9.9.5-3-Ubuntu <<>> -x 10.207.39.3
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 53494
;; flags: qr aa rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 1, ADDITIONAL: 2
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;3.39.207.10.in-addr.arpa. IN PTR
;; ANSWER SECTION:
3.39.207.10.in-addr.arpa. 86400 IN PTR lxc1-gns-vip.vmem.org.
;; AUTHORITY SECTION:
39.207.10.in-addr.arpa. 86400 IN NS vmem1.vmem.org.
;; ADDITIONAL SECTION:
vmem1.vmem.org. 86400 IN A 10.207.39.1
;; Query time: 0 msec
;; SERVER: 10.207.39.1#53(10.207.39.1)
;; WHEN: Sun Nov 02 18:36:12 CST 2014
;; MSG SIZE rcvd: 124
gstanden@vmem1:~$
These values can be checked using some utilities after installation as shown below (here run from lxcora02).
[grid@lxcora02 grid]$ ./runcluvfy.sh comp gns -postcrsinst -verbose
Verifying GNS integrity
Checking GNS integrity...
Checking if the GNS subdomain name is valid...
The GNS subdomain name "gns1.vmem.org" is a valid domain name
Checking if the GNS VIP belongs to same subnet as the public network...
Public network subnets "10.207.39.0" match the GNS VIP "10.207.39.3"
Checking if the GNS VIP is a valid address...
GNS VIP "10.207.39.3" resolves to a valid IP address
Checking the status of GNS VIP...
Checking if FDQN names for domain "gns1.vmem.org" are reachable
GNS resolved IP addresses are reachable
GNS resolved IP addresses are reachable
GNS resolved IP addresses are reachable
Checking status of GNS resource...
Node Running? Enabled?
------------ ------------------------ ------------------------
lxcora02 no yes
lxcora03 yes yes
GNS resource configuration check passed
Checking status of GNS VIP resource...
Node Running? Enabled?
------------ ------------------------ ------------------------
lxcora02 no yes
lxcora03 yes yes
GNS VIP resource configuration check passed.
GNS integrity check passed
Verification of GNS integrity was successful.
[grid@lxcora02 grid]$
Here is the same check run from lxcora03 as shown below.
[grid@lxcora03 grid]$ ./runcluvfy.sh comp gns -postcrsinst -verbose
Verifying GNS integrity
Checking GNS integrity...
Checking if the GNS subdomain name is valid...
The GNS subdomain name "gns1.vmem.org" is a valid domain name
Checking if the GNS VIP belongs to same subnet as the public network...
Public network subnets "10.207.39.0" match the GNS VIP "10.207.39.3"
Checking if the GNS VIP is a valid address...
GNS VIP "10.207.39.3" resolves to a valid IP address
Checking the status of GNS VIP...
Checking if FDQN names for domain "gns1.vmem.org" are reachable
GNS resolved IP addresses are reachable
GNS resolved IP addresses are reachable
GNS resolved IP addresses are reachable
Checking status of GNS resource...
Node Running? Enabled?
------------ ------------------------ ------------------------
lxcora02 no yes
lxcora03 yes yes
GNS resource configuration check passed
Checking status of GNS VIP resource...
Node Running? Enabled?
------------ ------------------------ ------------------------
lxcora02 no yes
lxcora03 yes yes
GNS VIP resource configuration check passed.
GNS integrity check passed
Verification of GNS integrity was successful.
[grid@lxcora03 grid]$
Also, there is an srvctl command that can be used as shown below from node lxcora02.
[grid@lxcora02 grid]$ srvctl config gns -a
GNS is enabled.
GNS is listening for DNS server requests on port 53
GNS is using port 5353 to connect to mDNS
GNS status: OK
Domain served by GNS: gns1.vmem.org
GNS version: 12.1.0.2.0
Globally unique identifier of the cluster where GNS is running: 381c21634d685f94ffd391a8528e606a
Name of the cluster where GNS is running: lxc1
Cluster type: server.
GNS log level: 1.
GNS listening addresses: tcp://10.207.39.3:11244.
GNS is individually enabled on nodes:
GNS is individually disabled on nodes:
[grid@lxcora02 grid]$
The output of "crsctl stat res -t" also shows GNS status as shown below in bold.
[grid@lxcora03 ~]$ crsctl stat res -t
--------------------------------------------------------------------------------
Name Target State Server State details
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.ASMNET1LSNR_ASM.lsnr
ONLINE ONLINE lxcora02 STABLE
ONLINE ONLINE lxcora03 STABLE
ora.DATA.dg
ONLINE ONLINE lxcora02 STABLE
ONLINE ONLINE lxcora03 STABLE
ora.FRA.dg
ONLINE ONLINE lxcora02 STABLE
ONLINE ONLINE lxcora03 STABLE
ora.IOPS.dg
ONLINE ONLINE lxcora02 STABLE
ONLINE ONLINE lxcora03 STABLE
ora.LISTENER.lsnr
ONLINE ONLINE lxcora02 STABLE
ONLINE ONLINE lxcora03 STABLE
ora.SYSTEMDG.dg
ONLINE ONLINE lxcora02 STABLE
ONLINE ONLINE lxcora03 STABLE
ora.UNDO1.dg
ONLINE ONLINE lxcora02 STABLE
ONLINE ONLINE lxcora03 STABLE
ora.net1.network
ONLINE ONLINE lxcora02 STABLE
ONLINE ONLINE lxcora03 STABLE
ora.ons
ONLINE ONLINE lxcora02 STABLE
ONLINE ONLINE lxcora03 STABLE
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.LISTENER_SCAN1.lsnr
1 ONLINE ONLINE lxcora02 STABLE
ora.LISTENER_SCAN2.lsnr
1 ONLINE ONLINE lxcora03 STABLE
ora.LISTENER_SCAN3.lsnr
1 ONLINE ONLINE lxcora03 STABLE
ora.MGMTLSNR
1 ONLINE ONLINE lxcora03 169.254.28.2 192.210
.39.11 192.211.39.11
,STABLE
ora.asm
1 ONLINE ONLINE lxcora02 Started,STABLE
2 ONLINE ONLINE lxcora03 Started,STABLE
3 OFFLINE OFFLINE STABLE
ora.cvu
1 ONLINE ONLINE lxcora03 STABLE
ora.gns
1 ONLINE ONLINE lxcora03 STABLE
ora.gns.vip
1 ONLINE ONLINE lxcora03 STABLE
ora.lxcora02.vip
1 ONLINE ONLINE lxcora02 STABLE
ora.lxcora03.vip
1 ONLINE ONLINE lxcora03 STABLE
ora.mgmtdb
1 ONLINE ONLINE lxcora03 Open,STABLE
ora.oc4j
1 ONLINE ONLINE lxcora03 STABLE
ora.scan1.vip
1 ONLINE ONLINE lxcora02 STABLE
ora.scan2.vip
1 ONLINE ONLINE lxcora03 STABLE
ora.scan3.vip
1 ONLINE ONLINE lxcora03 STABLE
ora.vmem1.db
1 ONLINE ONLINE lxcora02 Open,STABLE
2 ONLINE ONLINE lxcora03 Open,STABLE
--------------------------------------------------------------------------------
[grid@lxcora03 ~]$
There are alot of other options that can be run on the srvctl command as shown below for GNS verification and info.
[grid@lxcora03 ~]$ srvctl config gns -help
Displays the configuration information for the GNS daemon.
Usage: srvctl config gns [-detail] [-subdomain] [-multicastport] [-node <node_name>] [-port] [-status] [-version] [-query <name>] [-list] [-clusterguid] [-clustername] [-clustertype] [-loglevel] [-network] [-verbose]
-detail Print detailed configuration information
-subdomain Display subdomain served by GNS
-multicastport Display the port on which the GNS daemon is listening for multicast requests
-node <node_name> Display the configuration information for GNS on the specified node.
-port Display the port which the GNS daemon uses to communicate with the DNS server.
-network Display network on which GNS is listening
-status Display the status of GNS
-version Display the version of GNS
-query <name> Query GNS for the records belonging to a name.
-list List all records in GNS.
-clusterguid Display the globally unique identifier of the cluster where GNS is running
-clustername Display the name of the cluster where GNS is running
-clustertype Display the type of configuration of GNS on this cluster
-loglevel Print the log level of the GNS
-verbose Verbose output
-help Print usage
[grid@lxcora03 ~]$ srvctl config gns -clustertype
Connecting to the database from Ubuntu host laptop.
[oracle@lxcora02 ~]$ sqlplus system/Violin#1@//lxc1-scan.gns1.vmem.org:1521/VMEM1
SQL*Plus: Release 12.1.0.2.0 Production on Mon Nov 3 00:29:48 2014
Copyright (c) 1982, 2014, Oracle. All rights reserved.
Last Successful login time: Sun Nov 02 2014 21:28:06 -05:00
Connected to:
Oracle Database 12c Enterprise Edition Release 12.1.0.2.0 - 64bit Production
With the Partitioning, Real Application Clusters, Automatic Storage Management, OLAP,
Advanced Analytics and Real Application Testing options
SQL> select name, open_mode from v$database;
NAME OPEN_MODE
--------- --------------------
VMEM1 READ WRITE
SQL> exit
Disconnected from Oracle Database 12c Enterprise Edition Release 12.1.0.2.0 - 64bit Production
With the Partitioning, Real Application Clusters, Automatic Storage Management, OLAP,
Advanced Analytics and Real Application Testing options
[oracle@lxcora02 ~]$
Connect to DB from Ubuntu Host
The database is accessible from the Ubuntu host using the instantclient software from Oracle (free). Instructions on how to obtain and install the software are here. Connecting to the database using Oracle Instantclient is shown below.
gstanden@vmem1:~$ sqlplus system/Violin#1@//lxc1-scan.gns1.vmem.org:1521/VMEM1
SQL*Plus: Release 12.1.0.2.0 Production on Mon Nov 3 12:05:55 2014
Copyright (c) 1982, 2014, Oracle. All rights reserved.
Last Successful login time: Mon Nov 03 2014 09:41:48 -06:00
Connected to:
Oracle Database 12c Enterprise Edition Release 12.1.0.2.0 - 64bit Production
With the Partitioning, Real Application Clusters, Automatic Storage Management, OLAP,
Advanced Analytics and Real Application Testing options
SQL> exit
Disconnected from Oracle Database 12c Enterprise Edition Release 12.1.0.2.0 - 64bit Production
With the Partitioning, Real Application Clusters, Automatic Storage Management, OLAP,
Advanced Analytics and Real Application Testing options
gstanden@vmem1:~$ sqlplus system/Violin#1@//lxc1-scan.gns1.vmem.org:1521/VMEM1
SQL*Plus: Release 12.1.0.2.0 Production on Mon Nov 3 14:27:21 2014
Copyright (c) 1982, 2014, Oracle. All rights reserved.
Last Successful login time: Mon Nov 03 2014 12:05:57 -06:00
Connected to:
Oracle Database 12c Enterprise Edition Release 12.1.0.2.0 - 64bit Production
With the Partitioning, Real Application Clusters, Automatic Storage Management, OLAP,
Advanced Analytics and Real Application Testing options
SQL> select name, open_mode from v$database;
NAME OPEN_MODE
--------- --------------------
VMEM1 READ WRITE
SQL> select instance_name from v$instance;
INSTANCE_NAME
----------------
VMEM11
SQL> exit
Disconnected from Oracle Database 12c Enterprise Edition Release 12.1.0.2.0 - 64bit Production
With the Partitioning, Real Application Clusters, Automatic Storage Management, OLAP,
Advanced Analytics and Real Application Testing options
gstanden@vmem1:~$