Robin 12c RAC ASM Flex Cluster
This page is a WIP and will eventually be a complete detailed guide for deploying Oracle 12c ASM Flex Clusters on Robin Systems Containers. This build is done by installing Robin in a CentOS7 VM and then creating Robin Application Containers for Oracle inside the VM.
Build Robin Host Server VM
Build a VM for the Robin Host server. The VM needs to have the following amounts of storage allocated.
Run "yum update" to updated the VM
Snapshot VM After Initial Build (before any configurations)
Undefine and Destroy Default Virsh Network
[root@centos-72a ~]# ifconfig
enp0s17: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 10.207.39.10 netmask 255.255.255.0 broadcast 10.207.39.255
inet6 fe80::a00:27ff:fed7:13db prefixlen 64 scopeid 0x20<link>
ether 08:00:27:d7:13:db txqueuelen 1000 (Ethernet)
RX packets 241303 bytes 344983390 (329.0 MiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 42427 bytes 2899359 (2.7 MiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
lo: flags=73<UP,LOOPBACK,RUNNING> mtu 65536
inet 127.0.0.1 netmask 255.0.0.0
inet6 ::1 prefixlen 128 scopeid 0x10<host>
loop txqueuelen 0 (Local Loopback)
RX packets 4 bytes 340 (340.0 B)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 4 bytes 340 (340.0 B)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
virbr0: flags=4099<UP,BROADCAST,MULTICAST> mtu 1500 <-- Get rid of this network as shown below
inet 192.168.122.1 netmask 255.255.255.0 broadcast 192.168.122.255
ether 52:54:00:28:97:91 txqueuelen 0 (Ethernet)
RX packets 0 bytes 0 (0.0 B)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 0 bytes 0 (0.0 B)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
[root@centos-72a ~]# virsh net-destroy default <-- Destroys it now (but on reboot will reappear)
Network default destroyed
[root@centos-72a ~]# virsh net-undefine default <-- Destroys it permanentsly (gone on reboot also)
Network default has been undefined
[root@centos-72a ~]#
Install Guest Additions (Optional, VirtualBox VM)
Also, enable bi-directional drag-and-drop and bi-directional clipboard.
Add Two VMDK Files to Robin Host VM
Shutdown the VM and add the following virtual disk files using the VirtualBox GUI as described below.
250 Gb file for Robin Host Server block storage. Check the "SSD emulation" box.
150 Gb file for SCST filesystem file-backed LUN storage. Check the "SSD emulation" box.
Create an EXT4 Filesystem on the 150Gb Virtual Block device
Partition the smaller 150Gb disk and create an ext4 filesystem on the smaller 150Gb virtual block device which will be used for storing SCST file-backed LUN device files.
oracle@stlns01:~$ ssh root@centos-72a
root@centos-72a's password:
Last login: Wed Sep 7 12:17:58 2016 from stlns01.robinsystems.com
[root@centos-72a ~]# fdisk -l /dev/sd*
Disk /dev/sda: 268.4 GB, 268435456000 bytes, 524288000 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk label type: dos
Disk identifier: 0x000888c0
Device Boot Start End Blocks Id System
/dev/sda1 * 2048 1026047 512000 83 Linux
/dev/sda2 1026048 524287999 261630976 8e Linux LVM
Disk /dev/sda1: 524 MB, 524288000 bytes, 1024000 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk /dev/sda2: 267.9 GB, 267910119424 bytes, 523261952 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk /dev/sdb: 268.4 GB, 268435456000 bytes, 524288000 sectors <-- New 250Gb disk for Robin storage
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk /dev/sdc: 161.1 GB, 161061273600 bytes, 314572800 sectors <-- New 150Gb disk for SCST storage
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
[root@centos-72a ~]# gdisk /dev/sdc
GPT fdisk (gdisk) version 0.8.6
Partition table scan:
MBR: not present
BSD: not present
APM: not present
GPT: not present
Creating new GPT entries.
Command (? for help): n
Partition number (1-128, default 1):
First sector (34-314572766, default = 2048) or {+-}size{KMGTP}:
Last sector (2048-314572766, default = 314572766) or {+-}size{KMGTP}:
Current type is 'Linux filesystem'
Hex code or GUID (L to show codes, Enter = 8300):
Changed type of partition to 'Linux filesystem'
Command (? for help): p
Disk /dev/sdc: 314572800 sectors, 150.0 GiB
Logical sector size: 512 bytes
Disk identifier (GUID): 3F3BCA97-8C97-4DF6-9BB5-2714AD28168B
Partition table holds up to 128 entries
First usable sector is 34, last usable sector is 314572766
Partitions will be aligned on 2048-sector boundaries
Total free space is 2014 sectors (1007.0 KiB)
Number Start (sector) End (sector) Size Code Name
1 2048 314572766 150.0 GiB 8300 Linux filesystem
Command (? for help): w
Final checks complete. About to write GPT data. THIS WILL OVERWRITE EXISTING
PARTITIONS!!
Do you want to proceed? (Y/N): Y
OK; writing new GUID partition table (GPT) to /dev/sdc.
The operation has completed successfully.
[root@centos-72a ~]# mkfs -t ext4 /dev/sdc1
mke2fs 1.42.9 (28-Dec-2013)
Filesystem label=
OS type: Linux
Block size=4096 (log=2)
Fragment size=4096 (log=2)
Stride=0 blocks, Stripe width=0 blocks
9830400 inodes, 39321339 blocks
1966066 blocks (5.00%) reserved for the super user
First data block=0
Maximum filesystem blocks=2187329536
1200 block groups
32768 blocks per group, 32768 fragments per group
8192 inodes per group
Superblock backups stored on blocks:
32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208,
4096000, 7962624, 11239424, 20480000, 23887872
Allocating group tables: done
Writing inode tables: done
Creating journal (32768 blocks): done
Writing superblocks and filesystem accounting information: done
[root@centos-72a ~]#
Mount the SCST ext4 Filesystem
Mount the SCST ext4 filesystem and update the /etc/fstab as shown below.
[root@centos-72a ~]# cd /
[root@centos-72a /]# mkdir scst-storage
[root@centos-72a /]# mount -t ext4 /dev/sdc1 /scst-storage
[root@centos-72a /]# df -TH
Filesystem Type Size Used Avail Use% Mounted on
/dev/mapper/centos-root xfs 54G 4.5G 50G 9% /
devtmpfs devtmpfs 26G 0 26G 0% /dev
tmpfs tmpfs 26G 99k 26G 1% /dev/shm
tmpfs tmpfs 26G 9.2M 26G 1% /run
tmpfs tmpfs 26G 0 26G 0% /sys/fs/cgroup
/dev/mapper/centos-home xfs 189G 39M 189G 1% /home
/dev/sda1 xfs 521M 220M 302M 43% /boot
tmpfs tmpfs 5.1G 13k 5.1G 1% /run/user/1000
/dev/sr0 iso9660 60M 60M 0 100% /run/media/gstanden/VBOXADDITIONS_5.1.4_110228
tmpfs tmpfs 5.1G 0 5.1G 0% /run/user/0
/dev/sdc1 ext4 159G 63M 151G 1% /scst-storage
[root@centos-72a /]# vi /etc/fstab
[root@centos-72a /]# cat /etc/fstab
#
# /etc/fstab
# Created by anaconda on Wed Sep 7 10:34:58 2016
#
# Accessible filesystems, by reference, are maintained under '/dev/disk'
# See man pages fstab(5), findfs(8), mount(8) and/or blkid(8) for more info
#
/dev/mapper/centos-root / xfs defaults 0 0
UUID=dba3fdde-d5fa-4f01-8482-5436a3337daa /boot xfs defaults 0 0
/dev/mapper/centos-home /home xfs defaults 0 0
/dev/mapper/centos-swap swap swap defaults 0 0
/dev/sdc1 /scst-storage ext4 defaults 0 0
[root@centos-72a /]#
File /var/lib/lxc/vnode-39-116.robinsystems.com/config
[root@centos-72k vnode-39-116.robinsystems.com]# cat config
lxc.tty = 1
lxc.include = /usr/share/lxc/config/ubuntu.common.conf
# Robin Default Network Start
lxc.network.type = veth
lxc.network.flags = up
lxc.network.name = eth0
lxc.network.script.up = /usr/local/bin/lxc_ovs
lxc.network.script.down = /usr/local/bin/lxc_ovs
lxc.network.hwaddr = 00:16:0a:cf:27:74
lxc.network.ipv4 = 10.207.39.116/24
lxc.network.ipv4.gateway = 10.207.39.1
# Robin Default Network End
# Oracle Private Network Begin
lxc.network.type = veth
lxc.network.flags = up
lxc.network.name = eth1
lxc.network.script.up = /usr/local/bin/lxc_ovs
lxc.network.script.down = /usr/local/bin/lxc_ovs
lxc.network.hwaddr = 00:16:0a:57:c9:0d
lxc.network.ipv4 = 192.220.39.116/24
# Oracle Private Network End
lxc.utsname = vnode-39-116.robinsystems.com
lxc.rootfs = /usr/local/robin/instances/vnode-39-116.robinsystems.com
lxc.mount = /var/lib/lxc/vnode-39-116.robinsystems.com/fstab
lxc.cgroup.cpuset.cpus = 2,1
lxc.cgroup.memory.limit_in_bytes = 4294967296
lxc.rootfs = /usr/local/robin/instances/vnode-39-116.robinsystems.com
lxc.rootfs.backend = dir
# Devices, Filesystems, Mounts Begin
lxc.cgroup.devices.allow = a # Allow access to all devices
lxc.cgroup.devices.allow = a 10:236 rwm
lxc.cgroup.devices.allow = a 252:* rwm
lxc.cgroup.devices.allow = a 8:* rwm
lxc.mount.entry = /dev/mapper /usr/local/robin/instances/vnode-39-116.robinsystems.com/dev/mapper none defaults,bind,create=dir 0 0
lxc.mount.entry = /dev /usr/local/robin/instances/vnode-39-116.robinsystems.com/dev none defaults,bind,create=dir 0 0
lxc.mount.entry = shm dev/shm tmpfs size=3500m,nosuid,nodev,noexec,create=dir 0 0
lxc.mount.auto = proc sys cgroup
lxc.mount.auto = proc:rw sys:rw cgroup-full:rw
# Devices, Filesystems, Mounts End
[root@centos-72k vnode-39-116.robinsystems.com]#
File /usr/local/bin/lxc-ovs
This file will be installed by default when Robin Server is installed in the VM.
File /etc/scst.conf
[root@centos-72k ~]# cat /etc/scst.conf
# Automatically generated by SCST Configurator v3.1.0-pre1.
HANDLER vdisk_fileio {
DEVICE asm_sysdk_00 {
filename /asm0/asm_sysdk_00.img
}
}
TARGET_DRIVER copy_manager {
TARGET copy_manager_tgt {
LUN 1 asm_sysdk_00
}
}
TARGET_DRIVER iscsi {
enabled 1
TARGET iqn.2016-09.com.robinsystems:centos-72k.san.asm.luns {
enabled 1
rel_tgt_id 2
GROUP lxc1 {
LUN 0 asm_sysdk_00
INITIATOR iqn.1994-05.com.redhat:e1ab03b1eb5
}
}
}
[root@centos-72k ~]#
File /etc/sysctl.conf
[root@centos-72k ~]# cat /etc/sysctl.conf
# System default settings live in /usr/lib/sysctl.d/00-system.conf.
# To override those settings, enter new settings here, or in an /etc/sysctl.d/<name>.conf file
#
# For more information, see sysctl.conf(5) and sysctl.d(5).
# custom core file pattern
kernel.core_pattern = /var/crash/core.%e.%p.%h.%t
# Disable ipv6
net.ipv6.conf.all.disable_ipv6 = 1
net.ipv6.conf.default.disable_ipv6 = 1
net.ipv6.conf.lo.disable_ipv6 = 1
# Oracle Settings
# /etc/sysctl.conf - Configuration file for setting system variables
# See /etc/sysctl.d/ for additional system variables.
# See sysctl.conf (5) for information.
#
#kernel.domainname = example.com
# Uncomment the following to stop low-level messages on console
#kernel.printk = 3 4 1 3
##############################################################3
# Functions previously found in netbase
#
# Uncomment the next two lines to enable Spoof protection (reverse-path filter)
# Turn on Source Address Verification in all interfaces to
# prevent some spoofing attacks
# GLS 20141226 http://thenewstack.io/solving-a-common-beginners-problem-when-pinging-from-an-openstack-instance/
net.ipv4.conf.default.rp_filter=0
net.ipv4.conf.all.rp_filter=0
# Uncomment the next line to enable TCP/IP SYN cookies
# See http://lwn.net/Articles/277146/
# Note: This may impact IPv6 TCP sessions too
#net.ipv4.tcp_syncookies=1
# Uncomment the next line to enable packet forwarding for IPv4
# GLS 20141226 http://thenewstack.io/solving-a-common-beginners-problem-when-pinging-from-an-openstack-instance/
net.ipv4.ip_forward=1
# Uncomment the next line to enable packet forwarding for IPv6
# Enabling this option disables Stateless Address Autoconfiguration
# based on Router Advertisements for this host
#net.ipv6.conf.all.forwarding=1
###################################################################
# Additional settings - these settings can improve the network
# security of the host and prevent against some network attacks
# including spoofing attacks and man in the middle attacks through
# redirection. Some network environments, however, require that these
# settings are disabled so review and enable them as needed.
#
# Do not accept ICMP redirects (prevent MITM attacks)
#net.ipv4.conf.all.accept_redirects = 0
#net.ipv6.conf.all.accept_redirects = 0
# _or_
# Accept ICMP redirects only for gateways listed in our default
# gateway list (enabled by default)
# net.ipv4.conf.all.secure_redirects = 1
#
# Do not send ICMP redirects (we are not a router)
#net.ipv4.conf.all.send_redirects = 0
#
# Do not accept IP source route packets (we are not a router)
#net.ipv4.conf.all.accept_source_route = 0
#net.ipv6.conf.all.accept_source_route = 0
#
# Log Martian Packets
#net.ipv4.conf.all.log_martians = 1
#
# Oracle
#
kernel.shmmni = 4096
kernel.sem = 250 32000 100 128
fs.file-max = 6815744
fs.aio-max-nr = 1048576
net.ipv4.ip_local_port_range = 9000 65500
net.core.rmem_default = 262144
net.core.rmem_max = 4194304
net.core.wmem_default = 262144
net.core.wmem_max = 1048576
vm.nr_hugepages = 2060
kernel.panic_on_oops = 1
#Tim's Settings
kernel.shmall = 1073741824
kernel.shmmax = 4398046511104
[root@centos-72k ~]#
Configure HugePages on Robin Host VM
Run the following commands as shown below as root.
[root@centos-72a /]# grep Huge /proc/meminfo
AnonHugePages: 167936 kB
HugePages_Total: 0
HugePages_Free: 0
HugePages_Rsvd: 0
HugePages_Surp: 0
Hugepagesize: 2048 kB
[root@centos-72a /]# vi /etc/default/grub
[root@centos-72a /]# cat /etc/default/grub
GRUB_TIMEOUT=5
GRUB_DISTRIBUTOR="$(sed 's, release .*$,,g' /etc/system-release)"
GRUB_DEFAULT=saved
GRUB_DISABLE_SUBMENU=true
GRUB_TERMINAL_OUTPUT="console"
GRUB_CMDLINE_LINUX="crashkernel=auto rd.lvm.lv=centos/root rd.lvm.lv=centos/swap transparent_hugepage=never"
GRUB_DISABLE_RECOVERY="true"
[root@centos-72a /]#
Run the following commands to implement HugePages as shown below.
[root@centos-72a /]# sysctl -w vm.nr_hugepages=2060
vm.nr_hugepages = 2060
[root@centos-72a /]# echo 'vm.nr_hugepages=2060' > /etc/sysctl.d/hugepages.conf
[root@centos-72a /]# sysctl -p
[root@centos-72a /]#
Edit the /etc/default/grub file and add the "transparent_hugepage=never" kernel directive as shown below in bold. It is recommended to also remove the "rhgb quiet" while adding this so that boot and shutdown details are observable from the console.
Add these lines to the end of /etc/security/limits.conf in the Robin Host VM as shown below. These are calculated using the formula: (2048)x(# of hugepages) which in this example is (2048)x(2060) =
* soft memlock 4218880
* hard memlock 4218880
Check the current settings and then reboot to ensure the new settings take effect (disable of THP and activation of HugePages). The pre-reboot setting after the above commands but before reboot is shown below.
[root@centos-72a /]# grep Huge /proc/meminfo
AnonHugePages: 167936 kB
HugePages_Total: 2060
HugePages_Free: 2060
HugePages_Rsvd: 0
HugePages_Surp: 0
Hugepagesize: 2048 kB
[root@centos-72a /]#
Disable Transparent Huge Pages (THP) on Robin Host VM
[root@centos-72k ~]# cat /etc/default/grub
GRUB_TIMEOUT=5
GRUB_DISTRIBUTOR="$(sed 's, release .*$,,g' /etc/system-release)"
GRUB_DEFAULT=saved
GRUB_DISABLE_SUBMENU=true
GRUB_TERMINAL_OUTPUT="console"
GRUB_CMDLINE_LINUX="crashkernel=auto rd.lvm.lv=centos/root rd.lvm.lv=centos/swap transparent_hugepage=never"
GRUB_DISABLE_RECOVERY="true"
After updating /etc/default/grub as shown in the preceding step, next run the following command to implement the changes to transparent_hugepages kernel setting as shown below.
[root@centos-72a ~]# grub2-mkconfig -o /boot/grub2/grub.cfg
Generating grub configuration file ...
Found linux image: /boot/vmlinuz-3.10.0-327.28.3.el7.x86_64
Found initrd image: /boot/initramfs-3.10.0-327.28.3.el7.x86_64.img
Found linux image: /boot/vmlinuz-3.10.0-327.el7.x86_64
Found initrd image: /boot/initramfs-3.10.0-327.el7.x86_64.img
Found linux image: /boot/vmlinuz-0-rescue-b4e9202df0bb4b1687ccbbd09feb6fb6
Found initrd image: /boot/initramfs-0-rescue-b4e9202df0bb4b1687ccbbd09feb6fb6.img
done
[root@centos-72a ~]#
Reboot the Robin Host VM to verify changes take effect on reboot.
oracle@stlns01:~$ ssh root@centos-72a
root@centos-72a's password:
Last login: Wed Sep 7 12:54:19 2016 from stlns01.robinsystems.com
[root@centos-72a ~]# grep Huge /proc/meminfo
AnonHugePages: 0 kB <-- Transparent Huge Pages (THP) are 0 indicating disabled successfully
HugePages_Total: 2060
HugePages_Free: 2060 <-- HugePages quantity 2060 are enabled and ready for Oracle
HugePages_Rsvd: 0
HugePages_Surp: 0
Hugepagesize: 2048 kB
[root@centos-72a ~]#
Create the /etc/sysctl.d/oracle.conf file as shown below.
[root@centos-72a sysctl.d]# cat oracle.conf
# http://thenewstack.io/solving-a-common-beginners-problem-when-pinging-from-an-openstack-instance/
# Container Networking Support
net.ipv4.conf.default.rp_filter=0
net.ipv4.conf.all.rp_filter=0
net.ipv4.ip_forward=1
# Oracle
# GLS shmall 40% of physical memory in 4K pages
# kernel.shmall = 3174488
# GLS shmmax 50% of physical memory in bytes
# kernel.shmmax = 16253378560
kernel.shmmni = 4096
kernel.sem = 250 32000 100 128
fs.file-max = 6815744
fs.aio-max-nr = 1048576
net.ipv4.ip_local_port_range = 9000 65500
net.core.rmem_default = 262144
net.core.rmem_max = 4194304
net.core.wmem_default = 262144
net.core.wmem_max = 1048576
kernel.panic_on_oops = 1
[root@centos-72a sysctl.d]# sysctl -a | grep rmem
net.core.rmem_default = 262144
net.core.rmem_max = 4194304
net.ipv4.tcp_rmem = 4096 87380 6291456
net.ipv4.udp_rmem_min = 4096
[root@centos-72a sysctl.d]# sysctl -a | grep vmem
[root@centos-72a sysctl.d]# sysctl -a | grep wmem
net.core.wmem_default = 262144
net.core.wmem_max = 1048576
net.ipv4.tcp_wmem = 4096 16384 4194304
net.ipv4.udp_wmem_min = 4096
vm.lowmem_reserve_ratio = 256 256 32
[root@centos-72a sysctl.d]#
Transfer the /home/oracle/Downloads/Robin directory to the VM
Build DNS/DHCP Inside Robin Host VM (Optional)
The DNS/DHCP solution can be in the customer network, or it can be built into the VM for demo purposes. Since the Robin Host VM is a CentOS7 VM, the required files for a DNS/DHCP solution inside the VM are shown below. There is additional information here on doing this.
File /etc/named.conf
//
// named.conf
//
// Provided by Red Hat bind package to configure the ISC BIND named(8) DNS
// server as a caching only nameserver (as a localhost DNS resolver only).
//
// See /usr/share/doc/bind*/sample/ for example named configuration files.
//
acl "allowed" { 10.207.39.0/24; localhost; };
// options {
// allow-query { allowed; };
// allow-recursion {allowed;}; //who can access DNS server cache
// };
options {
listen-on port 53 { 10.207.39.1; 127.0.0.1; };
listen-on-v6 port 53 { ::1; };
directory "/var/named";
dump-file "/var/named/data/cache_dump.db";
statistics-file "/var/named/data/named_stats.txt";
memstatistics-file "/var/named/data/named_mem_stats.txt";
allow-query { localhost; 10.207.39.0/24; };
forwarders { 8.8.8.8; 127.0.0.1; };
recursion yes;
dnssec-enable no;
dnssec-validation no;
dnssec-lookaside auto;
bindkeys-file "/etc/named.iscdlv.key";
managed-keys-directory "/var/named/dynamic";
pid-file "/run/named/named.pid";
session-keyfile "/run/named/session.key";
};
logging { channel default_debug { file "data/named.run"; severity dynamic; }; };
zone "." IN {
type hint;
file "named.ca";
};
zone "robinsystems.com" IN {
type master;
file "data/fwd.robinsystems.com";
allow-update { key rndc-key; };
notify yes;
};
zone "39.207.10.in-addr.arpa" IN {
type master;
file "data/rev.robinsystems.com";
allow-update { key rndc-key; };
notify yes;
};
include "/etc/named.rfc1912.zones";
include "/etc/named.root.key";
include "/etc/rndc.key";
File /etc/rndc.key
key "rndc-key" {
algorithm hmac-md5;
secret "nMUCl4S2iljW5G7lpk5UXg==";
};
File /var/named/data/fwd.robinsystems.com
Also in addition to this file and the others, it is necessary to run the "setsebool" command so that DHCP can update the DNS zone files.
root@stlns01:/mnt/centos-root/var/named/data# cat fwd.robinsystems.com
$ORIGIN .
$TTL 86400 ; 1 day
robinsystems.com IN SOA stlns01.robinsystems.com. postmaster.robinsystems.com. (
1512181501 ; serial
60 ; refresh (1 minute)
1800 ; retry (30 minutes)
604800 ; expire (1 week)
86400 ; minimum (1 day)
)
NS stlns01.robinsystems.com.
$ORIGIN robinsystems.com.
_sflow._udp TXT "txtvers=1" "polling=20" "sampling=512"
SRV 0 0 6343 stlns01
$TTL 86400 ; 1 day
gns1 NS lxc1-gns-vip
lxc1 A 10.207.39.3
ns CNAME stlns01
stlns01 A 10.207.39.1
vnode-39-101 A 10.207.39.101
vnode-39-102 A 10.207.39.102
vnode-39-103 A 10.207.39.103
vnode-39-104 A 10.207.39.104
vnode-39-105 A 10.207.39.105
vnode-39-106 A 10.207.39.106
vnode-39-107 A 10.207.39.107
vnode-39-108 A 10.207.39.108
vnode-39-109 A 10.207.39.109
vnode-39-110 A 10.207.39.110
vnode-39-111 A 10.207.39.111
vnode-39-112 A 10.207.39.112
vnode-39-113 A 10.207.39.113
vnode-39-114 A 10.207.39.114
vnode-39-115 A 10.207.39.115
vnode-39-116 A 10.207.39.116
vnode-39-117 A 10.207.39.117
vnode-39-118 A 10.207.39.118
vnode-39-119 A 10.207.39.119
vnode-39-120 A 10.207.39.120
vnode-39-121 A 10.207.39.121
vnode-39-122 A 10.207.39.122
vnode-39-123 A 10.207.39.123
vnode-39-124 A 10.207.39.124
vnode-39-125 A 10.207.39.125
vnode-39-126 A 10.207.39.126
vnode-39-127 A 10.207.39.127
vnode-39-128 A 10.207.39.128
vnode-39-129 A 10.207.39.129
vnode-39-130 A 10.207.39.130
root@stlns01:/mnt/centos-root/var/named/data#
File /usr/lib/systemd/system/dhcpd.service
root@stlns01:/mnt/centos-root/usr/lib/systemd/system# more dhcpd.service
[Unit]
Description=DHCPv4 Server Daemon
Documentation=man:dhcpd(8) man:dhcpd.conf(5)
Wants=network-online.target
After=network-online.target
After=time-sync.target
[Service]
Type=notify
ExecStart=/usr/sbin/dhcpd -f -cf /etc/dhcp/dhcpd.conf -user dhcpd -group dhcpd --no-pid
[Install]
WantedBy=multi-user.target
root@stlns01:/mnt/centos-root/usr/lib/systemd/system# ls -l dhcpd.service
-rw-r--r--. 1 root root 313 Nov 19 2015 dhcpd.service
root@stlns01:/mnt/centos-root/usr/lib/systemd/system#
Build Robin Application Container for Oracle RAC
File /etc/sysctl.conf
[root@vnode-39-116 ~]# cat /etc/sysctl.conf
# Kernel sysctl configuration file for Red Hat Linux
#
# For binary values, 0 is disabled, 1 is enabled. See sysctl(8) and
# sysctl.conf(5) for more details.
# Controls IP packet forwarding
net.ipv4.ip_forward = 0
# Controls source route verification
net.ipv4.conf.default.rp_filter = 1
# Do not accept source routing
net.ipv4.conf.default.accept_source_route = 0
# Controls the System Request debugging functionality of the kernel
kernel.sysrq = 0
# Controls whether core dumps will append the PID to the core filename.
# Useful for debugging multi-threaded applications.
kernel.core_uses_pid = 1
# Controls the use of TCP syncookies
net.ipv4.tcp_syncookies = 1
# Controls the default maxmimum size of a mesage queue
kernel.msgmnb = 65536
# Controls the maximum size of a message, in bytes
kernel.msgmax = 65536
# Controls the maximum shared segment size, in bytes
kernel.shmmax = 4398046511104
# Controls the maximum number of shared memory segments, in pages
kernel.shmall = 1073741824
# Parameters added for Oracle db
fs.aio-max-nr = 1048576
fs.file-max = 6815744
kernel.shmall = 2097152
kernel.shmmax = 2147483648
kernel.shmmni = 4096
kernel.sem = 250 32000 100 128
net.ipv4.ip_local_port_range = 9000 65500
net.core.rmem_default = 262144
net.core.rmem_max = 4194304
net.core.wmem_default = 262144
net.core.wmem_max = 1048586
# (Orabuntu-LXC Settings)
# Disable netfilter on bridges.
net.bridge.bridge-nf-call-ip6tables = 0
net.bridge.bridge-nf-call-iptables = 0
net.bridge.bridge-nf-call-arptables = 0
kernel.panic_on_oops = 1
# Oracle 12c Settings
# http://docs.oracle.com/cd/E11882_01/relnotes.112/e23558/toc.htm
# 5.1.2 Multiple Private Interconnects and Oracle Linux
# net.ipv4.conf.ethX.rp_filter settings
net.ipv4.conf.eth0.rp_filter = 1
net.ipv4.conf.eth1.rp_filter = 2
[root@vnode-39-116 ~]#
File /etc/resolv.conf
[oracle@vnode-39-116 ~]$ cat /etc/resolv.conf
options attempts: 5
options timeout: 5
options rotate
search gns1.robinsystems.com
nameserver 10.207.39.3
[oracle@vnode-39-116 ~]$
File /etc/security/limits.conf
[oracle@vnode-39-116 ~]$ cat /etc/security/limits.conf
# /etc/security/limits.conf
#
#Each line describes a limit for a user in the form:
#
#<domain> <type> <item> <value>
#
#Where:
#<domain> can be:
# - a user name
# - a group name, with @group syntax
# - the wildcard *, for default entry
# - the wildcard %, can be also used with %group syntax,
# for maxlogin limit
#
#<type> can have the two values:
# - "soft" for enforcing the soft limits
# - "hard" for enforcing hard limits
#
#<item> can be one of the following:
# - core - limits the core file size (KB)
# - data - max data size (KB)
# - fsize - maximum filesize (KB)
# - memlock - max locked-in-memory address space (KB)
# - nofile - max number of open file descriptors
# - rss - max resident set size (KB)
# - stack - max stack size (KB)
# - cpu - max CPU time (MIN)
# - nproc - max number of processes
# - as - address space limit (KB)
# - maxlogins - max number of logins for this user
# - maxsyslogins - max number of logins on the system
# - priority - the priority to run user process with
# - locks - max number of file locks the user can hold
# - sigpending - max number of pending signals
# - msgqueue - max memory used by POSIX message queues (bytes)
# - nice - max nice priority allowed to raise to values: [-20, 19]
# - rtprio - max realtime priority
#
#<domain> <type> <item> <value>
#
#* soft core 0
#* hard rss 10000
#@student hard nproc 20
#@faculty soft nproc 20
#@faculty hard nproc 50
#ftp hard nproc 0
#@student - maxlogins 4
# Oracle
oracle soft nofile 1024
oracle hard nofile 65536
oracle soft nproc 16384
oracle hard nproc 16384
oracle soft stack 10240
oracle hard stack 32768
oracle hard memlock 3774874
# Oracle (Robin Default) Begin
# oracle soft nproc 2047
# oracle hard nproc 16384
# oracle soft nofile 1024
# oracle hard nofile 65536
# Oracle (Robin Default) End
# End of file
[oracle@vnode-39-116 ~]$
File /etc/hosts
[oracle@vnode-39-116 ~]$ cat /etc/hosts
127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4
::1 localhost localhost.localdomain localhost6 localhost6.localdomain6
10.207.39.37 centos-72k.robinsystems.com centos-72k
10.207.39.116 vnode-39-116.robinsystems.com vnode-39-116
10.207.39.118 vnode-39-118.robinsystems.com vnode-39-118
[oracle@vnode-39-116 ~]$
File /etc/rc.local
[oracle@vnode-39-116 ~]$ cat /etc/rc.local
#!/bin/sh
#
# This script will be executed *after* all the other init scripts.
# You can put your own initialization stuff in here if you don't
# want to do the full Sys V style init stuff.
ntpd
service ntpd start
chown oracle:asmadmin /dev/mapper/asm*
chmod 0660 /dev/mapper/asm*
touch /var/lock/subsys/local
[oracle@vnode-39-116 ~]$
File /root/create-users.sh
groupadd -g 54323 oper
groupadd -g 54324 backupdba
groupadd -g 54325 dgdba
groupadd -g 54326 kmdba
groupadd -g 54327 asmdba
groupadd -g 54328 asmoper
groupadd -g 54329 asmadmin
usermod -a -G dba,oper,backupdba,dgdba,kmdba,asmdba,asmoper,asmadmin oracle
File /root/create-dirs.sh
mkdir -p /u00/app/12.1.0.2/grid
mkdir -p /u00/app/oracle/product/12.1.0.2/db_1
chown -R oracle:oinstall /u00
chmod -R 775 /u00/
File /root/testdns.sh
[root@vnode-39-116 ~]# cat testdns.sh
echo "================================"
echo "google "
echo "================================"
nslookup google.com
ping -c 3 google.com
function CheckPing {
ping -c 3 google.com | grep packet | cut -f1 -d'%' | cut -f6 -d' '
}
Ping=$(CheckPing)
if [ $Ping -eq 0 ]
then
echo ''
echo "================================"
echo 'All Good (google)'
echo "================================"
else
echo 'Packet Failure'
fi
sleep 3
echo "================================"
echo "stlns01 "
echo "================================"
nslookup stlns01
ping -c 3 stlns01
function CheckPing {
ping -c 3 stlns01 | grep packet | cut -f1 -d'%' | cut -f6 -d' '
}
Ping=$(CheckPing)
if [ $Ping -eq 0 ]
then
echo ''
echo "================================"
echo 'All Good (stlns01)'
echo "================================"
else
echo 'Packet Failure'
fi
sleep 2
echo "================================"
echo "vnode-39-116 "
echo "================================"
nslookup vnode-39-116
ping -c 3 vnode-39-116
function CheckPing {
ping -c 3 vnode-39-116 | grep packet | cut -f1 -d'%' | cut -f6 -d' '
}
Ping=$(CheckPing)
if [ $Ping -eq 0 ]
then
echo ''
echo "================================"
echo 'All Good (vnode-39-116)'
echo "================================"
else
echo 'Packet Failure'
fi
sleep 2
echo "================================"
echo "vnode-39-118 "
echo "================================"
nslookup vnode-39-118
ping -c 3 vnode-39-118
function CheckPing {
ping -c 3 vnode-39-118 | grep packet | cut -f1 -d'%' | cut -f6 -d' '
}
Ping=$(CheckPing)
if [ $Ping -eq 0 ]
then
echo ''
echo "================================"
echo 'All Good (vnode-39-118)'
echo "================================"
else
echo 'Packet Failure'
fi
sleep 2
echo "================================"
echo "rsrac1-scan "
echo "================================"
nslookup rsrac1-scan
ping -c 3 rsrac1-scan
function CheckPing {
ping -c 3 rsrac1-scan | grep packet | cut -f1 -d'%' | cut -f6 -d' '
}
Ping=$(CheckPing)
if [ $Ping -eq 0 ]
then
echo ''
echo "================================"
echo 'All Good (rsrac1-scan)'
echo "================================"
else
echo 'Packet Failure'
fi
sleep 2
[root@vnode-39-116 ~]#
File /etc/nsswitch.conf
[root@vnode-39-116 ~]# cat /etc/nsswitch.conf
#
# /etc/nsswitch.conf
#
# An example Name Service Switch config file. This file should be
# sorted with the most-used services at the beginning.
#
# The entry '[NOTFOUND=return]' means that the search for an
# entry should stop if the search in the previous entry turned
# up nothing. Note that if the search failed due to some other reason
# (like no NIS server responding) then the search continues with the
# next entry.
#
# Valid entries include:
#
# nisplus Use NIS+ (NIS version 3)
# nis Use NIS (NIS version 2), also called YP
# dns Use DNS (Domain Name Service)
# files Use the local files
# db Use the local database (.db) files
# compat Use NIS on compat mode
# hesiod Use Hesiod for user lookups
# [NOTFOUND=return] Stop searching if not found so far
#
# To use db, put the "db" in front of "files" for entries you want to be
# looked up first in the databases
#
# Example:
#passwd: db files nisplus nis
#shadow: db files nisplus nis
#group: db files nisplus nis
passwd: files
shadow: files
group: files
#hosts: db files nisplus nis dns
hosts: files dns nis
# Example - obey only what nisplus tells us...
#services: nisplus [NOTFOUND=return] files
#networks: nisplus [NOTFOUND=return] files
#protocols: nisplus [NOTFOUND=return] files
#rpc: nisplus [NOTFOUND=return] files
#ethers: nisplus [NOTFOUND=return] files
#netmasks: nisplus [NOTFOUND=return] files
bootparams: nisplus [NOTFOUND=return] files
ethers: files
netmasks: files
networks: files
protocols: files
rpc: files
services: files
netgroup: nisplus
publickey: nisplus
automount: files nisplus
aliases: files nisplus
[root@vnode-39-116 ~]#
The GNS Cluster Pre- and Post-Verification Commands
./runcluvfy.sh comp gns -precrsinst -domain gns1.robinsystems.com -vip 10.207.39.3 -verbose -n vnode-39-116
./runcluvfy.sh comp gns -postcrsinst -verbose
Migrate ASM Flex Cluster to Robin Storage
Prepare Shared Filesystem for Robin RAC
Create the required Robin volume on the Robin host, mount the required Robin volume on the Robin host, and verify using gdisk or fdisk the correct vblock# of the volume, as shown below.
[root@centos-72k ~]# stormgr volume create cssvote1 1G -b 512 -m 'ssd' -M -P 1
{"compression": 1, "parent_snapshotname": null, "parent_volumename": null, "name": "cssvote1", "partitions": 1, "qgroupid": 1, "block_size": 512, "resource_poolid": 1, "multinode_mounting": true, "size": 1073741824, "media": 83, "protection": 0, "replication": 1, "parent_appname": null, "vtype": 0}
Volume cssvote1 created successfully.
[root@centos-72k ~]# stormgr volume mount cssvote1 default centos-72k
{"method":"mount","mounts_list":[{"appid":1,"zoneid":1472756430,"snapshotid":1,"nodeid":1,"volumeid":6,"mntpath":"/dev/vblock5"}]}
[root@centos-72k ~]# ls -l /dev/vblock*
brw-rw----. 1 root disk 252, 0 Sep 1 13:45 /dev/vblock0
brw-rw----. 1 root disk 252, 1 Sep 1 13:45 /dev/vblock1
brw-rw----. 1 root disk 252, 2 Sep 1 13:45 /dev/vblock2
brw-rw----. 1 root disk 252, 3 Sep 1 13:45 /dev/vblock3
brw-rw----. 1 root disk 252, 4 Sep 1 13:45 /dev/vblock4
brw-rw----. 1 root disk 252, 5 Sep 4 08:17 /dev/vblock5
[root@centos-72k ~]# gdisk -l /dev/vblock5
GPT fdisk (gdisk) version 0.8.6
Partition table scan:
MBR: not present
BSD: not present
APM: not present
GPT: not present
Creating new GPT entries.
Disk /dev/vblock5: 2097152 sectors, 1024.0 MiB
Logical sector size: 512 bytes
Disk identifier (GUID): F00D7A5C-BCF1-4D09-8C70-667720B811AD
Partition table holds up to 128 entries
First usable sector is 34, last usable sector is 2097118
Partitions will be aligned on 2048-sector boundaries
Total free space is 2097085 sectors (1024.0 MiB)
Number Start (sector) End (sector) Size Code Name
Create a Filesystem on the Shared Volume
From the Robin host, create an ext4 filesystem on the shared volume as shown below.
[root@centos-72k ~]# mkfs -t ext4 /dev/vblock5
mke2fs 1.42.9 (28-Dec-2013)
Filesystem label=
OS type: Linux
Block size=4096 (log=2)
Fragment size=4096 (log=2)
Stride=0 blocks, Stripe width=0 blocks
65536 inodes, 262144 blocks
13107 blocks (5.00%) reserved for the super user
First data block=0
Maximum filesystem blocks=268435456
8 block groups
32768 blocks per group, 32768 fragments per group
8192 inodes per group
Superblock backups stored on blocks:
32768, 98304, 163840, 229376
Allocating group tables: done
Writing inode tables: done
Creating journal (8192 blocks): done
Writing superblocks and filesystem accounting information: done
Mount the Shared Volume in the Robin RAC Containers
Mount the shared volume in the Robin RAC containers using the lxc-device command as shown below.
[root@centos-72k ~]# lxc-device --name=vnode-39-116.robinsystems.com -- add /dev/vblock5
[root@centos-72k ~]# lxc-device --name=vnode-39-118.robinsystems.com -- add /dev/vblock5
Create Mount Point for Shared Volume in the Robin RAC Containers
Create a mount point for the shared volume in the Robin RAC containers as shown below.
[root@centos-72k ~]# ssh vnode-39-116
Last login: Sat Sep 3 23:31:16 2016 from stlns01.robinsystems.com
[root@vnode-39-116 ~]# mkdir /orasharefs
[root@vnode-39-116 ~]# ssh vnode-39-118
root@vnode-39-118's password:
Last login: Sun Sep 4 08:11:31 2016 from 10.207.39.116
[root@vnode-39-118 ~]# mkdir /orasharefs
[root@vnode-39-118 ~]# cd /
[root@vnode-39-118 /]# chown oracle:oinstall orasharefs
[root@vnode-39-118 /]# exit
logout
Connection to vnode-39-118 closed.
[root@vnode-39-116 ~]# cd /
[root@vnode-39-116 /]# chown oracle:oinstall orasharefs
Verify Block in RAC Container and Mount Block
Verify the identity of the shared vblock within each container and then mount the vblock in each container as shown below.
[root@vnode-39-116 /]# ls -l /dev/vblock*
brw-rw----. 1 root disk 252, 0 Sep 1 13:45 /dev/vblock0
brw-rw----. 1 root disk 252, 1 Sep 1 13:45 /dev/vblock1
brw-rw----. 1 root disk 252, 2 Sep 1 13:45 /dev/vblock2
brw-rw----. 1 root disk 252, 3 Sep 1 13:45 /dev/vblock3
brw-rw----. 1 root disk 252, 4 Sep 1 13:45 /dev/vblock4
brw-r-----. 1 root root 252, 5 Sep 4 08:18 /dev/vblock5
[root@vnode-39-116 /]# fdisk -l /dev/vblock5
Disk /dev/vblock5: 1073 MB, 1073741824 bytes
64 heads, 32 sectors/track, 1024 cylinders
Units = cylinders of 2048 * 512 = 1048576 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000
[root@vnode-39-116 /]# mount -t ext4 /dev/vblock5 /orasharefs/
[root@vnode-39-116 /]# ssh vnode-39-118
root@vnode-39-118's password:
Last login: Sun Sep 4 08:28:43 2016 from 10.207.39.116
[root@vnode-39-118 ~]# ls -l /dev/vblock*
brw-rw----. 1 root disk 252, 0 Sep 1 13:45 /dev/vblock0
brw-rw----. 1 root disk 252, 1 Sep 1 13:45 /dev/vblock1
brw-rw----. 1 root disk 252, 2 Sep 1 13:45 /dev/vblock2
brw-rw----. 1 root disk 252, 3 Sep 1 13:45 /dev/vblock3
brw-rw----. 1 root disk 252, 4 Sep 1 13:45 /dev/vblock4
brw-r-----. 1 root root 252, 5 Sep 4 08:18 /dev/vblock5
[root@vnode-39-118 ~]# fdisk -l /dev/vblock5
Disk /dev/vblock5: 1073 MB, 1073741824 bytes
64 heads, 32 sectors/track, 1024 cylinders
Units = cylinders of 2048 * 512 = 1048576 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000
[root@vnode-39-118 ~]# mount -t ext4 /dev/vblock5 /orasharefs/
[root@vnode-39-118 ~]#
Check Status of votedisk and Relocate votedisk to Shared FS
Check status of votedisk then relocate it to the shared filesystem and recheck cluster status as shown below.
[root@vnode-39-116 orasharefs]# crsctl query css votedisk
## STATE File Universal Id File Name Disk group
-- ----- ----------------- --------- ---------
1. ONLINE 0d792826a0f14fb2bfe65432e9c1712e (/dev/mapper/asm_sysdk_00) [SYSD1]
Located 1 voting disk(s).
[root@vnode-39-116 orasharefs]# crsctl replace votedisk /orasharefs/cssvote1
Now formatting voting disk: /orasharefs/cssvote1.
CRS-4256: Updating the profile
Successful addition of voting disk 9ed9c126ded54f77bf27e0434d51c38d.
Successful deletion of voting disk 0d792826a0f14fb2bfe65432e9c1712e.
CRS-4256: Updating the profile
CRS-4266: Voting file(s) successfully replaced
[root@vnode-39-116 orasharefs]# crsctl query css votedisk
## STATE File Universal Id File Name Disk group
-- ----- ----------------- --------- ---------
1. ONLINE 9ed9c126ded54f77bf27e0434d51c38d (/orasharefs/cssvote1) []
Located 1 voting disk(s).
[root@vnode-39-116 orasharefs]# crsctl stat res -t
--------------------------------------------------------------------------------
Name Target State Server State details
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.ASMNET1LSNR_ASM.lsnr
ONLINE ONLINE vnode-39-116 STABLE
ONLINE ONLINE vnode-39-118 STABLE
ora.LISTENER.lsnr
ONLINE ONLINE vnode-39-116 STABLE
ONLINE ONLINE vnode-39-118 STABLE
ora.SYSD1.dg
ONLINE ONLINE vnode-39-116 STABLE
ONLINE ONLINE vnode-39-118 STABLE
ora.net1.network
ONLINE ONLINE vnode-39-116 STABLE
ONLINE ONLINE vnode-39-118 STABLE
ora.ons
ONLINE ONLINE vnode-39-116 STABLE
ONLINE ONLINE vnode-39-118 STABLE
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.LISTENER_SCAN1.lsnr
1 ONLINE ONLINE vnode-39-118 STABLE
ora.LISTENER_SCAN2.lsnr
1 ONLINE ONLINE vnode-39-116 STABLE
ora.LISTENER_SCAN3.lsnr
1 ONLINE ONLINE vnode-39-116 STABLE
ora.MGMTLSNR
1 ONLINE ONLINE vnode-39-116 169.254.119.173 192.
220.39.116,STABLE
ora.asm
1 ONLINE ONLINE vnode-39-116 Started,STABLE
2 ONLINE ONLINE vnode-39-118 Started,STABLE
3 OFFLINE OFFLINE STABLE
ora.cvu
1 ONLINE ONLINE vnode-39-116 STABLE
ora.gns
1 ONLINE ONLINE vnode-39-116 STABLE
ora.gns.vip
1 ONLINE ONLINE vnode-39-116 STABLE
ora.mgmtdb
1 ONLINE ONLINE vnode-39-116 Open,STABLE
ora.oc4j
1 ONLINE ONLINE vnode-39-116 STABLE
ora.scan1.vip
1 ONLINE ONLINE vnode-39-118 STABLE
ora.scan2.vip
1 ONLINE ONLINE vnode-39-116 STABLE
ora.scan3.vip
1 ONLINE ONLINE vnode-39-116 STABLE
ora.vnode-39-116.vip
1 ONLINE ONLINE vnode-39-116 STABLE
ora.vnode-39-118.vip
1 ONLINE ONLINE vnode-39-118 STABLE
--------------------------------------------------------------------------------
[root@vnode-39-116 orasharefs]#
Move ASM Password File from ASM to Shared Filesystem
For this prototype RAC the orapwasm password file is moved out of ASM and onto a shared filesystem as shown below.
[oracle@vnode-39-116 dbs]$ asmcmd
ASMCMD> pwcopy --dbuniquename +ASM +SYSD1/orapwasm /orasharefs
copying +SYSD1/orapwasm -> /orasharefs/orapwasm
ASMCMD-9456: password file should be located on an ASM disk group
ASMCMD> exit
[oracle@vnode-39-116 dbs]$ cd /orasharefs
[oracle@vnode-39-116 orasharefs]$ ls -l
total 20540
-rw-r-----. 1 oracle oinstall 21004800 Sep 4 08:51 cssvote1
drwx------. 2 root root 16384 Sep 4 08:17 lost+found
-rw-r-----. 1 oracle oinstall 7680 Sep 4 08:50 orapwasm
-rw-r-----. 1 oracle oinstall 1536 Sep 4 08:40 spfile+ASM.ora
[oracle@vnode-39-116 orasharefs]$
Move ASM spfile from ASM to Shared Filesystem
Move the ASM spfie from ASM to shared filesystem as shown below.
[root@vnode-39-116 ~]# chown oracle:oinstall /orasharefs
[root@vnode-39-116 ~]# ssh vnode-39-118
root@vnode-39-118's password:
Last login: Sun Sep 4 08:29:48 2016 from 10.207.39.116
[root@vnode-39-118 ~]# chown oracle:oinstall /orasharefs
[root@vnode-39-118 ~]# exit
logout
Connection to vnode-39-118 closed.
[oracle@vnode-39-116 ~]$ source grid_env
[oracle@vnode-39-116 ~]$ sqlplus "/ as sysasm"
SQL*Plus: Release 12.1.0.2.0 Production on Sun Sep 4 08:35:30 2016
Copyright (c) 1982, 2014, Oracle. All rights reserved.
Connected to:
Oracle Database 12c Enterprise Edition Release 12.1.0.2.0 - 64bit Production
With the Real Application Clusters and Automatic Storage Management options
SQL> set linesize 200
SQL> show parameter spfile
NAME TYPE VALUE
------------------------------------ ----------- ------------------------------
spfile string +SYSD1/rsrac1/ASMPARAMETERFILE
/registry.253.921618279
SQL>
SQL> create pfile='/u00/app/12.1.0.2/grid/dbs/init+ASM1.ora' from spfile;
File created.
SQL> create spfile='/orasharefs/spfile+ASM.ora' from pfile='/u00/app/12.1.0.2/grid/dbs/init+ASM1.ora';
File created.
SQL> exit
Disconnected from Oracle Database 12c Enterprise Edition Release 12.1.0.2.0 - 64bit Production
With the Real Application Clusters and Automatic Storage Management options
[oracle@vnode-39-116 dbs]$ ls -l /orasharefs
total 20532
-rw-r-----. 1 oracle oinstall 21004800 Sep 4 08:41 cssvote1
drwx------. 2 root root 16384 Sep 4 08:17 lost+found
-rw-r-----. 1 oracle oinstall 1536 Sep 4 08:40 spfile+ASM.ora
[oracle@vnode-39-116 dbs]$ cd $ORACLE_HOME
[oracle@vnode-39-116 dbs]$ mv init+ASM1.ora original.init+ASM1.ora
[oracle@vnode-39-116 dbs]$ cat original.init+ASM1.ora
+ASM1.__oracle_base='/u00/app/oracle'#ORACLE_BASE set from in memory value
+ASM2.__oracle_base='/u00/app/oracle'#ORACLE_BASE set from in memory value
*.asm_diskstring='/dev/mapper/asm*'
*.asm_power_limit=1
*.large_pool_size=12M
*.remote_login_passwordfile='EXCLUSIVE'
[oracle@vnode-39-116 dbs]$ vi init+ASM1.ora
[oracle@vnode-39-116 dbs]$ cat init+ASM1.ora
*.spfile='/orasharefs/spfile+ASM.ora'
[oracle@vnode-39-116 dbs]$ ssh vnode-39-118
Last login: Sat Sep 3 20:30:15 2016 from vnode-39-116.robinsystems.com
[oracle@vnode-39-118 ~]$ source grid_env
[oracle@vnode-39-118 ~]$ cd $ORACLE_HOME/dbs
[oracle@vnode-39-118 dbs]$ pwd
/u00/app/12.1.0.2/grid/dbs
[oracle@vnode-39-118 dbs]$ vi init+ASM2.ora
[oracle@vnode-39-118 dbs]$ cat init+ASM1.ora
*.spfile='/orasharefs/spfile+ASM.ora'
[oracle@vnode-39-118 dbs]$ ls -l /orasharefs
total 20532
-rw-r-----. 1 oracle oinstall 21004800 Sep 4 08:44 cssvote1
drwx------. 2 root root 16384 Sep 4 08:17 lost+found
-rw-r-----. 1 oracle oinstall 1536 Sep 4 08:40 spfile+ASM.ora
[oracle@vnode-39-118 dbs]$
Create Robin Shared Volume for SYSD1 ASM Diskgroup
Create the Robin shared volume for the ASM SYSD1 diskgroup and attach it to the Robin RAC containers as shown below.
[root@centos-72k ~]# robin host list
Zone Id | Hostname | Status | State | Pool | Roles | Cores | Memory | HDD | SSD | Instances | Tags
-----------+-----------------------------+--------+-------+---------+-------+-------+----------+---------+-----------+-----------+------
1472756430 | centos-72k.robinsystems.com | ONLINE | READY | default | C,S | 4 | 31.26 GB | 75.0 GB | 1000.0 GB | 2 | {}
[root@centos-72k ~]# stormgr volume create sysd1 10G -b 512 -m 'ssd' -M -P 1
{"parent_snapshotname": null, "vtype": 0, "parent_volumename": null, "replication": 1, "parent_appname": null, "multinode_mounting": true, "partitions": 1, "qgroupid": 1, "resource_poolid": 1, "block_size": 512, "media": 83, "compression": 1, "name": "sysd1", "protection": 0, "size": 10737418240}
Volume sysd1 created successfully.
[root@centos-72k ~]# stormgr volume mount sysd1 default centos-72k
{"method":"mount","mounts_list":[{"appid":1,"zoneid":1472756430,"snapshotid":1,"nodeid":1,"volumeid":7,"mntpath":"/dev/vblock7"}]}
[root@centos-72k ~]# ls -l /dev/vblock7
brw-rw----. 1 root disk 252, 7 Sep 4 09:29 /dev/vblock7
[root@centos-72k ~]# fdisk -l /dev/vblock7
Disk /dev/vblock7: 10.7 GB, 10737418240 bytes, 20971520 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
[root@centos-72k ~]# lxc-device --name=vnode-39-116.robinsystems.com -- add /dev/vblock7
[root@centos-72k ~]# lxc-device --name=vnode-39-118.robinsystems.com -- add /dev/vblock7
[root@centos-72k ~]#
Set Owner and Mode on Robin Shared Storage for ASM
Set the owner and mode for Robin shared storage to the same values currently in use by the SCST Linux SAN LUN currently in use by the +SYSD1 ASM diskgroup on the Robin host, and then verify that the owner and mode are correct in both containers as shown below.
[root@centos-72k ~]# ls -l /dev/vblock7
brw-r-----. 1 root root 252, 7 Sep 4 09:30 /dev/vblock7
[root@centos-72k ~]# ls -l /dev/mapper
total 0
lrwxrwxrwx. 1 root root 7 Sep 4 09:32 asm_sysdk_00 -> ../dm-4
lrwxrwxrwx. 1 root root 7 Sep 1 12:42 centos-home -> ../dm-2
lrwxrwxrwx. 1 root root 7 Sep 1 12:42 centos-root -> ../dm-0
lrwxrwxrwx. 1 root root 7 Sep 1 12:42 centos-swap -> ../dm-1
crw-------. 1 root root 10, 236 Sep 1 12:42 control
lrwxrwxrwx. 1 root root 7 Sep 1 12:00 docker-253:0-136056805-pool -> ../dm-3
[root@centos-72k ~]# ls -l /dev/dm-4
brw-rw----. 1 oracle asmadmin 253, 4 Sep 4 09:33 /dev/dm-4
[root@centos-72k ~]# chown oracle:asmadmin /dev/vblock7
[root@centos-72k ~]# chmod 0660 /dev/vblock7
[root@centos-72k ~]# ssh vnode-39-116
Last login: Sun Sep 4 08:18:47 2016 from centos-72k.robinsystems.com
[root@vnode-39-116 ~]# ls -l /dev/vblock7
[root@centos-72k ~]# ssh vnode-39-116
Last login: Sun Sep 4 09:34:10 2016 from centos-72k.robinsystems.com
[root@vnode-39-116 ~]# ls -l /dev/vblock7
brw-rw----. 1 oracle asmadmin 252, 7 Sep 4 09:30 /dev/vblock7
[root@vnode-39-116 ~]# ssh vnode-39-118
root@vnode-39-118's password:
Last login: Sun Sep 4 09:34:23 2016 from 10.207.39.116
[root@vnode-39-118 ~]# ls -l /dev/vblock7
brw-rw----. 1 oracle asmadmin 252, 7 Sep 4 09:30 /dev/vblock7
[root@vnode-39-118 ~]#
Add Mount Directive to /etc/fstab File on Robin RAC Containers
Add the mount directive for the shared Oracle filesystem on all Robin RAC containers as shown below.
[root@vnode-39-116 ~]# vi /etc/fstab
[root@vnode-39-116 ~]# ssh vnode-39-118
root@vnode-39-118's password:
Last login: Sun Sep 4 10:56:05 2016 from 10.207.39.116
[root@vnode-39-118 ~]# vi /etc/fstab
[root@vnode-39-118 ~]# cat /etc/fstab
/dev/root / rootfs defaults 0 0
none /dev/shm tmpfs nosuid,nodev 0 0
/dev/vblock5 /orasharefs ext4 defaults 0 0
[root@vnode-39-118 ~]# exit
logout
Connection to vnode-39-118 closed.
[root@vnode-39-116 ~]# cat /etc/fstab
/dev/root / rootfs defaults 0 0
none /dev/shm tmpfs nosuid,nodev 0 0
/dev/vblock5 /orasharefs ext4 defaults 0 0
[root@vnode-39-116 ~]#
Double Check all Relocated Files and Pointers to Files
Double check all relocated files, permissions, owners, and groups of relocated files (and directories) and pointers to files as shown below.
[root@vnode-39-116 ~]# crsctl query css votedisk
## STATE File Universal Id File Name Disk group
-- ----- ----------------- --------- ---------
1. ONLINE 075ea127f3404fcbbf84efbda7a76647 (/orasharefs/cssvote1) []
Located 1 voting disk(s).
[root@vnode-39-116 ~]# ls -l /orasharefs
total 20540
-rw-r-----. 1 oracle oinstall 21004800 Sep 4 11:14 cssvote1
drwx------. 2 root root 16384 Sep 4 10:55 lost+found
-rw-r-----. 1 oracle oinstall 7680 Sep 4 11:08 orapwasm
-rw-r-----. 1 oracle oinstall 1536 Sep 4 11:04 spfile+ASM.ora
[root@vnode-39-116 ~]# cat /u00/app/12.1.0.2/grid/dbs/init+ASM1.ora
*.spfile='/orasharefs/spfile+ASM.ora'
[root@vnode-39-116 ~]# ls -l /orasharefs/spfile+ASM.ora
-rw-r-----. 1 oracle oinstall 1536 Sep 4 11:04 /orasharefs/spfile+ASM.ora
[root@vnode-39-116 ~]# ssh vnode-39-118
root@vnode-39-118's password:
Last login: Sun Sep 4 11:11:13 2016 from 10.207.39.116
[root@vnode-39-118 ~]# crsctl query css votedisk
## STATE File Universal Id File Name Disk group
-- ----- ----------------- --------- ---------
1. ONLINE 075ea127f3404fcbbf84efbda7a76647 (/orasharefs/cssvote1) []
Located 1 voting disk(s).
[root@vnode-39-118 ~]# ls -l /orasharefs
total 20540
-rw-r-----. 1 oracle oinstall 21004800 Sep 4 11:15 cssvote1
drwx------. 2 root root 16384 Sep 4 10:55 lost+found
-rw-r-----. 1 oracle oinstall 7680 Sep 4 11:08 orapwasm
-rw-r-----. 1 oracle oinstall 1536 Sep 4 11:04 spfile+ASM.ora
[root@vnode-39-118 ~]# cat /u00/app/12.1.0.2/grid/dbs/init+ASM2.ora
*.spfile='/orasharefs/spfile+ASM.ora'
[root@vnode-39-118 ~]# ls -l /orasharefs/spfile+ASM.ora
-rw-r-----. 1 oracle oinstall 1536 Sep 4 11:04 /orasharefs/spfile+ASM.ora
[root@vnode-39-118 ~]# ls -l / | grep orasharefs
drwxr-xr-x. 3 oracle oinstall 4096 Sep 4 11:08 orasharefs
[root@vnode-39-118 ~]# exit
logout
Connection to vnode-39-118 closed.
[root@vnode-39-116 ~]# ls -l / | grep orasharefs
drwxr-xr-x. 3 oracle oinstall 4096 Sep 4 11:08 orasharefs
[root@vnode-39-116 ~]#
Restart All Robin RAC Containers
Restart all Robin RAC Containers to allow the new locations for the spfile and the pwdfile to become active. This can be done using the Robin CLI or the Robin GUI.
Login to All Robin RAC Containers and Verify New Settings
Login to all Robin RAC containers and verify that the /orasharedfs was mounted on boot, that the permissions of the /orasharefs and associated files are correct, that the new location of the votedisk is in use, and that the new location of the spfile was read and used, and finally the overall health of the cluster, as shown below.
[root@vnode-39-116 ~]# ls -l /dev/vblock[57]
brw-r-----. 1 root root 252, 5 Sep 4 10:55 /dev/vblock5
brw-rw----. 1 oracle asmadmin 252, 7 Sep 4 10:58 /dev/vblock7
[root@vnode-39-116 ~]# ls -l /orasharefs/
total 20540
-rw-r-----. 1 oracle oinstall 21004800 Sep 4 11:28 cssvote1
drwx------. 2 root root 16384 Sep 4 10:55 lost+found
-rw-r-----. 1 oracle oinstall 7680 Sep 4 11:08 orapwasm
-rw-r-----. 1 oracle oinstall 1536 Sep 4 11:04 spfile+ASM.ora
[root@vnode-39-116 ~]# ssh vnode-39-118
root@vnode-39-118's password:
Last login: Sun Sep 4 11:24:11 2016 from 10.207.39.116
[root@vnode-39-118 ~]# ls -l /dev/vblock[57]
brw-r-----. 1 root root 252, 5 Sep 4 10:55 /dev/vblock5
brw-rw----. 1 oracle asmadmin 252, 7 Sep 4 10:58 /dev/vblock7
[root@vnode-39-118 ~]# ls -l /orasharefs/
total 20540
-rw-r-----. 1 oracle oinstall 21004800 Sep 4 11:29 cssvote1
drwx------. 2 root root 16384 Sep 4 10:55 lost+found
-rw-r-----. 1 oracle oinstall 7680 Sep 4 11:08 orapwasm
-rw-r-----. 1 oracle oinstall 1536 Sep 4 11:04 spfile+ASM.ora
[root@vnode-39-118 ~]# su - oracle
[oracle@vnode-39-118 ~]$ source grid_env
[oracle@vnode-39-118 ~]$ crsctl query css votedisk
## STATE File Universal Id File Name Disk group
-- ----- ----------------- --------- ---------
1. ONLINE 075ea127f3404fcbbf84efbda7a76647 (/orasharefs/cssvote1) []
Located 1 voting disk(s).
[oracle@vnode-39-116 ~]$ source grid_env
[oracle@vnode-39-116 ~]$ sqlplus "/ as sysasm"
SQL*Plus: Release 12.1.0.2.0 Production on Sun Sep 4 11:30:17 2016
Copyright (c) 1982, 2014, Oracle. All rights reserved.
Connected to:
Oracle Database 12c Enterprise Edition Release 12.1.0.2.0 - 64bit Production
With the Real Application Clusters and Automatic Storage Management options
SQL> show parameter spfile
NAME TYPE VALUE
------------------------------------ ----------- ------------------------------
spfile string /orasharefs/spfile+ASM.ora
SQL> exit
Disconnected from Oracle Database 12c Enterprise Edition Release 12.1.0.2.0 - 64bit Production
With the Real Application Clusters and Automatic Storage Management options
[oracle@vnode-39-116 ~]$ ssh vnode-39-118
Last login: Sun Sep 4 11:08:23 2016 from 10.207.39.116
[oracle@vnode-39-118 ~]$ ls -l /dev/vblock[57]
brw-r-----. 1 root root 252, 5 Sep 4 10:55 /dev/vblock5
brw-rw----. 1 oracle asmadmin 252, 7 Sep 4 10:58 /dev/vblock7
[oracle@vnode-39-118 ~]$ ls -l /orasharefs/
total 20540
-rw-r-----. 1 oracle oinstall 21004800 Sep 4 11:30 cssvote1
drwx------. 2 root root 16384 Sep 4 10:55 lost+found
-rw-r-----. 1 oracle oinstall 7680 Sep 4 11:08 orapwasm
-rw-r-----. 1 oracle oinstall 1536 Sep 4 11:04 spfile+ASM.ora
[oracle@vnode-39-118 ~]$ source grid_env
[oracle@vnode-39-118 ~]$ sqlplus "/ as sysasm"
SQL*Plus: Release 12.1.0.2.0 Production on Sun Sep 4 11:31:20 2016
Copyright (c) 1982, 2014, Oracle. All rights reserved.
Connected to:
Oracle Database 12c Enterprise Edition Release 12.1.0.2.0 - 64bit Production
With the Real Application Clusters and Automatic Storage Management options
SQL> show parameter spfile
NAME TYPE VALUE
------------------------------------ ----------- ------------------------------
spfile string /orasharefs/spfile+ASM.ora
SQL> exit
Disconnected from Oracle Database 12c Enterprise Edition Release 12.1.0.2.0 - 64bit Production
With the Real Application Clusters and Automatic Storage Management options
[oracle@vnode-39-118 ~]$ crsctl query css votedisk
## STATE File Universal Id File Name Disk group
-- ----- ----------------- --------- ---------
1. ONLINE 075ea127f3404fcbbf84efbda7a76647 (/orasharefs/cssvote1) []
Located 1 voting disk(s).
[oracle@vnode-39-118 ~]$ crsctl stat res -t
--------------------------------------------------------------------------------
Name Target State Server State details
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.ASMNET1LSNR_ASM.lsnr
ONLINE ONLINE vnode-39-116 STABLE
ONLINE ONLINE vnode-39-118 STABLE
ora.LISTENER.lsnr
ONLINE ONLINE vnode-39-116 STABLE
ONLINE ONLINE vnode-39-118 STABLE
ora.SYSD1.dg
ONLINE ONLINE vnode-39-116 STABLE
ONLINE ONLINE vnode-39-118 STABLE
ora.net1.network
ONLINE ONLINE vnode-39-116 STABLE
ONLINE ONLINE vnode-39-118 STABLE
ora.ons
ONLINE ONLINE vnode-39-116 STABLE
ONLINE ONLINE vnode-39-118 STABLE
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.LISTENER_SCAN1.lsnr
1 ONLINE ONLINE vnode-39-118 STABLE
ora.LISTENER_SCAN2.lsnr
1 ONLINE ONLINE vnode-39-116 STABLE
ora.LISTENER_SCAN3.lsnr
1 ONLINE ONLINE vnode-39-116 STABLE
ora.MGMTLSNR
1 ONLINE ONLINE vnode-39-116 169.254.119.173 192.
220.39.116,STABLE
ora.asm
1 ONLINE ONLINE vnode-39-116 Started,STABLE
2 ONLINE ONLINE vnode-39-118 Started,STABLE
3 OFFLINE OFFLINE STABLE
ora.cvu
1 ONLINE ONLINE vnode-39-116 STABLE
ora.gns
1 ONLINE ONLINE vnode-39-116 STABLE
ora.gns.vip
1 ONLINE ONLINE vnode-39-116 STABLE
ora.mgmtdb
1 ONLINE ONLINE vnode-39-116 Open,STABLE
ora.oc4j
1 ONLINE ONLINE vnode-39-116 STABLE
ora.scan1.vip
1 ONLINE ONLINE vnode-39-118 STABLE
ora.scan2.vip
1 ONLINE ONLINE vnode-39-116 STABLE
ora.scan3.vip
1 ONLINE ONLINE vnode-39-116 STABLE
ora.vnode-39-116.vip
1 ONLINE ONLINE vnode-39-116 STABLE
ora.vnode-39-118.vip
1 ONLINE ONLINE vnode-39-118 STABLE
--------------------------------------------------------------------------------
[oracle@vnode-39-118 ~]$ exit
logout
Connection to vnode-39-118 closed.
[oracle@vnode-39-116 ~]$
Rebalance ASM +SYSD1 Diskgroup to Robin Storage
Set asm_diskstring for Old and New Storage Devices
Prepare for an ASM rebalance operation to move the +SYSD1 diskgroup from non-Robin storage to Robin vblock storage as shown below.
[oracle@vnode-39-116 ~]$ sqlplus "/ as sysasm"
SQL*Plus: Release 12.1.0.2.0 Production on Sun Sep 4 11:42:23 2016
Copyright (c) 1982, 2014, Oracle. All rights reserved.
Connected to:
Oracle Database 12c Enterprise Edition Release 12.1.0.2.0 - 64bit Production
With the Real Application Clusters and Automatic Storage Management options
SQL> show parameter asm_diskstring
NAME TYPE VALUE
------------------------------------ ----------- ------------------------------
asm_diskstring string /dev/mapper/asm*
SQL>
SQL> set linesize 200
SQL> column path format a40
SQL> select name, path, label, sector_size, header_status, mount_status, mode_status from v$asm_disk;
NAME PATH LABEL SECTOR_SIZE HEADER_STATU MOUNT_S MODE_ST
------------------------------ ---------------------------------------- ------------------------------- ----------- ------------ ------- -------
SYSD1_0000 /dev/mapper/asm_sysdk_00 512 MEMBER CACHED ONLINE
SQL> alter system set asm_diskstring='/dev/mapper/asm_sysdk_00','/dev/vblock*' scope=both sid='*';
System altered.
SQL> select name, path, label, sector_size, header_status, mount_status, mode_status from v$asm_disk;
NAME PATH LABEL SECTOR_SIZE HEADER_STATU MOUNT_S MODE_ST
------------------------------ ---------------------------------------- ------------------------------- ----------- ------------ ------- -------
/dev/vblock7 512 CANDIDATE CLOSED ONLINE
SYSD1_0000 /dev/mapper/asm_sysdk_00 512 MEMBER CACHED ONLINE
SQL>
Perform the ASM Rebalance from Old to New Storage
Perform the ASM rebalance from iscsi storage to Robin vblock storage as shown below.
SQL> select name, path, label, sector_size, header_status, mount_status, mode_status from v$asm_disk;
NAME PATH LABEL SECTOR_SIZE HEADER_STATU MOUNT_S MODE_ST
------------------------------ ---------------------------------------- ------------------------------- ----------- ------------ ------- -------
/dev/vblock7 512 CANDIDATE CLOSED ONLINE
SYSD1_0000 /dev/mapper/asm_sysdk_00 512 MEMBER CACHED ONLINE
SQL> alter diskgroup SYSD1 add disk '/dev/vblock7' drop disk SYSD1_0000 rebalance power 11;
Diskgroup altered.
SQL> select * from v$asm_operation;
GROUP_NUMBER OPERA PASS STAT POWER ACTUAL SOFAR EST_WORK EST_RATE EST_MINUTES ERROR_CODE CON_ID
------------ ----- --------- ---- ---------- ---------- ---------- ---------- ---------- ----------- -------------------------------------------- ----------
1 REBAL REBALANCE RUN 11 11 72 1130 882 1 0
1 REBAL COMPACT WAIT 11 11 0 0 0 0 0
SQL> /
no rows selected
SQL>
The ASM rebalance operation is complete when no rows are returned from the query of v$asm_operation view as shown above.
Verify Overall Health of RAC Post-ASM-Rebalance
Verify the health of the RAC cluster after completing the ASM rebalance and all the other operations of this completed procedure.
[oracle@vnode-39-116 ~]$ crsctl stat res -t
--------------------------------------------------------------------------------
Name Target State Server State details
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.ASMNET1LSNR_ASM.lsnr
ONLINE ONLINE vnode-39-116 STABLE
ONLINE ONLINE vnode-39-118 STABLE
ora.LISTENER.lsnr
ONLINE ONLINE vnode-39-116 STABLE
ONLINE ONLINE vnode-39-118 STABLE
ora.SYSD1.dg
ONLINE ONLINE vnode-39-116 STABLE
ONLINE ONLINE vnode-39-118 STABLE
ora.net1.network
ONLINE ONLINE vnode-39-116 STABLE
ONLINE ONLINE vnode-39-118 STABLE
ora.ons
ONLINE ONLINE vnode-39-116 STABLE
ONLINE ONLINE vnode-39-118 STABLE
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.LISTENER_SCAN1.lsnr
1 ONLINE ONLINE vnode-39-118 STABLE
ora.LISTENER_SCAN2.lsnr
1 ONLINE ONLINE vnode-39-116 STABLE
ora.LISTENER_SCAN3.lsnr
1 ONLINE ONLINE vnode-39-116 STABLE
ora.MGMTLSNR
1 ONLINE ONLINE vnode-39-116 169.254.119.173 192.
220.39.116,STABLE
ora.asm
1 ONLINE ONLINE vnode-39-116 Started,STABLE
2 ONLINE ONLINE vnode-39-118 Started,STABLE
3 OFFLINE OFFLINE STABLE
ora.cvu
1 ONLINE ONLINE vnode-39-116 STABLE
ora.gns
1 ONLINE ONLINE vnode-39-116 STABLE
ora.gns.vip
1 ONLINE ONLINE vnode-39-116 STABLE
ora.mgmtdb
1 ONLINE ONLINE vnode-39-116 Open,STABLE
ora.oc4j
1 ONLINE ONLINE vnode-39-116 STABLE
ora.scan1.vip
1 ONLINE ONLINE vnode-39-118 STABLE
ora.scan2.vip
1 ONLINE ONLINE vnode-39-116 STABLE
ora.scan3.vip
1 ONLINE ONLINE vnode-39-116 STABLE
ora.vnode-39-116.vip
1 ONLINE ONLINE vnode-39-116 STABLE
ora.vnode-39-118.vip
1 ONLINE ONLINE vnode-39-118 STABLE
--------------------------------------------------------------------------------
[oracle@vnode-39-116 ~]$ su - root
Password:
[root@vnode-39-116 ~]# ocrcheck
Status of Oracle Cluster Registry is as follows :
Version : 4
Total space (kbytes) : 409568
Used space (kbytes) : 2672
Available space (kbytes) : 406896
ID : 360352339
Device/File Name : +SYSD1
Device/File integrity check succeeded
Device/File not configured
Device/File not configured
Device/File not configured
Device/File not configured
Cluster registry integrity check succeeded
Logical corruption check succeeded
[root@vnode-39-116 ~]# crsctl query css votedisk
## STATE File Universal Id File Name Disk group
-- ----- ----------------- --------- ---------
1. ONLINE 075ea127f3404fcbbf84efbda7a76647 (/orasharefs/cssvote1) []
Located 1 voting disk(s).
[root@vnode-39-116 ~]# ssh vnode-39-118
root@vnode-39-118's password:
Last login: Sun Sep 4 11:28:54 2016 from 10.207.39.116
[root@vnode-39-118 ~]# crsctl query css votedisk
## STATE File Universal Id File Name Disk group
-- ----- ----------------- --------- ---------
1. ONLINE 075ea127f3404fcbbf84efbda7a76647 (/orasharefs/cssvote1) []
Located 1 voting disk(s).
[root@vnode-39-118 ~]# crsctl stat res -t
--------------------------------------------------------------------------------
Name Target State Server State details
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.ASMNET1LSNR_ASM.lsnr
ONLINE ONLINE vnode-39-116 STABLE
ONLINE ONLINE vnode-39-118 STABLE
ora.LISTENER.lsnr
ONLINE ONLINE vnode-39-116 STABLE
ONLINE ONLINE vnode-39-118 STABLE
ora.SYSD1.dg
ONLINE ONLINE vnode-39-116 STABLE
ONLINE ONLINE vnode-39-118 STABLE
ora.net1.network
ONLINE ONLINE vnode-39-116 STABLE
ONLINE ONLINE vnode-39-118 STABLE
ora.ons
ONLINE ONLINE vnode-39-116 STABLE
ONLINE ONLINE vnode-39-118 STABLE
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.LISTENER_SCAN1.lsnr
1 ONLINE ONLINE vnode-39-118 STABLE
ora.LISTENER_SCAN2.lsnr
1 ONLINE ONLINE vnode-39-116 STABLE
ora.LISTENER_SCAN3.lsnr
1 ONLINE ONLINE vnode-39-116 STABLE
ora.MGMTLSNR
1 ONLINE ONLINE vnode-39-116 169.254.119.173 192.
220.39.116,STABLE
ora.asm
1 ONLINE ONLINE vnode-39-116 Started,STABLE
2 ONLINE ONLINE vnode-39-118 Started,STABLE
3 OFFLINE OFFLINE STABLE
ora.cvu
1 ONLINE ONLINE vnode-39-116 STABLE
ora.gns
1 ONLINE ONLINE vnode-39-116 STABLE
ora.gns.vip
1 ONLINE ONLINE vnode-39-116 STABLE
ora.mgmtdb
1 ONLINE ONLINE vnode-39-116 Open,STABLE
ora.oc4j
1 ONLINE ONLINE vnode-39-116 STABLE
ora.scan1.vip
1 ONLINE ONLINE vnode-39-118 STABLE
ora.scan2.vip
1 ONLINE ONLINE vnode-39-116 STABLE
ora.scan3.vip
1 ONLINE ONLINE vnode-39-116 STABLE
ora.vnode-39-116.vip
1 ONLINE ONLINE vnode-39-116 STABLE
ora.vnode-39-118.vip
1 ONLINE ONLINE vnode-39-118 STABLE
--------------------------------------------------------------------------------
[root@vnode-39-118 ~]#
Procedure to migrate from non-Robin iscsi or other storage to Robin vblock storage is complete.
Verify Robin RAC /etc/resolv.conf File
The Robin RAC uses the following format of /etc/resolv.conf file. Note that because Robin RAC nodes do not use dhcp, the values in the /etc/resolv.conf file should not be overwritten on reboot. This format ensures that the Robin RAC containers can resolve the below types of addresses and DNS lookups.
Other RAC nodes and servers on the RAC public subnet via DNS (e.g. vnode-39-116, vnode-39-118, stlns01);
The RAC Grid Naming Service (GNS) RAC SCAN names (3 of them) (e.g. rsrac1-scan);
WAN entities (e.g. google.com, yum repositories, etc.) so that yum, svn, wget and similar commands can be used.
The format of this file is shown below, as well as output from a test script (testdns.sh) that verifies all connectivity types.
[root@vnode-39-118 ~]# cat /etc/resolv.conf
options attempts: 2
options timeout: 1
options rotate
search robinsystems.com gns1.robinsystems.com
nameserver 10.207.39.3
nameserver 10.207.39.1
[root@vnode-39-118 ~]# ./testdns.sh
================================
================================
;; Got recursion not available from 10.207.39.3, trying next server
Server: 10.207.39.1
Address: 10.207.39.1#53
Non-authoritative answer:
Name: google.com
Address: 216.58.216.206
PING google.com (216.58.216.206) 56(84) bytes of data.
64 bytes from 216.58.216.206: icmp_seq=1 ttl=53 time=21.4 ms
64 bytes from 216.58.216.206: icmp_seq=2 ttl=53 time=20.0 ms
64 bytes from 216.58.216.206: icmp_seq=3 ttl=53 time=29.1 ms
--- google.com ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2032ms
rtt min/avg/max/mdev = 20.014/23.537/29.149/4.014 ms
================================
All Good (google)
================================
================================
stlns01
================================
;; Got recursion not available from 10.207.39.3, trying next server
Server: 10.207.39.1
Address: 10.207.39.1#53
Name: stlns01.robinsystems.com
Address: 10.207.39.1
PING stlns01.robinsystems.com (10.207.39.1) 56(84) bytes of data.
64 bytes from 10.207.39.1: icmp_seq=1 ttl=64 time=0.398 ms
64 bytes from 10.207.39.1: icmp_seq=2 ttl=64 time=0.400 ms
64 bytes from 10.207.39.1: icmp_seq=3 ttl=64 time=0.425 ms
--- stlns01.robinsystems.com ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2002ms
rtt min/avg/max/mdev = 0.398/0.407/0.425/0.026 ms
================================
All Good (stlns01)
================================
================================
vnode-39-116
================================
;; Got recursion not available from 10.207.39.3, trying next server
Server: 10.207.39.1
Address: 10.207.39.1#53
Name: vnode-39-116.robinsystems.com
Address: 10.207.39.116
PING vnode-39-116.robinsystems.com (10.207.39.116) 56(84) bytes of data.
64 bytes from 10.207.39.116: icmp_seq=1 ttl=64 time=0.045 ms
64 bytes from 10.207.39.116: icmp_seq=2 ttl=64 time=0.104 ms
64 bytes from 10.207.39.116: icmp_seq=3 ttl=64 time=0.100 ms
--- vnode-39-116.robinsystems.com ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2001ms
rtt min/avg/max/mdev = 0.045/0.083/0.104/0.026 ms
================================
All Good (vnode-39-116)
================================
================================
vnode-39-118
================================
;; Got recursion not available from 10.207.39.3, trying next server
Server: 10.207.39.1
Address: 10.207.39.1#53
Name: vnode-39-118.robinsystems.com
Address: 10.207.39.118
PING vnode-39-118.robinsystems.com (10.207.39.118) 56(84) bytes of data.
64 bytes from 10.207.39.118: icmp_seq=1 ttl=64 time=0.026 ms
64 bytes from 10.207.39.118: icmp_seq=2 ttl=64 time=0.071 ms
64 bytes from 10.207.39.118: icmp_seq=3 ttl=64 time=0.071 ms
--- vnode-39-118.robinsystems.com ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2002ms
rtt min/avg/max/mdev = 0.026/0.056/0.071/0.021 ms
================================
All Good (vnode-39-118)
================================
================================
rsrac1-scan
================================
;; Got recursion not available from 10.207.39.3, trying next server
Server: 10.207.39.3
Address: 10.207.39.3#53
Name: rsrac1-scan.gns1.robinsystems.com
Address: 10.207.39.47
Name: rsrac1-scan.gns1.robinsystems.com
Address: 10.207.39.48
Name: rsrac1-scan.gns1.robinsystems.com
Address: 10.207.39.49
PING rsrac1-scan.gns1.robinsystems.com (10.207.39.47) 56(84) bytes of data.
64 bytes from 10.207.39.47: icmp_seq=1 ttl=64 time=0.029 ms
64 bytes from 10.207.39.47: icmp_seq=2 ttl=64 time=0.063 ms
64 bytes from 10.207.39.47: icmp_seq=3 ttl=64 time=0.078 ms
--- rsrac1-scan.gns1.robinsystems.com ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2000ms
rtt min/avg/max/mdev = 0.029/0.056/0.078/0.022 ms
================================
All Good (rsrac1-scan)
================================
[root@vnode-39-118 ~]#
Create Robin Volume for ASM +DATA1 Database Diskgroup
Create the required Robin volume, attach it to the Robin RAC containers, and use the asmca tool (or command line) to configure an ASM diskgroup for the database data as shown below.
[root@centos-72k ~]# robin login robin
Password:
User robin is logged in
[root@centos-72k ~]# robin host list
Zone Id | Hostname | Status | State | Pool | Roles | Cores | Memory | HDD | SSD | Instances | Tags
-----------+-----------------------------+--------+-------+---------+-------+-------+----------+---------+-----------+-----------+------
1472756430 | centos-72k.robinsystems.com | ONLINE | READY | default | C,S | 4 | 31.26 GB | 75.0 GB | 1000.0 GB | 2 | {}
[root@centos-72k ~]# robin instance list
ID | ResPool | App | Container | Hostname | IP Address | State | Host | Engine | Cores | Mem | Ctime
---+---------+----------+------------------------+--------------+---------------+---------+------------+--------+-------+--------+---------------------
2 | default | sjcora02 | sjcora02_sjcora02-db01 | vnode-39-116 | 10.207.39.116 | STARTED | centos-72k | lxc | 2 | 4.0 GB | 2016-09-01 12:35:08
1 | default | sjcora01 | sjcora01_sjcora01-db01 | vnode-39-118 | 10.207.39.118 | STARTED | centos-72k | lxc | 2 | 4.0 GB | 2016-09-01 12:31:10
[root@centos-72k ~]# stormgr volume create data1 40G -b 512 -m 'ssd' -M -P 1
{"parent_volumename": null, "parent_appname": null, "qgroupid": 1, "partitions": 1, "compression": 1, "name": "data1", "vtype": 0, "multinode_mounting": true, "replication": 1, "parent_snapshotname": null, "block_size": 512, "size": 42949672960, "resource_poolid": 1, "media": 83, "protection": 0}
Volume data1 created successfully.
[root@centos-72k ~]# stormgr volume mount data1 default centos-72k
{"method":"mount","mounts_list":[{"appid":1,"zoneid":1472756430,"snapshotid":1,"nodeid":1,"volumeid":8,"mntpath":"/dev/vblock9"}]}
[root@centos-72k ~]# ls -l /dev/vblock9
brw-rw----. 1 root disk 252, 9 Sep 4 12:36 /dev/vblock9
[root@centos-72k ~]# fdisk -l /dev/vblock9
Disk /dev/vblock9: 42.9 GB, 42949672960 bytes, 83886080 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
[root@centos-72k ~]# lxc-device --name=vnode-39-116.robinsystems.com -- add /dev/vblock9
[root@centos-72k ~]# lxc-device --name=vnode-39-118.robinsystems.com -- add /dev/vblock9
[root@centos-72k ~]# ls -l /dev/vblock7
brw-rw----. 1 oracle asmadmin 252, 7 Sep 4 12:37 /dev/vblock7
[root@centos-72k ~]# chown oracle:asmadmin /dev/vblock9
[root@centos-72k ~]# chmod 0660 /dev/vblock9
[oracle@vnode-39-116 ~]$ ssh vnode-39-118
Last login: Sun Sep 4 11:30:38 2016 from 10.207.39.116
[oracle@vnode-39-118 ~]$ ls -l /dev/vblock9
brw-rw----. 1 oracle asmadmin 252, 9 Sep 4 12:37 /dev/vblock9
[oracle@vnode-39-118 ~]$ fdisk -l /dev/vblock9
Disk /dev/vblock9: 42.9 GB, 42949672960 bytes
64 heads, 32 sectors/track, 40960 cylinders
Units = cylinders of 2048 * 512 = 1048576 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000
[oracle@vnode-39-118 ~]$ exit
logout
Connection to vnode-39-118 closed.
[oracle@vnode-39-116 ~]$ fdisk -l /dev/vblock9
Disk /dev/vblock9: 42.9 GB, 42949672960 bytes
64 heads, 32 sectors/track, 40960 cylinders
Units = cylinders of 2048 * 512 = 1048576 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000
[oracle@vnode-39-116 ~]$ asmca
Create +DATA1 ASM Diskgroup for Oracle RAC Database
The example below shows creation of the +DATA1 ASM diskgroup using the GUI asmca (ASM Configuration Assistant).
Create the Oracle RAC Database
The example below shows creation of the Oracle RAC database using the Oracle GUI "runInstaller" executable. The database can also be created manually using scripts if preferred without use of the GUI. This example uses the GUI installer as shown below.
Prepare Environment for Database Install
Because the linux user "oracle" owns both the GI and the DB it is necessary to set the .bashrc to set the correct $ORACLE_HOME by default before running the Oracle runInstaller so that the runInstaller will pick up the correct value for the $ORACLE_HOME for the database instead of for the Oracle GI. This is prepared as shown below.
oracle@stlns01:~/Downloads$ ssh oracle@vnode-39-116
oracle@vnode-39-116's password:
Last login: Sun Sep 4 12:45:37 2016 from stlns01.robinsystems.com
[oracle@vnode-39-116 ~]$ vi .bashrc
[oracle@vnode-39-116 ~]$ ssh vnode-39-118
Last login: Sun Sep 4 12:39:25 2016 from 10.207.39.116
[oracle@vnode-39-118 ~]$ vi .bashrc
[oracle@vnode-39-118 ~]$ cat .bashrc
# .bashrc
# Source global definitions
if [ -f /etc/bashrc ]; then
. /etc/bashrc
fi
# User specific aliases and functions
export TMP=/tmp
export TMPDIR=$TMP
export ORACLE_BASE=/u00/app/oracle
export GRID_HOME=/u00/app/12.1.0.2/grid
export DB_HOME=$ORACLE_BASE/product/12.1.0.2/db_1
export ORACLE_HOME=$DB_HOME <-- Ensure this is set to the "$DB_HOME" on all RAC containers
export ORACLE_TERM=xterm
export BASE_PATH=/usr/sbin:$PATH
export PATH=$ORACLE_HOME/bin:$BASE_PATH
export LD_LIBRARY_PATH=$ORACLE_HOME/lib:/lib:/usr/lib
export CLASSPATH=$ORACLE_HOME/JRE:$ORACLE_HOME/jlib:$ORACLE_HOME/rdbms/jlib
alias grid_env='. /home/oracle/grid_env'
alias db_env='. /home/oracle/db_env'
[oracle@vnode-39-118 ~]$