KVM OVS OEL 6.5

This page shows how to build and Oracle separation environment install in an Oracle Enterprise Linux (OEL) 6.5 KVM guest. An Oracle separation environment here means that a linux user named "grid" owns the Oracle ASM instance and Oracle networking installed software binaries, and a linux user named "oracle" owns the Oracle database installed software binaries, and that the overall install is a single-instance, non-RAC installation.

THIS PAGE IS STILL UNDER DEVELOPMENT.

Install Oracle Enterprise Linux 6.5 as KVM Guest

First install KVM and OpenvSwitch on your KVM host OS according to the instructions here. Note that if you have followed this wiki to setup the KVM and OpenvSwitch environment on an Ubuntu Server 14.04.1 KVM host, it will be possible to ssh into the new KVM guest as soon as it reboots after OS install is completed.

Use qemu-img to create the storage for the KVM guest as shown below.

root@vmem1:~# qemu-img create -f raw /var/lib/libvirt/images/oracle631-0.img 20G

Use the following kvm guest xml file to create the KVM guest. The bolded sections are the ones that will need modification according to specific settings. The "uuid" can be obtained by installing the "uuid" package and running it to generate a new UUID for the new KVM guest.

To use the UUID package to generate a new uuid for the KVM guest xml use the command as shown below.

gstanden@vmem1:~$ uuid

ea2e0f12-20c4-11e4-bfde-dbe8fcd80559

gstanden@vmem1:~$

Then paste this newly-generated uuid into the XML file listed below for a new UUID for the new KVM guest to be created. Other fields to be changed before creation are shown in bold and also enumerated below.

<name> (choose arbitrary name)
<uuid><uuid/> (gets populated at created time automatically)
<source file> (for install dvd/cdrom)
<source file> (for install img virtual disk file created in step above using qemu-img)

Below is the XML with the sections (bolded) that need to be edited before KVM guest creation.

gstanden@vmem1:~$ cat oracle651.xml

<name>oracle651</name>

<partition>/machine</partition>

</resource>

<os>

</os>

<acpi/>

<apic/>

<pae/>

</features>

<on_poweroff>destroy</on_poweroff>

<on_reboot>restart</on_reboot>

<on_crash>restart</on_crash>

<emulator>/usr/bin/kvm-spice</emulator>

</disk>

</disk>

</controller>

</controller>

</interface>

</serial>

</console>

</graphics>

</sound>

<video>

</video>

</memballoon>

</devices>

<qemu:commandline>

<qemu:arg value='-set'/>

<qemu:arg value='device.virtio-disk0.scsi=off'/>

<qemu:arg value='-set'/>

<qemu:arg value='device.virtio-disk0.config-wce=off'/>

<qemu:arg value='-set'/>

<qemu:arg value='device.virtio-disk0.x-data-plane=on'/>

</qemu:commandline>

</domain>

Create the KVM guest from the command line using the following as shown below.

gstanden@vmem1:~$ virsh --connect qemu:///system create oracle631.xml

Once this command has been launched successfully, a message will display showing the KVM guest has been successfully created.

Go into the virt-manager and open the console to the KVM guest being created and install the OS in the normal procedure for install OEL 6. If more information is needed there are excellent guides on installing OEL 6 on the internet. One of the best for installing virtualized OEL6 for Oracle work is the guide by Oracle Ace Director Tim Hall at the Oracle-Base site. In particular, use the section on installing the guest OS.

Configure KVM Guest for Terminal Console Output

Once the OEL 6 OS is installed, configure the KVM guest for console access as shown below and as described in this reference. This can be done most easily from a terminal session rather than the virt-install GUI session. If the wiki on how to setup KVM and OpenvSwitch referenced at the beginning of this wiki was followed, the new OEL6 KVM guest will already have a dhcp ip address assigned. Login to the KVM guest using the virt-install GUI and determine the IP, and bring up the eth0 interface if necessary. In this case, the dhcp ip address of the new guest is as shown below in bold.

Enable the network to start automatically by going into the virt-install GUI console of the KVM guest and selecting:

system --> preferences --> network connections --> (highlight "System eth0") --> (click on "Edit") --> (check "Connect Automatically") --> (click "Apply").

The system will prompt for the root password, enter it and save the change. Now eth0 will start automatically on reboot.

Restart the server using the virt-install console GUI for the KVM guest.

[gstanden@oracle631 ~]$ ifconfig

eth0 Link encap:Ethernet HWaddr 52:54:00:48:53:03

inet addr:10.207.39.52 Bcast:10.207.39.255 Mask:255.255.255.0

inet6 addr: fe80::5054:ff:fe48:5303/64 Scope:Link

UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1

RX packets:390 errors:0 dropped:0 overruns:0 frame:0

TX packets:378 errors:0 dropped:0 overruns:0 carrier:0

collisions:0 txqueuelen:1000

RX bytes:44055 (43.0 KiB) TX bytes:42843 (41.8 KiB)

lo Link encap:Local Loopback

inet addr:127.0.0.1 Mask:255.0.0.0

inet6 addr: ::1/128 Scope:Host

UP LOOPBACK RUNNING MTU:65536 Metric:1

RX packets:88 errors:0 dropped:0 overruns:0 frame:0

TX packets:88 errors:0 dropped:0 overruns:0 carrier:0

collisions:0 txqueuelen:0

RX bytes:7112 (6.9 KiB) TX bytes:7112 (6.9 KiB)

[gstanden@oracle631 ~]$

Once connected to the KVM guest edit the /boot/grub/grub.conf file as root as shown below. Note that in additon to adding the "console=ttyS0" at the end of the first kernel entry line also remove the "rhgb quiet" entries at the end of the kernel line so that verbose output is sent to console. The overall action is to replace "rhgb quiet" with "console=ttyS0" for the first kernel listed, "Oracle Linux Server Unbreakable Enterprise Kernel (3.8.13-16.2.1.el6uek.x86_64)". The finished edited file should look as shown below.

[gstanden@oracle631 ~]$ su - root

Password:

[root@oracle631 ~]# vi /boot/grub/grub.conf

[root@oracle631 ~]# cat /boot/grub/grub.conf

# grub.conf generated by anaconda

# Note that you do not have to rerun grub after making changes to this file

# NOTICE: You have a /boot partition. This means that

# all kernel and initrd paths are relative to /boot/, eg.

# root (hd0,0)

# kernel /vmlinuz-version ro root=/dev/mapper/vg_oracle631-lv_root

# initrd /initrd-[generic-]version.img

#boot=/dev/vda

default=0

timeout=5

splashimage=(hd0,0)/grub/splash.xpm.gz

hiddenmenu

title Oracle Linux Server Unbreakable Enterprise Kernel (3.8.13-16.2.1.el6uek.x86_64)

root (hd0,0)

kernel /vmlinuz-3.8.13-16.2.1.el6uek.x86_64 ro root=/dev/mapper/vg_oracle631-lv_root rd_NO_LUKS rd_LVM_LV=vg_oracle631/lv_root LANG=en_US.UTF-8 rd_LVM_LV=vg_oracle631/lv_swap rd_NO_MD SYSFONT=latarcyrheb-sun16 KEYBOARDTYPE=pc KEYTABLE=us rd_NO_DM console=ttyS0

initrd /initramfs-3.8.13-16.2.1.el6uek.x86_64.img

title Oracle Linux Server Red Hat Compatible Kernel (2.6.32-431.el6.x86_64)

root (hd0,0)

kernel /vmlinuz-2.6.32-431.el6.x86_64 ro root=/dev/mapper/vg_oracle631-lv_root rd_NO_LUKS rd_LVM_LV=vg_oracle631/lv_root LANG=en_US.UTF-8 rd_LVM_LV=vg_oracle631/lv_swap rd_NO_MD SYSFONT=latarcyrheb-sun16 crashkernel=auto KEYBOARDTYPE=pc KEYTABLE=us rd_NO_DM rhgb quiet

initrd /initramfs-2.6.32-431.el6.x86_64.img

[root@oracle631 ~]#

Now restart the KVM guest for the changes to take effect and connect to the console after starting the guest using the following command to verify console access is configured. The eth0 interface should start automatically on reboot.

The bolded below shows how to connect to the console of the KVM guest using virsh.

gstanden@vmem1:~$ virsh -c qemu:///system list

Id Name State

----------------------------------------------------

gstanden@vmem1:~$ virsh start oracle631

Domain oracle631 started

gstanden@vmem1:~$ virsh console oracle631

Connected to domain oracle631

Escape character is ^]

Initializing cgroup subsys cpuset

Initializing cgroup subsys cpu

Linux version 3.8.13-16.2.1.el6uek.x86_64 (mockbuild@ca-build44.us.oracle.com) (gcc version 4.4.7 20120313 (Red Hat 4.4.7-3) (GCC) ) #1 SMP Thu Nov 7 17:01:44 PST 2013

Command line: ro root=/dev/mapper/vg_oracle631-lv_root rd_NO_LUKS rd_LVM_LV=vg_oracle631/lv_root LANG=en_US.UTF-8 rd_LVM_LV=vg_oracle631/lv_swap rd_NO_MD SYSFONT=latarcyrheb-sun16 KEYBOARDTYPE=pc KEYTABLE=us rd_NO_DM console=ttyS0

e820: BIOS-provided physical RAM map:

BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable

Starting crond: [ OK ]

Starting atd: [ OK ]

Starting certmonger: [ OK ]

Oracle Linux Server release 6.5

Kernel 3.8.13-16.2.1.el6uek.x86_64 on an x86_64

oracle631.vmem.org login:

The above output from console confirms that console is now configured correctly to output to terminal.

Install the create-users.sh script to the /root/oracle-build-scripts directory as shown below.

gstanden@vmem1:~$ ssh root@10.207.39.52

root@10.207.39.52's password:

Last login: Sun Aug 10 15:24:17 2014 from 10.207.39.1

[root@oracle631 ~]# mkdir oracle-build-scripts

[root@oracle631 ~]# cd oracle-build-scripts/

[root@oracle631 oracle-build-scripts]# ls -lrt

total 0

[root@oracle631 oracle-build-scripts]# vi create-users.sh

Paste the following into the create-users.sh script as shown below.

[root@oracle631 oracle-build-scripts]# cat create-users.sh

groupadd -g 1000 oinstall

groupadd -g 1100 asmadmin

groupadd -g 1200 dba

groupadd -g 1300 asmdba

groupadd -g 1201 oper

groupadd -g 1301 asmoper

useradd -u 1100 -g oinstall -G asmadmin,asmdba,asmoper grid

useradd -u 1101 -g oinstall -G dba,asmdba,oper oracle

mkdir -p /u00/app/12.1.0/grid

mkdir -p /u00/app/grid

chown -R grid:oinstall /u00

mkdir -p /u00/app/oracle

chown oracle:oinstall /u00/app/oracle

chmod -R 775 /u00

[root@oracle631 oracle-build-scripts]#

Next it will be necessary to add an addition virtual disk to the KVM guest of size 20Gb which will become the /u00 file system inside the KVM guest. This can be done with the virt-install GUI, or it can be done using qemu-img to create the image file and then edit the XML to add the LUN to the KVM guest. Below are steps to do this as root using qemu-img.

root@vmem1:~# qemu-img create -f raw /var/lib/libvirt/images/oracle631-1.u00.img 20G

Formatting '/var/lib/libvirt/images/oracle631-1.u00.img', fmt=raw size=21474836480

root@vmem1:~#

Add the storage to the KVM guest using the virt-install GUI. The revised XML will be similar to that shown below. The boldeds section shows the newly added storage. Note that at the end of the xml file it is also necessary to add the new disk parameter for high-speed IOPS and bandwidth in that bolded section as well. Note that the disks indexes start with "0" (zero) and increment upward, so the new disk has index "1" (one).

gstanden@vmem1:~$ virsh edit oracle631

gstanden@vmem1:~$ virsh dumpxml oracle631

<name>oracle631</name>

<partition>/machine</partition>

</resource>

<os>

</os>

<acpi/>

<apic/>

<pae/>

</features>

<on_poweroff>destroy</on_poweroff>

<on_reboot>restart</on_reboot>

<on_crash>restart</on_crash>

<emulator>/usr/bin/kvm-spice</emulator>

</disk>

<disk type='file' device='disk'>

<driver name='qemu' type='raw' cache='none' io='native'/>

<source file='/var/lib/libvirt/images/oracle631-1.u00.img'/>

<target dev='vdb' bus='virtio'/>

<address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/>

</disk>

</disk>

</controller>

</controller>

</virtualport>

</interface>

</serial>

</console>

</graphics>

</sound>

<video>

</video>

</memballoon>

</devices>

<qemu:commandline>

<qemu:arg value='-set'/>

<qemu:arg value='device.virtio-disk0.scsi=off'/>

<qemu:arg value='-set'/>

<qemu:arg value='device.virtio-disk0.config-wce=off'/>

<qemu:arg value='-set'/>

<qemu:arg value='device.virtio-disk0.x-data-plane=on'/>

<qemu:arg value='-set'/>

<qemu:arg value='device.virtio-disk1.scsi=off'/>

<qemu:arg value='-set'/>

<qemu:arg value='device.virtio-disk1.config-wce=off'/>

<qemu:arg value='-set'/>

<qemu:arg value='device.virtio-disk1.x-data-plane=on'/> </qemu:commandline>

</domain>

Boot up the KVM guest and display properties of the new virtual disk /dev/vdb.

gstanden@vmem1:~$ ssh 10.207.39.52

gstanden@10.207.39.52's password:

Last login: Sun Aug 10 15:59:20 2014

[gstanden@oracle631 ~]$ ls -lrt /dev/vd*

brw-rw----. 1 root disk 251, 16 Aug 10 15:58 /dev/vdb

brw-rw----. 1 root disk 251, 0 Aug 10 15:58 /dev/vda

brw-rw----. 1 root disk 251, 2 Aug 10 15:58 /dev/vda2

brw-rw----. 1 root disk 251, 1 Aug 10 15:59 /dev/vda1

[gstanden@oracle631 ~]$ su - root

Password:

[root@oracle631 ~]# fdisk -l /dev/vdb

Disk /dev/vdb: 21.5 GB, 21474836480 bytes

16 heads, 63 sectors/track, 41610 cylinders

Units = cylinders of 1008 * 512 = 516096 bytes

Sector size (logical/physical): 512 bytes / 512 bytes

I/O size (minimum/optimal): 512 bytes / 512 bytes

Disk identifier: 0x00000000

[root@oracle631 ~]#

Create an LVM and filesystem on /dev/vdb inside the KVM guest, mount it at /u00, and update the fstab so the filesystem will mount on boot, as shown below.

[root@oracle631 /]# pvcreate /dev/vdb

Physical volume "/dev/vdb" successfully created

[root@oracle631 /]# vgcreate vg_u00 /dev/vdb

Volume group "vg_u00" successfully created

[root@oracle631 /]# lvcreate -l 100%FREE -n lv_oracle vg_u00

Logical volume "lv_oracle" created

[root@oracle631 /]# ls -lrt /dev/mapper

total 0

crw-rw----. 1 root root 10, 236 Aug 10 15:59 control

lrwxrwxrwx. 1 root root 7 Aug 10 15:59 vg_oracle631-lv_swap -> ../dm-1

lrwxrwxrwx. 1 root root 7 Aug 10 15:59 vg_oracle631-lv_root -> ../dm-0

lrwxrwxrwx. 1 root root 7 Aug 10 16:08 vg_u00-lv_oracle -> ../dm-2

[root@oracle631 /]# mkfs.ext4 /dev/mapper/vg_u00-lv_oracle

mke2fs 1.43-WIP (20-Jun-2013)

Filesystem label=

OS type: Linux

Block size=4096 (log=2)

Fragment size=4096 (log=2)

Stride=0 blocks, Stripe width=0 blocks

1310720 inodes, 5241856 blocks

262092 blocks (5.00%) reserved for the super user

First data block=0

Maximum filesystem blocks=4294967296

160 block groups

32768 blocks per group, 32768 fragments per group

8192 inodes per group

Superblock backups stored on blocks:

32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208,

4096000

Allocating group tables: done

Writing inode tables: done

Creating journal (32768 blocks): done

Writing superblocks and filesystem accounting information: done

[root@oracle631 /]# mount -t ext4 /dev/mapper/vg_u00-lv_oracle /u00

[root@oracle631 /]# df -TH

Filesystem Type Size Used Avail Use% Mounted on

/dev/mapper/vg_oracle631-lv_root ext4 19G 3.9G 14G 23% /

tmpfs tmpfs 522M 74k 522M 1% /dev/shm

/dev/vda1 ext4 500M 58M 416M 13% /boot

/dev/mapper/vg_u00-lv_oracle ext4 21G 47M 20G 1% /u00

Add the following code to the fstab so that /u00 will mount on boot automatically and reboot the KVM guest if desired to verify fstab is configured correctly. Note it is a good idea to make a backup of the fstab before editing it and to take care when editing fstab.

[root@oracle631 /]# cat /etc/fstab | grep u00

/dev/mapper/vg_u00-lv_oracle /u00 ext4 defaults 1 1

[root@oracle631 /]#

Install the oracle preinstall RPM. In this case, grid 12c is being install, so the Oracle 12 version of the preinstall RPM is used as shown below.

Edit the /etc/security/limits.conf file as shown below.

Run the supplementary list of packages yum install script.

Add space to the swap volume if necessary to ensure at least 4G of swap available using the guide here.

Ensure that the KVM guest has at least 4G of RAM allocated.

Ensure that umask is set to 022 in the /home/grid/.bashrc file.

Ensure that the FQDN hostname of the server is resolvable using either a DNS server or /etc/hosts. Note that although this is standalone, in the case of RAC DNS server is mandatory and /etc/hosts is not acceptable. If the hostname is not resolvable correctly, the network configuration assistant of the installer may hang. If it does, /etc/hosts or DNS can be configured and once the resolution information is available, it will be possible for the network configuration assistant to continue. Also, in this type of situation, there may be output on the crsctl stat res -t that says "not all endpoints connected" in the listener status. In this case, ensure all server hostname can be resolved successfully. If not, replace hostnames with IP address of server in listener.ora and tnsnames.ora.

Follow all the steps in the Violin Memory Oracle Best Practice Document April 2014 to prepare the system for Oracle 12c install on flash. Run the installer as shown below.

When the install completes, update the /home/grid/.bashrc with the $ORACLE_HOME, $GRID_HOME, $PATH and $ORACLE_SID environment variables.

Run the installer as shown below to install the Oracle database.

[oracle@oracle631 database]$ ./runInstaller

Starting Oracle Universal Installer...

Checking Temp space: must be greater than 120 MB. Actual 2524 MB Passed

Checking swap space: must be greater than 150 MB. Actual 5087 MB Passed

Checking monitor: must be configured to display at least 256 colors. Actual 16777216 Passed

Preparing to launch Oracle Universal Installer from /tmp/OraInstall2014-08-13_02-43-03PM. Please wait ...[oracle@oracle631 database]$ You can find the log of this install session at:

/u00/app/oraInventory/logs/installActions2014-08-13_02-43-03PM.log

[oracle@oracle631 database]$ exit

Run the root.sh script as instructed by the installer as shown below.

[root@oracle631 oracle]# /u00/app/oracle/product/11.2.0/dbhome_1/root.sh

Performing root user operation for Oracle 11g

The following environment variables are set as:

ORACLE_OWNER= oracle

ORACLE_HOME= /u00/app/oracle/product/11.2.0/dbhome_1

Enter the full pathname of the local bin directory: [/usr/local/bin]:

The file "dbhome" already exists in /usr/local/bin. Overwrite it? (y/n)

[n]:

The file "oraenv" already exists in /usr/local/bin. Overwrite it? (y/n)

[n]:

The file "coraenv" already exists in /usr/local/bin. Overwrite it? (y/n)

[n]:

Entries will be added to the /etc/oratab file as needed by

Database Configuration Assistant when a database is created

Finished running generic part of root script.

Now product-specific root actions will be performed.

Finished product-specific root actions.

[root@oracle631 oracle]#