Robin SLOB2 PDB Protocols
This page describes setting up SLOB2 for a PDB running on the Robin Systems platform. However, this protocol is generally applicable to any Oracle 12c PDB. This is a WIP where currently I am keeping rough notes that will be developed into a finalized page.
Download PDB-Aware SLOB2
The PDB-Aware SLOB2 is attached to this blogpost.
Modify Online Redo Log Groups
SLOB2 testing requires redo log groups of sufficient size which should be a minimum of 1G each, with two log members in each group, and 5 groups total. Typically an Oracle database is created by DBCA with three online redo log groups, non-multiplexed, and of a relatively small size of 50M. These much be changed as shown below.
Create Required Log Directories
This Robin Oracle system uses files for database storage. Since multiplexing of the online redo logs is being added, it will be necessary to first create the required second online redo log directory structure as the root use as shown below.
[root@vnode-90-20 /]# chown oracle:dba u00
[root@vnode-90-20 /]# mkdir -p /u00/app/oracle/oradata/cdb1/
[root@vnode-90-20 /]# chown -R oracle:dba /u00/app/oracle/oradata/cdb1/
Create Required Log Groups
It is necessary to create additional online redo log groups, and to modify existing redo log groups to enlarge the size of the redo log files and also to add a second logfile group member for multiplexing. To make such changes to a redo group actually requires that the redo log group be dropped and rebuilt to this new specification. This work can be done while the database is up provided that the redo logs are not switching too fast. If they are switching fast, then create temporarily a very very big online redo log group (large file size) that willl handle 20-30 minutes of redo activity to prevent switching. The super-jumbo redo log group can be dropped when this work is done. It's only purpose is to prevent redo logs from switching too fast while the below work is completed.
Check Current Oracle Redo Log Configuration
Login to the CDB as system or sys and check the current configuration of the online redo logs as shown below.
[oracle@vnode-90-20 ~]$ source oracle.env
[oracle@vnode-90-20 ~]$ sqlplus "/ as sysdba"
SQL*Plus: Release 12.1.0.2.0 Production on Fri Aug 19 11:18:21 2016
Copyright (c) 1982, 2014, Oracle. All rights reserved.
Connected to:
Oracle Database 12c Enterprise Edition Release 12.1.0.2.0 - 64bit Production
With the Partitioning, OLAP, Advanced Analytics and Real Application Testing options
SQL> set linesize 200
SQL> column file_name format a70
SQL> select * from v$log;
GROUP# THREAD# SEQUENCE# BYTES BLOCKSIZE MEMBERS ARC STATUS FIRST_CHANGE# FIRST_TIM NEXT_CHANGE# NEXT_TIME CON_ID
---------- ---------- ---------- ---------- ---------- ---------- --- ---------------- ------------- --------- ------------ --------- ----------
1 1 49 52428800 512 1 NO CURRENT 2312079 19-AUG-16 2.8147E+14 0
2 1 47 52428800 512 1 NO INACTIVE 2294552 19-AUG-16 2311260 19-AUG-16 0
3 1 48 52428800 512 1 NO INACTIVE 2311260 19-AUG-16 2312079 19-AUG-16 0
SQL> select * from v$logfile;
GROUP# STATUS TYPE MEMBER IS_ CON_ID
---------- ------- ------- ------------------------------------------------------------ --- ----------
3 ONLINE /u01/app/oracle/oradata/cdb1/redo03.log NO 0
2 ONLINE /u01/app/oracle/oradata/cdb1/redo02.log NO 0
1 ONLINE /u01/app/oracle/oradata/cdb1/redo01.log NO 0
Add New Redo Log Groups
In this step we have the situation of a typical DBCA-created DB that has 3 online redo log groups. Five online redo log groups are needed for the SLOB2 testing, of size 1G each, so two more groups are added in this step as shown below. We are continuing in the same SQL*Plus session so no login information to SQL*Plus is shown.
SQL> alter database add logfile group 4 '/u01/app/oracle/oradata/cdb1/redo04a.log' size 1G;
Database altered.
SQL> alter database add logfile member '/u00/app/oracle/oradata/cdb1/redo04b.log' to group 4;
Database altered.
SQL> set linesize 200
SQL> select * from v$log;
GROUP# THREAD# SEQUENCE# BYTES BLOCKSIZE MEMBERS ARC STATUS FIRST_CHANGE# FIRST_TIM NEXT_CHANGE# NEXT_TIME CON_ID
---------- ---------- ---------- ---------- ---------- ---------- --- ---------------- ------------- --------- ------------ --------- ----------
1 1 49 52428800 512 1 NO CURRENT 2312079 19-AUG-16 2.8147E+14 0
2 1 47 52428800 512 1 NO INACTIVE 2294552 19-AUG-16 2311260 19-AUG-16 0
3 1 48 52428800 512 1 NO INACTIVE 2311260 19-AUG-16 2312079 19-AUG-16 0
4 1 0 1073741824 512 2 YES UNUSED 0 0 0
SQL> column member format a70
SQL> select * from v$logfile;
GROUP# STATUS TYPE MEMBER IS_ CON_ID
---------- ------- ------- ---------------------------------------------------------------------- --- ----------
3 ONLINE /u01/app/oracle/oradata/cdb1/redo03.log NO 0
2 ONLINE /u01/app/oracle/oradata/cdb1/redo02.log NO 0
1 ONLINE /u01/app/oracle/oradata/cdb1/redo01.log NO 0
4 ONLINE /u01/app/oracle/oradata/cdb1/redo04a.log NO 0
4 INVALID ONLINE /u00/app/oracle/oradata/cdb1/redo04b.log NO 0
SQL> alter database add logfile group 4 '/u01/app/oracle/oradata/cdb1/redo04a.log' size 1G
2
SQL> c/group 4/group 5
1* alter database add logfile group 5 '/u01/app/oracle/oradata/cdb1/redo04a.log' size 1G
SQL> c/04a/05a
1* alter database add logfile group 5 '/u01/app/oracle/oradata/cdb1/redo05a.log' size 1G
SQL> /
Database altered.
SQL> alter database add logfile member '/u00/app/oracle/oradata/cdb1/redo04b.log' to group 5
2
SQL> c/04b/05b/
1* alter database add logfile member '/u00/app/oracle/oradata/cdb1/redo05b.log' to group 5
SQL> /
Database altered.
SQL> select * from v$log;
GROUP# THREAD# SEQUENCE# BYTES BLOCKSIZE MEMBERS ARC STATUS FIRST_CHANGE# FIRST_TIM NEXT_CHANGE# NEXT_TIME CON_ID
---------- ---------- ---------- ---------- ---------- ---------- --- ---------------- ------------- --------- ------------ --------- ----------
1 1 49 52428800 512 1 NO CURRENT 2312079 19-AUG-16 2.8147E+14 0
2 1 47 52428800 512 1 NO INACTIVE 2294552 19-AUG-16 2311260 19-AUG-16 0
3 1 48 52428800 512 1 NO INACTIVE 2311260 19-AUG-16 2312079 19-AUG-16 0
4 1 0 1073741824 512 2 YES UNUSED 0 0 0
5 1 0 1073741824 512 2 YES UNUSED 0 0 0
SQL> select * from v$logfile;
GROUP# STATUS TYPE MEMBER IS_ CON_ID
---------- ------- ------- ---------------------------------------------------------------------- --- ----------
3 ONLINE /u01/app/oracle/oradata/cdb1/redo03.log NO 0
2 ONLINE /u01/app/oracle/oradata/cdb1/redo02.log NO 0
1 ONLINE /u01/app/oracle/oradata/cdb1/redo01.log NO 0
4 ONLINE /u01/app/oracle/oradata/cdb1/redo04a.log NO 0
4 INVALID ONLINE /u00/app/oracle/oradata/cdb1/redo04b.log NO 0
5 ONLINE /u01/app/oracle/oradata/cdb1/redo05a.log NO 0
5 INVALID ONLINE /u00/app/oracle/oradata/cdb1/redo05b.log NO 0
7 rows selected.
The new logfile groups have 2 members each and have a status of INVALID (expected). To get the status INVALID to clear it's necessary to switch through them as shown below.
SQL> alter system switch logfile;
System altered.
SQL> select * from v$log;
GROUP# THREAD# SEQUENCE# BYTES BLOCKSIZE MEMBERS ARC STATUS FIRST_CHANGE# FIRST_TIM NEXT_CHANGE# NEXT_TIME CON_ID
---------- ---------- ---------- ---------- ---------- ---------- --- ---------------- ------------- --------- ------------ --------- ----------
1 1 49 52428800 512 1 NO ACTIVE 2312079 19-AUG-16 2315641 19-AUG-16 0
2 1 47 52428800 512 1 NO INACTIVE 2294552 19-AUG-16 2311260 19-AUG-16 0
3 1 48 52428800 512 1 NO INACTIVE 2311260 19-AUG-16 2312079 19-AUG-16 0
4 1 50 1073741824 512 2 NO CURRENT 2315641 19-AUG-16 2.8147E+14 0
5 1 0 1073741824 512 2 YES UNUSED 0 0 0
SQL> alter system switch logfile;
System altered.
SQL> select * from v$log;
GROUP# THREAD# SEQUENCE# BYTES BLOCKSIZE MEMBERS ARC STATUS FIRST_CHANGE# FIRST_TIM NEXT_CHANGE# NEXT_TIME CON_ID
---------- ---------- ---------- ---------- ---------- ---------- --- ---------------- ------------- --------- ------------ --------- ----------
1 1 49 52428800 512 1 NO ACTIVE 2312079 19-AUG-16 2315641 19-AUG-16 0
2 1 47 52428800 512 1 NO INACTIVE 2294552 19-AUG-16 2311260 19-AUG-16 0
3 1 48 52428800 512 1 NO INACTIVE 2311260 19-AUG-16 2312079 19-AUG-16 0
4 1 50 1073741824 512 2 NO ACTIVE 2315641 19-AUG-16 2315648 19-AUG-16 0
5 1 51 1073741824 512 2 NO CURRENT 2315648 19-AUG-16 2.8147E+14 0
SQL> alter system checkpoint global;
System altered.
SQL> select * from v$log;
GROUP# THREAD# SEQUENCE# BYTES BLOCKSIZE MEMBERS ARC STATUS FIRST_CHANGE# FIRST_TIM NEXT_CHANGE# NEXT_TIME CON_ID
---------- ---------- ---------- ---------- ---------- ---------- --- ---------------- ------------- --------- ------------ --------- ----------
1 1 49 52428800 512 1 NO INACTIVE 2312079 19-AUG-16 2315641 19-AUG-16 0
2 1 47 52428800 512 1 NO INACTIVE 2294552 19-AUG-16 2311260 19-AUG-16 0
3 1 48 52428800 512 1 NO INACTIVE 2311260 19-AUG-16 2312079 19-AUG-16 0
4 1 50 1073741824 512 2 NO INACTIVE 2315641 19-AUG-16 2315648 19-AUG-16 0
5 1 51 1073741824 512 2 NO CURRENT 2315648 19-AUG-16 2.8147E+14 0
SQL> alter system switch logfile;
System altered.
SQL> alter system checkpoint global;
System altered.
SQL> select * from v$log;
GROUP# THREAD# SEQUENCE# BYTES BLOCKSIZE MEMBERS ARC STATUS FIRST_CHANGE# FIRST_TIM NEXT_CHANGE# NEXT_TIME CON_ID
---------- ---------- ---------- ---------- ---------- ---------- --- ---------------- ------------- --------- ------------ --------- ----------
1 1 49 52428800 512 1 NO INACTIVE 2312079 19-AUG-16 2315641 19-AUG-16 0
2 1 52 52428800 512 1 NO CURRENT 2315676 19-AUG-16 2.8147E+14 0
3 1 48 52428800 512 1 NO INACTIVE 2311260 19-AUG-16 2312079 19-AUG-16 0
4 1 50 1073741824 512 2 NO INACTIVE 2315641 19-AUG-16 2315648 19-AUG-16 0
5 1 51 1073741824 512 2 NO INACTIVE 2315648 19-AUG-16 2315676 19-AUG-16 0
Stop switching when the INVALID status is gone and when logfile group 1 shows INACTIVE status. Be sure to run an "alter system checkpoint global;" in SQL*Plus before doing the next steps below.
Drop and Recreate Logfile Group 1
Ensure that logfile group 1 is not in the CURRENT state (we made sure of that above). If by mistake you try to do the following and group 1 is in the CURRENT state, no harm done, the command will just fail when attempting to drop a CURRENT online redo log group. You might get a different message if log group 1 is "INACTIVE" but you haven't yet issues and "alter system checkpoint global;" command. Whichever you might run into, now you know how to deal with it.
Drop logfile group 1 as shown below.
SQL> alter database drop logfile group 1;
Database altered.
SQL> select * from v$log;
GROUP# THREAD# SEQUENCE# BYTES BLOCKSIZE MEMBERS ARC STATUS FIRST_CHANGE# FIRST_TIM NEXT_CHANGE# NEXT_TIME CON_ID
---------- ---------- ---------- ---------- ---------- ---------- --- ---------------- ------------- --------- ------------ --------- ----------
2 1 52 52428800 512 1 NO CURRENT 2315676 19-AUG-16 2.8147E+14 0
3 1 48 52428800 512 1 NO INACTIVE 2311260 19-AUG-16 2312079 19-AUG-16 0
4 1 50 1073741824 512 2 NO INACTIVE 2315641 19-AUG-16 2315648 19-AUG-16 0
5 1 51 1073741824 512 2 NO INACTIVE 2315648 19-AUG-16 2315676 19-AUG-16 0
SQL> alter database add logfile group 4 '/u01/app/oracle/oradata/cdb1/redo04a.log' size 1G
2
SQL> c/group 4/group 1/
1* alter database add logfile group 1 '/u01/app/oracle/oradata/cdb1/redo04a.log' size 1G
SQL> c/04a/01a
1* alter database add logfile group 1 '/u01/app/oracle/oradata/cdb1/redo01a.log' size 1G
SQL> /
Database altered.
SQL> alter database add logfile member '/u00/app/oracle/oradata/cdb1/redo04b.log' to group 5
2
SQL> c/04b/01b
1* alter database add logfile member '/u00/app/oracle/oradata/cdb1/redo01b.log' to group 5
SQL> c/5/1
1* alter database add logfile member '/u00/app/oracle/oradata/cdb1/redo01b.log' to group 1
SQL> /
Database altered.
SQL> select * from v$log;
GROUP# THREAD# SEQUENCE# BYTES BLOCKSIZE MEMBERS ARC STATUS FIRST_CHANGE# FIRST_TIM NEXT_CHANGE# NEXT_TIME CON_ID
---------- ---------- ---------- ---------- ---------- ---------- --- ---------------- ------------- --------- ------------ --------- ----------
1 1 0 1073741824 512 2 YES UNUSED 0 0 0
2 1 52 52428800 512 1 NO CURRENT 2315676 19-AUG-16 2.8147E+14 0
3 1 48 52428800 512 1 NO INACTIVE 2311260 19-AUG-16 2312079 19-AUG-16 0
4 1 50 1073741824 512 2 NO INACTIVE 2315641 19-AUG-16 2315648 19-AUG-16 0
5 1 51 1073741824 512 2 NO INACTIVE 2315648 19-AUG-16 2315676 19-AUG-16 0
SQL> alter system switch logfile;
System altered.
SQL> alter system checkpoint global;
System altered.
SQL> select * from v$log;
GROUP# THREAD# SEQUENCE# BYTES BLOCKSIZE MEMBERS ARC STATUS FIRST_CHANGE# FIRST_TIM NEXT_CHANGE# NEXT_TIME CON_ID
---------- ---------- ---------- ---------- ---------- ---------- --- ---------------- ------------- --------- ------------ --------- ----------
1 1 53 1073741824 512 2 NO CURRENT 2315776 19-AUG-16 2.8147E+14 0
2 1 52 52428800 512 1 NO INACTIVE 2315676 19-AUG-16 2315776 19-AUG-16 0
3 1 48 52428800 512 1 NO INACTIVE 2311260 19-AUG-16 2312079 19-AUG-16 0
4 1 50 1073741824 512 2 NO INACTIVE 2315641 19-AUG-16 2315648 19-AUG-16 0
5 1 51 1073741824 512 2 NO INACTIVE 2315648 19-AUG-16 2315676 19-AUG-16 0
SQL> alter database drop logfile group 2;
Database altered.
Drop and Recreate Logfile Group 2
Modify the code from the Logfile Group 1 operations above by cutting and pasting the "add logfile group" command (WITHOUT the ";") and then change the command for Logfile Group 2 and run the commands as suggested below.
SQL> alter database add logfile group 1 '/u01/app/oracle/oradata/cdb1/redo01a.log' size 1G
2
SQL> c/1/2
1* alter database add logfile group 2 '/u01/app/oracle/oradata/cdb1/redo01a.log' size 1G
SQL> c/01a/02a
1* alter database add logfile group 2 '/u01/app/oracle/oradata/cdb1/redo02a.log' size 1G
SQL> /
Database altered.
SQL> alter database add logfile member '/u00/app/oracle/oradata/cdb1/redo01b.log' to group 1
SQL> c/01b/02b
1* alter database add logfile member '/u00/app/oracle/oradata/cdb1/redo02b.log' to group 1
SQL> c/group 1/group 2
1* alter database add logfile member '/u00/app/oracle/oradata/cdb1/redo02b.log' to group 2
SQL> /
Database altered.
SQL> select * from v$log;
GROUP# THREAD# SEQUENCE# BYTES BLOCKSIZE MEMBERS ARC STATUS FIRST_CHANGE# FIRST_TIM NEXT_CHANGE# NEXT_TIME CON_ID
---------- ---------- ---------- ---------- ---------- ---------- --- ---------------- ------------- --------- ------------ --------- ----------
1 1 53 1073741824 512 2 NO CURRENT 2315776 19-AUG-16 2.8147E+14 0
2 1 0 1073741824 512 2 YES UNUSED 0 0 0
3 1 48 52428800 512 1 NO INACTIVE 2311260 19-AUG-16 2312079 19-AUG-16 0
4 1 50 1073741824 512 2 NO INACTIVE 2315641 19-AUG-16 2315648 19-AUG-16 0
5 1 51 1073741824 512 2 NO INACTIVE 2315648 19-AUG-16 2315676 19-AUG-16 0
SQL> select * from v$logfile order by member;
GROUP# STATUS TYPE MEMBER IS_ CON_ID
---------- ------- ------- ---------------------------------------------------------------------- --- ----------
1 ONLINE /u00/app/oracle/oradata/cdb1/redo01b.log NO 0
2 INVALID ONLINE /u00/app/oracle/oradata/cdb1/redo02b.log NO 0
4 ONLINE /u00/app/oracle/oradata/cdb1/redo04b.log NO 0
5 ONLINE /u00/app/oracle/oradata/cdb1/redo05b.log NO 0
1 ONLINE /u01/app/oracle/oradata/cdb1/redo01a.log NO 0
2 ONLINE /u01/app/oracle/oradata/cdb1/redo02a.log NO 0
3 ONLINE /u01/app/oracle/oradata/cdb1/redo03.log NO 0
4 ONLINE /u01/app/oracle/oradata/cdb1/redo04a.log NO 0
5 ONLINE /u01/app/oracle/oradata/cdb1/redo05a.log NO 0
9 rows selected.
SQL> select * from v$log;
GROUP# THREAD# SEQUENCE# BYTES BLOCKSIZE MEMBERS ARC STATUS FIRST_CHANGE# FIRST_TIM NEXT_CHANGE# NEXT_TIME CON_ID
---------- ---------- ---------- ---------- ---------- ---------- --- ---------------- ------------- --------- ------------ --------- ----------
1 1 53 1073741824 512 2 NO CURRENT 2315776 19-AUG-16 2.8147E+14 0
2 1 0 1073741824 512 2 YES UNUSED 0 0 0
3 1 48 52428800 512 1 NO INACTIVE 2311260 19-AUG-16 2312079 19-AUG-16 0
4 1 50 1073741824 512 2 NO INACTIVE 2315641 19-AUG-16 2315648 19-AUG-16 0
5 1 51 1073741824 512 2 NO INACTIVE 2315648 19-AUG-16 2315676 19-AUG-16 0
Drop and Recreate Logfile Group 3
Drop and recreate logfile group 3 as shown below.
SQL> alter database drop logfile group 3;
Database altered.
SQL> alter database add logfile group 2 '/u01/app/oracle/oradata/cdb1/redo02a.log' size 1G
2
SQL> c/group 2/group 3/
1* alter database add logfile group 3 '/u01/app/oracle/oradata/cdb1/redo02a.log' size 1G
SQL> c/02a/03a
1* alter database add logfile group 3 '/u01/app/oracle/oradata/cdb1/redo03a.log' size 1G
SQL> /
Database altered.
SQL> alter database add logfile member '/u00/app/oracle/oradata/cdb1/redo02b.log' to group 2
2
SQL> c/02b/03b
1* alter database add logfile member '/u00/app/oracle/oradata/cdb1/redo03b.log' to group 2
SQL> c/group 2/group 3
1* alter database add logfile member '/u00/app/oracle/oradata/cdb1/redo03b.log' to group 3
SQL> /
Database altered.
SQL> select * from v$log;
GROUP# THREAD# SEQUENCE# BYTES BLOCKSIZE MEMBERS ARC STATUS FIRST_CHANGE# FIRST_TIM NEXT_CHANGE# NEXT_TIME CON_ID
---------- ---------- ---------- ---------- ---------- ---------- --- ---------------- ------------- --------- ------------ --------- ----------
1 1 53 1073741824 512 2 NO CURRENT 2315776 19-AUG-16 2.8147E+14 0
2 1 0 1073741824 512 2 YES UNUSED 0 0 0
3 1 0 1073741824 512 2 YES UNUSED 0 0 0
4 1 50 1073741824 512 2 NO INACTIVE 2315641 19-AUG-16 2315648 19-AUG-16 0
5 1 51 1073741824 512 2 NO INACTIVE 2315648 19-AUG-16 2315676 19-AUG-16 0
SQL> alter system checkpoint global;
System altered.
SQL> select * from v$log;
GROUP# THREAD# SEQUENCE# BYTES BLOCKSIZE MEMBERS ARC STATUS FIRST_CHANGE# FIRST_TIM NEXT_CHANGE# NEXT_TIME CON_ID
---------- ---------- ---------- ---------- ---------- ---------- --- ---------------- ------------- --------- ------------ --------- ----------
1 1 53 1073741824 512 2 NO CURRENT 2315776 19-AUG-16 2.8147E+14 0
2 1 0 1073741824 512 2 YES UNUSED 0 0 0
3 1 0 1073741824 512 2 YES UNUSED 0 0 0
4 1 50 1073741824 512 2 NO INACTIVE 2315641 19-AUG-16 2315648 19-AUG-16 0
5 1 51 1073741824 512 2 NO INACTIVE 2315648 19-AUG-16 2315676 19-AUG-16 0
SQL> select * from v$logfile;
GROUP# STATUS TYPE MEMBER IS_ CON_ID
---------- ------- ------- ---------------------------------------------------------------------- --- ----------
3 ONLINE /u01/app/oracle/oradata/cdb1/redo03a.log NO 0
2 ONLINE /u01/app/oracle/oradata/cdb1/redo02a.log NO 0
1 ONLINE /u01/app/oracle/oradata/cdb1/redo01a.log NO 0
4 ONLINE /u01/app/oracle/oradata/cdb1/redo04a.log NO 0
4 ONLINE /u00/app/oracle/oradata/cdb1/redo04b.log NO 0
5 ONLINE /u01/app/oracle/oradata/cdb1/redo05a.log NO 0
5 ONLINE /u00/app/oracle/oradata/cdb1/redo05b.log NO 0
1 ONLINE /u00/app/oracle/oradata/cdb1/redo01b.log NO 0
2 INVALID ONLINE /u00/app/oracle/oradata/cdb1/redo02b.log NO 0
3 INVALID ONLINE /u00/app/oracle/oradata/cdb1/redo03b.log NO 0
10 rows selected.
SQL> alter system switch logfile;
System altered.
SQL> /
System altered.
SQL> /
System altered.
SQL> /
System altered.
SQL> /
System altered.
SQL> alter system checkpoint global;
System altered.
SQL> select * from v$logfile;
GROUP# STATUS TYPE MEMBER IS_ CON_ID
---------- ------- ------- ---------------------------------------------------------------------- --- ----------
3 ONLINE /u01/app/oracle/oradata/cdb1/redo03a.log NO 0
2 ONLINE /u01/app/oracle/oradata/cdb1/redo02a.log NO 0
1 ONLINE /u01/app/oracle/oradata/cdb1/redo01a.log NO 0
4 ONLINE /u01/app/oracle/oradata/cdb1/redo04a.log NO 0
4 ONLINE /u00/app/oracle/oradata/cdb1/redo04b.log NO 0
5 ONLINE /u01/app/oracle/oradata/cdb1/redo05a.log NO 0
5 ONLINE /u00/app/oracle/oradata/cdb1/redo05b.log NO 0
1 ONLINE /u00/app/oracle/oradata/cdb1/redo01b.log NO 0
2 ONLINE /u00/app/oracle/oradata/cdb1/redo02b.log NO 0
3 ONLINE /u00/app/oracle/oradata/cdb1/redo03b.log NO 0
10 rows selected.
SQL>
SQL> l
1* select * from v$logfile
SQL> a order by group#, member
1* select * from v$logfile order by group#, member
SQL> /
GROUP# STATUS TYPE MEMBER IS_ CON_ID
---------- ------- ------- ---------------------------------------------------------------------- --- ----------
1 ONLINE /u00/app/oracle/oradata/cdb1/redo01b.log NO 0
1 ONLINE /u01/app/oracle/oradata/cdb1/redo01a.log NO 0
2 ONLINE /u00/app/oracle/oradata/cdb1/redo02b.log NO 0
2 ONLINE /u01/app/oracle/oradata/cdb1/redo02a.log NO 0
3 ONLINE /u00/app/oracle/oradata/cdb1/redo03b.log NO 0
3 ONLINE /u01/app/oracle/oradata/cdb1/redo03a.log NO 0
4 ONLINE /u00/app/oracle/oradata/cdb1/redo04b.log NO 0
4 ONLINE /u01/app/oracle/oradata/cdb1/redo04a.log NO 0
5 ONLINE /u00/app/oracle/oradata/cdb1/redo05b.log NO 0
5 ONLINE /u01/app/oracle/oradata/cdb1/redo05a.log NO 0
10 rows selected.
SQL> select * from v$log;
GROUP# THREAD# SEQUENCE# BYTES BLOCKSIZE MEMBERS ARC STATUS FIRST_CHANGE# FIRST_TIM NEXT_CHANGE# NEXT_TIME CON_ID
---------- ---------- ---------- ---------- ---------- ---------- --- ---------------- ------------- --------- ------------ --------- ----------
1 1 58 1073741824 512 2 NO CURRENT 2315978 19-AUG-16 2.8147E+14 0
2 1 54 1073741824 512 2 NO INACTIVE 2315966 19-AUG-16 2315969 19-AUG-16 0
3 1 55 1073741824 512 2 NO INACTIVE 2315969 19-AUG-16 2315972 19-AUG-16 0
4 1 56 1073741824 512 2 NO INACTIVE 2315972 19-AUG-16 2315975 19-AUG-16 0
5 1 57 1073741824 512 2 NO INACTIVE 2315975 19-AUG-16 2315978 19-AUG-16 0
SQL> exit
Disconnected from Oracle Database 12c Enterprise Edition Release 12.1.0.2.0 - 64bit Production
With the Partitioning, OLAP, Advanced Analytics and Real Application Testing options
[oracle@vnode-90-20 ~]$
Create the IOPS Tablespace for SLOB
The command to create the tablespace is shown below.
SQL> create tablespace IOPS datafile size 15G;
which will cause an OMF datafile to be created in the default file creation location as shown below.
[oracle@vnode-90-20 ~]$ sqlplus "/ as sysdba"
SQL*Plus: Release 12.1.0.2.0 Production on Fri Aug 19 11:18:21 2016
Copyright (c) 1982, 2014, Oracle. All rights reserved.
Connected to:
Oracle Database 12c Enterprise Edition Release 12.1.0.2.0 - 64bit Production
With the Partitioning, OLAP, Advanced Analytics and Real Application Testing options
SQL> column file_name format a70
SQL> select file_name, bytes from dba_data_files;
FILE_NAME BYTES
---------------------------------------------------------------------- ----------
/u01/app/oracle/oradata/cdb1/users01.dbf 5242880
/u01/app/oracle/oradata/cdb1/undotbs01.dbf 225443840
/u01/app/oracle/oradata/cdb1/system01.dbf 838860800
/u01/app/oracle/oradata/cdb1/sysaux01.dbf 859832320
SQL> alter session set container=RSPDB1;
Session altered.
SQL> alter pluggable database open;
alter pluggable database open
*
ERROR at line 1:
ORA-65019: pluggable database RSPDB1 already open
SQL> column file_name format a100
SQL> create tablespace IOPS datafile size 15G;
Tablespace created.
SQL>
SQL> select file_name, bytes from dba_data_files;
FILE_NAME BYTES
---------------------------------------------------------------------------------------------------- ----------
/u01/app/oracle/oradata/cdb1/RSPDB1/system01.dbf 272629760
/u01/app/oracle/oradata/cdb1/RSPDB1/sysaux01.dbf 597688320
/u01/app/oracle/oradata/cdb1/RSPDB1/RSPDB1_users01.dbf 5242880
/u01/app/oracle/oradata/CDB1/3A70EEBEB072848FE053145A090AD96F/datafile/o1_mf_iops_cvgk0zw9_.dbf 1.6106E+10
The tablespace for SLOB is created.
Review UNDOTBS for SLOB
SLOB can use up alot of UNDOTBS during load testing. It's often advisable to create a dedicated jumbo UNDOTBS for SLOB testing and/or to check if the UNDOTBS is in AUTOEXTEND mode as shown below.
SQL> l
1* select file_name, maxbytes, tablespace_name, autoextensible from dba_data_files
SQL> /
FILE_NAME MAXBYTES TABLESPACE_NAME AUT
-------------------------------------------------- ---------- ------------------------------ ---
/u01/app/oracle/oradata/cdb1/users01.dbf 3.4360E+10 USERS YES
/u01/app/oracle/oradata/cdb1/undotbs01.dbf 3.4360E+10 UNDOTBS1 YES
/u01/app/oracle/oradata/cdb1/system01.dbf 3.4360E+10 SYSTEM YES
/u01/app/oracle/oradata/cdb1/sysaux01.dbf 3.4360E+10 SYSAUX YES
SQL>
Disable Resource Manager
It is advisable to disable resource manager during the testing as shown below as explained here.
SQL> alter system set resource_manager_plan='' scope=both;
System altered.
SQL> execute dbms_scheduler.set_attribute('SATURDAY_WINDOW','RESOURCE_PLAN','');
PL/SQL procedure successfully completed.
SQL> execute dbms_scheduler.set_attribute('SUNDAY_WINDOW','RESOURCE_PLAN','');
execute dbms_scheduler.set_attribute('MONDAY_WINDOW','RESOURCE_PLAN','');
execute dbms_scheduler.set_attribute('TUESDAY_WINDOW','RESOURCE_PLAN','');
execute dbms_scheduler.set_attribute('WEDNESDAY_WINDOW','RESOURCE_PLAN','');
execute dbms_scheduler.set_attribute('THURSDAY_WINDOW','RESOURCE_PLAN','');
execute dbms_scheduler.set_attribute('FRIDAY_WINDOW','RESOURCE_PLAN','');
PL/SQL procedure successfully completed.
SQL>
PL/SQL procedure successfully completed.
SQL>
PL/SQL procedure successfully completed.
SQL>
PL/SQL procedure successfully completed.
SQL>
PL/SQL procedure successfully completed.
SQL>
PL/SQL procedure successfully completed.
SQL>
Run the wait_kit
The wait kit must be created (one time only) before running setup.sh for SLOB2 as shown below. Run make from the wait kit directory as shown below. There is a subdirectory under SLOB called "wait_kit" as shown.
[oracle@oracle631 wait_kit]$ make all
rm -fr *.o mywait trigger create_sem
cc -c -o mywait.o mywait.c
cc -o mywait mywait.o
cc -c -o trigger.o trigger.c
cc -o trigger trigger.o
cc -c -o create_sem.o create_sem.c
cc -o create_sem create_sem.o
cp mywait trigger create_sem ../
rm -fr *.o
[oracle@oracle631 wait_kit]$
Script: setup.sh
This section describes the configuration and running of the setup.sh script which creates the SLOB environment.
Preparing Script: setup.sh
Attached to this blog post is the SLOB2 bundle to which I made minor modification for use of SLOB2 with Oracle 12c pluggable databases (PDB). In the script there is a string PDBYOURS. First, make a backup copy of setup.sh (setup.sh.bak) and then vi the setup.sh and change all occurrences of PDBYOURS to "whatever the name of your PDB is". For example, in the work I am doing today, the PDB is "RSPDB1".
Once that edit is done, the setup.sh script will look simlar to that shown below. The entire script is shown below for completeness, but it is also attached to this blogpost, both separately and as part of the SLOB2 tar.gz bundle as well. I've highlighted in BOLD RED the 3 places in the setup.sh where the file must be updated with the name of your PDB.
[oracle@vnode-90-20 SLOB]$ cat setup.sh
#!/bin/bash
# DISCLAIMER OF WARRANTIES AND LIMITATION OF LIABILITY
# The software is supplied "as is" and all use is at your own risk. Peak Performance Systems disclaims
# all warranties of any kind, either express or implied, as to the software, including, but not limited to,
# implied warranties of fitness for a particular purpose, merchantability or non - infringement of proprietary
# rights. Neither this agreement nor any documentation furnished under it is intended to express or imply
# any warranty that the operation of the software will be uninterrupted, timely, or error - free. Under no
# circumstances shall Peak Performance Systems be liable to any user for direct, indirect, incidental,
# consequential, special, or exemplary damages, arising from or relating to this agreement, the software, or
# user#s use or misuse of the softwares. Such limitation of liability shall apply whether the damages arise
# from the use or misuse of the software (including such damages incurred by third parties), or errors of
# the software.
function test_conn() {
local ret=""
local connect_string="$1"
sqlplus -L "$connect_string" <<EOF > .test_conn.out 2>&1
SHOW PARAMETER db_name;
EXIT;
EOF
ret=$?
return $ret
}
function grant() {
local user=$1
sqlplus -s "$CONNECT_STRING" <<EOF
SET ECHO ON
ALTER SESSION SET CONTAINER=RSPDB1;
DROP USER $user CASCADE;
GRANT CONNECT TO $user IDENTIFIED BY $user;
GRANT DBA TO $user;
EXIT;
EOF
echo "DROP USER $user CASCADE;" >> drop_users.sql
}
function cr_seed () {
grant user0 > /dev/null 2>&1
echo "NOTIFY: Creating and loading seed table."
sqlplus -s user0/user0@${NON_ADMIN_CONNECT_STRING} <<EOF
SET ECHO ON
CREATE TABLE seed
(
custid NUMBER(8),
c2 VARCHAR2(128),
c3 VARCHAR2(128) ,
c4 VARCHAR2(128) ,
c5 VARCHAR2(128) ,
c6 VARCHAR2(128) ,
c7 VARCHAR2(128) ,
c8 VARCHAR2(128) ,
c9 VARCHAR2(128) ,
c10 VARCHAR2(128) ,
c11 VARCHAR2(128) ,
c12 VARCHAR2(128) ,
c13 VARCHAR2(128) ,
c14 VARCHAR2(128) ,
c15 VARCHAR2(128) ,
c16 VARCHAR2(128) ,
c17 VARCHAR2(128) ,
c18 VARCHAR2(128) ,
c19 VARCHAR2(128) ,
c20 VARCHAR2(128)
) PARALLEL CACHE PCTFREE 0 tablespace $TABLESPACE;
DECLARE
nrows NUMBER := 1;
x PLS_INTEGER := 0;
fluff VARCHAR2(128) := 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX';
BEGIN
FOR i IN 1 .. ${SCALE} LOOP
INSERT INTO seed VALUES (nrows, fluff, fluff, fluff, fluff, fluff, fluff, fluff, fluff, fluff, fluff, fluff, fluff, fluff, fluff, fluff, fluff, fluff, fluff, fluff);
nrows := nrows + 1;
x := MOD( nrows, 9973 );
IF ( x = 0 ) THEN
COMMIT;
END IF;
END LOOP;
COMMIT;
END;
/
EXIT;
EOF
echo ""
echo "NOTIFY: Seed table loading procedure has exited."
}
function drop_table() {
local user=$1
local pass=$2
sqlplus -s ${user}/${pass}@${NON_ADMIN_CONNECT_STRING} <<EOF
SET ECHO ON
DROP TABLE cf1 PURGE;
EXIT;
EOF
}
function drop_seed_table() {
sqlplus -s user0/user0@${NON_ADMIN_CONNECT_STRING} <<EOF
SET ECHO ON
DROP TABLE seed PURGE;
EXIT;
EOF
}
function cr_tab_and_load() {
local user=$1
local pass=$2
sqlplus -s ${user}/${pass}@${NON_ADMIN_CONNECT_STRING} <<EOF
SET ECHO ON
CREATE TABLE cf1
(
custid NUMBER(8),
c2 VARCHAR2(128),
c3 VARCHAR2(128) ,
c4 VARCHAR2(128) ,
c5 VARCHAR2(128) ,
c6 VARCHAR2(128) ,
c7 VARCHAR2(128) ,
c8 VARCHAR2(128) ,
c9 VARCHAR2(128) ,
c10 VARCHAR2(128) ,
c11 VARCHAR2(128) ,
c12 VARCHAR2(128) ,
c13 VARCHAR2(128) ,
c14 VARCHAR2(128) ,
c15 VARCHAR2(128) ,
c16 VARCHAR2(128) ,
c17 VARCHAR2(128) ,
c18 VARCHAR2(128) ,
c19 VARCHAR2(128) ,
c20 VARCHAR2(128)
) NOPARALLEL CACHE PCTFREE 99 TABLESPACE $TABLESPACE
STORAGE
(BUFFER_POOL RECYCLE INITIAL 1M NEXT 32K MAXEXTENTS UNLIMITED);
INSERT INTO cf1 SELECT * FROM user0.seed WHERE ROWNUM = 1 ;
COMMIT;
ALTER TABLE cf1 MINIMIZE RECORDS_PER_BLOCK;
TRUNCATE TABLE cf1 ;
COMMIT;
INSERT /*+ PARALLEL APPEND */ INTO cf1 SELECT * FROM user0.seed ORDER BY DBMS_RANDOM.VALUE();
COMMIT;
CREATE UNIQUE INDEX I_CF1 ON cf1(custid) NOPARALLEL PCTFREE 0 TABLESPACE $TABLESPACE;
ALTER INDEX i_cf1 SHRINK SPACE COMPACT;
EXEC DBMS_STATS.GATHER_TABLE_STATS('$user', 'cf1', estimate_percent=>100, block_sample=>TRUE, degree=>2);
EXIT;
EOF
}
function doit() {
local user=$1
local pass=$2
drop_table $user $pass >> drop_table.out 2>&1
cr_tab_and_load $user $pass >> cr_tab_and_load.out 2>&1
}
function test_sqlplus () {
if ( ! type sqlplus > /dev/null 2>&1 )
then
echo "FATAL: Please validate your environment. SQL*Plus is not executable in current \$PATH"
return 1
fi
return 0
}
function check_bom() {
local file=""
if [ ! -f ./misc/BOM ]
then
echo "FATAL: ${0}: ${FUNCNAME}: No BOM file in ./misc. Incorrect SLOB file contents."
return 1
fi
for file in `cat ./misc/BOM | xargs echo`
do
if [ ! -f "$file" ]
then
echo "FATAL: ${0}: ${FUNCNAME}: Missing ${file}. Incorrect SLOB file contents."
return 1
fi
done
return 0
}
#---------- Main body
if ( ! check_bom )
then
exit 1
fi
if ( ! test_sqlplus )
then
echo "FATAL: Cannot proceed in current environment."
exit 1
fi
export TABLESPACE="$1"
if [ -z "$1" ]
then
echo "FATAL: ${0} args"
echo "Usage : ${0}: <tablespace name> <optional: number of users>"
exit 1
fi
MAXUSER=`echo "$2" 2>&1 | sed 's/[^0-9]//g' 2> /dev/null`
if [ -z "$MAXUSER" ]
then
MAXUSER=128
fi
if [[ "$MAXUSER" -le 0 || "$MAXUSER" -gt 128 ]]
then
echo "FATAL: Must be non-zero and tested maximum is 128."
echo "Usage : ${0}: <tablespace name> <optional: number of users>"
exit 1
fi
LOAD_PARALLEL_DEGREE=${LOAD_PARALLEL_DEGREE:=1}
SCALE=${SCALE:=10000}
SQLNET_SERVICE_BASE=${SQLNET_BASE:=''}
SQLNET_SERVICE_MAX=${SQLNET_SERVICE_MAX:=''}
ADMIN_SQLNET_SERVICE=${ADMIN_SQLNET_CONNECT:=''}
SYSDBA_PASSWD=${SYSDBA_PASSWD:=''}
export NON_ADMIN_CONNECT_STRING="RSPDB1"
export CONNECT_STRING="RSPDB1"
source ./slob.conf
rm -f grant.out drop_table.out cr_tab_and_load.out drop_users.sql .test_conn.out
if [ -n "$ADMIN_SQLNET_SERVICE" ]
then
export CONNECT_STRING="sys/${SYSDBA_PASSWD}@${ADMIN_SQLNET_SERVICE} as sysdba"
export NON_ADMIN_CONNECT_STRING="@${ADMIN_SQLNET_SERVICE}"
else
export CONNECT_STRING="/ as sysdba"
fi
echo "
NOTIFY: Load Parameters (slob.conf):
LOAD_PARALLEL_DEGREE == $LOAD_PARALLEL_DEGREE
SCALE == $SCALE
ADMIN_SQLNET_SERVICE == \"$ADMIN_SQLNET_SERVICE\"
CONNECT_STRING == \"$CONNECT_STRING\"
NON_ADMIN_CONNECT_STRING == $NON_ADMIN_CONNECT_STRING
"
echo "NOTIFY: Testing connectivity to the instance to validate slob.conf settings."
if ( ! test_conn "$CONNECT_STRING" )
then
echo "FATAL: ${0}: cannot connect to the instance."
echo "FATAL: Error Output:"
echo ""
cat .test_conn.out 2>&1
echo ""
echo "FATAL: Please verify the instance is running and the settings"
echo "FATAL: in slob.conf are correct for your connectivity model."
exit 1
else
echo "NOTIFY: ${0}: Successful test connection: \"sqlplus -L $CONNECT_STRING\""
echo ""
rm -f .test_conn.out
fi
USER=""
PASS=""
cr_seed
cnt=1
x=0
echo -ne "NOTIFY: Setting up user \c"
while [ $cnt -le $MAXUSER ]
do
echo -ne " $cnt \c"
USER=user$cnt
PASS=user$cnt
( grant $USER >> grant.out 2>&1 ; doit $USER $PASS ) &
if [ $x -eq $(( LOAD_PARALLEL_DEGREE - 1 )) ]
then
echo ""
echo "NOTIFY: Waiting for background processes - `date`"
wait
[[ $cnt -ne $MAXUSER ]] && echo -ne "NOTIFY: Setting up user \c"
x=0
else
(( x = $x + 1 ))
fi
(( cnt = $cnt + 1 ))
done
wait
echo ""
doit user0 user0
drop_seed_table
wait
echo "NOTIFY: ${0}: Loading procedure complete (${SECONDS} seconds). Please check ./cr_tab_and_load.out for any errors"
echo ""
[oracle@vnode-90-20 SLOB]$
Running Script: setup.sh
Run the setup.sh script as shown below.
[oracle@vnode-90-20 SLOB]$ ./setup.sh IOPS 128 | tee /home/oracle/setup_SLOB.log
NOTIFY: Load Parameters (slob.conf):
LOAD_PARALLEL_DEGREE == 8
SCALE == 10000
ADMIN_SQLNET_SERVICE == ""
CONNECT_STRING == "/ as sysdba"
NON_ADMIN_CONNECT_STRING == RSPDB1
NOTIFY: Testing connectivity to the instance to validate slob.conf settings.
NOTIFY: ./setup.sh: Successful test connection: "sqlplus -L / as sysdba"
NOTIFY: Creating and loading seed table.
Table created.
PL/SQL procedure successfully completed.
NOTIFY: Seed table loading procedure has exited.
NOTIFY: Setting up user 1 2 3 4 5 6 7 8
NOTIFY: Waiting for background processes - Fri Aug 19 12:21:54 PDT 2016
NOTIFY: Setting up user 9 10 11 12 13 14 15 16
NOTIFY: Waiting for background processes - Fri Aug 19 12:22:10 PDT 2016
NOTIFY: Setting up user 17 18 19 20 21 22 23 24
NOTIFY: Waiting for background processes - Fri Aug 19 12:22:22 PDT 2016
NOTIFY: Setting up user 25 26 27 28 29 30 31 32
NOTIFY: Waiting for background processes - Fri Aug 19 12:22:34 PDT 2016
NOTIFY: Setting up user 33 34 35 36 37 38 39 40
NOTIFY: Waiting for background processes - Fri Aug 19 12:22:47 PDT 2016
NOTIFY: Setting up user 41 42 43 44 45 46 47 48
NOTIFY: Waiting for background processes - Fri Aug 19 12:22:59 PDT 2016
NOTIFY: Setting up user 49 50 51 52 53 54 55 56
NOTIFY: Waiting for background processes - Fri Aug 19 12:23:11 PDT 2016
NOTIFY: Setting up user 57 58 59 60 61 62 63 64
NOTIFY: Waiting for background processes - Fri Aug 19 12:23:24 PDT 2016
NOTIFY: Setting up user 65 66 67 68 69 70 71 72
NOTIFY: Waiting for background processes - Fri Aug 19 12:23:37 PDT 2016
NOTIFY: Setting up user 73 74 75 76 77 78 79 80
NOTIFY: Waiting for background processes - Fri Aug 19 12:23:49 PDT 2016
NOTIFY: Setting up user 81 82 83 84 85 86 87 88
NOTIFY: Waiting for background processes - Fri Aug 19 12:24:02 PDT 2016
NOTIFY: Setting up user 89 90 91 92 93 94 95 96
NOTIFY: Waiting for background processes - Fri Aug 19 12:24:15 PDT 2016
NOTIFY: Setting up user 97 98 99 100 101 102 103 104
NOTIFY: Waiting for background processes - Fri Aug 19 12:24:26 PDT 2016
NOTIFY: Setting up user 105 106 107 108 109 110 111 112
NOTIFY: Waiting for background processes - Fri Aug 19 12:24:38 PDT 2016
NOTIFY: Setting up user 113 114 115 116 117 118 119 120
NOTIFY: Waiting for background processes - Fri Aug 19 12:24:51 PDT 2016
NOTIFY: Setting up user 121 122 123 124 125 126 127 128
NOTIFY: Waiting for background processes - Fri Aug 19 12:25:03 PDT 2016
Table dropped.
NOTIFY: ./setup.sh: Loading procedure complete (210 seconds). Please check ./cr_tab_and_load.out for any errors
[oracle@vnode-90-20 SLOB]$
Run SLOB2
Run SLOB2 as shown below. Expected output from the logging file is shown using "tail -f" below.
[oracle@vnode-90-20 SLOB]$ nohup ./SLOB2-harness.sh > slob2-harness.out 2>&1 &
[1] 7516
[oracle@vnode-90-20 SLOB]$
[oracle@vnode-90-20 SLOB]$ ps -ef | grep harness
oracle 7516 1327 0 15:47 pts/5 00:00:00 /bin/bash ./SLOB2-harness.sh
oracle 7579 1327 0 15:47 pts/5 00:00:00 grep harness
[oracle@vnode-90-20 SLOB]$ tail -f slob2-harness.out
[oracle@vnode-90-20 SLOB]$ tail -f slob2-harness.out
nohup: ignoring input
20160819-154752 Info : Update Pct list : 0 10 20 30
20160819-154752 Info : SLOB Worker list : 001 002 004 006 008 010 012 014 016 018 020 022 024 026 028 030 032 034 036 038 040 042 044 046 048 052 056 060 064 068 072 076 080 088 096 112 128
20160819-154752 Info : 148 runs of SLOB will be required
20160819-154753 Info : Starting slob-harness...
20160819-154753 Info : Starting loop for UPDATE_PCT=0
20160819-154753 Info : Calling SLOB with 001 workers
20160819-155321 Info : Completed SLOB run for 001 workers: Tm 301
20160819-155321 Info : SLOB Harness is 0% complete
20160819-155321 Info : Calling SLOB with 002 workers
20160819-155841 Info : Completed SLOB run for 002 workers: Tm 300
20160819-155841 Info : SLOB Harness is 1% complete
20160819-155841 Info : Calling SLOB with 004 workers
20160819-160359 Info : Completed SLOB run for 004 workers: Tm 300
20160819-160359 Info : SLOB Harness is 2% complete
20160819-160359 Info : Calling SLOB with 006 workers
20160819-160921 Info : Completed SLOB run for 006 workers: Tm 300
20160819-160921 Info : SLOB Harness is 2% complete
20160819-160921 Info : Calling SLOB with 008 workers
20160819-161442 Info : Completed SLOB run for 008 workers: Tm 300
20160819-161442 Info : SLOB Harness is 3% complete
20160819-161442 Info : Calling SLOB with 010 workers
20160819-162003 Info : Completed SLOB run for 010 workers: Tm 300
20160819-162003 Info : SLOB Harness is 4% complete
20160819-162003 Info : Calling SLOB with 012 workers
20160819-162525 Info : Completed SLOB run for 012 workers: Tm 300
20160819-162525 Info : SLOB Harness is 4% complete
20160819-162525 Info : Calling SLOB with 014 workers
20160819-163051 Info : Completed SLOB run for 014 workers: Tm 300
20160819-163051 Info : SLOB Harness is 5% complete
20160819-163051 Info : Calling SLOB with 016 workers
20160819-163615 Info : Completed SLOB run for 016 workers: Tm 301
20160819-163615 Info : SLOB Harness is 6% complete
20160819-163615 Info : Calling SLOB with 018 workers
20160819-164141 Info : Completed SLOB run for 018 workers: Tm 301
20160819-164141 Info : SLOB Harness is 6% complete
20160819-164141 Info : Calling SLOB with 020 workers
20160819-164709 Info : Completed SLOB run for 020 workers: Tm 301
20160819-164709 Info : SLOB Harness is 7% complete
20160819-164709 Info : Calling SLOB with 022 workers
20160819-165237 Info : Completed SLOB run for 022 workers: Tm 301
20160819-165237 Info : SLOB Harness is 8% complete
20160819-165237 Info : Calling SLOB with 024 workers
20160819-165805 Info : Completed SLOB run for 024 workers: Tm 301
20160819-165805 Info : SLOB Harness is 8% complete
20160819-165805 Info : Calling SLOB with 026 workers
20160819-170338 Info : Completed SLOB run for 026 workers: Tm 301
20160819-170338 Info : SLOB Harness is 9% complete
20160819-170338 Info : Calling SLOB with 028 workers
20160819-170909 Info : Completed SLOB run for 028 workers: Tm 301
20160819-170909 Info : SLOB Harness is 10% complete
20160819-170909 Info : Calling SLOB with 030 workers
20160819-171446 Info : Completed SLOB run for 030 workers: Tm 302
20160819-171446 Info : SLOB Harness is 10% complete
20160819-171446 Info : Calling SLOB with 032 workers
20160819-172023 Info : Completed SLOB run for 032 workers: Tm 301
20160819-172023 Info : SLOB Harness is 11% complete
20160819-172023 Info : Calling SLOB with 034 workers
20160819-172601 Info : Completed SLOB run for 034 workers: Tm 301
20160819-172601 Info : SLOB Harness is 12% complete
20160819-172601 Info : Calling SLOB with 036 workers
20160819-173139 Info : Completed SLOB run for 036 workers: Tm 302
20160819-173139 Info : SLOB Harness is 12% complete
20160819-173139 Info : Calling SLOB with 038 workers
20160819-173719 Info : Completed SLOB run for 038 workers: Tm 302
20160819-173719 Info : SLOB Harness is 13% complete
20160819-173719 Info : Calling SLOB with 040 workers
20160819-174300 Info : Completed SLOB run for 040 workers: Tm 301
20160819-174300 Info : SLOB Harness is 14% complete
20160819-174300 Info : Calling SLOB with 042 workers
20160819-174845 Info : Completed SLOB run for 042 workers: Tm 301
20160819-174845 Info : SLOB Harness is 14% complete
20160819-174845 Info : Calling SLOB with 044 workers
20160819-175428 Info : Completed SLOB run for 044 workers: Tm 301
20160819-175428 Info : SLOB Harness is 15% complete
20160819-175428 Info : Calling SLOB with 046 workers
20160819-180011 Info : Completed SLOB run for 046 workers: Tm 302
20160819-180011 Info : SLOB Harness is 16% complete
20160819-180011 Info : Calling SLOB with 048 workers
20160819-180556 Info : Completed SLOB run for 048 workers: Tm 302
20160819-180556 Info : SLOB Harness is 16% complete
20160819-180556 Info : Calling SLOB with 052 workers
20160819-181145 Info : Completed SLOB run for 052 workers: Tm 301
20160819-181145 Info : SLOB Harness is 17% complete
20160819-181145 Info : Calling SLOB with 056 workers