Firewire / OCFS / RAC

Disclaimer: This document describes a setup for use as TEST / DEMO / DEVELOP environments and is not certified for production use.

How to set up a multi node RAC system on Linux using off the shelf hardware and firewire disks as shared storage.

document sample configuration setup

node1
AMD athlon 900Mhz
1024Mb RAM
6gb local disk
1x firewire card (TI chipset) ($25 cost)
2 network cards (only 1 will be used for this setup)

node2
Celeron 1Ghz
256Mb RAM
10gb local disk
1x firewire card (TI chipset)
2 network cards (only 1 will be used for this setup)

shared storage
1x 120Gb Western Digital External Firewire (ieee1394) disk ($300)

note : it is also possible to use a standard laptop and pcmcia firewire cards.
The above setup total cost was $1500.

Steps to get the system up and running in RAC configuration with 9.2.0.1 Oracle for Linux IA32

- Install operating system

Download/get/buy the Red Hat Advanced Server 2.1 distribution (or your own favorite distribution that will work with Oracle 9.2.0.1)

Install linux on the local harddisk (approx 25 minutes per node)

Configure the network as usual
sample :
hostname : node1
ip address : 192.168.1.50

hostname: node2
ip address : 192.168.1.51

- Install Oracle

on both nodes :

create a new user Oracle (group dba or oracle doesn't matter) on both machines /home/Oracle - this is on the local hard disk.
Download/get the Oracle for linux 9.2.0.1 CD's
mount the first CD and start the Oracle installer.

choose Cluster manager as the first product to install, and install it with only it's own nodename as public and private node for now.
this will take a minute or so, it's only a few Mb in size.

exit the installer and restart (runInstaller)
choose the database installation option and do a full software only installation (standard should also be ok)

- Configure Firewire / IEEE1394

go to http://otn.Oracle.com -> lnux technology center -> community code -> linux projects -> linux1394
download the fw-test-kernel-2.4.18-image.tar.gz file
on each node, as user root do
cd /
tar zxvf <location>/fw-test-kernel-2.4.18-image.tar.gz
modify /etc/grub.conf (or /etc/lilo.conf if you do not use grub as bootloader)
add something like

title Firewire Kernel (2.4.18)
        root (hd0,0)
        kernel /vmlinuz-2.4.18 ro root=/dev/hda3

reboot the system using this kernel on both nodes.

to make life easy the following should also be done :
edit /etc/modules.conf on both nodes and add

options sbp2 sbp2_exclusive_login=0
post-install sbp2 insmod sd_mod
post-remove sbp2 rmmod sd_mod

every time at bootup of the system load the firewire drivers on both nodes
eg

modprobe ohci1394
modprobe sbp2

if you do dmesg, you should see log messages along the lines of

Attached scsi disk sda at scsi0, channel 0, id 0, lun 0
SCSI device sda: 35239680 512-byte hdwr sectors (18043 MB)
sda: sda1 sda2 sda3

in this case that means there is an 18Gb disk with 3 partitions

the first time the firewire disk is configured/used, run fdisk on one node
and partition the disk as you like. (If both nodes had the modules loaded while running fdisk, I would suggest doing a reboot of the other system or unloading and reloading all the firewire and scsi modules to make sure the new partition table is loaded)

- Configuring OCFS

Download precompiled modules from http://otn.Oracle.com -> linux technology center -> community code -> linux -> ocfs
the file fw-kernel-ocfs.tar.gz
untar the file as root on both nodes.

- format the filesystem on the shared disk on one node :
use ocfsformat
eg
ocfsformat -f -l /dev/sda1 -c 128 -v ocfsvol -m /ocfs -n node1 -u 1011 -p 755 -g 1011
1011 is the uid and gid of the Oracle account and 755 is the directory permission
/dev/sda1 is the partition that will be used
-c 128 means we use a 128Kb clustersize, this can be 4,8,16,32,128,256,512,1024Kb

on both nodes make a directory /ocfs as user root (mountpoint)

- configuring/loading the kernel module
on each node create a config file /etc/ocfs.conf
eg
ipcdlm:
        ip_address = 192.168.1.50 (on node2 put 192.168.1.51)
        ip_port = 9999
        subnet_mask = 255.255.252.0
        type = udp
        hostname = node1 (on node2 put node2)
        active = yes

make sure that each node has the correct values

do insmod ocfs.o name=<nodename>
eg
insmod /root/ocfs.o name=node1 (this has to happen at every bootup of the system)

to mount, enter the following on each node :

mount -t ocfs /dev/sda1 /ocfs

you should now have a shared filesystem mounted on the nodes, owned by user Oracle.

- Configuring Oracle RAC

we now have a shared filesystem which we will use for the datafiles/logfiles and controlfiles, the modules are loaded,
Oracle is installed, so the following steps will configure the cluster manager software, create a database and we are all set.
to make life easier, it would be helpful to create a little script ENV in the home directory of the Oracle software that sets up the environment, eg

export ORACLE_HOME=/home/Oracle/9i
export ORACLE_SID=node1                                                      <- on node2 changes this to node2
export LD_LIBRARY_PATH=/home/Oracle/9i/lib
export PATH=$ORACLE_HOME/bin:$PATH

and source this file when you log in or from .login scripts as root or Oracle ( . ENV)

- cluster manager software configuration

as user Oracle or root. go to $ORACLE_HOME/oracm/admin
on each node create or change the following files

(1) cmcfg.ora

HeartBeat=10000
ClusterName=Oracle Cluster Manager, version 9i
PollInterval=300
MissCount=20
PrivateNodeNames=node1 node2
PublicNodeNames=node1 node2
ServicePort=9998
WatchdogSafetyMargin=3000
WatchdogTimerMargin=6000
HostName=node1 <- replace this with node2 on the other node

(2) ocmargs.ora
# Sample configuration file $ORACLE_HOME/oracm/admin/ocmargs.ora
watchdogd -d /dev/null -l 0
oracm /a:0

before starting the database, make sure the clustermanager software is running, (this can be put in the rc scripts if you wish)
as user root , on each node :
- setup the Oracle environment variables (source ENV)
cd $ORACLE_HOME/oracm/bin
./ocmstart.sh

- Configure Oracle init.ora and create a database

sample init.ora, in $ORACLE_HOME/dbs on each node, create a initnode1.ora and initnode2.ora (respectively)

db_block_size=4096
db_cache_size=37108864
cluster_database_instances=2 <- number of total nodes you want, 2 , 4 ...
filesystemio_options="directIO"
open_cursors=300
background_dump_dest=/home/Oracle/9i/admin/node1/bdump <- change node1 to node2
core_dump_dest=/home/Oracle/9i/admin/node1/cdump <- change node1 to node2
timed_statistics=TRUE
user_dump_dest=/home/Oracle/9i/admin/node1/udump <- change node1 to node2
db_domain=us.Oracle.com
remote_login_passwordfile=EXCLUSIVE
control_files=("/ocfs/ctrl")
dispatchers="(PROTOCOL=TCP)(SER=MODOSE)", "(PROTOCOL=TCP)(PRE=Oracle.aurora.server.GiopServer)", "(PROTOCOL=TCP)(PRE=Oracle.aurora.server.SGiopServer)", "(PROTOCOL=TCP)"
compatible=9.0.0
db_name=bob
java_pool_size=12428800
large_pool_size=10485760
shared_pool_size=47440512
processes=150
fast_start_mttr_target=300
resource_manager_plan=SYSTEM_PLAN
sort_area_size=524288
undo_management=AUTO
undo_tablespace=UNDO_ts <- change to undo_ts2 on node2
cluster_database=true;
instance_name=node1 <- change to node2 on node2
instance_number=1 <- change to 2 on node2
thread=1 <- change to 2 on node2

create the directories for the logfiles
eg

cd $ORACLE_HOME
mkdir admin ; cd admin ; mkdir node1 ; cd node1 ; mkdir udump ; mkdir bdump ; mkdir cdump
(do the same on node2, replacing node1 with node2)

make a link for the Oracle password file, this file does not yet have to exist. (on each node)

cd $ORACLE_HOME/dbs
ln -sf /ocfs/orapw orapw

Now we have the setup, next up is creating a database, I use the following shellscript to make life easier.
(create.sh) on either node

cd /home/Oracle/9i
. ENV
rm -f /ocfs/orapw
rm -f /ocfs/ctrl
orapwd file=/ocfs/orapw password=manager entries=5

sqlplus << EOF
/ as sysdba

startup nomount
create database bob
        maxinstances 5
        maxlogfiles 10
        character set "we8iso8859p1"
        datafile '/ocfs/system.dbf' size 400m reuse
        default temporary tablespace tempts tempfile '/ocfs/temp.dbf'
                size 15m reuse
        undo tablespace undo_ts datafile '/ocfs/undo.dbf' size 40m reuse
        logfile
                 '/ocfs/log1a.dbf' size 15m reuse,
                 '/ocfs/log1b.dbf' size 15m reuse;
@?/rdbms/admin/catalog.sql
@?/rdbms/admin/catproc.sql

create undo tablespace undo_ts2 datafile '/ocfs/undo2.dbf' size 40m reuse;
alter database add logfile thread 2 '/ocfs/log2a.dbf' size 15m reuse;
alter database add logfile thread 2 '/ocfs/log2b.dbf' size 15m reuse;
alter database enable thread 2;
shutdown immediate
exit
EOF

as user Oracle, run this script, if all goes well, it should create a database, add a second undo tablespace, add a second log thread and enable the thread. Run this on node1 only once.

the database is shut down, now you can start it up on both nodes
just do

sqlplus '/ as sysdba'
startup

you can configure your Oracle listener as normal on both nodes ($ORACLE_HOME/network/admin/listener.ora) and start this.

That's it ! Good luck.

- Extra
you can go up to 4 nodes in this setup, if you have more than 2 nodes it is required to purchase a firewire hub, you can get these for cheap ($70), a 4 or 5 port hub. just plug every linux box into the hub and plug the firewire disk in seperately. without a hub there is not enough power for the total cablelength on the bus. Configure nodes 3 and/or 4 the same way, follow the same steps, and make sure you create an extra set of logfiles and undo tablespace.