ACFS and Oracle VM: A silent revolution in storage management

(11gR2 only)

Slice and dice physical machines into small, much manageable, much performant pieces, is only one side behind virtualization. The other is: storage virtualization. In Oracle VM you can either add a virtual disk (dedicated to that VM) or a shared disk (that you can share across VMs).

After the virtual disks are visible to the VM you will treat it like a LUN and use the block device, build a raw or even create a partition and format it. This process takes forever to complete in big files (above 100 GB or so). If you want to create 10 mount points in a virtual machine, you can create 10 virtual disks, but they are not extensible. If something goes wrong, you have to unmount, delete the virtual disk and but it back on again, this time bigger. It’s a huge waste of space.

So what is the alternative?

Create the same 10 virtual disks, use them all in an ASM instance and build a disk group (with or without redundancy).

Imagine that these 10 virtual disks have 10 gigs each. With the new 11gR2 ASM capabilities you can use these 100 gigs in two ways: store database or clusterware files, or even create dynamic volumes that you can mount (with the “-t acfs” switch).

3 uses for the same 100 gigs:

- Database files

- Clusterware Files

- Dynamic Volumes

I can use 80 gigs for database files, and 10 gigs to mount for Oracle binaries. If by some lack of chance my 10 gigs are not enough, I can expand them, instead of having to painfully wait for another virtual disk that takes forever to be active on the VM.

Bottomline?

If you have a 700GB space for your VM on Oracle Virtual Server (OVS)  give it all to ASM 11gR2 and he will help you out on all kinds of needs you have.

LMC.

Putting ACFS to work: how to create a mount point in the new Oracle File System

This post is a translation of another one that I’ve originally wrote in portuguese. I’ve decided to share with the international Oracle community through my weblog.

ACFS stands for ASM Cluster File System. ACFS is a new file system created by Oracle that is created on top of an ASM Disk Group. The architecture is very simple and these mount points can contain both general purpose files (except  db files) or Oracle Home binaries, which is very good for sharing binaries.

pilha_asm

There are three ways to setup ACFS:

- Web (with Enterprise Manager)
- With  ASMCA (ASM Configuration Assistant) a graphic tool
- or through command line statements
We will show you examples using ASMCA because it’s nicer ;-)
Here’s the step-by-step:
1 – We had two 1GB LUNs/Disks for this example with which two raw devices were created:
[oracle@node1 ~]$ cat /etc/rc.local
#!/bin/sh
#
# This script will be executed *after* all the other init scripts.
# You can put your own initialization stuff in here if you don’t
# want to do the full Sys V style init stuff.

touch /var/lock/subsys/local

raw /dev/raw/raw1 /dev/sdb1
raw /dev/raw/raw2 /dev/sdc1
chmod 666 /dev/raw/raw1
chmod 666 /dev/raw/raw2
chown oracle:oinstall /dev/raw/raw1
chown oracle:oinstall /dev/raw/raw2

2 – Now it’s time to call ASMCA:

[oracle@node1 ~]$ . oraenv
ORACLE_SID = [+ASM] ?
ORACLE_HOME = [/home/oracle] ? /u01/11.2.0/grid
The Oracle base for ORACLE_HOME=/u01/11.2.0/grid is /u01/oracle
[oracle@node1 ~]$ asmca &

3 – Create one disk group (DG) with those two raw devices:
asmca - 01b - criar DG DATA
asmca - 01c - DG DATA criado
4 – We then create a ASM volume on top of this DG, naming it “volumezinho” (we show here several ways of doing so):
02_volumezinho
02a_volumezinho_sql
asmca - 02 - criar volumezinho
There are a few advanced options in ASMCA when you create a volume:
asmca - 02b - criar volumezinho advanced
5 – Create a directoy on the ext3 Linux file system so when can mount there the device created by “volumezinho”
[root@node1 ~]# mkdir /u02/oradata/acfs
[root@node1 ~]# chown oracle:oinstall /u02/oradata/acfs
6 – Now let’s create the ACFS mount point. We can choose between one of these two options:
asmca - 04 - acfs generico
But still you have to mount it manually:
asmca - 05 - acfs montado a mao

Et voilá!

Now we can go to the directory mounted on the ASM volume formated as ACFS, and create some bogus text file:
[oracle@node1 ~]$ df -m Filesystem           1M-blocks      Used Available Use% Mounted on /dev/mapper/VolGroup00-LogVol00 47551     16389     28708  37% / /dev/sda1                   99        11        83  12% /boot tmpfs                     1014       662       352  66% /dev/shm /dev/sdd1                  512       351       162  69% /u02/oradata/cluster-lmc /dev/asm/volumezinho-486 512       101       412  20% /u02/oradata/acfs/data_volumezinho [oracle@node1 ~]$ cd /u02/oradata/acfs/data_volumezinho [oracle@node1 data_volumezinho]$ ll total 68 -rw-r--r-- 1 oracle oinstall    89 Sep 24 14:33 ficheiro.txt drwx------ 2 root   root     65536 Sep 24 14:20 lost+found [oracle@node1 data_volumezinho]$ cat ficheiro.txt
Primeiro ficheiro de texto criado num volume ASM no qual foi montado um ACFS!!!
Yupii!!
[oracle@node1 data_volumezinho]$
Technology is an amazing thing. So many twists and turns to end up in the same place ...

Why does ASM Configuration Assistant can’t start my ASM instance?

[oracle@centos5 ~]$ asmca -silent -startup -configureParameter -param coiso=valor -sysAsmPassword Oracle.123
ASM failed to start. Check /oracle/cfgtoollogs/asmca/asmca-0909103PM0424.log for details.

[oracle@centos5 ~]$ asmca -silent -startup -configureParameter -param coiso=valor -sysAsmPassword Oracle.123

ASM failed to start. Check /oracle/cfgtoollogs/asmca/asmca-0909103PM0424.log for details.

And the log says:
(..) [UsmcaLogger.logException:171]  SEVERE:method oracle.sysman.assistants.usmca.backend.USMInstance:checkAndStartupInstance
(..) [UsmcaLogger.logException:172]  ORA-01034: ORACLE not available
(..) [UsmcaLogger.logException:173]  oracle.sysman.assistants.util.sqlEngine.SQLFatalErrorException: ORA-01034: ORACLE not available
Why??? And it goes on:
oracle.sysman.assistants.util.sqlEngine.SQLEngine.executeImpl(SQLEngine.java:1650)
oracle.sysman.assistants.util.sqlEngine.SQLEngine.executeQuery(SQLEngine.java:823)
oracle.sysman.assistants.usmca.backend.USMInstance.checkAndStartupInstance(USMInstance.java:602)
oracle.sysman.assistants.usmca.backend.USMInstance.doPreliminaryChecks(USMInstance.java:2597)
oracle.sysman.assistants.usmca.backend.USMInstance.configureASMParameters(USMInstance.java:1689)
oracle.sysman.assistants.usmca.service.UsmcaService.configureASMParameters(UsmcaService.java:252)
oracle.sysman.assistants.usmca.model.UsmcaModel.performConfigureASMParameters(UsmcaModel.java:746)
oracle.sysman.assistants.usmca.model.UsmcaModel.performOperation(UsmcaModel.java:512)
oracle.sysman.assistants.usmca.Usmca.execute(Usmca.java:156)
oracle.sysman.assistants.usmca.Usmca.main(Usmca.java:333)
(..) [UsmcaLogger.logExit:122]  Exiting oracle.sysman.assistants.usmca.backend.USMInstance Method : checkAndStartupInstance
(..) [UsmcaLogger.logInfo:141]  ASM failed to start. Check /oracle/cfgtoollogs/asmca/asmca-0909103PM0424.log for details.
(..) [UsmcaLogger.logExit:122]  Exiting oracle.sysman.assistants.usmca.model.UsmcaModel Method : performOperation
Why does the script tells me to read … the script itself?
Why do I feel like a guinea-pig?
I’ve logged and SR on Metalink.

Check LUNs in EMC

The powerpath command line to check the visible LUNs at your server is very simple:

as root:  /sbin/powermt display dev=all

as sudoer:  sudo /sbin/powermt display dev=all

And you’ll get an output like this:

Pseudo name=emcpowera
CLARiiON ID=CKM00084501075 [iddesc]
Logical device ID=6006016025DE2200F0AA849CAD41DE31 [LUN 1]
state=alive; policy=mypol; priority=0; queued-IOs=129
Owner: default=SP B, current=SP B Array failover mode: 1
==============================================================================
—————- Host ————— – Stor – — I/O Path – — Stats —
### HW Path I/O Paths Interf. Mode State Q-IOs Errors
==============================================================================
1 qla2xxx sdc SP A9 active alive 0 0
3 qla2xxx sdm SP B9 active alive 129 0

This way you can also check if multipath is in use or not.

ASM Best Practices

ASM is a world filled with myths. The only explanation I can find is ignorance. Ignorance may be  bliss in some contexts, but here is a myth creator! So I decided to share with you all the best document I could find about ASM one-o-one.

You download the document here and if you have some doubts feel free to shoot them at me:

asm_best_practices

Enjoy.

LMC.

Raw my LUN up with ASM

Scenario: you have a fibre channel and a couple of LUNs and you want to setup an ASM instance using those LUNs. Forget about ASMLib, just go with the raw (not the Smackdown ;-) ). We’ll use EMC as an example, but you can use any other manufacturer:

1. Check if you see the LUNs:

[oracle@Dgrnbdpitij06-old100 ~]$  ll /dev/emcpower*
crw-r–r– 1 root root  10,  62 Jul 15 05:18 /dev/emcpower
brw-r—– 1 root disk 120,   0 Jul 15 05:18 /dev/emcpowera
brw-r—– 1 root disk 120,   1 Jul 15 05:18 /dev/emcpowera1
brw-r—– 1 root disk 120,  16 Jul 15 09:11 /dev/emcpowerb
brw-r—– 1 root disk 120,  17 Jul 15 05:18 /dev/emcpowerb1
brw-r—– 1 root disk 120,  32 Jul 15 11:59 /dev/emcpowerc
brw-r—– 1 root disk 120,  33 Jul 15 05:18 /dev/emcpowerc1

$  ll /dev/emcpower*

crw-r–r– 1 root root  10,  62 Jul 15 05:18 /dev/emcpower

brw-r—– 1 root disk 120,   0 Jul 15 05:18 /dev/emcpowera

brw-r—– 1 root disk 120,  16 Jul 15 09:11 /dev/emcpowerb

brw-r—– 1 root disk 120,  32 Jul 15 11:59 /dev/emcpowerc

Nice. If you don’t, probably you’ve just created them using Navishpere and now either you need resync or just a good plain reboot.

2. No partitions please. Just raw-ger that!

$ sudo raw /dev/raw/raw1 /dev/emcpowerb

Password:
/dev/raw/raw1:  bound to major 120, minor 16

We’re using just one device (LUN) called “emcpowerb” whereas the letter “b” means it’s the second LUN created at the fibre.
You can repeat the process, creating as many raw devices as you would want for the available LUNs.

3. Now setup your perms:

$ sudo chown oracle:oinstall /dev/raw/raw1
$ sudo chmod 666 /dev/raw/raw1

$ ll /dev/raw/r*

crw-rw-rw- 1 oracle oinstall 162, 1 Jul 16 09:33 /dev/raw/raw1

4. Create (if you haven't already) an ASM instance:

There are two methods  (manual or DBCA), but I’ll recomend using DBCA just because it’s faster:

4.1. Startup the CRS Daemon

$ sudo /u01/app/oracle/product/database/10.2.0/bin/localconfig add

/etc/oracle does not exist. Creating it now.
Successfully accumulated necessary OCR keys.
Creating OCR keys for user ‘root’, privgrp ‘root’..

Operation successful.

Configuration for local CSS has been initialized
Adding to inittab
Startup will be queued to init within 30 seconds.
Checking the status of new Oracle init process…
Expecting the CRS daemons to be up within 600 seconds.
CSS is active on these nodes.

myserver-01

CSS is active on all nodes.
Oracle CSS service is installed and running under init(1M)

4.2. Get DBCA going

If you’re at the server console just type (as user “oracle”):

$ dbca

But you’re doing it remotly, just startup an X Emulator on your PC and type the following before invoking “dbca”:

$ export DISPLAY=<your_PC_ip_address>:0.0

After typing dbca you should see the following Welcome Screen:

asm1

Hit “Next” to go to the next screen:

asm2

In this screen choose the last option “Configure Automatic Storage Management” and hit “Next”:

asm3

In this screen you have to do 3 things:

- Type in an ASM password two times
- If you have a Grid Agent installed decide you would want to add this ASM instance as a target or not
- Hit the “ASM Parameters” button

asm4

At this screen type on the column “Value” in the second row the following text: “/dev/raw/*” (like on the image).
If you’re keyboard changed during editing this field just remember the / is where the “-” is, the delete is the back key and that the “*” is done with SHIFT+8.

Hit button “Close” and then the “Next” button:

asm5

Acklowledge this message and wait. In the meanwhile the following message will be displayed:

asm6

If the ASM Instance creation is successfull the following screen will appear:

asm7

Now you’ve successfully seted up ASM but it’s worth nothing unless you create diskgroups. To create the first one just hit the “Create New” button:

asm8

At this screen you will see right away the raw device because of the ASM parameter of the Disk Discovery Path was previously changed to “/dev/raw/*”. If you don’t see any raw devices at this screen try change the parameter by hitting the “Change Disk Discovery Path” button. If still you don’t see nothing, its because you have skipped or mistyped the “chown” or “chmod” commands previously. Check if the raw device has the permission stated before when we listed the raw devices. If there’s something you need to do again at the permissions level, exit DBCA, do whatever step you’ve skipped, and then invoke DBCA again, choose the last option and you’ll end up in the Disk Group creation screen again.

If you see the raw device use it to create a disk group called “TEST” and don’t forget to select “External” on the Redundancy area, since you only have one raw device you can’t produce redundancy at all!

Hit the “OK” button and you should see something like this:

asm9

And then back to the Disk Group creation screen:

asm91

You’re done! Exit and log on the database to test:

$ sqlplus / as sysdba

SQL> create tablespace my_asm_tbs datafile ‘+TEST’ size 5m;

Tablespace Created.

Done!

Now remember that you should create diskgroup with lots of disks instead of lots of disk groups with few disks.

Enjoy ASM!

Drop me only if I’m Mounted: ASM trick

Actually this is no trick. It’s a missconception. Oracle has created the ASM Disk Groups has entities that only can be the target of operations if they are mounted. So if you want to drop a diskgroup, you have to have it mounted, otherwise Oracle ASM won’t let you do it:

SQL> alter diskgroup data dismount;
Diskgroup altered.
SQL> drop diskgroup data;
drop diskgroup data
*
ERROR at line 1:
ORA-15039: diskgroup not dropped
ORA-15001: diskgroup "DATA" does not exist or is not mounted

You can see that the message is same if the diskgroup is not mounted or does not exist. For the ASM kernel it’s all the same, if you have dismounted or droped it, you can’t remove it.

Wrong Oracle Error Message in an ASM 10g instance when queriyng Data Dictionary V$ views

ASM is one of the best technologies Oracle has released in years. It’s simple and powerfull. But at a 10.2.0.4 ASM instance that has just been pulled up, querying the wrong V$ view should give you the message that view or table does not exist! Obvious isn’t it? No! Folks at Oracle Corporation decided to save on creating obvious error messages and turn it more into a black magic issue.

So if you query a wrong V$ at the Data Dictionary the message you’ll get might send you in the wrong direction:

[oracle@linux ~]$ . oraenv
ORACLE_SID = [orcl] ? +ASM
[oracle@linux ~]$ sqlplus / as sysdba
(...)
SQL> select name from v$disk_group;
ORA-01219: database not open: queries allowed on fixed table/views only

(ooops looks like I’ve f%ck up something!? …
… stay cool it’s just Oracle trying to mess you up!)

SQL> select name from v$asm_diskgroup;
NAME
-------------------------------------------
DG01
DG02
DG03

We all know the literary meaning of the word “Oracle” but this is a bit too much! This is Lord of the Rings kind of stuff. Well … Oracle Corp has a lot more from where this one came from -;)

ASMlib and Kernel version

Last time I had to install a ASMlib RPMs on my RHEL 4.4 box I found out that my Kernel version was:. 2.6.9-42.0.0.0-1. What on earth do those geeks are thinking to pull up such a version number?

I went to oracle.com’s page for ASMlib and found out that for RHEL 4 there were something like 400 different RPMs.

The information company is Oracle Corporation and their site sucks! How am I suposed to find the right RPM? Couldn’t those nerds made up some simple servlet that I would type my kernel version and a ZIP file would pop up for download? No, it’s too complicated for Oracle that once acused Informix of hiring lawyers instead of programmers. So now I think that Oracle should hire some programmers for their site instead of having fired everyone at Oracle local support.

So the hard time Oracle gave me at their stupid site grew on me the need to find something and since my version was 269-42 when I found it I was thrilled. I took the oracleasm-269-42 RPM along with the oracleasmlib and oracleasm-support one.

But when I’ve tryied to install it I got an error on “/etc/init.d/oracleasm configure” I’ve started a journey no one should want at a Linux box: Not knowing what went wrong, since I was darn sure I had the right stuff.

You know what? I learned the hard way geeks and nerds make stuff for geeks and nerds, and so those 3 zeros after the version number, that had absolutely no meaning for me, turned out to be the key.

So the RHEL install CD #3 had all the RPMs I needed for that stupid version: oracleasm-2.6.9-42.0.0.0-1 ! I had to do was get oracleasmlib RPM from the site and use the kernel and support one from the CD #3 and everything went like a charm.

SYSASM Role

SYSASM role exists to manage ASM instances and avoids overlaping between DBAs and storage administrators

SQL> CONNECT / AS SYSASM

For ASM instances, SYSDBA will be deprecated in the future:

  • Oracle Database 11g Release 1 behaves as in 10g
  • In future releases SYSDBA privileges restricted in ASM instances

ASM Discovery String in Linux

Having trouble finding the ASM disks that you have previously added but neither DBCA nor the ASM disk finder can’t dough them out of wherever they are? And do you use Linux?

If you are using ASMlib disks properly and previously added just use as a Discovery String for your new Disk Group the following string:

/dev/oracleasm/disks/*

Just like that: no quoting. If this doesn’t find your disks probably you haven’t added the disks correctly. Search for another post in this site about how to add a ASMlib disk in Linux.

ASM and Database version compatibility between 10g and 11g

Scenario:
You have two 10g databases in two different Oracle Homes and only one ASM 10g instance (in its own Oracle Home).

You need to:
Upgrade one of the databases to 11g keeping the same ASM

Follow the instructions on this simple document to make sure you don’t blow your storage to pieces:

ASM and Oracle11g version support

Main tags: DBUA, ASM, Oracle 11g, Upgrade