Enterprise Manager 11g Installation

People who care to surf around the posts on this blog have found out that step-by-steps, howtos, and installation guides are not the point because you have a bunch of those in Oracle official docs and tons of other blogs.
Having said that, last (and first!) time I’ve installed the all new Enterprise Manager 11g, I’ve captured the screen steps and I thought: “Why don’t fill the internet with mode stuff about stuff for people interested in stuff?”. So here it is.

The main changes I’ve found in the installation process were:

1. Look and feel like Oracle Database 11gR2
2. The installation process doesn’t make you create another helplessly old database instance: you provide a connection string for a previous one (10.2.0.4 or above). I’ve used my general purpose 11gR2 database and it went fine.
3. This one is a bit of a pain: it only works with Weblogic Server 10.3.2. You don’t have to create any domains, just install it and EM 11g will recognize your WLS 10.3.2


Manually Editing UDMs without EM

In case you use Oracle 10g in Enterprise Linux 5, you might run into a lot of weird issues. One of those is the inability of Enterprise Manager (EM) to collect host metrics. For EM the HostAvailability is zero percent. This will have a snowball effect on your EM repository.

This is a known bug (Bug 7284070), just because the syntax of the “tail” command has changed. How stupid can these errors be?

The workaround suggested by Oracle is to edit “$ORACLE_HOME/sysman/admin/scripts/allprocs.sh” and substitute the “tail +2″ command with “tail -n +2″.

But wait up! This is not enough! You might want to edit other perl scripts like $ORACLE_HOME/sysman/admin/scripts/osCpuUsage.pl and make the same syntax change in the “tail” command.

But if you haven’t reached this level of “troubleshiting” yet and you just have defined a nice set of User-Defined Metrics (UDMs), you will find yourself strangled by this “tail” issue. Since EM can’t know if the Host is Available, you will be able to edit or remove any UDM !!! The worst part is that I’m talking about Database UDMs and not Host UDMs, but still EM won’t let you alter any of them giving you back the following message:

You cannot edit User Defined Metric MY_UDM_01 because host myserver is currently unavailable. Try again later when it becomes available.

To take you out of your misery just go into SYSMAN schema and edit the MGMT_METRIC_THRESHOLDS table so you can alter the values of your warning and critical thresholds freely. But messing up directly with these tables should be avoided, so the best is trying to fix the tail issue first, but if everything else fails, go on and “hammer” those SYSMAN tables!

I/O Calibration in Oracle 11g

Very cool feature. For those who knew or used ORION probably it’s no big surprise, but still it’s very usefull.

You need to

1) alter one parameter: SQL>alter system set filesystemio_options=ASYNCH scope=spfile;  (needs a DB bounce)
2) Check another parameter: SQL> show parameter disk_asynch_io (if it’s TRUE, it’s OK)
3) Decide on the latency (in miliseconds) you will tolerate has the maximum for your storage system
4) Know how many physical disks you have
5) If using Linux and if  “cat /proc/sys/fs/aio-nr” is too close to cat /proc/sys/fs/aio-max-nr” edit (as root) the “/etc/sysctl.conf” file and increase the max parameter:

# vi/etc/sysctl.conf

add the following line: fs.aio-max-nr = 131072

This should be suficient for most systems, but if the value given by cat /proc/sys/fs/aio-nr” is too close to 131072, then you can increase the value of “fs.aio-max-nr” in “/etc/sysctl.conf”  file.

After this you can calibrate your I/O. You will need to do this in order to have realistic metric thresholds for you I/O.

To calibrate you have two ways:

* Calibrate with Enterprise Manager (EM)  interface
* Calibrate using the PL/SQL API

Calibrating your I/O with EM

This is supposed to be easy, but after you submit your calibration process you might wonder where it is. It’s the EM way! To get lost, or loose your context! Here are the steps to submit the process:

1) Go to Performance tab, and underneath the “Average Active Sessions” graphic click the “I/O” tab
2) After this you will need to page down again, and find that the I/O tab has a “I/O Calibration” button. Press it.
3) Now you just have to fill two text boxes: number of disks, and expected latency. Click Submit. You’re done.
4) Now go back and click the Server tab, and under “Oracle Scheduler” click “Jobs”, and then click the “Running” tab. You’ll find your Calibration Job here.

Calibrating your I/O with the PL/SQL API

Inside the DBMS_RESOURCE_MANAGER package there’s a procedure called CALIBRATE_IO that you must invoke with two variables going in:

SQL> desc DBMS_RESOURCE_MANAGER
PROCEDURE CALIBRATE_IO
Argument Name                  Type                    In/Out Default?
------------------------------ ----------------------- ------ --------
NUM_PHYSICAL_DISKS             BINARY_INTEGER          IN     DEFAULT
MAX_LATENCY                    BINARY_INTEGER          IN     DEFAULT
MAX_IOPS                       BINARY_INTEGER          OUT
MAX_MBPS                       BINARY_INTEGER          OUT
ACTUAL_LATENCY                 BINARY_INTEGER          OUT

Checking the Status and AfterMath of Calibration

After you submit your Calibration process either by EM or PL/SQL you can keep up to date of its status by querying the following view:

SQL> select * from v$io_calibration_status;

STATUS
-------------
CALIBRATION_TIME
---------------------------------------------------------------------------
IN PROGRESS

After a while the same query will give back a finishing status:

SQL> select * from v$io_calibration_status;

STATUS
-------------
CALIBRATION_TIME
---------------------------------------------------------------------------
READY
23-MAY-09 05.57.33.317 PM

And now it’s the time to check on the results, either by clicking again on the “I/O Calibration” button on EM and checking the “Existing Calibration Results” or by issuing the following query:

SQL> select * from dba_rsrc_io_calibrate

START_TIME
---------------------------------------------------------------------------
END_TIME
---------------------------------------------------------------------------
 MAX_IOPS   MAX_MBPS  MAX_PMBPS    LATENCY NUM_PHYSICAL_DISKS
---------- ---------- ---------- ---------- ------------------
23-MAY-09 05.37.25.117503 PM
23-MAY-09 05.57.33.316982 PM
 85         14         15         89                  1

All About Enterprise Manager Management

Well in fact it’s not all but just the most popular commands and concepts that I’ve come around to notice that been lacking on most of the customer sites I visit.

First the concepts:

Concept number 1: The Listener Dependency

Enterprise Manager it’s a JDBC dependent application that needs TNS port and hostname at it’s creation. So if you change the hostname or the listener, just wave goodbye to your current EM configuration. You have to deconfigure and configure again.

An example of this is when you choose to create EM at the database creation it’s useless if you haven’t already started up at least one TNS listener.

Concept number 2: The Database Framework of EM

At the service level EM it’s a java application that you either configure or deconfigure. But at the database level, EM has two components: the repository and the communication layer. For the repository there is a user called SYSMAN that has it’s objects stored in the SYSAUX tablespace; and for the communication layer there is the user DBSNMP. There are other objects linked to EM in the database:

    • View MGMT_VIEW

    • Role MGMT_USER

    • Public Synonyns MGMT_TARGET_BLACKOUTS and SETEMVIEWUSERCONTEXT

Since 10gR2 you must always use Enterprise Manager Configuration Assistant (EMCA) instead of the old tools like “RepManager”.

Statement Group 1: Configuration of the Service Layer of EM

To configure the service layer:

	$ emca -config dbcontrol db 

To deconfigure the service layer:

	$ emca -deconfig dbcontrol db 

To change the HTTP port:

$ emca -reconfig ports -DBCONTROL_HTTP_PORT 5500

To secure the HTTP connection (standalone):

	$ emctl secure dbconsole

To secure the HTTP connection (grid control):

	$ emctl secure oms

Statement Group 2: Configuration of the Database Layer of EM

Create the objects in the database (SYSMAN objects, etc):

	$ emca -repos create

Drop the database repository (maybe to re-create it afterwards):

	$ emca -repos drop

Statement Group 3: All-in-one statements

Take it all out:

	$ emca -deconfig dbcontrol db -repos drop

Or if you don’t want to be bored with questions:

$ emca -deconfig dbcontrol db -repos drop -SID my_sid -PORT 1521 -SYSMAN_PWD my_sysman_pass -SYS_PWD  my_sys_pass

Put it all back in with fewer questions:

$ emca -config dbcontrol db -repos create -SID my_sid -PORT 1521 -SYSMAN_PWD my_sysman_pass -SYS_PWD  my_sys_pass -DBSNMP_PWD my_dbsnmp_pass

Put it all back in the more interactive way:

	$ emca -config dbcontrol db -repos create

EM fails when trying to administer listeners

More than once I’ve tried to make some listener configuration through Enterprise Manager Console Control link called Net Services Administration and this error hits me:

"Error in getting data for creating new listener. Check if the listener.ora 
location '$ORACLE_HOME/network/admin' is a valid one."

Who the hell makes these messages? Who even told them that I want to create a listener?

It turned out this BUG is both on Windows and Linux and that it’s resolution is the typical black magic of 90% of Metalink solving notes.

Attention that some internet sites tell you to comment a line somewhere. Do n0t do it! The right recipe comes from the factory:

Metalink Note:   Doc ID:  406453.1

Subject:  Unable to Administer Listeners From DBcontrol (DBconsole)

Applies to: Enterprise Manager for RDBMS – Version: 10.2 to 10.2
This problem can occur on any platform.

Symptoms
Unable to Administer Listeners from DBcontrol Getting the Following Error:

Error Message

Error in getting data for creating new listener. Check if the listener.ora location”/u01/app/oracle/product/10.2.0/db_1/network/admin” is a valid one.

Cause
Removing SERVER from the comma separated list of values for “INSTALLEDCOMPONENTS” parameter from

$ORACLE_HOME/network/tools/NetProperties file.

IF in the list of installed components “SERVER” is not found, then we do not even try to read “listener.ora” and the error reported above is thrown.

Solution
1. ADD “SERVER” to “INSTALLEDCOMPONENTS” in NetProperties file In the comma separated list of values for

“INSTALLEDCOMPONENTS”, add SERVER as below :

INSTALLEDCOMPONENTS=ORACLENET,SERVER

2. Then logout of EM and login again, listener admin should work.

References
@ Bug 4428659 – EMDBGC ::ERROR IN GETTING DATA FOR CREATING NEW LISTENER.CHECK IF IT IS A VALID
@ Bug 5664957 – GC10203 TC10: NET SERVICE ADMIN: FAILS TO ENTER LISTENER ADMIN PAGE

Save EM Repository information when upgrading to 11g

Saving your EM information

This is something taken from the Oracle Documentation and something that can assault you in case you wonder: “If I loose my Enterprise Manager intell I’m doomed!”. Well for those who can’t aford to loose it here’s what Oracle tells you to do:

The emdwgrd utility resides in the ORACLE_HOME/bin directory in the Oracle Database 11g Release 1 (11.1) home. Before running the utility you must install Oracle Database 11g Release 1 (11.1) and invoke the script from the Oracle Database 11g Release 1 (11.1) home. The emdwgrd utility, however, requires that you set ORACLE_HOME to the old Oracle home.

The following procedure is for Linux and UNIX. To run it on Windows, simply substitute emdwgrd.bat for emdwgrd.
Follow these steps to save your Database Control files and data:

1.Install Oracle Database 11g Release 1 (11.1).
2.Set ORACLE_HOME to your old Oracle home.
3.Set ORACLE_SID to the SID of the database being upgraded.
4.Set PATH, LD_LIBRARY_PATH and SHLIB_PATH to point to the Oracle home from which the database is being upgraded.
5.Go to the Oracle Database 11g Release 1 (11.1) home.
6.Execute one of the following:

For a single-instance database, run the following command, where old_SID is the SID of the database being upgraded and save_directory is the path to the storage place you have chosen for your Database Control files and data:

emdwgrd -save -sid old_SID -path save_directory