We recently switched one of our client’s WebSphere Application Server and WebSphere Portal based infrastructure to a new physical environment. During our testing we found that we were getting stale connection exceptions in SystemOut.log and in the application logs. On digging a little deeper, our team tracked it down to the fact that the Cisco firewall was dropping our Oracle database connections after they had been inactive for a certain amount of time. When our team discussed the issue with the network team, they were essentially told that all was good with the network infrastructure. The application team tweaked some of the connection pool settings but that only helped alleviate the issue slightly and we started encountering performance issues. Eventually, we figured out a solution that works!

The solution that we ended up with was to work around this issue on the Oracle end. We modified the sqlnet.expire_time parameter in sqlnet.ora on the Oracle 10g server down from the default “30″ (30 minutes) to “1″ (1 minute).

The explanation on why this worked

The parameter SQLNET.EXPIRE_TIME is used to specify the time interval, in minutes, to send a probe to verify that client/server connections are active. If the probe finds a terminated connection, or a connection that is no longer in use, it returns an error, causing the server process to exit. This parameter is primarily intended for the database server so that it can free up server-side resources that are not in use.

A side-effect of the probe is that there is TCP-IP activity between the client and the server each time the probe is executed, and the firewall designates the link as an active one. By lowering the probe interval down to a minute, we managed to fool the firewall into leaving the database connections in the connection pool alone, and not terminate them even though the client may not execute a query using the connection for an extended period of time.

I am sure you can tweak the value of this parameter to determine the optimal value for your firewall and situation; dropping down the sqlnet.expire_time to a minute is probably too radical and redundant in reality. Even though the probe generated by this setting is small, it still does generate network traffic that may be meaningful in some situations.

Today I had to go in to configure multi-master replication between two Oracle databases as our Oracle replication expert was out at a client site. Well, to set up replication, I had to first connect to the database instances, and I found that one of the two instances was not starting up automatically even though the Windows service for that instance was started. I remembered that this was a problem that I had to run into a couple of years back but it took me a bit to figure out the core issue and resolve it. So I figured, I might as blog about it in case I run into it again a few years down the road and need to resolve the issue again.

First things first, I connected to the Oracle instance as a sysdba, and verified that I could start the instance up manually.

C:\Documents and Settings\Administrator>sqlplus "sys/sys_pwd@portal51 as sysdba"

SQL*Plus: Release 9.2.0.6.0 - Production on Thu Jul 24 21:34:39 2008

Copyright (c) 1982, 2002, Oracle Corporation.  All rights reserved.

Connected to an idle instance.

SQL> startup
ORACLE instance started.

Total System Global Area  236003236 bytes
Fixed Size                   454564 bytes
Variable Size             209715200 bytes
Database Buffers           25165824 bytes
Redo Buffers                 667648 bytes
Database mounted.
Database opened.
SQL> exit
Disconnected from Oracle9i Enterprise Edition Release 9.2.0.6.0 - Production
With the Partitioning, OLAP and Oracle Data Mining options
JServer Release 9.2.0.6.0 - Production

Once I started up the instance, I connected as a regular user and verified that I could connect. I then restarted the database instance Windows service, and found that once again the instance started up in idle mode. So, obviously something else was needed to have the instance start up automatically in the active mode. That turned out to be fairly straightforward. I opened up the registry editor, navigated to ‘HKEY_LOCAL_MACHINE\SOFTWARE\ORACLE\HOME0′, and found that the value of ‘ORA_PORTAL51_AUTOSTART’ (where PORTAL51 was our database SID) was FALSE. I changed that to TRUE, restarted the database instance, and sure enough the instance started up automatically!

PS: In case you are wondering, you did read it right. Yes, setting up multi-master Oracle replication does count as one of my skills – I had to pick it up many years when I could not get supposedly experienced Oracle DBAs to set it up in a timely enough fashion for my taste!

Yesterday, I had the need to reverse engineer an Oracle schema in Microsoft Visio on my now not-so-new Vista laptop. While I had the Oracle JDBC drivers copied onto my Vista laptop, I did not have the Oracle client installed. I first tried to install the Oracle 9i client, because the Oracle schema was in a Oracle 9i database instance. The Oracle 9i client installer warns about incompatibility with Vista and it certainly lived up to the billing. I saw several other users with postings about issues. I saw one obvious issue with folder permissions and changed ownership/permissions on the Oracle client install folder, but that did not resolve the issue all the way. While trying to configure an ODBC data source (only for reverse engineering the Oracle schema) , I ran into errors that had again been encountered by many. The first error was “The setup routines for the oracle in orahome92 odbc driver could not be loaded due to system error code 126″. Clicking ok on this error resulted in a secondary error alert – “Could not load the setup or translator library”. At this point, rather than spend more cycles on the issue, I switched to reverse engineering the schema using the eclipse plugin – Clay. Clay worked just fine for reverse engineering the schema (took all of 5 minutes), but the issue with it was that it did not allow me to move the foreign key connectors to make them route better. At this point, I was back looking at the Visio route and the solution was extremely simple!

All I had to do was -

• Uninstall the Oracle 9i client (more on this later)
• Install an Oracle 10g client that was supported for the Vista platform; download the client from here. I had to make sure to install the Oracle ODBC driver in this step.
• Launch the Oracle 10g Net Manager
  • Create an Oracle TNS listener
  • Configure a net service name for the Oracle schema that I was trying to reverse engineer. I had to check the checkbox for Use Oracle8 Release 8.0 Compatible Identification so that I could specify the SID. I tested the service to ensure I had it right so far.
• Create an ODBC data source where I specified Oracle in OraClient10g_home1 as the driver. When I did this, I got a few alerts for an unsupported operation, but it still seemed to work and brought up the dialog window for Oracle ODBC Driver Configuration. In this dialog, the key thing was to specify the correct net service name that I had configured earlier and that was all it took to get the Oracle ODBC driver to work.
• Next step was to launch Visio again, and reverse engineer using the ODBC data source that I had set up.
• And I had my desired reverse engineered Oracle schema in Visio!

All it took was the right Oracle client for Vista and me stumbling my way through the ODBC setup which does not come naturally to me! Now, you are probably wondering about my choice for my database modeling tool – yes, I am cheap – no Rational Data Architect for me!

PS: Uninstalling the 9i client for me did not work with the version of the Oracle universal installer that was installed on my Vista laptop. I ended up running the 9i client installer again to uninstall my 9i client. There must be several other solutions but this was the easiest one for me.

If you ever need to determine the row count for all tables in an Oracle Schema as I did, here is something to help you out.

Disclaimer: This is not my original work and I do not claim to be an Oracle DBA/expert, but I am writing it up as I continually need this and each time have to go through a discovery process on how to do it.

Save the script: Save the following snippet as a sql script file – for example, “OracleUserTableRowCount.sql” (unfortunately, I do not remember where I got this script from and cannot give credit where it is due).

Set heading off
Set feedback off
Set pagesize 0
Set termout off
Set trimout on
Set trimspool on
Set recsep off
Set linesize 100
Column d noprint new_value date_
Column u noprint new_value user_
Spool tmp
Select 'Select '''||table_name||' : ''||count(*) from '||table_name||';',
    to_char(sysdate, 'YYYYMMDDHH24MISS') d, user u
from user_tables
order by table_name
/
Spool off
Spool count_&user_._&date_
@tmp.LST
Spool off

Execution:

• Connect to the Oracle database using the user for whom you want to generate the table row count report. For example -

D:oraclescriptsrowCount>sqlplus icm/foo@wps6

SQL*Plus: Release 9.2.0.6.0 - Production on Tue Jul 17 11:29:25 2007

Copyright (c) 1982, 2002, Oracle Corporation.  All rights reserved.

Connected to:

Oracle9i Enterprise Edition Release 9.2.0.6.0 - Production

With the Partitioning, OLAP and Oracle Data Mining options

JServer Release 9.2.0.6.0 - Production

SQL>

• Next execute the script you created -

  SQL> @OracleUserTableRowCount.sql

  SQL>

  This will create a file with the row counts in the directory you started sql-plus from.

Recently I ran into the situation where I knew that an Oracle database table had a duplicate row, and needed to detect the duplicate row so that I could resolve the situation. You might say that the database design could and should enforce a uniqueness constraint to prevent this situation and you would be right – however given the situation how do you detect the duplicate row?

I don’t claim to be an expert in SQL or Oracle as many can attest but I can perform Google searches! Based on a quick search, I found that an effective way to detect duplicate rows is to join a table to itself.

SELECT

    ID,

    DN

FROM

    ldap_person A

WHERE

    rowid >

    (SELECT min(rowid) FROM ldap_person B

        WHERE

        B.dn = A.dn

    );

Please note that the where clause of the embedded select statement must specify all columns that make the row a duplicate. Once you have found the duplicate row, you can take the appropriate corrective action whether that is to delete the duplicate row or to update the row to resolve the duplicate.

I ran into a weird issue today with Oracle Enterprise Manager Console (EMC) today. I had the EMC (v9.2.0.6) open in the standalone mode, and when I would attempt to connect to a database instance entering my login credentials nothing would happen!!! I would enter my user name and password, and click “Ok” but I would fail to connect to the database instance with no error message. This had me baffled and thinking that we may have a JVM incompatibility issue given that I had run into a somewhat similar error with Oracle 8i install on Pentium 4 processors back in the day. You would click “Setup” to run the installer and nothing would happen – that issue turned out to be a Pentium 4 bug that affected the JIT compiler bundled with Oracle 8i. Anyways coming back to the issue from today, once I got over this hangup of suspecting a JVM issue, I ran the EMC in the debug mode -

oemapp debug console

I then attempted to connect to a database instance to produce the following exception stacktrace on the command window from where I ran the EMC -

NVFactory: _readNVPair expected )

NVFactory: _readNVPair expected )

NVFactory: _readNVLiteral expected LITERAL

java.lang.NullPointerException

at oracle.net.nl.NVPair.setAtom(NVPair.java)

at oracle.net.nl.NVFactory._readNVPair(NVFactory.java )

at oracle.net.nl.NVFactory.createNVPair(NVFactory.jav a)

at oracle.net.nl.NLParamParser.addNLPListElement(NLPa ramParser.java)

at oracle.net.nl.NLParamParser.(NLParamParser.java)

Searching for this exception in Google yielded exactly one result – http://www.dbforums.com/archive/index.php/t-807556.html and there I found the solution to my problem. As per this posting, the error is caused by misconfigured tnsnames.ora or listener.ora. Once I had this clue, fixing tnsnames.ora and getting past the issue was straightforward.