With the recent push to meet IBM’s requirements for business partner certifications, we have been working through the process of getting the required sales/technical certifications in a few areas that we were missing them. We certainly did not have all the sales certifications, and Xtivia has been working on getting them locked up. We already had the M197 IBM WebSphere Portal Family Sales Mastery and Related Lotus Products certification, but given that it was in my area of expertise, and I was curious what a sales certification test looked like, I took the test myself last week. And it lived up to my expectation – EASY, and I knocked it out in 33 minutes out of the available 90 minutes.

Today, I was asked by my boss if I could take the M159 WebSphere Sales Mastery Test for the Sales Professional v3 certification test as we needed it. Based on my earlier experience with M197, I fully expected this one to be a trivial task. IBM offers a virtual class “IBM TGVL: WebSphere Solutions & Competitive Selling” as prep for this test, but I was obviously too good to spend 16 hours preparing for a sales test (or so thought me and my inflated ego). Without further thought and effort, I launched the test and quickly realized that I was in trouble; not a good idea to skip the prep course! I spent 74 out of the allowed 75 minutes on the test, and my head was SPINNING – reading, and re-reading the questions and answers, and confusing the heck out of myself.

When I finally submitted my test, the Prometric site took over a full minute or more to grade it – that’s how tough the test was! The good news is that I passed it with some room to spare, but it certainly was not what I expected as a test for a sales guy. I guess if you do the prep course, you probably get exposed to all you need to know, and you can pass the test even if you are a sales guy. Yes, some major trash talking here! But seriously, I am not a believer in certifications proving that I can either implement or sell a particular solution – I have passed every cert test that I have ever taken – so no, it is not sour grapes talking! All these tests simply have some irrelevant questions – sometimes ingeniously framed to confuse candidates, but none that help in real-life scenarios.

However, at the end of it, I must admit that I am proudest that I managed to get this “WebSphere Solution Sales Professional” certification; more than when I got my “IBM Certified System Administrator – WebSphere Application Server, Network Deployment” or my “IBM Certified Solution Developer – IBM WebSphere Portal” or my “IBM Certified System Administrator – IBM WebSphere Portal” certifications. Yes, I know, blowing my horn hard, aren’t I!

PS: no malice intended towards any sales guys – I love them! Love them more when they close sales but love them nonetheless!

I am really excited about this new offering from our business unit – the official press release went out today and I could not resist blogging about it even though I am getting a bit salesy here! As Dennis said in the press release – we have been providing remote WebSphere administration and development services for various client installations that included WebSphere Application Server Network Deployment, WebSphere Portal, and IBM Web Content Management. These services ranged from complete outsourcing of all WebSphere administration and maintenance needs, to on-demand consulting and mentoring. So, in essence, we have now codified these services that we have been offering for years, into pre-packaged offerings that make it simpler for prospects and customers to understand the value proposition of Xtivia’s remote WebSphere services.

An excerpt from the news release -

Virtual-WebSphereAdmin provides a cost-effective solution for organizations seeking to optimize the availability and performance of their critical WebSphere-based business information systems. This is accomplished through a comprehensive remote WebSphere Administration service designed specifically to meet the WebSphere administration, maintenance, management and consulting needs of our clients. Xtivia’s Virtual-WebSphereAdmin service combines a sophisticated suite of monitoring and reporting tools with expert consulting services to deliver complete WebSphere performance management.

You can find more details about this service at http://www.xtivia.com/remote-services/virtual-websphereadmin or you can go directly to http://www.xtivia.com/request-information to get in touch with the Xtivia sales team about this service. I am confident that you will find it worth your time to do so!

We recently switched one of our client’s WebSphere Application Server and WebSphere Portal based infrastructure to a new physical environment. During our testing we found that we were getting stale connection exceptions in SystemOut.log and in the application logs. On digging a little deeper, our team tracked it down to the fact that the Cisco firewall was dropping our Oracle database connections after they had been inactive for a certain amount of time. When our team discussed the issue with the network team, they were essentially told that all was good with the network infrastructure. The application team tweaked some of the connection pool settings but that only helped alleviate the issue slightly and we started encountering performance issues. Eventually, we figured out a solution that works!

The solution that we ended up with was to work around this issue on the Oracle end. We modified the sqlnet.expire_time parameter in sqlnet.ora on the Oracle 10g server down from the default “30″ (30 minutes) to “1″ (1 minute).

The explanation on why this worked

The parameter SQLNET.EXPIRE_TIME is used to specify the time interval, in minutes, to send a probe to verify that client/server connections are active. If the probe finds a terminated connection, or a connection that is no longer in use, it returns an error, causing the server process to exit. This parameter is primarily intended for the database server so that it can free up server-side resources that are not in use.

A side-effect of the probe is that there is TCP-IP activity between the client and the server each time the probe is executed, and the firewall designates the link as an active one. By lowering the probe interval down to a minute, we managed to fool the firewall into leaving the database connections in the connection pool alone, and not terminate them even though the client may not execute a query using the connection for an extended period of time.

I am sure you can tweak the value of this parameter to determine the optimal value for your firewall and situation; dropping down the sqlnet.expire_time to a minute is probably too radical and redundant in reality. Even though the probe generated by this setting is small, it still does generate network traffic that may be meaningful in some situations.

I needed to implement Single Sign-On between IBM WebSphere Portal and HP Operations Dashboard (HPOD) without using a SSO product, and figured that we could do that using the LTPA token generated by WPE on login to the Portal. For LTPA token based SSO to work, we need to be able to decode the LTPA token on the HPOD front – HPOD is based on Jetspeed – in other words, we are looking at implementing SSO between WebSphere and Jetspeed. I was just getting ready to look up some info that I have from Jerry Cuomo on the LTPA token format, when I tried a quick Google search and found an even better answer. I stumbled upon a blog entry and functional code for LTPA token decoding at http://blog.offbytwo.com/2007/08/21/working-with-lightweight-third-party-authentication-ltpa/. I downloaded the code, exported the LTPA keys from a test WPE server, copied the 3DESKey and our LTPA encryption password into LtpaUtils, logged in to the WPE test server, determined the LTPA token cookie value for test purposes, and was able to decrypt it just fine using LtpaUtils. And thanks to Cosmin, all of this took about 20 minutes!

Today one of the portal administrators in my team was trying to reconfigure the LDAP server being used by an existing WebSphere Portal v6 install. The specific reconfiguration that he was doing was extremely simple – he was replacing one LDAP server with another identically configured LDAP server – so the only change was the name of the LDAP server. We were able to perform this change without disabling/re-enabling security in WebSphere Portal.

You end up changing two files to change the LDAP server name -

• security.xml: This file reconfigures the LDAP server being used by WebSphere Application Server. You can find this file in ${WP_PROFILE_ROOT}\config\cells\<cell_name>. You change the LDAP server name (highlighted in red) in the following lines -

<userRegistries xmi:type=”security:LDAPUserRegistry” xmi:id=”LDAPUserRegistry_1″ serverId=”wasadmin_dn” serverPassword=”{xor}wasadmin_pwd” realm=”ldap_server_name:389″ limit=”0″ ignoreCase=”true” type=”CUSTOM” sslEnabled=”true” sslConfig=”test/DefaultSSLSettings” baseDN=”dc=com” bindDN=”wpsbind_dn” bindPassword=”{xor}wpsbind_pwd” searchTimeout=”120″ reuseConnection=”true”>
<searchFilter xmi:id=”LDAPSearchFilter_1″ userFilter=”(&(uid=%v)(objectclass=inetOrgPerson))” groupFilter=”(&(cn=%v)(objectclass=groupOfUniqueNames))” userIdMap=”*:uid” groupIdMap=”*:cn” groupMemberIdMap=”groupOfUniqueNames:uniqueMember” certificateMapMode=”EXACT_DN” certificateFilter=”"/>
<hosts xmi:id=”EndPoint_1173225214609″ host=”ldap_server_name” port=”389″/>
</userRegistries>

• wmm.xml: This file reconfigures the LDAP server being used by WebSphere Portal WMM component. You can find this file in ${WP_SERVER_ROOT}\wmm. You change the LDAP server name (highlighted in red) in the following XML snippet -

<ldapRepository name=”wmmLDAP”
UUID=”LDAP1″
adapterClassName=”com.ibm.ws.wmm.ldap.ibmdir.IBMDirectoryAdapterImpl”
supportDynamicAttributes=”false”
configurationFile=”wmmLDAPServerAttributes.xml”
wmmGenerateExtId=”false”
supportGetPersonByAccountName=”true”
profileRepositoryForGroups=”LDAP1″
supportTransactions=”false”
adminId=”admin_dn”
adminPassword=”admin_pwd”
ldapHost=”ldap_server_name“
ldapPort=”389″
ldapTimeOut=”6000″
ldapAuthentication=”SIMPLE”
ldapType=”0″
sslEnabled=”true”
sslTrustStore=”C:\WebSphere\AppServer\etc\DummyServerTrustFile.jks”
dirContextsMaxSize=”20″
dirContextsMinSize=”5″
dirContextTimeToLive=”-1″
cacheGroups=”false”
groupsCacheTimeOut=”600″
cacheAttributes=”true”
attributesCacheSize=”2000″
attributesCacheTimeOut=”600″
cacheNames=”true”
namesCacheSize=”2000″
namesCacheTimeOut=”600″>

Once you have changed the LDAP server name in these two files to match the new server name, restart WebSphere Portal and you should be set to go.

Open up SystemOut.log in ${WP_SERVER_ROOT}\log and look for the following line during server startup to verify that you are using the new LDAP server -

[6/18/08 12:10:34:690 CDT] 0000000a LdapRegistryI A SECJ0419I: The user registry is currently connected to the LDAP server ldap://wpsldap:389.

In our case, the LDAP server name is “wpsldap”.

My Tip: In our test environments, we usually follow the practice of using virtual host names for the LDAP and database server. Rather than configuring the actual host names of the LDAP and database server being used by a given WebSphere Portal server, we use hosts file entries on Windows that configure wpsldap and wpsdb to point to the IP addresses of the relevant LDAP and database servers. Whenever we need to reconfigure a test WP server to use a different LDAP/database server, all we need to do is change the IP address in the hosts file to point to the new server. Simple, huh! And not exactly rocket science!

Recently our team needed to deploy the Portal Document Manager (PDM) portlet on a public WebSphere Portal page for guest user access. It turned out that this was not as trivial as expected if you don’t know what actions are required and there is nothing on IBM’s site documenting the steps involved in getting the document manager portlet to work for guest users. I figured I would capture the key steps here in case you need to do the same and are having a hard time with this task!

1. Enable public sessions in WebSphere Portal by adding public.session=true to the Navigator Service properties file. Refer to the WP Infocenter for help with this task.
2. Give anonymous user at least “User” level access to the “document manager” portlet.
3. Give anonymous user at least “User” level access to the “document library” being displayed in the “document manager” portlet.
4. Add “Everyone” to the “Authenticated” role of the “document manager” web application at the WebSphere Application Server level using the WAS admin console.

Disclaimer: I owe Dinesh on my team the credit for figuring out the last and most critical step in ensuring that the PDM portlet worked for guest or anonymous users. Kudos, Dinesh!

Yes, I am back with a blog entry after quite a gap here! I have been knocked out sick for 5 weeks now and I am sick of being sick! Better now and hoping to be back at a 100% soon. Enough excuses – onto the subject of this blog entry. We use Log4j’s DailyRollingFileAppender in a WebSphere Portal environment to generate daily log files for various custom portlets that we have deployed. This worked just fine on some WPE v6 test servers with multiple portlets logging to the same log file, and the daily rollover occurring every midnight. However, on some other WPE v6 test servers running the exact same applications, while logging did work the daily rollover did not. Instead of us having multiple log files with date timestamps in their names, we would only have today’s log file – the previous log files would simply disappear! This issue results from the fact that the DailyRollingFileAppender (DRFA) tries to rename the log file during rollover. However, when the rename fails, it ends up overwriting the log file, hence losing information. On checking the application server’s SystemErr.log, we found an error each midnight similar to -

log4j:ERROR Failed to rename [logfilename] to logfilename.2008.01.11].

This appears to be a fairly common issue with log4j 1.2.x in Java application servers running on Windows. While there is no good resolution for this issue there is a workaround that works pretty well.

We ended up getting the DatedFileAppender which is a log4j extension that is publicly available at http://minaret.biz/tips/datedFileAppender.html. DatedFileAppender avoids the need for renaming the log file by simply creating the log file initially with the date in the file name – this is a simple workaround for the problem that we were experiencing. Switching from the DailyRollingFileAppender to DatedFileAppender was fairly seamless.  In addition to installing the jar in WebSphere Portal, we had to switch a few lines of our log4j configuration. We changed the following lines -

log4j.appender.logFileAppender=org.apache.log4j.DailyRollingFileAppender
log4j.appender.logFileAppender.DatePattern=’.'yyyy-MM-dd
log4j.appender.logFileAppender.File=MyApps.log

to

log4j.appender.logFileAppender=biz.minaret.log4j.DatedFileAppender
log4j.appender.logFileAppender.Directory=.
log4j.appender.logFileAppender.Prefix=MyApps_
log4j.appender.logFileAppender.Suffix=.log
log4j.appender.sametimeSyncAppender.Period=month # this line is only needed if the period is not a day

There is a new technote out about an issue that I have seen in the past with WebSphere Portal v6 on a test server, where you find XA errors in SystemOut.log during server startup after WP6 was shutdown ungracefully – for example, on a Windows server reboot after installing a Windows update. Resolving the error was relatively straightforward and involved clearing the transaction logs.

The technote published by IBM explains the scenario, the observed symptoms, and what to do to resolve the issue. You can find this technote titled Shutdown immediate can cause corrupt transactions and affect subsequent server startup here. I figured that I would highlight it as I would have found it useful when I first encountered these errors.

My blogs are getting more and more infrequent recently – things have been more crazy lately at work and I have also been reading a few non-technical books, leaving me less time to write on my blog. Today – I decided to prioritize blogging over other things and write about an issue that had plagued a WebSphere Portal v6.0 cluster. Essentially, if you create a WP6 cluster following the instructions from IBM, you will end up with an issue whereby a portlet edit on one node in the cluster does not appear on other cluster nodes without a server restart on the other nodes.

With input from Michael Chaney (a fellow IBM Gold Consultant), we finally resolved the issue. I have not seen any reports of this issue on IBM’s site or the forums – so I figured I would write about it to possibly help others out. The issue is that by default in a WP6 cluster, portlet configurations are cached but the cache is not “cluster-aware” – as a result, your portlet edit on one server in the cluster fails to invalidate the corresponding cache entry on other servers in the cluster. As a result, your portlet edit does not show up on other servers without a server restart. Resolving this issue requires you to possibly perform the following two steps -

1. Turn on data replication for the dynamic cache service by following the instructions at Enabling cache replication.
2. Set cacheglobal.shared = true in the cache manager service if this is not already set.

Note that these settings are critical to to maintain data integrity between multiple WebSphere Portal nodes in the cluster.

PS: To be fair to IBM , the “Managing the cluster” section of the WP infocenter does have a blurb about “1″, but it does not spell out the consequences of not turning on data replication.

I am back writing about performance issues that we are running into with the secondary node in a WebSphere Portal v6 cluster. I had written sometime back about a startup performance issue on a secondary node in a cluster – at the time, we were not seeing any performance issues subsequent to server start-up. However, now we are seeing horrendous performance issues on the secondary node – running a JMeter test against the secondary node with 2 threads with a think time of 2-9 seconds causes the CPU on the secondary node to be pegged at 100%. Running the exact same test against the primary node with 10 threads causes the CPU utilization to hover between 10% and 20%. It is extremely puzzling to us why this should be the case with the exact same hardware and exact same network configuration. The situation on the secondary node eventually gets to the point where WebSphere Portal ends up with a ton of hung threads and stops responding totally.

We had some wild speculative theories about what could be causing the issue – our suspicion was around the distributed caching with us seeing a lot of Data Replication Service (DRS) activity. We saw an APAR that seemed reasonably close to our situation but that issue was supposedly fixed in an earlier WAS version than what we are running. We are chasing the issue with IBM support and hope to have it resolved soon; curious if there is anybody else running into this issue at all?