Most of the upgrade was fine, but one of the servers in this test environment had both Publisher 6.4 and Search Server (10gR3) on the same machine.  Publisher was not upgraded, but after upgrading search server to 10.3.0 , Publisher reported errors while connecting to the search server on its diagnotics page:

Other components, such as the portal and Collaboration server, were able to connect to the upgraded Search Server without any issues.  Since only Publisher was having this problem, we decided to look at the publisher log files next.  During the startup process, publisher was reporting the following error:

INFO   | jvm 1    | 2009/08/12 11:19:40 | 11:19:40,130 INFO  Initializing Search Server API with inxight = “/opt/oracle/portal/common/inxight/3.7.6/lang” and doc2text = “/opt/oracle/portal/ptcs/6.4/bin/native/doc2text”
INFO   | jvm 1    | 2009/08/12 11:19:40 | 11:19:40,366 ERROR BEA ALI Publisher startup failed
INFO   | jvm 1    | 2009/08/12 11:19:40 | java.lang.UnsatisfiedLinkError: Native library openkernelsearch_4-2j cannot be loaded.  Either the library or its dependencies cannot be found, or all copies of the library have been loaded already:
INFO   | jvm 1    | 2009/08/12 11:19:40 | [attempt 1]: /opt/oracle/portal/ptcs/6.4/lib/native/libopenkernelsearch_4-2j_0.so: ld.so.1: java: fatal: /opt/oracle/portal/common/icu/2.6/lib/native/libicuuc.so.26: wrong ELF class: ELFCLASS64

Ahahh!  The ELF class error indicates that the ICU (internationization) library was built as a 64-bit native library, but Publisher runs in a 32-bit JVM which will not load the 64-bit libraries. Looking at the timestamps on those libraries, it was apparent that these libraries were overwritten during the upgrade of search server, which does run in 64-bit mode.

After considering several options to fix this, we decided to use the previous versions of the common libraries that Publisher expects.  Luckily, we had followed the best practice of creating a backup of the pre-upgrade version of the portal directory.  We chose to restore the ‘common’ directory from the backup to a directory under the Publisher folder to end up with:

  <PORTAL_HOME>/ptcs/6.4/common

Then we needed to update the references in the publisher configuration to change the location of the common libraries.  This is configured in the following file:

  <PORTAL_HOME>/ptcs/6.4/settings/config/service.conf

So we edited this file and changed the location of the common libraries at the following line:

set.LD_LIBRARY_PATH=<PORTAL_HOME>/ptcs/6.4/common/inxight/3.7.6/lib/native:<PORTAL_HOME>/ptcs/6.4/common/outsidein/8.1.5/lib/native:<PORTAL_HOME>/ptcs/6.4/common/icu/2.6/lib/native:/<PORTAL_HOME>/ptcs/6.4/common/pthreads/2002.11.04/bin/native:<PORTAL_HOME>/ptcs/6.4/lib/native

Eventhough, it was just the ICU library that was conflicting, note that all the libraries were updated just to be consistent.   After saving the changes to service.conf and restarting Publisher, all was well again with the search server connection.  And we were merrily on our way to upgrading other portal components!

There are other components that run in a 32-bit JVM, so I would imagine that this would happen for any of those if they coexist with an upgraded Search Server.  Please share if you’ve experienced this issue with Publisher or other components with an upgrade.  If there are a lot of similar cases, this may be justification for a patch from Oracle.