A common, yet cryptic message awaits Publisher users who find the Rich Text Editor mysteriously stops working. When editing a Publisher document, the Rich Text window displays a blank box and cannot be clicked on or changed. The corresponding Javascript error message is anything but useful:

Error: Access is denied

Code: 0

URL: http://[servername]/portal/server.pt/gateway/PTARGS_xxxxxx

So what’s going on? Long story short, there’s a bug in the PTControls.js Javascript implementation. The script tries to set the default source of Rich Text Editor iFrame to “index.html”, making the erroneous assumption that index.html will always exist. In some situations, the resulting 404 error causes Internet Explorer to freak out and report “Access Denied”. Note that there are a couple of solutions to this problem.

1) If you consult the Publisher Release notes, you’ll be told this is a bug in IE, and to turn off ”Show Friendly HTTP Error Messages”. Unfortunately, a) this isn’t an IE bug, it’s a problem with the  Portal Javascript, b) this “fix” doesn’t always work in all environment and c) it’s often unpractical to push an IE configuration change out to your entire user base. Anyhow, if you have a controlled environment, with a limited number of users, you might be able to mask this issue by disabling “Show Friendly HTTP Error Messages” in Internet Explorer.

2) Make sure the default iFrame “src” attribute points to something that doesn’t generate a 404. You can do this by either pointing to a well-know file that is sure to exist (i.e. something like http://myimageserver/dummy.html), or just setting the “src” attribute to a javascript value that won’t cause the browser to bomb. In either case, you’re going to have to update two lines of code in PTControls.js at:

$IMAGESERVER_HOME/common/private/js/jscontrols/LATEST/PTControls.js

Make a quick backup of this file, and then search for lines that look like:

var src = (PTBrowserInfo.IS_NETSCAPE_DOM) ? ” : ‘ src=”‘ + ((this.defaultPageURL) ? this.defaultPageURL : ‘index.html’) + ‘”‘;

You should find two instances of the above. Now change those lines to read:

var src = “src=’javascript:false;’”;

Save the Javascript file, hit the Publisher Editor to confirm things are working, and Bob’s your uncle! With the error fixed, the editor window will allow access to the iFrame and its contents. We can all get back to creating announcements in Papyrus Fuscia, Size 5 like my muse, Ski Utah! If only <blink> was still supported in MSIE.

<context-param>

<param-name>com.bea.opencontrols.IMAGESERVER_URI</param-name>

<param-value>http://your.imageserver.url/bpmContext</param-value>

</context-param>

• You’ll also need to copy the following directory into your bpmContext folder of your imageserver:

$BPM_HOME/enterprise/webapps/workspace/opencontrols/plumtree

• Bounce the BPM workspace after making the changes above and you should see a marked improvement in workspace performance

First, a “Did you Know” on the BPM->Portal integration

• Did you know that the BPM->Portal integration was originally written by Plumtree way back when as a way to bolt Fuego BPM into the Plumtree Portal?  This original codebase is still more or less intact, and is still used for the integration today.
• Did you know that the BPM->Portal integration is built in Java and uses Java Server Faces (JSF)?
• Did you know that the BPM->Portal integration also makes use of the somewhat dated Plumtree  OpenControls libraries?
• Did you know that the code for the standalone BPM web-based client is a lot cleaner than the BPM->Portal integration?
• Did you know that there are servlet filters configured on the BPM Workspace that should do neato stuff like caching and compression auto-magically?  

OK, so the whole “Did you know” thing is a bit of stretch, and really just a vehicle to give you background on the issue we were researching.  So….

The Issue We Were Researching

It turns out that, out of the box, the portlet-based BPM workspace doesn’t handle some images too smartly.  Specifically, on every UI request, there are a bunch of static images that get reloaded all the way from the embedded Tomcat server that runs the BPM workspace.  Now, everybody and their brother knows that it’s just good common sense to cache images, so what gives?  Welp, let’s start at the very beginning and see.  After users reported slow-loading images in the BPM workspace, we used one of the best tools known to mankind to take a look all the HTTP requests being made on page load:

As the lovely highlighting above points out, some images in the workspace are getting loaded from the portal imageserver, and are being appropriately cached in the client browser.  We know they’re being cached by that handy-dandy 304 HTTP status code.  But what’s up with all these requests for images and javascript files with HTTP status code 200 (i.e. going back to the server to successfully load a static file on every page invocation)?  And why do all those URLs end with “WebResource.resource?r=foo”?  Blasphemy!  Time to do some poking around and set things straight.

How we researched the issue

This is the section of the blog post where I take you deep inside the black art of tearing apart a web application and making educated guesses to arrive at a testable hypothesis.  Strap in for a non-stop thrill ride.

Anytime I’m faced with a black box type problem (i.e. there’s something going wrong with a piece of commercial software to which I don’t have the source code), I more or less go through the following checklist:

• Is this something I’ve seen before?
• Is this something that somebody else has seen before?
• Is it going to be more painful to open a support ticket on this than to just figure it out myself?
• What do I know about the inputs and outputs of the black box?
• What do I know about how the application is built?
• How can I tear the application apart to learn something useful?

In this case, we were looking at a problem that we’d never noticed before, nor could we figure out a way to ask the interwebs if anybody else had seen it before.  I was pretty sure that going through the process of working a support ticket was going to be more painful that just fixing the problem (besides, what fun is working with support when you can break into the guts of a system?), so I asked myself, “What do I know about the inputs and outputs of the black box”?

• Input -> HTTP Request to a web page that loads mu
  ltiple static files and some dynamic content
• Desired output -> Cached static content
• Actual output -> Non-cached static content 

Then I thought about what I know about how the application is built:

• Java
• Web application
• JSF

Finally, how can I tear the application apart to learn something useful:

• Look at the inputs (i.e. look at the URLs being requested)
• Look at the web.xml and other configuration files of the application to see how everything is glued together
• Decompile the source to see what the hell is going on

So off we went.  The starting point in our maze was to take a good look at the problem image requests (Warning: I’m going to get all stream of consciousness on you here…welcome to my troubled world)

https://portal.foo.com/gateway_noise/bpm_portlet_server:123/workspace/jsf/menu/WebResource.resource?r=/a/path/to/an/image.gif

bpm_portlet_server:123/workspace/jsf/menu/WebResource?r=/a/path/to/an/image.gif

bpm_portlet_server:123/workspace/jsf/menu/WebResource?r=/a/path/to/an/image.gif

Looks like the name of the web app running on that Tomcat instance is workspace.  Maybe there’s a WAR file or something sitting around that I can play with.  Let me take a look in the Tomcat webapps directory on bpm_portlet_server.  Hmm, nothing here…WTF?  Ah, but I know that this is a special Tomcat server that’s configured all crazy-like because it’s embedded with the ALBPM distro. Let me poke around in the ALBPM directories to see if there’s anything interesting there.  Well how’s about that, there’s a directory at $BPM_HOME/enterprise/webapps named “workspace”.  Seems like this might be useful.  Let’s take a look.  OK, there’s no WAR file here, but I do see a WEB-INF directory, guess they’re just deploying an exploded WAR to the Tomcat server.  Let’s see open up the web.xml and see how this thing is configured:

Well that was pretty useless.   Except…hmm…what’s this:

Looks like there’s a servlet filter configured that should be caching the images.  Let me just decompile that guy real quick and see what he’s doing:

What the hell?  This thing should looks to be setting the cache expiration.  Let me go back to  Firebug and see what the max-age is on the images I’m loading:

Max-age=0

WTF?  I guess the portal gateway is somehow stripping off the cache headers for this image. Why don’t we just hit the web-app directly to see if the portal is screwing with the headers.  Back to browser/Firebug pointed at:

http://bpm_portlet_server:123/workspace/jsf/menu/WebResource?r=/a/path/to/an/image.gif

<Reload a few times>

<Look at FireBug to see cache headers>

Max-age=0

WTF?!?!?!  OK, guess that servlet filter isn’t working right – that’s annoying.

Let’s take a look at that web.xml again…

Yep..still noise.  Time to start randomly looking around in directories in the workspace web application.

<cd $BPM_HOME/webapps/workspace/jsf/>
<Look around at a million different files that aren’t very insteresting>
<Stumble upon …/workspace/jsf/view/viewPresentationNormal.xhtml>

<Open viewTableCell.xhtml> n>

fn:getImage(‘ATTACHMENT_ICON’)

OK, so we’ve finally managed to dive down to the where the app is making calls to get images. <Quick timeout to pat myself on back and IM friends a steady stream of profanities about how ridiculously ALBPM is built>.  Now, that fn: prefix is a tag libarary.  Let’s see what taglib is being used.  Obviously that stupid taglib isn’t included in hasAttachments.xhtml, that would be too simple.  So backtrace through the xhtml files until we get to viewPresentationPanel.xhtml which includes that line:

xmlns:fn=”http://bea.com/bpmWorkspace/functionsLibrary

Now to find where the actual code for this taglib lives…BACK to web.xml

/WEB-INF/facelets/bpmWorkspace.taglib.xml

fuego.workspace.jsfcomponents.FacletsFunctionLibrary

jar -xvf fuego.jsfcomponents.jar

oc:someMethod

So let’s see what the oc: tag library is all about.  ViewPresentationNormal.xhtml was nice enough to include the line:

xmlns:oc=”http://bea.com/opencontrols”

Back to our old friend web.xml.

jar -xvf opencontrols.jar

jar -xvf fuego.workspace.jar

jad workspace/jsfcomponents/WorkspaceTableRender.class

grep -i image WorkspaceTableRender.jad

grep -i img WorskpaceTableRender.jad

String urlBase = “/plumtree/common/private/opencontrols/image/table”;

jad com/bea/opencontrols/XPResourceRequest

<context-param>

<param-name>com.bea.opencontrols.IMAGESERVER_URI</param-name>
<param-value>http://your.imageserver.url/bpmContext</param-value>

</context-param>

You’ll may have noticed that I referred to “we” several times in the text above.  No, I haven’t started referring to myself in the royal third.  I actually had a partner in crime for some of this debugging exercise.  Some of you may know him as Howard, some of you might refer to him as Ross, but I just call him HRoss.  In any case, he’s one of the best portal consultants still working directly for Oracle.  So if you’re still shelling out the big bucks for an Oracle consultant (rather than a more reasonably priced alternative *cough* Function1 *cough*), ask for him by name.

Long Story Short (i.e. “Just the useful info, please”) 

There’s a bug in the JDK that causes problems when setting up certain types of Java server socket constructs on some Solaris 10 boxes.  This bug will likely manifest itself in one of the following ways in your ALUI environment:

1)  You’re running the ptlogging utility and see your logs filled up with something like:

PTSelectorThread_17679958      

com.plumtree.openkernel.impl.openhttp.core.network.PTSocketSelector     Unexpected exception.

java.io.IOException: Invalid argument

        at sun.nio.ch.DevPollArrayWrapper.poll0(Native Method)

        at sun.nio.ch.DevPollArrayWrapper.poll(Unknown Source)

        at sun.nio.ch.DevPollSelectorImpl.doSelect(Unknown Source)

        at sun.nio.ch.SelectorImpl.lockAndDoSelect(Unknown Source)

        at sun.nio.ch.SelectorImpl.select(Unknown Source)

        at sun.nio.ch.SelectorImpl.select(Unknown Source)

        at com.plumtree.openkernel.impl.openhttp.core.network.PTSocketSelector.run(PTSocketSelector.java:400)

        at java.lang.Thread.run(Unknown Source)

 

2) Portal starts up fine, but it can’t connect to any remote servers, giving you the same stacktrace as above:

java.io.IOException: Invalid argument

        at sun.nio.ch.DevPollArrayWrapper.poll0(Native Method)

        at sun.nio.ch.DevPollArrayWrapper.poll(Unknown Source)

        at sun.nio.ch.DevPollSelectorImpl.doSelect(Unknown Source)

        at sun.nio.ch.SelectorImpl.lockAndDoSelect(Unknown Source)

        at sun.nio.ch.SelectorImpl.select(Unknown Source)

        at sun.nio.ch.SelectorImpl.select(Unknown Source)

        at com.plumtree.openkernel.impl.openhttp.core.network.PTSocketSelector.run(PTSocketSelector.java:400)

        at java.lang.Thread.run(Unknown Source)

So what’s a good god-fearing person like yourself to do?  Well, if you act now, we’ll send you TWO workarounds for the price of one:

1) Have an SA up the hard File Descriptor (FD) limit on the server to 8193 or greater by editing /etc/system to have the line:

set rlim_fd_max=8193

Note: You’ll need to bounce the box after this change.  You can then verify it worked by running:

ulimit -n -H

Which should return a number >= 8193.  Sadly, this approach will probably have the SA asking you why you want to make the change, which means you’ll have to read the technical details below….so onto option #2

2) Tell the appropriate JVMs to use a different Socket Selector configuration.  You do this by passing the following option to the JVM:

-Djava.nio.channels.spi.SelectorProvider=sun.nio.ch.PollSelectorProvider

Depending on what ALUI component you’re updating, you may pass this option in different ways.  For instance, if you’re dealing with one of the back-end servers (Collab, Studio, etc), you’ll want to update wrapper.conf to add additional arguments like:

wrapper.java.additional.7=-Djava.nio.channels.spi.SelectorProvider=sun.nio.ch.PollSelectorProvider

Note: Replace “7″ in the above line with the appropriate number for your wrapper file.

Or you may just need to update a shell script somewhere that’s kicking off Tomcat/Weblogic/etc.  Note that these scripts all have their own Shell variables for holding additional Java arguments, so just look through them and update as appropriate.  If you have problems, feel free to post questions here and we’ll do our best to help out.

Long Story Long (i.e. “I’m kind of a geek, and I’m sitting at work with nothing else to do, so give me the details”)

So ALUI uses the NIO java packages that were introduced in Java 1.4.  FWIW, I always thought NIO stood for Non-blocking IO, but a little Googling reminds me that it’s actually New IO…silly me.  In any case, the NIO packages let you do some cool things with sockets to more efficiently manage high-volume connections.  The under-lying problem you’re running into is that out-of-the-box Selector implementation in the JDK uses /dev/poll to allocate 8192 File Descriptors (FD) for use by the selector, and 8192 exceeds the nofiles (Number of File Descriptors) limit on your server.  So, you can either bump up the server FD limit ala work-around #1 above, or tell Java to use a different selector implementation that doesn’t allocate all those FDs ala option #2 above.  If you’re interested in more detail, you can find the Sun bug on the issue here.

Until next time…thanks for reading! 

Well, it’s actually 2 bugs (maybe 3).  Let me explain:

When you answer “yes” to that question about security propagation, a job is created.  Here’s the problem:  the job is run as the user who created the folder, not the user changing the security.  What if you later delete that user?  Well, bug #1: automation server is hosed.  You’re going to get an error like this:

The Exception says “*** Job Operation #1 of 1 with ClassID 20 and ObjectID 898 cannot be run, probably because the operation has been deleted.”, and the error’s wrong (because it says “probably”, I won’t count this as a bug).  The real problem in this case is that the folder’s OWNER has been deleted, not the operation itself.

This gets me to Bug #2: when you delete a user, they’re removed from all groups, but apparently they’re not removed as OWNERS of any of the admin folder objects (and how could they?  What should they revert back to?).  Obviously, this is what causes the problem with automation server.  If this one was fixed somehow, bug #1 would become irrelevant.

Bug #3 (this one I haven’t confirmed yet, but is likely to exist given the testing and logs mentioned above): suppose Joe Content Manager creates an entire structure of folders.  Then Joe’s boss decides he shouldn’t be an admin for one of the child folders (say, an “Executive Committee” folder) and removes his access.  He changes the security and propagates it on that “Executive Committee” folder.  Later, he wants to add Mary Executive to have privileges to the folder.  He changes the ACL again and chooses to propagate.  Here’s the rub:  because the job is run as Joe, and Joe no longer has access, the job fails, and Mary can not be added.  In fact, it’s likely that Joe can’t even be added back, so that ACL is completely frozen unless the admin goes through every object and re-adds Joe back (or changes the owner of the folders through the DB).  Have I mentioned LockDown?

Anyway, want the SQL Scripts to fix the owners on the various folders?  Hit the jump.

This query will give you a list of all the ownerids in the PTADMINFOLDERS table:

select distinct ownerid, count(ownerid) from ptadminfolders group by ownerid;

This one will give you the folder count for all folders that have invalid owners:

select distinct ownerid, count(ownerid) from ptadminfolders where ownerid not in (select objectid from ptusers) group by ownerid;

And finally, this one will change all those ownerids to use the Administrator user.  [I thought this might be a security hole, because then users would be able to edit permissions on objects they shouldn't have access to (the opposite of Bug #3, above), but I can't think of any scenario where it would, since if a user has admin privileges on a folder, by definition, that user has admin privileges on all child folders as well.]:

update ptadminfolders set ownerid=1 where ownerid not in (select objectid from ptusers);

Unfortunately, I just ran into a major problem with this feature in a dev upgrade: the servlet for streaming all this content back doesn’t do any transformations on any of the URLs.  This is bad, since if you’re doing an upgrade, you’ve likely got hundreds or thousands of Content Items with relative links or images in them.  For example, suppose you have an existing Content Item with an image in it (that has been published to the same folder in Publisher).  The reference for this image will likely be:

<img src="myimage.jpg">

This worked fine with the old redirector, as the image tag would be transformed:

http://server/portal/server.pt/gateway/PTARGS_0_1_493_234_684_43/http%3B/ptpublisher%3B7087/publishedcontent/publish/folder1/folder2/image.jpg

But with the new content streaming functionality, this link does NOT get transformed the way you’d expect; the portal sees the relative link as relative to the SERVLET, not relative to the existing Content Item.  So the browser sees a URL like this:

http://server/portal/server.pt/gateway/PTARGS_0_1_493_234_684_43/http%3B/ptpublisher%3B7087/ptcs/PublishedContentServlet/image.jpg

… which obviously isn’t right.

In fact, Oracle seems to be aware of this: in the content.properties file, they changed the LTCUseRelativeURLs setting to “false” (it was “true” in 6.4).  That way, for NEW Content Items, the URLs are all absolute, which prevents this problem from coming up.

I was going to revoke my recommendation to upgrade to 6.5 for now, but came across a really easy fix to get all the benefits of 6.5 without this glaring bug.  Hit the link for the workaround.

The solution:  turn off the servlet in the “Published Content” Web Service.  Oracle left the redirector in there, and while this means you won’t get the scalability benefits of the new servlet, at least it’ll buy you time to check on the hundreds of Content Items you’ve already got.

Just change the Portlet URL for the “Published Content” Web Service from “PublishedContentServlet/proxy” to “published_content_redirect.jsp”, and all of the existing items should work again: