Just go to Administration: Select Utility…: Collaboration Administration: Collaboration Notification, and enable Health Monitoring:

It works, too (assuming your SMTP server doesn’t require authentication for internal addresses).  Collab is even kind enough to mark the mail as “High importance”:

On the other hand, for those of you that don’t appreciate this new little nugget of functionality, consider this irony:  Collaboration 4.5 uses email to tell you when email is broken.  Cosmic, man.

collab.server.administrator CNS.SECURITY com.bea.notification.security.SecurityManagerUnable to authenticate user with token '1|1288135304|xarwmCj1FbIzv/Mo/yyd0tEjkgI='

Don’t bother looking at the Security And Directory Service (really, what’s that thing for anyway?). The following screen shot explains it all:

Open your portal database, query PTSERVERCONFIG (“select VALUE from PTSERVERCONFIG where SETTINGID=65″), and stick that value in the “Message authentication code seed value” field on the “Aqualogic Notification Service”/”Login Tokens” page.

Maybe when I’m much less annoyed at this whole process I’ll dig more into the SAML2 Token Type.  And that elusive “Security and Directory Service”, which still seems to provide no more valuable service than being a red herring for bizarre portal issues. 

Here’s the general premise to ALI Directory Services: ALI 6.5 ships with this new Directory Services component that provides an LDAP service for Portal User accounts.  The idea is that historically, the portal has been great at synching users from external repositories (from AD, LDAP, or custom sources) into its own database. Once those users get synched and aggregated into the portal, though, they’re not exposed to any other services.  Directory Services aim to resolve that problem: 6.5 provides an LDAP server that uses the industry-standard LDAP protocol to expose users that have been synched to the portal.  So any other system can use LDAP to get user information.

Fantastic feature, right? But with the dearth of documentation out there, what may not be immediately obvious is that this Directory Service is also used by internal components such as the Notification Server.

I’ve only begun to scratch the surface with how all these components work together, but if you’re interested in reading about how they DON’T work together (saving yourself hours of diagnostic time), hit the jump.

This was the problem: The notification service was throwing the following exception:

Unable to retrieve user with UUID '{75AC0C94-1191-50A1-7217-1348987BA000}'
com.plumtree.security.InternalServerException
at com.plumtree.security.client.impl.util.ExceptionHelper.translate(ExceptionHelper.java:38)
at com.plumtree.security.client.impl.entity.UserManagerImpl.findUserByUuid(UserManagerImpl.java:79)
at com.plumtree.security.entity.UserManagerWrapper.findUserByUuid(UserManagerWrapper.java:48)

At first I thought the problem was with the “BEA ALI Security and Directory Service” component, but then started to focus on the “BEA ALI LDAP Directory” service (Don’t get me started on the worthlessness of the exception itself.)

Anyway, the moral of this story is that the port set in the Configuration Manager for the “AquaLogic Notification Service” (under “User and Group Directory”) didn’t match the port for the “ALUI Directory” (under “LDAP Listener Settings”).

Because we already had an LDAP server running on that machine, I had to change the port for the LDAP service to 2389.  But I also needed to change the port that Notification Server used to connect to it – which is fine, but isn’t that the point of the Configuration Manager – change a setting once and it’s reflected everywhere?

Obviously, the locally saved snapshot can get out of date over time, but it’s nice to have when you need it in a pinch (having been on sites that don’t allow you to connect to their network).

If you want an Excel copy of the matrix, you can download it here.  Keep in mind it’s only current as of the end of August 2008, but for prior versions of the portal, the information should be pretty static. For the latest version, visit one.bea.com (while it lasts).

Basically, the tool reports on every single file access and which Windows process is making the request.  As you can imagine, this produces a LOT of noise – Windows is constantly accessing hundreds of files from dozens of processes, but fortunately, FileMon provides some robust filtering to help you find what you’re looking for.

Let’s take an example from the use case in my last post: figuring out why I’m not getting notifications.  All I had to go on was this exception:

Hm… can’t find a file in ”/plumtree/collab/templates/”?  But where is it looking for that file?  I know these templates used to exist in /ptnotification/4.2/settings/templates, but /ptnotification/ doesn’t exist anymore.  And they’re not in the /cns/ directory OR the /ptcollab/ directory.

Join me after the link for the solution!

Again, the problem is we have an error that looks like it has to do with file access, but we have no idea where the system is even LOOKING for the file, let alone whether it exists or not.

Enter FileMon.  Basically, you start the tool and start capturing file access.  The initial problem is that it’s capturing HUGE amounts of information, so you use the “filter” button to rule out processes that can’t possibly be accessing the file you’re looking for, such as Windows Explorer or Windows Services:

When that’s done, you’re still going to have a really big list of file accesses, but at least it’s manageable.  From here, repeat the process you were doing (in my case, initiating a Notification request), and let FileMon do its thing.  Then, search the resulting list for the file you’re having problems with.  I found this entry with the file name showing up in PTSpy:

WOW - I wasn’t expecting that!  Notice that the process accessing the file is IIS.  And the path is in the \imageserver\ folder.  But, that was all the information I needed to realize the templates were stored on the image server, and chase down the original problem with IIS.

Download FileMon and see for yourself how useful (or at least interesting) it can be.  Note that this tool has been superseded by ProcessMon, which combines FileMon with Regmon (another Cool Tool I use all the time), but I haven’t checked it out yet.

I just couldn’t get Collab Notification working, so I fired up PTSpy and saw this (“Unable to retrieve the template named ‘/plumtree/collab/templates/NotifyLinkNotificationText.nst’”):

Through a long and arduous process (more on how in my next post), I found this out:

The 6.5 Notification server uses the image server to store notification templates in bea\alui\ptimages\imageserver\plumtree\collab\templates.  (You remember notification templates, right?)

This is interesting; ALI has really gone in the direction of making everything “remote” (remember, Adaptive Layouts are remote so they’re easy to edit and access in a central location). 

But if you’re using IIS as an image server, you’re in trouble.  Hit the link to find out why!

The problem at its core is that unlike most web servers, IIS 6 (and 7) don’t serve files with extensions that don’t have recognized MIME types in the OS (or IIS metabase).  So all your fancy “.nst” and “.inc” collab notification template files on the image server?  Useless!  IIS will simply return a 404 when you try to load the template with a browser (or, in this case, when Notification Server tries to load the template).

This wonderful security feature is described by Microsoft here, here, and here.  In a nutshell, you have to add .nst and .inc as known MIME types to IIS.

To do this, you open IIS Manager, right-click on the local computer name, and go to properties.  Then click the “MIME Types” button and use the “New” button in the popup to register “.nst” and “.inc” as “application/ms-octet-stream”.

Once you do this, Notification Server will be able to load those templates, and you should start getting notifications (I haven’t yet, but that’s yet another post for the ALI Security and Directory Service.  *sigh*).

By the way, to test whether your IIS instance needs this patch, use the following link: http://portal_server/imageserver/plumtree/collab/templates/NotifyLinkNotification.nst.  If you get a 404, you need to make the changes shown above.

UPDATE: I was whining about this little flaw to a good friend of mine, an Oracle Consultant extraordinaire, and he both complained that I never give him a shout-out on this blog (yeah, he doesn’t blog himself), and pointed out that if I had just RTFM, I’d have seen this in there.  So, let’s kill two birds with one stone:  Omid Krabbe, thanks for your tip: don’t do any install/upgrade without reading the documentation, no matter how many times you’ve done ‘em with previous versions!

So I did a reinstall of these components (Collab 4.5 and the Portal 6.5 notification service) and have a couple of tips to share in the upcoming posts.

First, remember to import those PTEs!  I got the following error in Collab (“HTTP Status 400: Invalid path /tasklist/ was requested”) when I first started everything up, and clicked the RSS button in a Collaboration project:

Turns out this was because PTSpy was reporting something about Notification Server not being able to “load the Notification Web Service ID”.  Once I imported the ALI Notification Service PTE file (bea\alui\cns\1.0\serverpackages\notification.pte) and restarted the service, I got this error cleared up.

But this was just the start of my problems.  Stay tuned for more on getting RSS working in Collab.

The authentication ID and the DB password are both written out to the file.  Obviously this isn’t a critical security hole (if a malicious user has access to your system, you’ve got other things to worry about), but a concern nonetheless.  Until the issue is resolved, I’d recommend just upping your log threshold to WARN by modifying the \ptcollab\4.5\settings\config\wrapper.conf file.  Change:

wrapper.logfile.loglevel=INFO

to:

wrapper.logfile.loglevel=WARN

Go Oracle!

Update June 9, 2008: Sorry ladies and gents, I just spent the day trying to upgrade our dev environment here.  After downloading hundreds of megabytes worth of installer packages that wouldn’t even open in Winzip, I tried again and was able to open them.  Sadly, they (at least Publisher and Collab, and I assume the rest of them) still required license keys.  I suspect what happened is that whovever uploaded the real updated installers didn’t FTP them as binary files (which would corrupt the files), and they rolled back to the original versions today.  I’m sure it’s coming, but I was also a little disappointed in this false start…

Update June 10, 2008:  OK, my fault.  The link above makes it pretty clear that the delicensed products are on Oracle’s download site and not available through BEA’s download center.

Well, no more.  The wicked witch is dead.  The clouds have parted.  Angels are singing.  Oracle has released generic license keys to legitimately licensed customers.

Now, people, I’m not saying this gives everyone free reign to steal Oracle products or use them in ways they’re not licensed for.  I have no idea what Oracle’s new licensing scheme is – particularly around using instances in development and DR environments (a common sore point with license keys) – so be good with these things.  I understand Oracle focuses more on customer audits than oppressive licensing schemes, so make sure you stay legit – contact your sales person if you’ve got any questions!

Without further ado, you can download the generic ALI keys here. 

Use them responsibly – our customers always maintained compliance at Plumtree without the keys, and I’m sure you all will again (well, I guess they did: I was just a consultant – what do I know?).