Social Connections Chicago – Here I come!


Had an email in my inbox a short while ago …

Congratulations! Your abstract for Social Connections 11 was accepted!

 

Looks like I will be in Chicago June 1-2 this year! Social Connections is a great conference, you should check it out and attend if you can – there is allot of great content and I am not just talking about my session … and you get to meet and interact with allot of people at the conference and make really, really, really great connections for yourself.

 

Oh yeah .. what will i be speaking on?

IBM Connections – Take Performance Tuning Beyond the Documentation

 

Connections 5.5, TLSv1.2, java.security and the tale of a log day


Let’s set up the background for our story first: Connections 5.5 CR2 on Windows. 3rd party products galore (Docs, Kudos, ProjExec, Text.IO/Ephox), heavy usage and then – above all – the off-and-on problem with the Rich Text widget. As my penchant for acronyms is well known by my friends, so I shall refer to this overall topic as TPP (this pesky problem) – and it kept rearing it’s ugly, mishapen and thoroughly ugly head off and on. We would squash it and then some other config change wold make it come back again.

I wanted to avoid having to switch WAS all the way to TLSv1.2 because of the well documented (potential) fall out for IBM Docs, Text.io and other products. If you want more background on that one, you can read up at the blogs of some of my colleagues – such as Nico, Ben and Robert. There are more, but you can start your education here and branch out.

So, our last defense this time is to enable TLS v. 1.2 ONLY on WebSphere which is a well documented process that actually does not take long – until it turned into the beginning of 8 hours of hell.. All went well until I tried to do a manual sync (syncnode) from any of the Nodes back to the Deployment Manager. I saw errors I had never seen before, all pointing back to SSL and formatting errors. A syncnode with the [-trace] switch wold give me 3000+ lines of juicy gibberish to wade through and no amount of searches on google helped me with anything. It all came back to this errors in the logs:

[Error parsing HTTP status line “\00”: java.util.NoSuchElementException].

After hours of pulling my hair I did what every IT guy does after a while – I looked for somebody to whine to and then beg for help. Multiple people responded, all felt bad for me but nobody was able to assist. In the end, it took my friend Nico going through a list of possible causes for TPP until he hit something that jiggled my memory: [Java Security].

The Cuplprit

This is where we go from prose back to techno talk – I dimply remembered that the install of ProjExec (btw, great project management tool – complicated but really, really good) has a requirement in it’s install documentation to edit the contents of the java.securty file of each node involved – the change is basically to change which SSLServerSocketFactory to use and here the change:

# Default JSSE socket factories

ssl.SocketFactory.provider=com.ibm.jsse2.SSLSocketFactoryImpl

ssl.ServerSocketFactory.provider=com.ibm.jsse2.SSLServerSocketFactoryImpl # WebSphere socket factories (in cryptosf.jar)

#ssl.SocketFactory.provider=com.ibm.websphere.ssl.protocol.SSLSocketFactory

#ssl.ServerSocketFactory.provider=com.ibm.websphere.ssl.protocol.SSLServerSocketFactory

 

The above shows what the change looks like, basically you un-comment the first two lines and comment the second two.

I reversed the change and – presto – TLSv1.2 works and the nodes can all talk to each other. We are working with the vendor to figure out if we really still need this change going forward. I am also thinking that this might have something to do with an SSL error on Activities file uploads I saw here and there – not sure.

So, the lessons of this days was:

  • If you are following documentation and other people can get it to work – it’s you, not the documentation
  • Peel back the onion: If you set it all correctly in WebSphere, step one pace back/up the chain of technology – it runs java, is java based -> you need to check up the chain to see what base java settings are in place, other than what you set yourself.
  • Don’t cry, it’s unbecoming
  • When friends who are kind enough to answer your Skype calls, LISTEN TO EACH QUESTION and think the answer through, you might not be seeing the forest because all those damn trees are in the way.
  • Say thank you – publicly. You might still be sitting there all night trying to figure out what went wrong

Technote: “Freemarker Template files are overwritten during IBM Connections CR2 install”


This happened to me, I was only saved by having a local back-up on my machine … don’t let it hit you!

Technote Link – swg21996243

Hving a good back-up before ANY upgrade, change etc. is important. If nothing else, do a backup config with WebSphere – that will capture all of the important files for you as well!

 

 

 

 

IBM WebSphere / Connections – Performance, Security and the SESN0008E Error


Just found a new issue today that has been vexing me for quite some time, I only found it because of the error [SESN0008E] and the fact that I had to add a whole new WebSphere Nodes to an existing environment so that these errors finally happend with a frequency that I finally noticed it:

logServletError SRVE0293E: [Servlet Error]-[atom-basic]: com.ibm.websphere.servlet.session.UnauthorizedSessionRequestException: SESN0008E: A user authenticated as anonymous has attempted to access a session owned by user:defaultWIMFileBasedRealm/CN=Joe Shmoe,OU=MYOU,dc=corp,dc=company,dc=com

We ran function testing on the new Node and performance was horrendous … I mean really horrendous. Performance has been bad overall before that, but at least tings worked. I investigated a few tech notes, there was some mentioning about the LTPA timeout being too short in combination with several other settings. This is an upgrade, same settings as other systems, should not be an issue … so I looked at other sites/pages here, here and here.

All of the tech notes mentioned [Security Integration] being at the heart of it. I checked all servers and noticed – none of the server that the Connections installer created had this settings set, all servers that I created manually had it. I looked into this a bit more and found out that the [Session Management] – [Security integration] is now a default setting for WebSphere and if you create a server manually it is automatically set. I ran a few third party products in separate servers that were all manually created … they probably brought the overall performance down.

So, I went through all servers that I had created, unset the settings (pic below) and then synced and restarted everything and …. voila, speed restored.

sessionManagement

IBM Connections – CCM Folders and File Loss … or not


Thought I’d share this one, it was a bit unique. Anyone out there who has had to do a restore of CCM and CCM files will know it is a pain. I will blog about how to do that separately .. It is not fun.

Here the scenario:

  • Client with IBM Connections 5.0, CR3, CCM – very active site,.
  • Many communities, allot of File Libraries. Many users use the Windows Plug-ins and access files and CCM libraries via the Windows desktop.
  • A user – somehow by mistake – deleted a folder inside a community library (CCM library!) containing a whole bunch of files – and this user did it via the desktop. Apparently what the user did how in Windows Explorer a folder was either moved or deleted (I was never able to exactly find out) and the folder with all files contained in it disappeared. Not to be found.

Long Story Short:

I was in the middle of restoring the Filenet databases from the day before to a separate environment to figure out if I could identify the files and ask the back-up team to restore them for me from tape. I took another look in the system and could not find the files in the acce interface.

But then I had an idea … Connections search indexes it all, no matter where it is.

So, using the Filenet restore I identified the files to look for, did a search for the filenames in Connections … and found them. We then did a “Move to Folder” for all the files (into a new folder in Libraries that we created) and all files were back where they belonged.

So – what this taught us is that deleting folders using the Windows plug-is does nothing to the actual files, it just appears to be removing the pointers to the database that the system needs to display the files … But Connections search still find them all. Like Pokemons … gotta catch’em all!

If I had known this earlier I could have saved myself a day of work …

 

 

 

Social Connections – Toronto Jun 6-7, 2016 – I am attending!


The next Social Connections Conference has been announced: June 6-7, 2016 in wonderful and clean TORONTO CANADA

http://socialconnections.info/

 

I already signed up and a submission for an abstract is already in ….

Anybody in the social media/social networking sphere should really attend this conference. Technical and Strategy without the marketing hype, that is why I really like to attend.

 

Go forth and attendeth!

Connections 5.5 – Install Problem for WebSphere Cluster Settings with UNC Shares


I just installed a new Connections V5.5 environment for a new client and came across this issue that I had encountered once before in previous versions when installing the IBM File viewer (look at my presentation from last year at MWLug 2015) .

Scenareo:

  • Connections 5.5,
  • Clustered Windows WebSphere servers (2 nodes on separate Windows server)
  • Windows File Share for shared file services (accessed using a UNC link i.e.: \\[fqhn of server]\[share name])

The Installer will go through and work without a problem, all apps are installed and the clusters in WebSphere created. When you run the WebSphere servers/JVMs for the first time you might notice a new folder created on the same drive as your WebSphere install, the name follows the above UNC naming for the shared file services location. In my case the folder created was [D:\FILESERVER\CnxData\messagestores\xxx).

Messagestores are the way that messaging engines running on WebSphere clustered servers communicate with each other by reading/writing log files (there is much more to it, but let’s keep this lite here …). Both Windows server will create the same folders and you will probably not see a whole lot of errors in the systemout.log files of the WebSphere servers because … those servers can access the files they expect, that they are not getting any inputs from other cluster members is not going to raise any errors inside of WebSphere.

In V5.0 what happens is that the installer creates a WebSphere variable and uses that variable in the cluster settings and then the system works and the UNC drive is read correctly. The V5.5 installer does not do this, it writes the location directly into the sib-engines.xml file of the cluster created and then things fall apart ….

 

What to do:

Basically you have to manually do what the installer should have done:

Create a WebSphere variable

  • I created the same one as V5.0 would have [MESSAGE_STORE_PATH] and gave it the value of the UNC folder location in WINDOWS format (using “\” slashes): i.e. [\\servername\share\messagestores]

Update the sib-engines.xml

  • Search for the sib-engine.cml files  on the Dmgr profile under: ..\WebSphere\AppServer\profiles\Dmgr01\config\cells\[cell name]\clusters\[Cluster Name]
  • Edit the last line in the file for each cluster to look something like this:
<fileStore xmi:id="SIBFilestore_1456105865384" uuid="5976E93BC88E6CB1" logSize="100" minPermanentStoreSize="200" maxPermanentStoreSize="500" minTemporaryStoreSize="200" maxTemporaryStoreSize="500" logDirectory="${MESSAGE_STORE_PATH}/UtilCluster/log" permanentStoreDirectory="${MESSAGE_STORE_PATH}/UtilCluster/store" temporaryStoreDirectory="${MESSAGE_STORE_PATH}/UtilCluster/store"/>

Note the use of “/” in this entry, do it that way!

Do the WAS Thing:

  • You need to then sync the nodes and restart all servers/clusters and then WebSphere will create the folders and subfolders is needs and all will be well ….

 

After a restart you can delete the incorrectly created folders, they do not contain any data you need, the data written into there is transactions and will be re-created when the servers restart.

Engage 2016 in Eindhoven, NL – Here I Come!


Just got this happy little email in my in-box:

 

Dear Victor,

More than 140 session proposals came in, and we had to make VERY tough decisions to get them reduced to 58.
We even added an extra track, to allow for 10 additional slots!
We are extremely happy to inform you that we accepted the following session:

Adm09. IBM Connections – Managing Growth and Expansion.

We look forward to seeing you soon at our 2-day event on WednesdayThursday, March 23-24, 2016, in Eindhoven, the Netherlands.
We picked out an awesome venue.

Now I have to book fights and get myself over there …..

If you are in Europe … if you are anywhere in the world, you will want to attend this user group!

 

For more info, go to this link HERE 

 

Who Said You Can’t Have Fun At Work?


I have been quite for a while. ALLOT of client work, the Connect2016 conference, family, Christmas, cookies .. allot of cookies. A new year, a new work-out plan and new projects!

This was one of those days – in a good way. Major client of mine who has been successfully running IBM Connections for several years and is fairly vested in the platform. They have dedicated company resources that look into adoption and training and how they can use the platform for the business and make it work. Not everything is rosy-sunshine-and-cloudless-days but overall this client as a whole just gets it.

This client uses Outlook as their email platform, but they have a very large Domino application presence which runs a major part of their company business and is thriving. Again, not everything is all sunshine but overall things go in the right direction – the client listens, tries hard and improves constantly. The discussions generally evolve around how to get “it” done and not why not to do “it”.

I come onsite regularly to have face-to-face meetings and do some staff training and mentoring and I found this one inconspicuous meeting invite in my calendar for today to talk about a new business venture they want to automate and have custom development done for. This is where things get exciting for me. It is not every day that you participate in these kinds of talks and do not have to wade through preconceived notions about wanting to re-invent the wheel and “using the latest in technology” and the “newest development platform” because it sounds good n marketing material. Instead, the meeting first evolves around educating me on their business (much to learn, young Paduan …), the client(s) and their future plans so I can help them to achieve their goals by using ( no, it’s time for a buzz-word) “leveraging” existing technologies along with some outside assistance/expertise that might not yet be available in-house. There are slides prepared with business processes and decision trees that need to be translated into program code and automated processes, wishlists about capabilities and the question “Victor, how do we best make this work”…one of those priceless Mastercard moments for any IT guy, really. “Tell me how to make it work” – the words we all want to hear – empowering, challenging, exciting – all wrapped up into one short sentence.

So now I find myself up in the middle of the night, thinking through the two new potentials (yes, hey have TWO new processes hey want to do) and how to best realize them. What partners in code/crime to assemble and how to best architect this solution using the best of the capabilities that Domino and Connections has to offer – I am so excited I can’t sleep – blogging helps categorize the process and organize my thoughts.

Now I will have to scratch together time between all the other work I have (NOT COMPLAINING, work is good and pays the bills – thank you customers!) to translate all my notes and the documentation they gave me into something I can put out to bid to a few partners I already have on mind and see what comes back. Knowing the people I plan to talk to, there are bound to be some ideas and improvements that I might never had considered.

The power of marrying IBM Domino and IBM Connections and using the best capabilities of both platforms. Sometimes it is just plain fun to be an IBM Champion …..

IBM Connections with Exchange Back-end – Chrome and Kerberos Delegation


First of all, thanks to my new found friend Michele Buccarello who had shared this document earlier last month on some very good pointers about how to integrate Exchange with IBM Connections.  With that document and some guesswork as to encryption settings between WAS and Exchange I was able to solve the problem – 90% of the way. We got it to work with IE and FireFox but Chrome was balking and getting into a log-out cycle. I used Fireshark to take a look and noticed it was an auth.redirect action by the HOMEPAGE app that was followed by a rest API call to Opensocial calendar settings .for my acocunt – and then righ back to the auth.redirect …. a classic redirect loop.
As things were working in FF and IE I knew it was not a system issue but rather a problem localized to Chrome so I looked up some technotes and knowledge base articles and here is how I solved it:
Chrome can be taught to work with Kerberos delegation just as IE and FF. For “normal” SPNEGO it takes it’s settings from IE and will accept them but with Exchange there is delegation going on (if you look at the Connections documentation it has you change two settings for both IE and FF, one of them refers to delegation) and Chrome needs to get a whitelist of which website it accepts delegation tickets from:
Option 1: Command line
Change the command line that starts Chrome to include a command switch:
chrome.exe –auth-negotiate-delegate-whitelist=*
Set the value to either [*] (make sure there are NO QUOTES surrounding the [*] as some documentation in various articles will have you enter it as) or any combination of the actual url you are connecting to i.e.: [*.domain.com] to limit it to anything inside the intranet domain or [connections.domain.com] for only the Connections website itself. Apparently this can also be a comma separated list of entries if that works for you.
Option 2: Create Windows Registry entry
Create this entry: [HKEY_LOCAL_MACHINE\SOFTWARE\Policies\Google\Chrome]
In it create a string entry: [AuthNegotiateDelegateWhitelist]
Any of the values used in the above command line example will work in this registry entry so I suggest to try it above first.
Enjoy – you’re welcome!