Running WAS 6.1.xx in a network deployment. The IBM provided plugin keystore's "plugin-key.kdb" password expires on april 26th along with the personal cert inside it.

So no problem right? Create new cert and set new password on the kdb, restash the password and off we go!

Well no! On restart of IBM HTTP Server we see

[Tue Apr 24 14:11:22 2012] 00b00004 00000001 - ERROR: lib_security: logSSLError: str_security (gsk error 408):  GSK_ERROR_BAD_KEYFILE_PASSWORD
[Tue Apr 24 14:11:22 2012] 00b00004 00000001 - ERROR: lib_security: initializeSecurity: Failed to initialize GSK environment
[Tue Apr 24 14:11:22 2012] 00b00004 00000001 - ERROR: ws_transport: transportInitializeSecurity: Failed to initialize security
[Tue Apr 24 14:11:22 2012] 00b00004 00000001 - ERROR: ws_server: serverAddTransport: Failed to initialize security
[Tue Apr 24 14:11:22 2012] 00b00004 00000001 - ERROR: ws_server: serverAddTransport: HTTPS Transport is skipped
[Tue Apr 24 14:11:22 2012] 00b00004 00000001 - ERROR: lib_security: logSSLError: str_security (gsk error 408):  GSK_ERROR_BAD_KEYFILE_PASSWORD
[Tue Apr 24 14:11:22 2012] 00b00004 00000001 - ERROR: lib_security: initializeSecurity: Failed to initialize GSK environment
[Tue Apr 24 14:11:22 2012] 00b00004 00000001 - ERROR: ws_transport: transportInitializeSecurity: Failed to initialize security
[Tue Apr 24 14:11:22 2012] 00b00004 00000001 - ERROR: ws_server: serverAddTransport: Failed to initialize security
[Tue Apr 24 14:11:22 2012] 00b00004 00000001 - ERROR: ws_server: serverAddTransport: HTTPS Transport is skipped

Here is the thing ... I can open the keystore using the new password with ikeyman and keytool. Using some "slightly dodgy" script, I can reverse the stash file and see that indeed the new password is set.

Then, even if I restore the old keystore files (plugin-key.kdb,plugin-key.crl,plugin-key.sth,plugin-key.rdb) they no longer work either!

So it must be permissions right? Well the permissions are the same as before, if I switch to apache user I can browse right through to the files and read them. I have even chown'ed them to apache:apache and/or chmod 777 and still its the same error!

Does anyone have a clue what is going on here? Its pretty urgent as our site will be without HTTPS in a couple of days if this isn't resolved - thats bad for a retail web site :)