AIX Grid Control 10.2.0.5 Communication and Monitoring Issue since 31-DEC-2010

Posted by jayatheertha.rao(at)oracle.com on Oracle Blogs See other posts from Oracle Blogs or by jayatheertha.rao(at)oracle.com
Published on Thu, 30 Dec 2010 02:13:46 -0700 Indexed on 2010/12/30 11:57 UTC
Read the original article Hit count: 1152

Detailed symptoms for Oracle Management Server (OMS) 10.2.0.5 on AIX

    • Oracle Management Service 10.2.0.5 instances on AIX 5L remain active and functional, but the OMS instances fail to communicate with the Grid Control Management Agents.

      An SSLPeerUnverified exception will be reported
       in the file $ORACLE_HOME/sysman/log/emoms.trc when OMS attempts to connect with an Agent:
      Javax.net.ssl.SSLPeerUnverifiedException: peer not authenticated
      at com.sun.net.ssl.internal.ssl.SSLSessionImpl.getPeerCertificateChain(DashoA12275)
      at oracle.sysman.emSDK.emd.comm.EMDClient.authenticateHTTPConnection(EMDClient.java:2002)
      at oracle.sysman.emSDK.emd.comm.EMDClient.getConnection(EMDClient.java:1877)
      at oracle.sysman.emSDK.emd.comm.EMDClient.getConnection(EMDClient.java:1810)
      at oracle.sysman.emSDK.emd.comm.EMDClient.verifyHttpConnection(EMDClient.java:2540)
      at oracle.sysman.emSDK.emd.comm.EMDClient.getResponseForRequest(EMDClient.java:2323)
      at oracle.sysman.emSDK.emd.comm.EMDClient.getUploadManagerStatus(EMDClient.java:4853)
      at oracle.sysman.eml.admin.rep.emdConfig.EmdConfigTargetsData.getEmdUploadData
      (EmdConfigTargetsData.java:1640)

      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

      This error may be reported when:
      - Accessing the Agent home page in Grid Control
      - Setting preferred credentials for a target monitored by the Agent
      - Managing metrics for a target monitored by the Agent

    • The jobs scheduled to be run by Agents can become non-responsive
      The OMS log file $ORACLE_HOME/sysman/log/emoms.trc can show:
      2010-12-31 00:06:58,204 [JobWorker 430:Thread-34] DEBUG emSDK.comm getStreamResponse.4015 - oracle.sysman.em
      SDK.emd.comm.CommException: java.io.IOException: javax.net.ssl.SSLPeerUnverifiedException: peer not authenticated
      oracle.sysman.emSDK.emd.comm.CommException: java.io.IOException: javax.net.ssl.SSLPeerUnverifiedException: peer not authenticated
      at oracle.sysman.emSDK.emd.comm.EMDClient.getStreamResponse_(EMDClient.java:4088)
      at oracle.sysman.emSDK.emd.comm.EMDClient.getStreamResponse(EMDClient.java:4009)
      at oracle.sysman.emSDK.emd.comm.EMDClient.remoteOperation(EMDClient.java:3404)
      at oracle.sysman.emdrep.jobs.CommandManager.requestRemoteCommand(CommandManager.java:765)
      at oracle.sysman.emdrep.jobs.commands.RemoteOp.executeCommand(RemoteOp.java:434)
      at oracle.sysman.emdrep.jobs.commands.RemoteOp.executeCommand(RemoteOp.java:491)
      at oracle.sysman.emdrep.jobs.BaseJobWorker.runStep(BaseJobWorker.java:614)
      at oracle.sysman.emdrep.jobs.BaseJobWorker.doOneOperation(BaseJobWorker.java:738)
      at oracle.sysman.emdrep.jobs.JobWorker.doOneOperation(JobWorker.java:306)
      at oracle.sysman.emdrep.jobs.JobWorker.run(JobWorker.java:288)
      at oracle.sysman.util.threadPoolManager.WorkerThread.run(Worker.java:261)

Detailed symptoms for Grid Control Management Agent 10.2.0.5 on AIX

  • Beginning 31-DEC-2010 00:00:00, 10.2.0.5 Management Agents running on the AIX 5L operating system will fail to monitor Oracle Application Server targets. As a result, the Availability Status for the Oracle Application Server targets will be in the "Metric Error" state.

    NOTE: The 10.2.0.5.0 Agents would experience these errors regardless of the version/platform of the OMS.

    The following metric error is seen in the console for the Oracle Application Server targets monitored by a Grid Control Management Agent 10.2.0.5 installed on AIX and experiencing a Root Certificate Authority issue:

    Message oracle.sysman.emSDK.emd.fetchlet.FetchletException: oracle.sysman.emSDK.emd.fetchlet.FetchletException: oracle.sysman.emSDK.emd.fetchlet.FetchletException: oracle.sysman.emSDK.emd.comm.CommException: java.io.IOException: javax.net.ssl.SSLPeerUnverifiedException: peer not authenticated 

    The Grid Control Management Agent log file $ORACLE_HOME/sysman/log/emagentfetchlet.log (or $ORACLE_HOME/hostname/sysman/log/emagentfetchlet.log for a clustered Agent) includes the following errors:

    2010-12-31 00:01:03,626 [nmefmgr_getJNIFetchlet] ERROR ias.ResponseMetric getResponseMetric.154 - Unable to
    compute application server status
    oracle.sysman.emSDK.emd.fetchlet.FetchletException: oracle.sysman.emSDK.emd.comm.CommException: java.io.IOEx
    ception: javax.net.ssl.SSLPeerUnverifiedException: peer not authenticated
    at oracle.sysman.ias.ias.ResponseMetric.getResponseMetric(ResponseMetric.java:108)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:79)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:618)
    at oracle.sysman.emd.fetchlets.JavaWrapperFetchlet.getMetric(JavaWrapperFetchlet.java:217)
    at oracle.sysman.emd.fetchlets.FetchletWrapper.getMetric(FetchletWrapper.java:382)

  • Beginning 31-DEC-2010, 10.2.0.5 Management Agents on the AIX 5L platform will fail to secure or re-secure with Oracle Management Service (OMS). This failure will cause installation of 10.2.0.5 Agents on the AIX 5L platform to fail.

    NOTE: The 10.2.0.5.0 Agents would experience these errors regardless of the version/platform of the OMS.

    The "emctl secure agent" command will fail with the following error, which will be written to the $ORACLE_HOME/sysman/log/secure.log file (or $ORACLE_HOME/hostname/sysman/log/secure.log for a clustered Agent) :

    2011-01-03 21:06:11,941 [main] ERROR agent.SecureAgentCmd main.207 - Failed
    to secure the Agent:
    javax.net.ssl.SSLPeerUnverifiedException: peer not authenticated
    at
    com.sun.net.ssl.internal.ssl.SSLSessionImpl.getPeerCertificateChain(DashoA6275)
    at
    oracle.sysman.emctl.secure.agent.SecureAgentCmd.checkUpload(SecureAgentCmd.java:478)
    at
    oracle.sysman.emctl.secure.agent.SecureAgentCmd.secureAgent(SecureAgentCmd.java:249)
    at
    oracle.sysman.emctl.secure.agent.SecureAgentCmd.main(SecureAgentCmd.java:200) 


For solution, refer to AIX Grid Control 10.2.0.5 SSL Communication and Monitoring Issue since 31-DEC-2010 (Doc ID 1275070.1)

© Oracle Blogs or respective owner

Related posts about Master Notes

Related posts about 10.2.0.5_aix_agent