glassfish server getting hang after sometime_问答_开发者

I am facing new problem in glassfish server now :(

After server starts successfully after sometime it gets hang and stops processing requests

following are the exceptions in server log

[#|2011-05-08T17:44:34.027+0300|WARNING|sun-appserver2.1|javax.enterprise.system.stream.err|_ThreadID=17;_ThreadName=Timer-18
;_RequestID=a47329c8-5efa-4e29-b90d-a5bda9c8f725;|
java.util.logging.ErrorManager: 5: Error in formatting Logrecord|#]

[#|2011-05-08T17:44:34.036+0300|WARNING|sun-appserver2.1|javax.enterprise.system.stream.err|_ThreadID=17;_ThreadName=Timer-18
;_RequestID=a47329c8-5efa-4e29-b90d-a5bda9c8f725;|
java.security.ProviderException: implNextBytes() failed
        at sun.security.pkcs11.P11SecureRandom.implNextBytes(P11SecureRandom.java:170)
        at sun.security.pkcs11.P11SecureRandom.engineNextBytes(P11SecureRandom.java:117)
        at java.security.SecureRandom.nextBytes(SecureRandom.java:413)
        at java.util.UUID.randomUUID(UUID.java:161)
        at com.sun.enterprise.admin.monitor.callflow.AgentImpl$ThreadLocalState.<init>(AgentImpl.java:162)
        at com.sun.enterprise.admin.monitor.callflow.AgentImpl$1.initialValue(AgentImpl.java:256)
        at com.sun.enterprise.admin.monitor.callflow.AgentImpl$1.initialValue(AgentImpl.java:255)
        at java.lang.ThreadLocal$ThreadLocalMap.getAfterMiss(ThreadLocal.java:374)
        at java.lang.ThreadLocal$ThreadLocalMap.get(ThreadLocal.java:347)
        at java.lang.ThreadLocal$ThreadLocalMap.access$000(ThreadLocal.java:225)
        at java.lang.ThreadLocal.get(ThreadLocal.java:127)
        at com.sun.enterprise.admin.monitor.callflow.AgentImpl.getThreadLocalData(AgentImpl.java:1070)
        at com.sun.enterprise.server.logging.UniformLogFormatter.uniformLogFormat(UniformLogFormatter.java:320)
        at com.sun.enterprise.server.logging.UniformLogFormatter.format(UniformLogFormatter.java:151)
        at java.util.logging.StreamHandler.publish(StreamHandler.java:179)
        at com.sun.enterprise.server.logging.FileandSyslogHandler.publish(FileandSyslogHandler.java:512)
        at java.util.logging.Logger.log(Logger.java:440)
        at java.util.logging.Logger.doLog(Logger.java:462)
        at java.util.logging.Logger.log(Logger.java:526)
        at com.sun.enterprise.deployment.autodeploy.AutoDeployControllerImpl$AutoDeployTask.run(AutoDeployControllerImpl.java
:391)
        at java.util.TimerThread.mainLoop(Timer.java:512)
        at java.util.TimerThread.run(Timer.java:462)
Caused by: sun.security.pkcs11.wrapper.PKCS11Exception: CKR_DEVICE_ERROR
        at sun.security.pkcs11.wrapper.PKCS11.C_GenerateRandom(Native Method)
        at sun.security.pkcs11.P11SecureRandom.implNextBytes(P11SecureRandom.java:167)
        ... 21 more
|#]

[#|2011-05-08T17:44:36.034+0300|SEVERE|sun-appserver2.1|javax.enterprise.system.tools.deployment|_ThreadID=17;_ThreadName=Tim
er-18;_RequestID=a47329c8-5efa-4e29-b90d-a5bda9c8f725;|"DPL8011: autodeployment failure while deploying the application : nul
l"|#]

[#|2011-05-08T17:44:38.034+0300|SEVERE|sun-appserver2.1|javax.enterprise.system.tools.deployment|_ThreadID=17;_ThreadName=Tim
er-18;_RequestID=a47329c8-5efa-4e29-b90d-a5bda9c8f725;|"DPL8011: autodeployment failure while deploying the application : nul
l"|#]

********

when it is in hang state i took thread dump and found 1 dead lock as below !!!

Found one Java-level deadlock:
=============================
&quot;SelectorThread-4242&quot;:
  waiting to lock monitor 0x0040ef60 (object 0x56cd6dd0, a org.apache.jasper.util.SystemLogHandler),
  which is held by &quot;Timer-2&quot;
&quot;Timer-2&quot;:
  waiting to lock monitor 0x0040f5d8 (object 0x568bc440, a com.sun.enterprise.server.logging.SystemOutandErrHandler$LoggingPrintStream),
  which is held by &quot;SelectorThread-4242&quot;

Java stack information for the threads listed above:
===================================================
&quot;SelectorThread-4242&quot;:
    at java.lang.Throwable.printStackTrace(Throwable.java:461)
    - waiting to lock &lt;0x56cd6dd0&gt; (a org.apache.jasper.util.SystemLogHandler)
    at java.lang.Throwable.printStackTrace(Throwable.java:452)
    at java.util.logging.ErrorManager.error(ErrorManager.java:78)
    - locked &lt;0x5a4fe388&gt; (a java.util.logging.ErrorManager)
    at com.sun.enterprise.server.logging.UniformLogFormatter.uniformLogFormat(UniformLogFormatter.java:364)
    at com.sun.enterprise.server.logging.UniformLogFormatter.format(UniformLogFormatter.java:151)
    at java.util.logging.StreamHandler.publish(StreamHandler.java:179)
    - locked &lt;0x568825e0&gt; (a com.sun.enterprise.server.logging.FileandSyslogHandler)
    at com.sun.enterprise.server.logging.FileandSyslogHandler.publish(FileandSyslogHandler.java:512)
    - locked &lt;0x568825e0&gt; (a com.sun.enterprise.server.logging.FileandSyslogHandler)
    at java.util.logging.Logger.log(Logger.java:440)
    at java.util.logging.Logger.doLog(Logger.java:462)
    at java.util.logging.Logger.log(Logger.java:485)
    at com.sun.enterprise.server.logging.SystemOutandErrHandler$LoggingByteArrayOutputStream.flush(SystemOutandErrHandler.java:368)
    - locked &lt;0x568b84f0&gt; (a com.sun.enterprise.server.logging.SystemOutandErrHandler$LoggingByteArrayOutputStream)
    at java.io.PrintStream.write(PrintStream.java:414)
    - locked &lt;0x568bc440&gt; (a com.sun.enterprise.server.logging.SystemOutandErrHandler$LoggingPrintStream)
    at com.sun.enterprise.server.logging.SystemOutandErrHandler$LoggingPrintStream.write(SystemOutandErrHandler.java:293)
    at sun.nio.cs.StreamEncoder$CharsetSE.writeBytes(StreamEncoder.java:336)
    at sun.nio.cs.StreamEncoder$CharsetSE.implFlushBuffer(StreamEncoder.java:404)
    at sun.nio.cs.StreamEncoder.flushBuffer(StreamEncoder.java:115)
    - locked &lt;0x568bc470&gt; (a java.io.OutputStreamWriter)
    at java.io.OutputStreamWriter.flushBuffer(OutputStreamWriter.java:169)
    at java.io.PrintStream.write(PrintStream.java:459)
    - locked &lt;0x568bc440&gt; (a com.sun.enterprise.server.logging.SystemOutandErrHandler$LoggingPrintStream)
    at java.io.PrintStream.print(PrintStream.java:602)
    at com.sun.enterprise.server.logging.SystemOutandErrHandler$LoggingPrintStream.print(SystemOutandErrHandler.java:205)
    at java.io.PrintStream.println(PrintStream.java:739)
    - locked &lt;0x568bc440&gt; (a com.sun.enterprise.server.logging.SystemOutandErrHandler$LoggingPrintStream)
    at com.sun.enterprise.server.logging.SystemOutandErrHandler$LoggingPrintStream.println(SystemOutandErrHandler.java:187)
    at org.apache.jasper.util.SystemLogHandler.println(SystemLogHandler.java:254)
    at org.apache.jasper.util.SystemLogHandler.println(SystemLogHandler.java:254)
    at org.apache.jasper.util.SystemLogHandler.println(SystemLogHandler.java:254)
    at org.apache.jasper.util.SystemLogHandler.println(SystemLogHandler.java:254)
    at java.util.logging.ErrorManager.error(ErrorManager.java:76)
    - locked &lt;0x5a4fe3a8&gt; (a java.util.logging.ErrorManager)
    at com.sun.enterprise.server.logging.UniformLogFormatter.uniformLogFormat(UniformLogFormatter.java:364)
    at com.sun.enterprise.server.logging.UniformLogFormatter.format(UniformLogFormatter.java:151)
    at com.sun.enterprise.server.logging.AMXLoggingHook.publish(AMXLoggingHook.java:198)
    at com.sun.enterprise.server.logging.FileandSyslogHandler.publish(FileandSyslogHandler.java:531)
    - locked &lt;0x568825e0&gt; (a开发者_开发知识库 com.sun.enterprise.server.logging.FileandSyslogHandler)
    at java.util.logging.Logger.log(Logger.java:440)
    at java.util.logging.Logger.doLog(Logger.java:462)
    at java.util.logging.Logger.log(Logger.java:551)
    at com.sun.enterprise.web.connector.grizzly.SelectorThread.doSelect(SelectorThread.java:1445)
    at com.sun.enterprise.web.connector.grizzly.SelectorThread.startListener(SelectorThread.java:1316)
    - locked &lt;0x56cdf878&gt; (a [Ljava.lang.Object;)
    at com.sun.enterprise.web.connector.grizzly.SelectorThread.startEndpoint(SelectorThread.java:1279)
    at com.sun.enterprise.web.connector.grizzly.SelectorThread.run(SelectorThread.java:1255)
&quot;Timer-2&quot;:
    at java.io.PrintStream.println(PrintStream.java:738)
    - waiting to lock &lt;0x568bc440&gt; (a com.sun.enterprise.server.logging.SystemOutandErrHandler$LoggingPrintStream)
    at com.sun.enterprise.server.logging.SystemOutandErrHandler$LoggingPrintStream.println(SystemOutandErrHandler.java:178)
    at org.apache.jasper.util.SystemLogHandler.println(SystemLogHandler.java:258)
    at org.apache.jasper.util.SystemLogHandler.println(SystemLogHandler.java:258)
    at org.apache.jasper.util.SystemLogHandler.println(SystemLogHandler.java:258)
    at org.apache.jasper.util.SystemLogHandler.println(SystemLogHandler.java:258)
    at java.lang.Throwable.printStackTrace(Throwable.java:462)
    - locked &lt;0x56cd6dd0&gt; (a org.apache.jasper.util.SystemLogHandler)
    at java.lang.Throwable.printStackTrace(Throwable.java:452)
    at com.sun.jbi.management.system.AdminService.heartBeat(AdminService.java:975)
    at com.sun.jbi.management.system.AdminService.handleNotification(AdminService.java:198)
    at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor$ListenerWrapper.handleNotification(DefaultMBeanServerInterceptor.java:1652)
    at javax.management.NotificationBroadcasterSupport.handleNotification(NotificationBroadcasterSupport.java:221)
    at javax.management.NotificationBroadcasterSupport.sendNotification(NotificationBroadcasterSupport.java:184)
    at javax.management.timer.Timer.sendNotification(Timer.java:1295)
    - locked &lt;0x57d2e590&gt; (a javax.management.timer.TimerNotification)
    at javax.management.timer.Timer.notifyAlarmClock(Timer.java:1264)
    at javax.management.timer.TimerAlarmClock.run(Timer.java:1347)
    at java.util.TimerThread.mainLoop(Timer.java:512)
    at java.util.TimerThread.run(Timer.java:462)

**Found 1 deadlock.**

Please suggest

Thanks Ali

got same issue

i resolve it cause my .properties file contained a simple quote ' , thus when Jersey logger tries to log it, it cannot parse the Json send back to the server.

So look which kind of String Jersey is trying to log and be carefull on special characters