AppletTalk.com

Thorkild Stray · Guest

Hi.

I am looking for advice at debugging a problem concerning the JVM dieing
with a segmentation fault. We're running release cxia32131-20030618 (which
I believe is SR5).

We've earlier had problems with constant crashing due to bugs in the
JIT-compiler, but thanks to the improved javacore-dumps and the Diagnostic
guide we've found suitable workaround for those. We still have one
problem, though, which I can't seem to figure out.

The setup:
We've got boxes running RedHat 2.1 AS with 2 cpus and plenty of ram. The
JVM is running with 1 GB heap where it normally GCes down to 500-600 MB
usage when it needs to. On top of the JVM, we're running Oracle
application server (oc4j).

The problem:
The system keeps running for 6-10 hours and then it crashes. The timing
seems rather random, but normally about twice a day.

Test-setup:
We have failed to recreate this error in our test setup. We've tried to
run some of the same traffic, but still, we can't seem to make it die in
the same way. This is also our main problem, since it would be much
simpler to do further detective work if we had a known way of making it
crash.

Work done:
According to the javacore files, the JVM is crashing with a SIGSEGV in
IBMJava2-131-sr5/bin/classic/libjvm.so. After disassembling and
recalculating the crash address according to where in memory it is mapped
in, we see that it crashes in the localMark-call. The thread that leads to
the crash is (again, according to the javacore file) the GC Daemon. At
first we thought this might have something to do with the parallell GC
system in the newer versions, but turning this off only meant that the
crash moved from parallellmark to localmark. If we look at the
stackpointer that is written to STDERR when it crashes, it also looks like
the same thing is happening each time. All crashes end with the JVM
giving a stackpointer that has the value ending i "c7a78", where the
complete adress (for example 0xbe9c7a78) always maps into something that
looks like the following line from the memory map:

2HPMEMMAPLINE b9bbf000-b9bc8000 rwxp 00001000 00:00 0

(which is the followed by this one:)
2HPMEMMAPLINE b9bc8000-b9bc9000 ---p 0000a000 00:00 0

After reading the Diagnostics guide and deciphering the memory map output,
this looks like the memory region of a thread.

I have also collected core dumps from the boxes and used jextract/jformat
to view their status. The core dumps are often quite reduced (since the
process does receive a segmentation fault, this doesn't seem all that
surprising to me), but some are complete. Those which are complete
complain about finding an object on the heap that has the length of 0, but
I can't seem to find any information about what kind of object that is. My
initial theory about that is that the reference to the object doesn't
contain anything about what kind of class it is, and so I would have to
have the content to be able to decide what class it is. This is, according
to jformat, a memory corruption. I don't know what happened to the object,
and I'd believe this should never happen. By the way, I am running the
java with verbose compiling, and none of the memory addresses are ever
used by the code being JIT-compiled.

My questions are:

1) Does anybody have any good ideas on how to trace this? I have looked at
using the tracing features (outlined in the Diagnostics-guide), but since
this is a production system I haven't found anything suitable which
doesn't slow the system down a lot.

2) Could this have anything to do with JITC? I can't run the production
system without any JIT compiling, but should I go through testing SKIP on
all the classes one-by-one? We've tried to both reduce the threshold on
how much it must be used before being JIT-compiled/optimized (and even
tried to use FORCE on all code) without any luck at crashing it.

Btw, great of IBM of releasing the Diagnostics guide. It has helped us
tremendously, and is really a very handy tool.

Any comments? Ideas?

--
Thorkild - allergic to SIGSEGVs.

Neil Masson · Guest

Thorkild,

I am glad you like the Diagnostics Guide. I suggest that first of all
you upgrade either to JDK 131 SR6 or JDK 141 SR1. Also check to see if
there are any Oracle upgrades.

It looks like there is corruption in the Java heap. This could be
caused by
- an internal error in the JVM code
- a memory overwrite in native code
- shortage of memory

Check for shortage of memory first. On RHAS2.1, you will see
MEMMAPLINE entries going up from 0x4000000 and thread stacks coming
down from 0xc0000000. Is there a gap between the entries coming up
and those coming down? If not you have run out of address space. The
quickest fix for this would be to reduce the Java heap size to 740MB
when it will fit below the 0x40000000 boundary.

Next enable the heap verification option. Set the trace point
-Dibm.dg.trc.print=st_verify_heap
which will check the heap before and after each GC cycle.

Memory overwrites from native code are very hard to find. Does
Oracle application server use native code?

The JIT addresses relate to where JITed methods are located. These
are not Java objects and so will not match anything in the Java heap.
The Garbage Collector is written in C, so you are not running a
JITed method at the point of failure. However it is possible that
the JIT might be wrongly allocating an object. Try setting
JITC_COMPILEOPT=NALL to disable all JIT optimisations.

Hope this helps.
Neil

Thorkild Stray wrote:

G.S. Link · Guest

What threading model are you using? What happens if you put a
LD_ASSUME_KERNEL=2.2.5 in your profile? What happens if you try the Sun
142 jdk?

Thorkild Stray · Guest

On Wed, 19 Nov 2003 14:07:49 +0000, Neil Masson wrote:

Thorkild Stray · Guest

On Wed, 19 Nov 2003 10:00:24 -0500, G.S. Link wrote:

G.S. Link · Guest

If this were mine I would probably replace both the kernel and the jre.
The new IBM 141 jre fixes several bugs. The 2.4.20+ kernel is an
improvement and the latest Sun 142 works much better. Things are changing
so fast in this area that it is difficult to keep up so I haven't been
able to test some of this. The point of my questions was to discover
where this problem lies. Is it in your code? Is it in the IBM jre or is
it in Java? If the problem shows up in both the Sun and IBM jre then it
is either in your code, in Java or in your hardware. If it only shows up
in one jre then it is either that jre or hardware. If you are not running
ecc memory this could be hardware. IBM Vast tends to produce complaints
like this. In almost all cases they can be traced to the software
stressing the hardware. Have you eliminated this as a factor?

Thorkild Stray · Guest

On Thu, 20 Nov 2003 10:17:54 -0500, G.S. Link wrote:

G.S. Link · Guest

The only other question I have is whether you have ever encountered the
situation with this bug where attempting to use a debugger fixes the
problem? Since it doesn't appear in Sun code and since your hardware is
solid that implicates the IBM code. Since Java is forked and partly OO my
experience has been that threading is the first place to look when you get
one of these problems. The object that has a length of zero may simply
not be there. I ran into this the other day in Smalltalk. The instance
was destroyed by one thread before it could be used by another thread. Of
course, putting in a debugger made the problem vanish. Another clue is
the segmentation fault. Missing code almost always ends up as a
catastrophic failure of this type. When you write a jvm for a machine with
x processors and a nonstandard kernel that may be doing strange threading
you may have timing problems and these things are always hard to find. If
you can eliminate timing as a possibility it will make this problem easier
to find. There are simply not that many places in that jvm, outside of
timing, where such a problem could be. Incidently, when it comes to
kernels, I trust Linus further than Red Hat. Thank you for doing your
homework before coming here! That really helps!

Neil Masson · Guest

Thorkild Stray · Guest

On Thu, 27 Nov 2003 15:04:51 +0000, Neil Masson wrote:

G.S. Link · Guest

JVM 131 is used to drive WebSphere Developer Workbench. I find that your
problem appears in two of three machines causing the workbench to crash.
As you found out it doesn't happen often enough to find the problem.