Solaris Core dump analysis

A Solaris server running Glassfish keep crashing in my company and even the Oracle consultant has no clue, after he analysed the core dump files.

I have no idea on how to tackle the problem either. But it happen again and again and I decided that it is time for me to learn how to analyse core dump files.

Of course Google is my best friend for a task like this, but some knowledge of reverse engineering helps alot here, without of which might mean getting nothing out even when presented with the correct information.

pflags

Run pflags and look for signals that caused a crash. Here I found a SIGSEGV signal.

pflags core.hostname04.703.26100.java > pflags.txt

and I do a grep and found in the file:

/243:  flags = DETACH
       sigmask = 0xfffffeff,0x0000ffff  cursig = SIGSEGV

The crash occured in thread number 243.

pstack

pstack core.hostname04.703.26100.java > pstack.txt

and look for 243. I got :

-----------------  lwp# 243 / thread# 243  --------------------
ffffffff7dda9840 jni_GetByteArrayRegion (1111471c8, fffffff80b6155c8, 0, 108, fffffff80b6054f8, 111147000) + f8

The function jni_GetByteArrayRegion is causing the crash. It is located in memory address ffffffff7dda9840.

pmap

pmap core.hostname04.703.26100.java > pmap.txt

portion of the pmap.txt:

....

FFFFFFFF7DB12000         16K r----
FFFFFFFF7DC00000       8192K r-x--  /opt/jdk1.6.0_24/jre/lib/sparcv9/server/libjvm.so
FFFFFFFF7E400000       2048K r-x--  /opt/jdk1.6.0_24/jre/lib/sparcv9/server/libjvm.so

........

So the culprit is /opt/jdk1.6.0_24/jre/lib/sparcv9/server/libjvm.so

I found an easier way to locate the executable that contains the offending function. I will write about it in a later post.

Since the library (/opt/jdk1.6.0_24/jre/lib/sparcv9/server/libjvm.so) is part of the Java Runtime Environment, I have filed a case with Oracle and they are currently investigating. I saved them some work infact :)

Have a nice day :)

fook sheng

Microsoft Windows memory management

Some notes on Microsoft Windows Memory Management (limited to 32 bit OS)

• Maximum memory available is 4G (because Microsoft is dumb enough to ignore segment registers. See http://social.technet.microsoft.com/Forums/en/w7itproperf/thread/9feee84c-44f8-48fa-b49f-9199473e3006)
• 32 bit applications can access up to 4G of memory but in actual fact it can only access 2 G, which is known as the user mode virtual address space (user space)
• User mode address space range from 0000 0000 - 7FFF FFFF
• The other 2G of memory is known as kernel mode address space (system space) and it is reserved for the OS
• Kernel mode address space range from 8000 0000 - FFFF FFFF
• System space contain the global OS code and data structures that are visible to each process

References

http://blogs.technet.com/b/askperf/archive/2007/09/28/memory-management-x86-virtual-address-space.aspx

NFS on MS Windows Server 2003

I am trying out Zen Cloud and a need for NFS as a shared resource arise. My colleague has a Windows server running and i found out that I can run NFS server on the Windows 2003 server.

http://blog.peacon.co.uk/wiki/Creating_an_NFS_Server_using_Windows_Server_2003


In short you need Services for Unix (SFU), which can be downloaded at:

http://www.microsoft.com/downloads/details.aspx?displaylang=en&FamilyID=896c9688-601b-44f1-81a4-02878ff11778


The version I had is Windows Services for UNIX Version 3.5.


Happy New Year and cheers :)