2GB barrier for a X64 app running on x64 CPU (Xeon 7650) with x64 OS (redhat 5.6) - why + what to check

I'm running the x64 version of some simulation app, on a very nice IBM x-server (4 8-core CPUs). The OS is Linux - redhat 5.6 x64 kernel. So this app crashes exactly when it needs more than 2 GB of memory (as evident from its own log files).

My question really is how to debug this issue - what relevant environment set开发者_JS百科tings should I look at? Is 'ulimit' (or sysctl.conf) relevant to this issue? What additional info can I post in order for you to help me?

---

This would be an application problem. Although the application is compiled as a 64-bit application, it still uses signed 32-bit integers for some things instead of proper pointers or the appropriate *_t types.

If you compile the application yourself, look for any "unsigned" or "truncated" warnings in the compilation output, and fix them.

---

The shmmax value defines the amount of memory that applications can use, you should check the value with this command:

cat /proc/sys/kernel/shmmax

If you need to increment, you can use:

echo 4096000000 > /proc/sys/kernel/shmmax

Bye