Working with Huge ArrayLists, Java OutOfMemoryError: Java heap space... use db?

I am having a memory problem that I completely understand the cause, but have no idea of a fix. I have attempted to use the -Xmx2g tag and make the heap size larger, but there开发者_开发问答 seems to be a hidden maximum. (if i used -Xmx512m I run out of space at the same time).

Assume I have 2 objects, an Area and a User. My Area object holds an ArrayList of users:

public class Area {
      int numUsers;
      ArrayList<User> userList;
}

My User class holds an ArrayList of friends:

public class User {
      int userID;
      int numFriends;
      ArrayList<User> friends;
}

Just using a single Area, with 1 million users, and an average of 200 friends per user, I run out of heap space after about 680,000 Users are created. Obviously if I lower the average number of friends/user to something closer to 100, I can store all of these objects in the heap.

What if I want to simulate 2 million Users in an area? Or Hundreds of area's?

With this much data, is a database the only feasible way to do simulations using the information?

---

Disk/database is only a solution if you can afford the factor 100.000 loss of random access performance (you might, there are lots of systems using a database). You can do much better with specialized data structures. Doing something special for fully connected subnetworks might save a lot of space.

---

You can, of course, run w/ more than 2m on 64bit java but that will not solve the issue. Btw, for Area, you probably need id (not numUsers), the number of users/friends can be obtained from list.size()

Database/disk storage is a natural solution for representing a lot of object, you can alternatively use cluster of servers (beside running a huge box w/ 500+GB of memory)

To answer the question you have to supply some more data: what is the point of areas/ friend graphs/etc.

---

If you can code your own struct(ure) using ByteBuffer (which is probably not an easy task) you can go beyond the 32bit limitations by java.io.MappedByteBuffer, ScatteringByteChannel/GatheringByteChannel. However, it's not a rookie task at any rate but if you like programming, I'd advise to try your hand at.

I wish you good luck with your studies.

---

To access more heap memory, move to a 64-bit OS and 64-bit JVM. If you are hitting problems with -Xmx512m, you are using 32-bit OS and/or JVM.