Load testing a multi-threaded system vs one using separate processes - why is the threaded version so much faster?

I'm working on a system开发者_StackOverflow社区 where three high level components interact

Client(PHP) -- Logger (Java) -- MainBackend (Java)

The PHP client creates a new Linux Logger process for each request received. Logger then sends a message over TCP to the MainBackend and begins logging messages it receives from MainBackend. Note that Logger is very lightweight and uses very little memory.

I load-tested this system, by increasing the number of users, N, acessing the system. I then wrote a version where Logger was multi-threaded so that only one process was used for each N simultaneous users and load-tested the threaded version.

The results were that the multi-threaded version was FAR faster, as in many times faster past a certain N. My question is why?

If it takes a certain time T to start each Linux process, why amn't I seeing a constant difference (T2 - T1) between the two graphs?

Is Linux just far less efficient at scheduling processes than Java is at scheduling threads?

EDIT: An important point I didn't mention is that the timing was all done from within Logger, so the time to start the process / virtual machine is not affecting the results - I did the experiment this way so as to have as few variables as possible.

---

JVM does runtime optimizations, especially for code executed repeatedly. This will take some time, a warm up period. The result can be insanely faster. You can time your task repeatedly in the same VM, you'll see that it's slow in the beginning but gets much faster in the end.

If you start a new JVM process for each small task, no optimization kicks in, before JVM is terminated.

---

Processes are more heavy weight than threads. Using an existing process or existing TCP connection is much faster than creating a new one each time. This is true of all operating systems. You would only create processes or connections on demand if performance wasn't an issue for you.

You will see some variation in the time it takes to do most tasks because the machine is trying to do several things at once. You should always expect to see some variation.

---

This is not only about time of creating a process or thread but about context switching too. You can find some numbers here: http://wiki.osdev.org/Context_Switching, http://en.wikipedia.org/wiki/Context_switch.

In short, structure with information about process is bigger than structure with data about thread.