Linux: what is the most scalable design for making a system call like fadvise in a thread?

My server has the following requirements:

1) each new connection to the server will trigger a series of N posix_fadvise calls. 2) the first few fadvise calls per connection should ha开发者_StackOverflow社区ppen ASAP 3) ability to re-order the fadvise calls if the client makes a subsequent requests.

I am thinking: thread pool with shared queue, where thread pool size is ~100. Any other suggestions?

---

Assuming you are talking about POSIX_FADV_WILLNEED:

posix_fadvise is already asynchronous. That is, it fires off the kernel's machinery to start paging in data in the background, but it does not actually wait for any data to get read. It returns immediately.

In other words, posix_fadvise is already a concurrency mechanism, so spawning your own threads to invoke it is non-sensical. And there is no way to "re-order" the calls, because once they have been handed to the kernel, the kernel will make up its own mind about how to re-order the disk accesses.

If you really want to roll your own, just have your threads making blocking read() calls over and over to read small-ish blocks (like 8k). (That is, read sequentially into the same 8k buffer repeatedly. Using the same buffer will keep it in L1 cache and avoid hammering the memory bus needlessly.) That will populate the page cache and give you some control over when it happens.

I am a little skeptical that your application needs such a mechanism, though...

---

There is no point in having multiple threads blocking in fadvise at the same time for the same underlying device, since they're all sharing the same request queue anyway.

This means that you should only need a single readahead thread, that takes readahead requests from a queue and executes them sequentially.