stack overflow method

In some operating system,for any process there is a stack and a heap.Both grows towards each other.There must be a guard band between them to check for overlapping.Can anyone give m开发者_如何学Ce some illustration about it.I want to write my own function for checking stack overflow error.

---

In a system like that, you would normally have a guard word or something similar at the top of the heap, something like 0xa55a or 0xdeadbeef.

Then, periodically, that guard word is checked to see if it's been corrupted. If so something has overwritten the memory.

Now this may not necessarily be a stack overflow, it may be a rogue memory write. But, in both those cases, something is seriously wrong so you may as well treat them the same.

Of course, more modern operating systems may take the approach of using the assistance of the hardware such as in the Intel chips. In those, you can set up a stack segment to a specific size and, if you try to write outside of there (using the stack selector), you'll get a trap raised.

The heap in that case would be using a different selector so as to be kept separate.

---

Many operating systems place a guard page (or similar techniques) between stack and heap to protect against such attack vectors. I haven't seen canaries (the method mentioned by paxdiablo) there yet, they're mostly used to guard against stack-internal overflows (aka to guard the return address).

Guard pages on Windows: http://msdn.microsoft.com/en-us/library/aa366549(VS.85).aspx

Linux had an interesting exploit based on this problem some time ago though: http://www.h-online.com/open/news/item/Root-privileges-through-Linux-kernel-bug-Update-1061563.html