[A] In Xinu, back-to-back layout of process stack memory implies that stack overflow in one process may corrupt the process stack of another. Without MMU (memory management hardware) support, it is difficult to prevent stack overflow corruption. To what extent can increased kernel vigilance such as checking of stack pointer values during context switching help avoid stack overflow? Can such a solution guarantee that stack overflow does not occur?
[B] An alternative approach that does not require hardware support is compiler based stack overflow prevention. Keep in mind that the primary goal is to contain the impact that an ill-behaving (by bug or malicious intent) process can have on other processes. For example, a compiler might inject extra range checking code when compiling function calls with the aim of detecting stack overflow at run-time before it occurs. Is this approach guaranteed to work? How does it compare against the kernel based run-time "stack monitor" approach? In your assessment, how do their overheads compare?
[C] What happens to the stack overflow problem if processes are laid out in a more traditional approach with each process having its own text, data, heap, and stack segments, with the memory between heap and stack being free?
Suppose two Xinu processes P1 and P2 have back-to-back stacks, with P2's stack sitting on top of P1's stack (i.e., overflow of P1's process stack will corrupt P2's process stack). Let us ignore for the moment that in our version of Xinu there is no separation between user space and kernel space — except in a conceptual sense — and even though there are per-process run-time stacks, they can also be read/written by any process in the system due to lack of elementary protection. Suppose P1 was written by a malicious user who wishes to "hijack" P2 for its own purpose (e.g., running part of its own code so that it monopolizes the CPU).
[A] Assume that P1 somehow knows the location of the return address in P2's process stack of the top-most function running in the stack. Suppose P1 overwrites the return address with an address that branches to its own code, for example the beginning of the function my_malicious_code(). Will doing so allow P1 to run my_malicious_code() without side effects? If not, what must P1 additionally do to hijack P2's run-time context so that its function my_malicious_code() runs correctly under P2's context?
[B] Suppose that P1 does not know where return addresses of P2's function calls in P2's process stack are located, but it still wants to hijack P2 for its own purpose by overwriting P2's stack making only use of the fact that P2's process stack follows P1's process stack. What might be a way for P1 to succeed without detailed knowledge?