Are software isolated processes superior to virtual memory for process isolation?

Question

So I've been building a small OS and thought a bit about the paging/virtual-memory concept. I believe it provides two distinctly different benefits:

(1) Just like how the processor has onboard cache, we can view memory as just a cache for the much larger hard disk. Page fault and disk swap is analogous to a CPU cache miss.

(2) It gives each process the illusion of having access to the entire address space, while isolating it so it cannot access or corrupt the memory of other processes, the kernel, etc.

So I get the point of #1. Though it is much less relevant today, now that, for example, my desktop has 32GB of ram.

But, #2 I have a several problems with:

(1) The first problem is that it increases the cost of a context switch because much of the TLB is probably flushed out when the new page directory is established. This means accessing virtual memory is probably much slower than just accessing physical memory directly.

(2) Virtual memory only solves a subset of the memory safety problem. You can still write a program that corrupts itself in all kinds of ways, but thankfully virtual memory protects the other processes.

A much better approach to achieve memory isolation would be if the kernel actually examined every program that it loaded. If the instructions were all memory safe, then I believe we wouldn't need virtual memory (correct me if I'm wrong). We could use plain old physical memory, with faster memory access, faster context switching, and faster data sharing, and less physical hardware. (I realize that x86 assembly cannot be used for this, it would have to be a higher-level assembly).

This approach actually has a name "Software Isolated Processes" or SIP and it looks like Microsoft experimented with this a while ago with project Singularity.

So my question is: are SIPs superior, or is there something that I'm missing about the benefits of virtual memory?