I totally see (and to an extent) can agree with where you are coming from. I just look at it as something that there are already in systems in place to prevent.
To get to that point someone would have to unlock the phone first. If the phone is locked (or set to lock after pressing the power button or xx minutes of inactivity) then you can't access the ICS Task Manager. That is, our privacy is as much as we want it to be. If you aren't worried, don't make a lock. If you do care or are worried, use one of the lock methods.