You'd have to re-root to remove any file that is installed in /system/xbin (i.e., busybox).

My busybox (on my N5) is also installed in /system/xbin (owned by root/shell) and it would take the ability to remount /system in read-write mode in order to remove it--which, of course, requires root access/capability.

Also, I think I'd let the Busybox app installer itself do the removal vs. trying to do it manually.