After you take a picture you'll see a thumbnail of it appear in the upper left corner. Tap that and then select the trashcan on the next screen.

I'm not sure what to tell you about lag time. Mine is almost instant.

 

If you want to be able to delete the picture as soon as you take it, then you need to make sure that Review is turned on. Open the camera app, click on the gear icon, and then click the larger gear that appears. Go to the 3rd tab under settings and turn Review to On. You will now have the option to delete pictures as you take them. Also, to decrease lag, make sure you tap to focus before taking the shot. You can also change your shooting mode to Sports which will increase the shutter speed.

 

Mine is pretty quick. It takes longer time to focus in lower light.
I think everyone expects too much out of camera's on a phone. I have a few camera's but only one takes instant photos and it cost over $2000.

Try the sports mode. Should give you quicker shutter speed as it's there for action shots. You will sacrifice quality though.

 

I've also been a little frustrated with the shutter lag on the S4. I was standing next to an iPhone user at a concert recently and was impressed how quick the iPhone was to snap a photo, almost instantaneously after touching the screen, compared to my S4.

If you're having trouble timing the photo just right, you might consider turning on Burst Shot in the main Camera settings, which takes a burst of shots as long as you hold down the button. This is good for fast-moving kids and sports when you're hoping to get one exact moment. [Note that those photos by default get saved to internal memory, not your SD card.]

While you're in the Settings menu turning on Burst Shot, you may also want to make the volume button your camera shutter button. That way you aren't awkwardly trying to hold the phone perfectly still while tapping the screen to take a photo. You can change this by going into Camera, Settings, gear, Volume Key = the camera key.

Another related option is "Best Shot" Mode, which takes 8 snaps in rapid succession and asks you to pick one to keep.

 

best i've found to reduce this lag is to hold the camera button, then release when you want to take the pic.