I've read up on programming for Android (not actually got round to doing it yet though!) and as mentioned, apps get killed when memory/resources are needed. There are different parts to applications: there are activities (usually screens you interact with), as well as services which run in the background. (There are not the only types, but will do for now)
The example usually given is an mp3 player - just because you've chosen an album to play, doesn't mean you should have to look at the music player screens whilst the album plays, you might want to browse the web or send a text. So the actual music playing part is handled by a service that can run without being displayed to the user. The music player UI though is an activity that the user interacts with - in the background though, it's using the service to actually play the music.
As for apps remembering their state - someone's already mentioned a finish() call. Basically, apps can be in a number of different active and non-active states, and usually the application is told what state it's being moved into. So if a user presses the home key, the application is told "Hey, you're not going to be active any more" - the application can then decide what to do, for example, write some data somewhere, capture some text the user has typed in a field. Or, ignore it completely!
Hope that makes sense (even though it's probably not 100% accurate
)