The easiest way to do it would be to get the advanced task killer. Search for it in the market, it's free.
Once it installs, open it up and it'll give you your selections, make sure music is checked, then click kill selected apps. As long as ATKiller is checked it will also kill the app so it's not running in the background.
There's a more complicated way to kill the music player, but i figure you want to easy way, so...yea.
As far as deleting music... Menu>all programs>music. Click the button on the bottom left corner of the screen that looks like 3 lines. Find whatever you're looking to delete and long press it until the pop up comes up. click delete and confirm. You're done.
An easier way to do this would be to mount it to your pc and add/delete whatever music you want...