You need a lot of it. Most apps write important data to the card. Your nandroid backups are stored there, backup app data, utilities, histories; if you've bought premium versions of free/paid apps, the license info is on the card, etc. It's impossible to answer your question without your phone in hand to examine, and even then, I myself don't have sufficient expertise to state definitively what you need to keep and what you don't.
You can safely decide which photos, videos, and download files you don't need and get rid of those, but I'd advise caution in that you're better off copying too much of the rest than not enough.