There is a lot of ongoing debate on the android community about task killers (apps that kill running applications and clear some RAM). People who seems to know the android system say that android already can free RAM automatically and efficiently (apparently this is especially true for Gingerbread and Ice cream sandwich) so basically you want most of your RAM full because it means your apps are already running in the background and you can call them really quickly when you need them.
That being said, there may be a difference between theory and practice. I would not be surprised that there is a threshold of free RAM to optimize efficiency (but is that 3MB, 100MB or more, I've got no clue). If there is no threshold, then 512MB is more than enough for current applications and the limitation is only the CPU speed and input/output data transmission.
For me, lag appears generally when I've been using the phone for some time so I would go for some free RAM is not bad.