You should be able to....
(1) - Open the Gmail app.
(2) - Menu -> Settings
(3) - Choose your account that you want to change notifications for. If you only have one then it will be the middle option between "General Settings" and "About Gmail"
(4) - Just under "Email Notifications" there will be a "Ringtone and Vibrate" option, click that and you can then change the ringtone to "None"
That should do it.