I was having a similar issue.I was told by HTC CS that the best way to record is for me to open the message or mail, add attachment, video, from camera and start recording from there so I don't exceed the limit the message can handle. They also suggested I keep the Resolution of QVGA (320x240) and MPEG4 Encoding type.
It can be a pain to point and shoot quick video but now I leave my vid camera settings on the 320x240 setting just in case because chances are if I want to record something really nicely, I'll have time to adjust the resolution.
HTH!
Extra note, in this manner I am allowed a full minute of recording to send in texts before it says I'll reach my limit.
The camera will do a countdown backwards if you use this way instead of the standard counting forward in time.
You want max message size to be the 1.2 M. Mine's set for phone storage. Keeping the video resolution at the QVGA allows me to take quick video if I need to.
I cannot seem to get my Handcent to take video like I said before but if I record within about the same time as I would have for the standard messaging app, I can still attach and send to friends over text. I imagine the same for email although it's super rare I email a video. If you record through the video app itself and choose share, you can CHOOSE Handcent to make it easier to send rather than having to attach.