Oh yeah. I do this all the time. To be precise, the prepaid data SIM I use are actually for those 3G data dongles usually of Huawei and ZTE manufacture. These data dongles got a cap which you can pull out, and insert a SIM into it. You purchase and load the SIM via text. This is quite common in Asia and Europe.
You can use such a SIM in your (unlocked) Android smartphone. It literally turns your phone into a mobile data only device. No text or voice, but you can get unlimited internet for a few days depending how on the package plan. I find this very useful for traveling, since the phone can be used both as a mobile wifi hotspot and internet device. I have to carry a second (unlocked) phone though, just for the voice and text.
CDMA is activated with a plan attached to the phone IMEID. GSM however, usually is not. The contract is based on the SIM.
But it's always good to inquire with your carrier.
Note, you can't use iPad micro SIM on an Android smartphone, which takes in regular SIMs. If the data SIM is for a Galaxy Tab, then you can fit it on an Android phone. Likewise, I've used regular phone SIMs on the Galaxy Tab, which lets you get voice and text off the Galaxy Tab like a giant phone.