The other thing is going to be getting the reviews to be unbiased and
objective.
I have seen millions (yes, literally millions - I am 8 months away from 40) of subjective reviews in my time, and I cannot count (anymore) the number of times I have followed someone's advice, only to find out their either 1) a moron, or 2) very differently aligned than I am, or 3) all of the above.
The ideas that UBR provided sound like a good start - but there is one other *critical* factor to consider.
Let's choose my phone, for example. DROID purchased from VZW over the phone (using my then current Samsung Alias, btw) on 6 Nov and arrived on my doorstep 7 Nov. Now, I didn't root until Jan (back injury less than 4 days after getting the phone, wasn't able to do much other than read a ton of Manga and play some games), but I can tell you this much:
The 2 most stable ROMs on my phone since I rooted have been:
1) Ultimate DROID from 7-9
2) Pete's Bugless Beast back before 2.1 was out and now again with FroYo
(note - is it any surprise that these ROMs, the original and the clone, both are the most stable ones?
)
Also, the base FroYo releases have been very kind to me in terms of stability.
Other ROMs that I have tried have sucked on my phone for whatever reason. For example, Cyanogen (and I know I am going to be lambasted for this, but I hope some of you will realize that this is my honest *objective* evaluation of ROMs on my phone) never works right. At some point in time apps start FCing (SportsTap is a major bad one here) and at other times my phone goes unresponsive. This happened with numerous 5.x releases and again last night with the 6.0 RC1.
However, I know plenty of people (including Mel Reyes, of Hacking Firefox fame) who are using and
loving Cyanogen Mod.
Same holds true for kernels - Taz's old kernels, then ChevyNo1's kernels, and finally, now, jldfg's kernels, all allow me to run at 1200 MHz without issues.
Most others allow me to run only up to 100 MHz....
So, that all being said, how do you plan to keep objectivity in these reviews?
Actually, I suppose the first question should be:
Do you plan to keep the reviews objective or subjective?