Root ROM Rating system in this subforum?

[YankeeDudeL]

It's def a great idea. I think we could nip a few probs in the bud by having some form of reputation system implemented, but I fear that may be opening a new can of worms. You guys are smart, though, you'll figure something out!

 

[UBRocked]

The other thing is going to be getting the reviews to be unbiased and objective.

I have seen millions (yes, literally millions - I am 8 months away from 40) of subjective reviews in my time, and I cannot count (anymore) the number of times I have followed someone's advice, only to find out their either 1) a moron, or 2) very differently aligned than I am, or 3) all of the above.

The ideas that UBR provided sound like a good start - but there is one other *critical* factor to consider.

Let's choose my phone, for example. DROID purchased from VZW over the phone (using my then current Samsung Alias, btw) on 6 Nov and arrived on my doorstep 7 Nov. Now, I didn't root until Jan (back injury less than 4 days after getting the phone, wasn't able to do much other than read a ton of Manga and play some games), but I can tell you this much:

The 2 most stable ROMs on my phone since I rooted have been:

1) Ultimate DROID from 7-9
2) Pete's Bugless Beast back before 2.1 was out and now again with FroYo

(note - is it any surprise that these ROMs, the original and the clone, both are the most stable ones? )

Also, the base FroYo releases have been very kind to me in terms of stability.

Other ROMs that I have tried have sucked on my phone for whatever reason. For example, Cyanogen (and I know I am going to be lambasted for this, but I hope some of you will realize that this is my honest *objective* evaluation of ROMs on my phone) never works right. At some point in time apps start FCing (SportsTap is a major bad one here) and at other times my phone goes unresponsive. This happened with numerous 5.x releases and again last night with the 6.0 RC1.

However, I know plenty of people (including Mel Reyes, of Hacking Firefox fame) who are using and loving Cyanogen Mod.

Same holds true for kernels - Taz's old kernels, then ChevyNo1's kernels, and finally, now, jldfg's kernels, all allow me to run at 1200 MHz without issues.

Most others allow me to run only up to 100 MHz....

So, that all being said, how do you plan to keep objectivity in these reviews?

Actually, I suppose the first question should be: Do you plan to keep the reviews objective or subjective?

I'm not seeing the issue John

What is not objective about your comments here? You are not favoring one developer over another because of any criteria except for how the ROM or Kernel ACTUALLY runs on your phone. So CM doesn't is not as stable for you as it is for me...that's a good thing in terms of reviewing the ROMs. If 5 people do a review and everyone has exactly the same experience...WITH EVERY SINGLE ROM/KERNEL...what's the point in having multiple reviewers?

If 5 people do a review and 5 people are blown away with battery life...someone that is looking for the best battery life will be very interested. ROMs like CM that often times for me do not run as fast as a ROM like BB, but provides other features like full screen option in the stock web browser (an important feature for me). Features vs. speed vs. stability are good comparisons to make. Saying that CM is the BEST ROM, is subjective and not all that helpful. Saying that CM is the best ROM for me because it incorporates different features that are important to me is what users would want to see (or so I imagine) so they could make a decision depending on how they feel in terms of features vs. speed vs. stability.

Saying CM 0.6.0 RC1 has more FC's than BB v0.4 is objective. Listing key features that CM has that BB does not (and vice versa) is objective. Other things like benchmarking is objective, where as battery life is somewhat subjective in nature (but if reviewed by multiple people, it could become a consensus opinion and therefore provide legitimacy to that subjective observation).

The key here is to have an agreed upon starting point. Wipe Boot, Data, System, and Cache partitions before installing. Provide a synopsis with the stock kernel first, then install a "user preferred" kernel that you would use normally (obviously, you would want to use a kernel that is @ whatever frequency you normally use). This would give an insight to users about various OC'ing kernels along with the ROM. Is this subjective...kind of...but could be extremely helpful as well.

The goal is not to be the United Nations here...it is to provide as objective evaluation as possible...but opinions, like the way the ROM "feels" to us are important as well. For example, BB0.4 feels smoother when scrolling through my 7 screens with Launcher Pro than CM does (true statement for me). Some users HATE any feelings of "lag" on their phone). As with anything we do with these phones...YMMV

 

[johnlgalt]

I'm not seeing the issue John

What is not objective about your comments here? You are not favoring one developer over another because of any criteria except for how the ROM or Kernel ACTUALLY runs on your phone. So CM doesn't is not as stable for you as it is for me...that's a good thing in terms of reviewing the ROMs. If 5 people do a review and everyone has exactly the same experience...WITH EVERY SINGLE ROM/KERNEL...what's the point in having multiple reviewers?

If 5 people do a review and 5 people are blown away with battery life...someone that is looking for the best battery life will be very interested. ROMs like CM that often times for me do not run as fast as a ROM like BB, but provides other features like full screen option in the stock web browser (an important feature for me). Features vs. speed vs. stability are good comparisons to make. Saying that CM is the BEST ROM, is subjective and not all that helpful. Saying that CM is the best ROM for me because it incorporates different features that are important to me is what users would want to see (or so I imagine) so they could make a decision depending on how they feel in terms of features vs. speed vs. stability.

Saying CM 0.6.0 RC1 has more FC's than BB v0.4 is objective. Listing key features that CM has that BB does not (and vice versa) is objective. Other things like benchmarking is objective, where as battery life is somewhat subjective in nature (but if reviewed by multiple people, it could become a consensus opinion and therefore provide legitimacy to that subjective observation).

The key here is to have an agreed upon starting point. Wipe Boot, Data, System, and Cache partitions before installing. Provide a synopsis with the stock kernel first, then install a "user preferred" kernel that you would use normally (obviously, you would want to use a kernel that is @ whatever frequency you normally use). This would give an insight to users about various OC'ing kernels along with the ROM. Is this subjective...kind of...but could be extremely helpful as well.

The goal is not to be the United Nations here...it is to provide as objective evaluation as possible...but opinions, like the way the ROM "feels" to us are important as well. For example, BB0.4 feels smoother when scrolling through my 7 screens with Launcher Pro than CM does (true statement for me). Some users HATE any feelings of "lag" on their phone). As with anything we do with these phones...YMMV

Problem is that it is still subjective, b/c as we both know, ROMs and especially kernels are phone (hardware) specific. Your DROID may like running CM 6 RC 1, but that doesn't mean that across the board it is going to run the same, give the same battery life and run the same using the same kernel.

The subjectivity versus objectivity can be seen in this example: Reviewer A likes CM and uses P3's kernels, stating that the ROM had absolutely no FCs, ran smoothly all the way down to 20% battery lie, and average battery life was 8-10 hours. My very first question is going to be "What apps are you running?"

Why? B/c I have had a very horrible time with CM ROMs. Although I know that Cyanogen is one of the granddaddies of ROM-from-source developers, and I know that his stuff is usually rock solid, I know also that one or more of the apps I *choose* to use is causing some sort of incompatibility. Take that with the fact that on my phone, Pete's runs pretty solidly without any funky problems, and you get me saying the same thing about Pete + jdlfg kernels that Reviewer A says about CM + P3.

So, now the question becomes, again, "Which ROM is right for me?"

However, the whole point of a review is subjective, to begin with, something that I 'conveniently' forgot. So my analysis is mostly moot (but not to me...)

 

[inssane]

I can tell you that I was a BB whore from 2.0.1, and right now, I find myself trying ROMs and simply running them based on overall stability, speed, and battery.
Scripts are nice, so are CMod features - but the bottom line is the ROM running well. I have learned to be more and more objective over time with these ROMs and even though I like Kangerade right now (most stable I have had on Froyo), I will always try something new.

Funny thing is that I never looked at JRummy with 2.1, so she's kinda my rebound off of BB - but I am in love again (for now)

 

[YankeeDudeL]

I can tell you that I was a BB whore from 2.0.1, and right now, I find myself trying ROMs and simply running them based on overall stability, speed, and battery.
Scripts are nice, so are CMod features - but the bottom line is the ROM running well. I have learned to be more and more objective over time with these ROMs and even though I like Kangerade right now (most stable I have had on Froyo), I will always try something new.

Funny thing is that I never looked at JRummy with 2.1, so she's kinda my rebound off of BB - but I am in love again (for now)

I can def relate. BB was the reason I wanted to root and used it from 7.4 to 7.9. Then I started experimenting w/ other ROMs, before unofficially settling on UD (yanno, before it proved to be an evil incarnate of other ROMs..lol). I just wanted my Droid to run as efficiently as possible, and that's what's most important today. I love CM's goodies, and BB has added some nice features as well, but I will ultimately go w/ whatever is smoothest (I'm impatient, lag is the enemy).

Right now I'm currently flitting between 3 diff ROMs: BB, CM, and SD (Succulent Desire, for noobies). Basically, the latest builds of those have proven to be the best for my Droid over what's currently available. We've had enough downtime between new ROMs that I've been able to give enough to properly evaluate which I like best.

 

[nstallion]

What if I want to check both answers? They aren't mutually exclusive!

 

[UBRocked]

What if I want to check both answers? They aren't mutually exclusive!

That cracked me up!

Okay, so we are hovering around 90% of the people that have voted want to see some sort of rating system...so I am determined to make a go of this.

I have had a few people bring up concerns over why this won't work...but haven't heard anyone that is totally against the idea...

Only big problem at this time is manpower. I haven't received any PM's from anyone wanting to volunteer to give reviews. If people want to get this going, we are going to need at least 15 people (in my estimates) that are willing and able to do reviews. I don't want to "recruit" for this so if you are interested, PM me

 

[UBRocked]

I am also thinking about incorporating a poll for each new ROM as it is released. We could run the poll for a set time or indefinitely (thoughts welcome here ). This is a double check to help ensure "everyone's" voice is heard as well as having more detailed "expert reviews" in the second post.

Options for the poll could include something like this:

5 Star ROM Rating: Elite
4 Star ROM Rating: Outstanding
3 Star ROM Rating: Average
2 Star ROM Rating: Not Ready for Primetime
1 Star ROM Rating: Not Usable

Edit: We could use "active polls" instead of using "stickies" for each New ROM (an idea I am not crazy about because it may look cluttered and distract from other potential stickies...like the two "How to's" posted by OMJ right now). If a new ROM is released, the poll for the previous release could have the voting suspended (IE...no active poll anymore = outdated release). I like this idea...any thoughts?

 

[supersaki]

Now I just want to know who thinks I am an idiot now.
LOL

I think it's a good idea, but I voted the 'idiot' choice just because it was there.

20 reviewers might be a good number to have, but I don't think that 10 or 20 people need to review each ROM for this to be successful (in fact, I think that would be counter productive as most people may not read more than the first couple of reviews to "get the gist". If we could get at least 2 and no more than 5 reviews per ROM, I think that would be sufficient. This way, people would not feel obligated to flash every ROM that comes out in order to provide a review (just not feasible or fair to ask that of anyone).

I agree. 20 is overkill. 3 would probably be a nice average.

I think all reviewers should come from a different 'ROM of choice'. (ie, not all reviewers running BB, or CM.) Along these lines, I don't think any comparisons should be made to other ROMs when doing the review, except for maybe stock 2.1 (or 2.2 when it comes out OTA). The reviewer can maybe add something in a 'closing comments' part of the review, but otherwise, the ROM should stand on it's own.

 

[inssane]

That cracked me up!

Okay, so we are hovering around 90% of the people that have voted want to see some sort of rating system...so I am determined to make a go of this.

I have had a few people bring up concerns over why this won't work...but haven't heard anyone that is totally against the idea...

Only big problem at this time is manpower. I haven't received any PM's from anyone wanting to volunteer to give reviews. If people want to get this going, we are going to need at least 15 people (in my estimates) that are willing and able to do reviews. I don't want to "recruit" for this so if you are interested, PM me

Why don't you throw up a thread asking for volunteers. I would just recommend that they have a decent post count with past posts relevant to ROMming. And ask them to PM you - know a number of people that you would need - and tell people when the decision will be made.
Maybe that would work.

 

[johnlgalt]

I am also thinking about incorporating a poll for each new ROM as it is released. We could run the poll for a set time or indefinitely (thoughts welcome here ). This is a double check to help ensure "everyone's" voice is heard as well as having more detailed "expert reviews" in the second post.

Options for the poll could include something like this:

5 Star ROM Rating: Elite
4 Star ROM Rating: Outstanding
3 Star ROM Rating: Average
2 Star ROM Rating: Not Ready for Primetime
1 Star ROM Rating: Not Usable

Edit: We could use "active polls" instead of using "stickies" for each New ROM (an idea I am not crazy about because it may look cluttered and distract from other potential stickies...like the two "How to's" posted by OMJ right now). If a new ROM is released, the poll for the previous release could have the voting suspended (IE...no active poll anymore = outdated release). I like this idea...any thoughts?

Have mods modify the OP threads when a new ROM is released to make a new Poll for that ROM, allowing voting to take place, and stickify it only until a new ROM is released, or it falls out of favor for whatever reason.

I know the stickies may be a distraction, but if the admins would support a ROM polls sub forum, then it would not be a distraction at all, it would be very unobtrusive, with links to he polls in the threads and the polls themselves in a completely separate sub-forum, complete with its own stickified posts.

 

[UBRocked]

Have mods modify the OP threads when a new ROM is released to make a new Poll for that ROM, allowing voting to take place, and stickify it only until a new ROM is released, or it falls out of favor for whatever reason.

I know the stickies may be a distraction, but if the admins would support a ROM polls sub forum, then it would not be a distraction at all, it would be very unobtrusive, with links to he polls in the threads and the polls themselves in a completely separate sub-forum, complete with its own stickified posts.

I liked the idea of using stickies for about a day...then rethought things. Couple of points...

1) If you add another sub-forum, that is yet another place that users have to go for information. I don't know that adding another layer is a good idea (plus, I doubt we would get Mod/Admin support for the additional sub-forum).

2) Adding stickies to the ROM Forum is not really necessary and will distract away from current stickies (the sticky on "how to flash a custom rom" that OMJ made really needs to be an attention getter, especially for the noobs). Also, we have our own ROM subforum for nothing other than ROM releases. If they don't stay on the first page, it is an old release or a VERY unpopular ROM. The current releases will be designated with an active poll and should be towards the top of the sub-forum anyway.

For now, let's plan on running the rating system in conjunction with the active polling (no stickies). We can always take a step back and evaluate what works well, and what doesn't. Doing it this way ensures that we can get this up and running without much if any Admin/Mod intervention. Once it's established and proves it's value, if we need to bring in the big guns, we can with a "here's what we've accomplished thus far, and here are the shortfalls" type of approach.

I'm not saying that my way is the best solution...but I think it's the best solution for now.

 

[inssane]

So it's late, and maybe I am just missing something or to tired right now, but...
If no subforum, no sticky - there will be an active poll for each ROM being rated, correct?
So if there are 10 ROMs, lets say, each one has a poll in the ROM subforum.
Popular ones stay up and others go down to the next page - but will these be open to everyone or just for reviewers.
I don't mean to muddy things up, and I am not expecting a concrete solution, because this is a WIP - I just need a little clarification on your implementation thoughts.

I agree another subforum is tedious for the community to navigate to UNLESS it is within the ROM subforum.
Stickies aren't terrible, but they will clog the first half of the page regardless.

Too bad we can't have ONE sticky that contains all of the reviews so that people could see them side by side, basically. The logistics of that actually working blow my mind right now, LOL.

Thanks for all the hard work people, I think we can make it work, and I am sure people will appreciate it - including developers.

 

[UBRocked]

So it's late, and maybe I am just missing something or to tired right now, but...
If no subforum, no sticky - there will be an active poll for each ROM being rated, correct?
So if there are 10 ROMs, lets say, each one has a poll in the ROM subforum.
Popular ones stay up and others go down to the next page - but will these be open to everyone or just for reviewers.
I don't mean to muddy things up, and I am not expecting a concrete solution, because this is a WIP - I just need a little clarification on your implementation thoughts.

I agree another subforum is tedious for the community to navigate to UNLESS it is within the ROM subforum.
Stickies aren't terrible, but they will clog the first half of the page regardless.

Too bad we can't have ONE sticky that contains all of the reviews so that people could see them side by side, basically. The logistics of that actually working blow my mind right now, LOL.

Thanks for all the hard work people, I think we can make it work, and I am sure people will appreciate it - including developers.

Yeah, this is the plan as of now:

No Stickies
No Sub-Forum specific to ROM Reviews

Each new ROM release will have an active poll so everyone in the community can vote on the ROMs.

The first available post not taken up by the developer (second post in most situations) will be reserved for the expert reviews. This way, when you open the ROM's thread, you will immediately see the "popular vote" followed by the ROM's release information, followed by expert reviews, and after that would be all of the normal posts discussing the ROM.

As you said, it's a WIP. This makes the most sense to me for a system that is as unobtrusive as possible and still provides a means for showing which releases are current, community reviews, and more detailed expert reviews.

An issue I'm still thinking through is what to do with a release when it becomes an "old" release. I saw a thread made by Fabolous that had showed it had a poll (it was for NexFro 1.1 I believe). When I opened up that thread, voting was closed for that poll. I don't want to delete the old ROM's polls but I don't want it to show that it has a poll from the main ROM Sub-forum page. I would like to change the thread icon to something we could use to signify that it is no longer current (like the icon used for this post).

Please continue to look at this from different angles and see if something doesn't make sense or seem right.

Edit: We could put together a sticky thread explaining the ROM reviewing system, as well as a "key" for understanding what signifies a current ROM release, an old release, etc. As always...just spitballin

 

[inssane]

OK that sounds good. I am still a little worried about people thrashing the polls because of loyalty to other developers, but I think if we did the scale you mentioned earlier rather than a "good" or "bad" poll, we should be OK.
Even though I am sure people may want to discuss old ROMs from time to time - I think closing a thread when it's old and letting it sink is the best way to do it.

I, and the developers most likely, would appreciate if people would post why they DIDNT like a ROM if they voted negatively. I can't stand 10 bad votes, but nothing is mentioned in the thread itself as to why.

 

[gnath9]

what about a sticky for "Old Roms" wich simply displays the overall rating of the rom and a link to the original thread containing the rom ... each time a rom sinks or becomes "aged" then it would be time to add to the links page. just my thoughts guys

 

[SoDroid]

what about a sticky for "Old Roms" wich simply displays the overall rating of the rom and a link to the original thread containing the rom ... each time a rom sinks or becomes "aged" then it would be time to add to the links page. just my thoughts guys

That's not a bad idea. One sticky which would include one link per rom in the OP. It would always be locked and only the OP would be updated. It would be the one stop shop for rom reviews.

 

[inssane]

I like those ideas.
+OLD ROMs+ (Sticky Thread)
-BB (Post)
--0.5 (link)
--0.4 (link)
--0.3 (link)
-UD (Post)
--v1 (link)
-Cyanogen (Post)
--5.0 (link)
--4.3 (link)

(sorry, I made up build numbers)
But something similar to the above where they would be somewhat archived with links for anyone who wants to compare a few older versions. Couldn't hurt.

 

[inssane]

Thanks to everyone who contributed - especially UBRocked for spearheading this whole operation and putting it together.

 

[OMJ]

Thanks to everyone who contributed - especially OMJ for spearheading this whole operation and putting it together.

I think you mean UBRocked. I have actually done very little towards this other than give UBRocked advise when he asked for it and a little of the adding polls and adding reviews to threads.

Its actually turned out very well. I must say I was pretty skeptical when I first heard the idea/plan but its been great. It would not have happened with the the tremendous amount of work that UBRocked has put in though, and the work that he continues to put in.

 

[GirLuvsDroid]

+1 for the kudos to UBRocked for all the work he put in on this, and how well it's worked out!

 

[inssane]

I think you mean UBRocked. I have actually done very little towards this other than give UBRocked advise when he asked for it and a little of the adding polls and adding reviews to threads.

Its actually turned out very well. I must say I was pretty skeptical when I first heard the idea/plan but its been great. It would not have happened with the the tremendous amount of work that UBRocked has put in though, and the work that he continues to put in.

YES I edited my post - I am an idiot and it was 4am, lol.

I think I just like your avatar better, lol - sorry UBR

"I'm on a dolphin doin flips n sh!t"

 

[UBRocked]

I think I just like your avatar better, lol - sorry UBR

Here's a question for you though...Is OMJ the guy in the foreground or the background of that picture? LOL I can't tell which one looks "cooler"

Thanks for the props guys! I too am surprised at how well it turned out. I am indexing ROMs right now and just now saw that this thread had new posts in it.

Many thanks to the ROM Reviewers that made this possible! Several of the ROM Revs have been OUTSTANDING and always working to review the next ROM. I will need to get some more "active" volunteers in the near future to help carry the load.

Now to finish indexing and figure out how to keep this going strong now that ROMs are being posted here as well as in the Developer's forums

 

[inssane]

Here's a question for you though...Is OMJ the guy in the foreground or the background of that picture? LOL I can't tell which one looks "cooler"

Thanks for the props guys! I too am surprised at how well it turned out. I am indexing ROMs right now and just now saw that this thread had new posts in it.

Many thanks to the ROM Reviewers that made this possible! Several of the ROM Revs have been OUTSTANDING and always working to review the next ROM. I will need to get some more "active" volunteers in the near future to help carry the load.

Now to finish indexing and figure out how to keep this going strong now that ROMs are being posted here as well as in the Developer's forums

He's in the back, because that guy is "on a boat like Leo" - I have that song on CD and I take it on the boat in the summer. I get strange looks, lol

Also - I will be active very soon My laptop HD died so I had to move and setup a desktop and it's still annoying to use because it's in the kitchen
Anyhow, I am planning a UD 8 - FY1.5 - Shadow

 

[OMJ]

Here's a question for you though...Is OMJ the guy in the foreground or the background of that picture? LOL I can't tell which one looks "cooler"

Im actually not in that picture.....Im T-Pain. shhhh dont tell anyone