das Monkey

Recently, I've found that I don't really hold this rating as highly as I used to. I don't know if it's a change in review styles or a change in my own preference, but either way, it got me to thinking. I really like the idea of having a special rating for the absolute best of the best, but I also recognize the variety of opinions in different reviewers and how one man's Skip It is another's Collectors Series. In addition to that, I think there needs to be a rating that separates kickass DVDs for kickass movies from simply kickass movies. To that end, I've always thought it was kind of silly to call a theatrical review "DVD Talk Collectors Series."

So, here's what I propose. An <i>individual</i> can give a DVD or film the highest rating possible. You can keep calling it DVD Talk Collectors Series or come up with a more suitable name. Then have another award. If a reviewer wants to give a DVD this ultimate kickass award, he has to have it "approved." What does that mean? I don't know. You could do it many different ways. My initial reaction is very simple: one other reviewer has to concur.

The reason that I suggest this is that some of our reviewers are much more prone to extreme opinions than others. And that's fine. It's their review; they can write whatever they want. However, the difference in standards between the reviewers makes it such that a list of DVD Talk Collectors Series titles isn't really that informative. This problem certainly exists with the other ratings levels as well, but it would be nice to have one rating that had a different and higher standard than all others. When you looked at the list of those titles, you would know it was really the special stuff, not just what one guy might have thought.

Does this sound somewhat reasonable? I realize there's an added overhead to this as two people would have to have seen the thing, but if it's that damn good, it seems pretty likely that more than one person should see it. And only one has to write the review; he just needs someone to confirm that it's really worthy of the honor. Many others may still disagree, but at least you cut out any rogue opinions. By doing this, you still maintain the individuality of the reviewers, and they can still give a DVD (or film) top honors; but you have another level of greatness that readers can look to with more consistency.

das

AgtFox

Sorry to cut it that short das, but I totally agree with you here. When I was doing reviews here I never used that for theatrical releases. It's just bad in my opinion and I was shocked to see it so freely used on recent Theatrical reviews here.

I agree there may need to be a special classification in the DVD section for those top tier ones, but you point out the weakness that there has to be at least 2 people on the team that have seen it.

Bill Geiger

Great Topic das.

While reading your opinion, I couldn't help but think of a DVDTalk Members rating. Possibly allowing the entire forum the ability to vote on a particluar film that has been reviewed (obviously once it has been released in theathers or on dvd of course.)

So, in one regard, you would have the reviewers opinion and the other having the forum members "opinion."

Hell, we could have our very own IMDB-"ish" Top 100 after a while.

abintra

Adam Tyner

The other sections were shoehorned into the current system and share the same ratings classifications. I agree that it is kind of silly to have a "Collectors'" rating for a theatrical release, and I also agree that the rating itself is overused. There isn't a clear definition as to what it represents, and the same can be said for any of the ratings.

When I toss out ratings, I write from the mindset of a reader who has some interest in the title.

Skip It: Unredeemably bad. Even if you were thinking about picking up this DVD, it's not worth it.

Rent It: It's too mediocre to buy (or overpriced), but if you're interested, you still might find it worth a rental.

Recommended: Decent. Definitely worth a rental; recommended as a purchase to those with a particular interest.

Highly Recommended: Highly recommended as a purchase.

DVD Talk Collectors' Series: An essential purchase. You have a gaping hole in your DVD collection if you don't own this title.

With that mindset, "Skip It" and "DVD Talk Collectors' Series" should both be given out infrequently. Most of the titles that will be reviewed are mediocre-to-average, so "Rent It" and "Recommended" catch most of those.

My breakdown is:

Skip It -- 22 times out of 343 reviews -- 6.41%
Rent It -- 129 times out of 343 reviews -- 37.61%
Recommended -- 119 times out of 343 reviews -- 34.69%
Highly Recommended -- 71 times out of 343 reviews - 20.70%
DVD Talk Collectors' Series -- 2 times out of 343 reviews -- 0.58%

I think the Collectors' Series rating for <i>A Bug's Life</i> is justified, but I'm not sure if I'd give <i>Superman</i> the same rating if I were looking at it again.

B.A.

das - you raise many valid points, particularly regarding the Collector Series rating for theatrical releases. I have always thought that was odd and a rather bad idea.

In regards to the Collector Series rating for DVDs, I think it would be a great idea to make sure that two or three of the reviewers agree that a particular release is worthy of that title.

Gil Jawetz

Our year end top 20 this past year was basically that. We had a very nice debate/vote in our own little forum and came up with a very cool list, I thought. Granted you may want something a little more regular, but that was a good step. I recall that you were down with it.

I like your idea, too, btw. I only wonder if after a while it's just too many ways of saying the same thing: reviews, group labels, years end lists... it's a lot of yammering!

das Monkey

Yeah, I'm just tossing the idea out there in hopes that someone has a better solution or that something better might evolve. If not, that's cool too. I just like the idea of having a "DVD Talk" rating that has some added power to it without having to immediately filter it through who actually wrote the review and wonder if it's just one man's rogue opinion, or if it's really worth me buying it that instant.

As for the Year End Top 20, that was pretty cool, and I look forward to it again. As you know, though, I think you guys made a particularly glaring error with that list that I still find really disappointing . Of course, the reason for the error appears to have been that enough reviewers didn't get to see the content, which again brings to light the inherent flaw in my proposal that is the subject of this thread.

With that in mind, as I said, I'm just tossing the idea out there. However, I would hope that if a reviewer felt so strongly about something as to give it the uber rating, he would also be willing to campaign to get at least one other reviewer to see the thing and confirm it (which 90% of the time is going to happen anyway). If not, is it really that good?

das

Gil Jawetz

That's the thing. A highest rating by consensus pick would basically gravitate towards the most mainstream, big-name titles, titles that everyone else already knows about. The reviewer panel here is a quirky bunch. There are anumber of people who do ont go for many of the mainstream titles. I do not own the Back to the Future, Indiana Jones or Star Wars films, plus no Spielberg at all. I like your idea but I think that a consensus on a DVD that you already know is great doesn't really get you much. I was psyched that I got some great feedback on Collector's Edition reviews of stuff like Scratch and Pride and Prejudice, films that people may not have seen if I hadn't really spent time on.

das Monkey

Keep in mind that I'm by no mean suggesting a "consensus." I recognize all the reasons that would be a terrible idea. (Also keep in mind that I'm talking about <i>adding</i> a category. You'd still be able to give <b>Scratch</b> and <b>Pride & Prejudice</b> the highest individual award, with potential for the uber award). I am, however, suggesting some kind of validation, so it's just not one single person. Even if you only have to find <i>one</i> other person to agree with you, given the variety of styles and interests and talents in your review group, I think it adds a significant amount of credibility to the rating. Sure, people are still going to disagree with the reviews, and that's fine; but it helps eliminate the case where the reviewer is so far in the minority with his opinion that he can't get one other person to support him. He can still give it the highest individual rating, but it will separated out from those titles that went to the next level. It's like in Roberts' Rules of Order ... the Chair won't consider your motion unless it's seconded. It might still get turned down, or may create a heated debate, or may be met with universal approval, but it won't even be considered without at least one other person believing it has merit. That's kind of what I'm getting at here.

I hesitate to post an example, since I don't want to offend anyone, but what the hell. This guy probably doesn't like me anyway, so no harm. I'll stick to TV, since it's my bag, and the same reviewer to keep it simple, and even two reviews I don't agree with to properly illustrate the point (and how my opinion of the review isn't relevant).

Take <i>24</i> - Season 2 and <i>The Dead Zone</i> - Season 2. Both received the DVD Talk Collectors Series rating. Both I personally don't agree with. However, I bet there are millions of people who agree with the former, including many of the DVD Talk Review staff. In contrast, I suspect the latter isn't supported by too many. If it is, then I've chosen a poor example, but let's run with it anyway. The point is that as an individual, the reviewer thinks these two sets are the best, and as an individual, I don't. Consequently, as it stands now, they are treated equally. I would argue that they shouldn't be, that the former contains a quality the latter doesn't have, and that difference would be revealed under the "approval" system I've proposed. In the end, I think that would be much more beneficial to the reader. I'm familiar with both of these sets, so my opinion is already set, but if I were the average reader, I would want the distinction to be made. I would want to know that one set had the approval of multiple people and one was simply liked by the reviewer. Maybe I'll like them both, maybe I won't, but I'd like that distinction to be made. If I tend to agree with the reviewer on things, I'll likely be inclined to check out both. However, if I typically don't agree with the reviewer, I won't be inclined to simply dismiss the <i>24</i> review.

I'm from the school of thought that reviews aren't just tossing your thoughts on paper for your own fun, but that they should actually assist others in finding something new to watch. The current system strives for that goal, but I think an update would go even further towards reaching it.

das

Holly E. Ordway

The essence of a review is that it's a personal evaluation with a supporting argument. When you see one of my reviews, I don't want you to just glance at the recommendation and walk away - I want you to read the review and see WHY I made that recommendation. Maybe you will agree, maybe you'll disagree, but in either case you will have had some points brought up that will hopefully allow you to make a better choice about whether to watch that DVD. Personally, I think that's a lot more useful than a consensus (I'm probably not the only person who finds the Internet Movie Database ratings completely pointless and useless.)

There is no such thing as a definitive or correct review, when it comes to the rating. I'll certainly agree that there are good-quality and poor-quality reviews. A good-quality review explains WHY a film got its recommendation, and backs it up; even if you disagree with the conclusion, you can see what it's based on. A poor-quality review doesn't support its conclusion, and just sort of says "it was good/it was bad" and slaps a recommendation down; that sort of review is not helpful even if you agree with it. But that's totally orthogonal to what the actual recommendation is.

I work hard to make my reviews helpful to readers by explaining why I liked or didn't like each film, and that's what should be meaningful. The recommendation is just the icing on the cake.

Anyway, we do have something already in place that serves the "consensus" need without diluting the reviewer's role: the "discuss this review" link. If you're interested in hearing more peoples' thoughts, then start a discussion and ask! You *can* have your cake and eat it too.

Gil Jawetz

Mmmm...cake... i may need to run down to the deli...

das Monkey

<b>Holly</b>,

I agree with everything you said. However, I fail to see how what I'm suggesting violates that in any way. The reviewer's role isn't diluted at all. My suggestion simply adds credence to the periodic official pimping of the DVD Talk Collectors Series. I'm not suggesting anything change with the reviews or the methods of reviewing; I'm only suggesting adding another tier, a level above the individual review, so when the forum issues an email that reads, "It's the chosen few DVDs released each year which receive DVD Talk's Highest Rating" we know those "chosen few" represent more than just one person's opinion. Because unlike the Highly Recommended or the Skip It, the DVD Talk Collectors Series is treated in such a manner as to be representative of more than just the individual reviewer.

And as I've said before, I'm not asking for a "consensus," nor am I looking for a plethora of opinions from different members. The "discuss this review" link serves a different purpose from what I propose, as does the DVD Talk forum.

das

Shannon Nutt

Das - since I wrote these reviews, let me defend my ratings. I love both these shows - I won't deny it. But even if I did, the actual DVD PACKAGE has to live up to the "Collector's Series" rating.

You obviously don't care for these programs...or at the very least enjoy 24 more than you do DEAD ZONE. But that's subjective...it's from the part of the review rating that is going to differ with every individual that watches the title.

But there's more to a DVD than just the program itself...there's the video quality, the audio quality and the bonus materials. These are more objective things that most people CAN agree on - or at least if they disagree, the margin isn't as great as with something like how you felt about the program itself.

For both 24 and Dead Zone, the A/V quality was exceptional for a TV series - both shows were presented in anamorphic widescreen, both were transfered well, and both were presented with impressive 5.1 Dolby Tracks.

The Extras on both shows were exceptional as well...in fact, if I had to choose between the two, I would say THE DEAD ZONE package is even more deserving of the "Collector's Series" status, since there is a commentary track for each and every show. (There are a wealth of other bonuses for each title as well...too numerous to relist here).

I should also point out that I don't give this rating out lighty. You've mentioned two of only three "Collector's Series" ratings I've given out in ALL of my reviews for DVDs. There are some movies that I LOVE (Shoot To Kill is one) that I didn't give this rating to because of the DVD lacked the criteria I feel is necessary (in Shoot To Kill's case, it was an absence of bonus material).

I think where the problem occurs is that different DVD Talk reviewers are using different criteria...but for me, a DVD has to score 4 or more stars in ALL of the review criteria (movie, video, audio, extras and rewatchability) for me to even consider giving it this rating.

das Monkey

I guess I did a poor job clarifying why I chose those two reviews as an example. It had nothing to do with you or the quality of your reviews. And of note, I actually like <I>The Dead Zone</i> more than I like <i>24</i>. That wasn't the point. I just tried to pick two reviews from the same reviewer that I had the same opinion on and were somewhat in the same ballpark (TV sets), so it would be moderately consistent. Like I said, it had nothing to do with you or your reviews or your review styles or your proclivity for rating things one way or another ... I just picked two reviews, and since <i>The Dead Zone</i> review was fresh in my memory (and I recalled the <i>24</i> - Season 2 review), I ran with it. My apologies for any confusion on that. Perhaps you'll see the bright side, that someone remembers your reviews in such detail. Anyway, you don't need to defend the reviews; I'm not criticizing them.

My point, as you've illustrated in your final sentence, is the differing criteria of differing reviewers. If DVD Talk is to treat the Collectors Series as something unique, far and above everything else that gets reviewed, I would want it to have a different standard. That's all I'm getting at. From the responses, I see I've done a terrible job articulating that.

das

Gil Jawetz

You think too much. When are you gonna just bite the bullet and join the review staff already?

das Monkey

I got to thinking the other day how one person's brain processes thoughts differently than anothers, and I wanted to share with you the conclusions. To avoid flooding the forum, I will present these conclusions over 17 parts.

Part I: Yeah, I know I think too much. It's my lot in life. Why do you think I'm an insomniac? But, hey, this place is somewhat of a home away from home, so I like to toss this stuff out there in hopes that it helps. If not, that's cool. But every once in a while, I stumble on a good idea. Don't you love those tooltips on the review covers? That's right baby ... all mine!

das

Shannon Nutt

Das - I knew you weren't bashing my reviews...I just thought it would be helpful if you understood my my criteria were, and also knew that there is no set criteria for reviewers one way or the other...which I think is what makes the review staff here unique. If we started to have to approve each other's "Collector's Series" ratings, we wouldn't have anything to fight with each other about.

Houstondon

I like Das and have found his opinions, on average, to be far more interesting and informative than average but I suppose the choice of words here implies something sinister. "Official pimping?!?" Maybe more examples would help with the discussion.

I have yet to hand out a Collector rating to a mainstream release. The closest I've come has been to Here and while sometimes, a reviewer goes too easy on a title, I doubt if readers can't figure that out quickly enough on their own without adding too many more layers of bureacracy.

das Monkey

Pardon my slang.

Pimp (v): To promote
Official (adj): Endorsed by the founder

I was simply referring to the periodic "press releases" (for lack of a better term) promoting the DVD Talk Collector Series as something far and above the typical DVD reviews on the site, as something unique and special that demands more of my attention than normal. I get an official e-mail every once in a while informing me about these Collector Series DVDs. That's all I meant by the phrase. I assure you there was no negative connotation intended. It's just the way I talk, and often my RL phrasing creeps into my online babbling.

I see this is apparently not one of my better ideas (actually, it is a good idea, just no one agrees with me ), so I'll let it go. It just seems to me that there should be a separation between the individual reviews and the official DVD Talk Collector Series. In its most simplistic terms, all I was really suggesting was to move from a 5-tiered rating system to a 6-tiered system, where the sixth tier was more than just a lone opinion. Tangential to that, and somewhat lost in this discussion, was having a better way to handle theatrical releases, which really don't make any sense in the current model. I meant no offense, although it appears my suggestion has created some anyway, and I regret that.

I'm not going anywhere, so rest assured ... we'll <i>always</i> have something to fight with each other about.

das

P.S.
That is most certainly going to be on my headstone.

Here Lies das Monkey
On average, better than average.

Houstondon

After I wrote that, I knew you'd pick up on the humor but I hated editing it to add the usual qualifiers (ie: Das' average post is far better than almost everyone else's average post) .

There is an extended discussion about your idea going on in the top secret reviewers forum still going on. I think changing the Collector rating is a bit too late since it has been a long time fixture but adding an uber Collector rating for something is workable for me.

Currently, the Collector rating is only used once in a while (my own usage falls at about .3%) but yes, some reviewers are more willing to use it than others. For some readers, that lowers the value of the rating overall while others probably just factor it in by reviewer. For me, a mainstream DVD would have to be great in just about every way, including content, before I'd assign it. With porn, I've stuck to handing it out to a handful of top notch values that were ahead of their time and a great value, although my recent Tera Tera Tera review broke that tradition since it was more of a media event (Tera Patrick's first release in a long, long time, with some of the best extras a Vivid release has ever offered). I will probably just adjust that rating but slavering fan boys of Tera's seem to wish a higher rating was possible, something it wouldn't deserve.

Gil Jawetz

The percentage thing is tricky. My percentage for collector's edition is WAAAAY higher than Don's and, even though I've been here longer, I've reviewed far fewer titles (although I don't know what percentage of his titles are non-porn). But then again I hunt and pick discs to review carefully since I don't have gobs of free time and I gravitate towards stuff that I really love. That's not to say I don't go out on a limb: I have more "skip it"s than "collectors". But if I have the chance to review a "One Flew over the cuckoo's nest" or "bridge on the river kwai" or "Gimme Shelter" then I'm gonna jump on it, even when its from my own collection.

Another thought on the collectors rating. Here's a quote from my Scratch review, which received that highest rating: So, I'm not suggesting, as some have said, that every DVD collection is bare without this disc in it. I'm saying that it's reached the highest mark of quality for some reason, detailed in the review, and anyone interested in the subject will really love it. There is no DVD that every collection truly must have. As I said, I'll never buy Star Wars, Indy or Back to the Future (or Schindler's List or Titanic) but if someone gave those the top rating I'd understand.

Das' idea isn't a bad one but i don't know if it would add anything. I know it's not a full consensus, which would never happen. And regular use of the "discuss this review" feature should deepen the reviewer-reader relationship in a pretty unique way.

Houstondon

This is a good point that comes up in most review websites: grade inflation. Gil is an excellent reviewer that shouldn't think twice about being overly generous in his ratings but the statistic junkies out there will always be willing to accuse (and that's not directed at Das as he knows better) someone who uses a higher rating more frequently. DVDTalk more prominently displays reviews with the Collector Series rating, listing them in emails and showing them more favorable treatment, so the danger is that readers get the wrong idea too.

Having pondered Das' original idea at length now, I can see some measure of benefit to his idea but I'm not sure how to pull it off for the lesser known titles some of have had the pleasure of reviewing (I've offered numerous reviewers a chance to borrow from my stash but we all keep so busy that it doesn't happen much). I honestly think most of my friends reviewing here (and elsewhere) treat a DVD they are reviewing with more of a critical eye than one they are just enjoying for kicks. If I read one of videophile's favorable reviews, I'm more likely to check out the movie with a favorable outlook, coloring my ability to be objective in a subsequent review. I can't speak for the others in this but such a system could easily become either a self fulfilling prophecy or a chance to go overboard with negative points that were not addressed in the original review.

Breakfast with Girls

Adam, I like your definitions. Maybe all the reviewers could vote on official ones in the reviewer forum and then there could be a link under the rating with "What does this rating mean?" or have a rating breakdown somewhere. Even better would be to include it on the reviewer form. There might be some confusion on some of the ratings.

Houstondon

With all the discussions that have taken place on ratings and their meaning, both in the reviewer forum and in the other forums, I don't think we need to be forced into certain definitions (as much as I like Adam's actual definitions). I think that people reading the reviews will get a pretty good feel for what the rating means in almost all current reviews (admittedly, I don't read every single review on the site but I have checked out many of them).

Most of the reviewers take advantage of the bio section we all have that allows us to explain our ideas as to ratings (if it's that important to us) so I'd be against forcing specific terms on the others.