Personally, I think much of the issues resides in the "judges round" where the submissions get cut down to five nominations per category.
Strangely enough, I can't find the ENnies judges for 2015 on the ENnies site (but have have found it with some digging off the ENnies site) but Kurt, Stacy, Jakub and Kayra have been judging for a number of years. At least five years for some (and is Stacey Muth related to Mathew Muth?)
2015 Judges |
2013 Judges |
All FOUR judges from 2016 were carried over to 2017. Reece Carter was added for 2017. I suspect we'd find a similar trend with the prior years which I can't find on the ENnies site. (I found the trend)
The first thing to do is limit judges to one year at a time - no consecutive years of judging. This really is a no brainer. Biases are natural but bringing in new blood every year will do well at balancing out those biases.
Now, choosing the judges is a popularity contest, and much like in real world politics, the incumbent is at an unfair advantage. One year terms - it really is that simple.
About the judging process? Is there a set of guidelines the judges are expected to follow? Is it public? It should be.
Product is submitted as physical product and shipped around the world at considerable cost to the publishers seeking to be considered. I suggest requiring all submissions be made in PDF. Why? Why not? The judges are volunteers reviewing product. They don't need physical products to do their review.
Switching to digital submissions would remove much of the cost involved. It would also remove the idea that judges are being monetarily compensated for their work, as the post judging Ebay sell off is certainly a thing. And hey, we'd save some trees.
I would also suggest disclosing the votes of both the judges (for the nomination process) and the voting public (for the awards) after the award ceremony. Nothing is better at showing the legitimacy of a process than full disclosure of the process.
If only one thing were to be done, just one - limit judges to one year terms.
So, what are your suggestions?
edit: so, this is a thing. no more than two consecutive years as a judge - I'd prefer one year but its a start:
Full disclosure is a great suggestion, Erik. I also wouldn't mind knowing exactly what criteria the materials are judged upon.
ReplyDeleteAgreed - disclosure, transparency, how the judges voted, and term limits. Heh, sounds a lot like politics. Oh wait, it is.
ReplyDeleteOne year limits means institutional knowledge doesn't get passed down (there's a lot of behind the scenes logistics crap). I'd limit at 3 were one to set a limit.
ReplyDeleteThe fact that you had to hunt for the Judges names is enough to know there isn't enough transparency in the process.
ReplyDeleteAs far as terms, I'd say two year term, staggered. Let them do it twice and learn some from predecessors on their way out.
The PDF thing is also a no-brainer: why do I have any reason to think that people aren't being Judges to get free stuff?
Erik, did you know that the ENworld front page shined a spotlight on several, but not all, entries? Several comments mentioned the obvious bias of seemingly ENworld endorsed products.
ReplyDeletehad no idea - is it still up or is there a screenshot floating around?
DeleteHere it is: http://www.enworld.org/forum/content.php?4255-Sean-s-Picks-of-the-Week-(0703-0707)-ENnie-Nominee-Week!#.WWOs-c9BTIU
DeleteFirst of all, there is some docs about the process and how the judges go about deciding, you can see it here.
ReplyDeletehttp://www.ennie-awards.com/blog/about-us/faq/
http://www.ennie-awards.com/blog/about-us/how-does-it-all-work/
A few comments.
The above links probably show all the rules for product voting. It's like the oscars--there are rules about what can count as a category, but you're not going to get a strict guide for the actual voters to vote on (vote on all these principal items on these five axes of quality, etc.), especially since the final vote is by the masses. I think there's too many variables here to make it anything other than a long weeding out process that can't be defined by a strict formula.
In this matter, I don't think we should disclose the votes of the individual judges for the nomination process. It's probably a very long and drawn out process, but more importantly I don't think this type of "transparency" is helpful at all. If the long process was made public, I think you'd end up getting bogged down in the people complaining about what the judges decide, accusations of bias mostly by zealous fans who think somebody got robbed of a vote, etc. There's a reason why most votes and nomination processes are by secret ballot. I'd be open to getting a final tally of the public votes since that can easily be tailed while keeping things anonymous.
I don't think forcing the ENnies to only accept PDF submissions is the right move. They clearly offer both avenues--you can submit in print or PDF. In some cases, part of the judging of a product might be based on the physical quality of the material. A game might have good presentation and quality accessories such as dice, etc. Also, there are some products that are limited if they are in PDF format. Monte Cook Games, for instance, is not releasing Invisible Sun in PDF form--should they suffer because of a "PDF only" rule. If you are concerned about judges possibly being biased by getting what little financial recompse from selling a product, perhaps require than it goes to the Ennie organization and then gets sold to support the org's operating costs, and doesn't compensate an individual judge.
Well, if Monte Cook releases a product clearly aimed only at a select few who fork over a major sum, that's their decision. As long as it is known ahead how you can submit to the Ennies, you can make your decisions, that's all.
DeleteAnd they still would have no problem delivering a PDF - they basically need one for the printer anyway.
Production values are naturally higher for high profile publishers, and I personally would not want them considered. Dice, gimmicks, and knick-knacks - if they really _define_ the value of a product, should we really care? I personally think: absolutely not.
But I still think trying to force the Ennies to be a PDF submission only is not wise. First of all, there are categories that won't work well for a PDF in any event--Miniatures, Best Family Game, Best Cover, Best Interior Art, Best Cartography, etc. (It's probably easier to see the maps spread out for instance, and some of these may have a lot of accessories).
DeleteRegardless of what you think, I think in some cases the physical product can help define the quality--maybe not for the writing categories, but others near depend on it.
I think though I'm kind of wondering what kind of "corruption" can come from a physical product being sent. It seems people are looking instead of a "trust, but verify" situation to "they are default corrupt, let's make more restrictions". It actually would cost more for the publishers to send print copies than PDFs (which they do accept), so I'm sure many products are already being sent that why. Why punish the works where in part the physical attributes help with the judgements. I don't get the "eBay selling" concern.
I want votes on real products. Digital is cool but mostly I want physical books. And in many cases the digital doesn't compare to the physical.
ReplyDeleteI'm fairly certain that PDF submissions are already welcomed, just not required.
ReplyDeleteAs with any awards, there will be people unhappy with the process and more unhappy with the results. I don't think that negates the value of the award, especially to those winning them.
Now, if there was an OSR related set of Tavern awards, I'd be all about that too!