Wattpad doesn't share many of the details of their judging process, but here's what we've managed to figure out about their process. Hint—there are definitely computers involved.
Does Wattpad Read All Wattys Entries?
Wattpad has been notoriously vague about their Wattys judging process, often just saying that their process "combines human review and data science." In their FAQ, when addressing whether every submission is read, they acknowledge that it would simply be impossible:
"The Wattys receive thousands of entries each year and would take us several lifetimes to read every single one."
The 2020 Wattys had 40,000 entries (the lowest since 2014). Previous years, 2019 and 2018, received 169,000 and 362,423 entries respectively.
It would take around 2.8 hours to read each entry in full, adding up to 112,000 hours—just under 13 years—of total reading time.
Last year, Wattpad listed 15 of their Wattys 2020 judges. Based on the 2020 numbers, that would amount to under 311 days of nonstop reading per judge just to get through each entry once.
While it's hard to know if there were additional judges outside this list, the entire company employs an estimated 2864 people. Even assuming 2000 of them were involved in judging stories, which is very unlikely, that would be 20 stories or 56 hours of reading per person. Again, that would be assuming each book was only read by one person, one time.
A more likely scenario is that these 15 judges are the primary human judges and could read a maximum of one book a day during the judging period. That would amount to a maximum of 2700 books being read by judges, about 6.8% of total entries.
Since the math is clearly not in their favor and Wattpad acknowledges reviewing all these entries by hand would be impossible, what are they doing instead?
How Are They Judging So Many Stories?
It's probably safe to assume that this is where Wattpad's data science comes into play. Some of this role is obvious: confirming that entries adhere to eligibility standards—that they are the correct length (over 50k words), were created before the cut-off date, haven't previously won an award, etc. That said, much of these criteria are controlled for in the application process.
To answer this question, we looked at the terms and conditions for other Wattpad contests.
In many of their contest terms, Wattpad uses the following (or similar) language to describe their selection process:
"Finalists are selected by a combination of data driven insights and manual editorial review from all eligible Entries received. ... Wattpad uses Machine Learning to score content according to quality sentence structure & grammar (the "ML Model"). Each word in a story is indexed by the type of word it is (e.g. noun, adverb, adjective, pronoun, etc...). The ML Model finds patterns in the grammar & sentence structure of that story. The ML Model then compares the grammar & sentence structure patterns to those patterns in Gutenberg Classics, and Handpicked Wattys Winners stories (what we tell the machine is quality). The machine will apply a score to how similar a story is to our quality story groups."
In other words, Wattpad trains their system to score stories based on how similar they are to their sample stories: previous Watty winners and the collection of Gutenberg Classics. Project Gutenberg, a site that shares the texts of famous public domain novels, includes classics like Frankenstein, Pride and Prejudice, and Moby Dick.
What Kinds of Books Would Score Well?
Wattpad has built its platform with a heavy lean toward the romance and YA genres, neither of which are particularly similar in style to classic novels, many of which are heavy in prose and written in older forms of English. It's unlikely that previous Wattpad hits like After and The Kissing Booth, both written in simple language by amateur writers, would score well under this system.
On the other hand, books with longer paragraphs, more descriptive prose, and less dialog may be a better fit for this rubric.
There may also be a benefit to better spelling and grammar, but it's hard to know. The Wattys FAQ insists that their system doesn't discriminate against made-up words or spelling variations, though they contradict themselves by saying human readers prevent these potential errors:
"Some of our data does involve using some machine-learning components to help evaluate content. It does not work like a spellcheck or discriminate against Fantasy and Science Fiction that tend to employ invented words more frequently or spelling differences between different versions of languages. It's why we partner humans with machines so we can make these sorts of adjustments and not penalize stories for doing story or language things."
So How Much Reading Are Judges Doing?
Considering how many books the team has to go through, it's probably fair to say that the judges are doing quite a bit of reading, but probably from a much smaller pool of long-listed finalists after they're culled by computer scoring. By the time a book makes it to the short list, it's likely that someone has at least read a substantial chunk of it, and it will likely be read in full in the final rounds.
Since Wattpad is quiet about its algorithms and the Wattys judging process in general, all of this is speculation based on the small amounts of information available to us.
Our takeaway? Don't put too much stock in The Wattys. The system is likely to favor more traditional, detail-heavy writing styles and a human may never have read your story during the judging process. It's just impossible for Wattys judges to read every story and computers aren't great judges of quality art.
Related Content
Are you entering the Wattys this year? Share your story in the comments...
Comments