Disney's cuddly toys of the future could be smart enough to pick out the perfect bedtime story, thanks to a clever new artificial intelligence engine that's been trained to judge short works of fiction.
Using neural networks designed to mimic the learning processes of human brains, the AI was trained using sample stories to recognise particular traits and patterns that would appeal to a broad range of readers.
The team from Disney Research and the University of Massachusetts Boston says it's the first large-scale study of its kind on getting bots to evaluate the quality of texts, and could eventually lead to AI systems that are able to tell a good story as well as spot one in a crowd of literary tales.
"Our neural networks had some success in predicting the popularity of stories," says one of the team, Boyang Albert Li from Disney Research. "You can't yet use them to pick out winners for your local writing competition, but they can be used to guide future research."
To get enough data for their system to work with, Disney researchers plugged in more than 28,320 of the answers from Q&A site Quora, focussing on those with most upvotes as an indicator of quality – many of the answers on the site are written in a story-like fashion, so it seemed a good fit for the purposes of this experiment.
Objectively judging the quality of a story is of course very tricky, so the upvotes added by Quora users acted like a sort of shorthand quality measure.
Three neural networks were then developed: one to look at individual chunks of the stories, one to analyse how those chunks fit together, and one to assess the stories more generally. As a combination, the networks were able to learn what separated the better stories from the rest.
The test was to see if the neural networks, once trained, could take a new story from the database and predict how many Quora upvotes it got – a sign that the AI was understanding what makes a story popular and what doesn't.
In the team's experiments, the neural networks proved better at judging stories than traditional machine-learning techniques, with the network judging the story as a whole, registering an 18 percent improvement.
We should note that as useful as Quora upvotes are, they are far from perfect in assessing the quality of a story. What's more, the research hasn't been peer-reviewed by other scientists yet.
Having said that, it's another example of the way advanced AI techniques like neural networks – which are capable of accepting and processing vast amounts of data in a layered way – can potentially make systems that are more human-like over time.
"The ability to predict narrative quality impacts on both story creation and story understanding," says Disney Research vice president Markus Gross, who wasn't directly involved in the research.
"To evaluate quality, the AI needs some level of understanding of the text. And if AIs are to create narratives, they need to be able to judge the quality of what they are producing."
The research is due to be presented at the International Joint Conference on Artificial Intelligence in Melbourne, Australia and is also available to view online.