Sesame Workshop Tackles Literacy With Technology

Elizabeth Jensen, The New York Times
Published: October 19, 2014

Elmo and his “Sesame Street” buddies could soon be having two-way conversations of sorts with children.

Sesame Workshop, the nonprofit producer of “Sesame Street,” and the children’s speech recognition company ToyTalk plan to announce Monday that they have signed a two-year research partnership agreement to explore how to use conversational technology to teach preschool literacy.

The agreement formalizes work the two have been undertaking for more than a year. Sesame Workshop has been testing prototype mobile apps that use ToyTalk’s proprietary PullString technology, a combination of speech recognition meant to understand children’s speech patterns, artificial intelligence and prewritten scripts responding to what a child has said.

The first products resulting from the partnership could be available early next year, said Miles Ludwig, managing director of Sesame Workshop’s Content Innovation Lab. Products that more formally teach children to read will take longer, however. When it comes to technology that tells children whether they pronounced a written word correctly — as opposed to, for example, asking them to come up with a word that rhymes with “cat” or discuss their feelings — “We need the accuracy to be very high,” Mr. Ludwig said.

Still, he said, “Down the road, I think that we can structure very specific early learning experiences” using speech recognition technology. He added, “It seems like the potential for literacy learning is tremendous, especially for those who are struggling to learn to read.”

In a September 2011 report published by Sesame Workshop’s Joan Ganz Cooney Center, a research lab, Marilyn Jager Adams, an expert in children’s literacy, argued that speech recognition technology could be a valuable and cost-effective literacy tool.

ToyTalk, founded in 2011 by former Pixar employees, already has its own apps, including the Winston Show, where children talk with animated characters. Those products target 4-to-8-year-olds, while Sesame Workshop focuses on preschoolers who typically do not speak as clearly and pause more often when searching for words.

ToyTalk’s system collects children’s speech patterns to feed into a continually updated database. “The more children talk to our characters, the better we get at understanding what they are saying,” said Oren Jacob, ToyTalk’s chief and co-founder.

Other adjustments must be made for preschoolers, as well. Sesame Workshop’s testing has discovered that children who see a two-dimensional Elmo on-screen assume it is a game with prompts, Mr. Ludwig said. When they see a live-action Elmo, however, they treat it “more like a Skype call with Grandma,” he said.