The Help the penguin toy was built in a student project during the first half of 2012. The final product is a fully working prototype; fully mobile, understanding and communicating using natural language, and fun to use!
Skip directly to:
Final 'Help the Penguin' prototype
The problem we wanted to solve was to support curious kids to find information when there is no one around to ask. Adults are often occupied with their own issues, and don't always have time to answer kids questions about things they find or think about. An intelligent device could help kids by answering simple questions, but might seem hard to approach and use. This is where our penguin enters the scene.
The penguin is the kid's friend and bridge to information. The kid can ask him about most things in the surrounding, and the penguin will tell them what they have found - problem solved? Not quite.. As we quickly realized, this kind of simple question-answer interaction gets boring rather quickly. We needed something more engaging for our kid users.
That is why the penguin comes with a story. He is old, has a dodgy memory, and frequently misplaces things. He gives the people he encounters a simple but sometimes challenging tasks: to locate one thing he has lost. Now there is a purpose to looking for and showing things to the penguin: you are helping him out. The penguin in turn helps you at his best ability, by giving you clues of what you are looking for. When you find it, you get greatly rewarded - before the penguin suggestions using your strong detective skills in finding the next things on his list.
Kid with penguin in hand and Cheeta Headset on
To communicate with the penguin, you need to use a headset. Since we wanted our interaction to be, or at least appear, physical and familiar for kids, we simply hid the headset inside of a Cheetah mask. This seemed to work well enough for our prototype, but is something we would work more on if we continued with this project.
Once you can hear the penguin, he will automatically start speaking to you. He will ask for your help finding a thing he is looking for, followed by a clue - something he can remember about it. For instance, he might say he is looking for "something red". The kid can then take lead in the interaction, by asking for something else, another clue, or for instance for the penguin to repeat what he said. The clues get progressively easier, so that everyone is able to eventually figure out what the penguin is looking for.
When the kid think he has found what the penguin is looking for, he or she needs to show it to the penguin. The penguin will tell the kid how, stating that since his vision is so bad, he needs to get a really good smell of the thing the kid is holding. The kid then puts the thing to the penguins nose, which, following a good 'sniff' by the penguin, leads him to tell the kid what they found, and if it was what they were looking for.
Behind the scenes the Help the Penguin toy uses the following main technologies:
Hardware - the core of Help the Penguin
The software of the penguin is written in Java. To understand what the user says, we use an Open Source Speech Recognition library called Sphinx 4. Based on what the user said, and a simple tracking of what current understanding we expect the user to have of what the penguin wants, we then respond by having the penguin speak back to the user. For this we ended up using the built in -say command on Mac, accessible by the terminal (or, from java, -exec).
For the hardware of the penguin toy we are using a microcontroller called Arduino. Plugged into this small computer sits an RFID-reader, a WiFi shield, and a simple 9 volt battery. The RFID reader then simply reads RFID-tagged objects that users are pulling to the penguins nose (where the RFID reader sits), which gets transmitted to our server using socket communication.
Further, on the Ardunio we also have a vibration motor that makes the penguin vibrate when he speaks, and a simple LED lamp that serves two functions: indicating normal function (i.e. battery is still alive), and signaling that an RFID tag is being read.
Back to top
Overly complicated interaction (Src: Wild Enterprise Wordpress Blog
Our toy relates to the theme of Science Fiction not by being an example of something that would actually capture your attention in move of such - there is too much flashy, completely unrealistic, technology there to compete with anyway - but rather as an important intermediete step towards the kind of future technology we want.
The future we envision is one where the human computer interaction is much less directed and constrained by computer devices. We don't think the interaction is going to take place at desks, in front of stationary technology, instead we think the technology will be distribuited and follow you and your needs. We don't think the use of technology will be so much limited to 'single player' use as it is today, but instead to a much larger extent invite groups of individuals to join in together to perform activites. Finally, the future we imagine is one where our technology interfaces finally become as intuitive the physical things that surround us, either by becoming more physical, or just by the use of more natural interaction methods.
We managed to get closer towards that ideal future with our penguin toy, and that is our contribution to the science fiction theme. Our technology is invisible - our users are not intertacting with a computer but with a friend in need of help. This is not only because the digital aspects of the penguin are hidden, but also because the penguin much more participates in the same physical world as our users than what ordinary stationary technology does. Finally, our interface showed that, at least for some tasks, speech technology can be at least as intuitive as traditional computer interaction using a computer mouse. Whereas it is still open for debate whether speech technology will be (part of) the future answer of natural interaction with technology, if the technology improves, we at least hold it as a viable option.