[The Toronto Star reports on a collaborative pilot project that uses virtual reality and artificial intelligence to help adult immigrants learn English. The impacts include not just the language learning but increased comfort and confidence; the story notes apparent advantages of using cartoon avatars rather than more perceptually realistic ones. See the original version of the story for two more pictures and for more context see the related April 2019 ISPR Presence News post “Believing VR is a Promising Tool for Learning Languages, Companies and Colleges Invest.” –Matthew]
[Image: From left: Yusra Doro, Manar Mustafa, and Adlais Hitou are seen with the VR headsets they use as a learning tool at the University of Toronto Mississauga campus. Credit: Giovanni Capriotti]
How VR and AI could revolutionize language training for newcomers
A new project is exploring the use of virtual reality and artificial intelligence in language acquisition for adult immigrants through simulated environment that mirrors real-life situations.
By Nicholas Keung, Immigration Reporter
August. 25, 2022
Adla Hitou is shamelessly showcasing her stellar work experience as she tries to convince the interviewer to hire her.
The Syrian newcomer answers every question that’s tossed her way.
“Give me that $3-trillion job,” she says, before bursting into laughter.
The assertive and fun-loving Hitou undertaking this mock job interview in virtual reality seems like an entirely different person from the timid mother of two who normally only whispers to others in her real-world classroom.
“I don’t feel nervous speaking in English in the virtual world because I just disappear. When I talk, I’m not afraid to make mistakes anymore. I just feel more confident,” says the 51-year-old Mississaugan, a former pharmacist who resettled in Canada in 2018 via Lebanon.
That’s music to the ears of the researchers behind an innovative pilot project that explores the use of virtual reality (VR) to improve newcomers’ spoken English — both their conversational skills and their confidence — through simulated environments that mirror real-life situations.
Funded by the federal Immigration Department and the University of Toronto Mississauga, the three-year, $550,000 project was developed by the university’s department of language studies in partnership with the Syrian Canadian Foundation.
“Learning a language is a social experience. Students need to be together and interact with one another in a more natural context. We wanted a place where students can go into a virtual environment where they are able to practise their target language as if they were in a shopping mall or in a doctor’s office,” said Paul Alexander, the project’s assistant principal investigator.
Alexander called the project’s use of VR in language learning “groundbreaking,” adding the technology appears to help participants overcome the self-consciousness in communicating in their second language.
“At this point, VR is a great equalizer. When students are using VR, they seem to feel this freedom to want to be able to speak. I think part of it is, when they’re in the VR world of it, they’re less concerned about themselves and about making mistakes, where they are actually being represented by a cartoon character.”
On this sweltering Saturday, instructor Anthony Faulkner and volunteers prepared the four female and three male adult learners on how to tackle questions at a job interview.
At first glance, there’s nothing atypical about the drill, a part of the English as a Second Language curriculum to help adult immigrants learn the language for their successful integration in their adopted country.
They talked about how to make a formal introduction of themselves, highlight their accomplishments and think on their feet when faced with the unexpected.
Sitting in a circle in the lab, Hitou — in a soft and gentle voice — told classmates in the physical world about her training as a pharmacist, her work experience as a project manager with the World Health Organization’s food and vaccination programs in Syria, and the civil war that forced her exile.
“I’m good at communication. I can take your ideas, relate the information and make a good presentation,” she said, adding personal details: “I love volunteering and do handcraft. I’ve made crochets and sold them at bazaars to raise money for charities. I am a bad seller but I have a kind heart.”
After the in-class session, with help of a team of volunteers, the participants were invited to put on their headsets, lift the hand-held controllers and enter Faulkner’s virtual office resembling an executive suite — wood panelling, rows of bookshelves and a bronze chandelier.
In an instant, Hitou, in her white hijab and blue one-piece dress, transformed into an avatar in the virtual world, revealing her long silver hair and sporting a black business suit.
“How do you deal with failures and mistakes?” asked Faulkner, standing in the middle of the classroom and moving his hand-held controllers in the air to make his avatar do a “hands-open, palms-up” gesture.
Caught by the surprise question, Hitou, behind her headset and facing a whiteboard in the physical world, confidently replied: “We are all humans. We have to learn from our mistakes and understand why we made the mistakes and failed. I do not give up.”
“This is so much more fun than learning English from books and notes in the traditional classroom,” she said later. “Now, I remember every word and thing that I see and learn in the virtual world when I go back to the real world.”
The virtual job interview is just one of many thematic VR experiences covered over the eight-week course by the research team that developed the customized scenarios with help from ENGAGE, a virtual platform that simulates the way people interact in the physical world for multi-user events, collaboration, training and education. One scene includes a dinner party at a virtual highrise loft; another involves the planning of a Canadian road trip where each newcomer is assigned to research a part of the country before going to the different booths in a virtual conference hall to make a presentation to their peers about these places and activities.
Instead of just viewing some Canadian landmarks on television as peers in a regular class might, the VR participants can take virtual tours of Niagara Falls, Toronto’s Eaton Centre and St. Lawrence Market through the VR 360 videos on YouTube VR.
Oakville’s Manar Mustafa, a computer engineer who fled war in Syria and came here in 2016, said she attended a regular English program at a newcomer settlement service agency but nothing can compare to VR learning.
“This is a perfect experience. I never used VR but everything feels so real in the VR world. I have not visited the Niagara Falls but now I have. It was right in front of me in the classroom and I didn’t even get wet,” the 36-year-old mother of four said with a chuckle.
“Initially, I felt dizzy (with motion sickness), but now I really love it. I feel very comfortable with it.”
Her classmate, Afghan journalist Abdul Mujib Ebrahimi, who only arrived in Canada last November, said he hopes to quickly translate what he has learned from the virtual world to the real world.
“The VR experience pushes you in a real situation. You are a character and you use your imagination to communicate with others. If I don’t know a word or how to say something, I just explain it in a different way for people to understand me,” said the 27-year-old from Badakhshan.
“I’m still trying to learn English and work with English, but I am more confident when I talk to real people.”
Alexander cautions that it’s too early to determine the effectiveness of VR in language learning.
“We have to be careful of the novelty effect of VR. Our students are enjoying the experience maybe because they’re trying VR for the first time and they can tell their friends and family about it,” he explained.
The VR class this summer is part of a series of projects that also include participants in traditional classrooms and artificial-intelligence-assisted learning in front of a computer. The AI session will be launched in winter.
There are about 80 newcomers waiting to get in the program, according to Marwa Khobieh, executive director of the Syrian Canadian Foundation, which has been running a joint English tutoring program for newcomers with volunteers from U of T since 2017.
She concedes that VR language learning is expensive, with the required infrastructure and equipment in its infancy, but if it succeeds, there’s a huge potential to use it to accelerate the learning and ultimately the integration of new immigrants.
“If we’re able to help newcomers learn and improve their language skills in two years instead of four, it’s worth the investment,” noted Khobieh. “Technology is our future and this can change the future of language training for newcomers.”
|
