[This vivid, engaging, personal story from Sixth Zone describes some of the complex practical, ethical and emotional issues surrounding the creation of AI-based “griefbots” that mimic loved ones who have died. See the original version for 11 more images and animations. –Matthew]
Looking for Closure, a Grandson Built a Ghost in the Machine
To make amends and bid farewell, Yu Jialin used AI to revive his grandfather. He still hasn’t told his parents.
By Tang Yucheng
April 5, 2023
Hunched over a desk, Yu Jialin stared transfixed at his computer. Fingers hovering nervously over the keyboard, his eyes darted back and forth as he struggled to frame the right test phrase.
Weeks of hard work, long nights, and frustrating setbacks on a deeply personal, off-the-books artificial intelligence project were about to bear fruit. A software engineer himself, Yu, 29, knew well the importance of exhaustive testing.
Finally, he settled on the sentence: “Hey, Grandpa. Guess who I am?”
After what felt like an eternity but was only a matter of minutes, a video materialized. And a man who looked just like Yu’s grandfather, who had passed away 12 years ago, stared back. “Who you are is not important at all,” the man on screen said. “Life is a beautiful miracle.”
The words brought a smile to Yu’s lips, but his elation was short-lived. The man in the video was not the same person, and was in fact a digital projection that Yu helped create using his grandfather’s text messages, photos, videos, and letters.
In a feat of technological wizardry even Yu initially struggled to grasp, he had just successfully tested an AI-powered chatbot that even simulated his grandfather’s conversational style. The electric hum that accompanied his voice, however, made it sound unmistakably artificial.
Yu was 17 when his beloved grandfather died, and that day is still etched in his memory. His father told him of the tragedy on the way back home from school, leaving him shattered. Soon after, though he was sorely missed, Yu recalls no one in the family mentioned him anymore, primarily to protect Yu’s grandmother.
“It seemed that everyone in the family was trying their best to forget Grandpa rather than remember him,” says Yu. Every time he mentioned his maternal grandfather unintentionally, his mother would immediately signal to change the topic.
Adding to the heartache, Yu’s last memories of the man who helped raise him and shape his character were tinged with bitterness. He still hasn’t forgotten the time he yelled at his grandfather over a trivial issue related to video games. And he often wondered what he would say to his grandfather if he could turn back time.
Then, he stumbled across griefbots — artificial intelligence programs or chatbots designed to simulate a conversation with a loved one who has died. Also known as mourning bots, griefbots use machine learning algorithms to glean information from a person’s chat history, social media posts, and digital footprint to mimic their personality and demeanor.
In China, Yu’s digital facsimile of his grandfather may well be among the first. But in other parts of the world, particularly the U.S., the trend to create digital simulations of the dead has slowly gained momentum.
Though still at a nascent stage, the technology already has experts divided: While some believe griefbots offer friends and families a way to cope with the pain of loss, others raise ethical considerations around data protection and privacy regulation, particularly about denying people agency posthumously.
Among the more recent examples of a griefbot was in 2021, when a heartbroken man used a platform called Project December to create a simulation of his fiancee, who died in 2012 at the age of 23.
According to the San Francisco Chronicle, Project December used a machine learning model provided by OpenAI, the organization behind ChatGPT, the chatbot whose incredibly sophisticated, near-human replies have now made it an internet sensation. OpenAI has since shut access to Project December, citing ethical considerations and possible misuse.
The moral implications notwithstanding, Yu, haunted by regret for over a decade, yearned to see his grandfather once more: to bid a final farewell, to thank him, and to ask forgiveness for a past left unresolved.
And an AI-powered chatbot offered the closest semblance of a reunion, despite it being virtual.
Into the unknown
The first inklings of inspiration to build such a machine emerged in 2020, a year after Yu graduated with a master’s degree in computer science from the U.S. and started work as a programmer in his hometown of Hangzhou in the eastern Zhejiang province.
That year, he read an essay about lip syncing — the practice of silently matching lip movements with pre-recorded media to create a convincing illusion of speech or singing.
It triggered an intriguing prospect: “Can I see Grandpa again using this technology?”
For some years after his death, Yu had rarely thought of his grandfather. But while at college, far away from home, the memories came flooding back. Now a professional software programmer, Yu wondered if he could put his skills to use. Soon after, he came across Project December.
Delving deeper, he began poring over essays and articles on creating a griefbot using available AI tech.
They included GPT-J to generate natural language text for chatbots; audio-driven animation to synchronize lip and facial movements with speech; text-to-speech (TTS) software to convert written text into spoken audio; and transfer learning technology, in which a model trained on one task is used as a starting point for training on another related task.
He also learned of two approaches to start constructing a griefbot. In the first, the subject being mimicked engages in dialogue with the machine, which collects data from these conversations to render a digital replica. But this requires the subject to be alive.
Yu had no choice but to opt for the second method. Here, data is collected from all traces left behind, both digital and physical — photos, videos, letters, audio, and other memorabilia.
His idea crystallizing, he began his endeavor one weekend in 2021 at his grandmother’s apartment, when no other relatives were present. Hesitantly, he asked her: “Grandma, I miss Grandpa so much. Can you show me his photos?”
She gladly obliged. And as they browsed through them all, Yu asked for stories behind each one. He recalls being cautious during the conversation, and says he constantly observed his grandmother’s expressions.
“When I noticed that she appeared to be sad, I changed the topic or told her a joke to distract her. I was like a steersman, controlling the direction of our conversation. I just didn’t want her to be sad,” says Yu.
Along with the photos, she also showed him some of his grandfather’s possessions, like his calligraphy work.
Slightly encouraged, Yu felt it was the right moment to tell his grandmother about his project to digitally recreate her husband. Though she has a smartphone, Yu underscores that she knows little about technology, let alone artificial intelligence.
He recalls taking great care not to use the word “resurrect.” For his grandmother to take the idea seriously, he needed to carefully choose his words. Steeling himself, he asked her, “Would you like to talk to Grandpa again?”
“She thought I was joking and stared at me as if I was insane. She was on the verge of scolding me.”
“I repeated the word ‘advanced technology’ to her three times,” says Yu. And she finally realized her grandson was being serious.
With her by his side, collecting information about his grandfather had gotten off to a good start. But he still needed more raw data to train the bot and make it resemble his grandfather more closely — his traits, speech patterns, and personality.
He found that source of data in the corner of a dusty drawer. It was an old mobile phone Yu used while in middle school, containing dozens of text messages that he and his grandfather exchanged. Soon after, he found more videos and pictures stored in his cloud backup.
One was a video of his grandfather commenting on cooking fish: “This 80-yuan fish should have been braised. Steaming is a waste.” In another, his grandfather asks Yu not to shoot videos while eating dinner: “Stop running around. Help your younger brother get the rice.”
Of all that he had accrued, his grandmother gave him what proved most valuable, the veritable key to unlocking his grandfather’s past: letters his grandparents exchanged when they were young.
With her permission, and brimming with excitement, Yu took the precious letters home to further train the griefbot.
Closer and closure
Written decades ago, the reams of correspondence proved more than just mere metadata.
They helped Yu discover a side of his grandfather, who died at the age of 73, that had remained hidden from him all these years — he was finally peering into a world of memories and experiences that had shaped the man he thought he knew well.
For example, Yu took for granted that his grandfather did not like discussing politics. “I remember that whenever we discussed geopolitics or complained about the local government, he seemed disinterested and said little,” he recalls.
But the letters, written when his grandfather was in his thirties, painted a different picture. He was cynical, he criticized the government, and he was passionate about reforms and student issues.
Though his grandfather doted on him, Yu says he was undoubtedly “the strict leader and the guardian of order” in the family. “No speaking during meals. No phone calls either. Finish eating within twenty minutes. Grandpa made a series of rules for the family,” says Yu.
The dog-eared pages shed light on a more gentle side to his grandfather’s otherwise stoic persona. In particular, his efforts to court Yu’s grandmother. “He complimented my grandmom without any logic and reason; he told her where he went and about the delicious food he ate; he wrote ‘I miss you’ a lot,” says Yu.
“It was like finding out that your boring office colleague becomes a DJ at night,” says Yu. Every letter he read offered new insight. It only further cemented his resolve to complete a project he had started to make amends for past mistakes.
While Yu discovered his grandfather anew, the AI-powered bot chewed through the letters between his grandparents, gradually building a digital copy of its intended subject.
After several weeks, an elementary prototype was ready.
Welcome to the machine
Early into its testing, Yu recalls entering the prompt: “Happy Teahouse went off air” — referring to his grandfather’s favorite show, in which the performers spoke in the Hangzhou dialect. “That’s a shame,” it replied.
Yu wasn’t impressed. “I couldn’t tell if the bot had understood what I said or not,” he explains. So in a subsequent test, he decided to give the bot more conversational context.
This time, Yu started with: “What is your favorite TV show?”
“Happy Teahouse,” it replied.
“Oh, but Happy Teahouse went off air.”
“That’s a shame. The show I want to watch is no longer available. I would have liked to see a few more episodes.”
“Now it finally understands me,” Yu recalls thinking after that exchange.
However, he yearned for more. He wanted to enhance his grandfather’s facial features and voice. But he couldn’t get very far. The bot couldn’t provide instantaneous two-way communication simply because multiple models wouldn’t work simultaneously.
So he settled for a delayed response. At this stage, the interaction resembled an email exchange: Yu would input text in a simple black-and-white interface, and about 10 minutes later, the system generated a video of his grandfather’s response, which it automatically saved to a file.
“The technology is limited,” explains Yu. “The speed of analyzing and processing data is not fast enough. It’s more like writing letters to my grandfather, which was one of the ways that he liked to communicate.”
Though such successful tests did present an illusion that the griefbot could “understand” what Yu was talking about, the bot often did not give him a sensible response. “Sometimes, the answers were just ridiculous,” he says.
To illustrate, Yu refers to the time his grandmother requested that he key in a message: “We are fine here. Don’t worry about us. How are you in that world?”
The bot replied: “I am fine here. Do you want to come and play?”
He frowned. Not only was the sentence callously phrased and unfeeling in tone, it also implied that his grandfather was inviting his grandmother to the afterlife.
“AI is uncontrollable,” Yu says. “You cannot predict what it will do in the next second. It may quarrel with you one moment and smile at you the next. It may flirt with a girl normally, but the next second swear at her and ask her to die.”
According to Julia Stoyanovich, a data scientist at New York University’s department of computer science and engineering, the unpredictability of the bot’s response was partly due to the fragility of data.
“You take data from multiple sources and put it together. How you map data and how you integrate the data can be very arbitrary. If you try the same process the next day, you might get an entirely different result,” she explains.
She also underscored that each griefbot required delicate customization — the bot simulating the same person should have different versions for different users. “We need to make sure that the information doesn’t leak across these boundaries of access,” she says. “It’s not about who creates it. It’s about who is going to use it.”
The unpredictability of responses implies that a griefbot is likely to hurt people. The bot’s response to Yu’s grandmother’s greeting could have upset her, which he could ill afford, particularly since he knew his parents would never condone his actions.
So Yu edited the machine’s response, removing the offensive sentence that invited his grandmother to the afterlife. Once complete, he saved the new video and took it to his grandmother’s apartment.
Proof of life
In silence, Yu’s grandmother watched her husband seemingly come back to life on screen. Once one video ended, she glanced at her grandson to indicate that he could play the next.
As the last video faded out, she seemed lost in thought. A few moments later, she looked at Yu and, her voice trembling, simply said: “Thank you,” before returning to her room.
Uncertain, Yu stood outside her door and listened but heard nothing. A while later, she walked out, headed to the kitchen, and started to prepare meals, as if nothing had happened. They never spoke about it again.
“My mom, dad, aunt, and her husband have always protected my grandmother by not mentioning my grandfather. But from what I know, that isn’t necessarily what she wants. Maybe she needed something like these videos to process the emotion and grief,” says Yu.
“That’s my interpretation,” he reflected. “Otherwise, why did she thank me?”
The videos also proved cathartic to Yu.
Through Yu’s time in primary school, his grandfather always dropped and then picked him up on his moped. But in middle school, something changed.
“I felt I wasn’t a child anymore. I was almost an adult,” he says. “If other kids saw me sit behind my grandfather on the moped, it would have been embarrassing.”
One day, Yu suddenly told his parents that he didn’t want his grandfather to pick him up at school anymore. Several days later, he heard his grandfather mention it to the family.
“Grandpa was unhappy and complained about it. He said: ‘He has grown up, and does not want me to pick him up anymore.’”
Also during middle school was the temper tantrum Yu directed at his grandfather — from which stemmed his plan to build a griefbot.
On that day, Yu was playing computer games at home. His grandfather came visiting out of the blue to pick him up for dinner. But Yu, a teenager in his rebellious phase, was addicted to the game. He recalls yelling at his grandfather hysterically after his computer crashed.
For all these years, that incident tormented Yu. “He could have called me and asked me to take a bus to his apartment to eat dinner. But he chose to walk to my house to pick me up. I knew he loved me so much,” he rues.
In the exchanges with the griefbot, Yu told his “grandfather” that he was sorry for the two incidents that left deep scars.
Asked to see the exchanges, Yu refused.
“I don’t want to share those videos,” he said, before adding: “But I think my grandfather forgave me in the end.”
Return zero
From the first seeds of an idea that used AI to his moment of epiphany, Yu spent about six months perfecting his grandfather’s digital replica. Though ultimately successful, he admits he had to navigate a maze of mistakes, anxiety, and conflicts within.
For days, he grappled with the question of whether his grandfather would have even permitted him to read his deeply personal letters if he were still alive.
“My answer was that I don’t believe he would have let me see them. Because to him, reputation meant everything,” says Yu. But he went ahead anyway. “I still wanted to make his persona more comprehensive.”
So he spent a lot of time adjusting his grandfather’s face. “Sometimes its responses were weird, and sometimes his face was distorted,” explains Yu. He recalls asking himself: “Is this offensive? Should I do that? Also in those moments, I realized it was a bot after all and not a human.”
Apart from telling his grandmother, Yu took great pains to keep the project away from the rest of his family, particularly his mother. “If Mom saw those mistakes and the flaws in the griefbot during this process, she would surely have scolded me and stopped the project,” he says. She would have also found those distorted images of her father very disrespectful.
He wanted to shut the project down the first weekend after he finished generating all the videos. But at the time, he was reluctant to say goodbye yet. So he kept the griefbot for another week.
His original purpose was always clear. “I only wanted to apologize and bid farewell. That was the decision I made at the beginning.”
But why not keep it running?
Yu says, “I was afraid of relying on this bot too much. I was afraid that I would not be able to move on if I kept conversing with it. These emotions might have overwhelmed me too much to work and live my life.”
Two weeks after bringing the griefbot online, it was time to make the hard call. Gazing wistfully at his computer screen, Yu mouthed the words, “Bye, Grandpa.” And hit delete.
|
