This month, a local TV-news station in Arizona ran an unsettling report: A mother named Jennifer DeStefano says that she picked up the phone to the sound of her 15-year-old crying out for her, and was asked to pay a $1 million ransom for her daughter’s return. In reality, the teen had not been kidnapped, and was safe; DeStefano believes someone used AI to create a replica of her daughter’s voice to deploy against her family. “It was completely her voice,” she said in one interview. “It was her inflection. It was the way she would have cried.” DeStefano’s story has since been picked up by other outlets, while similar stories of AI voice scams have surfaced on TikTok and been reported by The Washington Post. In late March, the Federal Trade Commission warned consumers that bad actors are using the technology to supercharge “family-emergency schemes,” scams that fake an emergency to fool a concerned loved one into forking over cash or private information.  

Such applications have existed for some time—my colleague Charlie Warzel fooled his mom with a rudimentary AI voice-cloning program in 2018—but they’ve gotten better, cheaper, and more accessible in the past several months alongside a generative-AI boom. Now anyone with a dollar, a few minutes, and an internet connection can synthesize a stranger’s voice. What’s at stake is our ability as regular people to trust that the voices of those we interact with from afar are legitimate. We could soon be in a society where you don’t necessarily know that any call from your mom or boss is actually from your mom or boss. We may not be at a crisis point for voice fraud, but it’s easy to see one on the horizon. Some experts say it’s time to establish systems with your loved ones to guard against the possibility that your voices are synthesized—code words, or a kind of human two-factor authentication.

One easy way to combat such trickery would be to designate a word with your contacts that could be used to verify your identity. You could, for example, establish that any emergency request for money or sensitive information should include the term lobster bisque. The Post’s Megan McCardle made this case in a story yesterday, calling it an “AI safeword.” Hany Farid, a professor at the UC Berkeley School of Information, told me he’s a fan of the idea. “It’s so low-tech,” he told me. “You’ve got this super-high-tech technology—voice cloning—and you’re like, ‘What’s the code word, asshole?’”

But we also should be wary of getting paranoid too quickly. A broader loss of trust in any audio, or video for that matter, could feed the “liar’s dividend,” or the idea that more public knowledge about fakes can make it easier for bad actors to undermine legitimate media. America doesn’t exactly have a surplus of trust right now: Faith in media and institutions, including organized religion and public schools, is polling miserably, at the same time that AI is amplifying the ability to spread false information online. “We want people to be aware of what’s possible,” Henry Ajder, an AI expert who has been studying synthetic voice technology for half a decade, told me. “We also don’t want to just absolutely terrify people.” If you do get an out-of-the-ordinary call, you can always just stay calm and ask commonsense questions that your loved ones should know how to answer, Ajder said.

Beyond the anecdotes, data about AI voice scams are practically nonexistent. Juliana Gruenwald, a spokesperson for the FTC, told me that the agency does not track how often AI or voice cloning is used in scams. Fraud-report statistics for the first three months of this year don’t show an increase in the number of scams involving the impersonation of family and friends. The FBI, which also keeps data on phone scams, did not respond to a request for comment.

[Read: The rise of AI Taylor Swift]

Still, there’s clearly genuine risk here. Last month, for a story about the proliferation of such clones on TikTok, I replicated Taylor Swift’s voice using just one minute of audio of her talking in an old interview on YouTube. It took five minutes and cost $1 using the online Instant Voice Cloning tool from ElevenLabs. (The company did not respond to a request for comment about how its software could be used in scams.) All the program needs is a short audio clip of the person speaking: Upload it, and the AI will do the rest. And you don’t have to be a high-profile figure to be vulnerable. It simply takes one public audio clip of you, perhaps pulled from a TikTok or an Instagram post or a YouTube vlog, and anyone can create an AI model of your voice that they can use however they choose. Our extensive digital histories, built over years of life online, can be used against us.

Although the technology feels like it’s lifted from a Philip K. Dick novel, this is, in a sense, a classic American story about the uncertainty of a new frontier. The historian Susan Pearson, who wrote The Birth Certificate: An American History, told me that when more Americans began moving from the countryside to cities in the mid-19th century, the country developed “a real cultural fascination” with swindlers and an “anxiety about being in these new large spaces, where all kinds of strangers are going to interact and you don’t necessarily know who you can trust.” We developed technologies like credit scores, for better or worse, so that we might know who we were doing business with. The expanse of the AI-powered internet is perhaps a corollary to that earlier fear.

We’re in a period of change, trying to figure out the benefits and costs of these tools. “I think this is one of those cases where we built it because we could and/or because we can make money from it,” Farid said. “And maybe nobody stopped to think whether they should be doing it.” There are some legitimate use cases for voice cloning: It could empower a person who has become impaired or lost their ability to use their own voice, for instance. In 2021, AI helped the actor Val Kilmer use his voice when he lost his natural ability to speak as a result of throat cancer. But the beneficial uses don’t necessarily require unregulated, free-for-all access, Farid pointed out.

Many critics of AI have said we should slow down and think a little more about what the technology might unleash if left alone. Voice cloning seems like an area in which we really ought to do so. Perhaps humans will evolve alongside AI and create new verification technologies that help us restore trust, but fundamentally, once we start doubting that the person on the other end of the line is really the person we want it to be, we’ve entered an entirely new world. Maybe we’re already there.  

US intelligence agents are after scores of Russians in connection with dastardly global plots and outright criminality.

Of the roughly 1,000 people who have been charged for their participation in the January 6, 2021, assault on the Capitol, nearly a quarter were indicted alongside a relative or romantic partner. All sorts of other close personal relationships run through the indictments: the two roommates from Ohio who have known each other since they were kids, the three high-school buddies from North Carolina, the three Marines from Camp Lejeune, a Florida man and his pastor, an electrician’s apprentice and his boss. The conventional wisdom about radicalization is that ideas attract people to extremist movements and to the violence those movements commit. Most adherents, though, never move beyond reading manifestos, watching videos, or plunging down internet rabbit holes. Very often, what differentiates those who commit overt violence is their personal ties to others in the movement. Because although extremist movements are ideological, extremist violence turns out to be strikingly social.

As a historian of American social movements, I’ve found that the perpetrators of violence tend to be pulled along not by ideas alone, as compelling as these people may find them, but also by the power of personal connections. These individual loyalties can create obligations so intense that they permit those who feel them to justify committing horrors.

[From the April 2023 issue: The new anarchy]

The phenomenon reaches far back into the 20th century. Think of the lynching of Abram Smith and Thomas Shipp. These two young Black men were accused of killing a white man, but their lynching was not determined by that. Instead, their murder, in August 1930 in Marion, Indiana, followed a grim pattern that became one of the most common forms of political violence in 20th-century America: an accusation of rape leveled by a white woman who had reason to lie, a quick arrest, the accused Black men’s imprisonment in a jail that police weren’t willing to defend when a mob descended. Once the jail’s doors had been battered down, Smith and Shipp were dragged from their cells and hanged from a maple tree on the courthouse lawn, two blocks away. The aftermath of the killings was captured in a commemorative photo sold as a postcard for 50 cents that showed members of the mob smiling for the camera, the young Black men’s brutalized bodies dangling above them.

A few days after the lynching, the executive secretary of the NAACP arrived in town to conduct a quiet investigation. His report on the murders, which was widely distributed through the organization’s press service, resulted in a storm of publicity that forced the county’s district attorney to respond. Together with the attorney general of Indiana, the D.A. launched an inquiry that stripped away the anonymity that normally protected lynch mobs’ leading participants.

The official investigation found that the assault on the jail began when the sheriff refused to hand Smith and Shipp over to the father and the uncle of the alleged rape victim, 18-year-old Mary Ball. Although the mob swelled to more than 1,000 people at its peak, witnesses attributed the worst of the violence to a core group of 21 men, 17 of whom, by my tally, lived close enough to one of the Ball brothers to be called their neighbors. White supremacy underpinned the killings of Smith and Shipp, but the rage of those who committed the atrocity was mobilized by the intimate ties of family and community.

Sixty-five years later, on April 19, 1995, a former soldier named Timothy McVeigh pulled a rented Ryder truck into the delivery zone of Oklahoma City’s Alfred P. Murrah Federal Building. He primed the fuses on the nearly 5,000-pound fertilizer bomb that he and his friend Terry Nichols had assembled in the truck’s cargo bed. Then he got out, locked the truck’s door, and walked toward the car he’d left in a parking lot several blocks away. He was still on his way there when the bomb detonated, demolishing the front of the building and killing 168 people, including 19 children, many of whom had been in a day-care center right above the delivery zone.

[Read: Alt history]

McVeigh saw his political purpose as guided by the anti-government extremism he’d embraced in the white-power militia movement. On the morning of the bombing, he carried with him pages from The Turner Diaries, the apocalyptic 1978 novel whose white-nationalist hero counters Washington’s mounting oppression by blowing up the FBI’s headquarters. In that respect, there would seem to be a straight line from his ideological conversion to the violence he committed that day. But he had been immersed in the movement for seven years before his attack on the Murrah building. For most of that time, he had engaged in only the pettiest of political acts—until he became bound by the personal ties that ran through the militia movement as strongly as they did in the mob that murdered Shipp and Smith.

Those connections started to form in the mid-1980s, when James Nichols brought Terry, his younger brother, into the militia that was taking root around their home in rural Michigan. Terry joined the Army in 1988. During his basic training at Fort Benning, in Georgia, he befriended McVeigh, whose own interest in white-power survivalism had been inspired, in large part, by The Turner Diaries. How much influence Terry then had on McVeigh is not completely clear, though McVeigh officially joined the movement during the first year they spent together, following a spell as a member of the Ku Klux Klan.

The two men had seemed to drift apart after Terry washed out of the Army in 1989, but they reconnected when McVeigh was discharged two years later. Terry was then living with James on their family farm. In early 1993, McVeigh also moved in. There, James revived his big-brother role as mentor, feeding his brother and his friend with movement propaganda, ferrying them to militia meetings, and stoking their anger in long conversations he tended to dominate. The two younger men took almost every step leading to the bombing together, except for the single act of driving the Ryder truck, which McVeigh did alone. When he was arrested, little more than an hour later, he listed the Nichols farm as his home and James as his next of kin.

In his lawyer’s telling, 24-year-old Hunter Seefried had no interest in Donald Trump’s January 6 rally at the Ellipse. He had voted for Trump. And he was sure that his father, Kevin, was right when he said that the Democrats had stolen the election. It just didn’t bother him enough to devote a day to protesting it. But his dad wanted the family to go with him, and saying no was likely to cause more trouble than it was worth. So, that morning, Hunter and his girlfriend made the two-hour drive with his parents from their small town in Delaware to Washington, D.C.

[Cynthia Miller-Idriss: Extremism has spread into the mainstream]

When the rally was over, Kevin insisted that they join the march on Congress, though they had planned on having lunch and heading home. Once they got to Capitol Hill, Hunter’s mother and his girlfriend faded into the crowd, while Hunter and his father worked their way to the front. They reached the west portico just as the first few rioters were climbing through a window that a Proud Boy had smashed open. Hunter carefully removed the last shards from the frame. Then he and his dad climbed in too.

They moved through the building together, up the stairs with the mob trying to chase down the Capitol Police officer Eugene Goodman, into an ugly confrontation with the backup officers Goodman had called. The two of them joined the mob’s front line, Kevin screaming at the officers to step aside or shoot him. When it became clear that the police wouldn’t back down, Hunter and Kevin left the building, 25 minutes after they’d entered.

Three months later, father and son were indicted together on seven misdemeanors and a felony charge of obstructing an official proceeding.

We don’t yet know enough about all of the extremist organizations that were involved in the January 6 attack to be able to trace their webs of affinity with the same precision. But what we already know is revealing: The Seefrieds are hardly alone.

The federal indictments identify the 14 Oath Keepers who, at the height of the riot, marched through the mob in military-style stack formation. Six of them had come to the Capitol with a relative who was also an Oath Keeper. Two others, both military veterans, had a tight friendship. One of the pair had also found a mentor in another Oath Keeper, 15 years his senior. “Love the hell outta you,” the younger man texted him on January 8. “You too, my dear friend!” the mentor texted back. “We stormed the gates of corruption together (although on opposite sides of the building) so between that and our first meeting and getting to know you since I can say we will always be brothers!”

Hunter Seefried was no Oath Keeper. He was a barely political young man following his father, whose own radicalization had not gone beyond following right-wing news sites and pro-Trump social media. Yet they were among the first rioters to breach the Capitol building, half an hour before the Oath Keepers started their march.

At Hunter’s sentencing hearing this past October, his lawyer argued that he didn’t deserve prison time; Hunter had put himself in the mob’s vanguard only because he was a dutiful son, and not as an insurrectionist. But America’s history of violent extremism makes that distinction meaningless. The Seefrieds turned to violence as so many people had before them, through the tangling together of dangerous ideas and intimate obligations. The more extremism spreads into the mainstream, the more likely that combination is to take hold again. As the personal pulls the ideological closer and closer to the center of American democracy, there is no limit to the damage this potent combination might do.