Artificial intelligence misinterprets its instructions to produce unintended side effects
“Check.”
Lieutenant Jeffrey Carson gazed at the black and white scramble and tried to make sense of the board. His opponent smirked, the navy-blue coat of her New Western Front uniform giving nothing away. Sarah MacIntyre was a seasoned veteran of the Interzonal Chess Events (or ICE for short) and played the part. In just a few minutes, she had tangled Lt Carson’s black soldiers around the board, leaving just enough of a gap for her own pieces to gain entry to the keep. Now the black pieces were comically strewn about, seemingly confused if their neighbours were friend or foe. Sarah liked to imagine the pieces with little swords and guns and other relics of their barbaric past.
Click. The magnetic snap of Lt Carson’s next move brought her attention back to the board. She frowned, cocked her head, and sailed her knight over the final defences.
“Checkmate,” she said with a grin as Lt Carson swore and pushed his chair away from the table, “but four moves more than last game!”
Lt Carson just scowled and wheeled over to the giant monitor to type in the most recent score. It was a dismal sight, with his column lit up in red apart from a single yellow mark. That was when Sarah had decided to play with her eyes closed, claiming she was training her intuition. And even then, he’d only managed to draw.
“Up for another game?” Sarah asked, already resetting the pieces.
“Not today – I’ve had enough of losing.” Lt Carson grabbed two Sonic Shakes from the cooler and zoomed back over to the table. Sarah was still staring at the board, mapping out potential games in her head.
“You know,” she said, “They used to use this to plan battles. We did, I mean. Well, not us, but – “
“Our more savage ancestors.” Lt Carson interrupted. “Yea, I read about that somewhere. Probably The Compendium, come to think of it.”
“What do you think it was like? Back when people used to fight.” Sarah shuddered at the thought. How could human beings do that to each other? The records said that national boundaries used to be drawn based on actual conflict. What a ridiculous way to settle things!
“Loud, I’d imagine. There used to be hundreds and thousands of people just firing at each other.”
“Do you think they ever felt bad about what they did?” Sarah continued. She tried hard to think about what it would feel like to take someone’s life. The attempt almost made her regurgitate her Sonic Shake.
“Sounds like they didn’t really have much of a choice.” Lt Carson had read a lot about the wars in humanity’s past. He was aware that his rank used to come from a military hierarchy. Now of course, it just meant his Fantasy Frisbee team was a force to be reckoned with. He loaded up a game on his Diskscreen and tapped idly. “Reckon we’d ever go back to it?”
“Of course not!” Sarah said horrified, “It’s absolutely barbaric! Besides, there’s nothing to fight over anymore.” That was true. It seemed that humans used to fight over land and resources and because they didn’t agree with each other. After they made the WAM, they couldn’t fight anymore. And then they realised there wasn’t really that much in the whole fighting thing in the first place.
Lt Carson whooped as his team scored again. Now this was something he was good at. Not war games, but simple commands to his robotic squadron. He looked up and saw Sarah was still staring down at the chessboard.
“Remember the story of the first guy who went against the WAM?” Lt Carson said, “It just scooped him up and dropped him back in New Zealand.”
“Yea, crazy to think people were actually against it, too.”
“Wouldn’t you be if your whole life was about war?”
Sarah thought for a bit, then walked over to the display monitor. She typed in a few commands and a video started playing. Lt Carson looked up for a second to see she’d pulled up a documentary on the WAM. But the siren from his Diskscreen made him look back and focus so his opponents didn’t score. Not today. He wasn’t losing again.
“The War Automation Machine or WAM for short, was an artificial intelligence program created by the fourth Grey Performance Trust. Initially built to effectively end conflicts, the WAM performed remarkably in training and test circumstances with a 99.7% success rate. It was then subsequently deployed in the Storm Wars of 2091 to finish off the insurgent forces. It was only after the WAM had infiltrated the general security systems and hijacked both sides that the potential bug in its code became clear.
The WAM was written to avoid human casualty at all cost, but the researchers were surprised to find out that it had learnt to associate any sort of defence measures and weaponry as being linked to human losses. In doing so, the WAM had secured the military capabilities of the entire world in a matter of minutes, jumping between different intelligence agencies’ databases and backdoors. The Storm Wars ended abruptly, as did all other conflicts. So driven was the WAM to its cause that it simply took away weaponry from humanity.
Without anything to fight with or even the freedom to harm each other, humans eventually turned to other contests. Sports saw an uprising, especially combat sports – though the WAM stopped anyone from getting any permanent injuries. Today, with civilisations on Earth, Luna, Mars, Europa and in the space between, the WAM helps detect incoming space debris and other dangers to human life.”
Sarah muted the documentary but kept watching as the camera panned in the 3D environment across the various cities humanity now occupied. After a while, her eyes were drawn back to the chessboard and the starkly contrasting warriors standing at attention. Crazy to think that this simplistic game used to foreshadow the deaths of thousands.
“Wooo!” Lt Carson yelled as his Diskscreen lit up with a victory jingle. He popped it back in its tray and downed the rest of his Shake. “Alright, let’s go!”
Sarah looked up in amusement. “You want another game? What happened to not wanting to lose again?”
Lt Carson chuckled and made the first move this time. “Ah, it’s just a game. A loss won’t kill me.”
Seventeen moves later, the Diskscreen was back out.