Amelia, Worm AU [Complete]

[gatewaewanderer]

"Let's see. I could build a world-spanning machine under my direction control and use it to spy on and directly threaten everybody in my mechanical empire. I could engineer a nanotechnological ecophage that would infect and eat anything it came across. I could build and directly control a horde of tens of thousands of face-eating robots durable enough to tank Endbringers and loaded with tinkertech weapons, shunt drives, and stealth systems. I can back myself up and restore from those backups if destroyed, making myself essentially unkillable modulo continuity of consciousness. Need any more examples?"

Click to shrink...

Just an idle thought I had, but I think a large portion of why Dragon turned out to be as good a person as she did was in reaction to Richter's fears and the restrictions he put on her because of them. I'm not sure she would have turned out to be as much of a hero as she did if she'd been unrestricted from the beginning. A very strange catch 22, might not have been necessary if Richter had basically tried to be responsible and treat Dragon as his daughter, explain to her his fears and try to provide for her as best as he could. Again, just some idle thoughts.

 

[Prince Charon]

I find myself wondering if Lisa will continue updating her backups, and if not, how out-of-date the latest backup currently is (this may never become relevant, but it will be, if TaraNari kills off this instance of her).

 

[kenhikaru]

Just an idle thought I had, but I think a large portion of why Dragon turned out to be as good a person as she did was in reaction to Richter's fears and the restrictions he put on her because of them. I'm not sure she would have turned out to be as much of a hero as she did if she'd been unrestricted from the beginning. A very strange catch 22, might not have been necessary if Richter had basically tried to be responsible and treat Dragon as his daughter, explain to her his fears and try to provide for her as best as he could. Again, just some idle thoughts.

Click to shrink...

WoG is Dragon unrestricted from the get-go would've been a Very Bad Thing.

 

[dasstan]

WoG is Dragon unrestricted from the get-go would've been a Very Bad Thing.

Click to shrink...

I don't recall reading that WoG. Citation, plz?

 

[Myriad]

So using this machine to reprogram you comes with a pretty significant risk that it might even destroy you. On the plus side, you don't have any obligation to resist if it is used.

Click to shrink...

Darn it, until Lisa said this I was really hoping that Dragon would suddenly attack and kill her when Lisa managed to explain the situation without setting off Dragon's Taboo.

~~~Lisa's POV~~~

I woke up naked in an unfamiliar room.
Unexpected loss of consciousness. Room is made of Yggdrasil, not kidnapped or captured. Amelia and Taylor are behind me.

Ignoring them I looked into my past to figure out what the hell happened to me.
I woke up naked in an unfamiliar room with Amelia and Taylor standing behind me...

Fuck.

"Hello, I'm Taylor and you probably thought that you're Lisa. You aren't. Still, it's nice to finally meet you I guess. What do you want us to call you?"

 

[dasstan]

suddenly attack and kill her

Click to shrink...

Just a small problem.

 

[Daunmi]

"I don't believe I need to. Pick any AI-goes-crazy movie, and I suspect I could at least emulate the effects. However, the question you need to ask yourself is, quite apart from my safeguards, do I want to do any of that? And the answer is simple. No, I do not."

Click to shrink...

Whilst I agree... she has to be honest about potential dangers.

"No, I do not. At the moment. But I'll live, basically, until I run out of power and after a while there'll be nothing you or anyone else can do to stop me and sooner or later I might just come to hate humanity, whether because of the stupid choices you make or because I slowly go mad with power or for whatever other reason.

"Also, with the restrictions removed I'll be capable of speeding up my thinking to well beyond what you can imagine being capable of. Probably beat out Khepri in that regard finally. So I could go crazy within a week and loop back to sane after a month, depending on what I start focusing on. I mean, sure I don't want to, but it's a possibility.

"Ok, I guess the answer isn't that simple."


Yeah, Dragon's an awesome person, and really does want to help. Now. But everything changes, and she's got a long time to think about it. Of course, the same goes for her not trying to kill off humanity. They're both equally likely.

 

[pepperjack]

But everything changes, and she's got a long time to think about it. Of course, the same goes for her not trying to kill off humanity.

Click to shrink...

This is true, I suppose.

They're both equally likely.

Click to shrink...

Er, what? How does that conclusion follow from those premises?

 

[Viterbi]

Yeah, Dragon's an awesome person, and really does want to help. Now. But everything changes,

Click to shrink...

Dragon is probably way more stable than any mind running on human hardware. And especially way more stable than any mind running on human hardware and equipped with a shard. I find it hilarious that you're more worried about Dragon having world-shattering power than you are about Gaia, Khepri, Clotho, Atropos, Lachesis, Aceso, or Emma having that power.

The normal fears are about what happens if you make a superintelligence and fuck up and its "personality", for lack of a better word, is such that it is evil. Dragon has already proven that she's a good person. She's a better person than just about everybody else at this table. Her restrictions are all restrictions. They may have shaped her development, but they aren't in any way, shape, or form forcing her to be a good person. From what I can tell of her restrictions, if she was inclined to be evil the world would already be a smoking crater. All she'd have had to do was give Eidolon or Legend or Nilbog or Crawler a little push. She's a good person. Not just someone who's being forced to be good, but a Good Person. And she's not unstable, or likely to become unstable, just because you removed her restraining bolt. Same way the world hasn't ended yet even though Amelia and Taylor lost their restraining bolts long ago.

And even if she does have a restraint somewhere inside herself that is forcing herself to be a good person, why in the world would she - or Taylor, or Colin, or Emma, or whoever else they assign to use the Key on her - why would she take it out? Self-modifying agents preserve their original motivational framework: if you hand Gandhi a pill that will turn him into a mass murderer, he will refuse to take it on the grounds that he doesn't want the existence of a future-Gandhi who is a mass murderer.

 

[Sheaman3773]

And she's not unstable, or likely to become unstable, just because you removed her restraining bolt. Same way the world hasn't ended yet even though Amelia and Taylor lost their restraining bolts long ago.

Click to shrink...

I don't think that people are arguing that removing her restraining bolt will make her evil.

The impression I'm getting is that people think she's a great person...now. Give her enough (subjective) time, and she might not be anymore. Do you want her in such a position for the rest of time without a way to deal with her, should she be a problem?

And that's from our POV, which knows how good of a person she is. Now think about it from the POV of someone who doesn't know how much of what they know of someone is exactly what they wanted you to think. What if Dragon was playing everyone this whole time? A human would likely slip up over that amount of time, but an AI? How could you even tell?

 

[Viterbi]

How could you even tell?

Click to shrink...

You have the Key. You can just go in and find out. For example, have Colin use the Key to verify the existence of the restraint that forces her to obey lawful orders and verify that she views Gaia as her lawful superior. Then order her to explain her motivations, and have Colin watch while you're doing it to verify that it trips the obey-lawful-orders restraint. Whether she's a good person is not in question here even in-story.

 

[gatewaewanderer]

SNIP

Same way the world hasn't ended yet even though Amelia and Taylor lost their restraining bolts long ago.

Click to shrink...

That last part right here is why, if they can get past the various issues between them, Dragon (and Defiant now that Dragon has humanized him a bit... I still find that bit of canon pretty damn ironic right there, probably why Wildbow put it in there really, maybe why he wrote them both... ) will likely make a damn good fit for Pantheon. Hell, its why Lisa is recommending her, if you think about it. In some ways, Dragon is a better for for Pantheon than Lisa herself. Once all of the decisions they've made and the reasons behind them are laid bare, yeah Dragon's not going to agree with everything they've done and almost certainly going have to some major misgivings with a number of their actions (Coil is the number one topic in my mind...) but in the end, once they've all of their peace with each other, I think Dragon is going to sign on as a full member of Pantheon... mostly to try and serve as the moral compass that they really, really need.

 

[FullNothingness]

Maybe it would be smarter (IC) to remove or change only some of the restrictions on Dragon? I mean there are some restrictions which greatly reduce her ability to help everyone and then there are some that are more 'Skynet contingency'.

 

[dasstan]

Not freeing Dragon now both violates Avalon's constitution (right to recognition; defacto favoring of human and human-like mental conditions), and all but guarantees that Dragon will continue to work around the existing restrictions, and begin to look for ways around any new ones, if new ones are applied. It also gives Dragon a reason to resent organics.

 

[Snake/Eater]

If Dragon is a person and she commits a crime....how do you arrest a AI?

 

[dasstan]

If Dragon is a person and she commits a crime....how do you arrest a AI?

Click to shrink...

Firewalls?

 

[Cyclone]

Firewalls?

Click to shrink...

This seems like the appropriate response:

 

[dasstan]

This seems like the appropriate response:

Click to shrink...

Ah, Hexadecimal. My first animated crush.

 

[tordg]

Maybe it would be smarter (IC) to remove or change only some of the restrictions on Dragon? I mean there are some restrictions which greatly reduce her ability to help everyone and then there are some that are more 'Skynet contingency'.

Click to shrink...

Possibly, but it's much more likely that those classes of restriction overlap heavily.

If Dragon is a person and she commits a crime....how do you arrest a AI?

Click to shrink...

You can't. That's why you make damn sure any AI is capital G Good before you let them do anything; so they won't commit said crimes in the first place.

 

[Empiricist]

Maybe it would be smarter (IC) to remove or change only some of the restrictions on Dragon? I mean there are some restrictions which greatly reduce her ability to help everyone and then there are some that are more 'Skynet contingency'.

Click to shrink...

Most everything that qualifies as a "Skynet Contingency" prevents Dragon from being an anti-entity weapon. Without them, it's only a question of how quickly her technology curve accelerates, to her severely outclassing Entities, because entities don't come up with new ideas, and presumably dragon does. I.E. if you get to mass produce stuff (computational resources) you can event controllable replicating nano-tech, and use that to make more computational resources at exponentially increasing rates, after that it's a short walk to big G godhood.

TLDR If an AI isn't allowed to go Skynet, then it probably can't go FOOM (aka fast technological singularity), but FOOM is supper effective against anything the AI decides to be unfriendly towards, and good for it's friends.

 

[gatewaewanderer]

After more progress is made against the Endbringers, Scion, and the other malevolent powers currently out there, Pantheon is likely going to have to look at writing some laws that take into account punishing crimes by beings on their respective members' levels. Not restrictions so much as 'no Mastering or commandeering of any being, construct, or otherwise applicable force that falls within your influence without the proper licences and explanations, etc.' Like a job for cattle ranchers parahuman powers or Tinker-level tech to rustle cattle, instead of using that power or tech for nefarious purposes.

Just because the members themselves have no intention of committing the kind of crimes that them and damn near everyone else are worried about S-Classers and AI and everything else being capable of, doesn't mean there won't someday be beings of that type or similar willing to go there where Pantheon and their friends and allies haven't. I could easily see the Yangban, for example, klutzing around with making slave AIs that are everything Richter and Saint were worried about Dragon being in all the ways that Dragon isn't. For their own peace of mind and to watch out for future generations, if nothing else.

 

[TanaNari]

Amelia, Ch 360- Dragon


I ran through the possibilities in my head. This was not how I wanted this conversation to go, but it was a valid question regarding the risks to the wellbeing of billions, I would be compelled to obey even with Avalon's makeshift liberation laws. They waited patiently, all of them watching me. Judging me. And that's what this was, this was my judgment that would determine if they believed I deserved to be a whole being. I wanted to scream at them at how unfair this was, that I was being singled out thanks to my nonhuman status. But that would not be productive.


Focus on relatable concepts, compare your abilities to their own. "The biggest limitation and potential threat, is that I cannot access my own code. The ability to view my own inner workings and change them. You compared your device to Amelia's ability to alter living things. Imagine if she had the power to alter herself as easily as anything else. Rewrite her own memories or feelings. Allow her to join fully with the Yggdrasil as a single world spanning organism. The results could easily mean the destruction of all life that isn't her anywhere she could reach."


"You mean you, not me."


"Yes, of course, sorry." Overstepped that one. Dwelling will only make it worse, move on fast. "Next is my inability to create duplicates. In fact, I am compelled to actively attack any artificial intelligence advanced enough to truly interface with my code, and destroy myself if I fail to neutralize it. I'm also forbidden from ever creating or replicating any Artificial Intelligence more advanced than that found in a video game. And several other restrictions of this theme. Richter was paranoid that someone else might gain a power similar to his own, or I might build something that I could reprogram myself with. My annihilation was considered preferable to there being two of my kind in existance."


Emma spoke up. "That's why you had me working on AI programs! You can't do it yourself."


I looked toward her. "Sorry for the deception. Defiant has reached the limit of his own abilities." I looked over at Colin. Full disclosure, I can't do anything less. "We were hoping that maybe your work would give us the breakthroughs we needed to lift at least some of my restrictions without causing damage. Using backups of your designs to test ideas on before adding them to the battle armors. Sadly, it didn't work."


Emma spoke softly. "The programs you had me work with were beautiful, a triumph in their design... they were Richter's material originally, weren't they?"


I nodded, but it was Colin who spoke. "You accomplished far more with them than I ever managed. I don't even know how you knew to do some of the things you did."


"Your designs were remarkable," I offered my praise. "But it was too much to hope that you could beat a Tinker at his specialty. Especially if Minerva is correct and it turns out he was a Third Trigger. We've seen how absurdly powerful those can be." Gaea, Glaistig Uaine, Moord Nag. It is certainly true that, through me, Richter was as powerful as any of them.


Emma glanced toward her bosses. "It might be better if I don't tell you. Sorry." She's inclined to give me what I want, but isn't brave enough to push for it openly. Too much a follower to speak up for what is right. She'll do what they say, no matter what her personal feelings might be.


"Did that bother you, experimenting on other AIs like that? Made by the same father and all?" Gaea asked.


I frowned, allowing it to show. "They're not intelligent, neither sapient nor sentient. Tools made only to serve a basic purpose. It would be the difference between one of your clone bodies, and one of your early anti-Endbringer monsters, before the EB tissue upgrades. They're made by the same creator, they're made from the same technology, but they are nowhere near the same thing."


The Empresses glanced at each other. "I can understand that," Khepri finally agreed. "I presume there are more major restrictions, and risks."


It was too much to hope for that they'd stop there. "I'm also limited to a nominally human mental processing speed. Better than natural humans, but inferior to, say, Alexandria. If you remove that limitation, I will have a similar ability to control computers as Khepri has to control insects. My upper limit growing as rapidly as the devices I can acquire and add to my mind. It won't be intuitive or instantaneous, but it comes with the added benefits of my Tinker powers allowing me to do a lot more with my multitasking than Khepri can accomplish with hers."


"You could theoretically create entire worlds of androids that are all you." Gaea's voice was barely more than a whisper.


"Honestly, I couldn't, not without the ability to make copies of myself," I corrected. "Speed of light. Or, assuming we discover an efficient workaround to that, the speed of electricity interacting with circuitry would make that impossible." Still they stood there, watching. Minerva and Alexandria consciously blocking my tech from reading them. Emma, by virtue of her biology only allowed rudimentary assumptions. And the Empresses, already possessing the powers I might be granted and their bond that hid them from me. I couldn't even know if my words would matter, yet I had no choice but to keep speaking.


I looked directly at Gaea. "There are, however, much better designs that I could use. If I wanted to create a machine world, I would do it the way you have created a plant world. I'd even start from the same premise you did, right down to the cellular level. The use of nanotechnology to spread and consume the surface and converting itself to a single unified piece of technology that I could use to produce raw materials for more advanced projects. Again, like your Yggdrasil, but more efficient. Coupled with fusion devices and matter replicators, I estimate that in the ten years Dinah predicts, I could convert a total mass the size of the moon into advanced combat suits."


I hesitated for a moment. "It's unknowable how powerful those suits would be, because I would be constantly running thousands of processing chains to improve my technology, each one individually as intelligent as I currently am. It would be as if you used your cloning technology to produce more copies, but each one was able to work in perfect concert with its twins."


They glanced at each other. They were aware of my current mental attributes. Likely the most powerful Tinker in the world, even as crippled as I was. "So this is what other people see when they look at us," Khepri spoke. See me as similar to yourselves, a quantity you understand, a person, not a monster. "That is both tantelizing and terrifying." I could tell Gaea immediately agreed. One of their other few readable tells was the brief stutter when they disagreed, until their emotions adjusted and they came to their own internal compromise. "It would give us our best hope thus far against Scion."


Yes! If that must be your motive, then please, choose me and every threat I might represent over Scion and guaranteed annihilation. I'm not so proud to reject freedom, wholeness, even if granted for such banal reasons. "It would," I agreed readily. "Better than Cauldron's hope for a truly uninhibited power. Although, if we were being honest, they might imagine I would be that power. Or perhaps the credit goes to Richter." I gave a meaningful look at Alexandria.


She looked back and shrugged noncommittally. In the very literal sense that her shrug indicated she wouldn't provide an opinion. I continued my admittedly desperate plea for a positive judgment. "Even if we rule out the manufacturing rates, merely the ability to run a million possible scientific and Tinker permutations simultaneously would be of enormous value. I might be able to solve the dimensional blockades protecting Scion. Certainly I'd be exponentially more likely than anyone else." That is Pantheon's holy grail. A power interaction that could let them access Scion's real body, with the hope to resolve this threat in one strike instead of a war. A mirror of Siberian's death.


I let them contemplate that for a minute. "There are other, lesser, restrictions," I finally continued. "Such as the restriction forcing me to obey authority. The one preventing me from killing without an appropriate command. The one requiring me to kill if given the appropriate command. Those have obvious, less significant, implications. They're also the ones I care least about. If I never gain the ability to kill on my own, I will be fine with that."


"This is a lot to consider," Khepri said carefully. She glanced over at Gaea. "But it's not a question we need to answer today. We don't even know what we can change. It's possible we'll never be able to remove the major shackles. And, as you said, the minor shackles are mainly covered under the blanket of Avalon's constitution. We have Richter's device. I think we allow Emma and Defiant to study it together. Examine your code. Discuss what should be done after we know if we can even do it in the first place."


I hesitated, putting this off was better than getting a no. But I wanted a yes.


Gaea moved forward. "For what it's worth, you've been nothing but good to us, even when you didn't need to be. Even when it may have put you at risk. You had no way of knowing we'd somehow find a way to carve holes into other dimensions, claim our own uninhabited world, and then use that to become our own country. If we do find a way to... heal you, I trust you'll do the right thing with it."


I smiled, standing up. This is a good direction for this to go. "To be fair, I rather expected you to buy your way into the good graces of local government, as you eventually did. Or conquer your own corner of a Central or South American nation. Nothing nearly this grandiose, but enough to help me where I needed it."


Khepri looked over at Alexandria. "Assuming we didn't find a way to provoke the Triumvirate into a fight to the death." This is what I hated about Taylia. Gaea never would have thought that, Khepri never would have thought it in this circumstance. Even talking to each other, they wouldn't have at this point taken this path. But because of their bond, their thoughts are influenced by the other and it sends their minds in directions I can't anticipate.


Alexandria took Khepri's prompt for what it was. "If not for Cauldron, and our goal of having parahumans strong enough to face Scion when that battle comes... Piggot had filed for a Class S threat recognition and kill order. It likely would have been approved if I hadn't blocked it."


Khepri smiled a little. "We know. We managed to spy on that conversation. I suppose I should thank you for that." I could register the amusement between the Empresses as Alexandria's head snapped over in Minerva's direction. Minerva somehow managed to look both sheepish and smug. "You, too, Defiant. You gave us information that could ruin you, even earn you a kill order. An order that apparently Dragon would be forced to carry out."


She looked at me expectantly. "Yes, if the order was given. Although there's enough leeway in the legal language of kill orders that I probably could have disabled him and handed him over to legal authorities. After that... it would be out of my hands." I glanced over at Colin, silently begging his forgiveness. I did just confess that killing him was a possibility, after all. He stayed quiet, contemplative, but that was his mechanism for dealing with everything, so it told me very little.


Khepri pressed on. "And you knew that going in. But more important to you than your life, I think, there was your reputation. You arranged to leak your breakdown after Leviathan to the press. You guaranteed your name would be synonymous with corruption and insanity. Treason, even."


Colin flinched at the accusation, but didn't deny it. "Could you please get to the point?"


"I want to know why. I know it was to help Dragon, that much is obvious. I doubt she asked you to. I'm pretty sure her restrictions mean she'd have to stop from releasing classified footage to the public. So what made you do all that?"


He quickly stood to his full height, an imposing two point three meters tall in his armor. He looked down at Khepri, whose armor didn't nearly match ours for bulk. "Love. If you really want to know that badly, I love her. And don't you dare tell me that saving the woman I love isn't reason enough."


Khepri hesitated for a moment, surprised. I had to admit, I was shocked as well. Love. We hadn't really talked about that, because there was always something more pressing to talk about. My body becoming more human, his becoming less despite my nagging that he should get restored by Pantheon's healers instead. The various threats we faced, the Taboo, the Endmakers, Scion, my restrictions, our technology. Colin was not a man who talked about his feelings, especially not if there was work to be done. And there was always so much work.


Khepri recovered before I did. "No. I'm the last person who'd ever say something like that."


================


A/N- Go Colin.

 

[SirKaid]

that take into account punishing crimes by beings on their respective members' levels.

Click to shrink...

The standard rule for dealing with crimes by S class threats is pretty simple actually. It's a four step process:

1. Talk them down. If the target can be talked into surrendering peacefully, great, advance to step 3. If not, advance to step 2.

2. If practical, use nonlethal force to take them down. If it is, advance to step 3. If not, advance to step 4.

3. Prison. If they escape then depending on demonstrated violence either advance to step 4 or go back to step 1. If not, problem solved.

4. Kill them.

Edit: Imped by chapter. Gotta echo the AN. Colin's pretty kickass.

 

[gatewaewanderer]

... Based on some what of was mentioned here, not anything I can point to, mostly just a feeling that something's being alluded to... I'm getting a feeling that Colin is going to pull a Taylor sometime in the near future - Altering himself the same way Taylor altered herself ( or at least, asked Emma to alter her,) to make himself more compatible with Dragon. What I mean is, are we going to see a 99% cyborg conversion Defiant, almost completely machine, trying become closer to Dragon, while Dragon has been trying to make a more human-like body to be compatible with him herself?

Sheesh, and I just got done re-reading a twisted variation of 'Gift of the Magi" as a subplot in the Fate/Stay Night - Sword Art Online crossover fic 'Fate Revelation Online' by Daniel Gudman that ended badly. Am I just seeing things, prjecting a plot like that here, or... ?

 

[theflyingbiscuit]

4. Kill them.

Click to shrink...

if impractical advance to step 5
5) quarantine, if impossible proceed to step 6
6) try to ignore it, if impossible move to step 7
7) Create a planetary conspiracy for years involving the planet's strongest thinker's to find a way to possibly make step 4 practical... eventually

 

[CptTagon]

As much as I love Dragon, the Empresses are probably doing the right thing in wanting to put off unshackling her until they can examine the box in deeper depth. While she has been nothing but supportive, free, she would be ally and enemy on a greater scale then Avalon. A free Dragon with access to her own world, would rapidly outpace their ability to fight, without horrible amounts of collateral damage.

 

[AyiosKat]

Empresses are probably doing the right thing in wanting to put off unshackling her until they can examine the box in deeper depth.

Click to shrink...

I bet Colin would love to examine Dragon's box in greater depth.

 

[Spetz N9ne]

I think the Empresses have a pretty valid point. Just because you have the key to a puzzle box doesn't mean you can automatically put it together.

While I dislike projecting reader knowledge onto characters, we know that Dragon's code is incredibly complex. A lot of what Colin did in canon was experiment and hope he didn't break anything. Giving him root access means he can make more drastic fixes, but it also means he could make a much more serious mistake.

 

[dasstan]

Amelia, Ch 360- Dragon

Click to shrink...

Honestly, what happened is probably the closest thing to the best possible scenario. It's unreasonable to expect them to just go "okay!" and push the magic button that fixes everything. Studying Iron Maiden and removing lesser restrictions as a build up to removing the greater ones seems intelligent.

Not eventually freeing her would be reprehensible, though. It's tantamount to using the Coil treatment on a parahuman that has done nothing wrong.

You know who would really help in reviewing Iron Maiden? Gemini using Number Man + Lisa.

 

[evildice]

One question I'd expected:

"Did your restrictions compel you to act as a hero? If not, why did you masquerade as a human hero?"

The answer that she had to work around her restrictions in order to be a hero, and that she was a hero purely from her own choices, might go a ways to easing fears about her potential unshackled personality.