guy recently linked this essay, its old, but i don’t think its significantly wrong (despite gpt evangelists) also read weizenbaum, libs, for the other side of the coin
guy recently linked this essay, its old, but i don’t think its significantly wrong (despite gpt evangelists) also read weizenbaum, libs, for the other side of the coin
So, ironically I think the author is falling in to the trap they’re complaining about. They’re talking about an “upload” as somehow copying a file from one computer to another.
Instead, consider transferring your brain to a digital system a little at a time. Old cells die, new cells are created. Do you ever lose subjective continuity during that process? Let one meat cell die and a digital cell grow. Do you stop being yourself once all your brain cells are digital? Was there ever a loss of phenomenalalaogy?
You’ve completely misunderstood their criticism of mind uploading.
The author asserts that you are not really your brain. If you copied your brain into a computer, that hapless brain would immediately dissociate and lose all sense of self because it has become unanchored from your body and your sociocultural and historical-materialist context.
You are not just a record of memories. You are also your home, your friends and family, what you ate for breakfast, how much sleep you got, how much exercise you’re getting on a regular basis, your general pain and comfort levels, all sorts of things that exist outside of your brain. Your brain is not you. Your brain is part of you, probably the most important part, but a computer upload of your brain would not be you.
Embodied cognition. I don’t see this as implying that what we’re doing isn’t computation (or information processing) in some sense. It’s just that the way we’re doing it is deeply, deeply different from how even neural networks instantiated on digital computers do it (among other things, our information processing is smeared out across the environment). That doesn’t make it not computation in the same way that not having a cover and a mass in grams makes a PDF copy of Moby Dick not a book. There are functional, abstract similarities between PDFs and physical books that make them the same “kinds of things” in certain senses, but very different kinds of things in other senses.
Whether they’re going to count as relevantly similar depends on which bundles of features you think are important or worth tracking, which in turn depends on what kinds of predictions you want to make or what you want to do. The fight about whether brains are “really” computers or not obscures the deeply value-laden and perspectival nature of a judgement like that. The danger doesn’t lie in adopting the metaphor, but rather in failing to recognize it as a metaphor–or, to put it another way, in uncritically accepting the tech-bro framing of only those features that our brains have in common with digital computers as being things worth tracking, with the rest being “incidental.”
I think I agree.
One metaphor I quite like is the brain as a ball of clay. Whenever you do anything the clay is gaining deformities and imprints and picking up impurities from the environment. Embodied cognition, right? Obviously the brain isn’t actually a ball of clay but I think the metaphor is useful, and I like it more than I like being compared to a computer. After all, when a calculator computes the answer to a math problem the physical structure of the calculator doesn’t change. The brain, though, actually changes! The computation metaphor misses this.
This is really useful for understanding memory, because every time you remember something you pick up that ball of clay and it changes.
What counts as “physical structure?” I can make an adding machine out of wood and steel balls that computes the answer to math problems by shuffling levers and balls around. A digital computer calculates the answer by changing voltages in a complicated set of circuits (and maybe flipping some little magnetic bits of stuff if it has a hard drive). Brains do it by (among other things) changing connections between neurons and the allocation of chemicals. Those are all physical changes. Are they relevantly similar physical changes? Again, that depends deeply on what you think is important enough to be worth tracking and what can be abstracted away, which is a value judgement. One of the Big Lies of tech bro narrative is that science is somehow value free. It isn’t. The choice of model, the choice of what to model, and the choice of what predictive projects we think are worth pursuing are all deeply evaluative choices.
In dwarf fortress you can make a computer out of dwarfs, gates, and levers, and it won’t change unless the dwarfs go insane from sobriety and start smashing stuff.
Great example! Failure modes are really important. Brains and dwarf fortresses might both be computers, but their different physical instations give them different ways to break down. Sometimes that’s not important, but sometimes it’s very important indeed. Those are the sorts of things that get obscures by these dogmatic all-or-nothing arguments.
Isn’t that what this article is about? That “brain as computer” is a value judgement, just like “brain as hydrolic system” and “brain as telegraph” were? These metaphors are all useful, I think the article was just critiquing the inability for people to think of brains outside of the orthodox computational framework.
I’m just cautioning against taking things too far in the other direction: I genuinely don’t think it’s right to say “your brain isn’t a computer,” and I definitely think it’s wrong to say that it doesn’t process information. It’s easy to slide from a critique of the computational theory of mind (either as it’s presented academically by people like Pinker or popularly by Silicon Valley) into the opposite–but equally wrong–kind of position that brains are doing something wholly different. They’re different in some respects, but there are also very significant similarities. We shouldn’t lose sight of either, and it’s important to be very careful when talking about this stuff.
Just as an example:
It strikes me as totally wrong to say that this process is free of computation. The computation that’s going on here has interesting differences from what goes on in a ball-catching robot powered by a digital computer, but it is computation.
Your analogy reminds me a bit of the Freud essay on the mystical writing pad
That should be a red flag to treat it with caution. Freud was a crank and his only contribution to psychology was being so wrong it inspired generations of scientists to debunk him.
What is the ratio of meat parts to machine parts at which point “that chair you once sat on” or “the dust bunnies you haven’t swept up yet even though you keep meaning to” are no longer materially a part of you and you subsequently lose your self as a result?
If someone loses their leg and gets a prosthetic does this alienate them from the birds that woke them up last week?
This trope in cyberpunk pisses me off to no end. Writers just out there saying “using a wheelchair makes you less human and the more wheelchair you use the less human you become.”
Like people are out there, living, surviving, retaining their sanity, in comas where they have no access to sensory input. Those people wake up and they’re still human after living in the dark for years. People who have no sensation or control below the neck go right on living without turning in to psycho-murderers because they’re so alienated from humanity because they can’t feel their limbs. How is it that people somehow lose their humanity and turn in to monsters just because they’ve got some metal limbs? You can cut out half of someone’s brain and there will still be a person on the other side. They might be pretty different, but there’s still a person there. people survive all kinds of bizarre brain traumas.
Corporate bloatware/adware in cyber-limbs. That’s the explanation in my cyberpunk setting.
Yeah. At least there’s been a movement in the genre towards “ok, it’s not cybernetic implants in general, it’s chronic pain from malfunctioning or poorly calibrated implants, it’s the trauma of a violent and alienated society intersecting with people who are both suffering and who have a massively increased material capacity to commit violence, etc” there. Like Mike Pondsmith himself has still got a bit of a galaxy brain take on it, but even he’s moved around to something like “cyberpsychosis is a confluence of trauma and machines that are actively causing pain, nervous system damage, etc and which need a cocktail of drugs to manage which also have their own health and psychiatric consequences.”
I don’t see that as an improvement or a recognition of what is wrong with “cyberpsychosis” and related concepts. People live with severe trauma, severe chronic pain, and severe psychiatric problems and manage to keep it together. Pondsmith is making up excuses to keep “wheelchairs make you evil” in his game instead of recognizing the notion for what it is and discarding it.
I think it’s a good example of ingrained, reflexive ableism. It’s a holdover from archaic 20th century beliefs about disabled people being less human, less intelligent, less capable. Cybernetics are not a good metaphor for capitalist alienation or any other kind of alienation. They are, no matter how you cut it, aids and accomodations for disability. You just cannot say that cyberware makes you evil without also saying that disabled people using aids in your setting are alienating themselves from humanity and becoming monsters. If you wanted to argue that getting wired reflexes, enhanced musculature, getting your brain altered so you can shut off empathy or fear, things that you do voluntarily to make yourself a better tool for capitalism, gradually resulted in alienation, go for it.
Ghost in the Shell does a good job with that. Kusanagi isn’t alienated from humanity because she’s a cyborg, but her alienation from humanity grows from questioning what it means for her to be a cyborg, a brain in a jar. She’s got super-human capabilities - she’s massively stronger and more resilient, she’s a wizard hacker augmented with cyberware that let’s her directly interface with the net in a manner most people simply don’t have the skills for. Her digestive and endocrine systems are under her conscious fine control. These things don’t make her an alien or a monster, they create questions in her mind about her identity, her personhood, and how she can even relate to normal humans as her perspective and understanding of the world moves further and further away from them.
And this isn’t a bad thing. It doesn’t lead her to self-destruction or a berserk rage. Instead it leads her to growth, change, and evolution. She ambiguously dies, but in dying brings forth new life. Her new form is not an enemy of humanity or a threat, but instead a new kind of being that is a child or inheritor of humanity, humanity growing past it’s limitations to seek new horizons of potential.
The key difference is Kusanagi has agency. The cybernetics don’t force her towards alienation. They don’t damage her mind and turn her in to a monster with no agency. Kusanagi’s alienation grows from her own lived experience, her own thoughts and learning. They grow from her interactions with the people in her life and her day to day experiences. Her cybernetics are an important part of that experience, but she is in control of her cyberbody. It is not controlling her and turning her in to a hapless victim.
Basically; the Cyberpunk paradigm says that using a wheelchair makes you violent and evil. The GitS paradigm says using a wheelchair makes you consider the world from a different perspective. In the former disability, both physical and mental (false dichotomy I know) is villainized and demonized. In the latter disability is a state that creates separation from “normal” people in a way that reflects the experiences of real disabled people, but is otherwise neutral.
At this point as I understand it his take is “alienation/isolation, trauma from a violent society, and denial of access to necessary medical care can eventually break someone, and someone who can bench press a car and has a bunch of reflex enhancers jacked directly into their spine is more likely to lash out in a dangerous way when their back’s to the wall, they think they’re going to die, and they panic,” with a whole lot of emphasizing social support networks as being important for surviving and enduring trauma like that.
It’s still not as good a take as “cyberpsychosis isn’t real, it’s just a bullshit diagnosis applied to people pushed past the brink by their material circumstances, acting in the way that a society that revolves around violence has ultimately taught them to act, who then just double down on it because they know they’re going to be summarily executed by the police who have no interest in deescalation or trying to take them alive, compounded with the fact that they can bench press a car and react to bullets fast enough to simply get out of the way, at least for a while” would be, but it’s earnest progress from someone who’s weirdly endearing despite being an absolute galaxy brained lib.
Weirdly, Cyberpunk 2077 seems to have had a better take on it than Pondsmith himself, with “cyberpsychos” mostly being just people with an increased capacity for violence dealing with intolerable material conditions until they fight back a little too hard against a real or perceived threat, with one who’s not even on a rampage and instead is just a heavily augmented vigilante hunting down members of a criminal syndicate that had murdered someone close to him. The police chatter also has the player branded a cyberpsycho when you get stars, reflecting the idea that it’s more a blanket term applied to anyone with augments who’s doing a violent crime than a real thing.
“It’s a bullshit diagnosis”, with “cyberpsychosis” being “excited delerium” for cyborgs, works for me.
This is one of the key things that has kept me away from Cyberpunk the game setting and one of my main problems with Shadowrun (not my only problem by any means).
I have no idea, but neither do you, which is kind of the point. How much of you is your brain, your body, your context?
Even if we can’t put a number on it, I think it’s trivial to assert that you are not just your brain. So, if you copied only your brain into some kind of computer, there would be parts of you that are missing because you left them behind with your meat.
Sure I do: that ratio does not exist, and no you don’t get alienated from your material context if you have a prosthetic limb. We’re made up of parts that perform functions, and we can remain ourselves despite the loss of large chunks of those - so long as someone remains alive and the brain keeps working they’re still themself, they’re still in there.
If someone could keep the functions of the brain ongoing and operating on machine bits they’d still be in there. It may be a transformative and lossy process, it may be unpleasant and imperfect in execution, but the same criticism applies to existing in general: at any point you may be forced out of your normal material context by circumstance and this is traumatic, you may lose the healthy function of large swathes of your body and this is traumatic, you may suffer brain damage and this is traumatic, you’re constantly losing and overwriting memories and this can be normal or it can be traumatic, etc, but through it all you are you and you’re still in there because ontologically speaking you’re the ongoing continuation of yourself, distinct from all the component parts that you’re constantly losing and replacing.
Does body dysmorphic disorder not exist? Or phantom limb? A full body prosthetic would undoubtedly be a difficult adjustment!
And would an upload be a person, legally speaking? Would your family consider the upload to be a person? That’s pretty alienating.
And people survive all of that stuff, and are still people. I really don’t understand what you’re getting at here.
I never implied that uploaded people wouldn’t be people. All I’m saying is that they’d be different people. It’s not like putting on new clothes.
I’m getting the impression that you would think they were a different person, and I would not, and that disagreement lies not in any measurable process but rather our personal beliefs.
I didn’t say “you are perfectly happy and have no material problems whatsoever dealing with a traumatic injury and imperfect replacement,” but rather that this doesn’t represent some sort of fundamental loss of self or unmooring from material contexts. People can lose so much, can be whittled away to almost nothing all the way up to the point where continued existence becomes materially impossible due to a loss of vital functions, but through that they still exist, they remain the same ongoing being even if they are transformed by the trauma and become unrecognizable to others in the process.
If you suffer a traumatic brain injury and lose a large chunk of your brain, that’s going to seriously affect you and how people perceive you, but you’re still legally the same person. If instead that lost chunk was instead replaced with a synthetic copy there may still be problems but less so than just losing it outright. So if that continues until you’re entirely running on the new synthetic replacement substrate, then you have continued to exist through the entire process just as you continue to exist through the natural death and replacement of neurons - for all we know copying and replacing may not even be necessary compared to just adding synthetic bits and letting them be integrated and subsumed into the brain by the same processes by which it grows and maintains itself.
A simple copy taken like a photograph and then spun up elsewhere would be something entirely distinct, no more oneself than a picture or a memoir.
Eh. I’d argue that in as much as “you” means anything, forks would both be equally the person, there’s no “original” who is more the person. It’s a point of divergence, both have equal claim to all experiences and history up to the point of divergence. Privileging the “original” over the “copy” is cultural prejudice, subjectively they’re the same person to the moment of divergence.
I don’t think that’s the right way to untangle that dilemma ethically, because it can lead people to jump to galaxy brained “solutions” like “but what if you can make sure only one of you exists at once?” that don’t make any sense or answer anything but are still active cognitohazards for some people.
You, as in the one that is in there right now, that instance would continue along its own discrete path for as long as it exists: if another instance were made and separated off that would be a person, that would be a non-contiguous you, but it would not be the same you that is there right now, a distinction that becomes important when dealing with cognitohazards like trying to terminate that instance as the new one is spun up so that “you” get to be the one in a machine instead and there’s no perceptual break between them.
I’d argue that the ethical way to deal with forking copies like that would be to find ways to keep them linked up and at least partially synced, effectively making them component parts of a larger contiguous whole instead of just duplicating someone in a way that inevitably means at least one of the copies gets fucked over by whatever circumstances inspired the copying. So instead of the you that’s here now and the you spun up in a decade on a computer, there’d be the you that’s here now and then also a new secondary brain that’s on that computer, both of which communicate, share some sensory data, and operate almost as if you’d just added more hemispheres to your brain. And at some point after that maybe you could start considering individual copies ablative the same way bits of brain are, things you don’t want to lose but which you can survive losing and can potentially repair and replace given time because of how redundant and distributed brain functions are.
What I’m trying to say is your full body prosthetic would need to look like you, feel like you, sound like you, and have a legal life like you. Imagine if your name was Unit 69420, you looked and sounded like a Star Wars droid, and were legally considered property instead of a person. I think you would definitely experience a fundamental loss of self and become unmoored from material contexts.
“If shitty things happen to you, then you will not like that and it will suck,” still doesn’t break the continuity of self. Fundamentally that same exact thing can happen to the current flesh and blood you and it would be horrible and destructive: you can be disfigured through an accident or through someone’s cruelty, you can be locked in a cage and dehumanized on the whim of the state-sanctioned professional violence men and given a farce of a trial by the dysfunctional shitshow that is the legal system, etc, but no one is going to argue that shitty things happening to you ontologically unpersons you in some sort of mystical fashion.
You can be reduced, you can be transformed, but you continue existing for as long as vital functions do. Talk about someone becoming someone else, or dying in truth long before they died in body, those are just poetic attempts at articulating sorrow and loss.
This is just god of the gaps. “we don’t know so it’s not possible”. Saying “just copy the brain” is a reductive understanding of what’s being discussed. If we can model the brain then modelling the endocrine system is probably pretty trivial.
I didn’t read it as being impossible? I think you could upload a human mind into a computer, but it can’t just be their brain. Your mind, your phenomenal self, is more than just your brain because your brain isn’t just a hard drive. That’s what I took away from the article, anyway.
You are some mix of your brain, your body, and your context. Whatever upload magic exists would need all of that to work.
Aight I think we might be stuck in a semantics disagreement here. I’m using brain to mean the actual brain organ plus whatever other stuff is needed to support brain function - the endocrine system, nervous system, whatever. The physical systems of cognition in the body. i do not mean literally only the brain organ with no other systems.
I think I can relate this to being trans.
I was never at home in my body before and now I am. That’s changed me a lot! My personality has shifted, my mannerisms, my habits, my attitude, my lifestyle, everything is so different! Changing my body changed my mind. A full body prosthetic would be the same.
We call them dead names for a reason.
This really sounds like mind-body dualism. Setting aside the body, which you could just stick the new brain back in your body and hook the nervous system back in, why would you be cut off from your sociocultural and historical material context? Go home. pet your cat with your robot arms. Hang out with your family, eat breakfast. Wherein lies the problem? All of those things exist inside your brain. Your brain is taking the very shoddy, very poor information being supplied by your sensory organs and assembling it in to approximations of what’s happening around you. We don’t interact with the world directly. Our interaction is moderated by our sensory organs and lots of non-conscious components of the mind, and they’re often wrong. If you hook up some cameras and a nervous system and some meat to this theoretical digital mind it has all the things a meat mind has. The same sensory information.
Like, really, where is the problem? A simulated brain is a brain is a brain. Put the same information in you’re going to get the same results out. Doesn’t matter if the signals are coming from a meat suit or a mathematical model of a meat suit, as long as they’re the right signals.
Okay, imagine your family does not recognize your mind upload as a person. That’s not hard to imagine, I bet most people today would struggle with that kind of barrier. Maybe they treat it like a memorial of a dead person and do not treat it like a member of the family, maybe they shun it because it’s creepy and don’t want anything to do with it, maybe they try to destroy it in the memory of who you were.
That’s your sociocultural and historical material context. The mind upload would become a new person entirely because of the insurmountable differences between being made of meat and being made of data. That upload could never be the person you are now, because the upload becomes someone different as soon as they stop being made of meat and stop eating hot chip and stop paying taxes etc.
That’s the day to day lived experience of like half the queer people in the world. I rejected part of my family and kicked them out of my life. i don’t have to imagine it, I’ve lived it.
You seem to have an assumption that some kind of digital transfer of consciousness would not physically interact with the world. You could just plug them in to a body. If you can run an entire brain on a computer hooking it in to a meat suit sounds fairly trivial. You can go right on eating hot chip and paying taxes. I think we’re operating from different perspectives here and I’m not sure what yours is. A digital mind could have a digital body and eat digital hot chip and they’d definitely be forced to pay taxes.
Okay, and are you the same person you were before that? I’ve experienced that too, and I don’t think I am the person I was before. I think I became someone else in order to survive. If I could somehow go back in time and meet myself we would hardly recognize each other!
No, they’d be property and someone else would have to pay taxes for owning them.
When I talk about the sociocultural context I’m talking about the fact that society would not treat the mind upload as a person. We have not developed far enough along our historical material context to recognize uploads as people. The trauma of that experience would necessitate becoming a different person entirely, and thus, they’re no longer an upload of you. That’s the problem with talking about mind uploads.
You are not just your brain (and its supporting structures). You are your context.
deleted by creator
Yeah, pretty much. I think people are stuck in this idea that we’d somehow take a picture of the mind all at once and there’d be a loss of continuity and then you’re not the same person, but like, we’ve already got the intelligence making machine, why not just swap the parts out with maths one at a time until it’s all maths? Why would that be a problem? I sincerely think that a lot of people who object to this either aren’t really thinking of the concept beyond what’s presented in pop sci-fi or are still stuck on the idea of an incorporeal mind separate from material reality.
deleted by creator
Yeah, it’s the illusion of self and continuity of consciousness thing. People assume that any upload would be necessarily destructive and that there’d be a loss of continuity. No one ever seems to think “well, wait, why couldn’t I be awake for the whole operation in a way that preserves subjective continuity?”
In the ship of theseus, brain being replaced cell by cell concept you don’t get your head chopped off and imaged, you’re very gradually shifting what hardware your mind is running on. The lights never go out, there’s never a drastic, shocking moment of change, you never wake up in a jar.
deleted by creator
Sure, maybe it’s not, but afaik there’s nothing that rules out the possibility. We can’t do it now, but I’m not aware of any physical laws or limits that make it impossible for us to do it at some point when we’ve got a better understanding of what’s happening and a lot of silicon to run the maths on.
deleted by creator
I genuinely don’t know how to explain what evolution is as a process most of the time “imagine a drop of water seeking the sea, but the drop of water really wants to fuck, and sometimes it gets hit by an asteroid?”
“Complex self-replicating systems reversing local entropy while undegoing variation caused by entropy until they lose equilibrium and can no longer self replicate” ?
It’s an incredibly simple concept. Water seeks the sea. And it’s also an incredibly complicated, obtuse concept. I think a huge part of the difficulty is cultural - we anthropomorphize and ascribe agency to everything, and evolution is the absolute and total absence of agency, the pure action of entropy