But our memories exist – I can say definitively “I know my childhood phone number.” It might be meaningless, but the information is stored in my head. I know it.

AI models don’t know your childhood phone number, even if you tell them explicitly, even if they trained on it. Your childhood phone number becomes part of a model of word weights that makes it slightly more likely, when someone asks it for a phone number, that some digits of your childhood phone number might appear (or perhaps the entire thing!).

But the original information is lost.

You can’t ask it to “forget” the phone number because it doesn’t know it and never knew it. Even if it supplies literally your exact phone number, it isn’t because it knew your phone number or because that information is correct. It’s because that sequence of numbers is, based on its model, very likely to occur in that order.