I haven’t vetted it yet, but fans of Lain tend to recommend Paranoia Agent and vice versa, and they sound similar enough to each other. But have you already seen Neon Genesis Evangelion? Because that’s another one that’s got a lot of vice-versa recommendations with Lain, and I’ve seen it twice now and I would say that it scratched a similar itch for me as Lain did, although Evangelion can at times also be more action-oriented than Lain, what with the mechs and all. (By the way, if you enjoyed Evangelion you should also consider Nadia: The Secret of Blue Water!)

I didn’t manage to get into Madoka Magica myself, but maybe that’s something for you, if you liked Lain for this teenage girl getting pulled into the web of this sort of dark underworld. I also think Lain has a lot of fan overlap with Revolutionary Girl Utena and FLCL, though I haven’t vetted more than the first two episodes of either of those. Utena and FLCL aren’t necessarily that similar to Lain in terms of tone or content, but they are late '90s to early '00s “auteur” type stuff, so if the “auteuriality” of Lain was a major draw for you — as it sounds like it was — then Utena and FLCL could also be up your alley.

…But I mean, that’s the problem with “like”, isn’t it? The whole question of “‘Like’ in what way?” Because I could even recommend Steins;Gate as a top-tier psychological mystery show involving a lot of tap-tapping on a computer keyboard, but the vibes just aren’t the same as Lain, 'cause Steins;Gate has a lot more levity, for one.

So yeah. The recommendations will always depend a lot on the specific point of comparison. But I still hope this is a good jumping off point.