If someone sees you put in your iphone passcode, can’t they still grab your phone and then use that to change your AppleID password, locking you out of your iCloud account?

I was under the impression that could still be an issue. Am I wrong?