Put a dollip of vegetable oil or wd-40 on a rag and scrub gently- the residue will come off. A little bit of soapy water on another rag will take off the oil after that. Dry thoroughly.

This works great for most adhesive residues- stickers, duct tape, etc on most surfaces.