I don’t know what a .webp file is but I don’t like it. They’re like a filthy prank version of the image/gif you’re looking for. They make you jump through all these hoops to find the original versions of the files that you can actually do anything with.
5mb to 100kb is not a typical result, so I would imagine that you are comparing apples to oranges (e.g. a very high quality jpeg vs a low quality webp)