Yeah, I think if anything it’s a similar issue that the literature being adapted is itself very frequently uninteresting and the adaptations, though certainly they usually add things, do not add nearly as much as they could.

Part of the reason for the huge emphasis on manga adaptation specifically is that you practically have the script and storyboard right there already, and it is rare for an anime to heavily emphasize elements requiring a lot of new storyboarding (exceptions include Bleach TYBW and Mob Psycho, and even still they are mostly the same).

But also of course another reason that applies to both cases is that capital is risk-averse, so producers overwhelmingly gravitate toward work that already has fans.