Plastic itself, sure. But it happily, greedily adsorbs pollutants (that is, picks them up on its surface where they can be released into whatever consumes them) and is full of additives like plasticisers and so on which have a variety of toxic effects, particularly around increased oxidative stress and endocrine disruption. The physical presence of the plastic inside our bodies is absolutely a problem but they do also poison us.