As dumble said, tell them that out of pocket lunch expenses are not in your budget at the time and you prefer to eat what you brought from home.

I’ve never actually attended a farewell lunch, especially for someone whom I barely know.

Hopefully this won’t affect your standing with your colleagues, but they should be accepting of your situation.