Can you succeed in swinging being overweight?

My husband and I are overweight. I can't tell you exactly how much we weigh, but I can assure you that it is a lot. We tried to lose weight, but we can't. Not because we don't want to or because we can't keep a diet, but because we are both having medical issues.
The weight is not a problem between us, the sex is awesome, but we realize we won't appeal to everyone.

We went once to a swinging club and we have seen people of a wide range of sizes. We just had a drink and left, because we were not ready to play.

Some friends of us who are into the LS for a long time told us that the weight might be an issue and they advised us to “dress to impress” if we want to succeed.

Is the weight such a big deal?

I know that insecure people are less sexier, but I can't seem to get this out of my mind.
I need some advice from people that are in our situation. Was it hard to find a couple to play with?
Should we stop thinking about getting into the lifestyle because of this?