Going with my girlfriend to a swingers club

We’re doing this for 5 months and I fell in love with the female part of the other couple. Last night we went out for a drink and nobody knows about this date. Do you think it’s a good idea to start a relationship with this woman? Or should we keep it casual and continue to see each other at the club? Do you think if we keep seeing each other this way, my relationship will be affected?