To stop a membership, set an End Date on it. Once that date has passed, the overnight sweep moves it to Finished and no further charges are raised.
End date, not delete
Money still owed
Ending a membership does not clear the balance. Charges already raised stay on the ledger and, if the member is still Active with automatic payments on, they will still be collected. That is usually what you want; when it is not, switch automatic payments off before you set the end date.
Writing off what is left
If you are letting somebody go without collecting the last charges, add a Credit on the Transactions tab for the outstanding amount. That brings the balance to zero and leaves an honest record of the write-off, rather than deleting charges that really were raised.
The member record itself
The member stays. Set their status to Inactive once everything is settled — that keeps their history available without their appearing in your active lists.