As long as you're not cleaning with lye, soap is generally fine. But if you're going to be a bit lax on cleaning, the only real downside in my opinion is potentially introducing flavors you didn't intend.
I think for the most part, you do you. If it looks visually fine, it's probably good enough.
My jeans are Calvin Klein. They are superior because they are mine.