The idea is "don't tolerate intolerance." Nazis are definitely intolerant, so it's ok to not tolerate them. Since Nazis promote death squads and gas chambers for people they don't like, punching them is very justified.
There are also some literal Nazis in the United States that are pro-Trump. They are also very punchable.
Does that mean that all trump supporters are punchable? No, it does not. But should it give you pause if you are on the same team as the Nazis? Yes, it should.
I was once doing work at a company that provided tech support and security for local businesses. There were a couple big instances of the companies being hacked with ransomware etc. On every occasion, we of course ask, "when was your last backup done?" And without fail, every one of them always responded, "backup?"