Home vs Public Speaking: Who Should Get the Greater Care?

In a recent podcast centered around speech with a "punch a Nazi" condoning libertarian, I disagreed on several points of speech etiquette. The conversation was moving along too quickly and, consequently, I didn't have time to object to a particular thought of his, which showed, I think, the generational gap between us (me, being nearly 17 years his senior and part of Gen-X to his Millennial):

In the greater context of condoning the reality that if you're a Nazi, saying Nazi things in public, you should expect to get punched: he stated that he thought he'd be less likely to be careful about what he said in his own home, not expecting to get punched for being offensive in that context, but more careful about what he said in a large group not knowing who he might offend out in public.

What I didn't have time to say, but would have liked to (and, I'm saying here) is that I'd handle it just exactly the opposite way. I would be more careful with what I said to guests in my home - not because it was my home and I'd expect to be safe from punches in that setting, but because, if someone was in my home, I presumably would:

1. Know them enough to know what would be too sensitive a topic to bring up with them, or want to get to know them better, and

2. That I'd care about not offending them, because I'd care about growing and/or maintaining our relationship.

However, in the public arena, I neither intimately know everyone's sensibilities nor do I care if I offended them, no more than they should care about offending me, which is why the basic decency of being free from the fear of physical harm, when speaking in a public sphere, must be held sacred.

How could I know every individual stranger's point of offense when speaking publicly, yet public speaking is necessary to a free market place of ideas so, it has to be that you are responsible to constructively manage your own points of offense when out in public.

The difference between the two spheres is relational. Where there is no relationship, there should be no emotional need to be safe guarded from offense. That person speaking is just a random person saying things with which you disagree. Where as, if we invite another into our warm and welcoming sanctuary - away from the cold, harsh relationless experience of the world - we ought to care deeply about making one another comfortable.

The difference between he and I is the understanding (or lack thereof) of a home's function and the place we allow various groups of people to take up in our emotional being. I do not need to care for all humans equally and thus only some are allowed into my safe haven of a home, or "safe space", as may be the term which is, culturally, more poignant. When I go out from that space I am responsible for guarding my emotional wellbeing among people who have no compulsion to take that care for me. 

I will link the podcast for full context when it becomes available.