Everyone's liberty, including speech, is constrained by a prohibition against doing harm to others. Harm can be psychological--and even sociological--as well as material. Identifying immaterial forms of harm can be more difficult, but there is no harm in being more deferential to other people's feelings.