Your critique of libertarianism runs very close to my critique of it. At bottom, if justice is liberty and justice is the highest ideal, how can any constraints on liberty — people doing what they want — be justified?

My studies have taught me that the ethic of justice must be mutual respect — people taking others into account. Mutual respect would maximize liberty as a practical matter, not by trying make a dogma of it.

A requirement of mutual respect follows from a belief in equality. That, however, makes it still ideological.

A requirement of mutual respect also follows from observation within material existence. There the “domain” of justice, the range of its referents, is limited to the process of effecting choices. That allows it to be a strictly rational ethic, limited to material existence, involving no beliefs, and therefore non-ideological.

If curious, the non-ideological form of that ethic is summarized in a “5 min read” here in Medium. You might also find “Libertarian, Yes; Libertarianism, No,” “Re-thinking Individualism,” or “Good Riddance, Ideology” interesting.