If you should have the vote and don't that is a state of not being free. But not everybody in society should vote.
Your question seems to be more about eligibility than anything else. Women in NJ had the right to vote very early on but they knew that that was a rare thing. Thus they felt not free despite having the vote; other women didn't because they were women. a black woman can feel doubly excluded for similar reasons. But most cases of the exclusion of women or blacks were in societies that had restrictions on poor whites and others so the situation is rather hard to evaluate. I would ask, What point is it for a woman or black to have that right but yet not vote or even care about the political situation??