Maybe you didn't explain something or I missed it, but:
1) It sounds like you can't hurt someone at all by hitting an armored location until/unless you completely destroy the armor at that location? That sounds wrong/unrealistic - it's certainly possible to penetrate armor or batter someone and hurt them without coming close to destroying the armor. If you mean that damage over hardness damages both the armor and the person wearing it, which would make more sense to me. For more realism, the damage resistance of the armor and of the person wearing the armor might be different numbers and depend on the attack type, as some hits might hurt the person without really hurting the armor, and so on.
2) The system for aiming at and protecting hit locations also becomes important of course, as well as having different effects for hitting different targets on a person.
Also, what you're describing sounds quite similar to what GURPS does, except GURPS is thoroughly developed/playtested/expanded. The hit locations, layering armor types and quality levels (see GURPS Low Tech), and weapons and shields which can also break in various ways, are all pretty well done there. However damage to armor itself doesn't have a great comprehensive damage system that I know of (though I expect there are some attempts on the GURPS forums).
But still, I'd try playing some GURPS. Probably 4E with 4E Low Tech. They've already done a lot of thinking and work about armor and hit locations and damaging stuff, which at the very least you could use as a reference.
I have tried tracking armor damage in detail in GURPS using various systems, and usually found it to be a whole lot of work, though I have still done it and liked the benefits even if I didn't like the work. And I'm the guy who thinks large GURPS combats on hex maps are fun and relatively easy to manage. But I don't think a realistic effect of armor damage is going to result in a major effect on tactics within one fight, because for the most part, armor tends to be more durable than the wearers. It's much easier to get around armor or pin or crush or disarm someone than it is to destroy their armor while they're wearing it, in general. Or at least, cut straps on the armor, pull it off them, get it twisted or something rather than physically destroying it.