It depends on the time-scale.
In the short term, knowledge of the terrain and the ability to strike-and-fade, target your opponent's vulnerable baggage train, and similarly make being a smaller and less mighty force quite tenable in resisting a greater one.
In the long term, your enemy can just send in heavily defended woodcutters, and attrit away your environment bit by bit.
And in the medium term, a moderately-leveled wizard doing a bombing run of Fireballs on a dry autumn night can do a lot of damage to your forest home. Now, if your barbarian tribe also has a strong druidic tradition, they have a much better chance of countering this, and also of making the ambushes even scarier, with wild-shaped druids being really amazing scouts...but a single druid allied with the incoming army could also go in as a heavily-advanced scout after the first major losses were taken, scout out the barbarian's defenses, and if they note any hidden settlements in their own prolonged scouting, making sure to report back via Message or Dream spells as to their exactly location, so when operation Fireball From Above starts, these settlements get hit first and hardest.
Basically, magic is a tactical element like terrain knowledge or numbers, and in most D&D worlds, the ability to marshall an actual army of fighting-men is pretty congruent with raising a few specialist casters. So, you should take into consideration who's got what magics, and what kind of magic-users just make certain historic tactics or strategies doomed on their face.