Yeah, but if you don't build them in every province, then that province is basically totally unprotected and anyone can simply waltz in, take over all your magic sites, and burn down your temple totally unopposed.
Cheapest fort is Palisades, 50 defence, 250 gold. Shit fort, but with that maybe your strategy could work. Standard forts are 800- 1000 gold a pop plus 5-6 turns of construction. No way you can spam them in every province. In My Nooobish Opinion, it's safer to have a solid Dominion and build unprotected temples only in regions far away from the frontline, were the game of province swapping WILL happen in war.
At 400 Gold, better to build them in a protected position and sparingly. Gold is a commodity, and balance is already quite different from Dom3, I can only imagine how different is from Dom2. If someone started spamming forts in all his provinces the only result I can foresee is a gimped army, weak research and slow expansion. That means, by the Red Queen hypothesis, death.
Of course, Noobish opinion. If someone wants to correct me, go wild.