The maximum population you can have in one town is 7650. This includes 20 farms, the castle and the well. There's a way to get a bit higher, it's the rare buildings you get in temporary adventure huts. If you want to hire mercenaries, you need to completely dedicate one of your towns to an obelisk of glory.
That town cannot have any troops assigned to it and if you want to keep a maxed library, maxed tavern and maybe a maxed temple in that town, you have to make a few choices. In that town, the workshop, barracks arn't that needed because you won't have enough population to train troops anyway, same would go with the stables.
It's you to decide which buildings you keep or give up if you want to maximize your obelisk of glory.
