The number of residents should not change unless you destroy farms. There is no need to keep them happy like Sim City. Just to guess what happened - I will guess that you are looking at your city screen - and there is a population number in the lower right hand corner, yes?
If I am correct - you are looking at the AVAILABLE population - not the total population. Population is used to build buildings and to support troops. For instance - it takes one population to support a swordman. An archer takes two. Upgrading a wall takes 1 or two. Building a library takes a couple.
