What I know when attacking another player when not in war is: walls, gates and repair squad can be destroyed, for the rest, declaring war is needed.
Note that the following is from my personnal experience, and it might be wrong. If someone attacks your town, and he always comes from a specific side (example north gate), you can block him from coming inside placing watch towers instead of walls right next to the entrance. This is a temporary solution and if your opponent capture a town south of you, he will be able to enter.