When i think of AoE or similar games, people without work always looked unnatural just standing around, and you had to search for workers who were not assigned any job (or click a button to find them).
It would be pretty nice if you could make some animations for poeple who dont have any task, for example just sitting and tearing grass from the floor, lay there with their hands behind thei head gazing into the sky, etc. If there are kids, maybe they could chase each other.
And a button for poeple who dont have a task would be nice, wich highlights them, and/or brings you near them. A general highlighting of these, such as a red shade might be helpfull aswell.