Kirman's ant model has been used to characterize the expectation formation of financial investors who are prone to herding. The model's original version suffers from the problem of N-dependence: its ability to replicate the statistical features of financial returns vanishes once the system size N is increased. In a generalized version of the ant model, the network structure connecting agents turns out to determine whether or not the model is N-dependent. We investigate a class of hierarchical networks in the generalized model that presumably reflect the institutional heterogeneity of financial markets. These network structures do overcome the problem of N-dependence, but at the same time they also increase system-wide volatility. Thus network structure becomes an auxiliary source of volatility in addition to the behavioral heterogeneity of interacting agents.