Despite measures on the European level to increase the compatibility between the HE sectors of the member states, the recent literature exposes variations in their efficiencies. To gain insights into these differences we split the efficiency term according to the two management levels each university is confronted with. Utilizing a recent advancement in the method to measure efficiency, we separate short-term (transient) and long-term (persistent) efficiency, while controlling for unobserved institution specific heterogeneity. While the first term reflects the efficiency of the individual universities working within the country, the second term echoes the influence of the country specific overall HE structure. The cross-country comparison displays if the overall efficiency difference between countries is related to individual performance of their universities or their HE structure. This allows more purposeful policy recommendation and expands the literature regarding the efficiency of universities in a fundamental way. Choosing Italy and Germany as two important illustrative examples we can take advantage of a novel dataset including characteristics of institutions in both countries for an exceptional long period of time from 2001 to 2011. We show that the Italian universities exhibit a higher overall efficiency value than their German counterparts. With the individual universities working at the upper bound of efficiency in both countries, the overall inefficiency as well as the gap between the countries is caused by persistent, structural inefficiency. To expedite a true European Area of Higher Education future measures should hence aim at the country specific structure, not solely at affecting the activities of single universities.