Evaluations of the Higher Education Sector are receiving increased attention, due to the rising expenditures and the absence of efficiency enhancing market pressure. To what extent universities are able to eliminate inefficiency is a question that has only partially been answered. This paper argues that heterogeneity among universities as well as persistent inefficiency hinder the institutions to achieve full efficiency - at least in the short run. Two standard and one novel specification of the Stochastic Frontier Analysis are applied to a new, comprehensive set of panel data to show how the standard efficiency evaluation changes when both aspects are taken into account. It is the first time that the idea of persistent inefficiency is considered in the analysis of the German Higher Education Sector. The comparison reveals that the disregard of heterogeneity distorts the estimation results towards lower efficiency values. The newly introduced specification improves the accuracy of the heterogeneity assumption and exposes that inefficiency tends to be long term and persistent rather than short term and residual. This implies that increasing efficiency requires a comprehensive change of the university structure.