The rural poor are often marginalized and restricted from access to markets, public services and information, mainly due to poor connections to transport and communication infrastructure. Despite these unfavorable conditions, agricultural technology investments are believed to unleash unused human and natural capital potentials and alleviate poverty by productivity growth in agriculture. Based on the concept of marginality we develop a theoretical model which shows that these expectations for productivity growth are conditional on human and natural capital stocks and transaction costs. Our model categorizes the rural farm households below the poverty line into four segments according to labor and land endowments. Policy recommendations for segment and location specific investments are provided. Theoretical findings indicate that adjusting rural infrastructure and institutions to reduce transaction costs is a more preferable investment strategy than adjusting agricultural technologies to marginalized production conditions.