Sub-Saharan Africa is one of the most severely affected regions regarding soil degradation, a global issue with the loss of nutrients caused by inappropriate management, leading to low agricultural productivity. Here we asked the question of how soil prokaryotic communities are affected by shifts in land use management and subsequent losses in soil organic carbon. We sampled soils from three sites in Zambia which have neighboring natural and managed sites. After the measurement of soil properties, soil DNA was sequenced, targeting the 16S rRNA gene. As expected, total carbon in soil was decreased in the managed sites, with significant reductions of bacterial biomass. However, the diversity indices in the managed soils were higher than in natural soils. Particularly, the relative abundance of nitrifiers was increased in the managed soils, most likely as a result of fertilization. However also other bacteria, e.g., those which formed tight interactions with the cultivated crops including the genera Balneimonas, and Bacillus, were increased in the managed soils. In contrast bacteria belonging to the family Chloroflexi, which were high in abundance in the natural soil were outcompeted by other prokaryotes in the managed soils most likely as a result of changes in the amount of soil organic carbon. Overall, our results suggest that we need to discuss the trends of prokaryotic diversity separately from those for prokaryotic abundance. Even when bacterial abundances were decreased in the managed soils, nitrifiers’ relative abundance and diversity increased in our experiment, suggesting the possible alteration of the nitrogen cycle in managed soils in sub-Saharan Africa.