This study assessed the concentration of heavy metal, such as lead (Pb), cadmium (Cd), Chromium (Cr), iron (Fe), Nickel (Ni), and Silver (Ag), in Vernonia amygdalina Delile and agricultural soils of three university farms located in Port Harcourt, Nigeria. The soils and plants were taken randomly to form composite samples and analyzed for heavy metals by the use of atomic absorption spectroscopy (AAS) and X-ray fluorescence (XRF). The study stations were agricultural soils and Vernonia amygdalina Delile from the Ignatius Ajuru University of Education (I), River State University (R) and University of Port Harcourt (U). The soil samples recorded mean concentration ranges for Fe as 19.71 ± 1.77 (I)–27.24 ± 3.56 mg/kg (R) in soils and 12.95 ± 1.68 (R)–18.18 ± 2.02 mg/kg (U) for the bitter leaf samples. The mean range for Pb concentration in the soil and bitter leaf were 4.35 ± 0.87–6.80 ± 0.86 mg/kg and 0.24 ± 0.64–2.19 ± 0.74 mg/kg, while Cd concentration in the soil and bitter leaf were 0.46 ± 0.28–1.42 ± 0.40 mg/kg and 0.17 ± 0.22–0.42 ± 0.08 mg/kg, respectively. The respective mean ranges for Cr concentration in the soil and bitter leaf were 5.91 ± 1.14–8.77 ± 0.88 mg/kg and 4.04 ± 0.64–5.92 ± 0.69 mg/kg, while Ni in soil and bitter leaf were 0.54 ± 3.38–10.26 ± 3.50 mg/kg and 0.042 ± 1.42–3.30 ± 0.88 mg/kg, while Ag was negligible. Heavy metal levels in soils and Vernonia amygdalina followed the order Fe > Cr > Pb > Ni > Cd and Fe > Cr > Ni > Pb > Cd, respectively, and were lower than WHO/FAO and EPA, except Cd, which was higher in soil and in Vernonia amygdalina. The ecological risk factor (ErF) was comparatively lower in soils than in the plant, while pollution load index (PLI) showed high heavy metal retention capacities in Vernonia amygdalina due to more anthropogenic influences. The metal transfer factor (TF) was highest in Fe, followed by Cr > Cd > Ni > Pb, while Pb had the highest chances of cancer risks from the incremental lifetime cancer risk (ILCR), especially in both soil and plant (mean ILCR, 2.07 × 10−2 and 2.45 × 10−3), while Cd had the least (mean ILCR, 9.64 × 10−5 and 3.36 × 10−5). Anthropogenic activities must be regulated and monitored by government relevant agencies to reduce heavy metal inputs into soils and avoid excessive accruals in food chain.