Adipose tissue inflammation, as defined by macrophage accumulation, is proposed to cause insulin resistance and systemic inflammation. Because the strength of this relationship for humans is unclear, we tested whether adipose tissue macrophage (ATM) burden is correlated with these health indicators. Using immunohistochemistry, we measured abdominal subcutaneous CD68+ (total ATM), CD14+ (proinflammatory/M1), and CD206+ (anti-inflammatory/M2) ATM in 97 volunteers (BMI 20–38 kg/m2, in addition to body composition, adipocyte size, homeostasis model assessment of insulin resistance, ADIPO-IR, adipose tissue insulin resistance measured by palmitate, plasma lipids, TNF, and IL-6 concentrations. There were several significant univariate correlations between metabolic parameters to IL-6 and ATM per 100 adipocytes, but not ATM per gram tissue; adipocyte size was a confounding variable. We used matching strategies and multivariate regression analyses to investigate the relationships between ATM and inflammatory/metabolic parameters independent of adipocyte size. Matching approaches revealed that the groups discordant for CD206 but concordant for adipocyte size had significantly different fasting insulin and IL-6 concentrations. However, groups discordant for adipocyte size but concordent for ATM differeded in that visceral fat, plasma triglyceride, glucose, and TNF concentrations were greater in those with large adipocytes. Multivariate regression analysis indicated that indexes of insulin resistance and fasting triglycerides were predicted by body composition; the predictive value of ATM per 100 adipocytes or per gram tissue was variable between males and females. We conclude that the relationship between ATM burden and metabolic/inflammatory variables is confounded by adipocyte size/body composition and that ATM do not predict insulin resistance, systemic inflammation, or dyslipidemia. ATM may primarily play a role in tissue remodeling rather than in metabolic pathology.