Prediction of mental states, such as stress and anxiety, can be important in situations where reduced job performance due to increased mental strain can lead to critical situations (e.g., front-line healthcare workers and first responders). While recent advances in biomedical wearable sensor technologies have allowed for collection of multiple physiological signals in everyday environments, numerous challenges emerge from such uncontrolled settings, including increased noise levels and artifacts, confounding effects from other psychological states (e.g., mental fatigue), as well as physical variables (e.g., physical activity). These factors can be particularly detrimental for heart rate variability (HRV) measures which, in controlled settings, have been shown to accurately track stress and anxiety states. In this paper, we propose two new ways of computing HRV proxies which we show are more robust to such artifacts and confounding factors. The proposed features measure spectral and complexity properties of different aspects of the autonomic nervous system, as well as their interaction. Across two separate “in-the-wild” datasets, the proposed features showed to not only outperform benchmark HRV metrics, but to also provide complementary information, thus leading to significantly greater accuracy levels when fused together. Feature ranking analysis further showed the proposed features appearing in 45–64% of the top features, thus further emphasizing their importance. In particular, features derived from the high frequency band showed to be most important in the presence of fatigue and physical activity confounding factors, thus corroborating their importance for mental state assessment in highly ecological settings.