Ion–atom interactions are a comparatively recent field of research that has drawn considerable attention due to its applications in areas including quantum chemistry and quantum simulations. In first experiments, atomic ions and neutral atoms have been successfully overlapped by devising hybrid apparatuses combining established trapping methods, Paul traps for ions and optical or magneto-optical traps for neutral atoms, respectively. Since then, the field has seen considerable progress, but the inherent presence of radiofrequency (rf) fields in such hybrid traps was found to have a limiting impact on the achievable collision energies. Recently, it was shown that suitable combinations of optical dipole traps (ODTs) can be used for trapping both atoms and atomic ions alike, allowing to carry out experiments in absence of any rf fields. Here, we show that the expected cooling in such bichromatic traps is highly sensitive to relative position fluctuations between the two optical trapping beams, suggesting that this is the dominant mechanism limiting the currently observed cooling performance. We discuss strategies for mitigating these effects by using optimized setups featuring adapted ODT configurations. This includes proposed schemes that may mitigate three-body losses expected at very low temperatures, allowing to access the quantum dominated regime of interaction.