Purpose: Although the use of soft cervical collars in the emergency department (ED), for whiplash-associated disorders (WAD), is controversial, it is still widely adopted. The purpose of our study was to investigate the impact of the early use of soft cervical collars on the return to the ED, within three months of a road traffic collision. Methods: We conducted a retrospective observational study on WAD patients from two EDs in Verona (Italy). Patients in the earlier acute phase of WAD (within 48 h from the trauma) were included; those with serious conditions (WAD IV) were excluded. As an end point, we considered patients who returned to the ED complaining of WAD symptoms within three months as positive outcome for WAD persistence. Results: 2162 patients were included; of those, 85.4% (n = 1847/2162) received a soft cervical collar prescription. Further, 8.4% (n = 156/1847) of those with a soft cervical collar prescription, and 2.5% (n = 8/315) of those without a soft cervical collar (p < 0.001) returned to the ED within three months. The use of the soft cervical collar was an independent risk factor for ED return within three months, with an OR, adjusted for possible clinical confounders, equal to 3.418 (95% CI 1.653–7.069; p < 0.001). After the propensity score matching, 25.5% of the patients (n = 25/98) using the soft cervical collar returned to the ED at three months, compared to the 6.1% (n = 6/98) that did not adopt the soft cervical collar. The use of a soft cervical collar was associated with ED return with an OR = 4.314 (95% CI 2.066–11.668; p = 0.001). Conclusions: Our study shows that the positioning of the soft collar in a cohort of patients with acute WAD, following a rear-end car collision, is an independent potential risk factor to the return to the ED. Clinically, the use of the collar is a non-recommended practice and seems to be related to an increased risk of delayed recovery. There is a need to inform healthcare providers involved in the ED of the aim to limit the use of the soft cervical collar. A closer collaboration between clinicians (e.g., physicians, physical therapists, nurses) is suggested in the ED. Future primary studies should determine differences between having used or not having used the collar, and compare early physical therapy in the ED compared with the utilization of the collar.