Sense of body ownership, i.e., the feeling that “my body belongs to me,” has been examined by both the rubber hand illusion (RHI) and full body illusion (FBI). In a study that examined the relationship between RHI and depersonalization, a symptom in which people experience a lower sense of body ownership, the degree of illusion was higher in people with a high depersonalization tendency. However, other reports have suggested that people with depersonalization disorder have difficulty feeling the sense of body ownership. Examination of depersonalization suggests that the negative body recognition in people with depersonalization may make them less likely to feel a sense of body ownership, but this has not yet been examined. In this study, by manipulating top-down recognition (e.g., instructing participants to recognize a fake body as theirs), we clarified the cause of the reduced sense of body ownership in people with a high depersonalization tendency. The FBI procedure was conducted in a virtual reality environment using an avatar as a fake body. The avatar was presented from a third-person perspective, and visual-tactile stimuli were presented to create an illusion. To examine the degree of illusion, we measured the skin conductance responses to the fear stimulus presented after the visual-tactile stimuli presentation. The degree of depersonalization was measured using the Japanese version of the Cambridge Depersonalization Scale. To manipulate the top-down recognition to the avatar, we provided self-association instructions before the presentation of the visual-tactile stimuli. The results showed that participants with a high depersonalization tendency had a lower degree of illusion (rho = -.589, p < .01) in the self-association condition, and a higher one (rho = .552, p < .01) in the non-association instruction condition. This indicates that although people with a high depersonalization tendency are more likely to feel a sense of body ownership through the integration of visual-tactile stimuli, top-down recognition of the body as one’s own leads to a decrease in the sense of body ownership.