Trinidadian thinker and activist C. L. R. James penned a criticism of Herman Melville’s work, Mariners, Renegades, and Castaways, while incarcerated in Ellis Island, New York, in the early 1950s. I investigate how the contradictory claims on labour and race, literary analysis, and communism in the last chapter come from what I call the prison–detention continuum: a historical continuity allocated to prison and detention facilities despite an overt difference between the two. The distinction survived so as to maintain racial classification and labour force from the times of slavery and plantation to the Cold War era. The physical statuses of those incarcerated were insecure when the McCarran–Walter Act legalized ideological surveillance and accelerated racism inside and outside the carceral spaces. In his book on Melville, James clarifies the difference between prison and detention by emphasizing labour’s role in Ellis Island. He situates his personal experience of maltreatment of his ulcer as a structural issue, produced by the way the officers obey their authorities without any principle. To foreground the docile individuals in the totalitarian society, he compares the inmates and officers on Ellis Island with the shipmates of the Pequod in Moby Dick. Furthermore, he regards that if labour is racialized, it will necessarily culminate in revolt. I argue that James’s reference to the Korean War POWs on Koje Island prefigures an interracial solidarity that becomes visible after the Bandung Conference of 1955.