Human activity has required, since its origins, stones as raw material for carving, construction and rock art. The study, exploration, use and maintenance of building stones is a global phenomenon that has evolved from the first shelters, manufacture of lithic tools, to the construction of houses, infrastructures and monuments. Druids, philosophers, clergymen, quarrymen, master builders, naturalists, travelers, architects, archaeologists, physicists, chemists, curators, restorers, museologists, engineers and geologists, among other professionals, have worked with stones and they have produced the current knowledge in heritage stones. They are stones that have special significance in human culture. In this way, the connotation of heritage in stones has been acquired over the time. That is, the stones at the time of their historical use were simply stones used for a certain purpose. Therefore, the concept of heritage stone is broad, with cultural, historic, artistic, architectural, and scientific implications. A historical synthesis is presented of the main events that marked the use of stones from prehistory, through ancient history, medieval times, and to the modern period. In addition, the main authors who have written about stones are surveyed from Ancient Roman times to the middle of the twentieth century. Subtle properties of stones have been discovered and exploited by artists and artisans long before rigorous science took notice of them and explained them.