Building materials like steel, cement, and concrete are mainly the most used man-made materials in the world and the core materials for the construction industry. The high demand for these materials is increasing rapidly, resulting in significant greenhouse gas emissions to the atmosphere and contributing to global warming; therefore, a crucial need exists to investigate their environmental impact and to find effective strategies to decarbonize the building material industry. This paper aims to review the available decarbonization strategies that can be implemented in the building materials industry, especially in the steel and cement industry, to decrease CO2 emissions associated with the production of building materials. It also proposes the use of alternative materials which have a reduced environmental impact compared to the traditional ones, like the use of fiber-reinforced polymer instead of steel for reinforcing structural elements or the use of alternative cement like geopolymer instead of ordinary Portland cement. The reported results of this study stated that the widespread production of concrete, cement, and steel leads to a negative impact on the environment. Additionally, the current standards for building materials are based on technical performance without considering their environmental impacts; therefore, using alternative materials helps to balance the technical and environmental performances. Utilizing geopolymer cement as a replacement for Portland cement reduced the CO2 emissions by 10 to 70%, while adopting Fiber-reinforced polymer instead of steel in structural members reduced the CO2 emissions by 4 to 75%, depending on the type of the FRP. Furthermore, reducing CO2 emissions either in steel or in the concrete industry requires developing new technologies, which will be a key factor in accomplishing developments in these sectors. A combination of decarbonization technologies should be implemented to achieve the desired reduction in CO2 emissions.