In ancient times, a small Roman fortress existed at this location. The town was first mentioned in historical documents in 1308 with the name Karom. The fortress of Karom was built on the ruins of the ancient Roman one. Until 1521, the Karom was a possession of the Hungarian noble families, of which the most well known were Bathory and Morovic. Turkish military commander Bali-beg conquered Karom in 1521, and in the next 170 years, the town was part of the Ottoman Empire. The Slavic name for the town - Karlovci, was first recorded in 1532/33. During the Ottoman rule, the town was mostly populated by Serbs, with the smaller part of population composed of Muslims. According to the Ottoman defterler from 1545, the population of Karlovci numbered 547 Christian (Serb) houses, thus it was the largest city with a Serb majority in the whole Ottoman Empire. The city also had three Orthodox churches and a monastery.
Between 16 November 1698 and 26 January 1699, the town of Karlovci was the site of a congress that ended the hostilities between the Ottoman Empire and the Holy League, a coalition of various European powers including Habsburg Monarchy, Poland, Venice and Russia; the congress produced the Treaty of Karlowitz. It was the first time a round table was used in international politics. After this peace treaty, the town was part of the Habsburg Monarchy and was included into the Military Frontier. According to the 1702 data, the population of the town was composed of 215 Orthodox and 13 Catholic houses, while according to the 1753 data, the population of the town numbered 3,843 people, of which 3,110 were ethnic Serbs. The town was also the spiritual, political and cultural centre of the Serbs in the Habsburg Monarchy. The Metropolitan of the Serb Orthodox Church resided in the town. To this day, the Serb Orthodox Patriarch retains the title of Metropolitan of Karlovci. The town also featured the earliest Serb (and Slavic in general) gymnasium founded on 3 August 1791. Three years after this, an Orthodox seminary was also founded in the town: it was the second oldest Orthodox seminary in the world (after the Spiritual Academy in Kiev), and it is still in existence.
At the Serb National Assembly in Karlovci in May 1848, Serbs declared the unification of the regions of Srem, Banat, Backa, and Baranja (including parts of the Military Frontier) into the province of Serbian Vojvodina. The first capital of Serbian Vojvodina was in Karlovci, until it was latter moved to Zemun, Veliki Beckerek, and Temisvar. In the same time the title of the Orthodox Metropolitan of Karlovci was raised to that of Patriarch, which thus established an Orthodox Patriarchate of Karlovci that existed until 1920 when it was joined with the Metropolitanate of Belgrade to form the new Patriarchate of Serbia. When Serbian Vojvodina was in 1849 transformed into the new province named Voivodeship of Serbia and Banat of Temeschwar, town of Karlovci was not included into this province, but was returned under the administration of the Military Frontier (Petrovaradin regiment that was part of Slavonian Krajina). With the abolishment of the Military Frontier in 1881, the town was included into Syrmia County of Croatia-Slavonia, the autonomous kingdom within Kingdom of Hungary and Austria-Hungary.
In 1918, the town became part of the Kingdom of Serbs, Croats and Slovenes. In the 1920s, it became the headquarters of Russian White emigres of General Wrangel whose monument remains to this day. It was also an early home to the Holy Synod of the Russian Orthodox Church Outside Russia. (Critics labeled this church the "Karlovtsy Synod" in its early days in an attempt to belittle its importance as an international Orthodox body.) Between 1929 and 1941, the town was part of Danube Banovina, a province of the Kingdom of Yugoslavia. During World War 2 (1941-1944), the town was occupied by the Axis Powers and it was attached to the Independent State of Croatia. During that time its name was changed to Hrvatski Karlovci. Since the end of the war, the town has been part of the Autonomous Province of Vojvodina. Between 1980 and 1989, Sremski Karlovci was one of the seven municipalities of Novi Sad City. Today, the municipality is not part of Novi Sad City, but a separate administrative unit of South Backa District.
Most recently, the government of Serbia announced its decision to move the Constitutional Court of Serbia to this town as part of the national strategy for decentralization. Furthermore the government decided to make this move because of the historic importance of this town in Serbian history as well because of its relative proximity to the capital, Belgrade.