It is a circular building, featureless from the outside, with a main gate that leads to the large courtyard inside. Surrounding the courtyard are 114 cave-like storage rooms, arranged in several levels. The lowest level, which partially lies underground, is used to store olive oil, while the upper levels are mainly used for barley and wheat. At the side of the main gate, a staircase leads to the top, where a walk completes a full circle around the top of the structure.
In a region of limited crops, granaries like this shelter the harvest from animals, thieves and weather extremes. The Qasr al-Haj was built in the 12th century by Sheikh Abu Jatla, who rented the storage rooms to several parties in exchange for a small amount of barley and wheat, which he then distributed among the poor.
The building originally comprised of 114 chambers, that corresponded to the number of chapters in the holy Koran. Each family had a space in that Qasr where they could keep food and grains for safe keeping. The qasr was in use even as late as the 1960s.
Since the building is still in use, visitors are encouraged not to enter the storage rooms without the guidance of the caretaker, or the permission from the owners. As of October 2011, it remains unclear, whether the 2011 Libyan Civil War damaged Qasr al-Haj.