The Castle Core was probably established as early as the 13th century in the form of a watchtower at a convenient location protecting the trade route linking the towns of Sušice and Horažďovice. There are no written records about the castle's earliest owners. The most recent hypothesis among historians situates the Romanesque foundations of the castle as the northernmost part of the domain of the counts of Bogen and later the Wittelsbachs. Only towards the end of the century did this territory probably become part of the Bohemian state once more. It is assumed that at that time the castle, still in the form of a purely residential-defensive tower, was inhabited by the lords of Velhartice. Rabí enters written history much later, in 1380, when the brothers Brenek and Půta of Rýzmberg acquired the territory by purchase or through family ties. In 1407 the estate was inherited by Brenek's sons, most notably Jan of Rýzmberg, who was known as an opponent of the Hussites. As a result the Hussite forces besieged and captured the castle twice. (During the second siege it was here that Jan Žižka lost his second eye.) After Jan's death Rabí passed to Vilém the Younger of Rýzmberg and later to his nephew Půta II Švihovský of Rýzmberg, who, thanks to his immense wealth and influential position, oversaw an exceptional building expansion of the complex (the reconstruction was probably carried out with the participation of the leading architect Benedikt Rejt). During his lifetime Rabí acquired its final form and became the centre of the Rýzmberg domain in the upper Otava basin.

Půta's sons, however, were forced to sell off the family fortune, and the castle subsequently changed hands frequently. In 1570 it was acquired by Adam Chanovský of Dlouhá Ves, whose family retained ownership until the first decade of the 18th century. The complex, by then considerably dilapidated and uninhabited, then passed to the noble Lamberg family, who donated the ruin in 1920 to the Society for the Preservation of Artistic, Historical and Natural Monuments in Horažďovice. In 1954 the castle was handed over to the state, and in 1978 it was declared a National Cultural Monument.

Benedikt Rejt (c. 1454–1534) was of south German origin. From 1489 he worked at Prague Castle, where he eventually succeeded Hans Spiess as court master builder. He rebuilt the Royal Palace there, constructing above all the Vladislav Hall, the Riders' Staircase, and the Louis Wing. The new artillery fortifications are also Rejt's work. Numerous stonemasons and builders trained under his direction in the Prague court workshop (W. Roskopf and others). It was precisely the lords of Rýzmberg who engaged Rejt as a fortification expert — the author of some of the most sophisticated and accomplished artillery defence systems of Bohemian late Gothic, enabling active defence — particularly in the construction of the outer fortifications of Švihov Castle and Rabí. In the later phase of his career he was also influenced by the work of Arnold of Westphalia, as the palace in Blatná demonstrates. Beyond present-day Czech territory, he built, for example, the magnificent four-winged Frankenstein Castle in Poland for the son of George of Poděbrady, Prince Hynek of Münsterberg.