Built from the mid-15th century to the early 16th century for the Viscounts of Rohan, Pontivy Castle is one of the last fortified castles built in Brittany. It is characteristic of Breton military architecture from the end of the 15th century.

Located at the northern end of the old town, Pontivy Castle was built from the mid-15th century to the early 16th century. Most of the construction was carried out on the initiative of Jean II de Rohan, known as the "Grand Viscount". Pontivy having become the capital of the Viscountcy of Rohan in 1396, Jean II needed a castle worthy of his rank.

With its thick walls, wide moats and massive towers projecting strongly from the curtain walls, Pontivy Castle is evidence of a military architecture which, at the end of the 15th century, attempted to adapt to the development of artillery fire. The presence of "Breton machicolations" or the use of schist and granite for construction are characteristic of the peninsula's fortified castles. The defensive function is combined with sustained attention to comfort and decoration: sculpted skylights, ornate water outlets, chimneys and latrines are still there to testify to this.

The castle permanently houses a contemporary installation by the Japanese artist Koki Watanabe as well as an exhibition dedicated to the sculptor Gaston Schweitzer (Montreuil-sous-Bois, 1879 - Paris, 1962).