The great river once again is responsible for the destiny of the city when it causes the relevant hydrogeological upheaval, causing the city to lose its physiognomy and reducing it to a small agricultural centre, provoking on more than one occasion the depopulation and abandonment of the city in the long centuries that follow the Roman era (III-IX century A.D.).
It would be the feudal lord Azzo II, of the Obertenghi dynasty, to give way to the rebirth of the city, building in the mid XI century a first defensive castle on the hill upon which stood the remains of a Roman castrum. In the following centuries the family saw a constant increase in its power and its territorial domains and consequently also the castle was enlarged and transformed into a fortress of significant proportions, as can be seen by the long walls that have survived intact to the present day. In 1249 the castle of the Marquis of the Estensi, who had taken the name from the town of which they had become the lords, underwent devastating destruction because of the siege it was subject to by its fiercest enemy Ezzelino da Romano, who had already seriously damaged it in 1238. It would be, however, the Padovani family to definitively push the Estensi family from the city several decades later, forcing Azzo VII to seek refuge in Ferrara (city of which he had already become the Lord) and to settle where the dynasty of the Estensi would know greater fame. With the exit from the scene of the family of Princes that had brought so much prosperity, well-being and prestige to the city and its territories in the period of its government, Ubertino da Carrara, Lord of Padua, rebuilt and enlarged the castle starting from 1339. However, during the domination of the Carraresi, Este no longer would hold the position of important political and administrative centre that on several occasions had been decisive for Italian medieval history of the first Estense period.