Ruffo was born at San Lucido in Calabria, then part of the Kingdom of Two Sicilies. His father, Litterio Ruffo, was duke of Baranello, and his mother, Giustiniana, was of the Roman family of Colonna. Fabrizio owed his education to his uncle, Tommaso Cardinal Ruffo, then dean of the College of Cardinals. In early life he secured the favour of Giovanni Angelo Braschi, who in 1775 became Pope Pius VI. Ruffo was placed by the pope among the chierici di camera, the clerks who formed the papal civil and financial service. He was later promoted to treasurer-general, a post which carried with it the ministry of war. Ruffo's conduct in office was diversely judged. Pietro Colletta, the historian of Naples, speaks of him as corrupt, and Jomini repeats the charge, but these can be dismissed as part of a hostile tradition. In fact, he was widely regarded as a reformer.
Ruffo's biographer, Sachinelli, says that he incurred hostility by restricting the feudal powers of some of the landowners in the Papal States. In 1791 he was removed from the treasurership, but was created cardinal on 2 September, though he was not in orders and in fact never became a priest. Ruffo went to Naples, where he was named administrator of the royal domain of Caserta, and received the abbey of Santa Sofia, Benevento in commendam.
When in December 1798 the French troops advanced on Naples, Ruffo fled to Palermo with the royal family. He was chosen to head a royalist movement in Calabria, where his family, though impoverished by debt, exercised large feudal powers. He was named vicar-general on 25 January 1800. On 8 February he landed at La Cortona with a small following, and began to raise the so-called "army of the faith" in association with Michele Pezza, "Fra Diavolo", and other brigand leaders. Backed by the Russian fleet of Admiral Ushakov, Ruffo had no difficulty in upsetting the Parthenopean Republican government established by the French, and by June had advanced to Naples. Possibly exceeding his authority, he promised the Neapolitan republicans immunity from reprisals and obtained their surrender in June, 1799. In the meantime Rear-Admiral Horatio Nelson appeared with his fleet; he called the cardinal to task for his leniency, and revoked the terms of surrender. The republicans, it was asserted, had surrendered under terms that were unclear. One of the main republican figures, former Admiral Francesco Caracciolo, was ignominiously executed on 30 June, and widespread reprisals and executions of other republican sympathizers in Naples followed.
The campaign gave rise to much controversy among nineteenth-century historians. Ruffo appears to have lost favor with the king by showing a tendency to spare the republicans. He resigned his vicar-generalship to the prince of Cassero, and during the second French occupation and the reigns of Joseph Bonaparte and Joachim Murat he lived quietly in Naples. Some notice was taken of him by Napoleon, but he never held an important post. After the restoration of the Bourbons he was received into favor. During the revolutionary troubles of 1822 he was consulted by the king, and was even in office for a very short time as a loyalist minister.