The earldom of Charleville (of the second creation) was granted to Charles William Bury (1764-1835) in 1806. He had inherited his wealth and estates at Charleville through his paternal grandmother, the sister and heiress of Charles Moore (1712-1764), the 1st earl of Charleville (of the first creation). Lord Charleville and his wife, Catherine Maria (widow of James Tisdall, County Louth), set about building the Gothic mansion Charleville Forest in 1800 on the site of an older 17th century house known as Redwood. The building project took many years and cost an enormous sum of money, which ultimately caused a financial burden for successive generations of the Bury family. They also continued with a lavish lifestyle, living for part of the year in London and travelling extensively on the continent.

Their son, Charles William, Lord Tullamore (1801-1851) married in Florence in 1821 and set up a second expensive household. When he inherited the estate in 1835, it was heavily encumbered. By 1844, it was unsustainable, Charleville was closed up and Lord and Lady Charleville headed for Berlin. Their son, Charles William George, 3rd earl of Charleville (1822-1859) succeeded to the estate in 1851 and returned to Charleville with his wife, Arabella at this time. Unfortunately, they both died within a couple of years of each other, leaving five young children as wards of chancery in Charleville Castle. The children's guardian was named as Alfred Bury, their uncle.

Charles William Francis Bury (1852-1874) inherited the earldom as a minor of seven years in 1859. When he came of age in 1873, there was much festivity and celebration in Tullamore, but sadly he died in New York a year later at the age of 22. The title then reverted to Alfred Bury (1829-1875), the youngest brother of the 3rd earl. He also only had one year as earl, dying in 1875 with no male heirs.

The earldom became extinct at this point but the estates passed to Lady Emily Howard-Bury (1856-1931), Alfred's niece. Her son, Col. Charles Kenneth Howard-Bury (1883-1963), who was born and raised in Charleville Castle, inherited Belevedere in Mullingar in County Westmeath from a cousin Charles Brinsley Marlay in 1912, and made it his permanent home. By the time he inherited Charleville in 1931, the family had ceased to live on the estate. In 1948 he arranged a large auction of all its furniture and paintings. On his death in 1963, he bequeathed Charleville to his cousin, Major William Hutton Bury (1914-1982) whose family have managed the estate since.