William Cavendish, 5th Duke of Devonshire

By his first wife, he had one son (William Cavendish, 6th Duke of Devonshire, sometimes called "The Bachelor Duke", who succeeded him and who died unmarried in 1858), and two daughters: Lady Georgiana "Little G" Cavendish, later the Countess of Carlisle (wife of George Howard, 6th Earl of Carlisle), and Lady Harriet "Harryo" Cavendish, later the Countess Granville (wife of Lord Granville Leveson-Gower, who would be created 1st Earl Granville). Both daughters left descendants and the title of Clifford barony fell into abeyance between them. The dukedom and estates would pass to a grandson of a younger brother of the 5th Duke of Devonshire; however, the 7th Duke of Devonshire would marry a daughter of the 6th Earl of Carlisle, who was thus a granddaughter of the 5th Duke and niece of the 6th Duke.

Georgiana Cavendish became a socialite who would gather around her a large circle of literary and political friends. Thomas Gainsborough and Joshua Reynolds painted her; the Gainsborough painting would be disposed of by the 5th Duke and would be recovered much later, after many vicissitudes.

By his second wife, Lady Elizabeth Foster, he had no legitimate issue, but the couple had two illegitimate children born before their marriage. A son, Augustus, was given the surname Clifford and became Sir Augustus Clifford and rose to be an admiral in the Royal Navy and the Gentleman Usher of the Black Rod in the House of Lords; his descendants would die out in the male line in 1895. The Duke's daughter by Lady Elizabeth, Caroline, was given a different surname from her brother, St. Jules. Caroline St. Jules would marry the Hon. George Lamb, a brother of the 2nd Viscount Melbourne (himself married to Lady Caroline Ponsonby, niece of Lady Georgiana Spencer, the 5th Duke's 1st wife). Caroline and George Lamb had no issue.

The 5th Duke had a daughter—Charlotte, given the surname Williams—by his mistress, Charlotte Spencer, the daughter of an indigent clergyman. His first child was born shortly after his marriage to Lady Georgiana Spencer (no relation to his mistress). Charlotte would later marry suitably.