Henry Yelverton, 19th Baron Grey de Ruthyn

The title of Baron Grey de Ruthyn had a long history and had belonged to the Earl of Kent for a period. It passed to the Earl of Sussex from 1717, the Yelvertons. Henry Yelverton, 18th Baron Grey de Ruthyn, 3rd Earl of Sussex died in 1799, with no sons. The Grey de Ruthyn title therefore passed to 19-year-old Henry, son of the Earl's daughter, Lady Barbara Yelverton (who had died in 1781) and her husband, Edward Thoroton Gould, who was the grandson of Robert Thoroton Esq. of Screveton Hall, Flintham, Nottinghamshire.[1][2] The younger Yelverton could not inherit the title of Earl of Sussex through his mother.

Lord Grey took his seat in the House of Lords as a Whig.

On 21 June 1809, he married Anna Maria Kellam, daughter of William Kellam, of Ryton-upon-Dunsmore, Warwick. Byron wrote from his European trip to his mother: "So Lord G— is married to a rustic. Well done! If I wed, I will bring home a Sultana, with half a dozen cities for a dowry, and reconcile you to an Ottoman daughter-in-law, with a bushel of pearls not larger than ostrich eggs, or smaller than walnuts."

The couple had one daughter, Barbara, born on 20 May 1810 (later Barbara Rawdon-Hastings, Marchioness of Hastings). In October of the same year, Grey died at his seat of Brandon House, near Coventry, aged 30.

Lord Byron had inherited Newstead Abbey with his title; the estate was leased to Lord Grey, from January 1803, until Byron came of age. Later that year, Byron stayed at Newstead Abbey for the summer whilst Grey was traveling abroad. When Grey returned, Byron stayed on, not returning for the Autumn term at Harrow. He and Grey became friends, spending their days and nights on shooting expeditions. Then Byron suddenly broke off their friendship and left Newstead.

Byron wrote to his half-sister, Augusta Leigh: "I am not reconciled to Lord Grey, and I never will. He was once my Greatest Friend, my reasons for ceasing that Friendship are such as I cannot explain, not even to you, my Dear Sister, (although were they to be made known to any body, you would be the first) but they will ever remain hidden in my own breast." Byron's mother was keen on a reconciliation. Byron wrote again to his sister of his troubles with his mother: "all our disputes have been lately heightened by my one with that object of my cordial, deliberate detestation, Lord Grey de Ruthyn." Byron's later apologetic letters to Grey and Grey's inability to understand his young friend's breaking-off of their relationship may point to a sexual relationship that Byron later regretted. They were not reconciled.

In April 1808, Lord Grey left Newstead at the end of his lease.