Mort married Theresa Laidley at St Lawrence's, Sydney, in 1841 — they had nine children together — and after her early death he married Marianne Macaulay in 1874. He kept two houses: Greenoaks, his Gothic-revival home at Darling Point, and the rural estate at Bodalla.
A devout Anglican, he was an unusually serious philanthropist for his era. He supported the Sydney Homœopathic Dispensary, opened the gardens of Greenoaks to the public, and treated his commercial ventures as much as community works as private enterprises. He died at Bodalla on 9 May 1878 — of pneumonia, contracted, by family account, while attending a sick employee in the night. He was buried at the Home Farm, in a spot he had chosen himself.
The southern Sydney suburb of Mortdale bears his name. Greenoaks was acquired in 1910 by the Anglican Diocese of Sydney and renamed Bishopscourt — the residence of the Archbishop ever since. The statue raised at Macquarie Place in 1883 — the first to honour any Australian — still stands.