1770, used by Capt. Cook and botanist Joseph Banks, supposedly an aborigine word from northeast Queensland, Australia, usually said to be unknown now in any native language. However, according to Australian linguist R.M.W. Dixon ("The Languages of Australia," Cambridge, 1980), the word probably is from Guugu Yimidhirr (Endeavour River-area Aborigine language) /gaNurru/ "large black kangaroo."
Kangaroo court is American English, first recorded 1850 in a Southwestern context (also mustang court), from notion of proceeding by leaps.