In lexicons of sea language going back to 1759, the bitter end is the part of a cable which is round about the bits (two great timbers used to belay cables) when the ship is at anchor.
See bit (n.1). So, when a cable is played out to the bitter end, there is no more left to play. The term began to be used c.1835 in non-nautical use and with probable influence of bitter (adj.).