Oh, and the next time you can reproduce this, would you try running /usr/bin/byobu-janitor, and see if that hooks up your ssh agent?
Oh, and the next time you can reproduce this, would you try running /usr/bin/ byobu-janitor, and see if that hooks up your ssh agent?