The issue happened again, but I am quite sure I found the issue.
I had cloned /etc/X11/xinit/xinitrc to create my own ~/.xinitrc but I forgot to include these line at the end of ~/.xinitrc
Code:
if [ "$SSH_AGENT_PID" ]; then
ssh-add -D < /dev/null
eval `ssh-agent -s -k`
fi
Since I did this, the intermittent has not happened