While switching from ssh-agent itself to GnuPG's ssh-agent capabilities, I had to prevent ssh-agent from starting when logging into Xfce. It seems the only way to do this is with the following command:

$ xfconf-query -c xfce4-session -p /startup/ssh-agent/enabled -n -t bool -s false