Is there a way to hide the Xref layers from appearing on the pick layers toolbar? I have many Xrefs in my drawing and they all show up grayed out in my pick list.

No, you can only use layer filters for the main Layer manager dialog (see Related tips).