My write-up is based on this post.
During my first node install, I made the mistake of configuring my Ceph network as my primary Proxmox network. Unfortunately, there doesn't appear to be any easy way to fix this. Here's how I fixed it.
Do not perform this process on a production Ceph environment.
Steps 1-6 are done on each node in the cluster.
1. We're going to purge Ceph - stop the Ceph processes.
You'd think that changing the network config in /etc/pve/ceph.conf would be enough but the Managers and Monitors appear to be keeping that from being the case.
systemctl stop ceph-mon.target ceph-mgr.target ceph-mds.target ceph-osd.target2. Remove systemd files
rm -rf /etc/systemd/system/ceph*3. Be sure that they all stopped.
killall -9 ceph-mon ceph-mgr ceph-mds4. Remove these Ceph library directories.
rm -rf /var/lib/ceph/mon/ /var/lib/ceph/mgr/ /var/lib/ceph/mds/5. Use the pveceph command to purge Ceph.
pveceph purge6. Create directories /var/lib/ceph/mon and /var/lib/ceph/mgr then change their ownership to the ceph user.
mkdir /var/lib/ceph/mon
mkdir /var/lib/ceph/mgr
chown ceph:ceph /var/lib/ceph/mon
chow ceph:ceph /var/lib/ceph/mgr7. Go back to your Proxmox GUI, to the Ceph tab and reconfigure Ceph.
8. If it does not create the Managers and Monitors, manually create those in the GUI as well. (Step 6 should prevent any No such file or directory or Permission Denied errors for this step)
What a bitch, I tell yah.