This is going to be a relatively short post, but after noticing a lack of documentation (or a lack of my google-fu in finding said documentation) regarding this particular issue, I figured I might as well document it here.
The other day, we had some problems getting the origin-node daemon for openshift started on one of our nodes. After looking at the logs, it appeared as though the problem was docker:
Oct 29 18:34:48 test.localhost systemd[1]: Starting Docker Application Container Engine...
Oct 29 18:34:48 test.localhost dockerd-current[1651]: time="2017-10-26T18:34:48.587544069Z" level=info msg="libcontainerd: new containerd process, pid: 1658"
Oct 29 18:34:50 test.localhost dockerd-current[1651]: time="2017-10-26T18:34:50.622278214Z" level=fatal msg="Error starting daemon: error initializing graphdriver: devmapper: Base Device UUID and Filesystem verification failed: devicemapper: Error running deviceCreate (ActivateDevice) dm_task_run failed"
Oct 29 18:34:50 test.localhost systemd[1]: docker.service: main process exited, code=exited, status=1/FAILURE
Oct 29 18:34:50 test.localhost systemd[1]: Failed to start Docker Application Container Engine.
Oct 29 18:34:50 test.localhost systemd[1]: Unit docker.service entered failed state.
Oct 29 18:34:50 test.localhost systemd[1]: docker.service failed.
Manually attempting to set up docker-storage fails as well.
[root@test ~]# docker-storage-setup
INFO: Found an already configured thin pool /dev/mapper/vg_docker-docker--pool in /etc/sysconfig/docker-storage
INFO: Waiting for device /dev/mapper/vg_docker-docker--pool to be available. Wait time remaining is 60 seconds
INFO: Waiting for device /dev/mapper/vg_docker-docker--pool to be available. Wait time remaining is 55 seconds
INFO: Waiting for device /dev/mapper/vg_docker-docker--pool to be available. Wait time remaining is 50 seconds
INFO: Waiting for device /dev/mapper/vg_docker-docker--pool to be available. Wait time remaining is 45 seconds
INFO: Waiting for device /dev/mapper/vg_docker-docker--pool to be available. Wait time remaining is 40 seconds
INFO: Waiting for device /dev/mapper/vg_docker-docker--pool to be available. Wait time remaining is 35 seconds
INFO: Waiting for device /dev/mapper/vg_docker-docker--pool to be available. Wait time remaining is 30 seconds
INFO: Waiting for device /dev/mapper/vg_docker-docker--pool to be available. Wait time remaining is 25 seconds
INFO: Waiting for device /dev/mapper/vg_docker-docker--pool to be available. Wait time remaining is 20 seconds
INFO: Waiting for device /dev/mapper/vg_docker-docker--pool to be available. Wait time remaining is 15 seconds
INFO: Waiting for device /dev/mapper/vg_docker-docker--pool to be available. Wait time remaining is 10 seconds
INFO: Waiting for device /dev/mapper/vg_docker-docker--pool to be available. Wait time remaining is 5 seconds
INFO: Timed out waiting for device /dev/mapper/vg_docker-docker--pool
ERROR: Already configured thin pool /dev/mapper/vg_docker-docker--pool is not available. If thin pool exists and is taking longer to activate, set DEVICE_WAIT_TIMEOUT to a higher value and retry. If thin pool does not exist any more, remove /etc/sysconfig/docker-storage and retry
This is actually a really easy fix. Basically, you will need to comment out or remove the DOCKER_STORAGE_OPTIONS in /etc/sysconfig/docker-storage (CentOS), and move or delete the contents of /var/lib/docker/
[root@test ~]# cat /etc/sysconfig/docker-storage
# DOCKER_STORAGE_OPTIONS="--storage-driver devicemapper --storage-opt dm.fs=xfs --storage-opt dm.thinpooldev=/dev/mapper/vg_docker-docker--pool --storage-opt dm.use_deferred_removal=true "
[root@test ~]# mv /var/lib/docker /var/lib/docker-bak
[root@test ~]# docker-storage-setup
Logical volume vg_docker/docker-pool changed.
All Fixed!