OSD full 발생 후 Node 자동 재부팅 발생
그 후, Volume 관련 OSD Pod 구동 불가 현상
위와 같이 OSD Pod의 CrashLoopBackOff 에러 확인 및 Pod 구동 불가 현상 발생
[haedong@k8s01:~]$ k get po -n rook-ceph
rook-ceph-osd-4-7f8fb7d455-2bksk 0/1 Init:CrashLoopBackOff 292 (3m15s ago) 27h
rook-ceph-osd-5-846b5b8fb5-5pdrc 0/1 Init:CrashLoopBackOff 292 (57s ago) 24h
rook-ceph-osd-6-b449b9f97-cpgfv 0/1 Init:CrashLoopBackOff 292 (114s ago) 24h
rook-ceph-osd-7-6bfc7899b9-6j2gk 0/1 Init:CrashLoopBackOff 292 (98s ago) 24h
OSD number 확인 후 log 확인
[haedong@k8s02:~]$ cd /var/log/pods/rook-ceph_rook-ceph-osd-7-6bfc7899b9-6j2gk_f306882c-d16a-459b-b19e-c18aa0da9322/activate
[haedong@k8s02:~]$ vim 283.log
{"log":"Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-7\n","stream":"stderr","time":"2022-02-24T00:24:58.9587
27749Z"}
{"log":"Running command: /usr/bin/ceph-bluestore-tool --cluster=ceph prime-osd-dir --dev /dev/ceph-8d868f14-e7d6-487c-ad1e-53329347
f36f/osd-block-4219042a-35e8-421f-b829-507fc9be4577 --path /var/lib/ceph/osd/ceph-7 --no-mon-config\n","stream":"stderr","time":"20
22-02-24T00:24:58.958752517Z"}
{"log":" stderr: failed to read label for /dev/ceph-8d868f14-e7d6-487c-ad1e-53329347f36f/osd-block-4219042a-35e8-421f-b829-507fc9be
4577: (2) No such file or directory\n","stream":"stderr","time":"2022-02-24T00:24:58.95875772Z"}
{"log":" stderr: 2022-02-24T00:24:58.955+0000 7f52c06f13c0 -1 bluestore(/dev/ceph-8d868f14-e7d6-487c-ad1e-53329347f36f/osd-block-42
19042a-35e8-421f-b829-507fc9be4577) _read_bdev_label failed to open /dev/ceph-8d868f14-e7d6-487c-ad1e-53329347f36f/osd-block-421904
2a-35e8-421f-b829-507fc9be4577: (2) No such file or directory\n","stream":"stderr","time":"2022-02-24T00:24:58.958761731Z"}
{"log":"Traceback (most recent call last):\n","stream":"stderr","time":"2022-02-24T00:24:58.959618018Z"}
{"log":" File \"/usr/sbin/ceph-volume\", line 11, in \u003cmodule\u003e\n","stream":"stderr","time":"2022-02-24T00:24:58.959634393
Z"}
log 내용 관련하여 볼륨 상태 확인 (lvm2 library가 없는 경우 yum install lvm2)
LV Status의 값 NOT available 확인
[haedong@k8s02:~]$ lvdisplay
--- Logical volume ---
LV Path /dev/ceph-8d868f14-e7d6-487c-ad1e-53329347f36f/osd-block-4219042a-35e8-421f-b829-507fc9be4577
LV Name osd-block-4219042a-35e8-421f-b829-507fc9be4577
VG Name ceph-8d868f14-e7d6-487c-ad1e-53329347f36f
LV UUID a7c3St-5yC9-qnP0-Uefy-1msp-pC2L-kSXrbf
LV Write Access read/write
LV Creation host, time rook-ceph-osd-prepare-k8s02-cwsbx, 2022-02-22 19:17:33 +0900
LV Status NOT available
LV Size <128.00 GiB
Current LE 32767
Segments 1
Allocation inherit
Read ahead sectors auto
--- Logical volume ---
LV Path /dev/ceph-d2a805b7-1b67-46a0-b527-299da655bd47/osd-block-1242e9ed-82f2-4c9c-8af4-c1a27d47c47e
LV Name osd-block-1242e9ed-82f2-4c9c-8af4-c1a27d47c47e
VG Name ceph-d2a805b7-1b67-46a0-b527-299da655bd47
LV UUID gvaJPt-lOII-QRgi-ZqzR-3j8R-HOKg-k0UFNv
LV Write Access read/write
LV Creation host, time rook-ceph-osd-prepare-k8s02-hvhjr, 2022-02-17 14:57:59 +0900
LV Status NOT available
LV Size <128.00 GiB
Current LE 32767
Segments 1
Allocation inherit
Read ahead sectors auto
--- Logical volume ---
LV Path /dev/ceph-291853e7-088b-4a27-a771-78aa6142acbe/osd-block-03b7b480-44c0-46f6-8c29-31c40cf3e585
LV Name osd-block-03b7b480-44c0-46f6-8c29-31c40cf3e585
VG Name ceph-291853e7-088b-4a27-a771-78aa6142acbe
LV UUID VbrlZa-ghQz-EVjN-gSKM-WtfQ-KpBa-lUKdxJ
LV Write Access read/write
LV Creation host, time rook-ceph-osd-prepare-k8s02-2prdv, 2022-02-03 14:02:04 +0900
LV Status NOT available
LV Size <256.00 GiB
Current LE 65535
Segments 1
Allocation inherit
Read ahead sectors auto
볼륨 그룹이
[haedong@k8s02:~]$ vgdisplay -A (Available 볼륨 그룹 출력)
[haedong@k8s02:~]$ vgdisplay (볼륨 그룹 전체 출력)
--- Volume group ---
VG Name ceph-aa89a261-f2b8-4008-97a8-3b3c90cd3552
System ID
Format lvm2
Metadata Areas 1
Metadata Sequence No 5
VG Access read/write
VG Status resizable
MAX LV 0
Cur LV 1
Open LV 0
Max PV 0
Cur PV 1
Act PV 1
VG Size <256.00 GiB
PE Size 4.00 MiB
Total PE 65535
Alloc PE / Size 65535 / <256.00 GiB
Free PE / Size 0 / 0
VG UUID 8YC1qM-Bjo2-srin-SA2N-g7gc-XG25-A3ENax
--- Volume group ---
VG Name ceph-369e561f-116a-4590-b6a0-6836207deea9
System ID
Format lvm2
Metadata Areas 1
Metadata Sequence No 5
VG Access read/write
VG Status resizable
MAX LV 0
Cur LV 1
Open LV 0
Max PV 0
Cur PV 1
Act PV 1
VG Size <128.00 GiB
PE Size 4.00 MiB
Total PE 32767
Alloc PE / Size 32767 / <128.00 GiB
Free PE / Size 0 / 0
VG UUID hrGJ4k-xwjc-sRKH-vyFo-cp45-VFiI-MeZnR8
--- Volume group ---
VG Name ceph-b75e4135-d7f4-4a9d-ad4c-860e5b99da5b
System ID
Format lvm2
Metadata Areas 1
Metadata Sequence No 5
VG Access read/write
VG Status resizable
MAX LV 0
Cur LV 1
Open LV 0
Max PV 0
Cur PV 1
Act PV 1
VG Size <128.00 GiB
PE Size 4.00 MiB
Total PE 32767
Alloc PE / Size 32767 / <128.00 GiB
Free PE / Size 0 / 0
VG UUID scqTqz-XqJE-SSWg-XTT6-4ido-v77w-lv0EeY
[haedong@k8s02:~]$ vgchange -ay
1 logical volume(s) in volume group "ceph-8d868f14-e7d6-487c-ad1e-53329347f36f" now active
1 logical volume(s) in volume group "ceph-d2a805b7-1b67-46a0-b527-299da655bd47" now active
1 logical volume(s) in volume group "ceph-291853e7-088b-4a27-a771-78aa6142acbe" now active
[haedong@k8s02:~]$ vgdisplay
[haedong@k8s02:~]$ lvdisplay
--- Logical volume ---
LV Path /dev/ceph-8d868f14-e7d6-487c-ad1e-53329347f36f/osd-block-4219042a-35e8-421f-b829-507fc9be4577
LV Name osd-block-4219042a-35e8-421f-b829-507fc9be4577
VG Name ceph-8d868f14-e7d6-487c-ad1e-53329347f36f
LV UUID a7c3St-5yC9-qnP0-Uefy-1msp-pC2L-kSXrbf
LV Write Access read/write
LV Creation host, time rook-ceph-osd-prepare-k8s02-cwsbx, 2022-02-22 19:17:33 +0900
LV Status available
# open 0
LV Size <128.00 GiB
Current LE 32767
Segments 1
Allocation inherit
Read ahead sectors auto
- currently set to 8192
Block device 251:0
--- Logical volume ---
LV Path /dev/ceph-d2a805b7-1b67-46a0-b527-299da655bd47/osd-block-1242e9ed-82f2-4c9c-8af4-c1a27d47c47e
LV Name osd-block-1242e9ed-82f2-4c9c-8af4-c1a27d47c47e
VG Name ceph-d2a805b7-1b67-46a0-b527-299da655bd47
LV UUID gvaJPt-lOII-QRgi-ZqzR-3j8R-HOKg-k0UFNv
LV Write Access read/write
LV Creation host, time rook-ceph-osd-prepare-k8s02-hvhjr, 2022-02-17 14:57:59 +0900
LV Status available
# open 0
LV Size <128.00 GiB
Current LE 32767
Segments 1
Allocation inherit
Read ahead sectors auto
- currently set to 8192
Block device 251:1
--- Logical volume ---
LV Path /dev/ceph-291853e7-088b-4a27-a771-78aa6142acbe/osd-block-03b7b480-44c0-46f6-8c29-31c40cf3e585
LV Name osd-block-03b7b480-44c0-46f6-8c29-31c40cf3e585
VG Name ceph-291853e7-088b-4a27-a771-78aa6142acbe
LV UUID VbrlZa-ghQz-EVjN-gSKM-WtfQ-KpBa-lUKdxJ
LV Write Access read/write
LV Creation host, time rook-ceph-osd-prepare-k8s02-2prdv, 2022-02-03 14:02:04 +0900
LV Status available
# open 0
LV Size <256.00 GiB
Current LE 65535
Segments 1
Allocation inherit
Read ahead sectors auto
- currently set to 8192
Block device 251:2
[haedong@k8s02:~]$ k get po -n rook-ceph
rook-ceph-osd-4-7f8fb7d455-2bksk 1/1 Running 0 28h
rook-ceph-osd-5-846b5b8fb5-5pdrc 1/1 Running 0 25h
rook-ceph-osd-6-b449b9f97-cpgfv 1/1 Running 0 25h
rook-ceph-osd-7-6bfc7899b9-6j2gk 1/1 Running 1 (25h ago) 31h