Rook ceph on k8s 환경에서 Object storage 설정 방법이다.
[root@k8s01 object-store-01]# vim object.yaml
#################################################################################################################
# Create an object store with settings for replication in a production environment. A minimum of 3 hosts with
# OSDs are required in this example.
# kubectl create -f object.yaml
#################################################################################################################
apiVersion: ceph.rook.io/v1
kind: CephObjectStore
metadata:
# ObjectStore 이름 설정
name: object-store-01
namespace: rook-ceph # namespace:cluster
spec:
# The pool spec used to create the metadata pools. Must use replication.
metadataPool:
failureDomain: host
replicated:
size: 3
# Disallow setting pool with replica 1, this could lead to data loss without recovery.
# Make sure you're *ABSOLUTELY CERTAIN* that is what you want
requireSafeReplicaSize: true
parameters:
# Inline compression mode for the data pool
# Further reference: https://docs.ceph.com/docs/master/rados/configuration/bluestore-config-ref/#inline-compression
compression_mode: none
# gives a hint (%) to Ceph in terms of expected consumption of the total cluster capacity of a given pool
# for more info: https://docs.ceph.com/docs/master/rados/operations/placement-groups/#specifying-expected-pool-size
#target_size_ratio: ".5"
# The pool spec used to create the data pool. Can use replication or erasure coding.
dataPool:
failureDomain: host
replicated:
size: 3
# Disallow setting pool with replica 1, this could lead to data loss without recovery.
# Make sure you're *ABSOLUTELY CERTAIN* that is what you want
requireSafeReplicaSize: true
parameters:
# Inline compression mode for the data pool
# Further reference: https://docs.ceph.com/docs/master/rados/configuration/bluestore-config-ref/#inline-compression
compression_mode: none
# gives a hint (%) to Ceph in terms of expected consumption of the total cluster capacity of a given pool
# for more info: https://docs.ceph.com/docs/master/rados/operations/placement-groups/#specifying-expected-pool-size
#target_size_ratio: ".5"
# Whether to preserve metadata and data pools on object store deletion
preservePoolsOnDelete: false
# The gateway service configuration
gateway:
# A reference to the secret in the rook namespace where the ssl certificate is stored
# sslCertificateRef:
# A reference to the secret in the rook namespace where the ca bundle is stored
# caBundleRef:
# The port that RGW pods will listen on (http)
port: 80
# The port that RGW pods will listen on (https). An ssl certificate is required.
# securePort: 443
# The number of pods in the rgw deployment
instances: 2
# The affinity rules to apply to the rgw deployment.
placement:
podAntiAffinity:
preferredDuringSchedulingIgnoredDuringExecution:
- weight: 100
podAffinityTerm:
labelSelector:
matchExpressions:
- key: app
operator: In
values:
- rook-ceph-rgw
# topologyKey: */zone can be used to spread RGW across different AZ
# Use <topologyKey: failure-domain.beta.kubernetes.io/zone> in k8s cluster if your cluster is v1.16 or lower
# Use <topologyKey: topology.kubernetes.io/zone> in k8s cluster is v1.17 or upper
topologyKey: kubernetes.io/hostname
# A key/value list of annotations
# nodeAffinity:
# requiredDuringSchedulingIgnoredDuringExecution:
# nodeSelectorTerms:
# - matchExpressions:
# - key: role
# operator: In
# values:
# - rgw-node
# topologySpreadConstraints:
# tolerations:
# - key: rgw-node
# operator: Exists
# podAffinity:
# podAntiAffinity:
# A key/value list of annotations
annotations:
# key: value
# A key/value list of labels
labels:
# key: value
resources:
# The requests and limits set here, allow the object store gateway Pod(s) to use half of one CPU core and 1 gigabyte of memory
# limits:
# cpu: "500m"
# memory: "1024Mi"
# requests:
# cpu: "500m"
# memory: "1024Mi"
# priorityClassName: my-priority-class
#zone:
#name: zone-a
# service endpoint healthcheck
healthCheck:
bucket:
disabled: false
interval: 60s
# Configure the pod probes for the rgw daemon
startupProbe:
disabled: false
livenessProbe:
disabled: false
readinessProbe:
disabled: false
# security oriented settings
# security:
# To enable the KMS configuration properly don't forget to uncomment the Secret at the end of the file
# kms:
# # name of the config map containing all the kms connection details
# connectionDetails:
# KMS_PROVIDER: "vault"
# VAULT_ADDR: VAULT_ADDR_CHANGE_ME # e,g: http://vault.my-domain.com:8200
# VAULT_BACKEND_PATH: "rook"
# VAULT_SECRET_ENGINE: "kv"
# VAULT_BACKEND: v2
# # name of the secret containing the kms authentication token
# tokenSecretName: rook-vault-token
# # UNCOMMENT THIS TO ENABLE A KMS CONNECTION
# # Also, do not forget to replace both:
# # * ROOK_TOKEN_CHANGE_ME: with a base64 encoded value of the token to use
# # * VAULT_ADDR_CHANGE_ME: with the Vault address
# ---
# apiVersion: v1
# kind: Secret
# metadata:
# name: rook-vault-token
# namespace: rook-ceph # namespace:cluster
# data:
# token: ROOK_TOKEN_CHANGE_ME
[root@k8s01 object-store-01]# kubectl create -f object.yaml
[root@k8s01 object-store-01]# kubectl -n rook-ceph get CephObjectStore
NAME PHASE
object-store-01 Connected
[root@k8s01 object-store-01]# vi storageclass-bucket-delete.yaml
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: rook-ceph-bucket
provisioner: rook-ceph.ceph.rook.io/bucket # driver:namespace:cluster
# set the reclaim policy to delete the bucket and all objects
# when its OBC is deleted.
reclaimPolicy: Delete
parameters:
objectStoreName: object-store-01
objectStoreNamespace: rook-ceph # namespace:cluster
#region: us-east-1
# To accommodate brownfield cases reference the existing bucket name here instead
# of in the ObjectBucketClaim (OBC). In this case the provisioner will grant
# access to the bucket by creating a new user, attaching it to the bucket, and
# providing the credentials via a Secret in the namespace of the requesting OBC.
#bucketName:
[root@k8s01 object-store-01]# kubectl create -f storageclass-bucket-delete.yaml
[root@k8s01 object-store-01]# kubectl -n rook-ceph get StorageClass
[root@k8s01 object-store-01]# vi object-bucket-claim-delete.yaml
apiVersion: objectbucket.io/v1alpha1
kind: ObjectBucketClaim
metadata:
name: ceph-delete-bucket
spec:
# To create a new bucket specify either `bucketName` or
# `generateBucketName` here. Both cannot be used. To access
# an existing bucket the bucket name needs to be defined in
# the StorageClass referenced here, and both `bucketName` and
# `generateBucketName` must be omitted in the OBC.
#bucketName:
generateBucketName: gtc
storageClassName: rook-ceph-bucket
additionalConfig:
# To set for quota for OBC
#maxObjects: "1000"
#maxSize: "2G"
[root@k8s01 object-store-01]# kubectl create -f object-bucket-claim-delete.yaml
[root@k8s01 object-store-01]# vi rgw-external.yaml
apiVersion: v1
kind: Service
metadata:
name: rook-ceph-rgw-object-store-01-external
namespace: rook-ceph # namespace:cluster
labels:
app: rook-ceph-rgw
rook_cluster: rook-ceph # namespace:cluster
rook_object_store: object-store-01
spec:
ports:
- name: rgw
nodePort: 30071
port: 80 # service port mentioned in object store crd
protocol: TCP
targetPort: 8080
selector:
app: rook-ceph-rgw
rook_cluster: rook-ceph # namespace:cluster
rook_object_store: object-store-01
sessionAffinity: None
type: NodePort
[root@k8s01 object-store-01]# kubectl create -f rgw-external.yaml
[root@k8s01 object-store-01]# kubectl -n rook-ceph get service
rook-ceph-rgw-object-store-01 ClusterIP 10.102.125.183 <none> 80/TCP 176m
rook-ceph-rgw-object-store-01-external NodePort 10.100.226.234 <none> 80:30071/TCP 3s
[root@k8s01 object-store-01]# vi object-user.yaml
#################################################################################################################
# Create an object store user for access to the s3 endpoint.
# kubectl create -f object-user.yaml
#################################################################################################################
apiVersion: ceph.rook.io/v1
kind: CephObjectStoreUser
metadata:
name: encore
namespace: rook-ceph # namespace:cluster
spec:
store: object-store-01
displayName: "encore"
# Quotas set on the user
# quotas:
# maxBuckets: 100
# maxSize: 10G
# maxObjects: 10000
# Additional permissions given to the user
# capabilities:
# user: "*"
# bucket: "*"
# metadata: "*"
# usage: "*"
# zone: "*"
[root@k8s01 object-store-01]# kubectl create -f object-user.yaml
NAME PHASE
encore Ready
Pod 목록 확인
[root@k8s01 object-store-01]# k -n rook-ceph get po
rook-ceph-tools-d6d7c985c-2trbn 1/1 Running 0 11d
rook-ceph-tools Pod 진입
[root@k8s01 object-store-01]# k -n rook-ceph exec -it rook-ceph-tools-d6d7c985c-2trbn -- /bin/bash
radosgw-admin user modify --uid={USER_ID} --access-key=${REDACTED}} --secret-key=
[rook@rook-ceph-tools-d6d7c985c-2trbn /]$ radosgw-admin user modify --uid=encore --access-key=${REDACTED} --secret-key=secret-key
위에서 설정한 accesskey, secretkey를 사용하여 Ceph object storage에 연결한다.