「Ceph」- 配置对象存储（Object Storage，S3，Swift）

解决方案

在实际的生产环境中：
1）建立部署独立的 RGW 服务，而不要与 MON 共用主机（除非负载较小）；
2）建议部署负载均衡，在负载均衡后端挂接多个 RGW 实例；

快速部署 RGW 实例

# for Ceph Octopus
# 默认 HTTP 80 访问；并自动创建 systemd 服务单元文件；
# ceph orch apply rgw myorg us-east-1 --placement="2 myhost1 myhost2"

通过客户端访问：通过 aws 命令；或其他兼容 S3 协议的 OSS 客户端；

环境概述

Ceph Cluster (with Three Nodes)
     ||
  Ceph RGW
     ||
 (S3, Swift)
     ||
  Ceph User

Ceph Cluster: ceph-node-01 172.31.252.201; ceph-node-02 172.31.252.202; ceph-node-03 172.31.252.203;
Ceph RGW: ceph-rgw-01 172.31.252.209;
Ceph Client: ubuntu-developing 172.31.252.100;

我们将在 ceph-rgw-01 中部署 RGW 服务，并通过 ubuntu-developing 访问对象存储；

第一步、配置网关服务

同步配置文件

on ceph-node-01 

# cephadm install ceph-common
# cephadm install radosgw

# scp /etc/ceph/ceph.conf 192.168.200.9:/etc/ceph/

添加网关凭证

目标：实现 RGW 访问 Ceph Cluster 的配置；

on ceph-node-01 

// 创建 Keyring 信息

ceph-authtool --create-keyring /etc/ceph/ceph.client.radosgw.keyring
chmod +r /etc/ceph/ceph.client.radosgw.keyring 

ceph-authtool /etc/ceph/ceph.client.radosgw.keyring -n client.radosgw.gateway --gen-key
ceph-authtool -n client.radosgw.gateway --cap osd 'allow rwx' --cap mon 'allow rwx' /etc/ceph/ceph.client.radosgw.keyring 

// 添加 Keyring 到集群

ceph auth add client.radosgw.gateway -i /etc/ceph/ceph.client.radosgw.keyring 

// 分配 Keyring 给节点

scp /etc/ceph/ceph.client.radosgw.keyring 172.31.252.209:/etc/ceph/ceph.client.radosgw.keyring

on ceph-rgw-01 

// ---------------------------------------------------------------------------- // 配置 RGW 实例

# vim /etc/ceph/ceph.conf
[client.radosgw.gateway]
host = ceph-rgw-01
keyring = /etc/ceph/ceph.client.radosgw.keyring
rgw socket path = /var/run/ceph/ceph.radosgw.gateway.fastcgi.sock
log file = /var/log/ceph/client.radosgw.gateway.log
rgw dns name = ceph-rgw-01.dc-laptop.d3rm.org
rgw print continue = false

关于 rgw dns name 参数：所指定的域名 及 其子域名 要能够解析到当前节点，否则 S3 将无法访问对象存储；

启动网关服务

// ---------------------------------------------------------------------------- // 启动 RGW 服务

# radosgw.gateway 为 client 的后半部分（参考 systemctl cat ceph-radosgw@.service 输出）；
systemctl start ceph-radosgw@radosgw.gateway.service
netstat -npl | grep 7480

第二步、创建访问用户

目标：实现 User 访问 RGW 的配置；

针对对象存储的访问：
1）若通过 S3 API 访问，则创建 S3 用户接口；
2）若通过 Swift 访问，则需要先创建 S3 接口的网关用户，然后在创建 Swift 接口的子用户；

鉴于 Ceph Swift 并非我们关注的重点，亦非我们当前技术栈之内，所相关内容较少提及；

on ceph-rgw-01

创建 S3 用户

# radosgw-admin -k /etc/ceph/ceph.client.radosgw.keyring -n client.radosgw.gateway \
    user create --uid=mona --display-name="Monika"
{
    "user_id": "mona",
    "display_name": "Monika",
    "email": "",
    "suspended": 0,
    "max_buckets": 1000,
    "subusers": [],
    "keys": [
        {
            "user": "mona",
            "access_key": "QP0ZJMOUW8H39JUYM9QY",
            "secret_key": "wrteSg9xIeNp7ukewOuMxaMay9zWt02ZNskop7mj"
        }
    ],
    ...
}

创建 Swift 子用户

# radosgw-admin -k /etc/ceph/ceph.client.radosgw.keyring -n client.radosgw.gateway \
    subuser create --uid=mona --subuser=mona:swift --access=full
{
    ...
    "subusers": [
        {
            "id": "mona:swift",
            "permissions": "full-control"
        }
    ],
    "keys": [
        {
            "user": "mona",
            "access_key": "QP0ZJMOUW8H39JUYM9QY",
            "secret_key": "wrteSg9xIeNp7ukewOuMxaMay9zWt02ZNskop7mj"
        }
    ],
    "swift_keys": [
        {
            "user": "mona:swift",
            "secret_key": "EA2FdtN2kuPVb0hDSS7mqlrYyw3tcnFBvhBAn5Ld"
        }
    ],
    ...
}

// 为子用户创建密钥

# radosgw-admin -k /etc/ceph/ceph.client.radosgw.keyring -n client.radosgw.gateway \
    key create --subuser=mona:swift --key-type=swift --gen-secret
{
    ...
    "subusers": [
        {
            "id": "mona:swift",
            "permissions": "full-control"
        }
    ],
    "keys": [
        {
            "user": "mona",
            "access_key": "QP0ZJMOUW8H39JUYM9QY",
            "secret_key": "wrteSg9xIeNp7ukewOuMxaMay9zWt02ZNskop7mj"
        }
    ],
    "swift_keys": [
        {
            "user": "mona:swift",
            "secret_key": "ZyyCp2WHc3LjbNBnOEV3OzuARIQqbZQEBxmFBItl"
        }
    ],
    ...
}

第三步、访问对象存储

通过 S3 访问

# apt-get install s3cmd

# s3cmd --configure
...
New settings:
  Access Key: QP0ZJMOUW8H39JUYM9QY
  Secret Key: wrteSg9xIeNp7ukewOuMxaMay9zWt02ZNskop7mj
  Default Region: US
  S3 Endpoint: ceph-rgw-01.dc-laptop.d3rm.org:7480
  DNS-style bucket+hostname:port template for accessing a bucket: %(bucket)s.ceph-rgw-01.dc-laptop.d3rm.org:7480
  Encryption password: 
  Path to GPG program: /usr/bin/gpg
  Use HTTPS protocol: False
  HTTP Proxy server name: 
  HTTP Proxy server port: 0
...

# s3cmd mb s3://testing
Bucket 's3://testing/' created

# s3cmd ls
ERROR: S3 error: 403 (SignatureDoesNotMatch)

# s3cmd put /etc/hosts s3://testing
upload: '/etc/hosts' -> 's3://testing/hosts'  [1 of 1]
 232 of 232   100% in    2s   104.72 B/s  done

# s3cmd ls s3://testing
2022-11-14 09:41       232   s3://testing/hosts

# s3cmd get s3://testing/hosts /tmp/hosts.s3
download: 's3://testing/hosts' -> '/tmp/hosts.s3'  [1 of 1]
 232 of 232   100% in    0s    66.99 kB/s  done

通过 Swift 访问

鉴于 Ceph Swift 并非我们关注的重点，亦非我们当前技术栈之内，所相关内容较少提及； 

# apt-get install python3-swiftclient

# radosgw-admin -k /etc/ceph/ceph.client.radosgw.keyring -n client.radosgw.gateway user info --uid="mona"
{
    ...
    "swift_keys": [
        {
            "user": "mona:swift",
            "secret_key": "ZyyCp2WHc3LjbNBnOEV3OzuARIQqbZQEBxmFBItl"
        }
    ],
    ...
}

# swift -A http://172.31.252.209:7480/auth/1.0 -U mona:swift -K ZyyCp2W... list
testing.ceph-rgw-01.dc-laptop.d3rm.org

# swift -A http://172.31.252.209:7480/auth/1.0 -U mona:swift -K ZyyCp2W... post second-bucket

# swift -A http://172.31.252.209:7480/auth/1.0 -U mona:swift -K ZyyCp2W... list
second-bucket
testing.ceph-rgw-01.dc-laptop.d3rm.org

Filed under: K4NZDROID - @ 2:20 AM