Learning RBAC The Hard Way

11/11/18, by Dario Ferrer

Let’s face it, you’re here because a search engine showed you some interesting code or because you saw a preview in a listing. So I’ll go straight to the point. RBAC in Kubernetes is still unknown to many and it is difficult to debug and get it running. We implemented a very simple RBAC solution for our Hubot pod, here is how it looks:

Assumptions

You may like this if you’re not new to K8s but more or less experienced. You know what is Kubernetes and you are already managing one or more clusters, maybe already in prod, you know that you need to secure it (also) from within and you know that RBAC is a pending subject that needs to be addressed sooner or later.

Brief overview of terminology

  • Api Object: All the following definitions can be considered RBAC API objects, and the RBAC API itself is considered a K8s apigroup.
  • Role: A K8s role is a group of rules that can be bind to a User or a ServiceAccount. The rules defined in the Role grant permissions, there are no deny ones. Roles are scoped to a Namespace.
  • ClusterRole: Exactly the same thing than the Role but they are cluster wide, so not scoped to a Namespace.
  • User: A User represents an external service who uses the K8s API, Kubernetes auth maps external auth services such as Google accounts or AWS users with K8s Users.
  • Group: It is just a string that defines groups of Users.
  • ServiceAccount: It is normally used by a service or a robot that uses the K8s API, a very common use case is to define a ServiceAccount for a pod, the pod then can use the API with the Role or ClusterRole bind to the ServiceAccount.
  • Rules: Is the definition of a permission grant, you define a number of Rules when creating or editing a Role, on the rules, you define which Resources and Resource Names you are granting access to as well as which actions, called Verbs you are allowing.
  • Resource: Is object within any apigroup. In practical terms, anything you can manage with the command line tool is a Resource. Some are namespaced like pods, and some are cluster wide like nodes. Run kubectl api-resources to get the full list of them.
  • Resource Names: You can also specify a particular Resource by its specific name, for example, within the secrets Resource, you may have defined a secret called hubot which you want to grant access to from a specific ServiceAccount. This is very useful to create fine grained access policies to our resources.
  • Verbs: Are the actions that you grant to each resource, for example use, list, get, etc.
  • Rolebindings: Once you have created a Role with Rules you have to bind it to a ServiceAccount, a User, or a Group, these 3 elements are considered the Subjects of a Rolebinding.
  • Clusterrolebindings: Same than the above but for ClusterRoles.

Here is an example of a role description, note that it is namespaced:

$ kubectl get roles hubot-role -o yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  creationTimestamp: 2018-11-12T14:45:01Z
  name: hubot-role
  namespace: kube-system
  resourceVersion: "14541453"
  selfLink: /apis/rbac.authorization.k8s.io/v1/namespaces/kube-system/roles/hubot-role
  uid: 88de900d-e689-11e8-8701-124bd0c62782
rules:
- apiGroups:
  - ""
  resourceNames:
  - hubot
  resources:
  - secrets
  verbs:
  - get
  - watch
  - list
  - use

Debugging role bindings

In Kubernetes you can bind roles to Users or to ServiceAccounts. There is a superb command called kubectl auth can-i that allows you to check whether your user, or any other specific user, can access certain resources to do certain actions. This command is basically simulating the API call so that the K8s API checks if we have permissions to perform the action or not. It is like a “dry run” command. Users are a great way to auth and access Kubernetes but for internal pods we mainly use ServiceAccounts and you are not able to impersonate those with the can-i command. This means that your options are to carefully examine the policy rules of a role or to test the API resources access from the targeted pod itself. This can be done by executing commands from the pods using the kubectl exec command, extremely useful for debugging purposes.

There is no way of get the Roles attached to a ServiceAccount by describing it. There is also no way of getting which ServiceAccounts are linked to a Role by describing the Role. The only way is to describe the Rolebindings, for example:

kubectl get rolebindings hubot-role-binding -o yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  creationTimestamp: 2018-11-12T15:18:54Z
  name: hubot-role-binding
  namespace: kube-system
  resourceVersion: "14538328"
  selfLink: /apis/rbac.authorization.k8s.io/v1/namespaces/kube-system/rolebindings/hubot-role-binding
  uid: 44bf54cc-e68e-11e8-8701-124bd0c62782
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: hubot-role
subjects:
- kind: ServiceAccount
  name: hubot-sa
  namespace: kube-system

It is very important to name your Rolebindings consistently as otherwise you’ll have to loop over all of them to find their related Roles and ServiceAccounts or Users. It was somehow frustrating to be forced to discover the bindings this way, but hey ho, that’s how it is and let’s move forward, a little bit of fixed naming convention will solve your issues.

GCP and AWS IAM integration

There is a lot of confusion around this topic. Both AWS and GCP use IAM (Identity and Access Management) Roles and policies to secure their own resources, the terminology is very close to the Kubernetes RBAC terminology, in the case of GCP it is even more confusing because GCP also has the concept of ServiceAccount in their own IAM solution. They are not making an extensive effort in making a clear distinction between K8s and cloud native terminology, which leads to misunderstanding. Let’s get some things clear:

  • AWS and GCP do not integrate directly with RBAC.
  • IAM Roles and ServiceAccounts have nothing to do with RBAC Roles and ServiceAccounts
  • What integrates with AWS and GCP is the kubelet daemon running on physical or virtual nodes, it does integrate by using authorisation plugins that use AWS or GCP credentials to manage native cloud resources such as load balancers or security groups.

So what we do in RBAC is to give permissions to certain Users, Groups, and ServiceAccounts to do certain actions that will result in calling cloud native resources. We then map such Users, Groups, and ServiceAccounts to specific Cloud provided IAM roles that have the necessary permissions to create, delete, use, etc, those native cloud resources.

  • Who controls this?

The cloud authenticator plugin does, and each one has its own rules, it is slightly more difficult in AWS, please check the plugin integration (page)[https://cloud.google.com/kubernetes-engine/docs/concepts/access-control] for details. Here is an example on a configmap that maps Kubernetes Groups and Users with AWS roles:

apiVersion: v1
kind: ConfigMap
metadata:
  name: aws-auth
  namespace: kube-system
data:
  mapRoles: |
    - rolearn: arn:aws:iam::XXXXXXXXXXXX:role/mayara-eks-services201808221217
      username: system:node:example_ec2_private_dns
      groups:
        - system:bootstrappers
        - system:nodes
    - rolearn: arn:aws:iam::XXXXXXXXXXX:role/mayara-eks-services-admin
      username: admin_role
      groups:
        - system:masters

  mapUsers: |

  mapAccounts: |

In the case of GCP, integration is a bit easier, you just have to reference as subjects the Google users (or Service Accounts) in the RBAC configuration directly:

kind: Rolebinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: example_binding
  namespace: default
subjects:
- kind: User
  name: example@mayara.io
  apiGroup: rbac.authorization.k8s.io

In both cases, AWS and GCP, we need to authenticate externally to K8s in order to use those mapped Groups and Users, kubelet will get the credentials automatically from AWS or GCP metadata API if it is running on a cloud instance.

  • So, apart from the necessary AWS or GCP permissions what permissions do I need to grant to a ServiceAccount to use cloud native resources?

Here is an example of RBAC permissions that an Ingress controller will need, in our case, we are creating a role for the ingress controller and we are binding it to a K8s ServiceAccount called ingress, in order to use this, we will have to set the ingress controller deployment to use the ingress ServiceAccount:

apiVersion: v1
kind: ServiceAccount
metadata:
  name: ingress
  namespace: kube-system
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRole
metadata:
  name: ingress-role
rules:
- apiGroups: [""]
  resources: ["secrets", "configmaps", "services", "endpoints"]
  verbs:
    - get
    - watch
    - list
    - proxy
    - use
    - redirect
    - create
    - update
- apiGroups: [""]
  resources: ["pods"]
  verbs:
    - list
- apiGroups: [""]
  resources: ["events"]
  verbs:
    - redirect
    - patch
    - post
    - create
    - update
- apiGroups:
    - "extensions"
  resources:
    - "ingresses"
    - "ingresses/status"
  verbs:
    - get
    - watch
    - list
    - proxy
    - use
    - redirect
    - update
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
  name: ingress-role
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: ingress-role
subjects:
- kind: ServiceAccount
  name: ingress
  namespace: kube-system

Principle of least privilege

Not actually in the scope of this post, but it is always a good thing to repeat this mantra over and over, just allow what you need, nothing else. Each pod should only be able to speak with a number of services, everything else has to be restricted, that is the point of RBAC. We update the rules at runtime, so opening things of the fly is a better option that open too much from the beginning.

What is it there by default?

Since K8s 1.6 RBAC itself is in the core (not a plugin anymore). There are a number of default roles created, they are all prefixed by system: so you can find them by running kubectl get roles|grep system and kubectl get clusterroles|grep system. They authorise all User and ServiceAccount calls to speak with the API, you can disable unauthorised calls by adding anonymous-auth=false to your kubelet configuration.

References

https://kubernetes.io/docs/reference/access-authn-authz/rbac/#default-roles-and-role-bindings

https://cloud.google.com/kubernetes-engine/docs/how-to/role-based-access-control

https://github.com/kubernetes-sigs/aws-iam-authenticator

https://cloud.google.com/kubernetes-engine/docs/concepts/access-control

11/11/18 Learning RBAC The Hard Way, by Dario Ferrer

comments powered by Disqus