Learning RBAC The Hard Way
11/11/18, by Dario Ferrer
Let’s face it, you’re here because a search engine showed you some interesting code or because you saw a preview in a listing. So I’ll go straight to the point. RBAC in Kubernetes is still unknown to many and it is difficult to debug and get it running. We implemented a very simple RBAC solution for our Hubot pod, here is how it looks:
You may like this if you’re not new to K8s but more or less experienced. You know what is Kubernetes and you are already managing one or more clusters, maybe already in prod, you know that you need to secure it (also) from within and you know that RBAC is a pending subject that needs to be addressed sooner or later.
Brief overview of terminology
- Api Object: All the following definitions can be considered RBAC API objects, and the RBAC API itself is considered a K8s apigroup.
- Role: A K8s role is a group of rules that can be bind to a User or a ServiceAccount. The rules defined in the Role grant permissions, there are no deny ones. Roles are scoped to a Namespace.
- ClusterRole: Exactly the same thing than the Role but they are cluster wide, so not scoped to a Namespace.
- User: A User represents an external service who uses the K8s API, Kubernetes auth maps external auth services such as Google accounts or AWS users with K8s Users.
- Group: It is just a string that defines groups of Users.
- ServiceAccount: It is normally used by a service or a robot that uses the K8s API, a very common use case is to define a ServiceAccount for a pod, the pod then can use the API with the Role or ClusterRole bind to the ServiceAccount.
- Rules: Is the definition of a permission grant, you define a number of Rules when creating or editing a Role, on the rules, you define which Resources and Resource Names you are granting access to as well as which actions, called Verbs you are allowing.
- Resource: Is object within any apigroup. In practical terms, anything you
can manage with the command line tool is a Resource. Some are namespaced
like pods, and some are cluster wide like nodes. Run
kubectl api-resourcesto get the full list of them.
- Resource Names: You can also specify a particular Resource by its specific
name, for example, within the
secretsResource, you may have defined a secret called
hubotwhich you want to grant access to from a specific ServiceAccount. This is very useful to create fine grained access policies to our resources.
- Verbs: Are the actions that you grant to each resource, for example
- Rolebindings: Once you have created a Role with Rules you have to bind it to a ServiceAccount, a User, or a Group, these 3 elements are considered the Subjects of a Rolebinding.
- Clusterrolebindings: Same than the above but for ClusterRoles.
Here is an example of a role description, note that it is namespaced:
$ kubectl get roles hubot-role -o yaml apiVersion: rbac.authorization.k8s.io/v1 kind: Role metadata: creationTimestamp: 2018-11-12T14:45:01Z name: hubot-role namespace: kube-system resourceVersion: "14541453" selfLink: /apis/rbac.authorization.k8s.io/v1/namespaces/kube-system/roles/hubot-role uid: 88de900d-e689-11e8-8701-124bd0c62782 rules: - apiGroups: - "" resourceNames: - hubot resources: - secrets verbs: - get - watch - list - use
Debugging role bindings
In Kubernetes you can bind roles to Users or to ServiceAccounts. There is a
superb command called
kubectl auth can-i that allows you to check whether your
user, or any other specific user, can access certain resources to do certain
actions. This command is basically simulating the API call so that the K8s API
checks if we have permissions to perform the action or not. It is like a “dry
run” command. Users are a great way to auth and access Kubernetes but for
internal pods we mainly use ServiceAccounts and you are not able to
impersonate those with the
can-i command. This means that your options are to
carefully examine the policy rules of a role or to test the API resources access
from the targeted pod itself. This can be done by executing commands from the
pods using the
kubectl exec command, extremely useful for debugging purposes.
There is no way of get the Roles attached to a ServiceAccount by describing it. There is also no way of getting which ServiceAccounts are linked to a Role by describing the Role. The only way is to describe the Rolebindings, for example:
kubectl get rolebindings hubot-role-binding -o yaml apiVersion: rbac.authorization.k8s.io/v1 kind: RoleBinding metadata: creationTimestamp: 2018-11-12T15:18:54Z name: hubot-role-binding namespace: kube-system resourceVersion: "14538328" selfLink: /apis/rbac.authorization.k8s.io/v1/namespaces/kube-system/rolebindings/hubot-role-binding uid: 44bf54cc-e68e-11e8-8701-124bd0c62782 roleRef: apiGroup: rbac.authorization.k8s.io kind: Role name: hubot-role subjects: - kind: ServiceAccount name: hubot-sa namespace: kube-system
It is very important to name your Rolebindings consistently as otherwise you’ll have to loop over all of them to find their related Roles and ServiceAccounts or Users. It was somehow frustrating to be forced to discover the bindings this way, but hey ho, that’s how it is and let’s move forward, a little bit of fixed naming convention will solve your issues.
GCP and AWS IAM integration
There is a lot of confusion around this topic. Both AWS and GCP use IAM (Identity and Access Management) Roles and policies to secure their own resources, the terminology is very close to the Kubernetes RBAC terminology, in the case of GCP it is even more confusing because GCP also has the concept of ServiceAccount in their own IAM solution. They are not making an extensive effort in making a clear distinction between K8s and cloud native terminology, which leads to misunderstanding. Let’s get some things clear: - AWS and GCP do not integrate directly with RBAC. - IAM Roles and ServiceAccounts have nothing to do with RBAC Roles and ServiceAccounts - What integrates with AWS and GCP is the kubelet daemon running on physical or virtual nodes, it does integrate by using authorisation plugins that use AWS or GCP credentials to manage native cloud resources such as load balancers or security groups.
So what we do in RBAC is to give permissions to certain Users, Groups, and ServiceAccounts to do certain actions that will result in calling cloud native resources. We then map such Users, Groups, and ServiceAccounts to specific Cloud provided IAM roles that have the necessary permissions to create, delete, use, etc, those native cloud resources.
- Who controls this?
The cloud authenticator plugin does, and each one has its own rules, it is slightly more difficult in AWS, please check the plugin integration (page)[https://cloud.google.com/kubernetes-engine/docs/concepts/access-control] for details. Here is an example on a configmap that maps Kubernetes Groups and Users with AWS roles:
apiVersion: v1 kind: ConfigMap metadata: name: aws-auth namespace: kube-system data: mapRoles: | - rolearn: arn:aws:iam::XXXXXXXXXXXX:role/mayara-eks-services201808221217 username: system:node:example_ec2_private_dns groups: - system:bootstrappers - system:nodes - rolearn: arn:aws:iam::XXXXXXXXXXX:role/mayara-eks-services-admin username: admin_role groups: - system:masters mapUsers: | mapAccounts: |
In the case of GCP, integration is a bit easier, you just have to reference as subjects the Google users (or Service Accounts) in the RBAC configuration directly:
kind: Rolebinding apiVersion: rbac.authorization.k8s.io/v1 metadata: name: example_binding namespace: default subjects: - kind: User name: firstname.lastname@example.org apiGroup: rbac.authorization.k8s.io
In both cases, AWS and GCP, we need to authenticate externally to K8s in order to use those mapped Groups and Users, kubelet will get the credentials automatically from AWS or GCP metadata API if it is running on a cloud instance.
- So, apart from the necessary AWS or GCP permissions what permissions do I need to grant to a ServiceAccount to use cloud native resources?
Here is an example of RBAC permissions that an Ingress controller will need, in
our case, we are creating a role for the ingress controller and we are binding
it to a K8s ServiceAccount called
ingress, in order to use this, we will
have to set the ingress controller deployment to use the
apiVersion: v1 kind: ServiceAccount metadata: name: ingress namespace: kube-system --- apiVersion: rbac.authorization.k8s.io/v1beta1 kind: ClusterRole metadata: name: ingress-role rules: - apiGroups: [""] resources: ["secrets", "configmaps", "services", "endpoints"] verbs: - get - watch - list - proxy - use - redirect - create - update - apiGroups: [""] resources: ["pods"] verbs: - list - apiGroups: [""] resources: ["events"] verbs: - redirect - patch - post - create - update - apiGroups: - "extensions" resources: - "ingresses" - "ingresses/status" verbs: - get - watch - list - proxy - use - redirect - update --- apiVersion: rbac.authorization.k8s.io/v1beta1 kind: ClusterRoleBinding metadata: name: ingress-role roleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: ingress-role subjects: - kind: ServiceAccount name: ingress namespace: kube-system
Principle of least privilege
Not actually in the scope of this post, but it is always a good thing to repeat this mantra over and over, just allow what you need, nothing else. Each pod should only be able to speak with a number of services, everything else has to be restricted, that is the point of RBAC. We update the rules at runtime, so opening things of the fly is a better option that open too much from the beginning.
What is it there by default?
Since K8s 1.6 RBAC itself is in the core (not a plugin anymore). There are a
number of default roles created, they are all prefixed by
system: so you can
find them by running
kubectl get roles|grep system and
clusterroles|grep system. They authorise all User and ServiceAccount calls
to speak with the API, you can disable unauthorised calls by adding
anonymous-auth=false to your kubelet configuration.
11/11/18 Learning RBAC The Hard Way, by Dario Ferrercomments powered by Disqus