Permission Denied: Reading RBAC Errors and Fixing ServiceAccount Permissions

You deploy a pod. It starts cleanly. Then it fails, with something like this in the logs:

Error from server (Forbidden): pods is forbidden: User "system:serviceaccount:staging:my-app"
cannot list resource "pods" in API group "" in the namespace "staging"

The image is correct. The pod is running. But something it tries to do against the Kubernetes API is not permitted.

This is a Forbidden error — an RBAC authorization failure. The frustrating thing is that the message already contains the full diagnosis. The identity, the denied action, the resource type, the namespace — it is all there. The problem is knowing how to read it.

The anatomy of a Forbidden error

When the API server denies a request, it returns a consistent 403 Forbidden message. Unpacking the example above:

User "system:serviceaccount:staging:my-app"
cannot list resource "pods"
in API group ""
in the namespace "staging"

Each field maps to something you will need in the fix:

Field	What it means
`system:serviceaccount:staging:my-app`	The identity. Namespace `staging`, ServiceAccount named `my-app`
`cannot list`	The denied verb: `get`, `list`, `watch`, `create`, `update`, `patch`, `delete`
`resource "pods"`	The Kubernetes resource type
`API group ""`	Empty string = core API group (`v1`). Others: `apps`, `batch`, `rbac.authorization.k8s.io`
`in the namespace "staging"`	Namespace-scoped. Cluster-scoped resources say "at the cluster scope" instead

All five fields are present in every Forbidden message. You need all five to write an effective RBAC rule.

Cluster-scoped variant. For nodes, namespaces, PersistentVolumes, and other cluster-scoped resources, the message looks like:

User "system:serviceaccount:monitoring:prometheus"
cannot list resource "nodes"
in API group ""
at the cluster scope

The phrase "at the cluster scope" tells you the fix requires a ClusterRole and ClusterRoleBinding, not a namespace-scoped Role.

Step 1: Confirm the ServiceAccount

Before writing any YAML, verify which ServiceAccount the pod is actually using. It may not be what you expect.

kubectl get pod <pod-name> -n <namespace> -o jsonpath='{.spec.serviceAccountName}'

If the output is empty or default, the pod is using the namespace default ServiceAccount. This is the single most common root cause: the application was written to expect certain permissions, the deployment never specifies a serviceAccountName, and the default ServiceAccount has no useful permissions bound to it.

Verify the ServiceAccount exists:

kubectl get serviceaccount -n <namespace>

If you need a dedicated ServiceAccount:

kubectl create serviceaccount my-app -n staging

Then set serviceAccountName: my-app in the pod spec. Adding permissions to the default ServiceAccount grants them to every pod in the namespace that does not specify one — almost never the right outcome in a production cluster.

Step 2: Check what the identity can already do

Do not write new permissions before auditing what is already granted. The kubectl auth can-i command tests authorization directly, impersonating the ServiceAccount:

kubectl auth can-i list pods \
  --as=system:serviceaccount:staging:my-app \
  --namespace=staging

Output is yes or no. For a full picture:

kubectl auth can-i --list \
  --as=system:serviceaccount:staging:my-app \
  --namespace=staging

This lists every verb/resource/API group combination the identity is currently allowed. Scan it before making changes — the permission may already exist under a different binding you were not aware of.

For cluster-scoped resources, drop the namespace flag:

kubectl auth can-i list nodes \
  --as=system:serviceaccount:monitoring:prometheus

Note

--as requires your own account to have impersonation privileges. If the can-i command itself returns a permission error, you need elevated access before you can proceed.

Step 3: Find existing RBAC bindings

Before creating new RBAC objects, check what already exists for this ServiceAccount:

kubectl get rolebindings -n staging -o wide

Look for the ServiceAccount in the SUBJECTS column. For ClusterRoleBindings:

kubectl get clusterrolebindings -o wide | grep my-app

Inspect a specific binding:

kubectl describe rolebinding <binding-name> -n staging

The Rules section of the referenced Role shows verbs, resources, and API groups. Cross-reference this with the error message to confirm exactly what is missing.

Step 4: Grant the missing permission

With the identity, verb, resource, API group, and scope confirmed, you can write the fix.

Option A: Role and RoleBinding (namespaced resources)

Use this for pods, services, configmaps, secrets, deployments, and any other namespace-scoped resource.

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: my-app-pod-reader
  namespace: staging
rules:
- apiGroups: [""]
  resources: ["pods"]
  verbs: ["get", "list", "watch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: my-app-pod-reader-binding
  namespace: staging
subjects:
- kind: ServiceAccount
  name: my-app
  namespace: staging
roleRef:
  kind: Role
  name: my-app-pod-reader
  apiGroup: rbac.authorization.k8s.io

Option B: ClusterRole and ClusterRoleBinding (cluster-scoped resources)

Use this for nodes, namespaces, PersistentVolumes, or when the ServiceAccount needs the same permissions across all namespaces.

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: prometheus-node-reader
rules:
- apiGroups: [""]
  resources: ["nodes", "nodes/metrics"]
  verbs: ["get", "list", "watch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: prometheus-node-reader-binding
subjects:
- kind: ServiceAccount
  name: prometheus
  namespace: monitoring
roleRef:
  kind: ClusterRole
  name: prometheus-node-reader
  apiGroup: rbac.authorization.k8s.io

Option C: Bind a built-in ClusterRole with a RoleBinding

Kubernetes ships built-in ClusterRoles for common use cases. You can scope them to a single namespace by using a RoleBinding instead of a ClusterRoleBinding:

apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: my-app-view
  namespace: staging
subjects:
- kind: ServiceAccount
  name: my-app
  namespace: staging
roleRef:
  kind: ClusterRole    # ClusterRole, referenced by a RoleBinding
  name: view
  apiGroup: rbac.authorization.k8s.io

Useful built-ins: view (read-only to most namespaced resources), edit (read/write, no RBAC), admin (full namespace admin, no cluster resources).

Step 5: Verify

After applying the RBAC objects, re-run the check:

kubectl auth can-i list pods \
  --as=system:serviceaccount:staging:my-app \
  --namespace=staging

Should return yes. For long-running controllers and operators, the change takes effect immediately — the API server evaluates RBAC on each request. If your application uses retry backoff, the next attempt will succeed without a restart.

Common scenarios

Helm charts with no RBAC templates

Many Helm charts install with a ServiceAccount but do not include a Role or RoleBinding. The chart assumes you will supply RBAC externally, or it worked in a permissive development cluster and was never tested with locked-down access. Check the chart's templates/ directory for role.yaml and rolebinding.yaml. If they are absent, you need to add them yourself, separate from the chart.

Init containers failing with Forbidden

Init containers use the pod's ServiceAccount, the same as the main container. If your init container reads a ConfigMap, calls the API, or checks a resource before the main container starts, it needs permissions too. The pod will show Init:Error and the init container logs will contain the Forbidden message.

Operators watching across namespaces

Operators typically need cluster-wide watch permissions for the resource types they manage. If the error message says "at the cluster scope" or the same error appears across multiple namespaces, a namespace-scoped Role will not solve it. Use a ClusterRole and ClusterRoleBinding.

API group confusion

Getting the apiGroups value wrong produces a rule that looks correct but grants nothing. The API group in the error message is the one to use in the rule:

Resource	apiGroups value
pods, services, configmaps, secrets, serviceaccounts	`""` (empty string)
deployments, replicasets, daemonsets, statefulsets	`apps`
jobs, cronjobs	`batch`
ingresses	`networking.k8s.io`
roles, rolebindings, clusterroles, clusterrolebindings	`rbac.authorization.k8s.io`
custom resources	group defined by the CRD (e.g. `monitoring.coreos.com`)

For custom resources, check the CRD:

kubectl get crd <name> -o jsonpath='{.spec.group}'

Dynamic and generated RBAC

Modern clusters increasingly have RBAC that nobody wrote by hand. Three patterns cause recurring confusion.

Operators that generate RBAC objects. Many operators — cert-manager, the Prometheus Operator, Flux, Argo CD — create Roles and RoleBindings dynamically as part of their reconciliation loop. The operator's own ServiceAccount needs create, update, and delete permissions on roles and rolebindings (or clusterroles and clusterrolebindings) to do this. When it does not, the Forbidden error is emitted by the operator, not by the workload it manages — which makes the error message confusing. The subject in the error is the operator's ServiceAccount. The fix is to extend the operator's own RBAC, not the managed workload's.

Helm charts with RBAC toggled off. Most Helm charts include a rbac.create value that defaults to true. When set to false — common in environments where platform teams manage RBAC separately — the chart skips its Role and RoleBinding templates entirely. The application deploys, starts cleanly, and fails at runtime when it first calls the API. Check the deployed values before looking elsewhere:

helm get values <release-name> -n <namespace>

If rbac.create: false is set, either flip it and upgrade the release, or create the RBAC objects manually to match what the chart would have generated. The chart's templates/ directory shows exactly what Role and RoleBinding it would produce.

Aggregated ClusterRoles. Kubernetes maintains three aggregated ClusterRoles — admin, edit, and view — whose rules are assembled at runtime from other ClusterRoles bearing matching labels. If your cluster runs custom resources (CRDs) and a controller or user is bound to view but cannot access those resources, the fix is not to modify view directly. Instead, create a new ClusterRole with the appropriate aggregation label and the rules you need:

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: my-crd-view
  labels:
    rbac.authorization.k8s.io/aggregate-to-view: "true"
rules:
- apiGroups: ["mygroup.example.com"]
  resources: ["myresources"]
  verbs: ["get", "list", "watch"]

Kubernetes merges this into view automatically. Any ServiceAccount bound to view picks up the new rules without a binding change. The same pattern works for edit and admin using their respective labels.

A production debugging session

Theory is cleaner than production. Here is a real scenario.

The situation: A team upgrades their Prometheus Operator Helm release from 45.x to 58.x. Within minutes, the operator pod logs start showing Forbidden errors. Alerting stops working. No application code changed.

Step 1: Read the error from the operator logs.

$ kubectl logs -n monitoring deploy/prometheus-operator | grep -i forbidden
E0610 09:14:02 operator.go:412] Failed to reconcile PrometheusRule:
  prometheusrules.monitoring.coreos.com is forbidden:
  User "system:serviceaccount:monitoring:prometheus-operator"
  cannot list resource "prometheusrules" in API group "monitoring.coreos.com"
  at the cluster scope

The subject is system:serviceaccount:monitoring:prometheus-operator. The resource is a CRD (prometheusrules) in the monitoring.coreos.com group. The scope is cluster-wide.

Step 2: Check what changed during the upgrade.

$ helm get manifest prometheus-operator -n monitoring | grep -A 5 "kind: ClusterRole"

The new chart version added prometheusrules to the list of resources the operator needs to watch — but the ClusterRole was not updated because rbac.create was set to false in the team's values file. The old ClusterRole predated the new resource type.

Step 3: Check what the chart would generate.

$ helm template prometheus-operator prometheus-community/kube-prometheus-stack \
    -f values.yaml \
    --set rbac.create=true \
  | grep -A 30 "kind: ClusterRole"

This renders the ClusterRole the chart would have created. Compare it against the live one:

$ kubectl get clusterrole prometheus-operator -o yaml

The live ClusterRole is missing the prometheusrules and alertmanagerconfigs entries added in the new chart version.

Step 4: Patch the ClusterRole.

$ kubectl edit clusterrole prometheus-operator

Add the missing resource entries to the existing rules block. The operator reconciles immediately — no restart needed.

$ kubectl logs -n monitoring deploy/prometheus-operator | grep -i forbidden

No output. Alerting resumes within 30 seconds.

Root cause: The team managed RBAC outside Helm (rbac.create: false), which is a valid pattern, but had no process for syncing their hand-managed ClusterRole against new chart versions. The fix is to either re-enable rbac.create: true and let Helm manage the lifecycle, or maintain a diff check as part of the upgrade runbook.

Scoping permissions efficiently

Fixing a Forbidden error is straightforward. Keeping permissions tight over time requires a few habits.

Name roles after the workload, not the permission. my-app-role is better than pod-reader. When you audit bindings six months later, workload-named roles are immediately traceable. Generic names like read-pods accumulate bindings from multiple unrelated workloads and become difficult to reason about.

Keep roles in the same namespace as the workload. A namespace-scoped Role and RoleBinding that lives alongside the deployment it serves is self-contained. When the workload is deleted or the namespace is torn down, the RBAC objects go with it. ClusterRoles and ClusterRoleBindings persist until explicitly deleted and are easy to forget.

Audit what a ServiceAccount can do before and after changes. Before granting a new permission, run kubectl auth can-i --list --as=... to baseline what already exists. After applying the change, run it again and diff the output. This prevents accidental accumulation of permissions across multiple incidents.

Prefer read verbs; add write verbs only on evidence. Start with get, list, watch. If the application needs to write, it will produce a second Forbidden error naming the specific write verb. Grant that verb then. This iterative approach produces a role that reflects actual usage rather than assumptions.

Check subresources explicitly. Some operations require subresource permissions that are not implied by the parent resource. A ServiceAccount that can get pods cannot necessarily get pods/log or create pods/exec. These must be listed separately:

rules:
- apiGroups: [""]
  resources: ["pods", "pods/log"]
  verbs: ["get", "list"]

If an application can connect to pods but cannot stream logs or exec into them, a missing subresource entry is the likely cause.

Use resourceNames to lock down access to specific objects. When a ServiceAccount needs access to one specific Secret or ConfigMap — not all of them in the namespace — use the resourceNames field:

rules:
- apiGroups: [""]
  resources: ["secrets"]
  verbs: ["get"]
  resourceNames: ["my-app-tls"]

This grants get on my-app-tls only. Note: resourceNames cannot be used with list or watch — those verbs always operate at the collection level.

What not to do

Apply the principle of least privilege. Grant only the specific verbs and resources the application actually needs — nothing more. When you are unsure, start with read-only (get, list, watch) and add write verbs only when a concrete failure requires them. An over-permissioned ServiceAccount is a liability: if the workload is compromised, the attacker inherits everything bound to it.

Do not bind cluster-admin to fix a Forbidden error. It resolves the symptom and creates a much larger problem. The application will have unrestricted access to the entire cluster, and you will no longer know what permissions it actually needs.

Do not add wildcards. verbs: ["*"] and resources: ["*"] in a role are RBAC sprawl. Every permission the application does not need is an attack surface it should not have. Grant specific verbs for specific resources.

Do not edit system roles. ClusterRoles prefixed with system: are managed by the control plane. Modifications are silently overwritten during upgrades.

Summary

Forbidden errors in Kubernetes are verbose by design. The message is not an obstacle — it is the diagnosis. Identity, verb, resource, API group, scope: all five are present in every error.

The workflow is five steps:

Read the error message and extract all five fields
Confirm the pod's ServiceAccount: kubectl get pod ... -o jsonpath='{.spec.serviceAccountName}'
Check what is already granted: kubectl auth can-i --list --as=...
Grant only the specific missing permission via Role + RoleBinding or ClusterRole + ClusterRoleBinding
Verify with kubectl auth can-i before restarting the pod

The error message tells you exactly what to do. The only remaining question is how narrowly you scope the fix.