IAM Roles for Service Accounts

AWS introduced IAM roles for service accounts feature in 2019. The blogs from AWS have detailed explanation for this feature:

Although the official AWS documentation is quite comprehensive, there are a few specific points that I would like to emphasize.

OIDC Provider

OpenID Connect or OIDC is an identity protocol that utilizes the authorization and authentication mechanisms of OAuth 2.0.

  • The OIDC provider (generally called the OpenID Provider or Identity Provider or IdP) performs user authentication, user consent, and token issuance.
  • The client or service requesting a user’s identity is normally called the Relying Party (RP).
    • It can be, for example, a web application, but also a JavaScript application or a mobile app.
  • The Token Endpoint is an OIDC Token endpoint
    • To obtain an Access Token, an ID Token, and optionally a Refresh Token, the RP (Client) sends a Token Request to the Token Endpoint to obtain a Token Response, as described in Section 3.2 of OAuth 2.0 [RFC6749], when using the Authorization Code Flow.
  • The UserInfo Endpoint is an OIDC UserInfo endpoint.
    • It responds with user attribute when service providers present access tokens that your Token endpoint issued.

Being built on top of OAuth 2.0, OpenID Connect uses tokens to provide a simple identity layer integrated with the underlying authorization framework. This integration implies the use of the following types of token:

  • ID Token: Specific to OIDC, the primary use of this token in JWT format is to provide information about the authentication operation’s outcome. Upon request, it may provide the identity data describing a user profile. The data about the authentication result and the user profile information are called claims. The user profile claims may be any data that is pertinent to the Relying Party for identification purposes, such as a persistent ID, email address, name, etc.
  • Access Token: Defined in OAuth2, this (optional) short lifetime token provides access to specific user resources as defined in the scope values in the request to the authorization server.
  • Refresh Token: Coming from OAuth2 specs, this token is usually long-lived and may be used to obtain new access tokens.

EKS

Create Pod using Kubernetes ServiceAccount

  1. Create an IAM role
  2. Create a Kubernetes service account
  3. Annotate the service account with aws role arn
1
2
3
4
5
6
7

apiVersion: v1
kind: ServiceAccount
metadata:
  annotations:
    eks.amazonaws.com/role-arn: arn:aws:iam::123456789012:role/test-role-for-service-account
  name: test-service-account
  namespace: default
  1. Create Pod using the Kubernetes service account
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16

apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    run: myapp
  name: myapp
spec:
  ...
  template:
    ...
    spec:
      serviceAccountName: test-service-account
      containers:
      - image: myapp:1.2
        name: myapp
        ...
  1. The Amazon EKS Pod Identity Webhook would inject environment variables and project volume aws-iam-token to the Pod.
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27

apiVersion: apps/v1
kind: Pod
metadata:
  name: myapp
spec:
  serviceAccountName: test-service-account
  containers:
  - name: myapp
    image: myapp:1.2
    env:
    - name: AWS_ROLE_ARN
      value: arn:aws:iam::123456789012:role/test-role-for-service-account
    - name: AWS_WEB_IDENTITY_TOKEN_FILE
      value: /var/run/secrets/eks.amazonaws.com/serviceaccount/token
    volumeMounts:
    - mountPath: /var/run/secrets/eks.amazonaws.com/serviceaccount
        name: aws-iam-token
        readOnly: true
  volumes:
  - name: aws-iam-token
    projected:
      defaultMode: 420
      sources:
      - serviceAccountToken:
          audience: sts.amazonaws.com
          expirationSeconds: 86400
          path: token

Question:

  1. Who issues the JWT token that injected to Pod via projected volume?

The IAM OIDC Provider

  1. What if the JWT token expires? Who will refresh the JWT token?

According to Kubernetes Official Document, The kubelet will:

  1. request and store the token on behalf of the Pod;
  2. make the token available to the Pod at a configurable file path; and refresh the token as it approaches expiration.
  3. The kubelet proactively requests rotation for the token if it is older than 80% of its total time-to-live (TTL), or if the token is older than 24 hours.

The application is responsible for reloading the token when it rotates. It’s often good enough for the application to load the token on a schedule (for example: once every 5 minutes), without tracking the actual expiry time.

Pod Access AWS Resources

  1. The Pod application wants to access AWS S3
  2. According to AWS Default Credential Provider Chain, Pod find Web Identity Token credentials from the environment.
  3. The WebIdentityTokenCredentialsProvider using action sts:AssumeRoleWithWebIdentity to get temporary credential
    • the provider want to assume role arn:aws:iam::123456789012:role/test-role-for-service-account
    • the provider has a JWT token which injected to Pod
  4. In some means, AWS STS checked with IAM and let the WebIdentityTokenCredentialsProvider assume role and return the temperory credential
  5. The Pod application now use the temporary credential to access S3

Question:

  1. What setting should we make on the role to allow sts:AssumeRoleWithWebIdentity ?

According to AWS IAM, A web identity session principal is a session principal that results from using the AWS STS AssumeRoleWithWebIdentity operation.

The trust relationship below would be attached to the AWS role arn:aws:iam::123456789012:role/test-role-for-service-account, which means the role trust the principal from AssumeRoleWithWebIdentity by the OIDC provider. In other words, the service account test-service-account could assume the role arn:aws:iam::123456789012:role/test-role-for-service-account

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14

{
   "Sid": "",
   "Effect": "Allow",
   "Principal": {
       "Federated": "arn:aws:iam::001797876521:oidc-provider/oidc.eks.us-west-2.amazonaws.com/id/55495BBBDD281DF88468ADFCFD2B3C2F"
   },
   "Action": "sts:AssumeRoleWithWebIdentity",
   "Condition": {
       "StringEquals": {
           "oidc.eks.us-west-2.amazonaws.com/id/55495BBBDD281DF88468ADFCFD2B3C2F:aud": "sts.amazonaws.com",
           "oidc.eks.us-west-2.amazonaws.com/id/55495BBBDD281DF88468ADFCFD2B3C2F:sub": "system:serviceaccount:default:test-service-account"
       }
   }
}

You can decode the IAM_TOKEN which is projected to the Pod

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29

$ IAM_TOKEN=$(kubectl exec -it example-pod -- cat /var/run/secrets/eks.amazonaws.com/serviceaccount/token)
$ jwt decode $IAM_TOKEN --json --iso8601
{
  "header": {
    "alg": "RS256",
    "kid": "6ef1fb2c5b3683248f37f991d8cd573e2a7be2bb"
  },
  "payload": {
    "aud": [
      "sts.amazonaws.com"
    ],
    "exp": "2023-03-28T09:06:15+00:00",
    "iat": "2023-03-27T09:06:15+00:00",
    "iss": "https://oidc.eks.us-west-2.amazonaws.com/id/55495BBBDD281DF88468ADFCFD2B3C2F",
    "kubernetes.io": {
      "namespace": "default",
      "pod": {
        "name": "example-pod",
        "uid": "1d38fb3d-83d4-46d2-ba33-57866ebf8a14"
      },
      "serviceaccount": {
        "name": "test-service-account",
        "uid": "ed0284be-f0ed-44b6-a53e-e708db226207"
      }
   },
    "nbf": "2023-03-27T09:06:15+00:00",
    "sub": "system:serviceaccount:default:test-service-account"
  }
}

See also: Creating a role for web identity or OpenID Connect Federation.

  1. Why should IAM trust the JWT token provided by the provider when it want to assume the role using sts:AssumeRoleWithWebIdentity?

In order to accept the token from a pod, there has to be a trust established between IAM and the OIDC provider. This is done via the eksctl tool or from AWS Management Console

After creating the OIDC identity provider, AWS IAM could authenticate AWS API calls with supported identity providers after receiving a valid OIDC JWT. This token can then be passed to AWS STS AssumeRoleWithWebIdentity API operation to get temporary IAM credentials.

TODO: check this

It is the same as using aws iam create-open-id-connect-provider and aws eks associate-identity-provider-config

1

$ aws iam create-open-id-connect-provider --url https://oidc.eks.us-west-2.amazonaws.com/id/55495BBBDD281DF88468ADFCFD2B3C2F ...
  1. How to verify JWT from the OIDC identity provider?

The OIDC JWT token we have in our Kubernetes workload is cryptographically signed, and IAM should trust and validate these tokens before the AWS STS AssumeRoleWithWebIdentity API operation can send the temporary credentials. As part of the Service Account Issuer Discovery feature of Kubernetes, EKS is hosting a public OpenID provider configuration document (Discovery endpoint) and the public keys to validate the token signature (JSON Web Key Sets – JWKS) at https://OIDC_PROVIDER_URL/.well-known/openid-configuration.

Reference