OpenStack Keystone Workflow & Token Scoping

While recently browsing the OpenStack documentation updates for the Folsom release, I came across a new (new to me anyway) Keystone diagram which provides a well deserved depiction of a typical end-user workflow using Keystone as an identity service provider. This diagram not only provides greater incite to this typical workflow, but it also illustrates the notion of scoped vs unscoped tokens. I've pasted the diagram below for convenience, but the original document can be found on the OpenStack documentation site.

Although this diagram paints a nice picture of a typical workflow, it leaves a bit to the imagination in terms of which APIs are used for each step. Moreover some of the steps are a bit misleading depending on which token type scheme you are using with Keystone.

This post aims to further solidify the steps in the workflow diagram above.

NOTE: This article was initially written in the later part of 2012 and thus is based on the state of OpenStack / Keystone at that point in time. Aspects of Keystone have changed since 2012, but this article's concepts should still apply. Enough folks found the article useful to maintain it -- hence it lives on here.

Step 1: Obtain an unscoped token from Keystone

To determine which tenants you have access to, you first need to use Keystone to obtain an unscoped token. By unscoped we mean the token is not tied to a particular tenant (soon to be called project) in the OpenStack Cloud. This unscoped token can be used to further query the Keystone service and determine which tenants you can access, but it should not be used with non-Keystone services such as the nova compute service -- only scoped tokens should be used with non-Keystone services.

To obtain an unscoped token, use the typical REST API for tokens, but do not specify a tenantName in the request body.

For example:

POST http://localhost:5000/v2.0/tokens

{
   "auth":{
      "passwordCredentials":{
         "username":"boden",
         "password":"my9password"
      }
   }
}

Upon successful authentication, your response should look something like that pasted below. The token id (hashbang path /access/token/id) is your unscoped token. This token id should be use as the value for the X-Auth-Token request header in subsequent requests to identity yourself.

{
   "access":{
      "token":{
         "expires":"2012-10-06T18:23:11Z",
         "id":"MIICDgYJKoZIhvcNAQcCoIr8JKYG0ywOaMc2lYqhIQhLApqJpOns="
      },
      "serviceCatalog":{
      },
      "user":{
         "username":"boden",
         "roles_links":[
         ],
         "id":"0e982248b8a14e4bb838f956b9b79e7a",
         "roles":[
         ],
         "name":"boden"
      }
   }
}

Step 2: Discover tenants you have access to

The next step is to use your unscoped token to determine which tenants you can access. Your tenancy is determined based on your role assignment -- for those tenants which you have an assigned role, you will have access to the tenant although your role may limit the operations you a perform for the given tenant in a given service endpoint. All operations (APIs) performed on a service endpoint should be performed with a scoped token.

To obtain a list of the tenants you have access to, use the GET /tenants Keystone API passing in your unscoped token in the X-Auth-Token header value.

For example:

GET http://localhost:5000/v2.0/tenants

HEADERS["X-Auth-Token":"MIICDgYJKoZIhvcNAQcCoIr8JKYG0ywOaMc2lYqhIQhLApqJpOns="]

The response from this call will include an array of tenants. For example the response shown below.

{
   "tenants":[
      {
         "description":"Default Tenant",
         "enabled":true,
         "id":"6f8945f2d47f4abea149b7a0176b12a8",
         "name":"DefaultTenant"
      },
      {
         "id":"d25825b1c6aa4fbb9bfa9c0aa35e3f55",
         "enabled":true,
         "description":"Something",
         "name":"NewTenant"
      }
   ],
   "tenants_links":[
   ]
}

Based on the list of tenants you have access to, determine which specific tenant you want to work with and obtain a scoped token for.

Step 3: Obtain a scoped token

Now that you have a list of tenants you can access and have determined which tenant you wish to work with, it's time to obtain a scoped token. A scoped token is a token which is tied to a specific tenant and provides metadata about that tenant and your roles within it. Only scoped tokens should be used when interfacing with a non-Keystone OpenStack service.

To obtain a scoped token, use the POST /tokens Keystone API as in step 1. There are two forms of this API:

• Use the same request body as in step 1 passing in your user id and credentials, but this time specify a tenantName to scope the token.
• Use a request body which contains both your unscoped token and the tenant name to scope the token. This later form allows you obtain a scoped token without POSTing your credentials again.

Both forms of this API are shown below.

Scope a token by resending credentials

POST http://localhost:5000/v2.0/tokens

{
   "auth":{
      "tenantName":"DefaultTenant",
      "passwordCredentials":{
         "username":"boden",
         "password":"my9password"
      }
   }
}

Scope a token using your existing unscoped token

POST http://localhost:5000/v2.0/tokens

{

   "auth":{
      "tenantName":"DefaultTenant",
      "token":{
         "id":"MIICDgYJKoZIhvcNAQcCoIr8JKYG0ywOaMc2lYqhIQhLApqJpOns="
      }
   }
}

The response will include a new scoped token and associated metadata as shown in the example response below:

{
   "access":{
      "token":{
         "expires":"2012-10-06T18:41:34Z",
         "id":"+dRsTPGAvw4yPrk-F7-eqcycJg",
         "tenant":{
            "description":"Default Tenant",
            "enabled":true,
            "id":"6f8945f2d47f4abea149b7a0176b12a8",
            "name":"DefaultTenant"
         }
      },
      "serviceCatalog":[
         {
            "endpoints_links":[
            ],
            "endpoints":[
               {
                  "adminURL":"http://localhost:9292/v1",
                  "region":"RegionOne",
                  "publicURL":"http://localhost:9292/v1",
                  "internalURL":"http://localhost:9292/v1",
                  "id":"ef3aa115fa104a33914d1ba05a8a1195"
               }
            ],
            "type":"image",
            "name":"glance"
         },
         {
            "endpoints_links":[
            ],
            "endpoints":[
               {
                  "adminURL":"http://localhost:1337",
                  "region":"RegionOne",
                  "publicURL":"http://localhost:1337",
                  "internalURL":"http://localhost:1337",
                  "id":"ee855645535e422fb316ed3eb652d94c"
               }
            ],
            "type":"compute",
            "name":"local-node-js"
         },
         {
            "endpoints_links":[
            ],
            "endpoints":[
               {
                  "adminURL":"http://localhost:35357/v2.0",
                  "region":"RegionOne",
                  "publicURL":"http://localhost:5000/v2.0",
                  "internalURL":"http://localhost:5000/v2.0",
                  "id":"882ad7040d8e46ef9cd036b8f685bb33"
               }
            ],
            "type":"identity",
            "name":"keystone"
         }
      ],
      "user":{
         "username":"boden",
         "roles_links":[
         ],
         "id":"0e982248b8a14e4bb838f956b9b79e7a",
         "roles":[
            {
               "name":"member"
            }
         ],
         "name":"boden"
      },
      "metadata":{
         "is_admin":0,
         "roles":[
            "9eaeea6cc78345738d3d7e12f6a9012e"
         ]
      }
   }
}

Notice the token metadata in the response is now scoped to the DefaultTenant tenant which was specified in the API request. Moreover notice the response includes an array of service endpoints. These endpoints identity the services your token has access to based on the service/endpoint catalog the Keystone service manages. As a consumer of the API, you now need to look through these service endpoints to determine which service you wish to use.

Let's quickly recap the main points of the endpoint/service catalog maintained by Keystone.

• Keystone manages a number of service definitions in its service catalog where a service is defined by a particular type (for example compute, image, etc.), nameand description. The service type indicates the capabilities of the service and will ultimately determine which APIs are hosted for endpoints of the given service.
• Keystone also manages a number of endpoint definitions in its service catalog where an endpoint contains a set of base URLs as well as the region where the endpoint resides. Region is typically associated with geographic location of the OpenStack Cloud hosting the endpoint and many deployments contain a Cloud per Region.
• Each endpoint is associated with a single service -- the endpoint's associated service provides a indication of the type of service the endpoint provides.
• Endpoint URLs are "base URLs" -- they provide the URL prefix to access the endpoint APIs. That said, actual API requests for a given endpoint are constructed by taking the endpoint base URL and appending the API URI to it in order to form a full URL (i.e. <base_endpoint_url>/<openstack_api_uri>).

With that in mind, we can see the response above contains a service/endpoint for compute and identity. This indicates that my scoped token can access APIs for the respective compute and identity services using the respective base endpoint URL given in the response. For more details on setting up services and endpoints in Keystone, see the OpenStack documentation.

IMPORTANT Remember that you now need to use the scoped token id in subsequent API calls passing it in via the X-Auth-Token header.

Step 4: Invoke the target endpoint service API

You now have a scoped token and know the URL of the endpoint API you wish to invoke. The next step is to actually invoke the service endpoint itself for the API(s) you wish to use. The diagram shows the endpoint service in step 4 using Keystone to validate your token. While this is true if you are using UUID based tokens, it is not true if you are using PKI. Let's outline the main differences.

UUID

With UUID based tokens, your token ID will be a UUID -- a unique string used to identity the token you hold. In this scheme the Keystone service you obtained the token from maintains an index of token UUIDs to their respective metadata and validity. As this token ID does not contain embedded metadata, the endpoint service must invoke Keystone (passing along your UUID based token ID) to validate the given token is authenticated and valid. In response Keystone will return the metadata associated with the token (much like the metadata return in the response from step 3) including your roles, tenancy, etc. which can then be used internally by the service while processing your API request. The important thing to note here is that the endpoint service is calling into Keystone for each API request it handles to validate your UUID based token.

PKI

The PKI token scheme was introduced in OpenStack Folsom RC1 and removes the need for the endpoint service to call into Keystone for each request. The bullets below outline how PKI works in a nutshell.

• PKI (Public Key Infrastructure) is based on a public/private certificate pair using X.509 technology.
• Keystone holds both a public and private certificate.
• Anybody can get the public certificate from Keystone via REST API call, but the private certificate key is not exposed outside of the Keystone service itself.
• When using PKI, each of the endpoint services will ask Keystone (via REST API) for the public key the first time the service is invoked. The endpoint service will then save off the public key (caching it) for later use.
• When Keystone builds your token in PKI mode, it creates the token JSON object containing your token metadata, encrypts the JSON using the private key, and then creates a signature of the encrypted token using MD5. This encrypted/hashed token becomes your token ID which is returned upon successful authentication (step 3 response) and is then used in your X-Auth-Token header. Take note that in this case your token ID includes the metadata for your token and not just a UUID as in the case of the UUID token scheme.
• When an endpoint service is invoked in PKI mode, it will verify the token signature and decrypt the token using the public key it has. Note that this is enough for the service to confirm validity since only Keystone has the private key and hence only Keystone could have encrypted the token using the private key.
• Since the decrypted token includes all the token metadata, the service no longer needs to invoke Keystone to get the token metadata and can validate the token internally.

The important take away is that PKI is much performant than the UUID scheme since the endpoint service does not need to make a REST call into Keystone to validate the token on a per request basis. As such it can increase service performance and reduce network "chatter".

You can setup the token scheme in your keystone.conf by specifying the token_format as PKI and including the certificate paths required to carry out PKI by Keystone. The snippet below shows an example development configuration which uses the test certificate keys provided by the Keystone tests (used for ad-hoc testing). In a production environment your own certificate keys would be used.

[signing]

#token_format = UUID

certfile = /home/boden/workspaces/openstack/keystone/tests/signing/signing_cert.pem

keyfile = /home/boden/workspaces/openstack/keystone/tests/signing/private_key.pem

ca_certs = /home/boden/workspaces/openstack/keystone/tests/signing/cacert.pem

#key_size = 1024

#valid_days = 3650

#ca_password = None

token_format = PKI

Step 5: Validate role metadata

Step 5 is really about the endpoint service using you token's metadata to verify you can access the requested service/operation. This is often referred to as Role Base Access Control (RBAC) and involves the service using a 'rule engine' to determine you token contains the proper role access based on the service's respective policy.json file. Although RBAC and concepts related to the policy.json file are beyond the scope of this post, it's worth noting that each endpoint service contains its own policy which is enforced on a per service basis -- that is, the internals of each service validate RBAC. For more details on how RBAC works, there is a good blog post on knowledgestack. You can also check the OpenStack docs which document the contents of the policy.json file on for each service type.

If you do not have the proper access to the operation you have requested, the service will deny your request and return an error.

Step 6: Service API request

In step 6, the endpoint service actually performs the operation you have requested as per the API URI. This might be a 'run instance', 'create volume', etc. depending on the API URI and service you are invoking. In the scheme of this post, there is nothing all that interesting to describe for this step.

Step 7: Return response

As you have probably guessed, step 7 returns an API response to the user. Again nothing interesting to dive into for this step in the context of this post.

FAQ

Can I work with (use) an unscoped token outside of Keystone when I'm invoking APIs on service endpoints?

It's not recommended -- almost all service implementations require a scoped token and therefore you will likely not get far using unscoped tokens.

How do non-Keystone endpoint services know the token I'm using is PKI based vs UUID based?

PKI based tokens are substantially larger in length than UUID based tokens as they contain the encrypted/signed token metadata whereas UUID based tokens are a fixed length string. Therefore the service checks the length of your token to determine.

Can a single Keystone service manage multiple OpenStack Cloud deployments (regions)?

Yes and in fact this is somewhat common in order to provide a single identity access point for consumers. However keep in mind that multiple Keystone services can share the same data store which allows you to provide multiple Keystone APIs with a single backing store (for example to support geo based access).