Azure interactive credentials not working

[Alexander Litvinov]

$ juju add-credential --verbose --debug azure
Enter credential name: alex
Select region [any region, credential is not region specific]: eastus
Select auth type [interactive]:
Enter subscription-id: xxxxx
Initiating interactive authentication.

To sign in, use a web browser to open the page https://microsoft.com/devicelogin and enter the code XXXXX to authenticate.
ERROR finalizing credential: creating service principal password: creating service principal password: Authorization_RequestDenied: Insufficient privileges to complete the operation.
11:17:31 DEBUG cmd supercommand.go:549 error stack:
Authorization_RequestDenied: Insufficient privileges to complete the operation.
github.com/juju/juju/provider/azure/internal/azureauth.(*ServicePrincipalCreator).createOrUpdateJujuServicePrincipal.func1:184: creating service principal password
github.com/juju/juju/provider/azure/internal/azureauth.(*ServicePrincipalCreator).createOrUpdateJujuServicePrincipal:195: creating service principal password
github.com/juju/juju/provider/azure/internal/azureauth.(*ServicePrincipalCreator).Create:155:
github.com/juju/juju/provider/azure.environProviderCredentials.deviceCodeCredential:240:
github.com/juju/juju/cmd/juju/cloud.finalizeProvider:443: finalizing credential
github.com/juju/juju/cmd/juju/cloud.(*addCredentialCommand).interactiveAddCredential:399:

Juju interactive credentials in fact is using Juju app with ID 60a04dc9-1857-425f-8076-5ba81ca53d66.

My theory is Juju is trying to create SP for this app 'Juju' with something like
az ad sp create --id $appid.
but our accounts (I have checked with 3 people in FE organization and we all have the same issue) doesn't have this permission.
Same as customer accounts, I believe, won't have this permission.

[Ian Booth (wallyworld)]

Is it possible to get a temporary account so we can diagnose the issue and determine what needs to be fixed? None of the accounts we are using have the issue and unless we can reproduce, it's hard to know exactly what to fix and if any fix we come up with will work.

[Alexander Litvinov (alitvinov)]

I am not the admin of our organization Azure, so I can't answer this question.
Better to ask the respective team.

[Vladimir Grevtsev (vlgrevtsev)]

Same for me here.

I was trying to deploy a Juju controller, but the credentials have failed to create (potentially, because of some organizational policies which I'm unaware about and do not have any permissions to modify)- so the standard juju bootstrap process doesn't work out of the box and there is no visible error message to indicate what exactly gone wrong (see screenshot attached).

From Juju's perspective, the error was simply as if the authentication process would have been cancelled.

Auth Types
  interactive
  service-principal-secret

Select auth type [interactive]:

Enter subscription-id:
xxx

Enter subscription-id:
Initiating interactive authentication.

To sign in, use a web browser to open the page https://microsoft.com/devicelogin and enter the code xxxxx to authenticate.
ERROR finalizing credential: looking for existing service principal for Azure Key Vault application: DeviceCodeCredential authentication failed
POST https://login.microsoftonline.com/6408b752-e868-44db-9221-765baeb8e9ec/oauth2/v2.0/token
--------------------------------------------------------------------------------
RESPONSE 400 Bad Request
--------------------------------------------------------------------------------
Response contained no body
--------------------------------------------------------------------------------

08:50:08 DEBUG cmd supercommand.go:549 error stack:
DeviceCodeCredential authentication failed
POST https://login.microsoftonline.com/6408b752-e868-44db-9221-765baeb8e9ec/oauth2/v2.0/token
--------------------------------------------------------------------------------
RESPONSE 400 Bad Request
--------------------------------------------------------------------------------
Response contained no body
--------------------------------------------------------------------------------

github.com/juju/juju/provider/azure/internal/azureauth.(*ServicePrincipalCreator).createOrUpdateServicePrincipal:207: looking for existing service principal for Azure Key Vault application
github.com/juju/juju/provider/azure/internal/azureauth.(*ServicePrincipalCreator).Create:150:
github.com/juju/juju/provider/azure.environProviderCredentials.deviceCodeCredential:215:
github.com/juju/juju/cmd/juju/cloud.finalizeProvider:443: finalizing credential
github.com/juju/juju/cmd/juju/cloud.(*addCredentialCommand).interactiveAddCredential:399:

[Vladimir Grevtsev (vlgrevtsev)]

It looks like that the necessary permissions required to use interactive authentication are not documented in the azure provider docs (https://juju.is/docs/juju/microsoft-azure mentions managed identities, but they're available only from 3.6 - while I'm using 3/stable which resolves to 3.5.3 at the moment of writing).

[Ian Booth (wallyworld)]

I can see the permissions listed on that doc page in the first table

- Microsoft.Compute/skus (read)
- Microsoft.Resources/subscriptions/resourceGroups (read, write, delete)
- Microsoft.Resources/deployments/ (write/read/delete/cancel/validate)
- Microsoft.Network/networkSecurityGroups (write, read, delete, other - join)
- Microsoft.Network/virtualNetworks/ (write, read, delete)
- Microsoft.Compute/virtualMachineScaleSets/ (write, read, delete, other - start action, other - deallocate action, other - restart action, other powerOff action)
- Microsoft.Network/virtualNetworks/subnets/ (read, write, delete, other - join)
- Microsoft.Compute/availabilitySets (write, read, delete)
- Microsoft.Network/publicIPAddresses (write, read, delete, other - join - optional for public services)
- Microsoft.Network/networkInterfaces (write, read, delete, other - join)
- Microsoft.Compute/virtualMachines (write, read, delete, other - start, power off, restart, deallocate)
- Microsoft.Compute/disks (write, read, delete)

[Ian Booth (wallyworld)]

About comment #4, you might see the screenshot in that comment, or you might see an error like this trying to do the device code workflow:

AADSTS650052: The app is trying to access a service 'cfa8b339-82a2-471a-a3c9-0fc0be7a4093'(Azure Key Vault) that your organization '5a8317e2-5d5b-4f10-b846-a5ba3f998ca4' lacks a service principal for. Contact your IT Admin to review the configuration of your service subscriptions or consent to the application in order to create the required service principal.

That just means you first need to et up the account to have a service principal created for the built-in Azure Key Vault application; the application has a "well known" ID cfa8b339-82a2-471a-a3c9-0fc0be7a4093

So from the CLI, assuming you are logged in as the user you want to create a credential for:

az ad sp create --id cfa8b339-82a2-471a-a3c9-0fc0be7a4093

Or you can do it from the portal.

[Ian Booth (wallyworld)]

This is hopefully fixed in this PR

https://github.com/juju/juju/pull/17995

The PR contains a full explanation of what was done, but basically:
1. we don't rely on the Canonical Juju enterprise app any more
2. we don't attempt to create a roll assignment to the built in Azure Key Vault app any more

Item 1 is replaced by creating a bespoke role definition and app on the user's subscription
Item 2 is replaced by the user doing it manually up front (interactive add-credential prints instructions)

I could only test this with a credential that worked the old way, so am not 100% sure the fix will be universally applicable. I'd appreciate it being tested and if there's an issue, add a comment to the bug.

[Ian Booth (wallyworld)]

This has now landed in the 3.6 edge snap

[Alexander Litvinov (alitvinov)]

I was able to add credential with interactive mode (with 2 logins) and custom role name.
(on first attempt I got this error "A custom role with the same name already exists in this directory." so I used custom on the second attempt. Maybe it is possible to reuse existing role?)

and then bootstrap with
juju bootstrap azure test2 --credential alextest2 --storage-pool name=foo --storage-pool type=azure --storage-pool encrypted=true --storage-pool vault-name-prefix=secret

Thank you for working on this issue.

[Ian Booth (wallyworld)]

Awesome, thanks for testing.

Is there an error code or any more detail with the "already exists" message. When I tested I created a few differently named credentials, and so would have reused the same role definition. If there's a more verbose error you can share, it might explain where we need to look.

[Alexander Litvinov (alitvinov)]

here is full debug log:

ubuntu@jujucli:~$ juju add-credential azure --debug --verbose
11:48:20 INFO juju.cmd supercommand.go:56 running juju [3.6-beta3 3aec73a1c7bab405b04c0ff97eac042fda55f178 gc go1.22.6]
11:48:20 DEBUG juju.cmd supercommand.go:57 args: []string{"/snap/juju/28431/bin/juju", "add-credential", "azure", "--debug", "--verbose"}
11:48:20 INFO cmd controller.go:493 This operation can be applied to both a copy on this client and to the one on a controller.
11:48:20 INFO cmd controller.go:509 No current controller was detected and there are no registered controllers on this client: either bootstrap one or register one.
11:48:20 DEBUG juju.provider.azure.internal.azurecli az.go:68 running az account show -o json
Enter credential name: azuretest
...
Select region [any region, credential is not region specific]: eastus
...
Select auth type [interactive]:
Enter subscription-id: XXXXX
Enter application-name (optional):
Enter role-definition-name (optional):
Note: your user account needs to have a role assignment to the
Azure Key Vault application (cfa8b339-82a2-471a-a3c9-0fc0be7a4093).
You can do this from the Azure portal or using the az cli:
  az ad sp create --id cfa8b339-82a2-471a-a3c9-0fc0be7a4093

Initiating interactive authentication.
To sign in, use a web browser to open the page https://microsoft.com/devicelogin and enter the code QQQQQ to authenticate.
ERROR finalizing credential: creating role definition: failed to create role definition: PUT https://management.azure.com/subscriptions/XXXXX/providers/Microsoft.Authorization/roleDefinitions/ZZZZZ
--------------------------------------------------------------------------------
RESPONSE 409: 409 Conflict
ERROR CODE: RoleDefinitionWithSameNameExists
--------------------------------------------------------------------------------
{
  "error": {
    "code": "RoleDefinitionWithSameNameExists",
    "message": "A custom role with the same name already exists in this directory. Use a different name."
  }
}
--------------------------------------------------------------------------------

11:49:13 DEBUG cmd supercommand.go:549 error stack:
PUT https://management.azure.com/subscriptions/XXXXX/providers/Microsoft.Authorization/roleDefinitions/ZZZZZ
--------------------------------------------------------------------------------
RESPONSE 409: 409 Conflict
ERROR CODE: RoleDefinitionWithSameNameExists
--------------------------------------------------------------------------------
{
  "error": {
    "code": "RoleDefinitionWithSameNameExists",
    "message": "A custom role with the same name already exists in this directory. Use a different name."
  }
}
--------------------------------------------------------------------------------

github.com/juju/juju/provider/azure/internal/azureauth.(*ServicePrincipalCreator).ensureRoleDefinition:238: failed to create role definition
github.com/juju/juju/provider/azure/internal/azureauth.(*ServicePrincipalCreator).Create:170: creating role definition
github.com/juju/juju/provider/azure.environProviderCredentials.deviceCodeCredential:260:
github.com/juju/juju/cmd/juju/cloud.finalizeProvider:443: finalizing credential
github.com/juju/juju/cmd/juju/clou...

[Alex Lutay (taurus)]

Hi @alitvinov,

I have bumped the same `RoleDefinitionWithSameNameExists` issue today on Juju 3.6-rc1.

TL;DR: the interactive mode with service-principal-secret-via-browser works for me IF
you fill the optional variables. Follow: https://charmhub.io/postgresql/docs/h-deploy-azure

It looks like Azure custom roles applicable for entire Azure Company (and not per Subscription).
Also it looks like Juju uses hardcoded value custom role name `Juju Application Role Definition`.
Once someone created it inside Canonical => RoleDefinitionWithSameNameExists.

Workaround: use random role-definition-name (even it is marked as optional on Juju CLI).

[Ian Booth (wallyworld)]

I raised bug https://bugs.launchpad.net/juju/+bug/2084858 to track the new issue.