Small IT things to remember

Background

In a project that is very heavy on databases and ssis projects the decision was made to split up the large system into smaller modules. It was also decided to keep each module in its own repository in the project.

When we started this project, we had in the project 25 different modules, the layout of each module was the same, a database, a ssis project and a webpage. It was decided to host this in Azure DevOps. Setting up 25 repositories that is identical from the start is not something I would like to do manually and if there were 25 modules it’s likely there will be more.

The decision was made to keep the build yaml pipelines in its own repository, this so all the pipelines could share resources like PowerShell scripts and test.

The first 25 repositories were created with AZ CLI and PowerShell. It was just one person that had the knowledge and the right to do it. This is not optimal and though the script was made easier to run it still was not optimal since others didn’t feel they had the knowledge to do it. Implementing a pipeline in the project would facilitate the straightforward creation of new modules by a broader range of contributors.

So, what should the pipeline do?

• Get a name for the module when triggering the pipeline.
• Create a repository with the given name.
• Set the folder structure for the module in the created repository.
• Create and registry a “build pipeline” in the pipeline repository.
• Configure Policies for the repositories and Areas for the boards.
• If something fails clean up what has been done.

Having most of this in PowerShell already with AZ cli and working, how much trouble can it be?

When designing the pipeline it was split into five stages.

• Setup and Validation.
• Create the repository and folder structure and default files.
• Create the pipeline.
• Create administration settings.
• Cleanup (in case of failure).

Since this was one big PowerShell script, the work started by breaking up the script into smaller scripts at the same time I built up the pipeline using the PowerShell@2 task in the pipeline.

Create repository

Since this is the main purpose of the pipeline this is where the work started. Starting small, just create a minimal pipeline that creates a repository.

- task: PowerShell@2
  inputs:
    targetType: 'inline'
    script: |
      az repos create --name ${{ parameters.ModuleName }} --project "Demo" --organization "https://dev.azure.com/xxxx" --output json 2>&1
    showWarnings: true
    pwsh: true
  env:
    AZURE_DEVOPS_EXT_PAT: $(System.AccessToken)

“ERROR: TF401027: You need the Git ‘CreateRepository’ permission to perform this action. Details: identity ‘Build\xxxxx-xxx-xxx-xxx-xxxx, scope ‘project’.

Almost success the build pipeline needs the rights to create repositories in the project. Project Settings – Repositories – All Repositories – Security and change “Create repository” to “Allow”.

Then the problem started….

Even though the rights for the Build Service were set correct it still said It missed the Create Repository rights.

I was stuck since if you need to do something in a Pipeline that needs a user right you can use the system access token. So why didn’t it work?
The system access token is the “pat” so that the build services can access resources is Azure Devops.

Spending time searching and asking ChatGPT and copilot and basically getting the same suggestions back all the time in what to do. It was first when I looked at the pipeline settings and noticed the settings “Protect access to repositories in YAML pipelines”

If “Protect access to repositories in YAML pipelines” is enabled (it’s enabled by default) the pipeline is limited to do things only in the repository where its located!

“Protect access to repositories in YAML pipelines” has the info “Also, generate a job access token that is scoped to repositories that are explicitly referenced in the YAML pipeline.” This means that we can’t go outside the repository where the pipeline is located in, we need to explicitly say in the pipeline that we will need to access a different repository. But the requirement for the pipeline was that the name of the repository should be set at execution of the pipeline.

So, what to do?

Asking ChatGPT with all this info it made the suggestion to use a Service Connection to azure. Exploring this option looked like the right way. The recommended way to create a service connection to Azure is in the “Project settings” in the “Pipeline section” klick “Service Connections” and then click “New service connection”. The connection to choose is “Azure Resource Manager”. The easiest is to choose the defaults (if you have the rights in azure)

Leave the Identity type and Credential to the default. Choose the scope level you want to use. Choose what resource group and give it a name to you standard. Choose if you want to give it access to all pipelines. This will create a connection and an app registration. You can then add Azure Devops to the App you can then add to your Azure Devops as a user and give it rights in the project like a normal user. The app registration will not have the same name as the Service connection.

This was the app that was created from Azure Devops service connection. I changed the group to Readers to give it minimal rights and then add only the rights in needs in the project. I added the rights to create repositories to the user in the project Demo

Changed the pipeline

- task: AzureCLI@2
  inputs:
    azureSubscription: 'AZDO-Demo-SC'
    scriptType: 'pscore'
    scriptLocation: 'inlineScript'
    inlineScript: |
      
      az repos create --name ${{ parameters.ModuleName }} --project "Demo" --organization "https://dev.azure.com/xxxx" --output json 2>&1

SUCCESS! The pipeline created the repository in the project. The small downside is that we need to use a license in Azure Devops to use this solution. I haven’t found another solution if the “Protect access to repositories in YAML pipelines” setting can’t be removed.

Using the AzureCLI@2 task with the input “azureSubscription: ServiceConnectionName” you can run Azure CLI commands against Azure Devops. The reason is that it will automatically login using the service connection with AzCLI tasks. Since the “user” needs access to the project it’s needed to add the service connection app as a user in the Azure Devops organization and to the project.

Git operations

Now that the pipeline can create the repository it’s time to prep the new repository with the default folder structure and files that all modules use. Let’s start with using a PowerShell task

- task: PowerShell@2
  inputs:
    targetType: 'inline'
    script: |
      Write-Host "Cloning repo: $env:REPO_NAME"

      # Construct the repo URL (adjust org/project as needed)
      $repoUrl = "https://dev.azure.com/psadi/Demo/_git/$env:REPO_NAME"

      # Use System.AccessToken for authentication
      git -c http.extraheader="AUTHORIZATION: bearer $env:SYSTEM_ACCESSTOKEN" clone $repoUrl
  env:
    SYSTEM_ACCESSTOKEN: $(System.AccessToken)
    REPO_NAME: ${{ parameters.repoName }}

The result wasn’t good

remote: TF401019: The Git repository with name or identifier Module_T does not exist or you do not have permissions for the operation you are attempting.

fatal: repository ‘https://dev.azure.com/xxxx/Demo/_git/Module_T/’ not found

And we are back to having errors that all say: don’t exist, but it does since it was just created. Since PowerShell didn’t work. Can we use same strategi as in the create repo task.

- task: AzureCLI@2
  inputs:
    azureSubscription: 'AZDO-Demo-SC'
    scriptType: 'pscore'
    scriptLocation: 'inlineScript'
    inlineScript: |
      $repoName = "${{ parameters.repoName }}"
      Write-Host "Git Clone"
      git clone https://dev.azure.com/psadi/Demo/_git/$repoName

  One small step closer but still not able to clone.

fatal: could not read Username for ‘https://dev.azure.com’: terminal prompts disabled

Can the service connection we created earlier? The answer seems to be yes. The pipeline should be able to request a bearer token from the service connection by

az account get-access-token --resource 499b84ac-1321-427f-aa17-267ca6975798

Where the guid is the same for everyone!
This bearer token is a short-lived PAT for the service connection and since it uses az the pipeline will be using the AzCLI task

- task: AzureCLI@2
  inputs:
    azureSubscription: 'AZDO-Demo-SC'
    scriptType: 'pscore'
    scriptLocation: 'inlineScript'
    inlineScript: |
      $token = az account get-access-token --resource 499b84ac-1321-427f-aa17-267ca6975798 --query accessToken -o tsv
      Write-Host "Cloning repo: ${{ parameters.repoName }}"

      # Construct the repo URL (adjust org/project as needed)
      $repoUrl = "https://dev.azure.com/xxxx/Demo/_git/${{ parameters.repoName }}"

      # Use System.AccessToken for authentication
      git -c http.extraheader="AUTHORIZATION: bearer $token" clone $repoUrl

The pipeline is all green.

But we don’t have to run Git operations with AzureCLI task. We can save the token as a variable in the pipeline so that we can use it in later tasks.

- task: AzureCLI@2
  name: SCToken
  inputs:
    azureSubscription: 'AZDO-Demo-SC'
    scriptType: 'pscore'
    scriptLocation: 'inlineScript'
    inlineScript: |
      $token = az account get-access-token --resource 499b84ac-1321-427f-aa17-267ca6975798 --query accessToken -o tsv
      Write-Output "##vso[task.setvariable variable=AdoSCToken;IsSecret=true]$token"

- task: PowerShell@2
  inputs:
    targetType: 'inline'
    script: |
      $token = "$(AdoSCToken)"
      Write-Host "Cloning repo: $env:REPO_NAME"

      # Construct the repo URL (adjust org/project as needed)
      $repoUrl = "https://dev.azure.com/psadi/Demo/_git/$env:REPO_NAME"

      # Use System.AccessToken for authentication
      git -c http.extraheader="AUTHORIZATION: bearer $token" clone $repoUrl
  env:
    REPO_NAME: ${{ parameters.repoName }}

To summarize

If the Azure Devops project has the setting “Protect access to repositories in YAML pipelines” enabled, the pipeline is locked down to its own repository if you don’t explicitly set what repository you want to use in the resource section

resources:
  repositories:
    - repository: aliasRepoName    # alias you choose
      type: git                    # Azure Repos Git
      name: MyProject/Repo         # project/repo name
      ref: main                    # branch to checkout

steps:
- checkout: aliasRepoName

 Using this will allow the pipeline to access that repo. But if the pipeline needs to use a dynamically repository name that is set when the pipeline is running, then there isn’t any way to set it like above.

When the pipeline gets the repository at runtime, one way to solve it is by using as service connection and authenticating to Azure Devops with that account. Could also use a PAT but then the solution is based on a user that needs to rotate the PAT from time to time. Using the service connection setup, that problem doesn’t exist. The service and build accounts require custom permissions throughout the solution. It took a lot of running and changing one thing and run it again to see if the error was solved.  That also goes for should a task be PowerShell or AzureCLI task. If you need to use the system.accesstoken or the token from the service connection. It was a frustrating but a very learning experience.