Sunday, March 14, 2021

PowerShell Module Quick Start

PowerShell modules are a great way to organize related functions into reusable unit. Today's post will provide a simple walk-through that creates a bare-bones module that you can start to work with.

Table of Contents

Why create a Module?

First of all, let's look at some of the motivations for creating a module - after all, you could simply move all your common utility methods into a 'helper' script and then import it into the session. Why go through the overhead of creating a module?

Not every script or utility function needs to be a module, but if you're creating a series of scripts that rely on some common helper functions, I can think of two primary reasons why you should consider creating a module:

  • Pathing: when you import a script using "dot sourcing", the "." means the current working folder. If you're import individual script files, you need to specify the relative path of the file you're importing. This strategy starts to fall apart if the file you're importing references additional imports. However, when you create a module, you import all the scripts into the session so pathing no longer becomes an issue.
  • Discoverability: PowerShell scripts can get fairly complex so if your helper utility contains dozens of smaller methods then developers consuming your script will need to 'grok' all of those functions, too. By creating a module, you can expose only the functions you want, which will make it easier for developers to understand and use.

1. Create the Manifest Module

A module is comprised of:

  • <module-name>.psd1: a manifest file that describes important details about the module (name, version, license, dependencies, etc)
  • <module-name>.psm1: the main entry point when the module is imported.

Important Note: The folder and module manifest name must match!

To simplify this process, Microsoft has provided the New-ModuleManifest cmdlet for us. They have some good guidance listed here on some additional settings you may want to provide, but here's the bare-bones nitty-gritty:

mkdir MyModule cd MyModule New-ModuleManifest -Path .\MyModule.psd1 -RootModule MyModule.psm1

2. Public vs Private Folders

When you're creating a module, it's best to think about what functions and features that you're exposing to your module consumers. This is very similar to the access-modifiers we put on classes in our .NET assemblies.

The less you expose, the easier it is for consumers to understand what your module does. Limiting what you expose can also protect you from accidentally introducing breaking changes to consumers - if all of your functions are public you won't know which methods that external team members might be using; keeping this list small can help you focus where version compatibility is required.

A good practice is to put our functions into two folders: public and private:

mkdir Private mkdir Public

3. Create the Module Script (psm1)

The last piece is the psm1 script. This simple script finds all the files and imports them into the session:

# MyModule.psm1  
# Get Functions 
$private = Get-ChildItem -Path (Join-Path $PSScriptRoot Private) -Include *.ps1 -File -Recurse
$public = Get-ChildItem -Path (Join-Path $PSScriptRoot Public) -Include *.ps1 -File -Recurse  

# Dot source to scope 
# load private scripts first 
($private + $public) | ForEach-Object {
     try {
         Write-Verbose "Loading $($_.FullName)"
         . $_.FullName
     }
     catch {
         Write-Warning $_.Exception.Message
     }
}  

# Expose public functions. Assumes that function name and file name match 
$publicFunctions = $public | Select-Object -ExpandProperty BaseName Export-ModuleMember -Function $publicFunctions

The last section of the script exposes the functions in your public folder as part of your module. This makes the assumption that the function name and file name are the same. An alternative to this approach is to set the Functions to export in the module manifest. I find this approach easier.

4. Using the Module

At this point, you're ready to start adding PowerShell scripts into your module. When you're ready to try it out, simply import the module by it's definition:

Import-Module .\MyModule.psd1 -Verbose

To verify that your commands are exposed correctly:

(Get-Module MyModule).ExportedCommands

During development of your module, it's important to realize that changes that are made to your scripts won't be visible until you reload the module:

Remove-Module MyModule -ErrorAction Silent Import-Module .\MyModule.psm1

What's Next?

There's lots more to creating a PowerShell module, such as setting the minimum supported PowerShell version, declaring dependencies, including .NET code assemblies, exposing types, etc. There are likely some additional considerations for publishing the module to a gallery, but unfortunately I'm not going to get into that for the purposes of this post. This article is helpful for creating the basic shell of a module, which you can use locally or on build servers.

Wrap Up

Creating a PowerShell module is a fairly quick process that helps to promote reuse between related scripts. With Microsoft's New-ModuleManifest cmdlet and the basic psm1 file provided above, you can fast track creating a Module.

Happy codin'

Saturday, March 06, 2021

Using the Azure CLI to Call Azure DevOps REST API

Suppose the Azure DevOps REST API that you want to call isn't in the list of az cli supported commands. Does this mean your script needs to toggle between az cli and invoking REST endpoints? Fear not, there's actually a built in az devops command "az devops invoke" that can call any Azure DevOps REST API endpoint.

The az devops invoke command is fairly easy to use, but the trick is discovering the command-line arguments you need to provide to pull it off. This post will walk you through that.

Table of Contents

Obtain a List of Available Endpoints

The az devops invoke command is neat alternative to using the REST API, but understanding what command-line arguments you'll need isn't obvious.

Let's start by finding out which endpoints are available by calling az devops invoke with no arguments and pipe this to a file for reference:

az devops invoke > az_devops_invoke.json 

This will take a few moments to produce. I've got a full listing of endpoints located here.

The list of endpoints are grouped by 'Area' and have a unique 'resourceName' and 'routeTemplate'. Here's an snippet:

 ... 
{ 
   "area": "boards", 
   "id": 
   "7f9949a0-95c2-4c29-9efd-c7f73fb27a63", 
   "maxVersion": 5.1, 
   "minVersion": 5.0, 
   "releasedVersion": "0.0", 
   "resourceName": "items", 
   "resourceVersion": 1, 
   "routeTemplate": "{project}/_apis/{area}/boards/{board}/{resource}/{*id}" 
},
{ 
   "area": "build", 
   "id": "5a21f5d2-5642-47e4-a0bd-1356e6731bee", 
   "maxVersion": 6.0, 
   "minVersion": 2.0, 
   "releasedVersion": "5.1", 
   "resourceName": "workitems", 
   "resourceVersion": 2, 
   "routeTemplate": "{project}/_apis/{area}/builds/{buildId}/{resource}" 
}
...

You can also use the JMESPath query syntax to reduce the list:

az devops invoke --query "[?area == 'build']"

Interesting note: If you study the source code for the az devops cli extension, you'll notice that all commands in the devops extension are using this same list as the underlying communication mechanism.

Finding the right endpoint

Finding the desired API in the list of endpoints might take a bit of research. All of the endpoints are grouped by 'area' and then 'resourceName'. I find that the 'area' keyword lines up fairly close with the API documentation, but you'll have to hunt through the endpoint list until you find the 'routeTemplate' that matches the API you're interested in.

Let's use the Get Latest Build REST API as an example. It's REST endpoint is defined as:

GET https://dev.azure.com/{organization}/{project}/_apis/build/latest/{definition}?api-version=6.0-preview.1

The routeTemplate is parameterized such that area and resource parameters correspond to the area and resourceName in the object definition.

From this, we hunt through all the 'build' endpoints until we find this matching endpoint:

{
  "area": "build",
  "id": "54481611-01f4-47f3-998f-160da0f0c229",
  "maxVersion": 6.0,
  "minVersion": 5.0,
  "releasedVersion": "0.0",
  "resourceName": "latest",
  "resourceVersion": 1,
  "routeTemplate": "{project}/_apis/{area}/{resource}/{definition}"
}

Invoking endpoints

Once you've identified the endpoint from the endpoint list, next you need to map the values from the route template to the command-line.

The mapping between command-line arguments and the routeTemplate should be fairly obvious. The values for "{area}" and "{resource}" are picked up from their corresponding command-line arguments, and the remaining arguments must be supplied as name-value pairs with the --route-parameters argument.

Using our Get Latest Build example, "{project}" and "{definition}" are provided on the command line like this:

az devops invoke `
     --area build `
     --resource latest `
     --organization https://dev.azure.com/myorgname `
     --route-parameters `
          project="MyProject" `
          definition=1234 `
     --api-version=6.0-preview

Adding query-string parameters

We can further extend this example by specifying query string parameters using the --query-parameters argument. In this example, we can get the latest build for a specific branch by specifying the branchName parameter:

az devops invoke `
     --area build --resource latest `
     --organization https://dev.azure.com/myorgname `
     --route-parameters project="MyProject" defintion=1234 `
     --query-parameters `
          branchName=develop `
     --api-version=6.0-preview

Note that while the CLI will validate route-parameters, it does not complain if you specify a query-string parameter that is misspelled or not supported.

Specifying the API Version

One of the challenges is knowing which API version to use. Frankly, I've had the most luck by specifying the latest version (eg 6.0-preview). As a general rule, the releasedVersion in the endpoint list should indicate which version to use, which is constrained by the 'maxVersion'.

If the releaseVersion is set to "0.0", then the preview flag is required.

Providing a JSON Body

To provide a JSON body for PUT and POST requests, you'll need to provide a JSON file using the --in-file and --httpMethod parameters.

For example, cancelling a build:

# Write the JSON body to disk
'{ "status": "Cancelling"}' | Out-File -FilePath .\body.json

# PATCH the build with a cancelling status
az devops invoke `
     --area build `
     --resource builds `
     --organization https://dev.azure.com/myorgname `
     --route-parameters `
          project="MyProject" `
          buildId=2345 `
     --in-file .\body.json `
     --http-method patch

Known Issues

By design, you would assume that the area and resourceNames in the list of endpoints are intended to be unique, but unfortunately this isn't the case. Again, referring to the source code of the extension, when trying to locate the endpoints by area + resource it appears to be a first-past-the-post scenario where only the first closest match is considered.

In this scenario, it would be helpful if we could specify the endpoint id from the command-line but this isn't supported yet.

To see the duplicates (it's not a small list):

$endpoints = az devops invoke | Select-Object -skip 1 | ConvertFrom-Json
$endpoints | Group-Object -Property area,resourceName | Where-Object { $_.Count -gt 1 }

The important thing to realize is that this list isn't unique to the az devops extension, it's actually a global list which is exposed from Azure DevOps. Perhaps how this list is obtained is something I'll blog about later.

Wrapping Up

While there are still somethings that are easier to do using the REST API, the Azure DevOps CLI offers a built-in capability to invoke the majority of the underlying APIs, though the biggest challenge is finding the right endpoint to use.

My personal preference is to start with the Azure DevOps CLI because I can jump in and start developing without having to worry about authentication headers, etc. I can also combine the results JMESPath filtering.