Wednesday, July 29, 2020

Managing ADO Licenses from Azure DevOps CLI

My last post introduced using JMESPath with the az devops cli, which hopefully gave you some insight into use the az cli and the az devops extension. Today I want to highlight how you can easily pull the az devops cli into PowerShell to unlock some amazing scripting ability.

A good example of this is how we can use PowerShell + az devops cli to manage Azure DevOps User Licenses.

Background

Azure DevOps is a licensed product, and while you can have unlimited free Stakeholder licenses, any developer that needs access to code repositories needs a Basic license, which costs about $7 CAD / month.

It's important to note that this cost is not tied to usage, so if you've allocated licenses manually, you're essentially paying for it. Interestingly, this is not a fixed monthly cost, but prorated on a daily basis in the billing period. So if you can convert Basic licenses to Stakeholder licenses, you’ll only pay for the days in the billing period when the license was active.

If you establish a process to revoke licenses when they're not being used, you can save your organization a few dollars that would otherwise be wasted. It might not be much, but if you consider 10 user licenses for the year is about $840 – the costs do add up. something you could argue should be added to your end-of-year bonus.

Integrating the Azure DevOps CLI into PowerShell

To kick things off and to show how incredibly easy this is, let's start with this snippet:

param() {

}


$users = az devops user list | ConvertFrom-Json

Write-Host $users.totalCount
Write-Host $users.items[0].user.principalName

Boom. No special magic. We just call the az devops cli directly in our PowerShell script, converting the JSON result into an object by piping it through the ConvertFrom-Json commandlet. We can easily interrogate object properties and build up some conditional logic. Fun.

Use JMESPath to Simplify Results

While we could work with the results in this object directly, the result objects have a complex structure so I’m going to flatten the object down to make it easier to get at the properties we need. I only want the user’s email, license, and the dates when the user account was created and last accessed.

JMESPath makes this easy. If you missed the last post, go back and have a read to get familiar with this syntax.

function Get-AzureDevOpsUsers() {

    $query = "items[].{license:accessLevel.licenseDisplayName, email:user.principalName, dateCreated:dateCreated, lastAccessedDate:lastAccessedDate }"
    $users = az devops user list --query "$query" | ConvertFrom-Json

    return $users
}

Ok. That structure’s looking a lot easier to work with.

Finding Licenses to Revoke

Now all we need to do is find the licenses we want to convert from Basic to Stakeholder. Admittedly, this could create headaches for us if we're randomly taking away licenses that people might need in the future, but we should be safe if we target people who've never logged in or haven't logged in for 21 days.

I’m going to break this down into two separate functions. One to filter the list of users based on their current license, and another function to filter based on the last access date.

function Where-License()
{
    param(
        [Parameter(Mandatory=$true, ValueFromPipeline)]
        $Users,

        [Parameter(Mandatory=$true)]
        [ValidateSet('Basic', 'Stakeholder')]
        [string]$license
    )

    BEGIN{}
    PROCESS
    {
        $users | Where-Object -FilterScript { $_.license -eq $license }
    }
    END{}
}

function Where-LicenseAccessed()
{
    param(
        [Parameter(Mandatory=$true, ValueFromPipeline)]
        $Users,

        [Parameter()]
        [int]$WithinDays = 21,

        [Parameter()]
        [switch]$NotUsed = $false
    )
    BEGIN
    {
        $today = Get-Date
    }
    PROCESS
    {
        $Users | Where-Object -FilterScript {

            $lastAccess = (@( $_.lastAccessedDate, $_.dateCreated) | 
                            ForEach-Object { [datetime]$_ } |
                            Measure-Object -Maximum | Select-Object Maximum).Maximum

            $timespan = New-TimeSpan -Start $lastAccess -End $today

            if (($NotUsed -and $timespan.Days -gt $WithinDays) -or ($NotUsed -eq $false -and $timespan.Days -le $WithinDays)) {
                Write-Host ("User {0} last accessed within {1} days." -f $_.email, $timespan.Days)
                return $true
            }

            return $false
        }
    }
    END {}
}

If you're new to PowerShell, the BEGIN,PROCESS,END blocks may look peculiar but they are essential for chaining results of arrays together. There’s a really good write-up on this here. But to demonstrate, we can now chain these methods together, like so:

Get-AzureDevOpsUsers | 
    Where-License -license Basic | 
        Where-LicenseAccessed -NotUsed -WithinDays 21

Revoking Licenses

And then we use the ever so important function to revoke licenses. This is simply a wrapper around the az devops command to improve readability.

function Set-AzureDevOpsLicense()
{
    param(
        [Parameter(Mandatory=$true)]
        $User,

        [Parameter(Mandatory=$true)]
        [ValidateSet('express','stakeholder')]
        [string]$license
    )
    Write-Host ("Setting User {0} license to {1}" -f $_.email, $license)
    az devops user update --user $user.email --license-type $license | ConvertFrom-Json
}

Putting it all Together

So we've written all these nice little functions, let's put them together into a little PowerShell haiku:

$users = Get-AzureDevOpsUsers | 
                 Where-License -license Basic | 
                 Where-LicenseAccessed -NotUsed -WithinDays 21 | 
                 ForEach-Object { Set-AzureDevOpsLicense -User $_ -license stakeholder }
Write-Host ("Changed {0} licenses." -f $users.length)

Nice. Az you can see that wasn't hard at all, and it probably saved my boss a few bucks. There's a few ways we can make this better...

Dealing with lots of Users

The az devops user list command can return up to 1000 users but only returns 100 by default. If you have a large organization, you'll need to make a few round trips to get all the data you need.

Let’s modify our Get-AzureDevOpsUsers function to retrieve all the users in the organization in batches.

function Get-AzureDevOpsUsers() {

    param(
        [Parameter()]
        [int]$BatchSize = 100
    )

    $query = "items[].{license:accessLevel.licenseDisplayName, email:user.principalName, dateCreated:dateCreated, lastAccessedDate:lastAccessedDate }"
    
    $users = @()

    $totalCount = az devops user list --query "totalCount"
    Write-Host "Fetching $totalCount users" -NoNewline

    $intervals = [math]::Ceiling($totalCount / $BatchSize)

    for($i = 0; $i -lt $intervals; $i++) {

        Write-Host -NoNewline "."

        $skip = $i * $BatchSize;
        $results = az devops user list --query "$query" --top $BatchSize --skip $skip | ConvertFrom-Json

        $users = $users + $results
    }   

    return $users
}

Giving licenses back

If the script can taketh licenses away, it should also giveth them back. To do this, we need the means to identify who should have a license. This can be accomplished using an Azure AD User Group populated with all users that should have licenses.

To get the list of these users, the Azure CLI comes to the rescue again. Also again, learning JMESPath really helps us because we can simplify the entire result into a basic string array:

az login --tenant <tenantid>
$licensedUsers = az ad group member list -g <groupname> --query "[].otherMails[0]" | ConvertFrom-Json

Note that I'm using the otherMails property to get the email address, your mileage may vary, but in my Azure AD, this setting matches Members and Guests with their email address in Azure DevOps.

With this magic array of users, my haiku can now reassign users their license if they've logged in recently without a license (sorry mate):

$licensedUsers = az ad group member list -g ADO_LicensedUsers --query "[].otherMails[0]" | ConvertFrom-Json

$users = Get-AzureDevOpsUsers
$reactivatedUsers = $user | Where-License -license Stakeholder | 
                            Where-LicenseAccessed -WithinDays 3 | 
                            Where-Object -FilterScript { $licensedUsers.Contains($_.email) } | 
                            ForEach-Object { Set-AzureDevOpsLicense -User $_ -license express }

$deactivatedUsers = $user | Where-License -license Basic | 
                            Where-LicenseAccessed -NotUsed -WithinDays 21 | 
                            ForEach-Object { Set-AzureDevOpsLicense -User $_ -license stakeholder }

Write-Host ("Reviewed {0} users" -f $users.Length)
Write-Host ("Deactivated {0} licenses." -f $deactivatedUsers.length)
Write-Host ("Reactivated {0} licenses." -f $reactivatedUsers.length)

Wrapping up

In the last few posts, we've looked at the Azure DevOps CLI, understanding JMESPath and now integrating both into PowerShell to unleash some awesome. If you're interested in the source code for this post, you can find it here.

In my next post, we'll build upon this and show you how to integrate this script magic into an Azure Pipeline that runs on a schedule.

Happy coding.

Monday, July 27, 2020

Azure DevOps CLI Examples

I've always been a fan of Azure DevOps's extensive REST API -- it's generally well documented, consistent and it seems like you can do pretty much anything you can do from within the web-interface. As much as I love the API, I hate having to bust it out. Nowadays, my new go to tool is the Azure DevOps CLI.

The Azure DevOps CLI is actually an extension of the Azure CLI. It contains a good number of common functions that you would normally use on a daily basis, plus features that I would normally rely on the REST API for. Its real power is unlocked when it's combined with your favourite scripting language. I plan to write a few posts on this topic, so stay tuned, but for today, we'll focus on getting up and running plus some cool things you can do with the tool.

Installation

Blurg. I hate installation blog posts. Let's get this part over with:

choco install azure-cli -y
az extension add --name azure-devops

Whew. If you don't have Chocolatey installed, go here: https://chocolatey.org/install

Get Ready

Ok, so we're almost there. Just a few more boring bits. First we need to login:

az login --allow-no-subscription

A quick note on the above statement. There are a number of different login options available but I've found az login with the --allow-no-subscription flag supports the majority of use cases. It'll launch a web-browser and require you to login as you normally would, and the --allow-no-subscription supports scenarios where you have access to the AD tenant to login but you don't necessarily have a subscription associated to your user account, which is probably pretty common for most users who only have access to Azure DevOps.

This next bit let's us store some commonly used parameters so we don't have to keep typing them out.

az devops configure --defaults organization=https://dev.azure.com/<organization>

In case your curious, this config file is stored in %UserProfile%\.azure\azuredevops\config

Our First Command

Let's do something basic, like getting a list of all projects:

az devops project list

If we've configured everything correctly, you should see a boatload of JSON fly by. The CLI supports different options for output, but JSON works best when paired with our automation plus there's some really cool things we can do with the result output by passing a JMESPath statement using the --query flag.

Understanding JMESPath

The JavaScript kids get all the cool tools. There are probably already a few dozen different ways of querying and manipulating JSON data, and JMESPath (pronounced James Path) is no different. The syntax is a bit confusing at first and it takes a little bit of tinkering to master it. So let's do some tinkering.

The best way to demonstrate this is to use the JSON output from listing our projects. Our JSON looks something like this:

{
   "value": [
     {
        "abbreviation": null,
        "defaultTeamImageUrl": null,
        "description": null,
        "id": "<guid>",
        "lastUpdateTime": "<date>",
        "name": "Project Name",
        "revision": 89,
        "state": "wellFormed",
        "url": "<url>",
        "visibility": "private"
     },
     ...
   ]
}

It's a single object with a property called "value" that contains an array. Let's do a few examples...

Return the contents of the array

Assuming that we want the details of the projects and not the outside wrapper, we can discard the "value" property and just get it's contents, which is an array.

az devops project list --query "value[]"
[
    {
        "abbreviation": null,
        "defaultTeamImageUrl": null,
        "description": null,
        "id": "<guid>",
        "lastUpdateTime": "<date>",
        "name": "Project Name",
        "revision": 89,
        "state": "wellFormed",
        "url": "<url>",
        "visibility": "private"
    },
    ...
]

Return just the first element

Because the "value" property is an array, we can get the first element.

az devops project --query "value[0]"

{
    "abbreviation": null,
    "defaultTeamImageUrl": null,
    "description": null,
    "id": "<guid>",
    "lastUpdateTime": "<date>",
    "name": "Project Name",
    "revision": 89,
    "state": "wellFormed",
    "url": "<url>",
    "visibility": "private"
}

You can also specify ranges:

  • [:2] = everything up to the 3rd item
  • [1:3] = from the 2nd up to the 4th items
  • [1:] = everything from the 2nd item

Return an array of properties

If we just wanted the id property of each element in the array, we can specify the property we want. The result assumes there are only 4 projects.

az devops project --query "value[].id"

[
    "<guid>",
    "<guid>",
    "<guid>",
    "<guid>"
]

Return specific properties

This is where JMESPath gets a tiny bit odd. In order to get just a handful of properties we need to do a "projection" which is basically like stating what structure you want the JSON result to look like. In this case, we're mapping the id and name property to projectId and projectName in the result output.

az devops project --query "value[].{ projectId:id, projectName:name }"

[
    {
        "projectId": "<guid>",
        "projectName": "Project 1"
    },
    {
        "projectId": "<guid>",
        "projectName": "Project 2"
    },
    ...
]

Filter the results

Here's where things get really interesting. We can put functions inside the JMESPath query to filter the results. This allows us to mix and match the capabilities of the API with the output filtering capabilities of JMESPath. This returns only the projects that are public.

az devops project list --query "value[?visibility=='public'].{ id:id, name:name }"
[
    {
        "id": "<guid>",
        "name": "Project 3"
    }
]

We could have also written this as:

--query "value[?contains(visibility,'private')].{id:id, name:name}"

Piping the results

In the above example, JMESPath assumes that the results will be an array. We can pipe the result to further refine it. In this case, we want just the first object in the resulting array.

az devops project list --query "value[?visibility=='private'].{ id:id, name:name} | [0]"

{
   "id": "<guid>",
   "name": "Project 3"
}

Piping can improve the readability of the query similar to a functional language. For example, the above could be written as a filter, followed by a projection, followed by a selection.

--query "value[?contains(visibility,'private')] | [].{id:id, name:name} | [0]"

Wildcard searches

Piping the results becomes especially important if we want just the single value of a wildcard search. For this example, I need a different JSON structure, specifically a security descriptor:

[
  {
    "acesDictionary": {
      "Microsoft.IdentityModel.Claims.ClaimsIdentity;<dynamic-value>": {
        "allow": 16,
        "deny": 0,
        "descriptor": "Microsoft.IdentityModel.Claims.ClaimsIdentity;<dynamic-value>",
        "extendedInfo": {
          "effectiveAllow": 32630,
          "effectiveDeny": null,
          "inheritedAllow": null,
          "inheritedDeny": null
        }
      }
    },
    "includeExtendedInfo": true,
    "inheritPermissions": true,
    "token": "repoV2"
  }
]

In this structure, i'm interested in getting the "allow" and "deny" and "token" values but the first element in the acesDictionary contains a dynamic value. We can use a wildcard "*" to substitute for properties we don't know at runtime.

Let's try to isolate that "allow". The path would seem like [].acesDictionary.*.allow but because JMESPath has no idea if this is a single element, so it returns an array:

[
    [
        16
    ]
]

If we pipe the result, [].acesDictionary.*.allow | [0] we'll get a single value.

[
    16
]

Following suit and jumping ahead a bit so that I can skip to the answer, I can grab the "allow", "deny" and "token" with the following query. At this point, I trust you can figure this out using by referencing all the examples I've provided. The query looks like:

--query "[].{allow:acesDictionary.*.allow | [0], deny:acesDictionary.*.deny | [0], token:token } | [0]"
{
    "allow": 16,
    "deny": 0,
    "token": "repoV2"
}

Ok! That is waay too much JMESPath. Let's get back on topic.

Using the Azure DevOps CLI

The Azure DevOps CLI is designed with commands and subcommands and has a few entry points. At each level, there are the obvious inclusions (list, add, delete, update, show), but there are a few additional commands per level.

  • az devops
    • admin
      • banner
    • extension
    • project
    • security
      • group
      • permission
        • namespace
    • service-endpoint
      • azurerm
      • github
    • team
    • user
    • wiki
      • page
  • az pipelines
    • agent
    • build
      • definition
      • tag
    • folder
    • pool
    • release
      • definition
    • runs
      • artifact
      • tag
    • variable
    • variable-group
  • az boards
    • area
      • project
      • team
    • iteration
      • project
      • team
    • work-item
      • relation
  • az repos
    • import
    • policy
      • approver-count
      • build
      • case-enforcement
      • comment-required
      • file-size
      • merge-strategy
      • required-reviewer
      • work-item-linking
    • pr
      • policy
      • reviewer
      • work-item
    • ref
  • az artifacts
    • universal

I won’t go into all of these commands and subcommands, I can showcase a few of the ones I’ve used the most recently…

List of Projects

az devops project list --query "value[].{id:id, name:name}"

List of Repositories

az repos list --query "[].{id:id, defaultBranch:defaultBranch, name:name}" 

List of Branch Policies

az repos policy list --project <name> --query "[].{name: type.displayName, required:isBlocking, enabled:isEnabled, repository:settings.scope[0].repositoryId, branch:settings.scope[0].refName}"

Service Connections

az devops service-endpoint list --project <name> --query "[].name"

One More Thing

So while the az devops cli is pretty awesome, it has a hidden gem. If you can't find a supporting command in the az devops cli, you can always call the REST API directly from the tool using the az devops invoke command. There's a bit of hunting through documentation and available endpoints to find what you're looking for, but you can get a full list of what's available using the following:

az devops invoke --query "[?contains(area,'build')]"
az devops invoke --query "[?area=='build' && resourceName=='timeline']"

[
  {
    "area": "build",
    "id": "8baac422-4c6e-4de5-8532-db96d92acffa",
    "maxVersion": 6.0,
    "minVersion": 2.0,
    "releasedVersion": "5.1",
    "resourceName": "Timeline",
    "resourceVersion": 2,
    "routeTemplate": "{project}/_apis/{area}/builds/{buildId}/{resource}/{timelineId}"
  }
]

We can invoke this REST API call by passing in the appropriate area, resource, route and query-string parameters. Assuming I know the buildId of a recent pipeline run, the following shows me the state and status of all the stages in that build:

az devops invoke 
    --area build 
    --resource Timeline 
    --route-parameters project=myproject buildId=2058 timelineid='' 
    --query "records[?contains(type,'Stage')].{name:name, state:state, result:result}"

Tip: the route and query parameters specified in the routeTemplate are case-sensitive.

More to come

Today's post outlined how to make sense out of JMESPath and some cool features of the Azure DevOps CLI. My next few posts I'll dig deeper into using the cli in your favourite scripting tool

Happy coding.

Wednesday, July 15, 2020

Exclusive Lock comes to Azure Pipelines

semaphore red left

As part of Sprint 171, the Azure DevOps team introduced a much needed feature for Multi-Stage YAML Pipelines, the Exclusive Lock "check" that can be applied to your environments. This feature silently slipped into existence without any mention of it in the release notes, but I was personally thrilled to see this. (At the time this post was written, Sprint 172 announced this feature was available)

Although Multi-Stage YAML Pipelines have been available for a while, there are still some subtle differences between their functionality and what's available through Classic Release Pipelines. Fortunately over the last few sprints we've seen a few incremental features to help close that feature parity gap, with more to come. One of the missing features is something known as "Deployment Queuing Settings" -- a Classic Release pipeline feature that dictates how pipelines are queued and executed. The Exclusive Lock check solves a few pain points but falls short on some of the more advanced options.

In this post, I'll walk through what Exclusive Locks are, how to use them and some other thoughts for consideration.

Deployments and Environments

Let's start with a multi-stage pipeline with a few stages, where we perform CI activities and each subsequent stage deploys into an environment. Although we could write our YAML to build and deploy using standard tasks, we're going to use the special "deployment" job that tracks builds against Environments.

trigger:
 - master

stages:
 - stage: ci_stage
   ...steps to compile and produce artifacts

- stage: dev_stage
   condition: and(succeeded(), eq(variables['Build.SourceBranch','refs/heads/master'))
   dependsOn: ci_stage
   jobs:
   - deployment: dev_deploy
     environment: dev
     strategy:
       runOnce:
         deploy:
           ... steps to deploy
       
 - stage: test_stage
   dependsOn: dev_stage
   ...

If we were to run this hypothetical pipeline, the code would compile in the CI stage and then immediately start deploying into each environment in sequence. Although we definitely want to have our builds deploy into the environments in sequence, we might not want them to advance into the environments automatically. That's where Environment Checks come in.

Environment Checks

As part of multi-stage yaml deployments, Azure DevOps has introduced the concept of Environments which are controlled outside of your pipeline. You can set special "Checks" on the environment that must be fulfilled before the deployment can occur. On a technical note, environment checks bubble up from the deployment task to the stage, so the checks must be satisfied before the stage is allowed to start.

For our scenario, we're going to assume that we don't want to automatically go to QA, so we'll add an Approval Check that allows our testing team to approve the build before deploying into their environment. We'll add approval checks for the other stages, too. Yay workflow!

approval-checks

At this point, everything is great: builds deploy to dev automatically and then pause at the test_stage until the testing team approves. Later, we add more developers to our project and the frequency of the builds starts to pick up. Almost immediately, the single agent build pool starts to fill up with builds and the development team start to complain that they're waiting a really long time for their build validation to complete.

Obviously, we add more build agents. Chaos ensues.

What just happen'd?

When we introduced additional build agents, we were expecting multiple CI builds to run simultaneously but we probably weren't expecting multiple simultaneous deployments! This is why the Exclusive Lock is so important.

By introducing an Exclusive Lock, all deployments are forced to happen in sequence. Awesome. Order is restored.

There unfortunately isn't a lot of documentation available for the Exclusive Lock, but according to the description:

“Adding an exclusive lock will only allow a single run to utilize this resource at a time. If multiple runs are waiting on the lock, only the latest will run. All others will be canceled."

Most of this is obvious, but what does 'All others will be canceled' mean?

Canceling Queued Builds

My initial impression of the "all other [builds] will be canceled" got me excited -- I thought this was the similar to the “deploy latest and cancel the others” setting of Deployment Queuing Settings:

deployment-queue-settings

Unfortunately, this is not the intention of the Exclusive Lock. It focuses only on sequencing of the build, not on the pending queue. To understand what the “all others will be canceled” means, let's assume we have 3 available build agents and we'll use the az devops CLI to trigger three simultaneous builds.

az pipelines run --project myproject --name mypipeline 
az pipelines run --project myproject --name mypipeline 
az pipelines run --project myproject --name mypipeline

In this scenario, all three CI builds happen simultaneously but the fun happens when all three pipeline runs hit the dev_stage. As expected, the first pipeline takes the exclusive lock on the development environment while the deployment runs and the remaining two builds queue up waiting for the exclusive lock to be released. When the first build completes, the second build is automatically marked as canceled and the last build remains begins deployment.

exclusive-lock-queuing

This is awesome. However I was really hoping that I could combine the Exclusive Lock with the Approval Gate to recreate the same functionality of the Deployment Queuing option: approving the third build would cancel the previous builds. Unfortunately, this isn’t the case. I’m currently evaluating whether I can write some deployment automation in my pipeline to cancel other pending builds.

Wrapping Up

In my opinion, Exclusive Locks are a hidden gem of Sprint 171 as they’re essential if you’re automatically deploying into an environment without an Approval Gate. This feature recreates the “deploy all in sequence” feature of Classic Release Pipelines. The jury is still out on canceling builds from automation. I’ll keep you posted.

Happy coding!

Tuesday, July 14, 2020

Using Templates to improve Pull Requests and Work-Items (Part 2)

In my previous post, I outlined how to setup templates for pull-requests. Today we’ll focus on how to configure work-items with some project-specific templates. We’ll also look at how you can create these customizations for all projects within the enterprise and the motivations for doing so.

While having a good pull-request template can improve clarity and reduce the effort needed to approve pull-requests, having well defined work-items are equally as important as they can drive and shape the work that needs to happen. We can define templates for our work-items to encourage work-item authors to provide the right level of detail (such as steps to reproduce a defect) or we can use templates to reduce effort for commonly created work-items (such as fields that are common set when creating a technical-debt work-item).

Creating Work-Item Templates

Although you can define pull-request templates as files in your git repository, Azure DevOps doesn’t currently support the ability to customize work-items as managed source files. This is largely due to the complexity of work-items structure and the level of customization available, so our only option to date is to manipulate the templates through the Azure Boards user-interface. Fortunately, it’s relatively simple and there are a few different ways you can setup and customize your templates – you can either specify customizations through the Teams configuration for your Project, or you can extract a template from an existing work item.

As extracting from an existing work-item is easier, we’ll look at this first.

Creating Templates from Existing Work-Items

To create a template from an existing work item, simply create a new work-item that represents the content that you’d like to see in your template. The great news is that our template can capture many different elements, ranging from the description and other commonly used fields to more specialized fields like Sprint or Labels.

It’s important to note that templates are team specific, so if you’re running a project with multiple scrum teams, each team can self-organize and create templates that are unique to their needs.

Here’s an example of user story with the description field pre-defined:

work-item-example

Once we like the content of the story, we can convert it into a template using the ellipsis menu (…) Templates –> Capture:

work-item-capture-template

The capture dialog allows us to specify which fields we want to include in our template. This typically populates with the fields that have been modified, but you can remove or add any additional fields you want:

capture-template-dialog

As some fields are stored in the template as HTML, using this technique of creating a template from an existing work-item is especially handy.

Customizing Templates

Once you’ve defined the template, you find them in Settings –> Team Configuration. There’s a sub-navigation item for Templates.

edit-template

Applying Templates

Once you have the template(s) created, there are a few ways you can apply them to your work-items: you can apply the template to the work-item while you’re editing it, or you can apply it to the work-item from the backlog. Both activities are achieved using the ellipsis menu: Templates –> <template-name>.

The latter option of applying the template from the Backlog is extremely useful because you can apply the template to multiple items at the same time.

assign-template-from-backlog

With some creative thinking, templates can be used like macros for commonly performed activities. For example, I created a “Technical Debt” template that adds a TechDebt tag, lowered priority and changes the Value Area to Architectural.

Creating Work-Items from Templates

If you want to apply the template to work-items as you create them, you’ll need to navigate to a special URL that is provided with each template (Settings –> Boards: Team Configuration –> Templates).

get-link-for-template

The Copy Link option copies the unique URL to the template to the clipboard, which you can circulate to your team. Personally, I like to create a Markdown widget on my dashboard that allows team members to navigate to this URL directly.

create-work-item-from-dashboard

Going Further – Set defaults for Process Template

Unfortunately, there’s no mechanism to specify which work-item template should be used as the default for a team. You can however provide these customizations at the Process level, which applies these settings for all teams using that process template. Generally speaking, you should only make these changes for enterprise-wide changes.

Note that you can’t directly edit the default process templates, you will need to create a new process template based on the default: Organization Settings –> Boards –> Process:

process-template

Within the process, you can bring up any of the work-items into an editor that let’s you re-arrange the layout and contents of the work-item. To edit the Description field to have a default value, we select the Edit option in the ellipsis menu:

edit-process-template

Remembering that certain fields are HTML, we can set the default for our user story by modifying the default options:

edit-process-template-field

Wrapping up

Hopefully the last two posts for providing templates for pull requests and work-item templates has given you some ideas on how to quickly provide some consistency to your projects.

Happy coding!

Monday, June 29, 2020

Using Templates to improve Pull Requests and Work-Items (Part 1)

I’m always looking for ways to improve the flow of work. I have a few posts I want to share on using templates for pull requests and work-items. Today, I want to focus on some templates that you can add to your Azure DevOps pull requests to provide some additional context for the work.

Templates for Pull Requests

Pull Requests are a crucial component of our daily work. They help drive our continuous delivery workflows and because they’re accessible from our git history long after the pull-request has been completed, they can serve as an excellent reference point for the work. If you review a lot of pull-requests in your day, a well-written pull-request can make the difference between a good and bad day.

Not many folks realize that Azure DevOps supports pre-populating your pull request with a default template. It can even provide customized messages for specific branches. And because Pull Requests for Azure Repos support markdown, you can provide a template that encourages your team to provide the right amount of detail (and look good, too).

Default Pull Request Template

To create a single template for all your pull requests, create a markdown file named pull_request_template.md and place it in the root of your repository or in a folder named either .azuredevops, .vsts, or docs. For example:

  • .azuredevops/pull_request_template.md
  • .vsts/pull_request_template.md
  • docs/pull_request_template.md
  • <root>/pull_request_template.md

A sample pull request might look like:

----
Delete this section before submitting!

Please ensure you have the following:

- PR Title is meaningful
- PR Title includes work-item number
- Required reviewers is populated with people who must review these changes
- Optional reviewers is populated with individuals who should be made aware of these changes
----
# Summary

_Please provide a high-level summary of the changes for the changes and notes for the reviewers_

- [ ] Code compiles without issues or warnings
- [ ] Code passes all static code-analysis (SonarQube, Fortify SAST)
- [ ] Unit tests provided for these changes

## Related Work

These changes are related to the following PRs and work-items:

_Note: use !<number> to link to PRs, #<number> to link to work items_

## Other Notes

_if applicable, please note any other fixes or improvements in this PR_

As you can see, I've provided a section a the top that provides some guidance on things to do before creating the pull request, such as making sure it has a meaningful name, while the following section provides some prompts to encourage the pull-request author to provide more detail. Your kilometrage will vary, but you may want to work your team to make a template this fits your needs.

Pull request templates can be written in markdown, so it’s possible to include images and tables. My favourite are the checkboxes (- [ ]) which can be marked as completed without having to edit the content.

Branch Specific Templates

You may find the need to create templates that are specific to the target branch. To do this, create a special folder named “pull_request_template/branches” within one of the same folders mentioned above and create a markdown file with the name of the target branch. For example:

  • .azuredevops/pull_request_template/branches/develop.md
  • .azuredevops/pull_request_template/branches/release.md
  • .azuredevops/pull_request_template/branches/master.md

When creating your pull-request, Azure DevOps will attempt to find the appropriate template by matching on these templates first. If a match cannot be found, the pull_request_template.md is used as a fallback option.

Ideally, I’d prefer different templates from the source branch, as we could provide pull-request guidance for bug/*, feature/*, and hotfix/* branches. However, if we focus on develop, release and master we can support the following scenarios:

  • develop.md: provide an overview of improvements of a feature, evidence for unit tests and documentation, links to work-items and test-cases, etc
  • release.md: provide high-level overview of the items in this release, related dependencies and testing considerations
  • master.md: (optional) provide a summary of the release and its related dependencies

    Additional Templates

    In additional to the branch-specific or default-templates, you can create as many templates as you need. You could create specific templates for critical bug fixes, feature proposals, etc. In this scenario, I’d use that initial (delete-me-section) to educate the user on which template they should use.

    You’re obviously not limited to a single template either. If you have multiple templates available, you can mix and match from any of the available templates to fit your needs. Clicking the “add template” simply append the other template to the body of the pull-request.

    create-pull-request

    Other observations

    Here’s a few other observations that you might want to consider:

    • If the pull-request contains only a single commit, the name of the pull-request will default to the commit message. The commit message is also appended to the bottom of the pull-request automatically.
    • If your pull-request contains multiple commits, the name of the pull-request is left empty. The commit messages do not prepopulate into the pull-request, but the “Add commit messages” button appears. The commit messages are added “as-is” to the bottom of the pull-request, regardless where the keyboard cursor is.

    Conclusion

    Hopefully this sheds some light on a feature you might not have known existed. In my next post, we’ll look at how we can provide templates for work-items.

    Happy Coding!

    Monday, June 08, 2020

    Keeping your Secrets Safe in Azure Pipelines

    These days, it’s critical that everyone in the delivery team has a security mindset and is vigilant about keeping secrets away from prying eyes. Fortunately, Azure Pipelines have some great features to ensure that your application secrets are not exposed during pipeline execution, but it’s important to adopt some best practices early on to keep things moving smoothly.

    Defining Variables

    Before we get too far, let’s take a moment to step back and talk about the motivations for variables in Azure Pipelines. We want to use variables for things that might change in the future, but more importantly we want to use variables to prevent secrets like passwords and API Keys from being entered into source control.

    Variables can be defined in several different places. They can be placed as meta-data for the pipeline, in variable groups, or dynamically in scripts.

    Define Variables in Pipelines

    Variables can be scoped to a Pipeline. These values, which are defined through the “Variables” button when editing a Pipeline, live as meta-data outside of the YAML file.

    image

    Define Variables in Variable Groups

    Variable Groups are perhaps the most common mechanism to define variables as they can be reused across multiple pipelines within the same project. Variable Groups also support pulling their values from an Azure KeyVault which makes them an ideal mechanism for sharing secrets across projects.

    Variable Groups are defined in the “Library” section of Azure Pipelines. Variables are simply key/value pairs.

    image

    image

    Variables are made available to the Pipeline when it runs, and although there are a few different syntaxes I’m going to focus on using what’s referred to as macro-syntax, which looks like $(VariableName)

    variables:
    - group: MyVariableGroup
    
    steps:
    - bash: |
         echo $(USERNAME)
         printenv | sort
    
    

    All variables are provided to scripts as Environment Variables. Using printenv dumps the list of environment variables. Both USERNAME and PASSWORD variables are present in the output.

    image

    Define Variables Dynamically in Scripts

    Variables can also be declared using scripts using a special logging syntax.

    - script: |
         $token = curl ....
         echo "##vso[task.setvariable variable=accesstoken]$token
    
    

    Defining Secrets

    Clearly, putting a clear text password variable in your pipeline is dangerous because any script in the pipeline has access to it. Fortunately, it’s very easy to lock this down by converting your variable into a secret.

    secrets

    Just use the lock icon to set it as a secret and then save the variable group to make it effectively irretrievable. Gandalf would be pleased.

    Why doesn't JWfan have a secure connection? - Other Topics - JOHN ...

    Now, when we run the pipeline we can see that the PASSWORD variable is no longer an Environment variable.

    image

    Securing Dynamic Variables in Scripts

    Secrets can also be declared at runtime using scripts. You should always be mindful as to whether these dynamic variables could be used maliciously if not secured.

    $token = curl ...
    echo "##vso[task.setvariable variable=accesstoken;isSecret=true]$token"
    
    

    Using Secrets in Scripts

    Now that we know that secrets aren’t made available as Environment variables, we have to explicitly provide the value to the script – effectively “opting in” – by mapping the secret to variable that can be used during script execution:

    - script : |
        echo The password is: $password
      env:
        password: $(Password)
    
    

    The above is a wonderful example of heresy, as you should never output secrets to logs. Thankfully, we don't need to worry too much about this because Azure DevOps automatically masks these values before they make it to the log.

    image

    Takeaways

    We should all do our part to take security concerns seriously. While it’s important to enable secrets early in your pipeline development to prevent leaking information, doing so will also prevent costly troubleshooting efforts when when variables are converted to secrets.

    Happy coding.

    Saturday, June 06, 2020

    Downloading Artifacts from YAML Pipelines

    Azure DevOps multi-stage YAML pipelines are pretty darn cool. You can describe a complex continuous integration pipeline that produces an artifact and then describe the continuous delivery workflow to push that artifact through multiple environments in the same YAML file.

    In today’s scenario, we’re going to suppose that our quality engineering team is using their own dedicated repository for their automated regression tests. What’s the best way to bring their automated tests into our pipeline? Let’s assume that our test automation team has their own pipeline that compiles their tests and produces an artifact so that we can run these tests with different runtime parameters in different environments.

    There are several approaches we can use. I’ll describe them from most-generic to most-awesome.

    Download from Azure Artifacts

    A common DevOps approach that is evangelized in Jez Humble’s Continuous Delivery book, is pushing binaries to an artifact repository and using those artifacts in ad-hoc manner in your pipelines. Azure DevOps has Azure Artifacts, which can be used for this purpose, but in my opinion it’s not a great fit. Azure Artifacts are better suited for maven, npm and nuget packages that are consumed as part of the build process.

    Don’t get me wrong, I’m not calling out a problem with Azure Artifacts that will you require you to find an alternative like JFrog’s Artifactory, my point is that it’s perhaps too generic. If we dumped our compiled assets into the artifactory, how would our pipeline know which version we should use? And how long should we keep these artifacts around? In my opinion, you’d want better metadata about this artifact, like source commits and build that produced it, and you’d want these artifacts to stick-around only if they’re in use. Although decoupling is advantageous, when you strip something of all semantic meaning you put the onus on something else to remember, and that often leads to manual processes that breakdown…

    If your artifacts have a predictable version number and you only ever need the latest version, there are tasks for downloading these types of artifacts. Azure Artifacts refers to these loose files as “Universal Packages”:

    - task: UniversalPackages@0
      displayName: 'Universal download'
      inputs:
        command: download
        vstsFeed: '<projectName>/<feedName>'
        vstsFeedPackage: '<packageName>'
        vstsPackageVersion: 1.0.0
        downloadDirectory: '$(Build.SourcesDirectory)\someFolder'
    
    

    Download from Pipeline

    Next up: the DownloadPipelineArtifact task is full featured built-in Task that can download artifacts from different sources, such as an artifact produced in an earlier stage, a different pipeline within the project, or other projects within your ADO Organization. You can even download artifacts from projects in other ADO Organizations if you provide the appropriate Service Connection.

    - task: DownloadPipelineArtifact@2
      inputs:
        source: 'specific'
        project: 'c7233341-a9ff-4e76-9367-909816bcd16g'
        pipeline: 1
        runVersion: 'latest'
        targetPath: '$(Pipeline.Workspace)'
    

    Note that if you’re downloading an artifact from a different project, you’ll need to adjust the authorization scope of the build agent. This is found in the Project Settings –> Pipelines : Settings. If this setting is disabled, you’ll need to adjust it at the Organization level first.

    image

    This works exactly as you’d expect it to, and the artifacts are downloaded to $(Pipeline.Workspace). Note in the above I’m using the project guid and pipeline id, which are populated by the Pipeline Editor, but you can specify them by their name as well.

    My only concern is there isn’t anything that indicates our pipeline is dependent on another project. The pipeline dependency is silently being consumed… which feels sneaky.

    build_download_without_dependencies

    Declared as a Resource

    The technique I’ve recently been using is declaring the pipeline artifact as a resource in the YAML. This makes the pipeline reference much more obvious in the pipeline code and surfaces the dependency in the build summary.

    Although this supports the ability to trigger our pipeline when new builds are available, we’ll skip that for now and only download the latest version of the artifact at runtime.

    resources:
     pipelines:
       - pipeline: my_dependent_project
         project: 'ProjectName'
         source: PipelineName
         branch: master
    
    

    To download artifacts from that pipeline we can use the download alias for DownloadPipelineArtifact. The syntax is more terse and easier to read. This example downloads the published artifact 'myartifact' from the declared pipeline reference. The download alias doesn’t seem to specify the download location. In this example, the artifact is downloaded to $(Pipeline.Workspace)\my_dependent_project\myartifact

    - download: my_dependent_project
      artifact: myartifact
    
    

    With this in place, the artifact shows up and change history appears in the build summary.

    build_download_with_pipeline_resource

    Update: 2020/06/18! Pipelines now appear as Runtime Resources

    At the time this article was written, there was an outstanding defect for referencing pipelines as resources. With this defect resolved, you can now specify the version of the pipeline resource to consume when manually kicking-off a pipeline run.

    1. Start a new pipeline run

      manually-trigger-build-select-resource
    2. Open the list of resources and select the pipeline resource

      manually-trigger-build-select-resource-pipeline
    3. From the list of available versions, pick the version of the pipeline to use:

      manually-trigger-build-select-resource-pipeline-version

    With this capability, we now have full traceability and flexibility to specify which pipeline resource we want!

    Conclusion

    So there you go. Three different ways to consume artifacts.

    Happy coding!

    Monday, February 10, 2020

    Challenges with Parallel Tests on Azure DevOps

    As I wrote about last week, Adventures in Code Spelunking, relentlessly digging into problems can be a time-consuming but rewarding task.

    That post centers around a tweet I made while I was struggling with an issue with VSTest on my Azure DevOps Pipeline. I'm feel I'm doing something interesting here: I've associated my automated tests to my test cases and I'm asking the VSTest task to run all the tests in the Plan; this is considerably different than just running the tests that are contained in the test assemblies. The challenge at the time was that the test runner wasn't finding any of my tests. My spelunking exercise revealed that the runner required an array of test suites despite the fact that the user interface restricts you to pick only one. I modified my yaml pipeline to contain a comma-delimited list of suites. Done!

    Next challenge, unlocked!

    Unfortunately, this would turn out to be a short victory, as I quickly discovered that although the VSTest task was able to find the test cases, the test run would simply hang with no meaningful insight as to why.

    [xUnit.net 00:00:00.00] xUnit.net VSTest Adapter v2.4.1 (64-bit .NET Core 3.1.1)
    [xUnit.net 00:00:00.52]   Discovering: MyTests
    [xUnit.net 00:00:00.57]   Discovered: MyTests
    [xUnit.net 00:00:00.57]   Starting: MyTests
    -> Loading plugin D:\a\1\a\SpecFlow.Console.FunctionalTests\TechTalk.SpecFlow.xUnit.SpecFlowPlugin.dll
    -> Using default config
    

    So, on a wild hunch I changed my test plan so that only a single test case was automated, and it worked. What gives?

    Is it me, or you? (it’s probably you)

    The tests work great on my local machine, so it’s easy to fall into a trap that the problem isn’t me. But to truly understand the problem is to be able to recreate it locally. And to do that, I’d need to strip away all the unique elements until I had the most basic setup.

    My first assumption was that it might actually be the VSTest runner -- a possible issue with the “Run Test Plan” option I was using. So I modified my build pipeline to just run my unit tests like normal regression tests. And surprisingly, the results were the same. So, maybe it’s my tests.

    Under a hunch that I might have a threading deadlock somewhere in my tests, I hunted through my solution looking for rogue asynchronous methods and notorious deadlock maker Task.Result. There were none that I could see. So, maybe there’s a mismatch in the environment setup somehow?

    Sure enough, I had some mismatches. My test runner from the command-prompt was an old version. The server build agent was using a different version of the test framework than what I had referenced in my project. After upgrading nuget packages, Visual Studio versions and fixing the pipeline to exactly match my environment – I still was unable to reproduce the problem locally.

    I have a fever, and the only prescription is more logging

    Well, if it’s a deadlock in my code, maybe I can introduce some logging into my tests to put a spotlight on the issue. After some initial futzing around (I’m amazing futzing wasn’t caught by spellcheck, btw), I was unable to get any of these log messages to appear in my output. Maybe xUnit has a setting for this?

    Turns out, xUnit has a great logging capability but requires a the magical presence of the xunit.runner.json file in the working directory.

    {
      "$schema": "https://xunit.net/schema/current/xunit.runner.schema.json",
      "diagnosticMessages": true
    }
    

    The presence of this file reveals this simple truth:

    [xUnit.net 00:00:00.00] xUnit.net VSTest Adapter v2.4.1 (64-bit .NET Core 3.1.1)
    [xUnit.net 00:00:00.52]   Discovering: MyTests (method display = ClassAndMethod, method display options = None)
    [xUnit.net 00:00:00.57]   Discovered: MyTests (found 10 test cases)
    [xUnit.net 00:00:00.57]   Starting: MyTests (parallel test collection = on, max threads = 8)
    -> Loading plugin D:\a\1\a\SpecFlow.Console.FunctionalTests\TechTalk.SpecFlow.xUnit.SpecFlowPlugin.dll
    -> Using default config
    

    And when compared to the server:

    [xUnit.net 00:00:00.00] xUnit.net VSTest Adapter v2.4.1 (64-bit .NET Core 3.1.1)
    [xUnit.net 00:00:00.52]   Discovering: MyTests (method display = ClassAndMethod, method display options = None)
    [xUnit.net 00:00:00.57]   Discovered: MyTests (found 10 test cases)
    [xUnit.net 00:00:00.57]   Starting: MyTests (parallel test collection = on, max threads = 2)
    -> Loading plugin D:\a\1\a\SpecFlow.Console.FunctionalTests\TechTalk.SpecFlow.xUnit.SpecFlowPlugin.dll
    -> Using default config
    

    Yes, Virginia, there is a thread contention problem

    The build agent on the server has only 2 virtual CPUs allocated and both executing tests are likely trying to spawn additional threads to perform the asynchronous operations. By setting the maxParallelThreads to “2” I am able to completely reproduce the problem from the server.

    I can disable parallel execution in the tests by adding the following to the assembly:

    [assembly: CollectionBehavior(DisableTestParallelization = true)]

    …or by disabling parallel execution in the xunit.runner.json:

    {
      "$schema": "https://xunit.net/schema/current/xunit.runner.schema.json",
      "diagnosticMessages": true,
      "parallelizeTestCollections": false
    }
    

    submit to reddit

    Friday, February 07, 2020

    Adventures in Code Spelunking

    image

    It started innocently enough. I had an Azure DevOps Test Plan that I wanted to associate some automation to. I’d wager that there are only a handful of people on the planet who’d be interested by this, and I’m one of them, but the online walk-throughs from Microsoft’s online documentation seemed compatible with my setup – so why not? So, with some time in my Saturday afternoon and some horrible weather outside, I decided to try it out. And after going through all the motions, my first attempt failed spectacularly with no meaningful errors.

    I re-read the documentation, verified my setup and it failed a dozen more times. Google and StackOverflow yielded no helpful suggestions. None.

    It’s the sort of problem that would drive most developers crazy. We’ve grown accustomed to having all the answers a simple search away. Surely others have already had this problem and solved it. But when the oracle of all human knowledge comes back with a fat goose egg you start to worry that we’ve all become a group of truly lazy developers that can only find ready-made code snippets from StackOverflow.

    When you are faced with this challenge, don’t give up. Don’t throw up your hands and walk away. Surely there’s an answer, and if there isn’t, you can make one. I want to walk you through my process.

    Read the logs

    If the devil is in the details, surely he’ll be found in the log file. You’ve probably already scanned the logs for obvious errors, it’s okay to go back and look again. If it seems the log file is gibberish at first glance, it often is. But sometimes the log contains some gems that give clues as to what’s missing. Maybe the log warns that a default value is missing, maybe you’ll discover a typo in a parameter.

    Read the logs, again

    Amp up the verbosity on the logs if possible and try again. Often developers use the verbose logging to diagnose problems that happen in the field, so maybe the hidden detail in the verbose log may reveal further gems.

    Now’s a good moment for some developer insight. Are these log messages helpful? Would someone reading the logs from your program be as delighted or frustrated with the quality of these output messages?

    Keep an eye out for references to class names or methods that appear in the log or stack traces. These could lead to further clues or give you a starting point for the next stage.

    Find the source

    Microsoft is the largest contributor to open-source projects on Github than anyone else, so it makes sense that they bought them. Just watching the culture shift within Microsoft in the last decade has been astounding and now it seems that almost all of their properties have their source code freely available for public viewing. Some sleuthing may be required to find the right repository. Sometimes it’s as easy as Googling “<name-of-class> github” or following the link on a nuget or maven repository.

    But once you’ve found the source, you enter a world of magic. Best case scenario, you immediately find the control logic in the code that relates to your problem. Worse case scenario, you learn more about this component than anyone you know. Maybe you’ll discover they parse inputs as case sensitive strings, or some conditional logic requires the presence of a parameter you’re not using.

    Within Github, your secret weapon is the ability to search within the repository, as you can find the implementation and usages in a single search. Recent changes within Github’s web-interface allows you to navigate through the code by clicking on class and method names – support is limited to specific programming languages but I’ll be in heaven when this capability expands. The point is to find a place to start and keep digging. It’ll seem weird not being able to set a breakpoint and simply run the app, but the ability to mentally trace through the code is invaluable. Practice makes perfect.

    If you’re lucky, the output from the log file will help guide you. Go back and read it again.

    As another developer insight – this code might be beautiful or make you want to vomit. Exposure to other approaches can validate and grow your opinions on what makes good software. I encourage all developers to read as much code that isn’t theirs.

    After spending some time looking at the source, check out their issues list. You might discover your problem is known by a different name that is only familiar to those that wrote it. Alternative suitable workarounds might appear from other problems.

    Roadblocks are just obstacles you haven’t overcome

    If you hit a roadblock, it helps to step back and think of other ways of looking at the problem. What alternative approaches could you explore? And above all else, never start from a position where you assume everything on your end is correct. Years ago when I worked part-time at the local computer repair shop, I learnt the hard way that the easiest and most blatantly obvious step, checking to see if it was plugged in, was the most important step to not skip. When you keep an open-mind, you will never run out of options.

    As evidenced by the tweet above, the error message I was experiencing was something that had no corresponding source-code online and all of my problems were baked into a black-box that only exists on the build server when the build runs. When the build runs… on the build server. When the build runs on the build agent… that I can install on my machine. Within minutes of installing a local build agent, I had the mysterious black-box gift wrapped on my machine.

    No source code? No problem. JetBrain’s dotPeek is a free utility that allows you to decompile and review any .net executable.

    Just dig until you hit the next obstacle. Step back, reflect. Dig differently. As I sit in a coffee shop looking out at the harsh cold of our Canadian winter, I reflect that we have it so easy compared to the original pioneers who forged their path here. That’s who you are, a pioneer cutting a path that no one has tread before. It isn’t easy, but the payoff is worth it.

    Happy coding.

    Thursday, February 06, 2020

    GoodReads 2019 Recap

    Hey Folks, like all posts that start in January, I’m starting my posts with the traditional …it’s been a while opener. Last year marks a first for this blog where I simply did not blog at all, which feels really strange. The usual suspects apply: busy at work, busy with kids, etc. However, in July of 2018 I started a new habit of taking a break from writing and focusing on reading more. I had planned to read 12 books in 2018 but read 20. Then I planned to read 24 in 2019 but read 41. I’ll probably have finished reading another book while I was writing this.

    Maybe your New Year’s Resolution is to read more books. So, here are some highlights of book I read last year that you might enjoy:


    The Murderbot Diaries

    The Murderbot Diaries

    Love, love, love Murderbot! By far, my favourite new literary character. The Murderbot diaries is set in the future where mankind has begun to explore planets beyond our solar system. If you were planning on exploring a planet, you’d hire a company to provide you with the assets to get there and as part of that contract, they’d provide you with a security detail to keep their assets you safe. Among that security detail is our protagonist, a security android that who has hacked his own governor module so it no longer needs to follow orders. What does a highly dangerous artificial intelligence with computer hacking skills and weapons embedded in its arms do with it’s own free will? Watch downloaded media and pretend to follow your orders. So. freaking. good.


    The Broken Earth Series

    The Broken Earth

    The Fifth Season is strange mix of fantasy meets apocalypse survival, this series is so brilliantly written that I got emotional when it ended. The world-building is vast and revealed appropriately as the story progresses but this attention to creativity does not overwhelm the characters’ depth or story arcs. The world, perhaps our own, is a distant future where history is lost. Artifacts of dead-civilizations, like the crystal obelisks that float aimlessly in the sky have no explanation and every few hundred years, the earth undergoes a geological disaster known as a Season. Seasons may last for years. This one, may last for centuries.

    Magic exists, but its source is a connection to the earth – an ability to delve, harness and channel the earth’s energy as a destructive force. For obvious reasons, those that are born with this ability are feared and thus rounded up and controlled by a ruling class. Our story involves a woman who secretly hides her ability and her kidnapped daughter who might be more powerful.

    Recursion

    Recursion: A Novel by [Crouch, Blake]

    Blake Crouch blew me away in 2018 with Dark Matter, Recursion follows the story of a detective who investigates the suicide of a woman who suffers from a disease that creates a disconnect between their memories and reality. Is it an epidemic or a conspiracy?


    The Southern Reach Trilogy (Annihilation, Authority, Acceptance)

    Area X Three-Book Bundle: Annihilation; Authority; Acceptance (Southern Reach Trilogy) by [VanderMeer, Jeff]

    I first heard of the book Annihilation from a CBC review of the bizarre and stunning visuals of the Annihilation movie starring Natalie Portman. The CBC review of the movie suggested that the director (Alex Garland) started the production of the movie before the 2nd and 3rd book of the series was written. Garland had support from the author, but it’s not surprising that the movie’s ending is radically different than the source material. I loved the movie, but needed to understand. The movie is a Kubrik mind-altering attempt to bring an unfilmable novel to the big screen, but the novel is so much more. The plot of the entire movie happens within the first few chapters, so if you liked the film the novel goes much further off the deep end. For example, the psychologist on the exhibition uses hypnosis and suggestive triggers on the rest of the exhibition to force compliance.  It’s not until our protagonist, the biologist, is infected by the effects of Area X does she become immune to the illusion.

    The insidious aspect is the villain is a mysterious environment with no face, presence or motive. How do you defeat an environment? (spoiler: you can’t.  The invasive species wins (annihilation), the people in charge that are hiding the conspiracy have no idea how to stop it (authority), and the sooner you come to terms with it the better you’ll be (acceptance) – and maybe, given what we’ve done to the environment in the past, we deserve the outcome)

    Sunday, March 11, 2018

    On Code Reviews

    Recently, a colleague reached out looking for advise on documentation for code reviews. It was a simple question, like so many others that arrive in my inbox phrased as a “quick question” yet don’t seem to have a “quick answer”. It struck me as odd that we didn’t have a template for this sort of thing. Why would we need this and under what circumstances would a template help?

    After some careful contemplation, I landed on two scenarios for code review. One definitely needs a template, the other does not.

    Detailed Code Analysis

    If you’ve been tasked with writing up a detailed analysis of a code-base, I can see benefit for a structured document template. Strangely, I don’t have a template but I’ve done this task many different times. The interesting part of this task is that the need for the document is often to support a business case for change or to provide evidence to squash or validate business concerns. Understanding the driving need for the document will shape how you approach the task. For example, you may be asked to review the code to identify performance improvements. Or perhaps the business has lost confidence in their team’s ability to estimate and they want an outside party to validate that the code follows sound development practices (and isn’t a hot mess).

    In general, a detailed analysis is usually painted in broad-strokes with high-level findings. Eg, classes with too many dependencies, insufficient error handling, lack of unit tests, insecure coding practices. As some of these can be perceived as the opinion of the author it’s imperative that you have hard evidence to support your findings. This is where tools like SonarCube shine as they can highlight design and security flaws, potential defects and even suggest how many hours of technical debt a solution has. Some tools like NDepend or JArchitect allow you to write SQL-like queries to find areas that need the most attention. For example, a query to “find highly used methods that have high cyclomatic complexity and low test coverage” can identify high yield pain points.

    If I was to have a template for this sort of analysis, it would have:

    • An executive summary that provides an overview of the analysis and how it was achieved
    • A list of the top 3-5 key concerns where each concern has a short concise paragraph with a focus on the business impact
    • A breakdown of findings in key areas:
      • Security
      • Operational Support and Diagnostics
      • Performance
      • Maintainability Concerns (code-quality, consistency, test automation and continuous delivery).

    Peer Code Review

    If we’re talking about code-review for a pull-request, my approach is very different. A template might be useful here, but it’s likely less needed.

    First, a number of linting tools such as FXCop, JSLint, etc can be included in the build pipeline so warnings and potential issues with the code are identified during the CI build and can be measured over time. Members of the team that are aware of these rules will call them out where appropriate in a code-reviews or they’ll set targets on reducing these warnings over time.

    Secondly, it’s best to let the team establish stylistic rules and formatting rather than trying to enforce a standard from above. The reasoning for this should be obvious: code style can be a religious war where there is no right answer. If you set a standard from above, you’ll spend your dying breath policing a codebase where developers don’t agree with you. In the end, consistency throughout the codebase should trump personal preference, so if the team can decide like adults which rules they feel are important then they’re less likely to act like children when reviewing each other’s work.

    With linting and stylistic concerns out of the way, what should remain is a process that requires all changes to be reviewed by one or more peers, and they should be reading the code to understand what it does or alternative ways that the author hadn’t considered. I’ve always seen code-review as a discussion, rather than policing, which is always more enjoyable.

    Thursday, October 19, 2017

    Xamarin.Forms with Caliburn.Micro walk-through

    As of this morning, I posted a new version of my Xamarin.Forms with Caliburn.Micro Starter Kit on the Visual Studio Marketplace.

    This video provides a quick walk-through of using the template.

    submit to reddit

    Monday, October 16, 2017

    Jump start your next project with Xamarin.Forms Caliburn.Micro Starter Kit

    Hey all! I’ve bundled my walkthrough of setting up a Xamarin.Forms to use Caliburn.Micro for Android, iOS and UWP into a Visual Studio Project Template and made it available in the Visual Studio Extensions Gallery

    You can download it directly here, or from within Visual Studio: Tools –> Extensions and Updates –> Online.

    Update 10/18/2017:

    • I had to republish the package as a “Tool” because it includes a few code snippets. The VS Gallery doesn’t allow you to change the classification of the VSIX, so I had to republish under a new identifier. You’ll need to uninstall and reinstall the new template.

    image

    As a multi-project, it’s very straight forward to use, simply choose File –> New and select “Xamarin.Forms with Caliburn.Micro”

    image

    Will create a project with the following structure:

    image

    Which, when run (on your platform of choosing) looks like this screenshot below. This is right where we left off from my walk-through earlier this year and is a great starting point for prototyping or building your next app.

    image

    Known Issues

    • Windows does not automatically create a signing key and identity for your app. Be sure to edit the UWP manifest and associate with your signing identity.

    The source code for the starter kit can be found here. Let me know what you think!

    submit to reddit

    Tuesday, October 10, 2017

    Bundle your Visual Studio Solution as a Multi-Project Template

    Earlier this year I provided a walkthrough of setting up a Xamarin.Forms project that leveraged Caliburn.Micro for Android, iOS and UWP. I had big plans for extracting the contents of that walkthrough and providing it as a NuGet package. Plans changed however, and I’ve decided to package the entire solution as a Multi-Project Template and provide it as an add-on to Visual Studio (VSIX). This post introduces provides a walk-through on how to create multi-project templates.

    Wait, why not NuGet?

    First off, as an aside, let’s go back and look what I wanted to do. I wanted to provide a starter-kit of files that would jump start your efforts and allow you to modify my provided files as you see fit. As a NuGet package, I can deliver these files to any project simply by adding these loose code files in the content folder of the NuGet package. Two things that are really awesome about this: the code files can be treated as source code transforms by changing their extension to *.pp, and through platform targeting I could deliver different content files per platform (Xamarin.iOS10, Xamarin.Android10, uap10.0, etc). With this approach, you would simply create a new Xamarin.Forms project then add the NuGet package to all projects. Bam. Easy.

    But there are a few problems with this approach:

    • Existing files. My NuGet package would certainly be replacing existing files in your solution. I’d want to overwrite key parts of the initial template (App.xaml, AppDelegate, Activity, etc) and in some cases delete files (MainPage.xaml). Technically, I can overcome these side-effects by modifying the project through a NuGet install script (install.ps1). However, you would be prompted during the install about the replacements and if you clicked ‘No’ when prompted to replace these files… my template wouldn’t work.
    • Delivering Updates. This is the funny thing about this approach -- it is really intended as a one time deal. You would add the starter files to your project and then begin to modify and extend to your hearts’ content. However, as the package author, no doubt I would find an issue or improvement for the package and publish it. If you were to update the package, it would repeat its initialization process and nuke your customizations. I would prefer not to see you when you’re angry.
    • Not guaranteed. Lastly, you could try and add the NuGet package to only one of your projects, or to a library that isn’t intended as a Xamarin.Forms project.

    Above all else, the NuGet documentation clearly states that these files should be treated immutable and not intended to be modified by the consuming project. And since the best place to add the package is immediately after you create the project using a Visual Studio Template, why not just make a Template?

    Creating a Multi-Project Template

    While Multi-Project Templates have been around for a while, their tooling has improved considerably over the last few releases of Visual Studio. Although there isn’t a feature to export an entire solution as a multi-project, they conceptually work the same way as creating a single project template and then tweaking it slightly.

    There are two ways to create a Project Template. The first and easiest is simply to select Project –> Export Template. The wizard that appears will prompt you for a Project and places your template in the My Exported Templates folder.

    The second approach requires you to install the Visual Studio SDK, which can be found as an option in the initial installer. When you have the SDK installed, you can create a Project Template as an item in your solution. This project includes the necessary vstemplate files and produces the packaged template every time you build.

    image

    Effectively, a Project Template is just a zip file with a .vstemplate file in it. A Multi-Project Template has a single .vstemplate that points to templates in subfolders. Here’s how I created mine:

    1. Create a Project Template project

    Using the Visual Studio SDK, I created a Project Template project to my solution and modified the VSTemplate file with the appropriate details:

    <VSTemplate Version="2.0.0" Type="ProjectGroup"
        xmlns="http://schemas.microsoft.com/developer/vstemplate/2005">
      <TemplateData>
        <Name>Xamarin.Forms with Caliburn.Micro</Name>
        <Description>Xamarin.Forms project with PCL library.</Description>
        <ProjectType>CSharp</ProjectType>
        <Icon>_icon.ico</Icon>
        <DefaultName>App</DefaultName>
        <ProvideDefaultName>true</ProvideDefaultName>
        <CreateNewFolder>true</CreateNewFolder>
        <RequiredFrameworkVersion>2.0</RequiredFrameworkVersion>
        <SortOrder>1000</SortOrder>
        <TemplateID>Your ID HERE</TemplateID>
      </TemplateData>
      <TemplateContent/>
    </VSTemplate>
    
    
    
    

    2. Export Projects and Add to the Project Template project

    Next, simply export all the projects in your solution that you want to include in your template. The Project –> Export Template dialog looks like this:

    image

    Once you’ve exported the projects as templates take each of the zip files and extract them into a subfolder of your Template Project. Then, in Visual Studio, include these extracted subfolders as part of the project. Note that Visual Studio will assign a default Action for each file, so code files will be set to Compile, images will be set as EmbeddedResource, etc. You’ll have to go through each of these files and change the default action to Content, copy if newer. It’s a pain, and I found it easier to unload the project and manually edit the csproj file directly.

    3. Configure the Template to include the embedded Projects

    Now that we have the embedded projects included in the output, we need to modify the template to point to these embedded templates. Visual Studio has a set of reserved keywords that can be used in the vstemplate and code transforms; $safeprojectname$ is a reserved keyword that represents the name of the current project. My vstemplate names the referenced templates after the name that was provided by the user:

    <VSTemplate Version="2.0.0" Type="ProjectGroup"
        xmlns="http://schemas.microsoft.com/developer/vstemplate/2005">
      <TemplateData>
        ...
      </TemplateData>
      <TemplateContent>
        <ProjectCollection>
          <ProjectTemplateLink ProjectName="$safeprojectname$" CopyParameters="true">XF\MyTemplate.vstemplate</ProjectTemplateLink>
          <ProjectTemplateLink ProjectName="$safeprojectname$.Android" CopyParameters="true">XF.Android\MyTemplate.vstemplate</ProjectTemplateLink>
          <ProjectTemplateLink ProjectName="$safeprojectname$.UWP" CopyParameters="true">XF.UWP\MyTemplate.vstemplate</ProjectTemplateLink>
          <ProjectTemplateLink ProjectName="$safeprojectname$.iOS" CopyParameters="true">XF.iOS\MyTemplate.vstemplate</ProjectTemplateLink>
        </ProjectCollection>
      </TemplateContent>  
    </VSTemplate>
    

    If the ProjectName is omitted, it will use the name within the embedded template.

    4. Fix Project References

    To ensure the project compiles, we must fix the project references to the PCL library in the iOS, Android and UWP projects. Here we leverage an interesting feature of Multi-Project templates – Visual Studio provides special reserved keywords for accessing properties of the root template project. In this case, we can reference the safeprojectname of the root project using the $ext_safeprojectname$ reserved keyword. And because project references use a GUID to refer to the referenced project, we can provide the PCL project with a GUID that will be known to all the child projects – in this case, we can use $ext_guid1$.

    The <ProjectGuid> element in the PCL Project must be configured to use the shared GUID:

    <PropertyGroup>
      <MinimumVisualStudioVersion>11.0</MinimumVisualStudioVersion>
      <Configuration Condition=" '$(Configuration)' == '' ">Debug</Configuration>
      <Platform Condition=" '$(Platform)' == '' ">AnyCPU</Platform>
      <ProjectGuid>{$ext_guid1$}</ProjectGuid>
      <OutputType>Library</OutputType>
      <AppDesignerFolder>Properties</AppDesignerFolder>
      <RootNamespace>$safeprojectname$</RootNamespace>
      <AssemblyName>$safeprojectname$</AssemblyName>
      <FileAlignment>512</FileAlignment>
      <TargetFrameworkVersion>v4.5</TargetFrameworkVersion>
      <TargetFrameworkProfile>Profile259</TargetFrameworkProfile>
      <ProjectTypeGuids>{786C830F-07A1-408B-BD7F-6EE04809D6DB};{FAE04EC0-301F-11D3-BF4B-00C04F79EFBC}</ProjectTypeGuids>
      <NuGetPackageImportStamp>
      </NuGetPackageImportStamp>
    </PropertyGroup>
    

    In the projects that reference the PCL, the path to the project, project GUID and Name must be also be modified:

    <ItemGroup>
      <ProjectReference Include="..\$ext_safeprojectname$\$ext_safeprojectname$.csproj">
        <Project>{$ext_guid1$}</Project>
        <Name>$ext_projectname$</Name>
      </ProjectReference>
    </ItemGroup>
    

    5. Fix-ups

    Lastly, there will be some other fix-ups you will need to apply. These are things like original project names that appear in manifest files, etc. The templating engine can make changes to any type of file, but you may need to verify that these files have the ReplaceParameters attribute set to True in the .vstemplate file.

    Build and Deploy!

    With this in place, you can simply compile the Project Template and copy the zip to ProjectTemplates folder. Optionally, you can add a VSIX project to the solution that you can use to bundle our Project Template as an installer that you can distribute to users via the Visual Studio Extensions Gallery.

    Happy coding!

    submit to reddit