Azure CLI for Deploying Customized Azure VMs

Posted on August 5, 2020 by yung chou

:'
This Azure CLI script is for ad hoc deploying customized Azure vms for testing including
- specified numebrs of vms and
- optionally a Bastion subnet for RDP/SSH over TLS directly with the Azure portal

To deploy,
1. Update the CUSTOMIZATION section, as preferred
2. Start an Azure Cloud Session,
   https://docs.microsoft.com/en-us/azure/cloud-shell/overview
3. Set the target subscription, if different form the current one
4. Copy and paste the statements of CUSTOMIZATION and STANDARDIZED ROUTINE to the Azure Cloud Shell session

© 2020 Yung Chou. All Rights Reserved.
'

# Session Start
az login

az account list -o table
subName='mySubscriptionName'
az account set -s $subName

################
# CUSTOMIZATION
################
prefix='da'

totalVMs=1
vmSize='Standard_B2ms'
region='southcentralus'
#az vm list-skus --location $region --output table
bastionSubnet='no'

# osType is a required setting
vmImage='ubuntults'
osType='linux'
#vmImage='win2016datacenter'
#vmImage='win2019datacenter'
#osType='windows'

# For testing
adminID='hendrix'
adminPwd='4testingonly!'
:'
Password must have the 3 of the following:
1 lower case character, 1 upper case character, 1 number and 1 special character

# if to interactively set
read -p "How many VMs to be deployed " totlaVMs
read -p "Enter the admin id for the $totalVMs VMs to be deployed " adminUser
read -sp "Enter the password for the $totalVMs VMs to be deployed " adminPwd
'
ipAllocationMethod='static'

# Use '' if not to open a service port
ssh=22
rdp=3389
http=80
https=443

#################################
# STANDARDIZED ROUTINE FROM HERE
#################################
echo "
Prepping for deploying:
$totalVMs $osType $vmImage vms in $vmSize size
each with $ipAllocationMethod public IP adderss
and port $ssh $rdp $http $https open
"

tag=$prefix$(date +%M%S)
echo "Session tag = $tag"

rgName=$tag
#echo "Creating the resource group, $rgName..."
az group create -n $rgName -l $region -o table
#az group delete -n $rgName --no-wait -y

# VIRTUAL NETWORK
vnetName=$rgName'-net'
subnetName='1' # 0..254
nsgName=$rgName'-vnet-nsg'
nsgRule=$rgName'-TestOnly'
priority=100

#echo "Creating the vnet, $vnetName..."
az network vnet create -g $rgName -n $vnetName -o none \
  --address-prefixes 10.10.0.0/16 \
  --subnet-name $subnetName --subnet-prefixes "10.10.$subnetName.0/24" 

# Bastion subnet
if [ $(echo $bastionSubnet | tr [a-z] [A-Z]) == 'YES' ]
then
  #echo "Adding the Bastion subnet..."
  az network vnet subnet create --vnet-name $vnetName -g $rgName -o none \
    -n AzureBastionSubnet --address-prefixes 10.10.99.0/24
fi

# NSG
#echo "Creating a NSG, $nsgName, associated with the vnet, $vnetName..."
az network nsg create -g $rgName -n $nsgName -o none
#echo "Creating a NSG rule, $nsgRule, associated with the NSG ,$nsgName..."
az network nsg rule create -g $rgName \
  --nsg-name $nsgName \
  -n $nsgRule \
  --protocol Tcp \
  --access Allow \
  --priority $priority \
  --destination-port-ranges $ssh $rdp $http $https \
  --description '*** FOR TESTING ONLY, NOT FOR PRODUCTION ***' \
  --verbose \
  -o table

# VM
time \
for i in `seq 1 $totalVMs`;
do

  vmName=$tag'-vm'$i
  echo "Prepping deployment for the vm, $vmName..."

  osDiskName=$vmName'-OSDisk'
  nicName=$vmName'-nic'
  vmIP=$vmName'-ip'

  az network public-ip create -g $rgName -n $vmIP \
    --allocation-method $ipAllocationMethod \
    --verbose \
    -o none
  echo "Allocated the $ipAllocationMethod public IP, $vmIP"

  az network nic create -g $rgName \
    -n $nicName \
    --vnet-name $vnetName \
    --subnet $subnetName \
    --network-security-group $nsgName \
    --public-ip-address $vmIP \
    --verbose \
    -o table
  echo  "Created the $nicName with the $ipAllocationMethod public IP, $vmIP"

  # CREATE VM AND RETURN THE IP
  if [ $(echo $osType | tr [a-z] [A-Z]) == 'LINUX' ]
  then
    echo "Configuring the Linux vm, $vmName, with password access"
    linuxOnly='--generate-ssh-keys --authentication-type all '
  else
    linuxOnly=''
  fi

  echo "Creating the vm, $vmName now..."
  pubIP=$(
    az vm create -g $rgName -n $vmName -l $region --size $vmSize \
      --admin-username $adminID --admin-password $adminPwd \
      --image $vmImage --os-disk-name $osDiskName \
      $linuxOnly \
      --nics $nicName \
      --query publicIpAddress \
      --verbose \
      -o tsv
  )
  #az vm show -d -g $rgName -n $vmName -o table
  echo  "
  Voilà! The VM, $vmName, has been deployed with the $ipAllocationMethod public IP, $pubIP
  "

done

# Deployed Resources
#az network vnet show -n $vnetName -g $rgName -o table
#az network vnet subnet list --vnet-name $vnetName -g $rgName -o table
#az network nic list -g $rgName -o table
az vm list -g $rgName -o table -d

# Clean up
:' To clean deployed resources
az group delete -n $rgName --no-wait -y
'

Finding an Azure VM Image Sku Using PowerShell

Posted on April 2, 2020 by yung chou

<#

The function, azure-vm-image-sku, returns the sku of a user-selected 
Azure VM image interactively. It calls the function, pick-one-item, 
which accepts an item-list and returns a selected item interactively.

This script is for demonstrating and learning Azure and PowerShell. 
The code is not optimized and does not handle all error messages. 

Usage:

# Get a sku in the default text mode
azure-vm-image-sku 

# Get a sku with GUI
azure-vm-image-sku -gui $true

# Get a sku with optional switches
azure-vm-image-sku `
  -region 'targetAzureRegion' `
  -publisher 'targetPublisherName' `
  -offer 'targetOffer' `
  -gui $true

Examples: 

azure-vm-image-sku -region 'south central us' -gui $true

azure-vm-image-sku `
  -region 'south central us' `
  -publisher 'microsoftwindowsserver'

azure-vm-image-sku `
  -region 'south central us' `
  -publisher 'microsoftwindowsserver' `
  -offer 'windowsserver' 

© 2020 Yung Chou. All Rights Reserved.

#>

function pick-one-item {

  param (
    [array  ]$thisList = @('East Asia','South Central US', 'West Europe', 'UAE North', 'South Afraica North'), 
    [string ]$itemDescription ='Azure Region', 
    [boolean]$gui = $false
    )

  if ($gui) {

    $thisOne = $thisList | Out-GridView -Title "$itemDescription List" -PassThru
  
  } else {
  
    if ($thisList.count -eq 1) { $thisOne = $thisList[0] 
    } else {
      
      $i=1; $tempList = @()

      foreach ( $item in $thisList )  {
          $tempList+="`n$i.`t$item" 
          $i++
      }

      do {
          write-host "`nHere's the $itemDescription list `n$tempList"
          $thePick = Read-Host "`nWhich $itemDEscription"
      } while(1..$tempList.length -notcontains $thePick)

      $thisOne = $thisList[($thePick-1)]
    }
  }

  write-host "$(get-date -f T) - Selecting '$thisOne' from the $itemDescription list " -f green -b black

  return $thisOne
  
}

function azure-vm-image-sku {

  param (

    [boolean]$gui = $false, 
    
    [string]$region = (pick-one-item `
      -thisList (Get-AzLocation).DisplayName `
      -itemDescription "Azure region" `
      -gui $gui ),

    [string]$publisher = (pick-one-item `
      -thisList (Get-AzVMImagePublisher -Location $region).PublisherName `
      -itemDescription "Azure $region publisher" `
      -gui $gui ),

    [string]$offer = (pick-one-item `
      -thisList (Get-AzVMImageOffer -Location $region -PublisherName $publisher).offer `
      -itemDescription "Azure $region $publisher's Offer" `
      -gui $gui ),  

    [string]$itemDescription = "Azure $region $publisher $offer Sku"

    )

  return $sku = (pick-one-item `
    -thisList (Get-AzVMImageSku -Location $region -PublisherName $publisher -Offer $offer).skus `
    -itemDescription $itemDescription `
    -gui $gui )

}

Creating Azure Managed Disk with VHD Using PowerShell

Posted on March 18, 2020 by yung chou


#region [CREATE MANAGED DISK WITH VHD]

write-host "
------------------------------------------------------------

This script is based on the following reference and for
learning Azure and PowerShell. I have made changes to the
original scrtip for clarity and portability.

Ref: Create a managed disk from a VHD file
https://docs.microsoft.com/en-us/azure/virtual-machines/scripts/virtual-machines-windows-powershell-sample-create-managed-disk-from-vhd

Recommend manually running the script statement by
statement in cloud shell.

© 2020 Yung Chou. All Rights Reserved.

------------------------------------------------------------
"

#region [CUSTOMIZATION]

#region [Needed only if an account owns multiple subscriptions]

Get-AzSubscription | Out-GridView  # Copy the target subscription name

# Set the context for subsequent operations
$context = (Get-AzSubscription | Out-GridView -Title 'Set subscription context' -PassThru)
Set-AzContext -Subscription $context | Out-Null
write-host "Azure context set for the subscription, `n$((Get-AzContext).Name)" -f green

#endregion

$sourceVhdStorageAccountResourceId = '/subscriptions/…/StorageAccounts/'
$sourceVhdUri = 'https://.../.vhd'

#Get-AzLocation
$sourceVhdLoc = 'centralus'

$mngedDiskRgName ="da-mnged-$(get-date -format 'mmss')"
#$mngedDiskRgName ='dnd-mnged'

#Provide the name of a to-be-created Managed Disk
$mngedDiskName = 'myMngedDisk'
$mngedStorageType = 'Premium_LRS' # Premium_LRS,Standard_LRS,Standard_ZRS
$mngedDiskSize = '128' # In GB greater than the source VHD file size

#endregion

if (($existingRG = (Get-AzResourceGroup | Where {$_.ResourceGroupName -eq $mngedDiskRgName})) -eq $Null) {
write-host "Resource group, $mngedDiskRgName, not found, creating it" -f y
New-AzResourceGroup -Name $mngedDiskRgName -Location $mngedDiskLoc
} else {
write-host "Using this resource group, $mngedDiskRgName, for the managed disk, $mngedDiskName" -f y
}

$diskConfig = New-AzDiskConfig `
-AccountType $mngedStorageType `
-Location $sourceVhdLoc `
-CreateOption Import `
-StorageAccountId $sourceVhdStorageAccountResourceId `
-SourceUri $sourceVhdUri

New-AzDisk `
-Disk $diskConfig `
-ResourceGroupName $mngedDiskRgName `
-DiskName $mngedDiskName `
-Verbose

#endregion [CREATE MANAGED DISK WITH VHD]

#region [CLean up]
# Remove-AzResourceGroup -Name $mngedDiskRgName -Force -AsJob
#endregion

Deploying Azure VM with Diagnostics Extension and Boot Diagnostics

Posted on March 9, 2020 by yung chou

This is a sample script for deploying an Azure VM with Diagnostics Extension and Boot Diagnostics, while each in a different resource group. The intent is to clearly illustrate the process with required operations, while paying minimal effort for code optimization.

Ideally an Azure VM, Diagnostic Extension, and Boot Diagnostics are to be deployed with the same resource group. However in production, it may be necessary to organize them into individual resource groups for standardization, which is what this script demonstrates.

The script can be run as it is. Or simply make changes in customization section and leave the rest in place. For VM Diagnostic Extension, the configuration file should be placed where the script is. Or update the variable, $diagnosticsConfigPath, accordingly. This script uses Storage Account Key for access which allows the storage account with a subscription different from that deploys the VM. A sample configuration file, diagnostics_publicconfig_NoStorageAccount.xml, is available and notice there is no <StorageAccount> element specified in this file.

Here’s the user experience up to finishing the [Deploying] section in the script. By default, an Azure VM is deployed with Boot Diagnostic enabled. The script upon a VM deployment changes and disables the Boot Diagnostic of the VM. For the following sample run, it took 3 minutes and 58 seconds.

Deploying Azure VM and setting Boot Diagnostics as disabled

Now with an Azure VM in place, the script adds VM Diagnostic Extension, followed by enabling Boot Diagnostics. Herr either extension uses a storage account in a resource group different form the VM’s. So this script creates 3 resource groups for: a VM itself, and the Diagnostics Extension and the Boot Diagnostics of the VM.

VM, Diagnostics, and Boot Diagnostics deployed with individual resource groups


write-host "
---------------------------------------------------------

This is a sample script for deploying an Azure VM
with Diagnostics Extension and Boot Diagnostics,
while each in a different resource group.

The intent is to clearly illustrate the process with
required operations, while paying minimal effort for
code optimization.

Ideally an Azure VM, Diagnostic Extension, and Boot Diagnostics
are to be deployed with the same resource group. However
in production, it may be necessary to organize them into
individual resource groups for standardization,
which is what this script demonstrates.

The script can be run as it is. Or simply make changes
in customization section, while leave the rest in place.
For VM Diagnostic Extension, the configuration file should
be placed where thi script is. Or update the variable,
$diagnosticsConfigPath, accordingly. This script uses a
Storage Account Key for access. This configuration allows
the storage account with a subscription different from that
deploys the VM. A sample configuration file,
diagnostics_publicconfig_NoStorageAccount.xml, is available at

https://1drv.ms/u/s!AuraBlxqDFRshVSl0IpWcsjRQkUX?e=3CGcgq

and notice there is no <StorageAccount> element specified in this file.

© 2020 Yung Chou. All Rights Reserved.

---------------------------------------------------------
"

Disconnect-AzAccount; Connect-AzAccount
# If multipel subscription
# Set-AzContext -SubscriptionId "xxxx-xxxx-xxxx-xxxx"

#region [Customization]

$cust=@{
initial='yc'
;region='southcentralus'
}

$diagnosticsConfigPath='diagnostics_publicconfig_NoStorageAccount.xml'

#region [vm admin credentials]

# 1.To hard-code
$cust+=@{
vmAdmin ='changeMe'
;vmAdminPwd='forDemoOnly!'
}
$vmAdmPwd=ConvertTo-SecureString $cust.vmAdminPwd -AsPlainText -Force
$vmAdmCred=New-Object System.Management.Automation.PSCredential ($cust.vmAdmin, $vmAdmPwd);
#>

# 2. Or interactively
#$vmAdminCred = Get-Credential -Message "Enter the VM Admin credentials."

#endregion

$tag=$cust.initial+(get-date -format 'mmss')
Write-host "`nSession ID = $tag" -f y

# Variables for common values
$vmRGName=$tag+'-RG'
$loc=$cust.region
$vmName=$tag+'vm'

$deployment=@{
vmSize='Standard_B2ms'
;dataDiskSzieInGB=5
;publisher='MicrosoftWindowsServer'
;offer='WindowsServer'
;sku='2016-Datacenter'
;version='latest'
;vnetAddSpace='192.168.0.0/16'
;subnetAddSpace='192.168.1.0/24'
}

#endregion

#region [Deployment Preping]

# Create a resource group
New-AzResourceGroup -Name $vmRGName -Location $loc
# Remove-AzResourceGroup -Name $vmRGName -AsJob

# Create a subnet configuration
$subnetConfig = `
New-AzVirtualNetworkSubnetConfig `
-Name 'default' `
-AddressPrefix ($deployment.subnetAddSpace) `
-WarningAction 'SilentlyContinue'

# Create a virtual network
$vnet = `
New-AzVirtualNetwork `
-ResourceGroupName $vmRGName `
-Location $loc `
-Name "$tag-vnet" `
-AddressPrefix $deployment.vnetAddSpace `
-Subnet $subnetConfig

# Create a public IP address and specify a DNS name
$pip = `
New-AzPublicIpAddress `
-ResourceGroupName $vmRGName `
-Location $loc `
-Name "$vmName-pip" `
-AllocationMethod Static `
-IdleTimeoutInMinutes 4

# Create an inbound network security group rule for port 3389
$nsgRuleRDP = `
New-AzNetworkSecurityRuleConfig `
-Name "$vmName-rdp" `
-Protocol Tcp `
-Direction Inbound `
-Priority 1000 `
-SourceAddressPrefix * `
-SourcePortRange * `
-DestinationAddressPrefix * `
-DestinationPortRange 3389 `
-Access Allow

# Create an inbound network security group rule for port 80,443
$nsgRuleHTTP = `
New-AzNetworkSecurityRuleConfig `
-Name "$vmName-http" -Protocol Tcp `
-Direction Inbound `
-Priority 1010 `
-SourceAddressPrefix * `
-SourcePortRange * `
-DestinationAddressPrefix * `
-DestinationPortRange 80,443 `
-Access Allow

$nsg= `
New-AzNetworkSecurityGroup `
-ResourceGroupName $vmRGName `
-Location $loc `
-Name "$vmName-nsg" `
-SecurityRules $nsgRuleRDP, $nsgRuleHTTP `
-Force

# Create a virtual network card and associate with public IP address and NSG
$nic = `
New-AzNetworkInterface `
-Name "$vmName-nic" `
-ResourceGroupName $vmRGName `
-Location $loc `
-SubnetId $vnet.Subnets[0].Id `
-PublicIpAddressId $pip.Id `
-NetworkSecurityGroupId $nsg.Id

$vmConfig = `
New-AzVMConfig `
-VMName $vmName `
-VMSize $deployment.vmSize `
| Set-AzVMOperatingSystem `
-Windows `
-ComputerName $vmName `
-Credential $vmAdmCred `
| Set-AzVMSourceImage `
-PublisherName $deployment.publisher `
-Offer $deployment.offer `
-Skus $deployment.sku `
-Version $deployment.version `
| Add-AzVMNetworkInterface `
-Id $nic.Id

#endregion

#region [Deploying]

$StopWatch = New-Object -TypeName System.Diagnostics.Stopwatch; $stopwatch.start()
write-host "`nDeploying the vm, $vmName, to $loc...`n" -f y

$vmStatus = `
New-AzVM `
-ResourceGroupName $vmRGName `
-Location $loc `
-VM $vmConfig `
-WarningAction 'SilentlyContinue' `
-Verbose

Set-AzVMBgInfoExtension `
-ResourceGroupName $vmRGName `
-VMName $vmName `
-Name 'bginfo'

$vm = Get-AzVM -ResourceGroupName $vmRGName -Name $vmName
# Set by default not to enable boot diagnostic
Set-AzVMBootDiagnostic `
-VM $vm `
-Disable `
| Update-AzVM
write-host "`nSet the vm, $vmName, with BootDiagnostic 'Disabled'`n" -f y

write-host '[Deployment Elapsed Time]' -f y
$stopwatch.stop(); $stopwatch.elapsed

#endregion

#region [Set VM Diagnostic Extension]
<# If using a diagnostics storage account name for the VM Diagnostic Extension, the storage account must be in the same subscription as the virtual machine. If the diagnostics storage account is in a different subscription than the virtual machine's, then enable sending diagnostics data to that storage account by explicitly specifying its name and key. #>
$vmDiagRGName=$tag+'vmDiag-RG'
$vmDiagStorageName=$tag+'vmdiagstore'

New-AzResourceGroup -Name $vmDiagRGName -Location $loc
#Remove-AzResourceGroup -Name $vmDiagRGName -AsJob

New-AzStorageAccount `
-ResourceGroupName $vmDiagRGName `
-AccountName $vmDiagStorageName `
-Location $loc `
-SkuName Standard_LRS

Set-AzVMDiagnosticsExtension `
-ResourceGroupName $vmRGName `
-VMName $vmName `
-DiagnosticsConfigurationPath $diagnosticsConfigPath `
-StorageAccountName $vmDiagStorageName `
-StorageAccountKey (
Get-AzStorageAccountKey `
-ResourceGroupName $vmDiagRGName `
-AccountName $vmDiagStorageName
).Value[0] `
-WarningAction 'SilentlyContinue'

$vmExtDiag = Get-AzVMDiagnosticsExtension -ResourceGroupName $vmRGName -VMName $vmName

#endregion

#region [Enable Boot Diagnostic]

# The resource group and the storage account are
# different from the vm's.

$vmBootDiagRGName=$tag+'bootDiag-RG'
$bootDiagStorageName=$tag+'bootdiagstore'

New-AzResourceGroup -Name $vmBootDiagRGName -Location $loc
#Remove-AzResourceGroup -Name $vmBootDiagRGName -AsJob

New-AzStorageAccount `
-ResourceGroupName $vmBootDiagRGName `
-AccountName $bootDiagStorageName `
-Location $loc `
-SkuName Standard_LRS

Set-AzVMBootDiagnostic `
-Enable `
-VM $vm `
-ResourceGroupName $vmBootDiagRGName `
-StorageAccountName $bootDiagStorageName `
| Update-AzVM

#endregion

#region [Session Summary]

($RGs = Get-AzResourceGroup | Where ResourceGroupName -like "$tag*") `
| ft ResourceGroupName, Location

($vms = Get-AzVM| Where ResourceGroupName -like "$tag*") `
| ft ResourceGroupName, Location, Name

($SAs = Get-AzStorageAccount | Where ResourceGroupName -like "$tag*") `
| ft ResourceGroupName, Location, StorageAccountName

#endregion

<# [Clean Up] 
Remove-AzResourceGroup -Name $vmRGName -AsJob 
Remove-AzResourceGroup -Name $vmDiagRGName -AsJob 
Remove-AzResourceGroup -Name $vmBootDiagRGName -AsJob 
#>

Azure, Ubuntu, and LVM on Encrypted Disks

Posted on December 29, 2019 by yung chou

:' 
Title: Azure, Ubuntu, and LVM on Encrypted Disks

Yung Chou © 2019

@yungchou
https://yungchou.wordpress.com
https://www.linkedin.com/in/yungchou/

This article depicts the logical flow with step-by-step
operations for configuring LVM on encrypted disks of an 
Azure Ubuntu VM. The configured LVM is a 3-disk stripe
set.
My intent is to provide clarity on the how-to with 
the following Azure CLI and BASH statements, while
considering little in optimizing and automating 
the process.

Notice that the flow of the steps and operations 
involves switching context between 

- a remote shell session to the Azure subscription 
- an ssh session to the deployed Azure Ubuntu VM

To follow the steps and execute the operations, first 
start a BASH shell session, log into Azure and set a 
subscription context, as applicable. Then simply 
populate the customization section with intended 
values, copy and paste selected statements into the 
Azure remote shell or an ssh session to the deployed 
Azure Ubuntu VM for execution, as directed.

STEP
0. LOG IN AZURE WITH A REMOTE SHELL SESSION
1. CUSTOMIZE SETTINGS
2. CREATE A RESOURCE GROUP FOR ALL RESOURCES
3. DEPLOY A VM FOR HOSTING LVM
4. ATTACH DATA DISKS 
5. ADD A KEY VAULT AND A KEY FOR ENCRYPTION

6. SSH INTO THE DEPLOYED VM,
   FORMAT AND MOUNT THE DATA DISKS,
   THEN EXIT AND BACK TO AZURE SESSION

7. BACK TO AZURE,
   ENABLE VM ENCRYPTION WITH KEK

8. SSH INTO THE DEPLOYED VM,
   CONFIGURE LVM, AND
   REBOOT THE VM

9. SSH INTO THE DEPLOYED VM UPON RESTART AND
   VERIFY LVM CONFIGURATION AND MOUNT POINT

X. OPTIONALLY CLEAN UP THE DEPLOYED RESOURCES

'
#---------------------------------------------
# 0. LOG IN AZURE WITH A REMOTE SHELL SESSION
#---------------------------------------------
az login -o table

# If multiple subscriptions, identifying an intended subscription
az account list -o table
# Then set the context
az account set --subscription 'subscription_name_or_id'

#-----------------------
# 1. CUSTOMIZE SETTINGS
#-----------------------
id=$(date +"%M%S") # Session ID

# Deploy a VM for hosting a LVM
rgName=rg$id
vmName=vm$id

sku='UbuntuLTS'  # Azure image sku, e.g. win2016datacenter
location='southcentralus'  # Azure region

# Listing VM sizes available in the Azure region
# az vm list-sizes -l $location | more

vmSize='Standard_B2ms'  # For configuring LVM, at least 8 GB RAM

diskSize=5    # GB
numOfDisks=3  # LUN {0...numOfdisks-1}
diskNamePrefix="disk$id-"

user='testing'
pwd='Notforproduction!'

kvName="kv$id"
keyName="key$id"
volType='DATA'      # Encrypting only data disks

#----------------------------------------------
# 2. CREATE A RESOURCE GROUP FOR ALL RESOURCES
#----------------------------------------------
az group create --name $rgName --location $location -o table

#--------------------------------
# 3. DEPLOY A VM FOR HOSTING LVM
#--------------------------------
az vm create \
    -g $rgName \
    -n $vmName \
    --image $sku \
    --size $vmSize \
    --admin-username $user \
    --admin-password $pwd \
    --verbose; \
az vm get-instance-view -g $rgName -n $vmName  -o table
#az vm list -g $rgName -d -o table

# VM Public IP
pubip=$(az vm show -d -g $rgName -n $vmName --query publicIps -o tsv)

#az vm open-port --port 80 --resource-group $rgName --name $vmName

#----------------------
# 4. ATTACH DATA DISKS 
#----------------------
let n=$numOfDisks-1

for lun in $(seq 0 $n)
do
az vm disk attach \
   --vm-name $vmName \
   -g $rgName \
   -n $diskNamePrefix$diskSize$lun \
   --size-gb $diskSize \
   --new \
   --verbose
done; \
az disk list -g $rgName -o table

#---------------------------------------------
# 5. ADD A KEY VAULT AND A KEY FOR ENCRYPTION
#---------------------------------------------
az keyvault create \
  -g $rgName \
  -n $kvName \
  --location $location \
  --enabled-for-disk-encryption \
  -o table \
  --verbose

az keyvault key create \
  --vault-name $kvName \
  --name $keyName \
  --ops decrypt encrypt sign unwrapKey verify wrapKey \
  --verbose
  
az keyvault key list --vault-name $kvName -o table

#----------------------------------------
# 6. SSH INTO THE DEPLOYED VM,
#    FORMAT AND MOUNT THE DATA DISKS,
#    THEN EXIT AND BACK TO AZURE SESSION
#----------------------------------------
user_at_ip=$user@$pubip

#-----------------------------------------
# Now ssh to the VM, format and mount the 
# disks before enabling encryption.
#-----------------------------------------
ssh $user_at_ip
clear
sudo su -

# No such file, if disks not added to the VM in Azure
ls -l /dev/disk/azure/scsi1/lun*
dmesg | grep SCSI

# Here, assuming disks already added in Azure portal as LUN {0...x}
echo 'y' | mkfs.ext4 /dev/disk/azure/scsi1/lun0
echo 'y' | mkfs.ext4 /dev/disk/azure/scsi1/lun1
echo 'y' | mkfs.ext4 /dev/disk/azure/scsi1/lun2

lsblk -f # FSTYPE and UUID populated

UUID0="$(blkid -s UUID -o value /dev/disk/azure/scsi1/lun0)"
UUID1="$(blkid -s UUID -o value /dev/disk/azure/scsi1/lun1)"
UUID2="$(blkid -s UUID -o value /dev/disk/azure/scsi1/lun2)"

mkdir /lun-0  # mount point for lun0
mkdir /lun-1  # moutn point for lun1
mkdir /lun-2  # moutn point for lun2

echo "UUID=$UUID0 /lun-0 ext4 defaults,nofail 0 0" >>/etc/fstab
echo "UUID=$UUID1 /lun-1 ext4 defaults,nofail 0 0" >>/etc/fstab
echo "UUID=$UUID2 /lun-2 ext4 defaults,nofail 0 0" >>/etc/fstab

more /etc/fstab | grep lun* 

mount -a      # Making sure disks mounted without issue
df -h /lun*   # Examining the mount point of each disk

exit  # exiting sudo
exit  # clsoing ssh and back to Azure remote shell session

#----------------------------------------
# 7. BACK TO AZURE,
#    ENABLE VM ENCRYPTION WITH KEK
#
#    The data disks to be encrypted must 
#    be mounted at this time in the VM.
#----------------------------------------
vaultResourceId=$(az keyvault show -g $rgName -n $kvName --query id -o tsv)

# The encryption may take a few minutes.
az vm encryption enable \
  -n $vmName \
  -g $rgName \
  --disk-encryption-keyvault $vaultResourceId \
  --key-encryption-keyvault $vaultResourceId \
  --key-encryption-key $keyName \
  --volume-type $volType \
  --encrypt-format-all \
  --verbose 

az vm encryption show \
  -g $rgName \
  -n $vmName \
  --query '[status,substatus]'

#------------------------------
# 8. SSH INTO THE DEPLOYED VM,
#   CONFIGURE LVM, AND
#   REBOOT THE VM
#------------------------------
ssh $user_at_ip
sudo su -

df -h /lun*

umount /lun-0  
umount /lun-1
umount /lun-2

fdisk -l | grep mapper
more /etc/crypttab  # Identifying the mapping

#----------------------------------------------
# Update the following mapping before creating
# physical volumes and volume group
#----------------------------------------------
echo 'y' | pvcreate /dev/mapper/???...???
echo 'y' | pvcreate /dev/mapper/???...???
echo 'y' | pvcreate /dev/mapper/???...???

volGroup="vg$id"
vgcreate $volGroup \
  /dev/mapper/???...??? \
  /dev/mapper/???...??? \
  /dev/mapper/???...???

vgdisplay -v $volGroup

logicalVol='stripes3'
lvcreate --extents 100%FREE --stripes 3 -n $logicalVol $volGroup
echo 'yes' | mkfs.ext4 /dev/$volGroup/$logicalVol

lvdisplay -a -m

mount -a
df -h /lun*

reboot

#----------------------------------------------
# 9. SSH INTO THE DEPLOYED VM UPON RESTRAT AND
#   VERIFY LVM CONFIGURATION AND MOUNT POINT
#----------------------------------------------
ssh $user_at_ip
sudo su -

lsblk -f
more /etc/fstab | grep mapper

# If all has gone well, LVM is up and running at this time.
vgdisplay -v $volGroup

exit # form sudo su -
exit # from the VM

#-----------------------------------------------
# X. OPTIONALLY CLEAN UP THE DEPLOYED RESOURCES
#-----------------------------------------------
az group delete -g $rgName --yes -y --no-wait
az logout

Feature Selection with Help from Boruta

Posted on November 19, 2018 by yung chou

Why

When developing a Machine Learning model, if there is a significant number of features to inspect, an initial and manual Exploratory Data Analysis may become tedious and nonproductive. One option is to facilitate the process by testing and identifying important variables based on statistical methods to help trim down features. And that is where Boruta comes in place.

What

A forest spirit in the Slavic mythology, Boruta (also called Leśny or Lešny) was portrayed as an imposing figure, with horns over the head, surrounded by packs of wolves and bears. Fortunately, in R Boruta is a helpful package for facilitating a feature selection process. Here’s a description from the documentation:

Boruta (CRAN) is an all relevant feature selection wrapper algorithm, capable of working with any classification method that output variable importance measure (VIM); by default, Boruta uses Random Forest. The method performs a top-down search for relevant features by comparing original attributes’ importance with importance achievable at random, estimated using their permuted copies, and progressively eliminating irrelevant features to stabilize that test.

How

The following is a sample routine in R demonstrating how I used Boruta to find a starting point for features selection. Some noticeable settings include:

The input data was train.imp.
doTrace=2 will log the activities and show progress to console.
maxRuns is how many times Boruta should run. In some circumstances (too short Boruta run, unfortunate mixing of shadow attributes, tricky dataset. . . ), Boruta may leave some attributes Tentative. For my particular case, the first 100 runs (which is a good initial value to start) confirmed most of the features with a few remain tentative. And I set it to 500 to finally resolve 80 features I was interested in and it took about half an hour.
TentativeRoughFix performs a simplified, weaker test for judging such attributes. This function should be used with discretion, since this weak test can lower the confidence of the final results.
getSelectedAttributes does what it sounds like.
attStats keep the statistics and the result of each resolved variable.

library(Boruta)

set.seed(1-0)
train.boruta <- Boruta(SalePrice~., data=train.imp, doTrace=2, maxRuns=500)

print(train.boruta)
plot(train.boruta , las=2, cex.axis=0.7, xlab='')
#plotImpHistory(boruta)

train.boruta.fix <- TentativeRoughFix(train.boruta)
train.boruta.selected.features <- getSelectedAttributes(train.boruta.fix, withTentative = F)

saveRDS(train.boruta.selected.features,'boruta/train.boruta.selected.features.rds')

train.boruta.selected.features.stats <- attStats(train.boruta.fix)
saveRDS(train.boruta.selected.features.stats, 'boruta/train.boruta.selected.features.stats.rds')

Also included are plots of Boruta output and attStats. Those confirmed important were in green and rejected in red. Unresolved variables were in yellow and classified as tentative which Boruta was not able to conclude their importance. And attStats kept and reported the statistics associated with the decisions.

Boruta uses Random Forest algorithm to provide educated sets of important and not so important features, respectively. Not only save time, but offer a repeatable and automatic way for initial exploratory data analysis.

Closing Thoughts

Feature selection is a critical task in developing a Machine Learning model. Extraneous features introduce multicollinearity, increase variance and lead to overfitting. Data is everything and feature selection is as critical. This is a task that can consume much of model development time. And for me, making the routine a code snippet and getting the mechanics in place help me become productive much quicker. A next logical step is to programmatically consume and integrate Boruta output to build and train a preliminary Machine Learning model to possibly establish a baseline of a target algorithm. Stay tuned for that.

Witnessing a clear cloudy day every day

Category Archives: code

Azure CLI for Deploying Customized Azure VMs

Finding an Azure VM Image Sku Using PowerShell

Creating Azure Managed Disk with VHD Using PowerShell

Deploying Azure VM with Diagnostics Extension and Boot Diagnostics

Azure, Ubuntu, and LVM on Encrypted Disks

Feature Selection with Help from Boruta

Why

What

How

Closing Thoughts

Share this:

Share this:

Share this:

Share this:

Share this:

Why

What

How

Closing Thoughts

Share this: