Microsoft Cortana Intelligence Suite Workshop Video Tutorial Series (5/5): Predictive Web Service

The last part of this video tutorial series includes three exercises. First, Exercise 6 uses Power BI Desktop, import the summary data from the Spark cluster and create a report with drag-n-drop to visualize the data. Exercise 7 is the exciting part, configures and deploys a sample web app and configures it to consume the predictive web service published in Exercise 1, followed by conducting a few simple tests. Finally, Exercise 8 shows how to clean up the deployed resources of the workshop.

Here you start.

Microsoft Cortana Intelligence Workshop encompasses a set of processes and supporting tools to architect, construct, package and deploy a predictive analytics solution. It is a friendly platform with no hardware to purchase, no software to configure. The workshop ultimately deploys a web application with a predictive analytics service. The app predicts the total number and the probability of flight delays between two cities based on date, time, carrier and real-time forecast weather information. It is a relative simple project, however includes all the essential components to formulate a modern and intelligent application.

Microsoft Cortana Intelligence Suite Workshop Video Tutorial Series by Yung Chou

The workshop is intended to be delivered as a whole-day event with presentation sessions and lab time. On the other hand, within 75 minutes the above video tutorial series can also offer you an experience and guide you through all the screens and interactions to successfully deploy the web service.

The next step is to apply what learned from this series to your work. Good luck.

Advertisements

Microsoft Cortana Intelligence Suite Workshop Video Tutorial Series (4/5): Azure Spark Cluster

The objective of Exercise 5 is to create a table, then store and prepare summary data for later visualization. You will find out it is simple and straightforward using a Spark notebook to interactively work on an Azure Spark cluster.

This video tutor series presents the live demonstrations of all the exercises to facilitate the learning of Microsoft Cortana Intelligence Suite. There are 5 parts:

Microsoft Cortana Intelligence Suite Workshop Video Tutorial Series (3/5): Azure Data Factory

Machine Learning, predictive analytics, web services and all the rest to make it happen are really about one thing. And that is to acquire, process and act on data. For the workshop, this is done with a Data Factory pipeline configured to automatically upload a dataset to the storage account of a Spark cluster where Azure Machine Learning is integrated to score the dataset. Importantly, this addresses a fundamental requirement relevant to data-centric applications involved cloud computing. Which is to securely, automatically and on demand moving data between an on-premises location and a designated one in the cloud. For IT today, cloud can be a source, a destination and a broker of data and the ability to securely move data between an on-premises facility and a cloud destination is imperative for a hybrid cloud setting and a backup-and-restore scenarios. And Azure Data Factory is a vehicle to achieve that ability.

image

The workshop video tutorial series is as listed below:

Specifically, Exercises 2 -4 are to accomplish three things:

  • Creating an Azure Data Factory service and pairing which with a designated
    on-premises (file) server
  • Constructing an Azure Data Factory Pipeline to automatically and securely
    move data from the designated on-premises server to a target Azure blob storage
    account
  • Enabling the developed Azure Machine Learning model to score the date
    provided by Azure Data Factory pipeline

Notice that the lab VM is also employed as an on-premises file server hosting a dataset to be uploaded to Azure. At one moment, you may be using the lab VM as a workstation to access Azure remotely, and the next on an on-premises file server installing a gateway. When following the instructions, be mindful where a task is carried out, as the context switching is not always apparently.

Microsoft Cortana Intelligence Suite Workshop Video Tutorial Series (2/5): Azure Machine Learning

This video tutorial series walks you through the development of a predictive analytics solution using Microsoft Cortana Intelligence Suite. The solution is realized as a web application to predict the number of delays with probability of a flight segment between two cities with a particular airline at a particular date and time. The content of this series is based on what is published at http://aka.ms/CortanaManual by and thanks to Todd Kitta.

The first video is an introduction of how the workshop is structured and highlights a few important items to get you prepared. There are additional four to cover all eight exercises as

Here, the video walks through the process and operations in Exercise 1 to build an Azure Machine Learning model, as below,

image

and package it as a web service for consumption. If having not tried Microsoft Azure Machine Learning Studio before, I hope you will enjoy and appreciate the build-in canvas and native drag-and-drop capability for creating and composing a model. It let you explore and realize your creativities in multiple dimensions.

There are nine tasks total. Here we go.

So, what qualifies the machine as being able to learn? What is learning anyway? Look for my upcoming blog posts to examine the concept of “learning” and more.

Microsoft Cortana Intelligence Suite Workshop Video Tutorial Series (1/5): Introduction

This series, based on the content developed by Microsoft, offers a learning path with minimal time and effort to acquire the essential operation-level knowledge of Microsoft Cortana Intelligence Suite. The workshop steps through a process to construct and deploy a web application with predictive analytics, while along the way introducing key functional components. By specifying an origin and a destination airports, a future date and time and an airline carrier, this application predicts a flight delay with probability based on the weather forecast. The video tutorial series runs about 75 minutes and has captured exactly when and what you will see on the screen, where and how to respond based on the instruction of each exercise in the workshop.

I believe this series will most benefit those who function in a technical leadership capacity including: enterprise architect, solution architect, cloud architect, application architect, DevOps lead, etc. and are interested in the solution architecture of an application of predictive analytics. Going through the recordings will provide you an end-to-end view and clarity on how to constructing and deploying a predictive analytics solution, hence a better understanding on the processes and technologies, integration points, packaging and publishing, resource skill profiles, critical path, cost model, etc.

Cortana Intelligence Suite is a set of processes and tools. This workshop outlines an approach where analytic models, data, analysis, visualization, packaging, publishing and deployment are delivered in an integrated fashion. In my view, this is a productive and the right way to start learning how to architect a predictive analytics solution. The above video is the first of five to accelerate your learning of Cortana Intelligence Suite, and highlight a few important items before starting the workshop.

Content Repo

The content of this workshop made available by Todd Kitta is at http://aka.ms/CortanaManual in github. The readme file of the workshop details the scenario, architecture, prerequisites and a list of links to the instructions of all eight exercises.

image

The above architecture diagram of the workshop depicts the functional components for a web application with predictive analytics. Here the lab VM is also employed as an on-premises file server as the source of a data pipeline securely connected to a created Azure Data Factory service to automatically upload data to be scored by the Azure Machine Learning model. At the center is a Spark HDInsight cluster for data analysis, while the data are visualized by Power BI. The predictive analytics model is integrated and package as a web service consumed by a web application.

Introduction

Let’s first pay attention to a few important items before doing the workshop. There are eight exercises in this workshop and I have grouped them into five videos: an introduction and four learning units.

I recommend reading the instruction of an exercise in its entirety before doing the exercise, this will help set the context and gain clarity the objectives of each exercise. To do the workshop, one will need an active Azure subscription. Notice that a free trial account does provide sufficient credit for doing the entire workshop.

image

The workshop environment is a collection of resources deployed to Azure, as shown above, including:

  • A VM with Internet connectivity for a student to log in and work on all the exercises, such that there is no need to download or install anything locally for this workshop
  • A Machine Learning workspace accessed via Microsoft Azure Machine Learning studio to develop an experiment of predictive analytics
  • A Spark cluster for hosting and analyzing data including a scored dataset and a summary table
  • A number of storage accounts for storing workshop data

These resources do incur a cost. And to minimize the cost, try deploying the workshop environment only when you are ready to work on the exercises and delete it once completed the workshop. The deployment will take about fifteen minutes, if not more. And do deploy all resources and create services into the same resource group, so all can be later removed by simply deleting the resources group. Personally, when doing the workshop, I will set aside at least a four-hour block, find a quiet room and get a great cup of coffee. It is indeed a lot to consume.

Enjoy the workshop. Let’s get started.

An Introduction of Windows 10 Credential Guard

Windows 10 Enterprise has introduced a set of new security features including Credential Guard which is a key for securing derived credentials and defend ‘credential theft and reuse’ attacks like Pass-the-Hash (PtH) and Pass-the-Ticket. This article is to provide a technical background and highlights how Credential Guard works. A good reference titled “Protect derived domain credentials with Credential Guard” is also available in Microsoft TechNet Library. Note that Credential Guard is technically part of Device Guard to be detailed in an upcoming article.

LSASS, a Known Secret Service in Windows

Credential Guard is to secure the data kept by Local Security Authority (LSA) Subsystem Service (LSASS) which is a privileged process in Windows and for:

  • Validating users for local and remote sign-ins
  • Enforcing local security policies

And on an Active Directory domain controller, LSASS is also responsible for providing Active Directory database lookups, authentication, and replication. The following illustrates an instance of Windows 10 running OS Build 10586, while the desktop Task Manager reported the status of LSASS. Also hereafter, unless stated otherwise, LSA and LSASS are used interchangeably.

image

image

image

The above shows the Task Manager reporting LSASS.exe running in a Windows 10 Enterprise desktop prior to enabling Credential Guard.

The Secret Service’s Secrets

Those secrets for authenticating requests on behalf of users are hashes and Ticket Granting Tickets (TGTs) which the secret service protects. In this article, the focus is on a hash, while much of the information is applicable to TGTs as well.

Hash

The term, hash (or a password hash, a hash value), in the context of computer security is a derived credential, namely an encrypted form of a user’s plain text password. During a session, a protected resource upon being initially accessed by a user will first try to authenticate the user. And upon a successful authentication, the process then authorizes the request based on access rights granted to the user and those security groups the user is a member of for this resource. In Windows security model, a password is never stored in plain text, instead it first goes through a hash function, and the resulting hash value is what LSA stores for reuse in a subsequent authentication request. A password hash can be stored in one of four forms: LAN Manager (LM), NT, AES key, or Digest. Although over the years, recent versions of Windows do not store LM and NT hashes anymore, however for backward compatibility some out-dated applications may still cause which to be stored. Windows security model stores hashes in one of the two places:

  • a local Security Accounts Manager (SAM) database
  • a networked Active Directory database, as applicable, namely the NTDS.DIT of a domain controller

LSA Secrets

And based on the supported authentication protocols and methods of the OS, LSA loads corresponding authentication packages which know how to realize a particular hash. And as mentioned, there may be various hashes which LSA stores in an encrypted forms in the local device registry for various usage scenarios and backward supportability. These stored hashes and other security artifacts like TGTs are collectively referred as LSA secrets.

There is normally a time-to-live element associated with a hash. And before a stored hash expired, LSA can and will reuse it upon an authentication request, instead of repeatedly prompting the user for credentials, hence provide an SSO experience to the user. Detailed description of authentication artifacts and processes are beyond the scope of this article and available elsewhere.

Recently, SSO has become a crucial component for a baseline criteria for user experience and adopting an application. And as malware becomes increasingly sophisticated, LSA secrets have become targets for identity theft and reuse attacks with stealth.

Pass-the-Hash (PtH) Attack

Once a user is authenticated, depending on the scenarios, there are multiple forms of a hash, i.e. LSA secrets, stored in LSASS process memory on a user’s behalf. And there are also tools including PowerShell to extract them. A password hash is stored for subsequent authentication needs including storage access, network transmission, etc. for this user, rather than to repeatedly prompt the user for the password again during a session. This is significant for implementing SSO, at the same tim for identity theft and reuse as well.

What makes a hash an interesting hacking target is a stored hash represents an already authenticated user. Additionally, a hash can roam within a network for remote access. And by passing a hash when accessing a remote resource, an authentication process can proceed without the need to have the user whose password previously generated the hash aware of. Such that this attack can allow a hacker to traverse a network, infect the next note and look for higher value accounts. And repeat the process to eventually acquire an account with domain administrator rights to ultimately own the associated Active Directory domain. Notice that Accessing LSA secrets does require a local administrator-level access.

Conceptually, a PtH attack is to first hack a local administrator-level account for accessing LSA secrets, i.e. stored hashes, then impersonates a user by presenting the user’s hash for accessing a remote resource without the need to involve the user at all. Notice that from an OS point of view there is no difference if a hash is presented on behalf of a legitimate user or a hacker via unauthorized access. This is why a PtH attack is effective, hard to detect and an imminent threat to an entire network.

Ultimately, an unauthorized access to LSA secrets including password hashes and TGTs can lead to the so called PtH and Pass-the-Ticket (PtT) attacks, respectively. And accessing LSA secrets is a privileged operation and requires local administrator rights. In other words, to conduct a PtH attack a hacker must first hack/phish an account with local administrator rights.

Assume Breach Security Model

So to conduct PtH or PtT, a hacker will need to first acquire local administrator access. And it seems one logical and essential defense is to focus on protecting the credential of a local administrator account. This is a right approach, however not an effective way. Statistics has shown that many users uses the same password for accessing multiple sites, potentially corporate and external sites. With a rootkit which can be installed by a careless click on a phishing email, malware can potentially be loaded before OS code upon forcing a reboot and compromise a device at a boot time. Anti-virus, encryption, real-time monitoring, etc. are all based on a presumption that hardware and the loaded OS are trustworthy. This is apparently not necessarily always the case.

Today, with hundreds of thousands, yes, hundreds of thousands new malware everyday and the alarming infection rate which we have learned from the past, in addition to Advanced Persistent Threats (APTs) are on the rise, ensuring a device and its OS maintain a pristine state from power on to off is critical. IT has quickly learned in the past few years from some high profile breaches that getting hacked is a reality and highly likely once targeted. And a new normal for IT has become “You have been hacked, you just don’t know it yet.”

In other words, IT needs to essentially assume breach, and take a serious look on how to prevent PtH attacks in this setting. And the first has to be ensuring the hardware and the platform (i.e. OS runtime) are trustworthy.

First, Hardware Boot Integrity and OS Code Integrity

I put these two together since from a user’s point of view both seem appearing as one start-up process once powering on a device. By the time a user sees the welcome or logon screen and types in a user ID and a password, the OS has been loaded and the system processes are auto-started, delayed or stopped per associated configurations. An important presumption for security measures to work after a device starts has always been that collectively the hardware (i.e. firmware) is loaded without being tempered, the OS code is loaded without being altered and all defined processes are set as designed. In other words, for all the security measures like antivirus, security policies, etc. engaged or imposed after a device is booted to be effective, the device must maintain both hardware and platform integrity. So the runtime environment is trustworthy and with predictable behaviors to possibly monitor identify and detect malware.

Windows 10 Device Guard employs UEFI 2.3.1 secure boot and a number of measures to ensure boot integrity and OS code integrity. This is to be detailed in upcoming post and beyond the scope of this article. Here, the point is that a meaningful discussion of protecting hash values and defending PtH attacks must be based on a device with trusted firmware and OS code.

Protecting Derived Credentials (Hashes)

imageThe introduction of Credential Guard in Windows 10 Enterprise edition offers a new approach to increase the security of derived credentials, i.e. password hashes and Kerberos TGTs, with virtualization-based securities to mitigate PtH and PtT attacks. There are various editions in Windows 10 for home, business and education users. Notice that Credential Guard is to better secure credentials in a managed and networked environment and does require Windows 10 Enterprise or Education edition.

Windows 10 Credential Guard Architecture

Credential Guard uses Virtualization-Based Security (VBS) to isolate secrets so that only privileged system software can access them. The following illustrates a conceptual model. There are particular hardware and configuration requirements as shown. Specifically, UEFI 2.3.1 or later is required and has a different disk partitions with those of BIOS. IT needs to plan the deployment of UEFI to those devices accordingly for employing Credential Guard.

image

Prior to Windows 10, all derived credentials are kept by LSA. The danger is that upon successfully compromising LSA, a hacker can get full access to LSA secrets. In Windows 10, the LSA is moved into a separate container serving as a VBS environment and runs as LSAIso. Windows 10 marks the first version of Windows to leverage hardware to create an area of high isolation. This makes it impossible for hackers to steal derived credentials. Let’s take a deeper look of VBS.

Enabling Credential Guard

In a Windows 10 Enterprise device, with UEFI Secure Boot and the hardware assisted virtualization turned on in the boot configuration, the settings are in the policy, as shown below.

image

VBS requires Secure Boot, and can optionally be enabled with the use of DMA Protections, as illustrated below. DMA protection requires hardware support and will only be enabled on correctly configured devices.  On Code Integrity setting, this enables VBS to protect and ensure the integrity of Kernel Mode.

image

With Windows 10 Version 1511, Credential Guard now has an UEFI lock setting. IT should test the setting against management scenarios to ensure supportability. Once Credential Guard is enabled, after a reboot LSAIso.exe should run and appear in Task Manager, as shown below.

image

image

Concept of Virtualization-Based Security

With Credential Guard, the local LSASS now works with LSAIso, a new and isolated LSA process that runs, stores and protects in a virtualized, i.e. isolated, environment within the same device. Notice that LSAIso does not host any device drivers, but a small subset of OS binaries that are needed for security. All of these binaries are signed with a certificate that is trusted by VBS and these signatures are validated before launching the file in the protected environment. These measures make Windows 10 virtualization-based security highly secure with minimal attack surface.

Remote Procedure Call

When processing a hash, LSA issues RPC to LSAIso. RPC, a program communication model introduced in 1984, offers a mechanism allowing a program in one device to make a procedure call to a program/procedure running in a remote environment. The calling (or caller) program becomes suspended after making the call and resumed upon receiving the results passed back from the called (or callee) program. Essentially, RPC offers a mechanism which enables a distributed application appearing as if the entire application resides locally. The employment of RPC simplifies the logical design and facilitates the implementation of a distributed application.

From a PRC’s point of view, LSA and LSAIso are two processes running in two separated environments and isolated from each other. When processing hash, LSA perform a RPC call to LSAIso, and waits for LSAIso to passed back the results to continue.  Notice the secrets stored by LSAIso are  protected with VBS and is not accessible to the rest of the operating system. In other words, “who” can call “whom” for “what” and “how” in this model are well defined. Credential Guard further enhances security by not allowing older variants of NTLM and Kerberos authentication protocols and cipher suites when using default derived credentials, including NTLMv1, MS-CHAPv2, and weaker Kerberos encryption types, such as DES.

Closing Thoughts

Credential Guard isolates and protects LSA secrets against PtH and PtT which have become popular “credential theft and reuse” attacks. Like many other Windows 10 security features, Credential Guard does have specific hardware, software and configuration requirements. IT need to start planning early since the hardware and the UEFI Secure Boot requirements may require a hardware upgrade, a disk layout change from BIOS to UEFI, an architecture change from x86 to x64, etc. The immediate task is to build inventories and assess business needs for Windows 10 security features, followed by planning hardware refresh and rolling out UEFI sooner than later.

Be Strategic, While Facing a Growing Trend of BYOD

By itself, Credential Guard secures hashes which is an important, yet just one component for protecting identity and defending malware. In Windows 10, a suite of security features are available including Devise Guard, Windows Hello, Microsoft Passport and combined with Enterprise Mobility Management (EMM) suite offering a new approach on device security from power on to off. Such that IT has the ability to manage not only on-premises devices, but also BYOD-based deployment via cloud. The key consideration is to make sure all management solutions are in convergence, and not end up with one solution for on-premises, one for cloud, and one for mobile.

EMM is a key enabler for authenticating users and enforcing device policies via cloud. It is a vehicle to embrace BYOD with minimal changes needed on existing on-premises infrastructure. For a sizeable company, some form of EMM solution is evitable facing raid adoption of mobility and BYOD. IT leadership needs to develop a roadmap for adopting EMM sooner than later, and start transforming  into a “mobile first, cloud first” operation model as Microsoft so passionately advocates.

Call to Action

  • Learn Windows 10 which Microsoft Virtual Academy offers self-paced and free online courses in all Microsoft products, technologies and solutions including Windows 10, security and “Windows as a Service” as shown below.

image