A Secure Software Supply Chain with Containers

The concept of software supply chain is not a new one. What may be new is that CI/CD (Continuous Integration/Continuous Delivery) with containers makes it conceptually easy to understand and technically practical to implement. Here’s a process diagram illustrates this approach with five steps.

image

CI/CD Process

A software supply chain is here the “master” branch of a release, while development activities at other branch level are not considered here. The start of a master branch is where and when code or a change is introduced, while on the other end of the master branch is a production runtime environment where applications run. The process, as shown above, are highlighted in 5 steps.

  1. A CI/CD process starts by committing and pushing code to a centralized repository. Code here encompasses all programmatic assets relevant to defining, configuring, deploying and managing a relevant application workload.
  2. Changes made on the master branch triggers the CI process to automatically (re)build and (re)test all assets relevant to the workload which the master branch delivers.
  3. The successful outcome will generate a container images which are automatically versioned, tagged, registered and published in a designated trusted registry. In Docker Enterprise Edition, this is implemented as a Docker Trusted Registry, or DTR. The function of a trusted registry is to secure and host container images. Important tasks carried out here are to at a binary level scan for known malware, check vulnerabilities and digitally sign a container image upon being successfully processed. The generation of a container image signifies application assets are successfully integrated and packaged, which signifies the end of CI part.
  4. CD kicks off, executes and validates the steps for deploying the workload to a target production environment.
  5. Upon substantiating containers, referenced container images are then as needed pulled to a local host and start the container instances, hence an application or service.

Notice that Continuous Delivery is a reference of capability and not the state. Continuous Delivery signifies the ability to maintain payload at a production ready and deployable condition at all time. It is not necessary suggesting payload once validated is deployed to production immediately.

One Version of Truth

A software supply chain starts with developers commit and push the code into the master branch of a release’s centralized repository. To have one version of truth, centralized management is essential. Nevertheless, as preferred we can operate a centralized repository in a distributed fashion like github. Further, a centralize source code hosting solution must properly address these priorities including role-based access control, naming and branching, high availability, network latency, single-point-of-failure, etc. With source control, promoted code can be versioned and tagged for asset and release management.

Triggering upon Pushing Changes

When changes introduced into the master branch which is the supply chain, CI must and will automatically kick off validation process which includes a set of defined criteria, namely test cases.

Test-Driven, a Must

Once the development criteria (or requirements) are set, test cases can be developed accordingly prior to writing the code. And this essentially establishes the acceptance criteria and force a developer to focus on designing code guided by what must be later validated. Which in essence designs in the quality.

When changes are made, there is no point to “manually” execute all test cases against all the code. Let me be clear. The regression tests are necessary, but the repetitions with manual labor is counterproductive and error-prone. A better approach is to programmatically automate all structured tests (i.e. the criteria are structured and stable, like canned test cases) and let a tester to do exploratory testing which may not be performed with scripted, expected or even logical steps, but with the intent to break the system. The automation makes regression tests consistent and efficient, while exploratory testing adds extra values by expanding the test coverage.

Master Branch Has No Downtime

A test-driven development in CI/CD holds a key objective that the master branch is always functional and ready to deliver. This may first appear to some of us as idealistic and over-committed. While in fact, considering an automobile production line, as material and parts are put together while moving through one station to another, an automobile is being built. If at any time, a station breaks down, it must be fixed at the scene since the whole production line is on hold. Fixing what stops a subject moving from one station to the next on the spot is necessary to keep the production producing.

Software supply chain, or a CI/CD pipeline, with containers is a digital version of the above-mentioned model where artifacts are definitions, configurations and scripts. As these artifacts are integrated, built and tested throughout the pipeline, the process to construct a service based on containers is validated. If a step fails the validation, the pipeline stops and the issue must be immediately addressed and resulted, so the process can continue to the next step, hence material flows through the pipeline. To CI/CD, a master branch is the pipeline and must be always kept functional and ready to deliver.

Containers Are Not the Deliverables

It should be noted here that artifacts passing through the CI/CD pipeline are neither container images, nor container instances. What the pipeline validates are a set of developed definitions, configurations and processes based on application requirements and presented in mark-up language like json or yaml. In Docker, they are dockerfile, compose yaml file and template-based scripts, for example, to define application architecture with configured Docker runtime environment for a target application delivered as containers upon instantiation.

Container images and instances are in a way by-products. Container images and instances are however not intended to be deliverables. A container image generated by a CI/CD pipeline should always first programmatically created by the initial CI and later reference or updated by CD. The key is that images must be pulled or generated by executing CI. With Docker, thanks for natively configuration management as code, a release may employ a particular version of a container image. And upgrade or fall back a software supply chain may be as easy as changing the reference version, followed by redeploying the associated payload.

Trusted Registry, the Connective Tissue

CI once successfully generated a container image should register and upload the image to a trusted registry for security scanning and digital signing, before CD takes over and later pulls or updates the images, as needed to complete the CI/CD pipeline. Technically CI starts from receiving code changes and ends at successfully register a container image.

Fail to register a resulting container image will prevent CD from progressing upon referencing the image. In other words, a trusted registry is like a connecting tissue holding and keeping CI and CD fully synchronized and functional with the associated container images. A generated container image does not flow through every step of a CI/CD pipeline, the image is however the focal point to the validity of produced results. As shown in the above diagram, I used a dashed line between CI and CD to indicate there is a dependency of the trusted registry. Failing a registration will eventually fail the overall process.

Closing Thoughts

The essence of CI is automatic testing against defined criteria at unit, function and integration levels. These criteria are basically test cases which should be developed prior to code development as acceptance criteria. This is a key. Development must fully address these test criteria at coding time to build in quality.

Software supply chain is a better way. Wait, make that a much better way than just “developing” applications. I remember those days when every release to production was a nightmare. And code promotion was an event full of anxiety, numerous crashes and many long hours. Good riddance, so long those days. CI/CD with containers presents a very interesting and powerful combination for quality and speed, which is unusual to be able to achieve both at the same time. Docker with Jenkins, github and Azure, a software supply chain is realistic and approachable. Which I will detail in an upcoming post. Stay tuned.

Advertisements

Containerization Use Cases

The term, containerization, here denotes not only packaging an application as a container, but practicing DevOps and employing container artifacts to generate container images and instantiate instances accordingly throughout an application lifecycle. The following is a list capturing some pertinent information on containerization.

image

There are a number of use cases that containerization may play a significant role as show below. And I use a 5-step process flow to depict the concept of software supply chain with containerization. image

Use Cases

Here, for each use case I highlight the issues and concerns, what business may pursue, and how containers may help address the problems. It is crucial to first get a clear vision of what the problems are and achievable goals. Frankly, there is always a solution for a problem. Depending on the available resources, how bad and how soon it needs to be done, the path to Rome may vary. Containerization is not an one-for-all solution.

Equally important is to determine what to measure and do benchmark current state before implementing changes. Containerization can and will change how you deliver services. There will be noticeable and fundamental impact on organization structure, company culture and team dynamics. Further containers-based deployment is for speed and scale. From the surface, changes made may initially appear disorganized, chaotic and without governance. Many will question the manageability and feel discomfort since containerization like DevOps, microservices and other emerging technologies are indeed different and disrupting. Therefore, realizing the ROI becomes even more critical. A project team must determine and get agreements with stake holders on KPIs, track them closely, report often to prove the business benefits early. Doing so,will not only improve the quality delivered, but drive consensus, strengthen the confidence and help validate and align with business goals.

image

image

image

image

image

image

image

image

Microservices architecture

Regarding containerization relevant to microservices architecture, not only most of the listed use cases are also contributing factors, but microservices architecture is a topic by itself and there has been much material already published and available online. It is included here for completeness and I will leave microservices architecture for the reader to investigate.

Closing Thoughts

Containerization is a technology backed up by Google’s years of implementation experience and strong credit. The adoption in IT industry has been aggressive and skills of Docker containers are becoming commodity like tcpip, virtualization and cloud. Identifying use cases and defining KPIs are in my view essential before implementation. Hope this article provide some values for achieving these tasks.

In my next blog, I will address more on a secure software supply chain with containerization which also have mush relevance to microservices architecture. Stay tuned.

A Simulated IoT device with Node-RED

In the last few months, I have gradually shifted to use Node-RED as the tool for demonstrating and prototyping Azure IoT solutions. In particular, I configure a dashboard to display the ambient information sent from the device and verify the data received by an Azure IoT Hub and stored in an Azure storage account using Azure Storage Explorer form my desktop. Ideally, I would configure all on an Arduino or a Raspberry Pi. To make it more portable, I also do it with a local Ubuntu VM, so no need to plug in anything and I can demo a simple IoT setup anytime and anywhere on demand with Internet connectivity. Briefly, here’s an outline of what I did.

1. Installing & Starting Node-RED

On my Ubuntu (16.04 LTS) VM, update and upgrade everything, followed by install Node-Red.

If you need to make a require node module globally available in Node_RED, edit the file, ~/.node-red/settings.js accordingly. Here, I made the module, math.js, globally available and used it to round the ambient data to two decimal points.

imageimage

Now, start Node-RED, as the below.

image

As activities being carried out in Node-RED, this session displays the log with diagnostics in real-time.

In Ubuntu, when close out the terminal session running Node-RED, somehow it also stops the Node-RED service. This is different than how it behaves in Raspberry Pi where closing a Node-RED terminal session will not stop the service.

2. Accessing Node-RED IDE

The default port for Node-RED IDE is 1880, as shown below accessing the service from localhost. If preferred, authentication can be enabled and port changed by following what is stated in documentation and the above-mentioned settings.js file.

image

By default, there are a number of nodes installed as show on the left panel. And you may install addition nodes to better fit the needs.

3. Install additional Nodes

There are ample nodes and flows in Node-RED web site which you may install a from and contribute to. In addition to install these node modules with a command line interface, doing it interactively is also an option. In the IDE, click the upper right waffle within the Node-RED session and click ‘Manage palette’. If you do not see the option, update npm to the latest should make this option appear. As shown below, the Nodes tab presents the nodes installable directly or already installed currently. The

image

and on the install tab, you may keyword-search the Node-RED repository for relevant modules. Below, I search the modules relevant to Azure.

image

A few modules, I frequently install including:

4. Develop & Deploy a Node-RED Process Flow

To create a flow, start dragging selected nodes from the left panel to the canvas and construct flows by connecting the nodes. There copious amount of contents with how-to instructions on Node-RED in Internet already. Or if you like to do it in an old fashioned way, like me, by reading the document. The following is a simulated device with a few nodes to send and display ambient data to an Azure IoT Hub call thisiothub, as the following.

SNAGHTMLe2519f

Global Variables

Here, I added a config node to set the global variable to set the baseline temperature, humidity and pressure for a simulation run. Node-RED will always initialize a config node prior to executing all flows presented on the canvas.

Timestamp

The timestamp node sets the time interval for sending data. When developing and troubleshooting, I set it to a long period between messages to minimize the noises. When demoing, I will then set it based on a customer’s requirements. Each time, the timestamp triggers, the connected nodes are consequently executing the programmed the logic, respectively.

The Functions

In this setting, each emission by the timestamp node has the following effects.

  • This IoT Device function prepares the message payload and updates current ambient data which are global variables.
  • The temperature, humidity and pressure functions pipe the data stored in the global variables to a configured Node-RED dashboard.

This IoT Hub

This node has the host name of a target Azure IoT hub, here thisiothub, and the device connection information is provided in the function, This IoT Device.

msg.payload

This is a debug node. Once dragged to the canvas, it will automatically rename itself to msg.payload. Once connected, this node becomes a standardout of Node-RED. And you can examine the output in the debug tab in the right panel. In the screen capture above, you will find that I rounded the data to two decimal points and send it with mqtt.

SNAGHTML118120f

Gauges and Charts

A main reason motivating me to use Node-RED is the simplicity to configure and deploy a dashboard directly on an IoT device. Data visualization is essential for an IoT solution which is all about data. The ability to deploy a dashboard right there and when on demand is a significant time saver and a noticeable advantage. It did however took me some practices to correctly place those gauges and charts the way I wanted. Once configured, the dashboard is published automatically at http://node-red-instance/ui and here is what I got.

image

Verifying the Data Sent to Azure IoT Hub

There are two tools I use for managing and examining the activities between an IoT device and Azure IoT Hub. Namely, iothub-explorer for Linux and Device Explorer for Windows. The latter is a sharp tool for Windows users to examine data, .

image

And a convenient way to get the connection strings.

image

I also deployed a sample web app which plots the temperature and the humidity data received form thisiotdevice in real time, as shown.

image

So either from Azure IoT Hub or directly on the device, we may present the data visually.

Some Gotcha

Ubuntu frequently stopped Node-RED when a deployment had failed connecting to Azure IoT Hub, and the node will also lost the host name data. And I had to frequently restart the services and re-enter the Azure IoT Hub host name in the node configuration. And it is better to leave the terminal session where you started the Node-RED service visible at all time to restart the service as needed. I once spent hours troubleshooting a flow, researching material and was not able to figured out why, only to later find out the Node-RED service exited its session upon a failed deployment behind the scene.

Closing Thoughts

Node-RED is a great learning and prototyping tool. And once learned, you can create process logic based on data flows relatively easily. It is visual and a picture is always worth a thousand words

<

p align=”left”>Azure IoT Hub is the Swiss army knife for formulating an IoT solution. It does the heavy lifting for registering, securing and managing devices with interfaces to integrate other Azure or 3rd-party services. The recent announcement of Azure IoT Edge opens up many scenarios and opportunities to increase ROI by processing data right where they are collected. Which is what I plan to include to the next version of my Node-RED flow. Stay tuned.

NIST Guidance on Container Security

Here, a selected few of NIST documents which I’ve found very informative may help those who seek formal criteria, guidelines and recommendations for evaluating containerization and security.

NIST SP 800-190

Application Container Security Guide

Published in September of 2017, this document (800-190) reminds us the potential security concerns and how to address those concerns when employing containers. 800-190 details the major risks and the countermeasures of container technologies include image, registry, orchestrator, containers and host OS.

Worth pointing out that in section 6 of 800-190 recommends organizations should apply all, while listing out  exceptions and additions in planning and implementation to, the NIST SP 800-125 Section 5 recommendations in a container technology context.

NISTIR 8176

Security Assurance Requirements for Linux Application Container Deployments

Published in October of 2017, this document (8176) explains the execution model of Linux containers and assumes the attack model is that the vulnerability in the application code of the container or its faulty configuration has been exploited by an attacker. 8176 also examines securing containers based on hardware and configurations including namespace, cgroups and capabilities. Addressing the functionality and assurance requirements for the two types container security solutions, 8176 complements NIST 800-190 which provides the security guidelines and counter measures for application containers, .

NIST SP 800-180

NIST Definition of Microservices, Application Containers and System Virtual Machines

As of January of 2018, this document (800-180) is not yet finalized, while the draft was published in February of 2017 and the call for comments had ended in the following month.

The overwhelming interests on container technologies and their applications have energized organizations for seeking new and improved ways to add values to their customers and increase ROI. At the same time, as containers, containerization and microservices have become highly popular terms and over and over again being abused in our daily business conversations, the lack of rigorous and recognized criteria to clearly define what containers and microservices are has been in my view a main factor confusing and perhaps misguided many. For those who seek definitions and clarity before examining a solution, the agony of being in a state of confusion suffocated by the ambiguity of technical jargons indiscreetly applied to statements can be, or for me personally is, a very stressful experience. And some apparently has had enough and urged us that “There is no such thing as a microservice!”

With 800-180 serving a similar role to what NIST 800-145 to cloud computing, we now have a set of criteria to reference as a baseline for carrying out a productive conversation on containers., microservices and related solutions. And that’s a good thing.

NIST SP 800-125

Guide to Security for Full Virtualization Technologies

Like many NIST documents, this document (800-125) first gives the background information by explaining what full virtualization is, the motivations of employing it and how it works, before depicting the use cases, requirements and security recommendations for planning and deployment. Although today most business and technical professionals in the IT industry are to some degree versed in virtualization technologies. 800-125 remains an interesting read and provides an insight into virtualization and security. There are two associated documents, as below, point out important topics on virtualization to for a core knowledge domain of the subject.

  • NIST SP 800-125A Security Recommendations for Hypervisor Deployment on Servers
  • NIST SP 800-125B Secure Virtual Network Configuration for Virtual Machine (VM) Protection

Microsoft Cortana Intelligence Suite Workshop Video Tutorial Series (1/5): Introduction

This series, based on the content developed by Microsoft, offers a learning path with minimal time and effort to acquire the essential operation-level knowledge of Microsoft Cortana Intelligence Suite. The workshop steps through a process to construct and deploy a web application with predictive analytics, while along the way introducing key functional components. By specifying an origin and a destination airports, a future date and time and an airline carrier, this application predicts a flight delay with probability based on the weather forecast. The video tutorial series runs about 75 minutes and has captured exactly when and what you will see on the screen, where and how to respond based on the instruction of each exercise in the workshop.

I believe this series will most benefit those who function in a technical leadership capacity including: enterprise architect, solution architect, cloud architect, application architect, DevOps lead, etc. and are interested in the solution architecture of an application of predictive analytics. Going through the recordings will provide you an end-to-end view and clarity on how to constructing and deploying a predictive analytics solution, hence a better understanding on the processes and technologies, integration points, packaging and publishing, resource skill profiles, critical path, cost model, etc.

Cortana Intelligence Suite is a set of processes and tools. This workshop outlines an approach where analytic models, data, analysis, visualization, packaging, publishing and deployment are delivered in an integrated fashion. In my view, this is a productive and the right way to start learning how to architect a predictive analytics solution. The above video is the first of five to accelerate your learning of Cortana Intelligence Suite, and highlight a few important items before starting the workshop.

Content Repo

The content of this workshop made available by Todd Kitta is at http://aka.ms/CortanaManual in github. The readme file of the workshop details the scenario, architecture, prerequisites and a list of links to the instructions of all eight exercises.

image

The above architecture diagram of the workshop depicts the functional components for a web application with predictive analytics. Here the lab VM is also employed as an on-premises file server as the source of a data pipeline securely connected to a created Azure Data Factory service to automatically upload data to be scored by the Azure Machine Learning model. At the center is a Spark HDInsight cluster for data analysis, while the data are visualized by Power BI. The predictive analytics model is integrated and package as a web service consumed by a web application.

Introduction

Let’s first pay attention to a few important items before doing the workshop. There are eight exercises in this workshop and I have grouped them into five videos: an introduction and four learning units.

I recommend reading the instruction of an exercise in its entirety before doing the exercise, this will help set the context and gain clarity the objectives of each exercise. To do the workshop, one will need an active Azure subscription. Notice that a free trial account does provide sufficient credit for doing the entire workshop.

image

The workshop environment is a collection of resources deployed to Azure, as shown above, including:

  • A VM with Internet connectivity for a student to log in and work on all the exercises, such that there is no need to download or install anything locally for this workshop
  • A Machine Learning workspace accessed via Microsoft Azure Machine Learning studio to develop an experiment of predictive analytics
  • A Spark cluster for hosting and analyzing data including a scored dataset and a summary table
  • A number of storage accounts for storing workshop data

These resources do incur a cost. And to minimize the cost, try deploying the workshop environment only when you are ready to work on the exercises and delete it once completed the workshop. The deployment will take about fifteen minutes, if not more. And do deploy all resources and create services into the same resource group, so all can be later removed by simply deleting the resources group. Personally, when doing the workshop, I will set aside at least a four-hour block, find a quiet room and get a great cup of coffee. It is indeed a lot to consume.

Enjoy the workshop. Let’s get started.

Microsoft Azure Stack Technical Preview 1: Introduction & Feature Overview

This is something I had wanted to do for a while. Finally did welcome Charles Joy, a Principal Program Manager on the Azure Stack team, back to the show and we discussed the recent release of Microsoft Azure Stack Technical Preview 1. It’s a fun episode. Enjoy it.

image