A glance at ONNX Runtime (ORT)

ONNX Runtime is a high-performance cross-platform inference engine to run all kinds of machine learning models. It supports all the most popular training frameworks including TensorFlow, PyTorch, SciKit Learn, and more. ONNX Runtime aims to provide an easy-to-use experience for AI developers to run models on various hardware and software platforms. Beyond accelerating server-side inference, ONNX Runtime for Mobile is available since ONNX Runtime 1.5. Now ORT Web is a new offering with the ONNX Runtime 1.8 release, focusing on in-browser inference.

In-browser inference with ORT Web

Running machine-learning-powered web applications in browsers has drawn a lot of attention from the AI community. It is challenging to make native AI applications portable to multiple platforms given the variations in programming languages and deployment environments. Web applications can easily enable cross-platform portability with the same implementation through the browser. Additionally, running machine learning models in browsers can accelerate performance by reducing server-client communications and simplify the distribution experience without needing any additional libraries and driver installations.

How does it work?

ORT Web accelerates model inference in the browser on both CPUs and GPUs, through WebAssembly (WASM) and WebGL backends separately. For CPU inference, ORT Web compiles the native ONNX Runtime CPU engine into the WASM backend by using Emscripten. WebGL is a popular standard for accessing GPU capabilities and adopted by ORT Web for achieving high performance on GPUs.

Figure 1: ORT web overview.

WebAssembly (WASM) backend for CPU

WebAssembly allows you to use server-side code on the client-side in the browser. Before WebAssembly only JavaScript was available in the browser. There are some advantages of WebAssembly compared to JavaScript such as faster load time and execution efficiency. Furthermore, WebAssembly supports multi-threading by utilizing SharedArrayBuffer, Web Worker, and SIMD128 (128-bits Single Instruction Multiple Data) to accelerate bulk data processing. This makes WebAssembly an attractive technique to execute the model at near-native speed on the web.

We leverage Emscripten, an open-source compiler toolchain, to compile ONNXRuntime C++ code into WebAssembly so that they can be loaded in browsers. This allows us to reuse the ONNX Runtime core and native CPU engine. By doing that ORT Web WASM backend can run any ONNX model and support most functionalities native ONNX Runtime offers, including full ONNX operator coverage, quantized ONNX model, and mini runtime. We utilize multi-threading and features in WebAssembly to further accelerate model inferencing. Note that SIMD is a new feature and isn’t yet available in all browsers with WebAssembly support. The browsers supporting new WebAssembly features could be found on the webassembly.org website.

During initialization, ORT Web checks the capabilities of the runtime environment to detect whether multi-threading and SIMD features are available. If not, there is a fallback version based on the environment. Taking Mobilenet V2 as an example, the CPU inference performance can be accelerated by 3.4x with two threads together with SIMD enabled, comparing the pure WebAssembly without enabling these two features.

Figure 2: 3.4x performance acceleration on CPU with multi-threading and SIMD enabled in WebAssembly (Test machine: Processor Intel(R) Xeon(R) CPU E3-1230 v5 @ 3.40GHz, 3401 Mhz, 4 Core(s), 8 Logical Processor(s)).

WebGL backend for GPU

WebGL is a JavaScript API that conforms to OpenGL ES 2.0 standard, which is supported by all major browsers and on various platforms including Windows, Linux, macOS, Android, and iOS. The GPU backend of ORT Web is built on WebGL and works with a variety of supported environments. This enables users to seamlessly port their deep learning models across different platforms.

In addition to portability, the ORT WebGL backend offers superior inference performance by deploying the following optimizations: pack mode, data cache, code cache, and node fusion. Pack mode reduces up to 75 percent memory footprint while improving parallelism. To avoid creating the same GPU data multiple times, ORT Web reuses as much GPU data (texture) as possible. WebGL uses OpenGL Shading Language (GLSL) to construct shaders to execute GPU programs. However, shaders must be compiled at runtime, introducing unacceptably high overhead. The code cache addresses this issue by ensuring each shader will be compiled only once. WebGL backend is capable of quite a few typical node fusions and has plans to take advantage of the graph optimization infrastructure to support a large collection of graph-based optimizations.

All ONNX operators are supported by the WASM backend but a subset by the WebGL backend. You can get supported operators by each backend. And below are the compatible platforms that each backend supports in ORT Web.

Figure 3: Compatible platforms that ORT Web supports.

Get started

In this section, we'll show you how you can incorporate ORT Web to build machine-learning-powered web applications.

Get an ONNX model

Thanks to the framework interoperability of ONNX, you can convert a model trained in any framework supporting ONNX to ONNX format. Torch.onnx.export is the built-in API in PyTorch for model exporting to ONNX and Tensorflow-ONNX is a standalone tool for TensorFlow and TensorFlow Lite to ONNX model conversion. Also, there are various pre-trained ONNX models covering common scenarios in the ONNX Model Zoo for a quick start.

Inference ONNX model in the browser

There are two ways to use ORT-Web, through a script tag or a bundler. The APIs in ORT Web to score the model are similar to the native ONNX Runtime, first creating an ONNX Runtime inference session with the model and then running the session with input data. By providing a consistent development experience, we aim to save time and effort for developers to integrate ML into applications and services for different platforms through ONNX Runtime.

The following code snippet shows how to call ORT Web API to inference a model with different backends.

const ort = require('onnxruntime-web');// create an inference session, using WebGL backend. (default is 'wasm')const session = await ort.InferenceSession.create('./model.onnx', { executionProviders: ['webgl'] });// feed inputs and runconst results = await session.run(feeds);

Figure 4: Code snippet of ORT Web APIs.

Some advanced features can be configured via setting properties of object `ort.env`, such as setting the maximum thread number and enabling/disabling SIMD.

// set maximum thread number for WebAssembly backend. Setting to 1 to disable multi-threadsort.wasm.numThreads = 1;// set flag to enable/disable SIMD (default is true)ort.wasm.simd = false;

Figure 5: Code snippet of properties setting in ORT Web.

Pre- and post-processing needs to be handled in JS before inputs are fed into ORT Web for inference. ORT Web Demo shows several interesting In-Browser vision scenarios powered by image models with ORT Web. You can find the code source including image input processing and inference through ORT Web. Another E2E tutorial is created by the Cloud Advocate curriculum team about building a Cuisine Recommender Web App with ORT Web. It goes through exporting a Scikit-Learn model to ONNX as well as running this model with ORT Web using script tag.

Figure 6: A cuisine recommender web app with ORT Web.

Looking forward

We hope this has inspired you to try out ORT Web in your web applications. We would love to hear your suggestions and feedback. You can participate or leave comments in our GitHub repos (ONNX Runtime). We continue to work on and improve the performance, model coverage as well as adding new features. On-device training is another interesting possibility we want to research for ORT Web. Stay tuned for our updates.

The post ONNX Runtime Web—running your machine learning model in browser appeared first on Microsoft Open Source Blog.

Spring is an important part of the Java ecosystem and we've been working together with VMware to make it easier for developers to run Spring applications. We started back in early 2016 with our Spring on Azure integrations with Pivotal. The developer response from our customers was very positive, and they asked us if we could address other challenges with running Spring apps at scale such as infrastructure management, dynamic scaling, monitoring and observability, application lifecycle management, and more.

In 2018 we started working on Azure Spring Cloud, a fully managed service for Spring Boot applications that lets developers focus on their apps. We announced the preview at SpringOne in 2019 and we've continued to improve the service and experience with new capabilities. This year at SpringOne, you'll see even more innovation to help enterprise developers run their Spring applications. Microsoft has our biggest presence ever at SpringOne this year and we're excited to share a range of sessions across topics, all in close partnership with VMware.

For more information, visit the Tech Community blog.

The post Join Microsoft and the Spring community at SpringOne 2021 appeared first on Microsoft Open Source Blog.

For a reasonably long time, DDP was only available on Linux. This was changed in PyTorch 1.7. In PyTorch 1.7 the support for DDP on Windows was introduced by Microsoft and has since then been continuously improved. In this article, we'd like to show you how it can help with the training experience on Windows.

Walkthrough

For reference, we'll set up two machines with the same spec on Azure, with one being Windows and the other being Linux, then perform model training with the same code and dataset.

We use this very nice resource in Azure called a Data Science Virtual Machine (DSVM). This is a handy VM image with a lot of machine learning tools preinstalled. At the time of writing, PyTorch 1.8.1(Anaconda) is included in the DSVM image, which will be what we use for demonstration.

You can search directly for this resource:

You can also follow the normal VM creation process and choose the desired DSVM image:

In this article, we use the size "Standard NC24s_v3", which puts four NVIDIA Tesla V100 GPUs at our disposal.

To better understand how DDP works, here are some basic concepts we need to learn first.

One important concept we need to understand is "process group", which is the fundamental tool that powers DDP. A process group is, as the name suggests, a group of processes. Each of the processes is responsible for the training workload of one dedicated GPU. Additionally, we need some method to coordinate the group of processes (more importantly, the GPUs behind them), so that they can communicate with each other. This is called "backend" in PyTorch (–dist-backend in the script parameter). In PyTorch 1.8 we will be using Gloo as the backend because NCCL and MPI backends are currently not available on Windows. See the PyTorch documentation to find more information about "backend". And finally, we need a place for the backend to exchange information. This is called "store" in PyTorch (–dist-url in the script parameter). See the PyTorch documentation to find out more about "store".

Other concepts that might be a bit confusing are "world size" and "rank". World size is essentially the number of processes participating in the training job. As we mentioned before, each process is responsible for one dedicated GPU. Thus, world size also equals to the total number of GPUs used. Pretty straightforward, right? Now let's talk about "rank". Rank can be seen as an index number of each process, which can be used to identify one specific process. Note that a process with rank 0 is always needed because it will act like the "controller" which coordinates all the processes. If the process with rank 0 doesn't exist, the entire training is a no-go.

With the necessary knowledge in our backpack, let's get started with the actual training. We use a small subset of ImageNet 2012 as the dataset. Let's assume we have downloaded and placed our dataset at some location in the filesystem, we'll use “D:\imagenet-small” for this demonstration.

Obviously, we also need a training script. We use the imagenet training script from PyTorch Examples repo and ResNet50 as the target model. The training script here can be seen as a normal training script, plus the DDP power provided packages like “torch.distributed” and “torch.multiprocessing”. The script doesn't contain too much logic and you can easily set up your own script based on it. You can also refer to this Getting Started tutorial for more inspiration.

On a single machine, we can simply use FileStore which is easier to set up. The complete command looks like this:

> python main.py D:\imagenet-small --arch resnet50 --dist-url file:///D:\pg --dist-backend gloo --world-size 1 --multiprocessing-distributed --rank 0

You probably noticed that we are using "world-size 1" and "rank 0". This is because the script will calculate the desired world size and rank based on the available GPUs. Here the actual world size used is the same as the number of GPUs available, which is four. The rank of each process will also be automatically assigned with the correct number, starting from zero.

If you're not a fan of command-line arguments, you can also use environment variables to initialize the DDP arguments. This might be helpful if you need to automate the deployment. More details can be found in the "Environment variable initialization" section of the PyTorch documentation.

If everything goes well, the training job will start shortly after.

Troubleshooting

If something doesn't go well, here are some troubleshooting tips that might be helpful:

• If you're using FileStore on Windows, make sure the file used is not locked by other processes, which can happen if you forcefully kill the training processes. This can lead to freezing of the DDP training process, because the script fails to initialize the FileStore. A workaround is to manually kill previous training processes and delete the file before you conduct the next training.
• If you're using TcpStore, make sure the network is accessible and the port is in fact available. Otherwise, the training may freeze because the script fails to initialize the TcpStore. The process with rank zero will bind and listen on the port you provided. Other processes will try to connect to that port. You can use network monitoring tools like "netstat" to help debugging the TCP connection issue.
• You can use tools like nvidia-smi to monitor the GPU load while performing the training. Ideally, we want all the GPUs fully utilized and running at 100 percent usage. If you find that the GPU load is low, you may want to increase the batch size and/or the number of DataLoader workers.
• Be aware that the number of GPUs used in DDP also affects the effective batch size. For example, if we use 128 as batch size on a single GPU, and then we switch to DDP with two GPUs. We have two options: a) split the batch and use 64 as batch size on each GPU; b) use 128 as batch size on each GPU and thus resulting in 256 as the effective batch size. Besides the limitation of the GPU memory, the choice is mostly up to you. You can tweak the script to choose either way. Remember to also adjust the initial learning rate if you choose option b) and expect a similar training result.

Benchmark

Back to our benchmarking mission. First, we tried to perform the training without using DDP to establish a baseline. Then we tried the DDP setup with two GPUs, then finally with four GPUs. These are the results:

To better visualize it, we plot it as the chart below:

As we can see from the data, the acceleration from additional GPUs meets our overall expectations. Using two GPUs cuts training duration to almost half. And using four GPUs makes it nearly one-quarter.

In terms of accuracy, here's the loss curve we see on both Windows and Linux:

We can tell from the loss curve that the shortening of training time does not end up with a bad training result. We can still expect the model to be gradually trained over time.

This is of course only a small demonstration of how DDP on Windows can bring users a performance boost that is comparable to the one on Linux, without compromising accuracy. We at Microsoft are working closely with PyTorch team to keep improving the PyTorch experience on Windows. The support of DDP on Windows is a huge leap ahead in terms of training performance. We'd like to encourage people to try it and we'd love to hear your feedback.

The post Introducing Distributed Data Parallel support on PyTorch Windows appeared first on Microsoft Open Source Blog.

Microsoft has worked with the Eclipse community for many years and we joined the Eclipse Foundation in 2016. Today I'm excited to share Microsoft is continuing to advance our support of the Eclipse Foundation AISBL by expanding our participation to a Strategic Member. I will also be joining the Foundation's board of directors and I look forward to working alongside our industry partners.

The Eclipse Foundation has a long history of providing a strong, collaborative culture supporting open-source-licensed projects, and many of those projects are important to Microsoft, our partners, and our customers. It is important for Microsoft to support the organization that supports those projects, and to work within the organization towards those collective goals.

Collaboration in community

The Eclipse Foundation has deep expertise in vendor-neutral governance, infrastructure, marketing, community building, and developer advocacy work. The team showed initiative and forethought, and pivoted to become a European-based international non-profit organization to align with its membership. The Eclipse Foundation is a natural place for Microsoft to collaborate on new initiatives beginning with European partners.

The Eclipse Foundation remains a vital cornerstone of the Java ecosystem. Microsoft is committed to Java developers and the health of the Java ecosystem, actively participating in Eclipse Adoptium (formerly AdoptOpenJDK) and other projects. Expanding our involvement with the Foundation as a Strategic Member will help advance modern Java initiatives in the spirit of open source.

The Eclipse Foundation also has close ties with core parts of the Java community with the Eclipse IDE, Jakarta EE (the successor to Java EE), and MicroProfile projects hosted there. For Microsoft and its partners, the Eclipse Foundation was the logical choice for AdoptOpenJDK to continue that mission. As a vendor-neutral, multi-vendor initiative, Eclipse Adoptium continues to be a leading provider of fully compatible, high-quality distributions of Java runtimes based on OpenJDK source code.

The Eclipse Foundation is expanding its role through working groups and many of these working groups are important to Microsoft and its partners. Recent work around the Eclipse Dataspace Connector and Eclipse Tractus-X are examples of new work beginning at the Eclipse Foundation in working groups in which Microsoft has an interest in participating.

Looking ahead

Open source non-profits serve an important role in the community, providing the structure to enable projects to reach their next opportunity of growth. Companies support such non-profits because it supports their engagement in the non-profit's projects, supports relationships with collaborating partners, and supports the developer communities at large. Having a rich ecosystem of healthy non-profits supporting different groups of Open-Source-Initiative-licensed projects and their project ecosystems is a must. At Microsoft, we are committed to continuing to support and participate across the non-profit ecosystem, as well as engage in projects themselves.

The post Microsoft expands support with The Eclipse Foundation appeared first on Microsoft Open Source Blog.

Training deep learning models requires ever-increasing compute and memory resources. Today we release torch_ort.ORTModule, to accelerate distributed training of PyTorch models, reducing the time and resources needed for training. To provide flexibility for the developer, torch-ort is available for both NVIDIA and AMD GPUs. The torch-ort package can be used with other deep learning optimizers like DeepSpeed to provide additional performance gains on training tasks.

Delivered via the torch-ort package from https://github.com/pytorch/ort, the ORTModule class is a simple wrapper for torch.nn.Module. ORTModule supports transformer models such as the GPT and BERT series, with support for other modalities to come soon. Today, you can fine-tune the most popular language models with a labeled dataset for a target task; augment self-supervised training of a model with a specific corpus, or experiment with pre-training new models from scratch.

Performance

As well as using torch-ort for large workloads inside Microsoft, we have benchmarked fine-tuning the most popular HuggingFace models, showing up to 37 percent improvement in throughput, with ORTModule alone, and up to 86 percent when combined with DeepSpeed.

These experiments were run on Azure's world-class Azure ND A100 v4 Infrastructure, with its optimal bandwidth between GPUs inside and across machines.

The graphs above show throughput in training samples per second. The actual time your training job takes depends on the number of training samples you have and the type of CPU/GPU you are using. Before starting the training processing, ORTModule does a one-time optimization of your model. This has a fixed cost that is amortized across the run.

The combination of ORTModule and DeepSpeed also enables Ask Here First to train the 2.7B parameter GPT-Neo model for a custom natural language task, where previously this large model could not be trained on the available hardware. AskHereFirst, a spin-off by Columbia University, uses powerful AI-based natural language query solutions for structured data stores that can be used to dramatically simplify the search in a wide range of industries such as finance, media, marketing, and sports.

Training GPT-NEO for our custom natural language task was not possible before we employed ORTModule and DeepSpeed. We have now produced fine-tuned 2.7B parameter GPT-NEO models to map natural language inputs into structured queries for a number of our applications."Professor Vishal Misra, Columbia University and founder, Ask Here First

Hardware portability

There are different hardware platform options for running distributed training workloads. The torch_ort.ORTModule works with NVIDIA and AMD GPUs.

We are releasing the torch-ort package for NVIDIA using CUDA 10.2 or CUDA 11.1. This can be used to accelerate the PyTorch training execution on both NVIDIA GPUs on Azure or on a user's on-prem environment.

We are also releasing the preview package for torch-ort with ROCm 4.2 for use on AMD GPUs.

Simple developer experience

Getting started with ORTModule is simple. You download and install the torch-ort package and wrap your model with ORTModule, as demonstrated in the following code example.

Your PyTorch training loop is unmodified except for wrapping the torch.nn.Module in ORTModule.

Because the PyTorch training loop is unmodified, ORTModule can be seamlessly integrated with other libraries in the PyTorch ecosystem, such as torch.autocast and NVIDIA apex.

How does it work?

On the first call to forward, two optimized computation graphs are generated: one for the forward prediction pass and one for the backward gradient calculation pass. All other parts of the training loop are executed by native PyTorch. The optimizations in these graphs, such as optimized kernels, subgraph operator fusion, and reduction of memory copies between CPU and GPU provide the speed up.

For more information

Go for a technical deep dive into ORTModule and read about our partnership with AMD. Also, see documentation, samples, or reach out to the team.

The post Accelerate PyTorch training with torch-ort appeared first on Microsoft Open Source Blog.

ONNX Runtime is an open-source project that is designed to accelerate machine learning across a wide range of frameworks, operating systems, and hardware platforms. Today, we are excited to announce a preview version of ONNX Runtime in release 1.8.1 featuring support for AMD Instinct GPUs facilitated by the AMD ROCm open software platform. Users can now use AMD Instinct GPUs with ONNX Runtime to accelerate distributed training for large-scale DNN models. AMD ROCm becomes the latest ONNX Runtime execution provider, continuing the Microsoft mission to endorse choice and versatility in targeting different compute devices and server platforms.

Figure 1: Selection interface showing AMD GPU support.

The ROCm Open Software Platform

ROCm is AMD’s open software platform for GPU-accelerated high-performance computing and machine learning workloads. Since the first ROCm release in 2016, the ROCm platform has evolved to support additional math, AI and machine learning, and communication libraries and tools, a wider set of Linux distributions, and a range of new GPUs. This includes the AMD Instinct MI100 GPU, the first AMD data center accelerator based on the compute-optimized AMD CDNA architecture.

The primary focus of ROCm has been high-performance computing at scale. The combined capabilities of ROCm and the AMD Instinct family of data center accelerators are well suited to accelerate AI/ML training using ONNX Runtime.

Accelerated training with ONNX Runtime on AMD GPUs

Large transformer models like GPT2 have proven themselves state of the art in natural language processing (NLP) tasks like NLP understanding, generation, and translation. They are also proving useful in applications like time-series prediction and computer vision. Due to their size, these models need to be trained in a largescale, distributed GPU environment. ONNX Runtime, with support from AMD (rocBLAS, MIOpen, hipRAND, and RCCL) libraries, enables users to train large transformer models in mixedprecision in a distributed AMD GPU environment. Thus, ONNX Runtime on ROCm supports training state-of-art models like BERT, GPT-2, T5, BART, and more using AMD Instinct GPUs. Data scientists, researchers, students, and others in the community have an option to accelerate workloads using ONNX Runtime on AMD GPUs. This includes AMD Instinct MI100, AMD Radeon Instinct MI50, and AMD Radeon Pro VII GPUs.

Today, we are happy to announce the preview of Python packages supporting ONNX Runtime on ROCm, making it easy to get started with ROCm and ONNX Runtime.

Training performance acceleration

In this preview, we have demonstrated clear performance gains with ONNX runtime using AMD GPUs for fine-tuning GPT2 using HuggingFace on eight AMD Instinct MI100 GPUs. We see an 18 percent performance gain in these experiments relative to standalone PyTorch along and validated well-matched loss curves.

Figure 2: Using ONNX runtime gets 18 percent perf gains over standalone PyTorch. Configuration details are listed below.

Figure 3: Training loss comparing the PyTorch and PyTorch and ONNX Runtime experiments.

In general, the preview ONNX Runtime-ROCm library can be used in multi-node MI100 AMD GPU configurations with high-speed interconnects for inter-GPU communications. As we proceed to our official release, we expect users to see excellent performance across a wide range of Transformer models and ML/AI workloads, offering users a highly performant choice for their datacenter applications.

Getting started with ONNX runtime on AMD GPUs

With Python packages

In a ROCm enabled environment, users can get off to a quick start with a pip install:

https://onnxruntimepackages.z14.web.core.windows.net/onnxruntime_stable_torch190.rocm42.html

Plugging their Pytorch script to ONNX runtime only* requires wrapping the model with

More details are available at pytorch/ort: Accelerate PyTorch models with ONNX Runtime.

*PyTorch model should use standard PyTorch to support export to ONNX.

With Dockerfiles

Users can also take advantage of a simple Dockerfile to get pre-configured packages of the ROCm libraries, Pytorch, and ONNX Runtime.

The stable ONNX runtime 1.8.1 release is now available at ort/Dockerfile.ort-torch181-onnxruntime-stable-rocm4.2-ubuntu18.04 at main pytorch/ort

More details are available at pytorch/ort.

More information about ONNX Runtime

• Read our recent blog, “ONNX Runtime 1.8: mobile, web, and accelerated training,” introducing the extended capabilities of ONNX runtime release 1.8.1.
• Check out examples demonstrating accelerating large transformer models using ONNX runtime.
• View instructions for accelerating general PyTorch workloads using ONNX runtime at Accelerate PyTorch models with ONNX Runtime.

More Information about ROCm Open Software Platform

• A list of ROCm supported GPUs and operating systems.
• General documentation on the ROCm platform.
• ROCm Learning Center.
• General information on AMD's offerings for HPC and machine learning.

Configuration and performance benchmarking

Hardware setup

• Server Type: HPE Apollo 6500 Gen10 Plus
• 8 x AMD Instinct MI100 with 2nd Gen Infinity Fabric Link (4 GPUs/ring) and PCIe Gen4 (across rings)
• GPU Memory: 32 GB
• CPU: 2 x AMD EPYC 7662 | AMD
• Main Memory: 512 GB (HPE 32GB 2Rx4 PC4-3200AA-R)
• SSD: HPE 1.92TB NVMe Read-Intensive Smart Carrier U.3 PE8010 SSD
• Ethernet: Intel I350 1GbE 4-port BASE-T

HuggingFace configuration

Repository

HuggingFace Transformers (branch blog-commit).

Dockerfile

ort/Dockerfile.ort-torch181-onnxruntime-stable-rocm4.2-ubuntu18.04

HuggingFace GPT2

The flag below enables wrapping with ONNX Runtime.

Author information: Jeff Daily is a Principal Member of Technical Staff, Deep Learning Software for AMD. Weixing Zhang is a Principal Software Engineer, AI Frameworks at Microsoft. Suffian Khan is a Software Engineer, AI Frameworks at Microsoft.

Their postings are their own opinions and may not represent AMD's or Microsoft's positions, strategies or opinions. Links to third-party sites are provided for convenience and unless explicitly stated, neither AMD nor Microsoft is responsible for the contents of such linked sites and no endorsement is implied.

The post ONNX Runtime release 1.8.1 previews support for accelerated training on AMD GPUs with the AMD ROCm™ Open Software Platform appeared first on Microsoft Open Source Blog.

About the co-author: Alejandro leads teams of machine learning engineers focused on the scalability and extensibility of machine learning deployment and monitoring products with over five million installations. Alejandro is also the Chief Scientist at the Institute for Ethical AI and Machine Learning, where he leads the development of industry standards on machine learning explainability, adversarial robustness, and differential privacy. With over 10 years of software development experience, Alejandro has held technical leadership positions across hyper-growth scale-ups and has a strong track record building cross-functional teams of software engineers.

As organizations adopt machine learning in production, they face growing challenges that arise when the number of production machine learning models starts to increase. In this article, we provide a practical tutorial that will enable AI practitioners to leverage production-ready workflows to deploy their machine learning models at scale. More specifically, we will demonstrate the benefits of open source tools and frameworks like ONNX Runtime, Seldon Core, and HuggingFace, as well as how these can be integrated with Azure Kubernetes Services to achieve robust and scalable machine learning operations capabilities.

By the end of this blog post, you will have a simple, repeatable, and scalable process to deploy complex machine learning models. You will learn by example, deploying the OpenAI GPT-2 natural language processing (NLP) model as a fully-fledged microservice with real-time metrics, and robust monitoring capabilities. Try out our GPT-2 Azure AKS Deployment Notebook that demonstrates the full process.

The steps that will be carried out in this blog are outlined in the image below, and include the following:

1. Fetch the pre-trained GPT2 Model using HuggingFace and export to ONNX.
2. Setup Kubernetes Environment and upload model artifact.
3. Deploy ONNX Model with Seldon Core to Azure Kubernetes Service.
4. Send inference requests to Kubernetes deployed GPT2 Model.
5. Visualize real-time monitoring metrics with Azure dashboards.

Furthermore, the tools that we'll be using in this framework will be the following:

• Seldon Core: A machine learning model deployment and monitoring framework for Kubernetes which will allow us to convert our model artifact into a scalable microservice with real-time metrics.
• ONNX Runtime: An optimized runtime engine to improve the performance of model inference, which we'll be using to optimize and run our models.
• Azure Kubernetes Service (AKS): Azure's managed Kubernetes service, where we will be running the deployed machine learning models.
• Azure Monitor: Azure's service for managed monitoring, where we will be able to visualize all the performance metrics.
• HuggingFace: An ecosystem for training and pre-trained transformer-based NLP models, which we will leverage to get access to the OpenAI GPT-2 model.

Let's get started.

1. Fetch the trained GPT-2 Model with HuggingFace and export to ONNX

GPT-2 is a popular NLP language model trained on a huge dataset that can generate human-like text. We will use Hugging Face's utilities to import the pre-trained GPT-2 tokenizer and model. First, we download the tokenizer as follows.

Now we can download the GPT2 Tensorflow model and export it to deploy it:

Finally, we convert it and optimize it for ONNX Runtime with the command below:

One of the main advantages of using the ONNX Runtime is the high-performance inference capabilities and broad compatibility that it brings. The ONNX Runtime enables practitioners to use any machine learning framework of their choice, and convert it to the optimized Open Neural Network Exchange (ONNX) format. Once these models are converted, the ONNX Runtime can be used to deploy it to a variety of targets including desktop, IoT, mobile, and in our case Azure Kubernetes through Seldon Core.

This framework has benefited from a broad range of rich benchmarks that showcase the high throughput and low latencies that can be achieved, which you can see in "Accelerate your NLP pipelines using Hugging Face Transformers and ONNX Runtime", as well as in "Faster and smaller quantized NLP with Hugging Face and ONNX Runtime".

2. Setup Kubernetes environment and upload model artifact

Seldon Core is one of the leading open-source frameworks for machine-learning model deployment and monitoring at scale on Kubernetes. It allows machine learning practitioners to convert their trained model artifacts or machine learning model code into fully-fledged microservices. All models deployed with Seldon are enabled with advanced monitoring, robust promotion strategies, and scalable architectural patterns. We will be using Seldon Core to deploy our GPT-2 model.

Seldon provides out-of-the-box a broad range of Pre-Packaged Inference Servers to deploy model artifacts to TFServing, Triton, ONNX Runtime, etc. It also provides Custom Language Wrappers to deploy custom Python, Java, C++, and more. In this blog post, we will be leveraging the Triton Prepackaged server with the ONNX Runtime backend. In order to set up Seldon Core in you can follow Seldon core setup instructions.

Setting Up the Azure Kubernetes Environment

The following diagram depicts our target architecture utilizing Azure Kubernetes Service (AKS)fully-managed Kubernetes service provided on Azure which removes the complexity of managing infrastructure and allows developers and data scientists to focus on machine learning models.

We recommend creating an AKS cluster with three node pools and installed CSI driverrefer to the following notebook for the scripts which you can run yourself (Azure Setup Notebook):

• CPU System Nodepool: running Kubernetes system components.
• CPU User Nodepool: running Seldon Operator and Istio components.
• GPU User Nodepool: running machine learning model inference with GPU hardware optimizations.
• Azure Blob CSI driver mounting Azure Storage Account for model hosting.

Upload model artifacts

Seldon is able to automatically download your model artifacts from an object store, so we will start by uploading our model in Azure blob storage. To abstract away details of storage connection from SeldonDeployment, we will be able to use the PersistentVolume reference in our model manifest, which will be mounted with our model container. For details on setting up PersistentVolume for Azure Blob with Blob CSI driver refer to our example here. We can then upload ONNX model file to Azure Blob following the default directory structure as per the Triton model repository format:

3. Deploy to Kubernetes (AKS) with Seldon Core

Now, we are ready to deploy our model using Seldon Core's Triton pre-packaged server. For that we need to define and apply a SeldonDeployment for prepackaged Triton server Kubernetes manifest file as shown below:

Some of the key attributes to notice:

• implementation field is set to `TRITON_SERVER`.
• 'model_url' points to PVC to download the model (<pvc>://<name>).
• 'name' field is GPT-2 to instruct what model to download.
• 'componentSpecs' override Pod spec fields such as limits/requests and tolerations to instruct Kubernetes scheduler to run the Pods on GPU nodes.
• 'protocol' field is using the widely adopted inference protocol.
• annotations direct Azure Monitor to scrape real-time metrics.

Once you deploy it, you can verify the logs as follows:

4. Run inference requests with your deployed model

Now that we have deployed our model, we are able to perform text generation in real-time. This can be done by sending REST requests directly against our productionized model; however, we'll have to carry out a couple of steps first: namely tokenization of our input, then sending the request, and then decoding the resulting tokens. This is shown in detail below.

• Tokenize the input sentence using Hugging Face GPT-2 pre-trained tokenizer:

• Now we can send the tokens by constructing the input payload:

• Our GPT-2 model will return the probability distribution for the next token over the vocabulary for the input vector (logits). Following the "greedy" approach we decode the response to a string and append to the input sentence.

• We repeat this to generate the full synthetic sentence:

The full end-to-end implementation could be found in our GPT-2 Notebook.

5. Visualize monitoring metrics with Azure Monitor

We are now able to visually monitor the real-time metrics generated by our Seldon Model by enabling Azure Monitor Container Insights in the AKS cluster. We can navigate to the insights blade page and check whether resources/limits configured for SeldonDeployment are within the healthy thresholds and monitor the changes in Memory or CPU during model inference.

In addition to container metrics, we can collect the specialized metrics generated by Seldon Triton orchestrator. Azure Monitor Container Insights provides out-of-the-box ability to scrape Prometheus metrics from declared endpoints, no need to install and operate Prometheus server. To learn more see Microsoft docsPrometheus metrics with Container insights. For our case we can now visualize the real-time metrics with specialized dashboards as demonstrated in our GPT-2 Notebook and the following example dashboard:

Uncover repeatable ML deployment processes today

In this blog, we have been able to cover a repeatable and scalable process to deploy a GPT-2 NLP model as a fully-fledged microservice with real-time metrics to enable observability and monitoring capabilities at scale using Seldon Core in Azure.

More specifically, we were able to:

• Fetch the trained GPT-2 Model with HuggingFace and export to ONNX.
• Setup Kubernetes Environment and Upload Model Artifact.
• Deploy ONNX Model with Seldon Core to Azure Kubernetes Service.
• Send requests to generate text with deployed GPT-2 Model.
• Visualize monitoring metrics with Azure dashboards.

These workflows are continuously being refined and evolved through the Seldon Core open source project, and advanced state-of-the-art algorithms on outlier detection, concept drift, explainability, and more are improving continuouslyif you are interested in learning more or contributing please feel free to reach out. If you are interested in further hands-on examples of scalable deployment strategies of machine learning models, you can check out:

• Production machine learning monitoring: Outliers, drift, explainers, and statistical performance.
• Real-time machine learning at scale using SpaCy, Kafka, and Seldon Core.
• Seldon Core quick-start documentation.

The post Simple steps to create scalable processes to deploy ML models as microservices appeared first on Microsoft Open Source Blog.

"With its resource-efficient and high-performance nature, ONNX Runtime helped us meet the need of deploying a large-scale multi-layer generative transformer model for code, a.k.a., GPT-C, to empower IntelliCode with the whole line of code completion suggestions in Visual Studio and Visual Studio Code."

Large-scale transformer models, such as GPT-2 and GPT-3, are among the most useful self-supervised transformer language models for natural language processing tasks such as language translation, question answering, passage summarization, text generation, and so on. After successfully shipping the first deep learning model for IntelliCode completion, our recent research effort brings GPT-C, a multi-layer generative decoder transformer architecture part of our DeepDev transformer platform for code and text from Developer Division (DevDiv) Data&AI Applied Science team, to empower IntelliCode with the whole line of code completion suggestions in Visual Studio and Visual Studio Code.

To meet the need of computing power required by large-scale transformers, our initial aim was to deploy the GPT-C model in production by leveraging Azure Machine Learning service with a cluster of virtual machines powered by NVIDIA Tesla V100 GPUs. However, there were some limitations:

• Cloud-based deployment requires transmitting user code over the network for inference, which increases the risks of exposing sensitive data.
• The service is not accessible in disconnected or offline mode. This limitation requires developers to stay connected to the internet during their work, which may not be a choice for people who work in areas with poor internet connections.
• Typical language models aim to generate full token sequences left-to-right using a beam search decoding algorithm to search for the best solutions in a batch-oriented manner. GPT-C is no exception. This scenario imposes a large memory overhead, resulting in high latency and serving costs. A 12-layer generative transformer model requires 374 MB in memory usage, takes around 80 ms GPU time per inference call. This cost of scaling it to our large user base would make it impractical.

With its resource-efficient and high-performance nature, ONNX Runtime can help address these limitations in GPT-C model production.

Large scale transformer model with ONNX Runtime

ONNX (Open Neural Network Exchange) and ONNX Runtime play an important role in accelerating and simplifying transformer model inference in production. ONNX is an open standard format representing machine learning models. Models trained with various frameworks, e.g. PyTorch, TensorFlow, can be converted to ONNX. Built based on the ONNX standard, ONNX Runtime is an optimized inference engine for efficiently running any model converted to the ONNX format across different hardware and operating systems with minimum effort. Due to this framework interoperability nature of ONNX, ONNX Runtime improves the development efficiency from model training to inference. Through various optimization techniques, ONNX Runtime can run all kinds of models with optimal performance across hardware platforms.

To deliver the IntelliCode line completion experience at a low cost, we decided to deploy GPT-C on the client-side. This means that the GPT-C model needs to be run on CPU efficiently with a wide range of client devices. Thanks to ONNX Runtime, our first attempt significantly reduces the memory usage from about 370MB to 80MB. ONNX Runtime enables transformer optimizations that achieve more than 2x performance speedup over PyTorch with a large sequence length on CPUs. PyTorch offers a built-in ONNX exporter for exporting PyTorch model to ONNX. On top of that, ONNX Runtime builds the GPT2 conversion tool for simplifying the conversion experience for GPT2 models with the past states. Our GPT-C transformer model is easily converted from PyTorch to ONNX by leveraging this tool, then runs with ONNX Runtime with good performance. In addition to the model itself, beam search is another important component in our deployment. In the initial version, beam search modules were implemented in managed code (C# and Typescript). It scores and re-ranks the output tensors received from the previous ONNX Runtime model inference step. When the scoring and re-ranking are done, the model retrieves the output tensors from the beam search module and conducts another round of inference. Due to the inefficiency of managed code implementation, the E2E client-side GPT-C inference suffers from a relatively poor response time of around 1 second CPU time for each line completion inference.

To improve the E2E performance of client-side GPT-C further, we extended the GPT2 conversion tool to support GPT-2 models with native one-step beam search. This was collaborative work between the DevDiv Data&AI Applied Science team, Microsoft Turing team, and the ONNX Runtime team. Consequently, we improved both aspects of training and of deploying GPT-2 models, which makes it simpler and more efficient for GPT-2 models with native one-step beam search to fully access hardware acceleration through ONNX Runtime.

One-step beam search optimization through ONNX Runtime for large scale transformer model

As shown in Figure 1, GPT-C is leveraging the native one-step beam search in its compute graph. Specifically, one-step beam search is compiled as TorchScript code that serves as a bridge between the GPT-C beam search module and ONNX Runtime. Then GPT2 conversion tool calls to the ONNX conversion APIs to convert one-step beam search into ONNX operators and appends to the end of the converted GPT-C transformer model ONNX compute graph. After GPT-2 models with native one-step beam search are converted to the whole ONNX graph, ONNX Runtime quantization is applied to further reduce the size of the model. When deploying the GPT-C ONNX model, the IntelliCode client-side model service retrieves the output tensors from ONNX Runtime and sends them back for the next inference step until all beams reach the end of the line.

Figure 1. How GPT-C Model deployed in Visual Studio and Visual Studio Code

We measured the latency of the GPT-C ONNX model on both CPU and GPU configurations. CPU performance measurement was done on a laptop machine with an Intel Core i7-8650U CPU. Compared with the initial attempt client-side GPT-C, performance gains up to 4.0x with around 300 ms per inference.

For GPU, we used one NVIDIA V100-PCIE-16GB GPU on an Azure Standard_NC12s_v3 VM and tested it in FP16 configuration. Compared with PyTorch, ONNX Runtime showed both significant memory efficiency and performance speedup with up to 5x and 4x, respectively.

Technical insights about one-step beam search ONNX optimization

Considering beam search requires multiple steps with certain stop conditions while the ONNX graph is static, we standardize the interface by exporting only one step of the beam search to ONNX. To enable multi-step beam search, all we need is a simple loop with a proper stop condition. Unfortunately, we ran into problems as the beam search algorithm requires loop operations in selecting beams and a set to store finished beams, which aren't natively supported in ONNX spec yet. To overcome this, at each step, we use two matrices to store the beam indices and scores at each step. In addition, we use a vector of indicators to track if the input beams are finished.

Figure 2. Left: Input beams. Right: candidate output beams. Meaning of tuples: (index, scores and or probabilities, finish and or unfinished). Here beam size (k) is 3 and index 2 is an <end-of-text> token. For the 3^rd row, the input is finished (i.e., reach <end-of-text>), so we construct and insert k "fake" candidates with the 1^st beam carrying the same score as the input with a pad index (or arbitrary index). The 2^nd and after beams are given a score of -Inf, which will be dropped when finding the top-k (shadowed) from all candidates.

As the example shown in Figure 2, input beams feed into the model to get a probability distribution of the next tokens. Since model inference is expensive, we only run the model on the unfinished beams, denoted as k0 (k0 = 2 in the example), and select top-k (the beam size) candidates for each unfinished input beam, which results in a k0 x k table. Then the finished beams are constructed and inserted back into the candidate pool or table by using the ONNX scatter operator. Therefore, we end up with a table of k x k candidates. For the next round of beam search, the next k-beams are selected by the top-k operator, which automatically discards finished beams with a -inf score, avoiding the use of branching yet to be supported in ONNX to handle finished and unfished inputs separately.

By putting beam search into the ONNX graph, we benefit from ONNX Runtime's optimization and reduce the overhead of transforming data between ONNX Runtime and the scripting language, which helps reduce model inference latency. Another benefit is to help bridge the gap between model training and deployment. As we know, it is common that the programming language for the large-scale transformer model in production may not be the same as the one used to train the model. For example, we found that most models are trained and tested using Python packages such as PyTorch but deployed in C#, C++, or Javascript packages. This means that the exact same beam search algorithm has to be implemented in different languages, which would cause inconsistency and maintenance issues. Given beam search algorithm is mostly standardized, exporting beam search directly to the ONNX graph avoids substantial code changes during deployment.

Try it now

We are delighted to offer this innovation to the public developer and data science community. You can now leverage high-performance inference with ONNX Runtime for a given GPT-2 model with one step beam search with the following steps:

1. Train a model with or load a pre-trained model from GPT-2.
2. Convert the GPT-2 model with one-step beam search to ONNX format.

Run the converted model with ONNX Runtime on the target platform of your choice. Check out this end-to-end tutorial.

Ongoing work

We will continue optimizing the performance of the large-scale transformer model in ONNX Runtime. There are still opportunities for further improvements, such as integrating the multi-step beam search into the ONNX model.

We have completed the internal preview of IntelliCode line completion and released it to preview in Visual Studio 2022.

The post Journey to optimize large scale transformer model inference with ONNX Runtime appeared first on Microsoft Open Source Blog.

Easily migrate and modernize Linux and open source databases to Azure

Azure Migrate is your one-stop-shop in Azure for migrating and modernizing your virtual machines like Windows or Linux Servers, databases, data, web apps, and virtual desktops. Azure Migrate features free Azure migration tools with features like agentless datacenter discovery, Azure readiness analysis, cost estimation, app modernization, and app dependency visualization as well as popular migration tools from our ISV partners to help you in the discovery, assessment, and migration phases of your migration and modernization journey in one central location with end-to-end visibility.

In the below demo video, we migrated and modernized an open-source Java app, Airsonic, and its backend MySQL database, both running on-premises on Linux Virtual Machines, to Azure. To modernize the MySQL database, we moved the data from the on-premises virtual machine into Azure Database for MySQL, using the Azure Database Migration Service. Azure Database for MySQL is a managed database, so once you have the data in Azure Database for MySQL, you don't have to worry about managing a virtual machine and you get the benefits of built-in scalability, high availability, enterprise-grade SLAs, and cost optimization.

For the modernizing the app, we containerized it using the Azure Migrate App Containerization tool, so you can achieve faster application development cycles, ease of deployment, and quick scalability offered by containers, all without making any code changes to the app. Also, check out this MySQL migration guide, to get detailed step-by-step guidance on how to migrate MySQL workloads to Azure Database for MySQL.

To learn more, watch the Microsoft Mechanics video below, which shows you step-by-step how to migrate and modernize your Linux and open source databases to Azure.

Get full support for Linux and high availability, industry-leading SLAs for open source databases

Azure supports all major Linux distributions including Red Hat, SUSE, Ubuntu, CentOS, Debian, Oracle Linux, and Flatcar Linux and open-source databases like MySQL, PostgreSQL, Cassandra, MariaDB, and more. More than 60 percent of Azure Marketplace solutions run on Linux. Then, beyond the workload level, Azure also contributes back to the upstream Linux and Kubernetes communities, that many of the modern and cloud-native architectures rely on.

Microsoft has done a ton of work for performance, reliability, manageability, and security to make Azure the best home for running any open source workload. Starting at the foundational level, Microsoft is working with the leading Linux Distros to optimize the kernels and hypervisors of Azure, including tuning the kernel for Azure hypervisors. Microsoft also works closely with Red Hat for managed services like Azure Red Hat OpenShift and SUSE with SAP enhancements. So when you bring your workloads to Azure, there is a benefit every step of the way, from onboarding to operation and you gain more security than you might have had on-premises, in your private cloud, or in another cloud. And whether you are starting greenfield or bringing what you already have running to Azure, we've got you covered.

Here is a summary of the key advantages of running Linux workloads and open source database services on Azure:

• Support for all major Linux distros like Red Hat, SUSE, Ubuntu, Oracle Linux, Debian, CentOS, CoreOS, and OpenSUSE.
• Azure-tuned kernels provide 25 percent faster network throughput.
• Unique and integrated support experience. Our support teams work with Red Had and the SUSE support team, to triage your support cases together.
• Minimal administration required for MySQL, PostgreSQL, and MariaDB with fully managed databases based on the latest community editions.
• The best total cost of ownership, high availability, and built-in intelligence provided by managed databases.
• Enterprise scalability with Hyperscale (Citus) which enables scaling of PostgreSQL across multiple servers and Query parallelization across those servers for faster responses.

Learn more

Find just about everything related to Linux running on Azure. And once you're ready to try migration or modernize your apps and open source databases, you can use Azure Migrate to find the tools to migrate your databases. Get guidance on how to migrate and modernize your workloads, apps, and databases on Azure Migration Center and enroll in Azure Migration Program to get expert help. And we also have a ton of learning content available on Microsoft Learn to help you easily migrate and modernize your applications to Azure.

The post How to migrate and modernize Linux workloads and open source databases to Azure appeared first on Microsoft Open Source Blog.

To learn more about how Microsoft is making it easier to build scalable applications and contributing to open source projects visit our most recent blog post, "Gain flexibility to run open source applications your way with Microsoft Azure."

Azure is the cloud for open source developers

You have endless possibilities on your cloud-native journey as new technologies and patterns are emerging every day. Azure supports developers to build on their terms, with integrated support for open source tools, languages, and third-party integrations of choice. We are constantly improving Azure for all. Our open cloud allows you to be free from infrastructure management, without the fear of vendor lock-in so that you can focus on delivering impact. Learn from developers around the globe why they chose Azure from our Developers Tell All video below.

At Microsoft Build, we shared new technologies to enable developers to develop flexibly and innovate quickly on Microsoft Azure. Check out our session, "Run Open Source Applications your way with Microsoft Azure," and our demos to see how we are empowering developers. Get the latest updates on the open source blog to learn how our engineers are innovating with open source and follow us on Twitter.

The post Empowering you to achieve more with open source on Azure appeared first on Microsoft Open Source Blog.