What is Node.js Monitoring?

The term “service monitoring,” means tasks of collecting, processing, aggregating, and displaying real-time quantitative data about a system.

Monitoring gives us the ability to observe our system’s state and address issues before they impact our business. Monitoring can also help to optimize our users’ experience.
To analyze the data, first, you need to extract metrics from your system — like the Memory usage of a particular application instance. We call this extraction instrumentation.
We use the term white box monitoring when metrics are provided by the running system itself. This is the kind of Node.js monitoring we’ll be diving into.

The four metrics to watch

Every service is different, and you can monitor many aspects of them. Metrics can range from low-level resources like Memory usage to high-level business metrics like the number of signups.
We recommend you to watch these metrics for all of your services:

• Error Rate: Because errors are user facing and immediately affect your customers.
• Response time: Because the latency directly affects your customers and business.
• Throughput: The traffic helps you understanding the context of increased error rates and the latency too.
• Saturation: It tells how “full” your service is relative to capacity. If the CPU usage is 90%, can your system handle more traffic.

Instrumentation

You can instrument your system manually, but most of the commercial monitoring solutions provide out of the box instrumentations.
In many cases, instrumentation means adding extra logic and code pieces that come with a performance overhead.
With Node.js monitoring and instrumentation, you should aim to achieve low overhead, but it doesn’t necessarily mean that a bigger performance impact is not justifiable for better system visibility.

The risk of instrumenting your code

Instrumentations can be very specific and usually need expertise and more development time. Also, a bad instrumentation can introduce bugs into your system or generate an unreasonable performance overhead.
Instrumenting your code can also produce a lot of extra lines and bloat your applications codebase.

Picking your Node.js Monitoring Tool

When your team picks a monitoring tool you should consider the following factors:

• Expertise: Do you have the expertise? Building a monitoring tool and writing a high-quality instrumentation and extracting the right metrics is not easy.
• Build or buy: Building a proper monitoring solution needs lots of expertise, time and money while obtaining an existing solution can be easier and cheaper.
• SaaS or on-premises: Do you want to host your monitoring solution? Can you use a SaaS solution — what’s your data compliance and protection policy? Using a SaaS solution can be a good pick, for example, when you want to focus on your product instead of tooling. Both open source and commercial solutions are usually available as hosted or on-premises setup.
• Licensing: Do you want to ship your monitoring toolset with your product? Can you use a commercial solution? You should always check licensing.
• Integrations: Does it support my external dependencies like databases, orchestration system and npm libraries?
• Instrumentation: Does it provide automatic instrumentation? Do I need to instrument my code manually? How much time would it take to do it on my own?
• Microservices: Do you build a monolith or a distributed system? Microservices needs specific tools and philosophy to debug and monitor them effectively. Do you need to distribute tracing or security checks?

Based on our experience, in most of the cases an out-of-the-box SaaS or on-premises monitoring solution like Trace gives the right amount of visibility and toolset to monitor and debug your Node.js applications.
But what can you do when you cannot choose a commercial solution for some reason, and you want to build your own monitoring suite? In this case, Prometheus comes into the picture!

Node Monitoring with Prometheus

Prometheus is an open source solution for Node.js monitoring and alerting. It provides powerful data compressions and fast data querying for time series data.
The core concept of Prometheus is that it stores all data in a time series format. Time series is a stream of immutable time-stamped values that belong to the same metric and the same labels. These labels cause the metrics to be multi-dimensional.

Data collection and metrics types

Prometheus uses the HTTP pull model, which means that every application needs to expose a GET /metrics endpoint that can be periodically fetched by the Prometheus instance.
Prometheus has four metrics types:

• Counter: Cumulative metric that represents a single numerical value that only ever goes up.
• Gauge: Represents a single numerical value that can arbitrarily go up and down.
• Histogram: Samples observations and counts them in configurable buckets.
• Summary: similar to a histogram, samples observations, it calculates configurable quantiles over a sliding time window.

In the following snippet, you can see an example response for the /metrics endpoint. It contains both the counter (nodejs_heap_space_size_total_bytes) and histogram (http_request_duration_ms_bucket) types of metrics:

# HELP nodejs_heap_space_size_total_bytes Process heap space size total from node.js in bytes.# TYPE nodejs_heap_space_size_total_bytes gaugenodejs_heap_space_size_total_bytes{space="new"} 1048576 1497945862862 nodejs_heap_space_size_total_bytes{space="old"} 9818112 1497945862862 nodejs_heap_space_size_total_bytes{space="code"} 3784704 1497945862862 nodejs_heap_space_size_total_bytes{space="map"} 1069056 1497945862862 nodejs_heap_space_size_total_bytes{space="large_object"} 0 1497945862862# HELP http_request_duration_ms Duration of HTTP requests in ms# TYPE http_request_duration_ms histogramhttp_request_duration_ms_bucket{le="10",code="200",route="/",method="GET"} 58http_request_duration_ms_bucket{le="100",code="200",route="/",method="GET"} 1476http_request_duration_ms_bucket{le="250",code="200",route="/",method="GET"} 3001 http_request_duration_ms_bucket{le="500",code="200",route="/",method="GET"} 3001 http_request_duration_ms_bucket{le="+Inf",code="200",route="/",method="GET"} 3001

Prometheus offers an alternative, called Pushgateway, to monitor components that cannot be scrapped because they live behind a firewall or are short-lived jobs.
Before a job gets terminated, it can push metrics to this gateway, and Prometheus can scrape the metrics from there later on.
To set up Prometheus to periodically collect metrics from your application check out the following example configuration.

Monitoring a Node.js application

When we want to monitor our Node.js application with Prometheus, we need to solve the following challenges:

• Instrumentation: Safely instrumenting our code with minimal performance overhead.
• Metrics exposition: Exposing our metrics for Prometheus with an HTTP endpoint.
• Hosting Prometheus: Having a well configured Prometheus running.
• Extracting value: Writing queries that are statistically correct.
• Visualizing: Building dashboards and visualizing our queries.
• Alerting: Setting up efficient alerts.
• Paging: Get notified about alerts with applying escalation policies for paging.

Node.js Metrics Exporter

To collect metrics from our Node.js application and expose it to Prometheus we can use the prom-client npm library.
In the following example, we create a histogram type of metrics to collect our APIs’ response time per routes. Take a look at the pre-defined bucket sizes and our route label:

 // Init const Prometheus = require('prom-client') const httpRequestDurationMicroseconds = new Prometheus.Histogram({ name: 'http_request_duration_ms', help: 'Duration of HTTP requests in ms', labelNames: ['route'], // buckets for response time from 0.1ms to 500ms buckets: [0.10, 5, 15, 50, 100, 200, 300, 400, 500] })

We need to collect the response time after each request and report it with the route label.

// After each response httpRequestDurationMicroseconds .labels(req.route.path) .observe(responseTimeInMs)

We can then register a route a GET /metrics endpoint to expose our metrics in the right format for Prometheus.

// Metrics endpoint app.get('/metrics', (req, res) => { res.set('Content-Type', Prometheus.register.contentType) res.end(Prometheus.register.metrics()) })

After we collected our metrics, we want to extract some value from them to visualize.
Prometheus provides a functional expression language that lets the user select and aggregate time series data in real time.
The Prometheus dashboard has a built-in query and visualization tool:

Let’s look at some example queries for response time and memory usage.

Query: 95th Response Time

We can determinate the 95th percentile of our response time from our histogram metrics. With the 95th percentile response time, we can filter out peaks, and it usually gives a better understanding of the average user experience.

histogram_quantile(0.95, sum(rate(http_request_duration_ms_bucket[1m])) by (le, service, route, method))

Query: Average Response Time

As histogram type in Prometheus also collects the count and sum values for the observed metrics, we can divide them to get the average response time for our application.

avg(rate(http_request_duration_ms_sum[1m]) / rate(http_request_duration_ms_count[1m])) by (service, route, method, code)

For more advanced queries like Error rate and Apdex score check out our Prometheus with Node.js example repository.

Alerting

Prometheus comes with a built-in alerting feature where you can use your queries to define your expectations. However, Prometheus alerting doesn’t come with a notification system. To set up one, you need to use the Alert manager or another external process.
Let’s see an example of how you can set up an alert for your applications’ median response time. In this case, we want to fire an alert when the median response time goes above 100ms.

# APIHighMedianResponseTime ALERT APIHighMedianResponseTime IF histogram_quantile(0.5, sum(rate(http_request_duration_ms_bucket[1m])) by (le, service, route, method)) > 100 FOR 60s ANNOTATIONS { summary = "High median response time on {{ $labels.service }} and {{ $labels.method }} {{ $labels.route }}", description = "{{ $labels.service }}, {{ $labels.method }} {{ $labels.route }} has a median response time above 100ms (current value: {{ $value }}ms)", }

Kubernetes integration

Prometheus offers a built-in Kubernetes integration, supported inside Azure Container Service as well. It’s capable of discovering Kubernetes resources like Nodes, Services, and Pods while scraping metrics from them.
It’s an extremely powerful feature in a containerized system, where instances are born and die all the time. With a use case like this, HTTP endpoint-based scraping would be hard to achieve through manual configuration.
You can also provision Prometheus easily with Kubernetes and Helm. It only needs a couple of steps.
To start, we need a running Kubernetes cluster! As Azure Container Service provides a hosted Kubernetes, I can provision one quickly:

# Provision a new Kubernetes cluster az acs create -n myClusterName -d myDNSPrefix -g myResourceGroup --generate-ssh-keys --orchestrator-type kubernetes# Configure kubectl with the new cluster az acs kubernetes get-credentials --resource-group=myResourceGroup --name=myClusterName

After a couple of minutes when our Kubernetes cluster is ready, we can initialize Helm and install Prometheus:

helm inithelm install stable/prometheus

For more information on provisioning Prometheus with Kubernetes check out the Prometheus Helm chart.

Grafana

As you can see, the built-in visualization method of Prometheus is great to inspect our queries output, but it's not configurable enough to use it for dashboards.
As Prometheus has an API to run queries and get data, you can use many external solutions to build dashboards. At RisingStack, one of our favorites is Grafana.
Grafana is an open source, pluggable visualization platform. It can process metrics from many types of systems, and it has built-in Prometheus data source support. In Grafana, you can import an existing dashboard or build you own.

Dashboard with Grafana

Next steps

If you want to learn more about Node.js and reliability engineering, make sure to check out the free whitepaper from RisingStack.
At the Node Summit this week? Check out this blog post for ways to the connect with the Microsoft team there.

The post Node.js performance monitoring with Prometheus appeared first on Microsoft Open Source Blog.

At RisingStack we use Kubernetes with tens of microservices to provide our Node.js monitoring solution Trace for our SaaS customers.
During the last couple of months, we were asked by many enterprises with strict data compliance requirements to make our product available as a self-hosted service. So we had to find a solution that makes it easy for them to install Trace as a single piece of software and hides the complexity of our infrastructure. It’s challenging because Trace contains many small applications, databases, and settings. We wanted to find a solution that is not only easy to ship but also highly configurable. As Kubernetes is configuration based, we started to look for templating solutions that brought up new challenges. That’s how we found Helm, which provides a powerful templating and package management solution for Kubernetes. Thanks to this process, Trace is now available as a self-hosted Node.js monitoring solution, and you can have the same experience in your own Azure cloud as our SaaS customers.In this blog post, I’ll show how we packed our Kubernetes microservices app with Helm and made them easy to reproduce in various environments.

apiVersion: apps/v1beta1 kind: Deployment metadata: name: metrics-processor spec: replicas: 3 spec: containers: - name: metrics-processor image: myco/metrics-processor:1.7.9 env: - name: DB_URI value: postgres://my-uri

One of the best parts of Helm is that it comes with an open-source repository maintained by the community, where you can find hundreds of different pre-packed solutions from databases like MongoDB and Redis, to applications like WordPress and OpenVPN.
With Helm, you can install complex solutions like a Jenkins master-slave architecture in minutes.

helm install --name my-jenkins stable/Jenkins

Helm doesn’t just provision your Kubernetes resources in the correct order. It also comes with lifecycle hooks, advanced templating, and the concept of sub-charts. For the complete list, I recommend checking out their documentation.

# Provision a new Kubernetes cluster az acs create -n myClusterName -d myDNSPrefix -g myResourceGroup --generate-ssh-keys --orchestrator-type kubernetes# Configure kubectl with the new cluster az acs kubernetes get-credentials --resource-group=myResourceGroup --name=myClusterName

helm init

helm install --name my-ci --set Master.ServiceType=NodePort,Persistence.Enabled=false stable/jenkins

For the sake of simplicity and security, I disabled persistent volume and service exposing in this example.

That’s it! To visit our freshly installed Jenkins, follow the instructions in the Helm install output or use the kubectl pot-forward <pod-name> 8080 terminal command.

In a really short amount of time, we just provisioned a Jenkins master into our cluster, which also runs its slaves in Kubernetes. It is also able to manage our other Kubernetes resources so we can immediately start to build CI pipelines.

Questions? Let me know in the comments.

The post Packing a Kubernetes Microservices App with Helm on Azure Container Service appeared first on Microsoft Open Source Blog.