Out-of-memory (OOM) in Kubernetes – Part 4: Pod evictions, OOM scenarios and flows leading to them

Mihai Albert3 Comments

This is part 4 of a four-part article that looks into what happens in detail when Kubernetes runs into out-of-memory (OOM) situations and how it responds to them. You can find the article’s full table of contents here as part of the first post.

Pod evictions

What are pod evictions in Kubernetes? It’s an automatic action that Kuwbernetes takes on a node experiencing low resources whereby one or more pods are terminated in an effort to reduce the pressure. As this article deals with memory, we’ll talk exclusively about memory as the resource a node is experiencing a shortage of.

Previously (in OOM killer and Cgroups and the OOM killer) we’ve seen how the OOM killer will ensure the available memory on the node doesn’t go below critical levels. So it’s quite apparent these 2 mechanisms – pod evictions and the OOM killer – have the same goal: ensure the node doesn’t end up without any memory left. So why have both of them active at the same time?

Kubernetes doesn’t have direct control on the OOM killer. Remember that it’s a Linux kernel feature. What Kubernetes does – and to be more specific the Kubelet on each node – is adjust the “knobs” for the OOM killer: e.g. by setting different values for oom_score_adj it alters the latter’s behavior as to which victim gets chosen first. This still doesn’t answer the initial question of why are both mechanisms needed, but only goes to tell that Kubernetes has to “live with” the OOM killer. Let’s go ahead for now, and as we get further along the answer will gradually appear.

Continue reading →

Out-of-memory (OOM) in Kubernetes – Part 3: Memory metrics sources and tools to collect them

Mihai AlbertLeave a comment

This is part 3 of a four-part article that looks into what happens in detail when Kubernetes runs into out-of-memory (OOM) situations and how it responds to them. You can find the article’s full table of contents here as part of the first post.

Metrics components

There are several components that generate and consume metrics for a Kubernetes cluster. In this section, we’ll go over some of those components that can provide useful memory metrics for us, see what they do and how they interconnect, look at the endpoints they expose, and the format data is provided in. We’ll briefly discuss some Prometheus components now, even if they’re not builtin inside Kubernetes, as we’ll make use of them further into the article. Grafana and Prometheus will be treated in depth separately in their own section later on.

There is as of now (Dec 2021) overlapping functionality between some of the components we’ll go over, since some are slated for decommissioning. Note also that things are moving fast in the Kubernetes land, so by the time you read this some of the items presented might have already changed.

Keep in mind that we’ll not discuss “grand” topics such as types of monitoring pipelines (for which you can find a starter here) nor the Kubernetes monitoring architecture (which is discussed in a proposal document – although dated by now – here). Nor will we touch upon components that don’t have a direct bearing on the problem we’re trying to solve, such as the Custom Metrics API. We’ll stick to the point and only do detours when strictly required.

Continue reading →

Out-of-memory (OOM) in Kubernetes – Part 2: The OOM killer and application runtime implications

Mihai Albert4 Comments

This is part 2 of a four-part article that looks into what happens in detail when Kubernetes runs into out-of-memory (OOM) situations and how it responds to them. You can find the article’s full table of contents here as part of the first post.

Overcommit

We need to discuss overcommit, otherwise some of the stuff that comes next won’t make sense, particularly if – like me – you’re coming from the Windows world. What does overcommit mean? Simply put, the OS hands out more memory to processes than what it can safely guarantee.

An example: we have a Linux system running with 12 GB RAM and 4 GB of swap. The system can be bare-metal, a VM, a WSL distribution running on Windows, etc as it doesn’t matter from the standpoint of how overcommit works. Assume the OS uses 1 GB for its kernel and various components it runs. Process A comes along and requests to allocate 8 GB. The OS (Operating System) happily confirms, and process A gets a region of 8 GB inside its virtual address space that it can use. Next process B shows up and asks for 9 GB of memory. With overcommit turned on – which is usually the default setting – the OS is happy to process this request successfully as well, so now process B has a region of 7 GB inside its own virtual address space. Of course, if we sum 1+8+9 this shoots over the total quantity of memory that the OS knows it has at its disposal (12+4), but as long as processes A and B don’t need to use all their memory at once, all is well.

Continue reading →

Out-of-memory (OOM) in Kubernetes – Part 1: Intro and topics discussed

Mihai Albert3 Comments

Intro

You’ve seen it all before: that important application service running inside Kubernetes goes down at the most inconvenient time. There’s a tug of war between the DevOps team – that maintains Kubernetes did nothing wrong and the service simply exited with an error message (“can’t they just fix their code?”) – and the Development team – that maintains all the logs show the application was alive and well, and there must be something that Kubernetes did that resulted in the service to be taken down (“can’t they just make that Kubernetes thing stable?”). Sometimes an “OOMKilled” tell-tale status is observed for one of the containers, other times some Kubernetes pods are seen in an “Evicted” state. There are times when there’s some strange 139 exit code, which gets mapped to something called “SIGSEGV”, but it doesn’t solve the mystery either. The investigation continues, and nodes are found that show weird numbers like 107% memory usage at times. Is that number normal? Is there some glitch in how Kubernetes reports memory on that very node? You start suspecting there might be a memory issue, and so you turn to the Kubernetes logs to get more details. But where are those to begin with? This continues, with more questions coming up as you go along and hours spent trying to figure out what went wrong. And there’s always management that at the end of the day wants an answer to “how can we prevent this from happening in the future?”…

This 4-part article will look into what happens in detail when Kubernetes runs into out-of-memory (OOM) situations and how it responds to them. The discussion will include the concepts involved, details around how the various mechanisms work, the metrics that describe memory and the tools used to gather them, diagrams to show how things come together, and in-depth analysis about some of the situations that can be encountered.

Why would you want to read this article? You’re using Kubernetes,…

Continue reading →

A tool to leak memory on Windows, Linux and Within Containers

Mihai AlbertLeave a comment

We went over a crude app that allocated and touched memory in a previous post. Allocations were done using int arrays, and touching the memory was done efficiently, only writing every 1024th element in each int array, which in turn would write each time to a subsequuent memory page (4KB in size, equivalent to 1024 x 4 bytes per int element) and force it into the working set. Yet the memory block size was fixed, the delay between allocations was hardcoded and all the allocated memory was eventually touched. It lacked control.

So what we’ll set out to do is write a new app that allocates (leaks) memory, that allows:

  • Custom delay between subsequent allocations
  • Adjustable size of the memory block allocated at once
  • Memory block size specified in MB
  • Variable touch “fill” ratio, specifying how much of the committed memory gets touched per each memory block allocated
  • A threshold above which no more allocations will be performed, specified in MB (or 0 to allocate indefinitely)
  • Command arguments to specify values for all the parameters described above

What we’ll get is a tool that can leak memory in a controlled fashion, with multiple “knobs” that can alter its behavior.

Continue reading →

Automate Retrieving Azure VM Data Using Azure Resource Graph (ARG) and C#

Mihai AlbertLeave a comment

In this post, we’ll look into using Azure Functions to run C# code. We’ll keep things simple, and we’ll only use a timer-based function. The function itself will simply write output data to a storage account, as a simple JSON file. The payload is something we’ve looked at before in detail: the list of all Azure VMs with all their private and public IPs.

Why go over the same thing that was discussed previously? First, we’ve used several methods to gather the data in that post (Powershell, Azure Resource Graph Explorer (ARGE), Azure CLI) but neither involved C#. Secondly, the methods depicted weren’t automation-ready, in the sense that one would still have to perform things manually to get the results (for Powershell, run the script; for Azure CLI, run the commands; for ARGE, run the query and manually export the results).

This time, C# code within an Azure Function will automatically write our output .json file, without any sort of manual intervention, via a timer-based trigger. The underlying technology to get all the networking data for the Azure VMs will be Azure Resource Graph (ARG).

Continue reading →

Get the List of All AWS EC2 Instances With All Their Private and Public IPs

Mihai Albert2 Comments

You want to retrieve a list of all your AWS EC2 instances, complete with all their private and public IPs. You want to do this across your whole AWS organization, thereby targeting all the AWS accounts and all the regions enabled for each. And you’d also want to use C# for the job.

This post will detail how to do this, starting from explaining the base concepts and going all the way to building the code. The output – alongside basic console text – will consist of a .json file containing the following attributes for each EC2 instance:

  • Instance id
  • Instance type
  • All private IP addresses
  • All public IP addresses
  • Parent AWS account name
  • Parent AWS account id

If you’d like to skip directly to the C# code for retrieving private and public IP addresses for all EC2 instances using IAM users, go here. For the C# code that uses SSO users, go here.

Continue reading →

Get the List of All Azure VMs With All Their Private and Public IPs

Mihai Albert5 Comments

You want to retrieve a list with all your Azure VMs, complete with all their private and public IPs. You’d also like to get this fast, without having to start some script and come the next day to find the results (or worse, to discover that it errored out, and you only have 5 minutes left to produce the report). There are bits and pieces around the web – like this query that retrieves just one public IP per each VM – regardless if they have multiple assigned – but no private IP whatsoever. There are also Powershell scripts around, but they take too long or provide incomplete information.

And since Azure has, at this time, resources deployed using two possible models (ASM and ARM), you need to be careful about what you use to get each set of VMs, as the tools used to retrieve the info for one are incompatible with the other.

You might also want to query across thousands of VMs spread out in hundreds of Azure subscriptions that make up your tenant. How about a solution that takes less than a second to get all this information:

TL;DR Jump here to see how to extract all the Azure VMs + all their private/public IPs in a matter of seconds.

Continue reading →

Use Microsoft Graph to Retrieve Users Photos and Store Them as Azure Blobs With C#

Mihai Albert2 Comments

A strange request comes your way. The photos of everyone in your organization’s Office 365 tenant need to be provided as .jpg files. They have to be imported into a proprietary app, and named based on an attribute that uniquely identifies to whom each belongs. It’s not clear yet if they just want all the photos as a giant .zip archive, or if their app runs in Azure and needs the photos stored in a storage account. You come across the Get-UserPhoto Powershell cmdlet, which seems to offer a quick solution, but it fails to deliver against those users that still have their mailbox on-premises. How to go about it?

TL;DR The sample code that retrieves photos from Office 365 and stores them both locally on disk, as well as in Azure Blob storage, can be found here.

The main goal of this post is to get users’ pictures from an Office 365 tenant. We’ll subsequently store them in 2 places: on disk and in an Azure storage account as block blobs. We’ll only target ‘live’ users, meaning those that have their account enabled and also have an employeeNumber. Our scenario further detailed:

Continue reading →

Automatically Recover Zoom Licenses

Mihai AlbertLeave a comment

Suppose you’ve purchased Zoom licenses for your users. However not everybody is using the services provided by having her/his user type as “Licensed”, instead some of the users can go along just fine with “Basic”. As you don’t want to pay for unused licenses, you don’t buy one for each and every user you have, and rely on assigning licenses to only those people that need it. However if auto-provisioning is enabled, any user automatically logging in will trigger an account to be created in Zoom. At this point you have a problem:

  • If a license is automatically handed out to every newly provisioned user, you’ll eventually run out of licenses and new users will be denied the services offered through one
  • If no license is automatically assigned to newly provisioned users, but instead any new account is provisioned as “Basic”, these won’t be able to use the licensed services

As you want to avoid buying more licenses, you’re left with only one alternative: free up licenses not used. You can go to Admin -> Account Management -> Reports -> Inactive Hosts and see those who haven’t logged in to their account in a specific timeframe and choose to remove their license. There are 2 problems with this approach, at least as of May 2020: 1) you have to process users one by one and 2) the timeframe of the report is limited to 1 month, so trying to get the users not active in the last 3 months is not that straightforward.

Luckily there’s an API that Zoom built, whose details are here that allows automating this process. Read on for how to build a script that leverages it for retrieving user data and for removing licenses for users matching a specific criteria.

TL;DR: Are you looking for a script that automatically identifies the users that haven’t logged in recently and removes their Zoom license ? Jump to the Powershell script.

Continue reading →