Stop and Start your AKS Big Data Cluster and Save $$$

I remember when I first started deploying Big Data Clusters, they were on Azure Kubernetes Service utilizing the $200 credit for first time sign ups. By the time I got around to figuring out how to deploy the BDC, not only was my $200 credit gone, but I started to incur cost out of pocket.

If only there was a feature that would allow me to stop the VMs in AKS whenever I wasn’t using them. Well, I’m excited to share that Microsoft AKS (Azure Kubernetes Service) came out with a neat feature (currently in preview at the time of the publishing of this post) that allows you to stop and start your AKS cluster by running a simple command. Of course I had to try it out on BDCs and to my surprise it worked. Well, sort of. Let me explain…

To create setup my test environment, I created two BDCs running on AKS. The first one is a regular dev/test BDC running on a 3-node cluster (aks-dev-test). The second is a dev/test BDC with HA (high availability) option running on a 3-node cluster (aks-dev-test-ha). If you’re wondering, “huh?”, I deployed both of the options in the BDC wizard within Azure Data Studio (see screenshot below).

The first BDC (aks-dev-test) worked fine. I ran the, az aks stop -n bdc -g bdc command and it stopped all 3 of the nodes. When I tried to do the same thing with the HA option BDC (aks-dev-test-ha), it stopped fine, but when I ran the start command (az aks start -n bdc -g bdc), the only pods that came up were the controller pods. I assume something with the HA function in the BDC that just doesn’t allow for a smooth “stop/start” in AKS.

So if you’re running a BDC for dev/test purposes (non-HA option), you can cut down on the cost by using the “stop/start” command. For example, shutting it down for the weekend, or if you don’t plan on using the BDC for a while.

A couple things to note:

Per Microsoft AKS documentation page:

• You have 12 months before Microsoft removes it. So if you stop your AKS cluster, go on vacation for 12 months, come back and on the first day of the 13th month you try to run “az aks start…”….guess what? (Why do I have N’SYNC’s Bye Bye Bye in my head?)
• When the AKS cluster is stopped, the only two commands you can run are Start or Delete. If you’re looking to update the cluster, you’ll have to start it first.
• This feature only works on VM scaled sets backed clusters, not private clusters (which is great for the BDC on AKS scenario because BDCs deployed on AKS are deployed across VM scaled sets clusters)

How To Stop / Start your BDC on AKS

The process is quite simple. Here are the steps I took to stop (then start) my BDC running on AKS:

1. If you do not have the az preview extension installed, install it by running the below command:

az extension add --name aks-preview

2. If you already have the az preview extension installed, then it’s a good habit to always update it to ensure you have the latest enhancements. You can update it by running the below command:

az extension update --name aks-preview

3. Once you have an updated version of the az preview extension, and since this feature is in preview, you will have “Register” the feature by running the below command:

az feature register --namespace "Microsoft.ContainerService" --name "StartStopPreview"

4. When step 3 is complete, refresh the provider:

az provider register --namespace Microsoft.ContainerService

5. Now we’re reading to use the new feature. Let’s first stop our BDC on AKS by running the following command:

az aks stop --name myClusterName --resource-group myResourceGroupName

One quick note before stopping the AKS cluster. You can see your nodes under the VM scale set (under your resource group), and in my case I can see all 3 of my nodes (see screenshot below):

Now, let’s run the stop command.

You can see in the screenshots below that show the stop command in Azure Data Studio, and the “Stopping” status in the Azure portal.

If you run a kubectl get nodes while the stop command is running you can see that the Status of the nodes is “Ready,SchedulingDisabled” (see screenshot below). This means that no new pods can be scheduled on the nodes.

Once the AKS cluster has stopped, the status will change to “Succeeded”. So if I go back to Azure Data Studio and run a kubectl get nodes, I end up with this:

“Unable to connect to the server: dial tcp: lookup bdc-bdc-cf2467-bfe0e51b.hcp.eastus.azmk8s.io: no such host”

And if I go to into my Azure portal dashboard under the Virtual Machine scale set, I do not see any of the 3 nodes:

6. To start the BDC on AKS, it’s as simple as replacing the “stop” in the command with “start”:

az aks start --name myAKSClusterName --resource-group myResourceGroupName

Once the command is done executing, I can run a kubectl get nodes and kubectl get pods -n bdc:

After a couple minutes all the pods are in a “running” status and ready. At this time you can log into your Big Data Cluster.

There you go! If you’re new to BDCs and want to give AKS a try, you can do so and use the stop/start command and stretch your $200 credit pretty far. Hopefully this puts some money back in your pocket!