Self-Imposed Exile from Instagram and WhatsApp

Around the same time last year and till the middle of 2018, I wasn’t so addicted to social media. I still don’t think of myself as someone addicted to social media. For example, I’m very inactive on Facebook and Twitter. And I don’t understand why people even use Snapchat at all! Yes, I’m that dumb/old school. But, I get addicted to people. And I am not sure which is worse.

The Preface

I had been spending a lot of time on Instagram lately. Besides following some motivational speakers, quotes pages, school-time crushes (truth is I always have a crush on someone 😉), I talked with a few people on the app. And few of these are my favourite people! But it’s not your usual chat app. And I don’t know what is with it that even when there are no notifications, I would open up the app and mindlessly scroll through the feed.

While mindlessly scrolling without any notifications, I knew I was not making the best use of the time but, I also started feeling like my friends were too busy in their lives. At one point, I started feeling like they don’t want me around. I think I had a pretty bad anxiety situation that I had dragged myself into. And things were getting worse for me with every passing day. I had to get back to being sane again.

In the past, I have uninstalled Instagram a number of times because I started observing the mindless scrolling behaviour. But with the advent of chat feature, it became impossible to remove the app since it would mean cutting off the communication with closest people.

What did I do about it?

I wasn’t feeling well last Saturday so I messaged a friend about it, took medicines, had a chat with my wife and went to bed. I was in Europe (Brno and Brussels) last month and along with bringing back a lot of chocolates for family and friends, I had brought back a really bad flu as a personal souvenir - especially for me!

Next morning, I and my wife were supposed to be travelling to a nearby town for her work but I decided to stay back due to the flu & fever. Rest of that Sunday, I didn’t turn on the data on my phone since I wasn’t feeling well. Maybe it was flu getting into my head that I felt like staying offline on my phone for some more time and I challenged myself to do so for a week. So, just like that, without notifying anyone, I kept the data on my phone off for an entire week.

What was I hoping to achieve?

Well, quite a few hypothetical things actually. Here’s a list:

  • I was hoping to get a call from the people I thought were really close to me. Not many. Just 1 or 2 persons who I communicated with on a daily basis through Instagram and WhatsApp because I thought I was important to them.

  • I was hoping to get my thoughts sorted. I am turning 30 in a couple of days and I feel like I have not accomplished anything that would make me feel proud. Like really proud!

  • Writing is thinking. And I recently started doing it. I wanted to write more about what’s going on in my head.

  • Hours of boring time. At least 1-1.5 hours a day because that’s how much I used to spend on Instagram. I hoped for some ideas to come up when my mind was busy being bored.

  • Read more. I managed to read 10 books last year and have challenged myself to read 15 this year. Since I have not made any significant progress in the first two months of the year, I was hoping to make some during the quiet period.

What did I really achieve or, how did the experiment really go?

  • First off, the flu fucked me bad enough to take two days of sick leave from work. But it was a blessing in disguise. No internet on the phone, no work, not an ounce of energy to even take a shower - boredom unleashed (if cough let its leash loose.)

  • No phone calls. Not from the few close people, not from anyone. But, to be fair to them, I went off without a warning. Just the way most people die. Isn’t it very common to miss someone when they’re dead? I’m not sure how much of time and effort it would take to simply check on someone you love. Not even 5 minutes per person per day, I guess. Anyway, this was neither surprising nor disappointing. I just gotta stop whining and welcome myself to the 21st century living. 😉

  • I did manage to sort my thoughts to an extent and gain clarity on what’s important to me and what’s not. I’m surprised to realize that this is possible for me.

  • I also wrote more. Not that I have a life as interesting as Tom Marvolo Riddle but I still blurted out a bunch of stuff in a diary/journal/whatever-the-fuck-you-call-it I maintain since a few months.

  • Boring time! This was really crucial. Besides coming up with ideas to do certain things, I also felt more confident about myself due to the detox. I had some ideas and more importantly, I felt like I can take them to fruition! I’m yet to start working on any of them but hey isn’t positive attitude nice shit to have on your side?

  • I finally started reading the book I was very excited to read when it released in early 2019. I read about 100 pages. Quite slow than my usual pace but this is not some runaway fiction novel. It’s a book that explains some of the most complex texts in easy to understand language.

Okay, what else?

  • I started spending less time on a commode because I wasn’t mindlessly scrolling through Instagram feed while shitting.

  • While I spent most mornings coughing the fuck out of me, I think mornings will become quieter if the cough dies away and I keep the data off during the first few hours of the day.

  • It’s Spring season in India. That means, it’s more beautiful outside the house than inside (unless you’re just married). The sky is more blue than usual. Or maybe the lack of blue light of smartphone has made the natural blue light look sexier to me!

  • More books, more boredom, more writing. At least that’s what I plan to continue with. I know I can’t stay off the social media forever. In fact, I turned on the data this morning. But I don’t feel the itch to keep checking my phone frequently.

  • I have been staying away from mainstream media since over a year. But in the past week when tensions between India and Pakistan were high, I was watching a few news channels. And all the while I was feeling like turning it off already.

  • I couldn’t stop thinking of the asshole I had once become when I had issues with my ex-girlfriend (she’s my wife now - I’m a one-woman man). And I missed that me real bad. I still do. That was a positive change in me during a crazy time.

Parting thoughts (mostly crap)

I know you’ve got better things to do than reading the crap I write.

Maybe it’s just me (I know I’m getting old) but I don’t really like the way the advent of social media has changed the way we do some very basic yet special things. For example, not a long time ago, we would call up someone on a birthday; these days a “Happy Birthday” on WhatsApp/Facebook is all we do and mostly we don’t even mean it! I’d rather not wish that person instead of faking a wish!

Or we would go to a cafe and have a talk without clicking tonnes of selfies or mindlessly scrolling through the feeds and looking like two strangers just sharing the same fucking table. Or we would actually savour the food at a restaurant instead of clicking pics from different angles and eating the meal while simultaneously checking the number of likes we got!

Maybe it’s just me but relations feel more real outside the virtual world than inside. When the distance is the barrier, it makes sense to use these platforms. But being glued to the phone all the time has made me more depressed and anxious than I ever thought was possible!

That’s it!

I’m done. I don’t know why I wrote this post. I don’t know if I did the right thing by writing this here. I don’t know if this detox was the right thing to do or if I should have continued it for more than a week. Heck, I don’t know anything. But a week of self-imposed Instagram exile made me feel good about myself. Better than I have felt in a long time.

And in this entire week, there was only one friend who pinged me a few times on WhatsApp to know about my whereabouts and if I had blocked them or if I was dead (too bad for you that I’m still alive). Guess I’m lucky to have one such friend. It’s scary how small the circle of people gets as we grow old. Or maybe I’m being a wrong kind of an asshole in my life so far. 😉

Until next time… 😄

Centos Container Pipeline OpenShift Upgrade Part 5 - 'The Lessons'

This blog is the last part in the series of blogs on OpenShift OKD upgrade we performed for the CentOS Container Pipeline project. If you haven’t read the earlier parts, go ahead and do that first.

If you have already read the previous parts in the series, it’s easy to conclude that we screwed up. We thought we had broken out of that habit after moving to the new architecture. After all, we had a stable environment for two and half months and only issues we faced were from our buggy code. But bad habits are difficult to get rid of!

Obviously, we learned a few lessons in the process. Few too many!

Lesson 0 - Friday is for beers

This one doesn’t need any detail. Unless it’s a major bug fix or security patch, Friday should be meant for beer, or coffee, or tea, or <put your favorite drink here>.

Lesson 1 - OpenShift knowhow needs to be improved

We need to improve our knowledge about OpenShift works. RTFM is not only required but we need to make more time to do it. I’m not sure how many problems we could have avoided had we read more documentation but, we could have surely made informed decisions when we faced them.

Lesson 2 - Homogeneous production and testing environment

Every bit of testing we did was on an infrastructure which we had absolute control over right from spinning up the VMs to deploying our service in the OpenShift cluster. However, the production environment is something that CentOS infrastructure team provides and there are some differences between the two because the errors we saw in production environment never showed up while performing same actions on the test environment.

We need start using a small subset of systems provided by CentOS infrastructure team for pre-production testing and validation.

Lessons 3 - Have reasonable expectations from the upstream

The OpenShift upstream is possibly loaded with a bunch of questions and issues from users like our project. It would be too much to expect them to look into every issue beyond “best effort” basis. This takes us back to Lesson 1.

Lesson 4 - Talk to the experts

Before making a cluster level change, it might be a good idea to write down the plan and the steps, and run it by a few experts to get an opinion or maybe even a better idea to do what we’re trying to do.

That’s it!

That’s all I have to share about the fiasco we did during the weekend of December 14, 2018. If you have any suggestions/ideas/feedback for us, let us know. I know there’s a lot we can and we need to learn about the various tools involved in this exercise.

On a personal note, this is the most I’ve ever written in a given week. If you read it this far, I hope I managed to make you think - “This guy is a professional screw-it-up artist”.

Until next time… 😄

Centos Container Pipeline OpenShift Upgrade Part 4 - 'Fire in the Hole'

This blog is fourth part in the series of blogs on OpenShift OKD upgrade we performed for the CentOS Container Pipeline project. If you haven’t read the earlier parts, go ahead and do that first.

I closed the previous post on a high wherein we managed to fix multiple issues that were preventing us from doing the OKD upgrade in production. After reaching the stage, we decided to do OKD 3.9 to OKD 3.10 upgrade and then upgrade to OKD 3.11. In the development environment, I already attempted an OKD 3.10 to OKD 3.11 upgrade. It was a walk in the park just like the experience I had while upgrading from OKD 3.7.2 to OKD 3.9.0.

Mistake #0 - Choosing Friday for production upgrade

This statement is intentionally written to not make any sense; just like our decision to do production upgrade on Friday. I don’t think any amount of penance can free us from this sin of disturbing production Gods on a Friday. Amen.

Song of Error on loop mode

We had taken a maintenance window of about 4 hours and were confident of finishing our job in under 2 hours. But there won’t be any fun if things went as per the plan. As if a song of error was playing on loop mode, we were repeatedly seeing the same error and not able to make any sense of what’s going wrong.

TASK [Read node config]
ok: []                                                                                                                          
fatal: []: FAILED! => {"changed": false, "msg": "file not
found: /etc/origin/node/node-config.yaml"}
ok: []
ok: []
ok: []
ok: []
ok: []
ok: []
ok: []
ok: []
ok: []

We were seeing this error for the master. And every time we checked the master after OpenShift Ansible threw above error, this file /etc/origin/node/node-config.yaml was undoubtedly present! The logs were not particularly helpful in pointing out the real cause of the problem. Above all, this was the first time we were seeing this error. Our development environment didn’t throw this error for OKD 3.11.

Now, if you have read previous posts in this series, it’s not the first time I’m saying that logs didn’t make a lot of sense. This, probably, boils down to the point I mentioned in an earlier post about the first and one of the most important lessons this exercise taught us - “We lack OpenShift expertise within our team.”

Uninstall OKD 3.9, install OKD 3.11

After struggling with the plan of upgrading to OKD 3.10 in a recommended manner, we decided to go with the unconventional path in production because with every passing minute during which our production was a mess, we were not doing what our service was supposed to do - build container images for the open-source projects.

Jenkins PV did bite us

After bringing up OKD 3.11 in production, we tried to attach the same PV to Jenkins that was being used in OKD 3.9 cluster. To our surprise, the PV got attached immediately and Jenkins started up. This was something we struggled to accomplish in the development environment where PV wouldn’t get attached unless we modified the permissions or deleted the data on PV. And once the 3.9 to 3.10 upgrade worked fine, we stopped thinking about Jenkins PV altogether.

In production, however, first thing Jenkins did after connecting with the PV was to delete the data from previous builds jobs. Not sure how that was any better than us formatting the PV ourselves!

etcd wanted its share of biting us as well

The production environment was running on top of OKD 3.11 while development environment was on OKD 3.10. We didn’t have much idea of how things were with OKD 3.11. And that fact was very well exploited by etcd.

etcd restarts when a particular version of docker is used which caused API service to be unavailable for some time. The seed-job in CentOS Container Pipeline service is a build job that:

  • runs on Jenkins,
  • parses the container index,
  • creates/updates jobs on Jenkins
    • triggers build for newly created jobs
    • silently updates existing jobs (if there is any change)

It uses oc command line tool to do its job. Now while the etcd restarts caused API server to be unavailable, our seed-job kept doing its job unaware of the situation and the oc commands during this time were pretty much talking to /dev/null (metaphorically). The seed-job needs to be altered to consider this scenario. We faced this issue with etcd in spite of using a version of docker newer than the one mentioned in errata.

Unwarranted emails sent to users

We didn’t realize that we were hitting the etcd issue until it was a bit too late. We formatted Jenkins PV once after realizing that Jenkins had anyway deleted all the data that might have been of any use. After that, we triggered the seed-job to parse my fork of container index so that it didn’t send emails to the users. Things seemed to be under control so, we switched the seed-job to refer to main container index. But the jobs that were not created during the earlier run of seed-job due to etcd restarts got created in the subsequent run and few users were notified with an email that said “First build of the container image” as the cause of build.

That’s it!

Wow, this one turned out to be longer than expected. Overall, we had a crazy weekend (yes we were now working on a Saturday). We failed to send weekly scan emails as well. The taste of stability, that we got after moving to OpenShift based infrastructure in late September, was swiftly taken away by this incident. It taught us a number of lessons in the process.

Next and the last post in this series is going to be a quick gist of the lessons we learned during this exercise.

Until next time… 😄

Centos Container Pipeline OpenShift Upgrade Part 3 - 'Ready for Production!'

This blog is a part of the series of blogs on OpenShift OKD upgrade we performed for the CentOS Container Pipeline project. If you haven’t read the first part and the second part, this post might not make a lot of sense. Go ahead, read them first.

I ended the previous post with describing the unconventional path we unwillingly decided to go with and how that led us to figure the cause of the issue we were facing while following recommended path for OKD 3.9 to 3.10 upgrade.

Uninstall, install, repeat

As mentioned in the previous post, the idea was to:

  • uninstall OKD 3.9,
  • install OKD 3.11,
  • ensure that PV used for Jenkins in 3.9 setup was usable with 3.11 as well without data loss

Just like OKD 3.10, OKD 3.11 also uses openshift_node_groups dictionary in place of openshift_node_labels and takes the configuration for kubelet arguments with it.

While installing OKD 3.11, we encountered an issue wherein Ansible playbook would exit with an error related to OpenShift master. The error was:

{"changed": false, "msg": "Node start failed."}

This kept on repeating every time we tried to install OKD 3.11 on the nodes where OKD 3.9 was earlier installed.

Fixed two issues in one shot

We opened an issue on GitHub for the installation error. Unlike the failure with OKD 3.10 upgrade, this time the logs were more helpful in making sense of the underlying issue.

It seemed like we made a mistake in configuring the value for openshift_node_groups variable. My takeaway from the way we fixed the issue is that the value in the dictionary has to be a list even if it’s just one value and not multiple values.

After this we faced another error that had to do with the value we had set for the key kubeletArguments.image-gc-high-threshold in openshift_node_groups. This is something I found from the logs. After trying few values that I thought might help origin-node service to start well, I gave up and removed every configuration related to garbage collector from the openshift_node_groups variable.

This not only helped us perform a successful installation of OKD 3.11, but also turned on a bulb in my head that illuminated something like, “Remove the garbage collector configuration and try OKD 3.9 to OKD 3.10 upgrade the recommended way!” And so I did.

Successful OKD 3.9 to OKD 3.10 upgrade

It turned out that the bulb illuminated the right thing. Apart from facing a few minor issues (minor because we didn’t need to open a GitHub issue), we managed to upgrade our OKD 3.9 cluster OKD 3.10 using the OpenShift Ansible playbooks! It was a “F**k yeah” moment for me.

Over next few days, we managed to successfully perform OKD 3.9 to OKD 3.10 upgrade in the development environment before we decided to replicate the steps in production.

That’s it!

Since the recommended upgrade path worked well for us, we didn’t bother to test if doing uninstall and install of different versions of OKD and then using the same PV for a persistent Jenkins deployment worked well or not. We felt ecstatic with the way things worked out. Little did we know how the Jenkins thing was going to be one among several dogs that were planning on biting us!

By the time we reached this point, it was more than 3 weeks since we started off. I visited the Serengeti National Park during this timeframe and if we count the vacation, it was more than 4 weeks since we started.

Until next time… 😄

Centos Container Pipeline OpenShift Upgrade Part 2 - 'The Steps'

This blog is a part of the series of blogs on OpenShift OKD upgrade we performed for the CentOS Container Pipeline project. If you haven’t read the first part, this post might not make a lot of sense. Go ahead, read that first.

In the second installment of the series, I’m going to describe the steps we divided our plan into. Or rather, the steps we had to break things into because upgrading from OKD 3.9 to OKD 3.11 was not possible without upgrading to OKD 3.10 first.

Note: The steps mentioned in this post were executed on a development/pre-production environment that lives on Red Hat infrastructure and not on CentOS infrastructure where the production environment lives.

OKD 3.9 to OKD 3.11 via OKD 3.10

Upgrade prerequisites section in OKD 3.11 in-place cluster upgrade documentation says that an existing OKD cluster needs to be upgraded to “latest asynchronous release of version 3.10” before attempting an upgrade to OKD 3.11. You can go ahead and search for those exact words in the quotes on the docs page to be sure that I’m not lying!

OKD 3.9 to OKD 3.10 upgrade

Having successfully upgraded from OKD 3.7.2 to OKD 3.9, I was confident that upgrading OKD 3.9 to OKD 3.10 would be as simple as a walk in the park. But just as life has its ups and downs at most unexpected times, OKD had plans of making us go down a rabbit hole when we least expected it to. With OKD 3.10:

  • the openshift_node_labels got deprecated,
  • having a proper DNS was mandatory,
  • the openshift_node_kubelet_args got deprecated

DNS setup in development environment

Even before trying to perform an upgrade, we had to configure proper DNS in our development environment.

Our development environment, running in Red Hat infrastructure, had OpenShift OKD cluster running as VMs on top of KVM and libvirt.

Till OKD 3.9, using IP addresses as hostnames in Ansible inventory file was just fine. But with OKD 3.10, it was mandatory to use hostnames that would resolve through a DNS server. Even /etc/hosts entries wouldn’t be considered valid unless a proper DNS was set up.

Our production environment didn’t have to be altered because DNS worked fine there but in our development environment, we had to set up a DNS server. This blog post by a fellow Red Hatter came to the rescue!

First attempt at upgrade

After having grasped the changes required to make to the Ansible inventory file work well with OpenShift Ansible playbooks for OKD 3.10, we attempted the upgrade. The initial attempt was futile and we faced error due to a typo in the variable name.

Serious errors during the upgrade

After the typo issue, next error we faced was more serious. Serious because it didn’t seem like upstream had any clue about a fix either. We opened an issue on GitHub and added a bunch of information that we found while troubleshooting the error by ourselves.

Logs complained about No existing node-config.yaml, exiting in spite of having the /etc/origin/node/node-config.yaml in place. The issue, as we later understood, was a misconfiguration of kubelet arguments in the new way of doing things with OKD 3.10.

But this one took a really long time because the error message didn’t seem to have the correct indication of the underlying issue and, as you can see from the GitHub issue, we tried a number of things in absence of help from upstream.

This issue taught us our first and one of the most important lessons - “We lack OpenShift expertise within our team.”

It’s one thing to use OpenShift and another to know it inside-out. However, it’s not an easy thing to fix due to the huge size and fast pace of development of Kubernetes, and hence that of OpenShift.

The unconventional path

Due to the above error, we faced during OKD 3.9 to OKD 3.10 upgrade, we started to lean towards doing something that’s not mentioned in the documentation and quite possibly the least favorable way to upgrade. We planned to:

  • uninstall OKD 3.9 from the cluster
  • install OKD 3.11 on the same cluster
  • preserve the Jenkins build data on the PV so that when we attach the same PV to Jenkins on OKD 3.11, we can possibly resume from where we left

That’s it

We managed to do some crazy stuff while going down the “unconventional” path mentioned above. But the craziest thing was to find a solution to doing an OKD 3.9 to OKD 3.10 upgrade in a clean way. That is, we managed to find a solution and the cause of the issue we faced while following the recommended path for OKD 3.9 to 3.10 upgrade.

Until next time… 😄

Using Buildah to build container images on Openshift

In CentOS Container Pipeline, we use Jenkinsfile strategy of OpenShift to build container images, scan them and push them to the registry. To build a container image, we need to make sure that the pod started by Jenkins is privileged (akin to docker run --privileged) and that we’re sharing the Docker daemon socket (/var/run/docker.sock) from the host with the container. Anyone with security in mind would dislike the idea (and I hope Dan Walsh doesn’t read how insecurely we’ve plumbed certain things here.)

After having stabilized the service on its new architecture, we wanted to explore if we can use buildah to build the containers and eliminate the need of sharing Docker socket with all the pods that spin up on an OpenShift node (we also managed to blow up the service after stabilizing it but, that was not intentional.)

Challenge #1 - Running buildah inside a container

Since we use OpenShift to build images and Jenkinsfile strategy to do things other than just building them, we had to make sure that image building can be done inside the pod that’s dynamically brought up by Jenkins server on the OpenShift cluster. To be able to do this, we had to use a newer version of buildah than one provided by CentOS repos.

Challenge #2 - Go package is only available via SCL in RHEL/CentOS

CentOS repos come with an outdated version of buildah so we had to build the buildah binary. To build it inside the container was non-trivial (at least for me) and it took me some time to finally come up with a script to do it.

Challenge #3 - Buildah still failed with ERRO[0000] 'overlay' is not supported over overlayfs

In spite of managing to overcome first two challenges, we were hitting an error with overlayfs:

$ buildah images
ERRO[0000] 'overlay' is not supported over overlayfs    
ERRO[0000] 'overlay' is not supported over overlayfs    
'overlay' is not supported over overlayfs: backing file system is unsupported for this graph driver
'overlay' is not supported over overlayfs: backing file system is unsupported for this graph driver

The error was fixed by using a tip from this GitHub comment.

It works!

I’m sure that the solutions/workarounds that I have mentioned above are not perfect. But that’s what we managed to get our issues fixed with and, more importantly, be able to build container images using buildah from inside a pod/container on OpenShift. Eventually, our Jenkinsfile was modified to use buildah bud instead of docker build.

Open challenges

Although we’re able to use buildah bud to build images, we still need to find a way to be able to scan the resulting container images. As it is right now, docker run doesn’t work to spin up images created using buildah unless the image has been first pushed to an external registry and pulled from there. But I’m guessing this is because the Docker socket is being shared from the host system and buildah is working within the confines of the container (which is a good thing, I guess).

So the next thing we’re trying to figure out is to use podman run instead of docker run to get done with scanning the container image. However, we need to do this from inside the container and we’re already seeing the error reported in this GitHub issue.

That’s it

We are still working on this and are not using buildah bud in production. But we plan to change that soon. If you have any ideas/suggestions that can help us do things in a better way, let us know! We hangout on #centos-devel IRC channel.

Until next time… 😄

Centos Container Pipeline OpenShift Upgrade Part 1 - 'The Plan'

This blog post and others in the series are going to be about the OpenShift cluster upgrade that our team performed for CentOS Container Pipeline during December 2018. To find all the blogs on the topic, check the openshift_upgrade_2018 tag.

OpenShift Origin was renamed to OKD (Origin Community Distribution of Kubernetes) so I’ll be using the term OKD in this and future posts to refer to OpenShift Origin.

What we had

We had a stable and functional OKD cluster running on OKD 3.9 that was being used as the base for CentOS Container Pipeline service. Having things stable at infrastructure level was still a new thing for us. Before we revamped the CentOS Container Pipeline to run on top of OKD, things were quite shaky. We used to keep firefighting most of the time because the heterogeneous cluster was far from stable!

We had setup Prometheus and Grafana with the help of openshift-ansible playbooks. We also had a custom Grafana dashboard to monitor the number of builds (both successful and failed), the number of builds in different OKD stages like New, Pending, Running, Complete and Failed, and a graphical representation of the amount of free memory on the nodes in OKD cluster.

What we wanted

We wanted more monitoring and alerting. In the sense, we wanted to be notified when some build is taking a long time, or when there are more than x failed builds in a short duration, or if there were more than x builds hanging around in Pending state, so on and so forth. The intention was to be able to stay on top of things unlike how screwed up they were in earlier architecture where many times it was the users who notified us about something blowing up in our environment while we were busy writing new bugs (read ‘features’). 😉

The Plan

As mentioned earlier, we were already using Prometheus for monitoring and had to configure Alert Manager to add rules based on which it would send us alerts. But instead of doing that, we stumbled upon the fact that OKD 3.11 was going to ship with built-in Prometheus Cluster Monitoring and we thought to ourselves, let’s use this instead of writing our own rules.

To be honest, the assumption was that, in OKD 3.11, Prometheus would have some default set of rules which would make our job easier. Now making assumptions is probably never a good idea in software engineering. But we still managed to do that.

We decided that instead of writing rules for the Prometheus setup that we have in OKD 3.9, we would upgrade our cluster to OKD 3.11 and see what’s available by default and add rules on top of it. Of course, we didn’t know at the time that a better idea would have been to RTFM the upcoming shiny thing called as Prometheus Cluster Monitoring instead of diving into the task of upgrading the cluster.


In this post, I mentioned what we wanted to achieve and what we thought would help us do that and what we assumed in process. In future posts, I’ll mention what we did, how we messed up and how we made sure that the taste of stability was taken away. To complement that, we screwed the production on a Friday when the production Gods are not in a mood to be disturbed.

Until next time… 😄

Centos Container Pipeline Status Update November 2018

It’s been two months since we deployed the CentOS Container Pipeline service based on OpenShift in production. Since deployment, the service is running on top of OKD (new name for OpenShift Origin) 3.9. We haven’t faced any major issues that would keep us fire-fighting. In essence, we’ve been able to focus on implementing things that were available in previous architecture but are not yet available in current one.

At the moment, we’re working on:

  • Writing the APIs for fetching logs from Jenkins server running atop OKD. This is to provide users with the logs from lint, build, scan phases of the pipeline. We’re writing:
    • A backend which will fetch the information from Jenkins
    • And a frontend that will be accessible to users via
  • Add container images for the projects on CentOS-Dockerfiles repo which were not already available on the container index.
  • Generate statistics for number of times a container image has been pulled from
  • Add more functional tests so as to have a more reliable CI.
  • Moving from OKD 3.9 to 3.11 so that we can use built-in Prometheus Cluster Monitoring instead of writing our own rules.

One of the major pain points has been upgrading OKD from 3.9 to 3.10. We opened a GitHub issue for it but haven’t been able to reach a resolution yet.

That’s it for now. We will share more as we make progress.

Restructuring CentOS Container Pipeline using OpenShift - Part 2

If you have not read the first part of this series on CentOS Container Pipeline service then please go ahead and do it!

In this post, I’ll be talking about the issues we faced and how we resolved them when we tried to build a large sized container-index with the newly developed service.

What we did?

One fine morning I had a thought of trying to build container images for all the projects that build their images with our service already. But I wanted to do it with the new deployment of the service which is based on OpenShift.

First things first, I modified the container-index to make sure that notification emails don’t get sent out to the actual users but land in my inbox instead. Otherwise it would have been one hell of a testing for me! 😉

I had a 3-node cluster (1 master and 2 nodes) which I was going to test things against. So far, we had tested things only on a small sized index of maybe 5-10 images. Obviously that was a mistake and we should have done this testing of full container-index sooner rather than when we’re so close to deploying things into production.

What we observed?

When I triggered the full index to be built on my deployment of the service, I saw that the seed-job that parses all the yaml entries in the container-index picked up the jobs and started parsing them. At the same time, the jobs that it had already parsed started getting served by OpenShift and Jenkins, i.e., builds were triggered.

Some of the issues we observed:

  • Numerous jobs were failing due to unavailability of Docker daemon socket to build the image.
  • Many images were taking hours to build completely instead of taking a few minutes.
  • Seed job failed to parse a particular entry and in event failed to parse the remaining index.
  • Jenkins used to get restarted and thus number of pods were left orphaned while Jenkins console reported that it cannot connect to the pods.
  • Scanners failed to scan certain images.
  • Weekly scan doesn’t run

How we fixed the issue?

Docker daemon issue

It turned out that OpenShift template used to spin up seed-job and build jobs (jobs that would build container image for parsed entries) had same name. The pod template for seed-job didn’t have Docker socket configuration because it doesn’t need the Docker daemon for anything. So the build jobs triggered while seed-job was still under execution used the same pod template which didn’t have Docker socket available.

Figuring out the issue took an embarassingly long time and was finally fixed by this PR.

Builds taking hours to complete

This was happening as a result of 200+ build requests coming in in quick successcion. During development, we tested with a small index and this never became an issue.

To fix this issue, we introduced batch processing of the jobs. With this in place, we default to processing only five jobs at a time, poll the OpenShift API every 30 seconds to see how many jobs are still new/pending/building and process the next batch if there are 2 or less projects being processed.

The benefits of batch processing from what I observed are: - Quicker build times - Abiliity to larger container-index even on a single node OpenShift cluster (like minishift). This is not tested yet but I tried to build entire index on just two node cluster things worked just fine!

Pull request for batch-processing is available here.

Seed job failing to parse particular entry

This issue has to do with OpenShift’s rules for naming the builds. You can’t have an upper-case letter in the build’s name. So we had to make sure that all build jobs being created were converted to lower case before creating them on OpenShift.

PR for this is available here.

Jenkins restart issue

This was relatively simple to fix. We provided more memory to the Jenkins deployment and things started working fine. However, with batch processing in place, I think we can even do with less amount of memory because there’s not going to be a lot of builds being processed at the same time.

One thing I learned (thanks Pradeeptp) was that you can’t increase the number of pods of Jenkins to address this issue because Jenkins does not scale well horizontally. Increasing its compute resources (CPU/Memory) is the best way to mitigate issues related to it.

Scanners failing for certain images

This issue showed up because of more than one reasons: - We parsed the output of commands like yum check-update from stdout. When the output for single package spanned over multiple lines, things broke. - CentOS 6 and CentOS 7 have different ways to check for updates. One uses yum check-update and other uses yum check-updates. - We use /bin/python as an entrypoint for the scanners when they run scripts inside the image under test but using /usr/bin/python ensures that things are compatible across both CentOS 6 & CentOS 7

These issues are being fixed by this PR.

Weekly scan failures

We found that weekly scans were not getting triggered by Jenkins at all. Now the weekly scan jobs are configured as Jenkins Pipeline which are to be run as cron on weekend. But Jenkins doesn’t run them unless they are already triggered once. There’s a JIRA ticket for this issue that I can’t find right now. It’s closed with “Won’t fix”.

So we came up with another approach which is using the OpenShift cron to run a script that triggers build for all such weekly scan jobs. PR for this is available here.

What’s next?

At the moment, we are busy fixing this issues we discovered by trying to build entire container index on the cluster. We have fixed most of them and have PRs open for remaining ones.

Next, we’re going to get the email notification piece working and deploy things to a more production like environment before finally moving things to production.

That’s it!

If you know of better solution/approach to fix aforementioned problems we faced, please let us know. You can catch us on #centos-devel IRC channel on the Freenode server. I hang out on the channel as “dharmit”.

Until next time… 😉

Restructuring CentOS Container Pipeline using OpenShift - Part 1

In this post I’m going to talk about why we are restructuing the CentOS Container Pipeline service and how OpenShift is key to it. There’s bunch of material available on the Internet so I don’t really need to write about OpenShift.

I gave a talk at about our service and got some great feedback. Most common feedback was that such a service would be immensely useful to the opensource community. But deep down, I knew that it’s not possible to scale the current implementation of service to serve a large community. It needed a rejig to be useful to the users and not a pain in bad places for its administrators! 😉

What does the service do?

Before I talk about the issues and how we’re handling them in new implementation, I’ll quickly jot down the features of the service.

  • Pre-build the artifacts/binaries to be added to the container image
  • Lint the Dockerfile for adherence to best practices
  • Build the container image
  • Scan the image for:
    • list RPM updates
    • list updates for packages installed via other package managers:
      • npm
      • pip
      • gem
    • check integrity of RPM content (using rpm -Va)
    • point out capabilities of container created off the resulting image by examining RUN label in Dockerfile
  • Weekly scanning of the container images using above scanners
  • Automatic rebuild of container image when the git repo is modified
  • Parent-child relationship between images to automatically trigger rebuild of child image when parent image gets updated
  • Repo tracking to automatically rebuild the container image in event of an RPM getting updated in any of its configured repos

Issues with the old implemention

Our old implementation of service has a lot of plumbing. There are workers written for most of the features mentioned above.

  • Pre-build happens on CentOS CI (ci.c.o) infrastructure. In this stage, we build the artifacts/binaries and push it to a temporary git repo. The job configuration then uses this git repo to build container images while another job on ci.c.o keeps looking for update in the upstream git repo.

  • Lint worker runs as a systemd service on one node.

  • Build worker runs as a container on another node and triggers a build within an OpenShift cluster.

  • Scan worker runs as a systemd service and uses atomic scan to scan the containers. This in turn spins up a few containers which we need to delete along with their volumes to make sure that host system disk doesn’t get filled up.

  • Weekly scanning is a Jenkins job that checks against container index, and underlying database of the service before triggering a weekly scan

  • Repo tracking works as a Django project and heavily relies on database which we have almost always failed to successfully migrate whenever the schema was changed.

All of the above is spread across four systems which are quite beefy! Yet, we couldn’t manage to do parallel container builds to serve more requests. A couple of teams evaluated our project to bring up their own pipeline because they didn’t want to use public registry. However, they found the service implementation too complex to understand, deploy, and maintain!

How are we handling (or planning) things in new implementation?

In the new implementation of the service which is still to be moved under the official repo, we are using OpenShift exclusively for everything, for every feature that the service provides.

Although we’re far from done, we have successfully implemented and tested that these features work fine in an OpenShift cluster:

  • Pre-build
  • Lint
  • Build
  • Scan
  • Weekly scan

We’re relying heavily on the OpenShift and Jenkins integration. Every project in the container index has an OpenShift Pipeline of its own in the single OpenShift project that we use. All of the implemented features work as various stages in the OpenShift Pipeline.

For logging, we’re using the EFK (Elasticsearch - Fluentd - Kibana) integration in OpenShift. To be honest, we’re still learning how to use Kibana!

This new implementation hasn’t been deployed in a production environment yet. However, it’s relatively straightforward to deploy than the old implementation and can even be deployed on a minishift environment.

In progress items

We are still working on things like:

  • proper CI (unit and functional tests for the code and service)
  • setting up monitoring and alerting stack using OpenShift’s integration with Prometheus
  • providing useful information to the users in the emails that are sent after every build and weekly scan
  • rewrite the entire repo tracking piece to make it as independent of database as we can

That’s it

This blog went on to be longer than I wanted it to be! But it’s a gist of what we’ve been doing as CentOS Container Pipeline team since past few months. In coming posts, I’ll be talking about individual implementation details!

Until next time… 😄

Talking Containers at IIT Gandhinagar

Series: Speaking

I’m an engineering graduate from India. IITs are the best engineering institutes in India. NITs follow the suite. Although some might want to debate, that’s not the point of this post. I pursued my engineering education from a non-IIT and non-NIT institute. I think most engineering graduates in India have dreamed of being in IIT/NIT and I’m no exception. Studying at IIT was my dream as well but, it’s not an easy feat to get into one.

So when I got an opportunity to speak at IIT Gandhinagar, I was obviously excited. Topic was containers. And the expected audience was extremely dispersed. I was expected to address under-grad, post-grad and doctarate students from Computer Science and Electrical Engineering.

Considering the vast audience, I decided to start from the beginning about how containers evolved, how they compare with VMs, how Docker popularized them, how Kubernetes took over the orchestration world and how containers have made a lasting impact on the development and deployment story. I also discussed about monoliths and microservices.

The Talk

First, the slides:

I started with a show of hands to know how many in the audience had heard of or played with containers. Surprisiginly there were just about 2-3 people. Total audience was around 25-30, I guess. So my decision to start from how it all began proved to be right.

As I went ahead with the slides, I got some questions around the slide that talked about Docker commands.

Although there weren’t many questions around the rest of the slides, it did seem like the topics mentioned in the slides were likely unheard of earlier by the audience.

At the end of the talk, one of the PhD students told me that she found my talk interesting+informative but she’s not much into the architecture stuff. At that time, I was a bit confused about why she said so and didn’t get a chance to talk further. However, I don’t think my talk was around computer architecture. It did talk about the microservices pattern but due to lack of time, I didn’t dive deep into it.

Informal discussions with faculties

I went to IIT Gandhinagar upon invitation from one of the faculty members. And two more faculty members attended my talk. I got an opportunity to interact with three faculties and the discussions were pretty interesting. Having talked at few other colleges, I felt that, in general, challenges they face are pretty much the same.

General opinion of the faculties was that most students weren’t really aware of much of the things happening in the world of containers. One of the surprising discussion was around how students felt that it was more rewarding to work on making web/mobile apps than working on Operating Systems, Networking, and other things that keep you closer to the hardware. Having worked with people doing both sort of things, I certainly feel like working on the latter is rewarding as well. Sure the hype is around making next breakthrough (read multi-million dollar) app but, the core things that make them possible are seeing development as well!

We even planned on having a workshop to have hands-on about containers and developing an app based on microservices pattern. Idea of using OpenShift to do CI/CD, build images and eventually deploy the code seemed interesting to the faculty members.

Above all, and probably the best part, the faculties were more than willing to host community meetups at IIT Gandhinagar campus. I’m really looking forward to hosting a meetup at IIT Gandhinagar in near future! I’m sure many engineers would be excited to visit an IIT. 😉

Mistakes on my part

  • I didn’t manage to get a single picture of the talk.
  • I forgot to bring any laptop stickers for the students.

That’s it

Being a non-IIT grad, it was surely an exciting experience to be speaking to IIT students about the technologies that are changing the landscape. IIT Gandhinagar’s location is pretty amazing and far, far away from any sort of pollution one can think of - hence my kind of place. 😉

Until next time… 😄 2018 and my Europe visit

Series: Speaking

This year’s was my first time at DevConf and also my first time speaking at a conference. It was attended by about 1500+ people! That’s massive, isn’t it? I was super excited from the moment my talk was accepted.

Note: This is not an event report. It’s more of an informal description about my experience attending the conf and visiting Europe!

Giving the talk

I was speaking about CentOS Containr Pipeline - the project I have been working on since some time at Red Hat. So far we had been quietly working on developing the service but at, we wanted to talk about it and tell more people that we’re ready to on-board them to our service. In a nutshell, motto of our project is to help open-source projects build and update their container images so that they can focus on developing awesome stuff instead. 😉

Since it was my first time speaking at a conference, I was obviously nervous. To top that, I saw some really well versed fellows in the list of people interested in attending my talk. I was excited and nervous at the same time! To my surprise, my talk managed to get pretty high in the list of popular talks. At its peak, it was on 13th spot before sliding to 21st spot on the popular talks page. There were about 250 talks in all.

Except going a bit fast, I think it went good. One of the tips I received for the talk was to “learn to breathe while speaking.” 😆

Here’s my talk that explains what we provide to any open-source project that joins us:

I got some interesting questions at the end of the talk and, in general, people loved our work. Few people even wanted to do an on-premise installation of our service. It’s out there on GitHub so practically anyone can do that but, we’ve not made effort in deploying it beyond our infra so, YMMV! 😆

People interactions

Besides the talk, I had a great time talking with people. I’m more of an introvert but I love talking with people. It’s just the starting a conversation is extremely difficult for me! So giving a talk sort of helped me have a topic to talk with at least those people who attended it (my talk had about 100 attendees.) Besides that various colleagues whom I knew only by name were also present at the conf.

At the end of the conference, I think I socialized about 3 months worth of my average socilzation in just 3 days! Since I love talking with people and giving talks (doing meetup talks on regular basis) this was most positive takeaway from the trip!

Travelling Europe

If you have seen the iconic Bollywood movie “Dilwale Dulhania Le Jayenge”, you would know that most of us in India are really fascinated with the idea of traveling to Europe. 😉

I traveled to a few cities on the sidelines of conference. My friends who had been there earlier had warned me about how cold it might get in the part of Europe I was visiting. Fortunately, it didn’t get so cold and I ended up loving the weather instead! The part of world where I live, we don’t feel winters colder than around 10 C. Even then, noons generally end up getting warmer.

But in Europe, it was cold by my standards for the entire day. And I absolutely loved it. To be honest, I was not excited about cold as it might seem from the post. I was planning to cut my trip short and come back to India just because the weather conditions seemed so scary to me (my team’s still making fun of me for this). Glad I didn’t make that change.

Properly laid out public transport system in all the cities I visited (Brno, Prague, Budapest and Vienna) was a surprise to me. The accuracy of Google Maps in India is somewhat flaky at times but in Europe, it was the most used and relied upon app for me. I was shocked when Google Maps even pointed that a tram/metro was running a minute or two late than it’s scheduled time.

European architecture has always fascinated me. It’s elegant and beautiful in its own way. Places like Prague Castle, Parliament building in Budapest, and your average buildings across the city left me dumbstruck on multiple occassions.

History is another reason why I was always so fascinated about visiting Europe. From what I could gather during this short and haphazard trip, World War II has left a lasting impact on Europe. I did study about it during my schooling but it was from Indian point of view. In Europe, you get to know that even after seven decades, World War II is stil among the most crucial events in history.

European countryside is something you normally keep wallpapers of (because wallpapers of Maria Sharapova, Yami Gautam and the likes are distracting.) I didn’t get to explore much of European countryside during the visit. And since it was winter, I didn’t feel disappointed about it. Had it been European summer, I’d have been cursing instead. This is something I want to be able to do if I get a chance to visit Europe again.

That’s it

Okay, I need to stop somewhere. My trip to the conference and Europe in general was filled with lots of experiences and if I try to write it all down, it’ll get too long.

Knowing more about European history by visiting various memorials, palaces, churches, etc. is one thing I wish I had done better. Tasting different beers is something I did well on my first visit already. Realizing I could enjoy cold weather (not Antarctic cold!) was a pleasant surprise! Being a pure vegetarian guy who doesn’t even eat eggs was a major pain in the ass. Meeting so many awesome fellow Red Hatters was amazing.

Hopefully, I’ll be writing about DevConf 2019 and maybe FOSDEM as well in about a year’s time. 😉

Until next time… 😆

Setup a 2-node OpenShift cluster

Recently I started working more than usual on OpenShift. I’ve contributed to the minishift project and also spoke about OpenShift at a local meetup. But now we’re planning to move to microservices architecture based on OpenShift.

I use CentOS’s DevCloud infrastructure to setup test instances. And it is on the same infra that I brought up my first real OpenShift cluster deployed using openshift-ansible. I used this hosts file along with the playbook available under byo directory for OpenShift 3.6.1. It’s a 2-node cluster where one system behaves as master and other as a node.


For a smooth installation and setup, I had to ensure a few things like:

  • Install NetworkManager and firewalld on both the nodes.

  • Start NetworkManager and firewalld manually on both the nodes. To get this working, I had to set SELinux to permissive. I didn’t dig much into this but I think with proper context, I could have got it working in Enforcing mode as well.

  • Ensure that /var partition has about 40 GB of free space. I think the requirement is 15 GB on master and 40 GB on node. But I ensured 60 GB of space for /var on both nodes.

  • Modify /etc/hosts file on both nodes so that they are able to access each other by their hostnames.

  • Install the RPM python-rhsm-ceritificates so that Ansible can pull an image from

After finishing these steps, perform the installation:

$ ansible-playbook openshift-ansible/playbooks/byo/config.yml

I did it off the release-3.6 branch.

Post-install setup

Add a new user

Having used simple OpenShift cluster in past, it was a bit of struggle this time to get into the OpenShift console. minishift does this very nicely so that end user doesn’t have to bother.

It also took me some time to add a user to the cluster. I probably couldn’t find my way around the huge OpenShift documentation.

We just need to execute below as system:admin user:

$ oc create user dharmit --full-name="Dharmit Shah"
$ htpasswd /etc/origin/master/htpasswd dharmit

# and if you want the user to have cluster-admin privileges
$ oadm policy add-cluster-role-to-user cluster-admin dharmit

This user can now login to the OpenShift web console using the credentials you’ve just set. It took me really long to get up to this step!

Deploying an image to OpenShift cluster

Now I wanted to run a simple beanstalkd image on OpenShift cluster. All I did was use to build an image and pull it. After successfully logging in and navigating to the project page, OpenShift shows you an option at the top called “Add to Project”. I chose “Deploy Image” option from this and gave the image name to be deployed.

OpenShift automatically created a DeploymentConfig and created a service based on the metadata of the image. It also provides the name of the service. This name can be used by other objects in the same project to access beanstalkd seamlessly!

Still learning

I’m still learning OpenShift through the docs. The Persistent Volumes concept has been interesting! I’m working on creating an architecture wherein various workers can run in tandem as containers and read/write things to a remote NFS server configured as a PV.

There’s lot to learn and do before we can achieve a microservices based architecture. I’ll try to keep this space udpated. 😉

An Extra Mile

Last week, most of my time was occupied in debugging an issue whose cause was not very obvious when we hit it. A teammate of mine opened a pull request that failed CI check every single time. He updated the PR about four times but, CI never felt satisfied. Finally when there was no more code to add to the PR and CI seemed stubborn, there was a need to dig deeper into it.

A few points (disclaimers, maybe):

  • Our project’s CI check is not very comprehensive. It’s mostly a bunch of functional tests that were six months back with then master branch as base for tests.

  • There are no unit tests.

  • CI check prints a lot of logs and does that only when the check fails. When everything’s OK, it doesn’t print more than basic information. When something fails assertion checks, it prints logs like crazy:

    • logs from Ansible provisioning
    • logs from all workers in our service
    • nothing in a structured manner
  • Tests were written by a single developer with minimal input from others in the team. Although that developer did a great job, we didn’t add more than a few tests or refactor the existing ones.

I had played almost no role in coming up with the tests. So when I had to go figure why CI checks were failing, I had to learn from scratch about how the current setup worked.

Our code-base takes good deal of time to get deployed in test environment. We use Ansible playbook for deploying things in dev, pre-prod and prod environments. Doing deployment on freshly provisioned nodes would take up to 30 minutes while deployment on existing VMs would take about 15 minutes.

After doing the deployment, I had to go to IPython shell and create a few dictionaries that were then served to the test suite. These dictionaries mainly contained IP addresses of the VMs on which tests were to be executed. Finally, the test suite, created using nosetests, was executed.

Surely, I was seeing similar results in dev environment as CI when the test suite was executed. Logs made no sense! Assertions failed and it seemed like they were getting executed before the tasks finished. All I was wondering was, how are assertions going to succeed when they check result of a task that’s still not finished? And it seemed that way to my teammates as well.

I was about to tell the team that this process is becoming frustrating and not taking us anywhere when a strange voice in my head asked me to look one more time - the extra mile!

It took me about 15-20 minutes to find error in one of the worker services. Surprisingly, the error didn’t appear in the CI logs! Nor did it appear in the log file we generate for every job that passes through the service. It might have seemed obvious that I should’ve checked the particular service right from the beginning. I didn’t do that as our team was living in an assumption that we’re spitting out logs for every service in the CI tests; specially when something fails.

Key lessons that I took away:

  • There’s immense scope for improvement in our existing CI framework.

  • We’re doing ourselves and our users a major disservice if we keep focusing on adding features and not on fixing a core part like tests.

  • Log aggregation needs to improve big time. More importantly, we need to ensure that useless actions are not logged!

  • Break down the CI check into multiple pieces. One Jenkins job can do the deployment, other to run unit tests, yet another to run functional tests, so on and so forth. If any one of the jobs fails, we stop the entire thing. That helps us save time and, more importantly, figure what piece is exactly failing.

  • Use less Python magic wherever we can. For example, Ansible playbook can be directly executed from shell instead of using Python.

  • Focus on this right away instead of waiting for a major issue to bring things down!

  • Going an extra mile is often rewarding. 😉

I wrote this blog to bring out my thoughts. In process, if it helps someone else, I’d be surprised! But if you have any suggestions or comments about how we can improve, please let me know in comments.

Until next time. 😄

Teach myself Go through a side project - Hacker News on Terminal

I have had a pretty bad track record at learning new programming language by myself, at consistently writing a blog, at continuing working on (or being excited about) the side project I’m doing. That sounds like having a bad track record at pretty much everything that matters in the world of open source and programming. 😞

It took me really long to learn Python. I remember starting from print Hello World a number of times because it had been really long since I last wrote anything in Python. I’ve been bad at coming up with side project ideas or contributing to many open source projects. It wasn’t until my job required me to write Python full time that I started getting good at it. 😏

Same goes for Go. I have been willing to learn it since mid 2015. Coincidentally, the job change I made around the time required me to learn Go along the way. But I left that job too soon and moved back into the Python world. I did contribute little something to a Go based project that I like. And, secretly, I feel happy when every release announcement of the project requires a mention about the command I contributed. But that’s pretty much it.

So what the heck am I gonna do now? Well, turns out I’m going to try yet another time. 😄

Recently I came across this tweet which resonated with me in opposite way of how it was intended to. 😉:

We all have some spare time and I figured I could use some of it to write something in Go.

So, I planned to do a tiny side project that’s not going to be of much use to anyone except me. But, it would help me learn Go - the growth I’ve wished to make to my profile. This time, I’m going to try and blog as I make progress on the project. That way, I might actually end up making a helpful contribution to myself.

The project’s hosted on GitHub and, at the moment, has nothing but a short README of planned features.

Hope to take this to a closure. Best of luck to me! 😉

Udaipur Trip 2017!

This Diwali, me and my wife decided to visit Udaipur. We started on the morning of New Year day, which is celebrated right after Diwali in Gujarat.

Day 1

mostly travel and getting trapped in city traffic

On the way, we visited a Jain tirth called Rishabhdev which seemed more of a Digambara tirth than Shwetambara tirth. The temple was nice. I had already been there once about 10+ years back. But didn’t remember much of it.

After reaching Udaipur around 5 PM, we went out to have food. And in the quest of visiting couple of places of interest, I got us trapped in city traffic near Fateh Sagar Lake. I took the car to really narrow lanes which were not well suited to drive a car through it. 😉

Day 2

we absolutely regretted visiting Udaipur

Our first full day in Udaipur started with breakfast with our hosts (we booked homestay via Airbnb).

Our first stop was Sajjan Garh Palace or the Monsoon Palace. Except the view from the top and an ambient cafe, we didn’t find anything really interesting in the palace itself. Kullad chai at the cafe was nice!

Our next stop was the Sajjangarh Biological Park which is just beneath the Sajjan Garh Palace. Park itself is pretty nice but the crowd and the time of day ensured that we wouldn’t see many animals. Those we saw, were in rest mode after their lunch. Apparently, like humans, even animals prefer having a nap after lunch. 😉

The biological park is pretty big and considering the hot weather we visited Udaipur in, it was only wise to roam around the park in a golf car.

Next we went to Gulab Bagh which is the biggest garden in Udaipur. We wanted to visit the Gulab Bagh Public Library inside the garden but it was closed. Maybe due to Diwali holidays. Gulab Bagh was nice to walk through. It’s a lush green and peaceful area in the most visited part of the city.

From Gulab Bagh, we walked to Udaipur City Palace which is probably the most visited spot in the city. The rush inside the palace museum was too much for me and my wife to handle. And there was persistent nuisance from some fellow tourists - a group of 6-7 guys. It got suffocating due to the rush and we decided to leave the palace after seeing only a couple of rooms in the museum.

Thoroughly disappointed because of the insane crowd we were seeing this day, me and my wife went back to the homestay and decided to rest the next day.

Day 3

it was surprisingly awesome!

We decided a day before to relax at the room and not go out to any tourist spots in the city. Out of habit, I woke up early and decided to read a book on Mewar history that I found at the hosts’ place.

Few pages into the book, I got really excited reading about the history. By the time my wife woke up, I had mentally decided to not relax on this day. 😄

After some discussion with our host, we decided to visit Eklingji temple because of the historical significance it has in Mewar. The temple is about 25 km from Udaipur and our host told us that it might not be as overcrowded as the city itself. We ensured to visit during the darshan timings so that we can take a look at all the parts of the temple. It turned out to be a really good decision on our part and we also got a book from outside the temple that had more historical facts about the temple detailed in it.

While checking out the nearby places around Eklingji, I found that there are two more temples which are as old as Eklingji (built in 8th century CE) or older than it. These were:

  • Sahastra Bahu temples
  • Shantinath Jain temple

While Sahastra Bahu temples were built in early 10th century CE, the Jain temple was built around 3000 years back. The history, carving and landscape of this temples completely amazed us. And they were not even 10% as crowded as the main city.

I’m glad I read up some history and ended up visiting places of historical significance. We surely witnessed some amazing architecture that has stood through the centuries.

In the evening we visited Maharana Pratap Smarak. It is a museum full of history and miniatures of Chittorgarh, Kumbhalgarh, City Palace, and battlefield of Haldighati. The view from the top of Moti Magri hill where the museum rests is also nice!

Day 4

we were left dumbstruck

While reading about Mewar’s history the previous day, I stumbled upon Jaisamand Lake. Since it was a few hours detour on the highway back to Ahmedabad, we visited this lake on the last day.

The road from Udaipur to Jaisamand lake is absolutely amazing. It’s got a lot of turnings and a terrain that rises and falls very, very often. Driving car on this road was nothing short of a pleasure!

Upon reaching Jaisamnd Lake, we were completely blown away by its expanse. It is an artifical lake and second-largest of its type in India (per some Udaipur locals, second-largest in Asia). There’s also an island in the lake which hosts a resort. Should be interesting staying there. 😉

It surprised us how they created such a huge lake so many years back! Technological advancements back then were almost inexistent yet, the constructions (lakes and palaces) from that era were surprisingly amazing.

That’s it!

Udaipur is about 4-5 hours drive from Ahmedabad. Roads are pretty good but, if you’re driving at night, beware of the potholes that show up surprisingly on the highway. There’s a good number of restaurants on the highway.

We plan to visit Udaipur again during off-season period. I’ll be updating the post with few pics of the places I’ve mentioned. Until next time! 😄

Talking Python with students

Series: Speaking

Those who know me are aware that I’ve given talks at various Meetups in the past. I’ve presented at colleges as well. But I’ve never written about it. This time, I decided to write my thoughts (in general). Also I’d write about my experience while talking at Nirma University, Ahmedabad during past two days about career options for Python programmers.

One of the reasons I’m always eager to talk at colleges/universities is that it gives me an opportunity to go back to that lively part of the world where I myself had a great time. School and college days are the days when most of us make best friends and memories. I love and enjoy interacting with the students; and, more often than not, end up learning something from their questions/feedback.

Another reason I like to go there and talk is that I see a pretty big gap between students’ knowledge and industry expectations.

General thoughts (not specific to any college/university)

There’s no shortage of engineers in India - for pretty much any domain. But most companies don’t hire because they don’t find students’ knowledge up to the mark. Students excel at academics but have very less or no clue of what’s really going on in the industry.

For example, Python is gaining momentum in Indian IT industry since past few years. And in spite of being such a beautiful and easy-to-learn programming language, numerous students are unaware of it. I’ve interacted with number of students in past few years, and more than few have mentioned that they get so scared of programming while studying C and C++ that they contemplate giving up on programming! In worse case, sometimes they even make up their mind to do MBA instead. Most, if not all, have felt that Python should be the first programming being taught in college/university instead of C. Debatable topic, nonetheless.

Having studied these programming languages myself, I agree with the notion of making Python the first programming language to be taught instead of C. It helps get students excited about programming. And then it would make more sense to teach C and C++.

Experience at Nirma University


Nirma University was my dream university as a 10+2 student. But, due to health reasons, I was bedridden for almost four months before the final (board) exams and didn’t get to work hard enough to realize the dream. Naturally I was super excited about giving a talk at Nirma! 😄

The students I interacted with were from Electronics & Communication (EC) branch and had a fractional course on Python in their fifth semester curriculum. As a part of it, I was invited as an industry person to talk to them about how Python is used in industry and what kind of jobs/roles/opportunities are open for Python programmers.

To be honest, it’s my favorite part to talk to students about the plethora of opprtunities that Python, Linux and open source in general would open up for them.

Topics I covered (link to slides):

  • History of Python
  • Which organizations use Python and how they use them
  • Detailed discussion about
    • Web Frameworks
      • How Instagram uses Django. Link to their engineering blog
    • How hot the domain of Data Science is and awesome collection of projects under the SciPy ecosystem
    • How Machine Learning and Artificial Intelligence are changing things. I recommended them to watch Person of Interest and Eagle Eye to get a better understanding. 😉
    • Python for Embedded Systems
  • How to leverage MOOC content to learn more about Python and its applications!
  • …and more

While talking about web frameworks, I explained what backend and frontend are. We take the 24x7x365 availability of our favorite apps (WhatsApp, Instagram, FB, Twitter, etc.) lightly; even, for granted. With examples, it was easy to explain them that it’s not really an easy task and it requires good deal of engineering effort. Ironicaly, WhatsApp faced glitches the same day in the afternoon and hashtag #whatsappdown was trending in India for almost rest of the day! I guess they’ll never forget frontend-backend for rest of their career. 😉

That’s it!

I’ve uploaded the slides to Slideshare. If you are one of the students from Nirma University that attended the seminar, consider providing your feedback on the form at the end of the slides.

Let me know what you think in the comments below! Until next time. 😄

Use 'git subtree' to create new repository from a sub-directory

As a part of my ongoing Ansible blog series, I plan to walk-through the provisioning code of CentOS Containr Pipeline service which lives under provisions directory.

My plan:

  • Walk-through the existing code base
  • Make changes to the code without touching my fork of the repo
  • Push and test the new code

For the second part, i.e., modifying the code without touching my fork, I needed to create a new git repository from the said provisions directory.

A bit of Google-fu and this stackoverflow answer came to rescue! All I did was, use this git subtree command to create a new branch that holds the code I’m interested in:

$ git subtree split -P provisions -b provisions-only

Here, provisions is the directory I want to create a new git repo out of and provisions-only is the branch that will hold the code that’s inside provisions directory

Create a new directory outside the directory holding my fork of the repo and do git init:

$ mkdir ~/repos/cccp-provisions
$ cd ~/repos/cccp-provisions
$ git init

And git pull from the fork to new directory:

$ git pull ~/repos/container-pipeline-service provisions-only

Here, ~/repos/container-pipeline-service is the path to the forked repo that I don’t want to touch and provisions-only is the branch in that repo that holds the code I’m interested in.

Whenever there’s a change in the big repo (container-pipeline-service) and I want to copy the changes to cccp-provisions repo, all I need to do is execute the same git subtree command I used above in the big repo, and do git pull --rebase in cccp-provisoins :

$ cd ~/repos/container-pipeline-service

$ git diff --unified=0
diff --git a/ b/
index fd32557..fb14f4f 100755
--- a/
+++ b/
@@ -0,0 +1,2 @@
diff --git a/provisions/ b/provisions/
index f956524..67f70dc 100644
--- a/provisions/
+++ b/provisions/
@@ -0,0 +1,2 @@

$ git add -A

$ git commit -m "Test"

$ git subtree split -P provisions -b provisions-only
Created branch 'provisions-only'

$ cd ~/repos/cccp-provisions

$ $ git pull --rebase ~/repos/container-pipeline-service provisions-only
remote: Counting objects: 3, done.
remote: Compressing objects: 100% (3/3), done.
remote: Total 3 (delta 2), reused 0 (delta 0)
Unpacking objects: 100% (3/3), done.
From /home/dshah/repos/container-pipeline-service
 * branch            provisions-only -> FETCH_HEAD
Updating d203d40..3dc4752
Fast-forward | 2 ++
 1 file changed, 2 insertions(+)

What I did above was: modify two files; one inside the provisions directory and one outside it; committed the changes it to my master branch and using the git subtree command, created a provisions-only branch like we did earlier. When I go to the cccp-provisions directory and do git pull --rebase, git knows that only one of the two files modified in that commit is of our interest and so it changes only one file:

Fast-forward | 2 ++
 1 file changed, 2 insertions(+)

That was pretty cool for me! Now I can go ahead and walk-through the provisioning bits and also make changes to it without affecting my fork of the repo.

Ansible Series: Introducing Playbooks

Series: Ansible

In previous post, we did a combination of trivial and non-trivial tasks on the command line:

  • install httpd package on the servers in group servers,
  • modify a line in /etc/httpd/conf/httpd.conf file to set our desired value for port and
  • set the httpd service into restarted mode.

At the end of the post, I mentioned that we can automate these tasks by writing a playbook. That’s what we’ll do for this post.

Ansible Playbooks

As official documentation suggests, playbooks are Ansible’s configuration, deployment and orchestration language. They provide a very nice analogy as well:

If Ansible modules are the tools in your workshop, playbooks are your instruction manuals, and your inventory of hosts are your raw material.

With the help of playbooks, we can:

  • download and install packages
  • configure services
  • start/stop/restart server processes
  • perform rolling upgrades
  • interact with load balances and monitoring systems

Playbooks can be used for various things and can have various tasks written in them. It would be nearly impossible to cover each and every aspect of it.

In this post, we will cover its basics by creating a playbook out of the tasks we performed in last post.

Plyabooks are written in YAML format. Each playbook can have one or more ‘plays’ in it. Every play is targeted at a group of hosts (serversin our example.)

Let’s create a playbook (store it in, say, playbook.yaml file) for the tasks we performed in previous post:

- hosts: servers
      - name: Install httpd
        yum: name=httpd state=present

      - name: Change port to listen on
            path: /etc/httpd/conf/httpd.conf
            regexp: "^Listen"
            state: present
            line: "Listen {{ http_port }}"

      - name: Restart and enable the service
        systemd: name=httpd state=restarted enabled=yes

Here we have written only one play and that is for the group servers. tasks is a list of ad-hoc Ansible commmands that we want to execute on the remote hosts in the group servers. Earlier we executed these ad-hoc commands from the command line with ansible command. There are three tasks in this playbook:

  • First task uses yum module to install the httpd package.
  • Second task uses lineinfile module to replace the specific line in the file /etc/httpd/conf/httpd.conf with the one we’re interested in.
  • Third task restarts and enables the httpd server using systemd module.

Let us first uninstall httpd package from the remote systems. This is not really necessary to run the playbook but, we’re doing it to validate that all tasks we did earlier are performed as expected by the playbook.

$ ansible -m yum --args="name=httpd state=absent" servers

And now run the playbook:

$ ansible-playbook playbook.yaml

PLAY [servers]

TASK [Gathering Facts]
ok: [host1]
ok: [host2]

TASK [Install httpd]
changed: [host1]
changed: [host2]

TASK [Change port to listen on]
changed: [host2]
changed: [host1]

TASK [Restart and enable the service]
changed: [host1]
changed: [host2]

host1                      : ok=4    changed=3    unreachable=0    failed=0   
host2                      : ok=4    changed=3    unreachable=0    failed=0

See how the name we set for every task in the playbook.yaml file is shown in above output. As an aside, execute the very same command again and observe the output. Everything that shows changed in above output will instead show as ok because nothing changed in remote systems as everything was setup just a few seconds back. 😉

That’s it for this post

In upcoming posts, we’ll be creating an example application and deploying that using Ansible Playbooks. Although it won’t be as complex as most real-life applications and their deployments, it’ll give a fair idea of how Ansible can be used to deploy non-trivial applications.

If you have any feedback/suggestions, leave it in the comment section at the bottom of the post. Until next time. 😉

Ansible Series: The inventory file - variables, aliases and more

Series: Ansible

In the previous post on inventory file, we saw that inventory file is central location that stores information about the remote hosts. It’s not necessary that we would always want to deal with remote hosts. Ansible can also work on the control system (localhost).

$ cat /etc/ansible/hosts


And let’s try the ping module:

$ ansible -m ping all
localhost | UNREACHABLE! => {
    "changed": false, 
    "msg": "Failed to connect to the host via ssh: Permission denied
    "unreachable": true

} | SUCCESS => {
    "changed": false, 
    "ping": "pong"

} | SUCCESS => {
    "changed": false, 
    "ping": "pong"


Oops, that failed to ping localhost - the only ping that we would expect to work even if everything else failed. Let’s see the msg part of the output - Failed to connect to the host via ssh: Permission denied(publickey,gssapi-keyex,gssapi-with-mic).. Why would I want to connect to localhost over SSH? Let’s set ansible_connection in the inventory file:

$ cat /etc/ansible/hosts
localhost ansible_connection=local


That tells Ansible to connect to localhost using connection type local. So, no more SSH’ing into localhost:

$ ansible -m ping all 
localhost | SUCCESS => {
    "changed": false, 
    "ping": "pong"

} | SUCCESS => {
    "changed": false, 
    "ping": "pong"

} | SUCCESS => {
    "changed": false, 
    "ping": "pong"


Voila! That was successful.

As you might have guessed by now, default ansible_connection type is SSH. And it can be changed by setting ansible_connection to desired (and valid) value in the inventory file. Other connection types include:

  • SSH protocol types
    • smart (default)
    • ssh
    • paramiko
  • Non-SSH connection types
    • local
    • docker

Throughout the series, we’ll mainly be working using the default SSH connection type. We’ll be deploying containers as well and will see if we need to use the docker connection type then. 😄

Host Variables

We can easily assign host variables in the inventory file. These variables can then be used in playbooks.

$ cat /etc/ansible/hosts
[servers]    http_port=8080    http_port=9000

Above snippet defines http_port to different values for the two hosts. Let’s bring up httpd webserver on these hosts on the specified port.

$ ansible -m yum --args="name=httpd state=present" servers

$ ansible -m lineinfile --args="regexp=\"^Listen\" path=/etc/httpd/conf/httpd.conf state=present line=\"Listen {{http_port }}\" " servers

$ ansible -m systemd --args="name=httpd state=restarted" servers

That’s handful of commands to execute. Let’s see what each one does.

  • First we install httpd package using the yum module. state=present installs a package if it’s not already installed on the system.
  • Next, we replace a line in file (lineinfile) that starts with Listen (regexp="^Listen"). The \ are used as escape characters so that a " is not considered as closing quote.
  • Finally we start httpd service in those systems using systemd module.

Now if we curl these systems, we’ll be greeted with default Apache web server page. All we need to do is curl or curl

Group Variables

If we’d like to set variables for entire group, servers in our example, all we need to do is:

$ cat /etc/ansible/hosts


And then executing similar commands as those we did earlier would start httpd server for us on port 8080 of both the systems.


If you’ve used Linux command line for some time, chances are you have already heard of alias. It’s similar concept in Ansible as well.

An alias helps you define name for a host. In our example, we’ve specified IP addresses of the hosts in our inventory file. If we’d like to ping only one system, we’ll have to do:

$ ansible -m ping

That doesn’t look cool. Let’s set aliases for our two hosts:

$ cat /etc/ansible/hosts
host1 ansible_host=
host2 ansible_host=

And now we can simply do:

$ ansible -m ping host1

That’s it for this post

In this post, we looked at some non-trivial ad-hoc commands of Ansible. It looks cool but, executing a bunch of commands every time on the command line is not really fun. Instead, we can use Ansible Playbooks to perform all tasks we did above with a single command. We’ll soon look at Ansible Playbooks!

As always, if you have any comments/feedback/suggestion, please let me know below! Until next time. 😉

Ansible Series: The inventory file - working with remote systems

Series: Ansible

In this post we’ll talk about the concept of Inventory. When we installed Ansible, an example inventory file was automatically placed at the location /etc/ansible/hosts for us. This is the default inventory file for Ansible.

Inventory file

An inventory file consists of the information of various remote hosts that Ansible knows of. This file needs to be configured before we can start using Ansible to work with remote systems.

Hosts specified in the inventory file can either belong to a group or be ungrouped. A group is specified like below:


Ungrouped hosts should be specified before specifying any grouped hosts. You can either provide FQDN or IP address of the host. Make sure that the remote host(s) is/are reachable from the system you’re using to run Ansible commands through the FQDN or IP provided in inventory file.

Inventory file can be of different formats. The one you saw above and that we’re going to follow throughout the series is INI-like syntax (Ansible’s default). You can read about other formats in Ansible’s documentation.

Playing with remote systems

For this post, we’re going to work with two remote systems. You can work this out in various ways by creating two Linux virtual machines on your laptop or a cloud provider. Two systems I’m going to work with have IP addresses and

My inventory file (/etc/ansible/hosts):

$ cat /etc/ansible/hosts

Let’s start throwing some Ansible magic to them:

$ ansible -m ping all | UNREACHABLE! => {
    "changed": false, 
    "msg": "Failed to connect to the host via ssh: Permission denied (publickey,gssapi-keyex,gssapi-with-mic).\r\n", 
    "unreachable": true

    "changed": false, 
    "msg": "Failed to connect to the host via ssh: Permission denied
    "unreachable": true


Oops, that was embarassing. Our first real Ansible command failed. 😢

Ah, our host system doesn’t have SSH access to the remote systems! We need to configure that first by enabling password-less SSH access from host system to both the remote systems. This can be done by creating ssh keys and copying them to remote system.

And then execute same command again:

$ ansible -m ping all | SUCCESS => {
    "changed": false, 
    "ping": "pong"

} | SUCCESS => {
    "changed": false, 
    "ping": "pong"


Now that worked like a charm. One thing you need to ensure when configuring SSH access is that, by default, Ansible will use same user to connect to remote system via SSH as that on the host you’re executing Ansible from. That means, if on the host system you’re logged in as user randomuser, Ansible will try to connect to remote system as the user randomuser only.

But what is above command doing anyway? It’s using the module ping on hosts that belong to group all. In response, since SSH connectivity is OK, it’s getting a pong response and the result is SUCCESS.

But we didn’t configure all group; we configured servers group. By default, Ansible has two groups: all and ungrouped. Hosts that are not a part of any user defined group belong to the group ungrouped and all contains all hosts in the inventory file.

Let’s do something else on this remote systems. Let’s do yum -y update (assuming they are running CentOS/RHEL) on these systems:

$ ansible -m yum --args="name='*' state=latest" all

We use yum module of Ansible and ask it to work on all installed packages (name='*') such that their updated to their latest versions (state=latest). * is a regular expresssion which indicates “all installed packages” to yum.

Expect this command to take some time and print ugly output. It’s going to return only when the updates have been downloaded and installed on the remote system. Time it will take depends on: number of updates available, internet speed and type of disk (HDD vs. SSD).

Let’s try shell module which executes the specified command on remote systems:

$ ansible -m shell --args="date" all | SUCCESS | rc=0 >>
Fri Sep 29 15:00:30 UTC 2017 | SUCCESS | rc=0 >>
Fri Sep 29 15:00:30 UTC 2017

$ ansible -m shell --args="df -h" all | SUCCESS | rc=0 >>
Filesystem      Size  Used Avail Use% Mounted on
/dev/vda1        10G  1.2G  8.9G  12% /
devtmpfs        900M     0  900M   0% /dev
tmpfs           920M     0  920M   0% /dev/shm
tmpfs           920M   17M  904M   2% /run
tmpfs           920M     0  920M   0% /sys/fs/cgroup
tmpfs           184M     0  184M   0% /run/user/0 | SUCCESS | rc=0 >>
Filesystem      Size  Used Avail Use% Mounted on
/dev/vda1        10G  1.2G  8.9G  12% /
devtmpfs        900M     0  900M   0% /dev
tmpfs           920M     0  920M   0% /dev/shm
tmpfs           920M   17M  904M   2% /run
tmpfs           920M     0  920M   0% /sys/fs/cgroup
tmpfs           184M     0  184M   0% /run/user/0
tmpfs           184M     0  184M   0% /run/user/1000

rc=0 that we see in the first line of output for both systems indicates that return code of executing the command was 0 (which means there were no errors.)

That’s it for this post

There’s more that can be talked about inventory file: host variables, group variables, aliases, and more. But I’ll keep it for the next post to avoid making this too long. I personally feel bored of reading really long posts written across the Internet. 😉

As always, if you have any comments/feedback/suggestion, please let me know below! Until next time. 😉

Ansible Series: Set things up

Series: Ansible

As I mentioned in my previous post, I will be starting with Ansible series. Best way to find all posts in Ansible series to use this link.

If you’re reading this post, chances are you already know what Ansible is and what it is used for. So, instead of repeating what numerous other posts across the Internet already have to say, I’d try to put it in short as to what we use Ansible for and how it helps us.

We use Ansible as a configurtation and deployment tool. It helps us configure systems with the packages and tools we need on it to deploy the end product. We use Ansible Playbooks (will talk about this in later posts) to install packages, start services and ensure things are up and fine. We plan to use its “Continuous Deployment” feature to automate deployments on different environments (pre-prod, prod, etc.)

Let’s get into setting up Ansible on our system and perform some tasks with it. Ansible is generally used to configure remote systems but, for this post, we’ll be using it on localhost only. That is, install it on localhost and perform operations on localhost as well.

We’re going to use CentOS 7 system for the purpose of this series. Except installation, all other steps should work just fine irrespective of the underlying distro.


$ yum -y install ansible

At the time of writing this, above command will install version for us. If you’d rather want to install the latest (bleeding edge) version, follow below steps:

$ yum -y install epel-release
$ yum -y install python-pip
$ pip install ansible

Benefit of installing via yum is that packages in official RHEL/CentOS repositories undergo a good deal of testing to ensure enterprise stability and security.

What we get upon Ansible installation?

Once you’ve installed Ansible, on your command line, type ansible and hit Tab key twice to load all possible commands that start with ansible.

Ones we will be discussing in this series are:

  • ansible: runs a specified task on target host(s).

  • ansible-console: drops you into a shell that works as REPL (Read-Eval-Print-Loop). It allows running ad-hoc tasks against a chosen inventory.

  • ansible-doc: provides documentation on the command prompt. It’s really helpful for quick references where we are not completely sure but have a rough idea.

    ansible-doc yum would print help for the yum module whereas ansible-doc -s yum would print a snippet which can then be copied to playbook and modified.

  • ansible-galaxy: helps us manage roles using Ansible Galaxy.

  • ansible-playbook: is the most interesting and heavy lifter among all. It executes a playbook passed to it as an argument. We’ll have quite a few posts on this. 😉

  • ansible-pull: pulls playbook from VCS server and run them on the machine executing ansible-pull. It helps invert default push architecture of Ansible into pull architecture.

  • ansible-vault: helps safeguard sensitive information stored in a data file used by Ansible.

Performing actions on localhost

As mentioned in the beginning of this post, we’ll perform some actions on localhost using ansible command.

  • Install httpd on our CentOS system:

    $ ansible localhost -m yum --args="name=httpd state=present"

    This tells ansible to execute on localhost, use the module yum and pass additional arguments "name=httpd state=present" to the yum module. As a result of executing this command, httpd package will be installed on the localhost.

  • Start httpd server we just installed:

    $ ansible localhost -m systemd --args="name=httpd state=started"

    This tells ansible to start the httpd server we installed in previous command. It does so using systemd module.

    Check if it was actually started:

    $ curl localhost
  • Enable httpd server to start at boot time:

    $ ansible localhost -m systemd --args="name=httpd enabled=true"

    You can verify that httpd server starts upon boot by rebooting the localhost using the same curl command mentioned earlier.

That’s it for this post

In the next post, we will take a look at the concept inventory in Ansible. If you have any comments/feedback/suggestion, please let me know below! Until next time. 😉

Writing Series of Technical Posts

When I decided to make some use of this domain, one (and probably most important for me) thing that I had in mind was to write a series of posts/articles on tech that I work on or that interests me. I had already decided on using Hugo but was not sure how to group articles in the form of a series.

That’s when I came across Nate Finch’s website and read his post on how he managed to group articles into a series. Knowing how greatly I suck at frontend, I decided to clone his repo and modify things to suit my requirements. I didn’t want my discomfort with frontend get in the way of writing!

In coming few days, I’ll be writing a tech series, starting with Ansible. At the moment, I have not decided how many posts I’ll write or what specific topics I’ll dive into or how frequently I’ll write. The only thing I’ve decided to do is to write.

Through writing, I want to share what I know (with the hope of being helpful to someone some day) and learn from the feedback/suggestions of readers.


This is not really my first attempt at blogging. I’ve tried and failed numerous times so far. 😃 This might or might not be another such attempt at writing.

I enjoy hosting and talking at meetups. During one such recent meetup, I volunteered to give a talk about technologies I had not played much with.

It was the result of challenging myself to prepare for this talk that I decided to start writing - as a challenge. And as every other living creature, I hope to succeed. 😄

I intend to write about my learnings on various technologies that I play around with. I hope it helps someone else out there some time.

I don’t think of myself as an expert at any of the technologies I intend to write about; so if you have any comments/suggestions/feedback, please feel free to ping me. 😉