Dec 28, 2018
This blog is the last part in the series of
blogs on OpenShift OKD upgrade we
performed for the CentOS Container Pipeline project. If you haven’t read the
earlier parts, go ahead and do that first.
If you have already read the previous parts in the series, it’s easy to
conclude that we screwed up. We thought we had broken out of that habit after
moving to the new
architecture.
After all, we had a stable environment for two and half months and only issues
we faced were from our buggy code. But bad habits are difficult to get rid of!
Obviously, we learned a few lessons in the process. Few too many!
Lesson 0 - Friday is for beers
This one doesn’t need any detail. Unless it’s a major bug fix or security
patch, Friday should be meant for beer, or coffee, or tea, or <put your
favorite drink here>.
Lesson 1 - OpenShift knowhow needs to be improved
We need to improve our knowledge about OpenShift works. RTFM is not only
required but we need to make more time to do it. I’m not sure how many problems
we could have avoided had we read more documentation but, we could have surely
made informed decisions when we faced them.
Lesson 2 - Homogeneous production and testing environment
Every bit of testing we did was on an infrastructure which we had absolute
control over right from spinning up the VMs to deploying our service in the
OpenShift cluster. However, the production environment is something that CentOS
infrastructure team provides and there are some differences between the two
because the errors we saw in production environment never showed up while
performing same actions on the test environment.
We need start using a small subset of systems provided by CentOS infrastructure
team for pre-production testing and validation.
Lessons 3 - Have reasonable expectations from the upstream
The OpenShift upstream is possibly loaded with a bunch of questions and issues
from users like our project. It would be too much to expect them to look into
every issue beyond “best effort” basis. This takes us back to Lesson 1.
Lesson 4 - Talk to the experts
Before making a cluster level change, it might be a good idea to write down the
plan and the steps, and run it by a few experts to get an opinion or maybe even
a better idea to do what we’re trying to do.
That’s it!
That’s all I have to share about the fiasco we did during the weekend of
December 14, 2018. If you have any suggestions/ideas/feedback for us, let us
know. I know there’s a lot we can and we need to learn about the various tools
involved in this exercise.
On a personal note, this is the most I’ve ever written in a given week. If you
read it this far, I hope I managed to make you think - “This guy is a
professional screw-it-up artist”.
Until next time… 😄
Dec 27, 2018
This blog is fourth part in the series of
blogs on OpenShift OKD upgrade we
performed for the CentOS Container Pipeline project. If you haven’t read the
earlier parts, go ahead and do that first.
I closed the previous post on a high wherein we managed to fix multiple issues
that were preventing us from doing the OKD upgrade in production. After
reaching the stage, we decided to do OKD 3.9 to OKD 3.10 upgrade and then
upgrade to OKD 3.11. In the development environment, I already attempted an OKD
3.10 to OKD 3.11 upgrade. It was a walk in the park just like the experience I
had while upgrading from OKD 3.7.2 to OKD 3.9.0.
Mistake #0 - Choosing Friday for production upgrade
This statement is intentionally written to not make any sense; just like our
decision to do production upgrade on Friday. I don’t think any amount of
penance can free us from this sin of disturbing production Gods on a Friday.
Amen.
Song of Error on loop mode
We had taken a maintenance window of about 4 hours and were confident of
finishing our job in under 2 hours. But there won’t be any fun if things went
as per the plan. As if a song of error was playing on loop mode, we were
repeatedly seeing the same error and not able to make any sense of what’s going
wrong.
TASK [Read node config]
**************************************************************************************************************************************
ok: [osn1.example.com]
fatal: [osm1.example.com]: FAILED! => {"changed": false, "msg": "file not
found: /etc/origin/node/node-config.yaml"}
ok: [osn2.example.com]
ok: [osn3.example.com]
ok: [osn4.example.com]
ok: [osn5.example.com]
ok: [osn6.example.com]
ok: [osn7.example.com]
ok: [osn8.example.com]
ok: [osn9.example.com]
ok: [osn10.example.com]
We were seeing this error for the master. And every time we checked the master
after OpenShift Ansible threw above error, this file
/etc/origin/node/node-config.yaml was undoubtedly present! The logs were not
particularly helpful in pointing out the real cause of the problem. Above all,
this was the first time we were seeing this error. Our development environment
didn’t throw this error for OKD 3.11.
Now, if you have read previous posts in this series, it’s not the first time
I’m saying that logs didn’t make a lot of sense. This, probably, boils down to
the point I mentioned in an earlier post about the first and one of the most
important lessons this exercise taught us - “We lack OpenShift expertise within
our team.”
Uninstall OKD 3.9, install OKD 3.11
After struggling with the plan of upgrading to OKD 3.10 in a recommended
manner, we decided to go with the unconventional path in production because
with every passing minute during which our production was a mess, we were not
doing what our service was supposed to do - build container images for the
open-source projects.
Jenkins PV did bite us
After bringing up OKD 3.11 in production, we tried to attach the same PV to
Jenkins that was being used in OKD 3.9 cluster. To our surprise, the PV got
attached immediately and Jenkins started up. This was something we struggled to
accomplish in the development environment where PV wouldn’t get attached unless
we modified the permissions or deleted the data on PV. And once the 3.9 to 3.10
upgrade worked fine, we stopped thinking about Jenkins PV altogether.
In production, however, first thing Jenkins did after connecting with the PV
was to delete the data from previous builds jobs. Not sure how that was any
better than us formatting the PV ourselves!
etcd wanted its share of biting us as well
The production environment was running on top of OKD 3.11 while development
environment was on OKD 3.10. We didn’t have much idea of how things were with
OKD 3.11. And that fact was very well exploited by etcd.
etcd restarts when a particular version of
docker is used which
caused API service to be unavailable for some time. The seed-job in CentOS
Container Pipeline service is a build job that:
- runs on Jenkins,
- parses the container index,
- creates/updates jobs on Jenkins
- triggers build for newly created jobs
- silently updates existing jobs (if there is any change)
It uses oc command line tool to do its job. Now while the etcd restarts
caused API server to be unavailable, our seed-job kept doing its job unaware
of the situation and the oc commands during this time were pretty much
talking to /dev/null (metaphorically). The seed-job needs to be altered to
consider this scenario. We faced this issue with etcd in spite of using a
version of docker
newer
than the one mentioned in errata.
Unwarranted emails sent to users
We didn’t realize that we were hitting the etcd issue until it was a bit too
late. We formatted Jenkins PV once after realizing that Jenkins had anyway
deleted all the data that might have been of any use. After that, we triggered
the seed-job to parse my fork of container index so that it didn’t send
emails to the users. Things seemed to be under control so, we switched the
seed-job to refer to main container index. But the jobs that were not created
during the earlier run of seed-job due to etcd restarts got created in the
subsequent run and few users were notified with an email that said “First build
of the container image” as the cause of build.
That’s it!
Wow, this one turned out to be longer than expected. Overall, we had a crazy
weekend (yes we were now working on a Saturday). We failed to send weekly scan
emails as well. The taste of stability, that we got after moving to OpenShift
based infrastructure in late September, was swiftly taken away by this
incident. It taught us a number of lessons in the process.
Next and the last post in this series is going to be a quick gist of the
lessons we learned during this exercise.
Until next time… 😄
Dec 27, 2018
This blog is a part of the series of
blogs on OpenShift OKD upgrade we
performed for the CentOS Container Pipeline project. If you haven’t read the
first
part
and the second
part,
this post might not make a lot of sense. Go ahead, read them first.
I ended the previous post with describing the unconventional path we
unwillingly decided to go with and how that led us to figure the cause of
the issue we were facing while following recommended path for OKD 3.9 to 3.10
upgrade.
Uninstall, install, repeat
As mentioned in the previous post, the idea was to:
- uninstall OKD 3.9,
- install OKD 3.11,
- ensure that PV used for Jenkins in 3.9 setup was usable with 3.11 as well
without data loss
Just like OKD 3.10, OKD 3.11 also uses openshift_node_groups dictionary in
place of openshift_node_labels and takes the configuration for kubelet
arguments with it.
While installing OKD 3.11, we encountered an issue wherein Ansible playbook
would exit with an error related to OpenShift master. The error was:
{"changed": false, "msg": "Node start failed."}
This kept on repeating every time we tried to install OKD 3.11 on the nodes
where OKD 3.9 was earlier installed.
Fixed two issues in one shot
We opened an issue on
GitHub for the
installation error. Unlike the failure with OKD 3.10
upgrade, this
time the logs were more helpful in making sense of the underlying issue.
It seemed like we made a mistake in configuring the value for
openshift_node_groups variable. My takeaway from the way we fixed the
issue
is that the value in the dictionary has to be a list even if it’s just one
value and not multiple values.
After this we faced another error that had to do with the value we had set for
the key kubeletArguments.image-gc-high-threshold in openshift_node_groups.
This is something I found from the logs. After trying few values that I thought
might help origin-node service to start well, I gave up and removed every
configuration related to garbage
collector from
the openshift_node_groups variable.
This not only helped us perform a successful installation of OKD 3.11, but
also turned on a bulb in my head that illuminated something like, “Remove the
garbage collector configuration and try OKD 3.9 to OKD 3.10 upgrade the
recommended way!” And so I did.
Successful OKD 3.9 to OKD 3.10 upgrade
It turned out that the bulb illuminated the right thing. Apart from facing a
few minor issues (minor because we didn’t need to open a GitHub issue), we
managed to upgrade our OKD 3.9 cluster OKD 3.10 using the OpenShift Ansible
playbooks! It was a “F**k yeah” moment for me.
Over next few days, we managed to successfully perform OKD 3.9 to OKD 3.10
upgrade in the development environment before we decided to replicate the steps
in production.
That’s it!
Since the recommended upgrade path worked well for us, we didn’t bother to test
if doing uninstall and install of different versions of OKD and then using the
same PV for a persistent Jenkins deployment worked well or not. We felt
ecstatic with the way things worked out. Little did we know how the Jenkins
thing was going to be one among several dogs that were planning on biting us!
By the time we reached this point, it was more than 3 weeks since we started
off. I visited the Serengeti National
Park during this
timeframe and if we count the vacation, it was more than 4 weeks since we
started.
Until next time… 😄
Dec 26, 2018
This blog is a part of the series of
blogs on OpenShift OKD upgrade we
performed for the CentOS Container Pipeline project. If you haven’t read the
first
part,
this post might not make a lot of sense. Go ahead, read that first.
In the second installment of the series, I’m going to describe the steps we
divided our plan into. Or rather, the steps we had to break things into because
upgrading from OKD 3.9 to OKD 3.11 was not possible without upgrading to OKD
3.10 first.
Note: The steps mentioned in this post were executed on a
development/pre-production environment that lives on Red Hat infrastructure and
not on CentOS infrastructure where the production environment lives.
OKD 3.9 to OKD 3.11 via OKD 3.10
Upgrade
prerequisites
section in OKD 3.11 in-place cluster upgrade documentation says that an
existing OKD cluster needs to be upgraded to “latest asynchronous release of
version 3.10” before attempting an upgrade to OKD 3.11. You can go ahead and
search for those exact words in the quotes on the docs page to be sure that I’m
not lying!
OKD 3.9 to OKD 3.10 upgrade
Having successfully upgraded from OKD 3.7.2 to OKD 3.9, I was confident that
upgrading OKD 3.9 to OKD 3.10 would be as simple as a walk in the park. But
just as life has its ups and downs at most unexpected times, OKD had plans of
making us go down a rabbit hole when we least expected it to. With OKD 3.10:
- the
openshift_node_labels got deprecated,
- having a proper DNS was mandatory,
- the
openshift_node_kubelet_args got deprecated
DNS setup in development environment
Even before trying to perform an upgrade, we had to configure proper DNS in our
development environment.
Our development environment, running in Red Hat infrastructure, had OpenShift
OKD cluster running as VMs on top of KVM and libvirt.
Till OKD 3.9, using IP addresses as hostnames in Ansible inventory file was
just fine. But with OKD 3.10, it was mandatory to use hostnames that would
resolve through a DNS server. Even /etc/hosts entries wouldn’t be considered
valid unless a proper DNS was set up.
Our production environment didn’t have to be altered because DNS worked fine
there but in our development environment, we had to set up a DNS server. This
blog
post
by a fellow Red Hatter came to the rescue!
First attempt at upgrade
After having grasped the changes required to make to the Ansible inventory file
work well with OpenShift Ansible playbooks for OKD 3.10, we attempted the
upgrade. The initial attempt was futile and we faced error due to a typo in
the variable
name.
Serious errors during the upgrade
After the typo issue, next error we faced was more serious. Serious because it
didn’t seem like upstream had any clue about a fix either. We opened an issue
on GitHub and
added a bunch of information that we found while troubleshooting the error by
ourselves.
Logs complained about No existing node-config.yaml, exiting in spite of
having the /etc/origin/node/node-config.yaml in place. The issue, as we later
understood, was a misconfiguration of kubelet arguments in the new way of doing
things with OKD 3.10.
But this one took a really long time because the error message didn’t seem to
have the correct indication of the underlying issue and, as you can see from
the GitHub issue, we tried a number of things in absence of help from upstream.
This issue taught us our first and one of the most important lessons - “We lack
OpenShift expertise within our team.”
It’s one thing to use OpenShift and another to know it inside-out. However,
it’s not an easy thing to fix due to the huge size and fast pace of development
of Kubernetes, and hence that of OpenShift.
The unconventional path
Due to the above error, we faced during OKD 3.9 to OKD 3.10 upgrade, we started
to lean towards doing something that’s not mentioned in the documentation and
quite possibly the least favorable way to upgrade. We planned to:
- uninstall OKD 3.9 from the cluster
- install OKD 3.11 on the same cluster
- preserve the Jenkins build data on the PV so that when we attach the same PV
to Jenkins on OKD 3.11, we can possibly resume from where we left
That’s it
We managed to do some crazy stuff while going down the “unconventional” path
mentioned above. But the craziest thing was to find a solution to doing an OKD
3.9 to OKD 3.10 upgrade in a clean way. That is, we managed to find a solution
and the cause of the issue we faced while following the recommended path for
OKD 3.9 to 3.10 upgrade.
Until next time… 😄
Dec 20, 2018
In CentOS Container Pipeline, we use Jenkinsfile
strategy
of OpenShift to build container images, scan them and push them to the
registry. To build a container image, we need to make sure that the pod started
by Jenkins is privileged (akin to docker run --privileged) and that we’re
sharing the Docker daemon socket (/var/run/docker.sock) from the host with
the container. Anyone with security in mind would dislike the idea (and I hope
Dan Walsh doesn’t read how insecurely we’ve
plumbed certain things here.)
After having stabilized the service on its new architecture, we wanted to
explore if we can use buildah to build the containers and eliminate the need
of sharing Docker socket with all the pods that spin up on an OpenShift node
(we also managed to blow up the service after stabilizing
it
but, that was not intentional.)
Challenge #1 - Running buildah inside a container
Since we use OpenShift to build images and Jenkinsfile strategy to do things
other than just building them, we had to make sure that image building can be
done inside the pod that’s dynamically brought up by Jenkins server on the
OpenShift cluster. To be able to do this, we had to use a newer version of
buildah than one provided by CentOS
repos.
Challenge #2 - Go package is only available via SCL in RHEL/CentOS
CentOS repos come with an outdated version of buildah so we had to build the
buildah binary. To build it inside the container was non-trivial (at least
for me) and it took me some time to finally come up with a script to do
it.
Challenge #3 - Buildah still failed with ERRO[0000] 'overlay' is not supported over overlayfs
In spite of managing to overcome first two challenges, we were hitting an error
with overlayfs:
$ buildah images
ERRO[0000] 'overlay' is not supported over overlayfs
ERRO[0000] 'overlay' is not supported over overlayfs
'overlay' is not supported over overlayfs: backing file system is unsupported for this graph driver
'overlay' is not supported over overlayfs: backing file system is unsupported for this graph driver
The error was fixed by using a tip from this GitHub
comment.
It works!
I’m sure that the solutions/workarounds that I have mentioned above are not
perfect. But that’s what we managed to get our issues fixed with and, more
importantly, be able to build container images using buildah from inside a
pod/container on OpenShift. Eventually, our Jenkinsfile was modified to use
buildah
bud
instead of docker build.
Open challenges
Although we’re able to use buildah bud to build images, we still need to find
a way to be able to scan the resulting container images. As it is right now,
docker run doesn’t work to spin up images created using buildah unless the
image has been first pushed to an external registry and pulled from there. But
I’m guessing this is because the Docker socket is being shared from the host
system and buildah is working within the confines of the container (which is
a good thing, I guess).
So the next thing we’re trying to figure out is to use podman run instead of
docker run to get done with scanning the container image. However, we need to
do this from inside the container and we’re already seeing the error reported
in this GitHub issue.
That’s it
We are still working on this and are not using buildah bud in production. But
we plan to change that soon. If you have any ideas/suggestions that can help us
do things in a better way, let us know! We hangout on #centos-devel IRC
channel.
Until next time… 😄
Dec 19, 2018
This blog post and others in the series are going to be about the OpenShift
cluster upgrade that our team performed for CentOS Container Pipeline
during December 2018. To find all the blogs on the topic, check the
openshift_upgrade_2018
tag.
OpenShift Origin was renamed to OKD (Origin Community Distribution of
Kubernetes) so I’ll be using the term OKD in this and future posts to refer to
OpenShift Origin.
What we had
We had a stable and functional OKD cluster running on OKD 3.9 that was being
used as the base for CentOS Container Pipeline service. Having things stable at
infrastructure level was still a new thing for us. Before we revamped the
CentOS Container
Pipeline
to run on top of OKD, things were quite shaky. We used to keep firefighting
most of the time because the heterogeneous cluster was far from stable!
We had setup Prometheus and Grafana with the help of openshift-ansible
playbooks. We also had a custom Grafana dashboard to monitor the number of
builds (both successful and failed), the number of builds in different OKD
stages like New, Pending, Running, Complete and Failed, and a graphical
representation of the amount of free memory on the nodes in OKD cluster.
What we wanted
We wanted more monitoring and alerting. In the sense, we wanted to be notified
when some build is taking a long time, or when there are more than x failed
builds in a short duration, or if there were more than x builds hanging around
in Pending state, so on and so forth. The intention was to be able to stay on
top of things unlike how screwed up they were in earlier architecture where
many times it was the users who notified us about something blowing up in our
environment while we were busy writing new bugs (read ‘features’). 😉
The Plan
As mentioned earlier, we were already using Prometheus for monitoring and had
to configure Alert Manager to add rules based on which it would send us alerts.
But instead of doing that, we stumbled upon the fact that OKD 3.11 was going to
ship with built-in Prometheus Cluster
Monitoring
and we thought to ourselves, let’s use this instead of writing our own rules.
To be honest, the assumption was that, in OKD 3.11, Prometheus would have some
default set of rules which would make our job easier. Now making assumptions is
probably never a good idea in software engineering. But we still managed to do
that.
We decided that instead of writing rules for the Prometheus setup that we
have in OKD 3.9, we would upgrade our cluster to OKD 3.11 and see what’s
available by default and add rules on top of it. Of course, we didn’t know at
the time that a better idea would have been to RTFM the upcoming shiny thing
called as Prometheus Cluster Monitoring instead of diving into the task of
upgrading the cluster.
Conclusion
In this post, I mentioned what we wanted to achieve and what we thought would
help us do that and what we assumed in process. In future posts, I’ll mention
what we did, how we messed up and how we made sure that the taste of stability
was taken away. To complement that, we screwed the production on a Friday when
the production Gods are not in a mood to be disturbed.
Until next time… 😄
Nov 26, 2018
It’s been two months since we deployed the CentOS Container Pipeline service
based on OpenShift in production. Since deployment, the service is running on
top of OKD (new name for OpenShift Origin) 3.9. We
haven’t faced any major issues that would keep us fire-fighting. In essence,
we’ve been able to focus on implementing things that were available in previous
architecture but are not yet available in current one.
At the moment, we’re working on:
- Writing the APIs for fetching logs from Jenkins server running atop OKD. This
is to provide users with the logs from lint, build, scan phases of the
pipeline. We’re writing:
- A backend which will fetch the information from Jenkins
- And a frontend that will be accessible to users via
registry.centos.org.
- Add container images for the projects on CentOS-Dockerfiles
repo which were not already available
on the container index.
- Generate statistics for number of times a container image has been pulled
from registry.centos.org.
- Add more functional tests so as to have a more reliable CI.
- Moving from OKD 3.9 to 3.11 so that we can use built-in Prometheus Cluster
Monitoring instead of writing our own rules.
One of the major pain points has been upgrading OKD from 3.9 to 3.10. We opened
a GitHub issue
for it but haven’t been able to reach a resolution yet.
That’s it for now. We will share more as we make progress.
Sep 11, 2018
If you have not read the first
part
of this series on CentOS Container Pipeline service then please go ahead and do
it!
In this post, I’ll be talking about the issues we faced and how we resolved
them when we tried to build a large sized
container-index with the newly
developed service.
What we did?
One fine morning I had a thought of trying to build container images for all
the projects that build their images with our service already. But I wanted to
do it with the new deployment of the service which is based on OpenShift.
First things first, I modified the
container-index
to make sure that notification emails don’t get sent out to the actual users
but land in my inbox instead. Otherwise it would have been one hell of a
testing for me! 😉
I had a 3-node cluster (1 master and 2 nodes) which I was going to test things
against. So far, we had tested things only on a small sized index of maybe 5-10
images. Obviously that was a mistake and we should have done this testing of
full container-index sooner rather than when we’re so close to deploying things
into production.
What we observed?
When I triggered the full index to be built on my deployment of the service, I
saw that the seed-job that parses all the yaml entries in the container-index
picked up the jobs and started parsing them. At the same time, the jobs that it
had already parsed started getting served by OpenShift and Jenkins, i.e.,
builds were triggered.
Some of the issues we observed:
- Numerous jobs were failing due to unavailability of Docker daemon socket to
build the image.
- Many images were taking hours to build completely instead of taking a few
minutes.
- Seed job failed to parse a particular entry and in event failed to parse the
remaining index.
- Jenkins used to get restarted and thus number of pods were left orphaned
while Jenkins console reported that it cannot connect to the pods.
- Scanners failed to scan certain images.
- Weekly scan doesn’t run
How we fixed the issue?
Docker daemon issue
It turned out that OpenShift template used to spin up seed-job and build jobs
(jobs that would build container image for parsed entries) had same name. The
pod template for
seed-job
didn’t have Docker socket configuration because it doesn’t need the Docker
daemon for anything. So the build jobs triggered while seed-job was still under
execution used the same pod template which didn’t have Docker socket available.
Figuring out the issue took an embarassingly long time and was finally fixed by
this PR.
Builds taking hours to complete
This was happening as a result of 200+ build requests coming in in quick
successcion. During development, we tested with a small index and this never
became an issue.
To fix this issue, we introduced batch processing of the jobs. With this in
place, we default to processing only five jobs at a time, poll the OpenShift
API every 30 seconds to see how many jobs are still new/pending/building and
process the next batch if there are 2 or less projects being processed.
The benefits of batch processing from what I observed are:
- Quicker build times
- Abiliity to larger container-index even on a single node OpenShift cluster
(like minishift). This is not tested yet but I tried
to build entire index on just two node cluster things worked just fine!
Pull request for batch-processing is available
here.
Seed job failing to parse particular entry
This issue has to do with OpenShift’s rules for naming the builds. You can’t
have an upper-case letter in the build’s name. So we had to make sure that all
build jobs being created were converted to lower case before creating them on
OpenShift.
PR for this is available
here.
Jenkins restart issue
This was relatively simple to fix. We provided more memory to the Jenkins
deployment and things started working fine. However, with batch processing in
place, I think we can even do with less amount of memory because there’s not
going to be a lot of builds being processed at the same time.
One thing I learned (thanks Pradeeptp) was
that you can’t increase the number of pods of Jenkins to address this issue
because Jenkins does not scale well horizontally. Increasing its compute
resources (CPU/Memory) is the best way to mitigate issues related to it.
Scanners failing for certain images
This issue showed up because of more than one reasons:
- We parsed the output of commands like yum check-update from stdout. When
the output for single package spanned over multiple lines, things broke.
- CentOS 6 and CentOS 7 have different ways to check for updates. One uses yum
check-update and other uses yum check-updates.
- We use /bin/python as an entrypoint for the scanners when they run scripts
inside the image under test but using /usr/bin/python ensures that things
are compatible across both CentOS 6 & CentOS 7
These issues are being fixed by this
PR.
Weekly scan failures
We found that weekly scans were not getting triggered by Jenkins at all. Now
the weekly scan jobs are configured as Jenkins Pipeline which are to be run as
cron on weekend. But Jenkins doesn’t run them unless they are already triggered
once. There’s a JIRA ticket for this issue that I can’t find right now. It’s
closed with “Won’t fix”.
So we came up with another approach which is using the OpenShift cron to run a
script that triggers build for all such weekly scan jobs. PR for this is
available
here.
What’s next?
At the moment, we are busy fixing this issues we discovered by trying to build
entire container index on the cluster. We have fixed most of them and have PRs
open for remaining ones.
Next, we’re going to get the email notification piece working and deploy things
to a more production like environment before finally moving things to
production.
That’s it!
If you know of better solution/approach to fix aforementioned problems we
faced, please let us know. You can catch us on #centos-devel IRC channel on the
Freenode server. I hang out on the channel as “dharmit”.
Until next time… 😉
Jul 31, 2018
In this post I’m going to talk about why we are restructuing the CentOS
Container Pipeline service and how
OpenShift is key to it. There’s bunch of material available on the Internet so
I don’t really need to write about OpenShift.
I gave a talk at
DevConf.cz about our
service and got some great feedback. Most common feedback was that such a
service would be immensely useful to the opensource community. But deep down, I
knew that it’s not possible to scale the current implementation of service to
serve a large community. It needed a rejig to be useful to the users and not a
pain in bad places for its administrators! 😉
What does the service do?
Before I talk about the issues and how we’re handling them in new
implementation, I’ll quickly jot down the features of the service.
- Pre-build the artifacts/binaries to be added to the container image
- Lint the Dockerfile for adherence to best practices
- Build the container image
- Scan the image for:
- list RPM updates
- list updates for packages installed via other package managers:
- check integrity of RPM content (using
rpm -Va)
- point out capabilities of container created off the resulting image by
examining
RUN label in Dockerfile
- Weekly scanning of the container images using above scanners
- Automatic rebuild of container image when the git repo is modified
- Parent-child relationship between images to automatically trigger rebuild
of child image when parent image gets updated
- Repo tracking to automatically rebuild the container image in event of an
RPM getting updated in any of its configured repos
Issues with the old implemention
Our old implementation of service has a lot of plumbing. There are workers
written for most of the features mentioned above.
Pre-build happens on CentOS CI (ci.c.o)
infrastructure. In this stage, we build the artifacts/binaries and push it to
a temporary git repo. The job configuration then uses this git repo to build
container images while another job on ci.c.o keeps looking for update in the
upstream git repo.
Lint worker runs as a systemd service on one node.
Build worker runs as a container on another node and triggers a build within
an OpenShift cluster.
Scan worker runs as a systemd service and uses atomic
scan to scan the containers. This
in turn spins up a few containers which we need to delete along with their
volumes to make sure that host system disk doesn’t get filled up.
Weekly scanning is a Jenkins job that checks against container
index,
registry.centos.org and underlying database of
the service before triggering a weekly scan
Repo tracking works as a Django project and heavily relies on database which
we have almost always failed to successfully migrate whenever the schema was
changed.
All of the above is spread across four systems which are quite beefy! Yet, we
couldn’t manage to do parallel container builds to serve more requests. A
couple of teams evaluated our project to bring up their own pipeline because
they didn’t want to use public registry. However, they found the service
implementation too complex to understand, deploy, and maintain!
How are we handling (or planning) things in new implementation?
In the new implementation of the
service which is still to be moved under the official
repo, we are using
OpenShift exclusively for everything, for every feature that the service
provides.
Although we’re far from done, we have successfully implemented and tested that
these features work fine in an OpenShift cluster:
- Pre-build
- Lint
- Build
- Scan
- Weekly scan
We’re relying heavily on the OpenShift and Jenkins integration. Every project
in the container index has an OpenShift Pipeline of its own in the single
OpenShift project that we use. All of the implemented features work as various
stages in the OpenShift Pipeline.
For logging, we’re using the EFK (Elasticsearch - Fluentd - Kibana) integration
in OpenShift. To be honest, we’re still learning how to use Kibana!
This new implementation hasn’t been deployed in a production environment yet.
However, it’s relatively straightforward to deploy than the old implementation
and can even be deployed on a minishift
environment.
In progress items
We are still working on things like:
- proper CI (unit and functional tests for the code and service)
- setting up monitoring and alerting stack using OpenShift’s integration with
Prometheus
- providing useful information to the users in the emails that are sent after
every build and weekly scan
- rewrite the entire repo tracking piece to make it as independent of database
as we can
That’s it
This blog went on to be longer than I wanted it to be! But it’s a gist of what
we’ve been doing as CentOS Container Pipeline team since past few months. In
coming posts, I’ll be talking about individual implementation details!
Until next time… 😄
Mar 21, 2018
Series: Speaking
I’m an engineering graduate from India. IITs are the best engineering
institutes in India. NITs follow the suite. Although some might want to debate,
that’s not the point of this post. I pursued my engineering education from a
non-IIT and non-NIT institute. I think most engineering graduates in India have
dreamed of being in IIT/NIT and I’m no exception. Studying at IIT was my dream
as well but, it’s not an easy feat to get into one.
So when I got an opportunity to speak at IIT Gandhinagar, I was obviously
excited. Topic was containers. And the expected audience was extremely
dispersed. I was expected to address under-grad, post-grad and doctarate
students from Computer Science and Electrical Engineering.
Considering the vast audience, I decided to start from the beginning about how
containers evolved, how they compare with VMs, how Docker popularized them, how
Kubernetes took over the orchestration world and how containers have made a
lasting impact on the development and deployment story. I also discussed about
monoliths and microservices.
The Talk
First, the
slides:
I started with a show of hands to know how many in the audience had heard of or
played with containers. Surprisiginly there were just about 2-3 people. Total
audience was around 25-30, I guess. So my decision to start from how it all
began proved to be right.
As I went ahead with the slides, I got some questions around the slide that
talked about Docker commands.
Although there weren’t many questions around the rest of the slides, it did
seem like the topics mentioned in the slides were likely unheard of earlier by
the audience.
At the end of the talk, one of the PhD students told me that she found my talk
interesting+informative but she’s not much into the architecture stuff. At that
time, I was a bit confused about why she said so and didn’t get a chance to
talk further. However, I don’t think my talk was around computer architecture.
It did talk about the microservices pattern but due to lack of time, I didn’t
dive deep into it.
I went to IIT Gandhinagar upon invitation from one of the faculty members. And
two more faculty members attended my talk. I got an opportunity to interact
with three faculties and the discussions were pretty interesting. Having talked
at few other colleges, I felt that, in general, challenges they face are pretty
much the same.
General opinion of the faculties was that most students weren’t really aware of
much of the things happening in the world of containers. One of the surprising
discussion was around how students felt that it was more rewarding to work on
making web/mobile apps than working on Operating Systems, Networking, and
other things that keep you closer to the hardware. Having worked with people
doing both sort of things, I certainly feel like working on the latter is
rewarding as well. Sure the hype is around making next breakthrough (read
multi-million dollar) app but, the core things that make them possible are
seeing development as well!
We even planned on having a workshop to have hands-on about containers and
developing an app based on microservices pattern. Idea of using OpenShift to do
CI/CD, build images and eventually deploy the code seemed interesting to the
faculty members.
Above all, and probably the best part, the faculties were more than willing to
host community meetups at IIT Gandhinagar campus. I’m really looking forward to
hosting a meetup at IIT Gandhinagar in near future! I’m sure many engineers
would be excited to visit an IIT. 😉
Mistakes on my part
- I didn’t manage to get a single picture of the talk.
- I forgot to bring any laptop stickers for the students.
That’s it
Being a non-IIT grad, it was surely an exciting experience to be speaking to
IIT students about the technologies that are changing the landscape. IIT
Gandhinagar’s location is pretty amazing and far, far away from any sort of
pollution one can think of - hence my kind of place. 😉
Until next time… 😄
Feb 28, 2018
Series: Speaking
This year’s DevConf.cz was my first time at
DevConf and also my first time speaking at a conference. It was attended by
about 1500+ people! That’s massive, isn’t it? I was super excited from the
moment my talk was accepted.
Note: This is not an event report. It’s more of an informal description
about my experience attending the conf and visiting Europe!
Giving the talk
I was speaking about CentOS Containr Pipeline -
the project I have been working on since some time at Red Hat. So far we had
been quietly working on developing the service but at DevConf.cz, we wanted to
talk about it and tell more people that we’re ready to on-board them to our
service. In a nutshell, motto of our project is to help open-source projects
build and update their container images so that they can focus on developing
awesome stuff instead. 😉
Since it was my first time speaking at a conference, I was obviously nervous.
To top that, I saw some really well versed fellows in the list of people
interested in
attending
my talk. I was excited and nervous at the same time! To my surprise, my talk
managed to get pretty high in the list of popular talks. At its peak, it was on
13th spot before sliding to 21st spot on the popular talks
page. There were about 250 talks in
all.
Except going a bit fast, I think it went good. One of the tips I received for
the talk was to “learn to breathe while speaking.” 😆
Here’s my talk that explains what we provide to any open-source project that
joins us:
I got some interesting questions at the end of the talk and, in general, people
loved our work. Few people even wanted to do an on-premise installation of our
service. It’s out there on GitHub so practically anyone can do that but, we’ve
not made effort in deploying it beyond our infra so, YMMV! 😆
People interactions
Besides the talk, I had a great time talking with people. I’m more of an
introvert but I love talking with people. It’s just the starting a conversation
is extremely difficult for me! So giving a talk sort of helped me have a topic
to talk with at least those people who attended it (my talk had about 100
attendees.) Besides that various colleagues whom I knew only by name were also
present at the conf.
At the end of the conference, I think I socialized about 3 months worth of my
average socilzation in just 3 days! Since I love talking with people and giving
talks (doing meetup talks on regular basis) this was most positive takeaway
from the trip!
Travelling Europe
If you have seen the iconic Bollywood movie “Dilwale Dulhania Le
Jayenge”, you would know that most of us in
India are really fascinated with the idea of traveling to Europe. 😉
I traveled to a few cities on the sidelines of conference. My friends who had
been there earlier had warned me about how cold it might get in the part of
Europe I was visiting. Fortunately, it didn’t get so cold and I ended up loving
the weather instead! The part of world where I live, we don’t feel winters
colder than around 10 C. Even then, noons generally end up getting warmer.
But in Europe, it was cold by my standards for the entire day. And I absolutely
loved it. To be honest, I was not excited about cold as it might seem from the
post. I was planning to cut my trip short and come back to India just because
the weather conditions seemed so scary to me (my team’s still making fun of me
for this). Glad I didn’t make that change.
Properly laid out public transport system in all the cities I visited (Brno,
Prague, Budapest and Vienna) was a surprise to me. The accuracy of Google Maps
in India is somewhat flaky at times but in Europe, it was the most used and
relied upon app for me. I was shocked when Google Maps even pointed that a
tram/metro was running a minute or two late than it’s scheduled time.
European architecture has always fascinated me. It’s elegant and beautiful in
its own way. Places like Prague Castle, Parliament building in Budapest, and
your average buildings across the city left me dumbstruck on multiple
occassions.
History is another reason why I was always so fascinated about visiting Europe.
From what I could gather during this short and haphazard trip, World War II
has left a lasting impact on Europe. I did study about it during my schooling
but it was from Indian point of view. In Europe, you get to know that even
after seven decades, World War II is stil among the most crucial events in
history.
European countryside is something you normally keep wallpapers of (because
wallpapers of Maria Sharapova, Yami Gautam and the likes are distracting.) I
didn’t get to explore much of European countryside during the visit. And since
it was winter, I didn’t feel disappointed about it. Had it been European
summer, I’d have been cursing instead. This is something I want to be able to
do if I get a chance to visit Europe again.
That’s it
Okay, I need to stop somewhere. My trip to the conference and Europe in general
was filled with lots of experiences and if I try to write it all down, it’ll
get too long.
Knowing more about European history by visiting various memorials, palaces,
churches, etc. is one thing I wish I had done better. Tasting different beers
is something I did well on my first visit already. Realizing I could enjoy
cold weather (not Antarctic cold!) was a pleasant surprise! Being a pure
vegetarian guy who doesn’t even eat eggs was a major pain in the ass. Meeting
so many awesome fellow Red Hatters was amazing.
Hopefully, I’ll be writing about DevConf 2019 and maybe FOSDEM as well in about
a year’s time. 😉
Until next time… 😆
Dec 14, 2017
Recently I started working more than usual on OpenShift. I’ve
contributed to
the minishift project and also spoke
about OpenShift at a local meetup. But now we’re planning to move to
microservices architecture based on OpenShift.
I use CentOS’s
DevCloud infrastructure to setup test
instances. And it is on the same infra that I brought up my first real
OpenShift cluster deployed using
openshift-ansible. I used
this hosts file along with the playbook
available under byo
directory
for OpenShift 3.6.1. It’s a 2-node cluster where one system behaves as master
and other as a node.
Installation
For a smooth installation and setup, I had to ensure a few things like:
Install NetworkManager and firewalld on both the nodes.
Start NetworkManager and firewalld manually on both the nodes. To get
this working, I had to set SELinux to permissive. I didn’t dig much into this
but I think with proper context, I could have got it working in Enforcing
mode as well.
Ensure that /var partition has about 40 GB of free space. I think the
requirement is 15 GB on master and 40 GB on node. But I ensured 60 GB of
space for /var on both nodes.
Modify /etc/hosts file on both nodes so that they are able to access each
other by their hostnames.
Install the RPM python-rhsm-ceritificates so that Ansible can pull an image
from registry.access.redhat.com.
After finishing these steps, perform the installation:
$ ansible-playbook openshift-ansible/playbooks/byo/config.yml
I did it off the release-3.6 branch.
Post-install setup
Add a new user
Having used simple OpenShift cluster in past, it was a bit of struggle this
time to get into the OpenShift console. minishift does this very nicely so that
end user doesn’t have to bother.
It also took me some time to add a user to the cluster. I probably couldn’t
find my way around the huge OpenShift documentation.
We just need to execute below as system:admin user:
$ oc create user dharmit --full-name="Dharmit Shah"
$ htpasswd /etc/origin/master/htpasswd dharmit
# and if you want the user to have cluster-admin privileges
$ oadm policy add-cluster-role-to-user cluster-admin dharmit
This user can now login to the OpenShift web console using the credentials
you’ve just set. It took me really long to get up to this step!
Deploying an image to OpenShift cluster
Now I wanted to run a simple beanstalkd image on OpenShift cluster. All I did
was use registry.centos.org to build an image
and pull it. After successfully logging in and navigating to the project page,
OpenShift shows you an option at the top called “Add to Project”. I chose
“Deploy Image” option from this and gave the image name to be deployed.
OpenShift automatically created a DeploymentConfig and created a service based
on the metadata of the image. It also provides the name of the service. This
name can be used by other objects in the same project to access beanstalkd
seamlessly!
Still learning
I’m still learning OpenShift through the docs. The Persistent Volumes concept
has been interesting! I’m working on creating an architecture wherein various
workers can run in tandem as containers and read/write things to a
remote NFS server configured as a PV.
There’s lot to learn and do before we can achieve a microservices based
architecture. I’ll try to keep this space udpated. 😉
Dec 3, 2017
Last week, most of my time was occupied in debugging an issue whose cause was
not very obvious when we hit it. A teammate of mine opened a pull request that
failed CI check every single time. He updated the PR about four times but, CI
never felt satisfied. Finally when there was no more code to add to the PR and
CI seemed stubborn, there was a need to dig deeper into it.
A few points (disclaimers, maybe):
Our project’s CI check is not very comprehensive. It’s mostly a bunch of
functional tests that were six months back with then master branch as base
for tests.
There are no unit tests.
CI check prints a lot of logs and does that only when the check fails. When
everything’s OK, it doesn’t print more than basic information. When something
fails assertion checks, it prints logs like crazy:
- logs from Ansible provisioning
- logs from all workers in our service
- nothing in a structured manner
Tests were written by a single developer with minimal input from others in
the team. Although that developer did a great job, we didn’t add more than a
few tests or refactor the existing ones.
I had played almost no role in coming up with the tests. So when I had to go
figure why CI checks were failing, I had to learn from scratch about how the
current setup worked.
Our code-base takes good deal of time to get deployed in test environment. We
use Ansible playbook for deploying things in dev, pre-prod and prod
environments. Doing deployment on freshly provisioned nodes would take up to 30
minutes while deployment on existing VMs would take about 15 minutes.
After doing the deployment, I had to go to IPython shell and create a few
dictionaries that were then served to the test suite. These dictionaries mainly
contained IP addresses of the VMs on which tests were to be executed. Finally,
the test suite, created using nosetests, was executed.
Surely, I was seeing similar results in dev environment as CI when the test
suite was executed. Logs made no sense! Assertions failed and it seemed like
they were getting executed before the tasks finished. All I was wondering was,
how are assertions going to succeed when they check result of a task that’s
still not finished? And it seemed that way to my teammates as well.
I was about to tell the team that this process is becoming frustrating and not
taking us anywhere when a strange voice in my head asked me to look one more
time - the extra mile!
It took me about 15-20 minutes to find error in one of the worker services.
Surprisingly, the error didn’t appear in the CI logs! Nor did it appear in the
log file we generate for every job that passes through the service. It might
have seemed obvious that I should’ve checked the particular service right from
the beginning. I didn’t do that as our team was living in an assumption that
we’re spitting out logs for every service in the CI tests; specially when
something fails.
Key lessons that I took away:
There’s immense scope for improvement in our existing CI framework.
We’re doing ourselves and our users a major disservice if we keep focusing on
adding features and not on fixing a core part like tests.
Log aggregation needs to improve big time. More importantly, we need to ensure
that useless actions are not logged!
Break down the CI check into multiple pieces. One Jenkins job can do the
deployment, other to run unit tests, yet another to run functional tests, so
on and so forth. If any one of the jobs fails, we stop the entire thing. That
helps us save time and, more importantly, figure what piece is exactly
failing.
Use less Python magic wherever we can. For example, Ansible playbook can be
directly executed from shell instead of using Python.
Focus on this right away instead of waiting for a major issue to bring things
down!
Going an extra mile is often rewarding. 😉
I wrote this blog to bring out my thoughts. In process, if it helps someone
else, I’d be surprised! But if you have any suggestions or comments about how
we can improve, please let me know in comments.
Until next time. 😄
Nov 20, 2017
I have had a pretty bad track record at learning new programming language by
myself, at consistently writing a blog, at continuing working on (or being
excited about) the side project I’m doing. That sounds like having a bad track
record at pretty much everything that matters in the world of open source and
programming. 😞
It took me really long to learn Python. I remember starting from print Hello
World a number of times because it had been really long since I last wrote
anything in Python. I’ve been bad at coming up with side project ideas or
contributing to many open source projects. It wasn’t until my job required me
to write Python full time that I started getting good at it. 😏
Same goes for Go. I have been willing to learn it since mid 2015.
Coincidentally, the job change I made around the time required me to learn Go
along the way. But I left that job too soon and moved back into the Python
world. I did contribute little
something to a Go based
project that I like. And, secretly, I feel happy when every release
announcement of the project requires a mention about the command I contributed.
But that’s pretty much it.
So what the heck am I gonna do now? Well, turns out I’m going to try yet
another time. 😄
Recently I came across this tweet which resonated with me in opposite way of
how it was intended to. 😉:
We all have some spare time and I figured I could use some of it to write
something in Go.
So, I planned to do a tiny side project that’s not going to be of much use to
anyone except me. But, it would help me learn Go - the growth I’ve wished to
make to my profile. This time, I’m going to try and blog as I make progress on
the project. That way, I might actually end up making a helpful contribution to
myself.
The project’s hosted on GitHub and, at the
moment, has nothing but a short README of planned features.
Hope to take this to a closure. Best of luck to me! 😉
Nov 5, 2017
This Diwali, me and my wife decided to
visit Udaipur. We started on the
morning of New Year day, which is celebrated right after Diwali in
Gujarat.
Day 1
mostly travel and getting trapped in city traffic
On the way, we visited a Jain
tirth called
Rishabhdev which seemed more of a
Digambara tirth than
Shwetambara tirth.
The temple was nice. I had already been there once about 10+ years back. But
didn’t remember much of it.
After reaching Udaipur around 5 PM, we went out to have food. And in the quest
of visiting couple of places of interest, I got us trapped in city traffic near
Fateh Sagar Lake. I took the car to really narrow lanes which were not well suited to drive a car through it. 😉
Day 2
we absolutely regretted visiting Udaipur
Our first full day in Udaipur started with breakfast with our hosts (we booked
homestay via Airbnb).
Our first stop was Sajjan Garh
Palace or the Monsoon Palace.
Except the view from the top and an ambient cafe, we didn’t find anything
really interesting in the palace itself. Kullad chai at the cafe was nice!
Our next stop was the Sajjangarh Biological
Park which is just
beneath the Sajjan Garh Palace. Park itself is pretty nice but the crowd and
the time of day ensured that we wouldn’t see many animals. Those we saw, were
in rest mode after their lunch. Apparently, like humans, even animals prefer
having a nap after lunch. 😉
The biological park is pretty big and considering the hot weather we visited
Udaipur in, it was only wise to roam around the park in a golf car.
Next we went to Gulab Bagh
which is the biggest garden in Udaipur. We wanted to visit the Gulab Bagh
Public Library inside the garden but it was closed. Maybe due to Diwali
holidays. Gulab Bagh was nice to walk through. It’s a lush green and peaceful
area in the most visited part of the city.
From Gulab Bagh, we walked to Udaipur City
Palace which is probably
the most visited spot in the city. The rush inside the palace museum was too
much for me and my wife to handle. And there was persistent nuisance from some
fellow tourists - a group of 6-7 guys. It got suffocating due to the rush and
we decided to leave the palace after seeing only a couple of rooms in the
museum.
Thoroughly disappointed because of the insane crowd we were seeing this day, me
and my wife went back to the homestay and decided to rest the next day.
Day 3
it was surprisingly awesome!
We decided a day before to relax at the room and not go out to any tourist
spots in the city. Out of habit, I woke up early and decided to read a book on
Mewar history that I found at the hosts’
place.
Few pages into the book, I got really excited reading about the history. By the
time my wife woke up, I had mentally decided to not relax on this day. 😄
After some discussion with our host, we decided to visit
Eklingji temple because of the
historical significance it has in Mewar. The temple is about 25 km from Udaipur
and our host told us that it might not be as overcrowded as the city itself. We
ensured to visit during the darshan
timings so that
we can take a look at all the parts of the temple. It turned out to be a really
good decision on our part and we also got a book from outside the temple that
had more historical facts about the temple detailed in it.
While checking out the nearby places around Eklingji, I found that there are
two more temples which are as old as Eklingji (built in 8th century CE) or
older than it. These were:
- Sahastra Bahu temples
- Shantinath Jain temple
While Sahastra Bahu
temples were built in
early 10th century CE, the Jain temple was built around 3000 years back. The
history, carving and landscape of this temples completely amazed us. And they
were not even 10% as crowded as the main city.
I’m glad I read up some history and ended up visiting places of historical
significance. We surely witnessed some amazing architecture that has stood
through the centuries.
In the evening we visited Maharana Pratap
Smarak. It is a
museum full of history and miniatures of Chittorgarh, Kumbhalgarh, City Palace,
and battlefield of Haldighati. The view from the top of Moti
Magri hill where the museum rests is
also nice!
Day 4
we were left dumbstruck
While reading about Mewar’s history the previous day, I stumbled upon
Jaisamand Lake. Since it was a few
hours detour on the highway back to Ahmedabad, we visited this lake on the last
day.
The road from Udaipur to Jaisamand lake is absolutely amazing. It’s got a lot
of turnings and a terrain that rises and falls very, very often. Driving car on
this road was nothing short of a pleasure!
Upon reaching Jaisamnd Lake, we were completely blown away by its expanse. It
is an artifical lake and second-largest of its type in India (per some Udaipur
locals, second-largest in Asia). There’s also an island in the lake which hosts
a resort. Should be interesting staying there. 😉
It surprised us how they created such a huge lake so many years back!
Technological advancements back then were almost inexistent yet, the
constructions (lakes and palaces) from that era were surprisingly amazing.
That’s it!
Udaipur is about 4-5 hours drive from Ahmedabad. Roads are pretty good but, if
you’re driving at night, beware of the potholes that show up surprisingly on
the highway. There’s a good number of restaurants on the highway.
We plan to visit Udaipur again during off-season period. I’ll be updating the
post with few pics of the places I’ve mentioned. Until next time! 😄
Nov 3, 2017
Series: Speaking
Those who know me are aware that I’ve given talks at various
Meetups in the past. I’ve presented
at colleges as well. But I’ve never written about it. This time, I decided to
write my thoughts (in general). Also I’d write about my experience while talking at
Nirma University, Ahmedabad during past two days about
career options for Python programmers.
One of the reasons I’m always eager to talk at colleges/universities is that it
gives me an opportunity to go back to that lively part of the world where I
myself had a great time. School and college days are the days when most of us
make best friends and memories. I love and enjoy interacting with the students;
and, more often than not, end up learning something from their
questions/feedback.
Another reason I like to go there and talk is that I see a pretty big gap
between students’ knowledge and industry expectations.
General thoughts (not specific to any college/university)
There’s no shortage of engineers in India - for pretty much any domain. But
most companies don’t hire because they don’t find students’ knowledge up to the
mark. Students excel at academics but have very less or no clue of what’s
really going on in the industry.
For example, Python is gaining momentum in Indian IT industry since past few
years. And in spite of being such a beautiful and easy-to-learn programming
language, numerous students are unaware of it. I’ve interacted with number of
students in past few years, and more than few have mentioned that they get so
scared of programming while studying C and C++ that they contemplate giving
up on programming! In worse case, sometimes they even make up their mind to do
MBA instead. Most, if not all, have felt that Python should be the first
programming being taught in college/university instead of C. Debatable topic,
nonetheless.
Having studied these programming languages myself, I agree with the notion of
making Python the first programming language to be taught instead of C. It
helps get students excited about programming. And then it would make more sense
to teach C and C++.
Experience at Nirma University
Nirma University was my dream university as a 10+2 student. But, due to health
reasons, I was bedridden for almost four months before the final (board) exams
and didn’t get to work hard enough to realize the dream. Naturally I was super
excited about giving a talk at Nirma! 😄
The students I interacted with were from Electronics & Communication (EC)
branch and had a fractional course on Python in their fifth semester
curriculum. As a part of it, I was invited as an industry person to talk to
them about how Python is used in industry and what kind of
jobs/roles/opportunities are open for Python programmers.
To be honest, it’s my favorite part to talk to students about the plethora of
opprtunities that Python, Linux and open source in general would open up for
them.
Topics I covered (link to
slides):
- History of Python
- Which organizations use Python and how they use them
- Detailed discussion about
- Web Frameworks
- How Instagram uses Django.
Link
to their engineering blog
- How hot the domain of Data Science is and awesome collection of projects
under the SciPy ecosystem
- How Machine Learning and Artificial Intelligence are changing things. I
recommended them to watch Person of
Interest and Eagle
Eye to get a better understanding.
😉
- Python for Embedded Systems
- How to leverage MOOC content to learn more about Python and its applications!
- …and more
While talking about web frameworks, I explained what backend and frontend are.
We take the 24x7x365 availability of our favorite apps (WhatsApp, Instagram,
FB, Twitter, etc.) lightly; even, for granted. With examples, it was easy to
explain them that it’s not really an easy task and it requires good deal of
engineering effort. Ironicaly, WhatsApp faced glitches the same day in the
afternoon and hashtag #whatsappdown
was trending in India for almost rest of the day! I guess they’ll never forget
frontend-backend for rest of their career. 😉
That’s it!
I’ve uploaded the slides to
Slideshare. If you
are one of the students from Nirma University that attended the seminar,
consider providing your feedback on the form at the end of the slides.
Let me know what you think in the comments below! Until next time. 😄
Oct 15, 2017
As a part of my ongoing Ansible blog
series, I plan to walk-through the
provisioning code of CentOS Containr
Pipeline service which
lives under provisions directory.
My plan:
- Walk-through the existing code base
- Make changes to the code without touching my fork of the repo
- Push and test the new code
For the second part, i.e., modifying the code without touching my fork, I
needed to create a new git repository from the said
provisions
directory.
A bit of Google-fu and this
stackoverflow answer came to rescue! All I did was, use this git subtree
command to create a new branch that holds the code I’m interested in:
$ git subtree split -P provisions -b provisions-only
Here, provisions is the directory I want to create a new git repo out of and
provisions-only is the branch that will hold the code that’s inside
provisions directory
Create a new directory outside the directory holding my fork of the repo and do
git init:
$ mkdir ~/repos/cccp-provisions
$ cd ~/repos/cccp-provisions
$ git init
And git pull from the fork to new directory:
$ git pull ~/repos/container-pipeline-service provisions-only
Here, ~/repos/container-pipeline-service is the path to the forked repo that
I don’t want to touch and provisions-only is the branch in that repo that
holds the code I’m interested in.
Whenever there’s a change in the big repo (container-pipeline-service) and I
want to copy the changes to cccp-provisions repo, all I need to do is execute
the same git subtree command I used above in the big repo, and do git pull
--rebase in cccp-provisoins :
$ cd ~/repos/container-pipeline-service
$ git diff --unified=0
diff --git a/README.md b/README.md
index fd32557..fb14f4f 100755
--- a/README.md
+++ b/README.md
@@ -0,0 +1,2 @@
+Test
+
diff --git a/provisions/README.md b/provisions/README.md
index f956524..67f70dc 100644
--- a/provisions/README.md
+++ b/provisions/README.md
@@ -0,0 +1,2 @@
+Test
+
$ git add -A
$ git commit -m "Test"
$ git subtree split -P provisions -b provisions-only
Created branch 'provisions-only'
3dc47520e2d7a0bb603b6f0ac9dd60143f99f331
$ cd ~/repos/cccp-provisions
$ $ git pull --rebase ~/repos/container-pipeline-service provisions-only
remote: Counting objects: 3, done.
remote: Compressing objects: 100% (3/3), done.
remote: Total 3 (delta 2), reused 0 (delta 0)
Unpacking objects: 100% (3/3), done.
From /home/dshah/repos/container-pipeline-service
* branch provisions-only -> FETCH_HEAD
Updating d203d40..3dc4752
Fast-forward
README.md | 2 ++
1 file changed, 2 insertions(+)
What I did above was: modify two files; one inside the provisions directory
and one outside it; committed the changes it to my master branch and using
the git subtree command, created a provisions-only branch like we did
earlier. When I go to the cccp-provisions directory and do git pull
--rebase, git knows that only one of the two files modified in that commit is
of our interest and so it changes only one file:
Fast-forward
README.md | 2 ++
1 file changed, 2 insertions(+)
That was pretty cool for me! Now I can go ahead and walk-through the
provisioning bits and also make changes to it without affecting my fork of the
repo.
Oct 3, 2017
Series: Ansible
In previous post, we
did a combination of trivial and non-trivial tasks on the command line:
- install
httpd package on the servers in group servers,
- modify a line in
/etc/httpd/conf/httpd.conf file to set our desired value
for port and
- set the
httpd service into restarted mode.
At the end of the post, I mentioned that we can automate these tasks by writing
a playbook. That’s what we’ll do for this post.
Ansible Playbooks
As official
documentation suggests,
playbooks are Ansible’s configuration, deployment and orchestration language.
They provide a very nice analogy as well:
If Ansible modules are the tools in your workshop, playbooks are your
instruction manuals, and your inventory of hosts are your raw material.
With the help of playbooks, we can:
- download and install packages
- configure services
- start/stop/restart server processes
- perform rolling upgrades
- interact with load balances and monitoring systems
Playbooks can be used for various things and can have various tasks written in
them. It would be nearly impossible to cover each and every aspect of it.
In this post, we will cover its basics by creating a playbook out of the tasks
we performed in last post.
Plyabooks are written in YAML
format. Each playbook
can have one or more ‘plays’ in it. Every play is targeted at a group of hosts
(serversin our example.)
Let’s create a playbook (store it in, say, playbook.yaml file) for the tasks
we performed in previous post:
---
- hosts: servers
tasks:
- name: Install httpd
yum: name=httpd state=present
- name: Change port to listen on
lineinfile:
path: /etc/httpd/conf/httpd.conf
regexp: "^Listen"
state: present
line: "Listen {{ http_port }}"
- name: Restart and enable the service
systemd: name=httpd state=restarted enabled=yes
Here we have written only one play and that is for the group servers. tasks
is a list of ad-hoc Ansible commmands that we want to execute on the remote
hosts in the group servers. Earlier we executed these ad-hoc commands from
the command line with ansible command. There are three tasks in this
playbook:
- First task uses
yum module to install the httpd package.
- Second task uses
lineinfile module to replace the specific line in the file
/etc/httpd/conf/httpd.conf with the one we’re interested in.
- Third task restarts and enables the
httpd server using systemd module.
Let us first uninstall httpd package from the remote systems. This is not
really necessary to run the playbook but, we’re doing it to validate that all
tasks we did earlier are performed as expected by the playbook.
$ ansible -m yum --args="name=httpd state=absent" servers
And now run the playbook:
$ ansible-playbook playbook.yaml
PLAY [servers]
**************************************************************************************************************************************************************
TASK [Gathering Facts]
******************************************************************************************************************************************************
ok: [host1]
ok: [host2]
TASK [Install httpd]
********************************************************************************************************************************************************
changed: [host1]
changed: [host2]
TASK [Change port to listen on]
*********************************************************************************************************************************************
changed: [host2]
changed: [host1]
TASK [Restart and enable the service]
***************************************************************************************************************************************
changed: [host1]
changed: [host2]
PLAY RECAP
******************************************************************************************************************************************************************
host1 : ok=4 changed=3 unreachable=0 failed=0
host2 : ok=4 changed=3 unreachable=0 failed=0
See how the name we set for every task in the playbook.yaml file is shown
in above output. As an aside, execute the very same command again and observe
the output. Everything that shows changed in above output will instead show
as ok because nothing changed in remote systems as everything was setup
just a few seconds back. 😉
That’s it for this post
In upcoming posts, we’ll be creating an example application and deploying that
using Ansible Playbooks. Although it won’t be as complex as most real-life
applications and their deployments, it’ll give a fair idea of how Ansible can
be used to deploy non-trivial applications.
If you have any feedback/suggestions, leave it in the comment section at the
bottom of the post. Until next time. 😉
Sep 30, 2017
Series: Ansible
In the previous post
on inventory file, we saw that inventory file is central location that stores
information about the remote hosts. It’s not necessary that we would always
want to deal with remote hosts. Ansible can also work on the control system
(localhost).
$ cat /etc/ansible/hosts
localhost
[servers]
172.29.33.22
172.29.33.23
And let’s try the ping module:
$ ansible -m ping all
localhost | UNREACHABLE! => {
"changed": false,
"msg": "Failed to connect to the host via ssh: Permission denied
(publickey,gssapi-keyex,gssapi-with-mic).\r\n",
"unreachable": true
}
172.29.33.23 | SUCCESS => {
"changed": false,
"ping": "pong"
}
172.29.33.22 | SUCCESS => {
"changed": false,
"ping": "pong"
}
Oops, that failed to ping localhost - the only ping that we would expect to
work even if everything else failed. Let’s see the msg part of the output -
Failed to connect to the host via ssh: Permission
denied(publickey,gssapi-keyex,gssapi-with-mic).. Why would I want to connect
to localhost over SSH? Let’s set ansible_connection in the inventory file:
$ cat /etc/ansible/hosts
localhost ansible_connection=local
[servers]
172.29.33.22
172.29.33.23
That tells Ansible to connect to localhost using connection type local. So,
no more SSH’ing into localhost:
$ ansible -m ping all
localhost | SUCCESS => {
"changed": false,
"ping": "pong"
}
172.29.33.23 | SUCCESS => {
"changed": false,
"ping": "pong"
}
172.29.33.22 | SUCCESS => {
"changed": false,
"ping": "pong"
}
Voila! That was successful.
As you might have guessed by now, default ansible_connection type is SSH. And
it can be changed by setting ansible_connection to desired (and valid) value
in the inventory file. Other connection types include:
- SSH protocol types
- smart (default)
- ssh
- paramiko
- Non-SSH connection types
Throughout the series, we’ll mainly be working using the default SSH connection
type. We’ll be deploying containers as well and will see if we need to use the
docker connection type then. 😄
Host Variables
We can easily assign host variables in the inventory file. These variables can
then be used in playbooks.
$ cat /etc/ansible/hosts
[servers]
172.29.33.22 http_port=8080
172.29.33.23 http_port=9000
Above snippet defines http_port to different values for the two hosts. Let’s
bring up httpd webserver on these hosts on the specified port.
$ ansible -m yum --args="name=httpd state=present" servers
$ ansible -m lineinfile --args="regexp=\"^Listen\" path=/etc/httpd/conf/httpd.conf state=present line=\"Listen {{http_port }}\" " servers
$ ansible -m systemd --args="name=httpd state=restarted" servers
That’s handful of commands to execute. Let’s see what each one does.
- First we install
httpd package using the yum module. state=present
installs a package if it’s not already installed on the system.
- Next, we replace a line in file (
lineinfile) that starts with Listen
(regexp="^Listen"). The \ are used as escape characters so that a " is
not considered as closing quote.
- Finally we start
httpd service in those systems using systemd module.
Now if we curl these systems, we’ll be greeted with default Apache web
server page. All we need to do is curl 172.29.33.22:8080 or curl
172.29.33.23:9000.
Group Variables
If we’d like to set variables for entire group, servers in our example, all
we need to do is:
$ cat /etc/ansible/hosts
[servers]
172.29.33.22
172.29.33.23
[servers:vars]
http_port=8080
And then executing similar commands as those we did earlier would start httpd
server for us on port 8080 of both the systems.
Aliases
If you’ve used Linux command line for some time, chances are you have already
heard of alias. It’s similar concept in Ansible as well.
An alias helps you define name for a host. In our example, we’ve specified IP
addresses of the hosts in our inventory file. If we’d like to ping only one
system, we’ll have to do:
$ ansible -m ping 172.29.33.22
That doesn’t look cool. Let’s set aliases for our two hosts:
$ cat /etc/ansible/hosts
[servers]
host1 ansible_host=172.29.33.22
host2 ansible_host=172.29.33.23
And now we can simply do:
$ ansible -m ping host1
That’s it for this post
In this post, we looked at some non-trivial ad-hoc commands of Ansible. It
looks cool but, executing a bunch of commands every time on the command line is
not really fun. Instead, we can use Ansible Playbooks to perform all tasks we
did above with a single command. We’ll soon look at Ansible Playbooks!
As always, if you have any comments/feedback/suggestion, please let me know
below! Until next time. 😉
Sep 29, 2017
Series: Ansible
In this post we’ll talk about the concept of Inventory. When we installed
Ansible, an example inventory file was automatically placed at the location
/etc/ansible/hosts for us. This is the default inventory file for Ansible.
Inventory file
An inventory file consists of the information of various remote hosts that
Ansible knows of. This file needs to be configured before we can start using
Ansible to work with remote systems.
Hosts specified in the inventory file can either belong to a group or be
ungrouped. A group is specified like below:
[groupname]
host1
host2
Ungrouped hosts should be specified before specifying any grouped hosts. You
can either provide FQDN or IP address of the host. Make sure that the remote
host(s) is/are reachable from the system you’re using to run Ansible commands
through the FQDN or IP provided in inventory file.
Inventory file can be of different formats. The one you saw above and that
we’re going to follow throughout the series is INI-like syntax (Ansible’s
default). You can read about other formats in Ansible’s
documentation.
Playing with remote systems
For this post, we’re going to work with two remote systems. You can work this
out in various ways by creating two Linux virtual machines on your laptop or a
cloud provider. Two systems I’m going to work with have IP addresses
172.29.33.22 and 172.29.33.23.
My inventory file (/etc/ansible/hosts):
$ cat /etc/ansible/hosts
[servers]
172.29.33.22
172.29.33.23
Let’s start throwing some Ansible magic to them:
$ ansible -m ping all
172.29.33.23 | UNREACHABLE! => {
"changed": false,
"msg": "Failed to connect to the host via ssh: Permission denied (publickey,gssapi-keyex,gssapi-with-mic).\r\n",
"unreachable": true
}
172.29.33.22 | UNREACHABLE! => {
"changed": false,
"msg": "Failed to connect to the host via ssh: Permission denied
(publickey,gssapi-keyex,gssapi-with-mic).\r\n",
"unreachable": true
}
Oops, that was embarassing. Our first real Ansible command failed. 😢
Ah, our host system doesn’t have SSH access to the remote systems! We need to
configure that first by enabling password-less SSH access from host system to
both the remote systems. This can be done by creating ssh keys and copying them
to remote system.
And then execute same command again:
$ ansible -m ping all
172.29.33.23 | SUCCESS => {
"changed": false,
"ping": "pong"
}
172.29.33.22 | SUCCESS => {
"changed": false,
"ping": "pong"
}
Now that worked like a charm. One thing you need to ensure when configuring SSH
access is that, by default, Ansible will use same user to connect to remote
system via SSH as that on the host you’re executing Ansible from. That means,
if on the host system you’re logged in as user randomuser, Ansible will try
to connect to remote system as the user randomuser only.
But what is above command doing anyway? It’s using the module ping on hosts
that belong to group all. In response, since SSH connectivity is OK, it’s
getting a pong response and the result is SUCCESS.
But we didn’t configure all group; we configured servers group. By default,
Ansible has two groups: all and ungrouped. Hosts that are not a part of any
user defined group belong to the group ungrouped and all contains all hosts
in the inventory file.
Let’s do something else on this remote systems. Let’s do yum -y update
(assuming they are running CentOS/RHEL) on these systems:
$ ansible -m yum --args="name='*' state=latest" all
We use yum module of Ansible and ask it to work on all installed packages
(name='*') such that their updated to their latest versions (state=latest).
* is a regular expresssion which indicates “all installed packages” to yum.
Expect this command to take some time and print ugly output. It’s going to
return only when the updates have been downloaded and installed on the remote
system. Time it will take depends on: number of updates available, internet
speed and type of disk (HDD vs. SSD).
Let’s try shell module which executes the specified command on remote
systems:
$ ansible -m shell --args="date" all
172.29.33.22 | SUCCESS | rc=0 >>
Fri Sep 29 15:00:30 UTC 2017
172.29.33.23 | SUCCESS | rc=0 >>
Fri Sep 29 15:00:30 UTC 2017
$ ansible -m shell --args="df -h" all
172.29.33.22 | SUCCESS | rc=0 >>
Filesystem Size Used Avail Use% Mounted on
/dev/vda1 10G 1.2G 8.9G 12% /
devtmpfs 900M 0 900M 0% /dev
tmpfs 920M 0 920M 0% /dev/shm
tmpfs 920M 17M 904M 2% /run
tmpfs 920M 0 920M 0% /sys/fs/cgroup
tmpfs 184M 0 184M 0% /run/user/0
172.29.33.23 | SUCCESS | rc=0 >>
Filesystem Size Used Avail Use% Mounted on
/dev/vda1 10G 1.2G 8.9G 12% /
devtmpfs 900M 0 900M 0% /dev
tmpfs 920M 0 920M 0% /dev/shm
tmpfs 920M 17M 904M 2% /run
tmpfs 920M 0 920M 0% /sys/fs/cgroup
tmpfs 184M 0 184M 0% /run/user/0
tmpfs 184M 0 184M 0% /run/user/1000
rc=0 that we see in the first line of output for both systems indicates that
return code of executing the command was 0 (which means there were no errors.)
That’s it for this post
There’s more that can be talked about inventory file: host variables, group
variables, aliases, and more. But I’ll keep it for the next post to avoid
making this too long. I personally feel bored of reading really long posts
written across the Internet. 😉
As always, if you have any comments/feedback/suggestion, please let me know
below! Until next time. 😉
Sep 28, 2017
Series: Ansible
As I mentioned in my previous
post, I will be
starting with Ansible series. Best way to find all posts in Ansible series to
use this link.
If you’re reading this post, chances are you already know what Ansible is and
what it is used for. So, instead of repeating what numerous other posts across
the Internet already have to say, I’d try to put it in short as to what we use
Ansible for and how it helps us.
We use Ansible as a configurtation and deployment tool. It helps us configure
systems with the packages and tools we need on it to deploy the end product.
We use Ansible Playbooks (will talk about this in later posts) to install
packages, start services and ensure things are up and fine. We plan to use its
“Continuous Deployment” feature to automate deployments on different
environments (pre-prod, prod, etc.)
Let’s get into setting up Ansible on our system and perform some tasks with it.
Ansible is generally used to configure remote systems but, for this post, we’ll
be using it on localhost only. That is, install it on localhost and perform
operations on localhost as well.
We’re going to use CentOS 7 system for the purpose of this series. Except
installation, all other steps should work just fine irrespective of the
underlying distro.
Installation
$ yum -y install ansible
At the time of writing this, above command will install version 2.3.2.0-2.el7
for us. If you’d rather want to install the latest (bleeding edge) version,
follow below steps:
$ yum -y install epel-release
$ yum -y install python-pip
$ pip install ansible
Benefit of installing via yum is that packages in official RHEL/CentOS
repositories undergo a good deal of testing to ensure enterprise stability and
security.
What we get upon Ansible installation?
Once you’ve installed Ansible, on your command line, type ansible and hit Tab
key twice to load all possible commands that start with ansible.
Ones we will be discussing in this series are:
ansible: runs a specified task on target host(s).
ansible-console: drops you into a shell that works as REPL
(Read-Eval-Print-Loop). It allows running ad-hoc tasks against a chosen
inventory.
ansible-doc: provides documentation on the command prompt. It’s really
helpful for quick references where we are not completely sure but have a
rough idea.
ansible-doc yum would print help for the yum module whereas
ansible-doc -s yum would print a snippet which can then be copied to playbook
and modified.
ansible-galaxy: helps us manage roles using Ansible
Galaxy.
ansible-playbook: is the most interesting and heavy lifter among all. It
executes a playbook passed to it as an argument. We’ll have quite a few posts
on this. 😉
ansible-pull: pulls playbook from VCS server and run them on the machine
executing ansible-pull. It helps invert default push architecture of Ansible
into pull architecture.
ansible-vault: helps safeguard sensitive information stored in a data file
used by Ansible.
As mentioned in the beginning of this post, we’ll perform some actions on
localhost using ansible command.
Install httpd on our CentOS system:
$ ansible localhost -m yum --args="name=httpd state=present"
This tells ansible to execute on localhost, use the module yum and
pass additional arguments "name=httpd state=present" to the yum module. As
a result of executing this command, httpd package will be installed on the
localhost.
Start httpd server we just installed:
$ ansible localhost -m systemd --args="name=httpd state=started"
This tells ansible to start the httpd server we installed in previous
command. It does so using systemd module.
Check if it was actually started:
$ curl localhost
Enable httpd server to start at boot time:
$ ansible localhost -m systemd --args="name=httpd enabled=true"
You can verify that httpd server starts upon boot by rebooting the
localhost using the same curl command mentioned earlier.
That’s it for this post
In the next post, we will take a look at the concept inventory in Ansible. If
you have any comments/feedback/suggestion, please let me know below! Until next
time. 😉
Sep 26, 2017
When I decided to make some use of this domain, one (and probably most
important for me) thing that I had in mind was to write a series of
posts/articles on tech that I work on or that interests me. I had already
decided on using Hugo but was not sure how to group
articles in the form of a series.
That’s when I came across Nate Finch’s website and read his
post on how he managed to group
articles into a series. Knowing how greatly I suck at frontend, I decided to
clone his repo and modify things to suit my
requirements. I didn’t want my discomfort with frontend get in the way of
writing!
In coming few days, I’ll be writing a tech series, starting with
Ansible. At the moment, I have not decided how many
posts I’ll write or what specific topics I’ll dive into or how frequently I’ll
write. The only thing I’ve decided to do is to write.
Through writing, I want to share what I know (with the hope of being helpful to
someone some day) and learn from the feedback/suggestions of readers.
Sep 24, 2017
This is not really my first attempt at blogging. I’ve tried and failed numerous
times so far. 😃 This might or might not be another such attempt at
writing.
I enjoy hosting and talking at meetups. During one such recent meetup, I
volunteered to give a talk about technologies I had not played much with.
It was the result of challenging myself to prepare for this talk that I decided
to start writing - as a challenge. And as every other living creature, I hope
to succeed. 😄
I intend to write about my learnings on various technologies that I play around
with. I hope it helps someone else out there some time.
I don’t think of myself as an expert at any of the technologies I intend to
write about; so if you have any comments/suggestions/feedback, please feel free
to ping me. 😉
