Restructuring CentOS Container Pipeline using OpenShift - Part 1

In this post I’m going to talk about why we are restructuing the CentOS Container Pipeline service and how OpenShift is key to it. There’s bunch of material available on the Internet so I don’t really need to write about OpenShift.

I gave a talk at about our service and got some great feedback. Most common feedback was that such a service would be immensely useful to the opensource community. But deep down, I knew that it’s not possible to scale the current implementation of service to serve a large community. It needed a rejig to be useful to the users and not a pain in bad places for its administrators! 😉

What does the service do?

Before I talk about the issues and how we’re handling them in new implementation, I’ll quickly jot down the features of the service.

Issues with the old implemention

Our old implementation of service has a lot of plumbing. There are workers written for most of the features mentioned above.

All of the above is spread across four systems which are quite beefy! Yet, we couldn’t manage to do parallel container builds to serve more requests. A couple of teams evaluated our project to bring up their own pipeline because they didn’t want to use public registry. However, they found the service implementation too complex to understand, deploy, and maintain!

How are we handling (or planning) things in new implementation?

In the new implementation of the service which is still to be moved under the official repo, we are using OpenShift exclusively for everything, for every feature that the service provides.

Although we’re far from done, we have successfully implemented and tested that these features work fine in an OpenShift cluster:

We’re relying heavily on the OpenShift and Jenkins integration. Every project in the container index has an OpenShift Pipeline of its own in the single OpenShift project that we use. All of the implemented features work as various stages in the OpenShift Pipeline.

For logging, we’re using the EFK (Elasticsearch - Fluentd - Kibana) integration in OpenShift. To be honest, we’re still learning how to use Kibana!

This new implementation hasn’t been deployed in a production environment yet. However, it’s relatively straightforward to deploy than the old implementation and can even be deployed on a minishift environment.

In progress items

We are still working on things like:

That’s it

This blog went on to be longer than I wanted it to be! But it’s a gist of what we’ve been doing as CentOS Container Pipeline team since past few months. In coming posts, I’ll be talking about individual implementation details!

Until next time… 😄