In response to the Rad Hat© killing free and open-source CentOS® 8, CloudLinux announced the commitment to release a free, open-sourced, community-driven, 1:1 binary compatible fork of RHEL® 8 (and future releases) in the Q1 of 2021. CloudLinux will invest $1 million annually in development and establish a community initiative around its RHEL fork intended as a safe haven for CentOS users left stranded by the RedHat announcement.
- Question 1: Can you explain to our readers what significant changes happened at CentOS and why they should care?
RedHat announced that they would terminate support for CentOS 8 as of December 2021. This termination gives only a year for people who are currently using CentOS 8 to migrate to another distribution. After December 2021, no new security updates will come to CentOS.
Redhat also pitched CentOS stream, where RedHat plans to test all the new features for RHEL. I believe that the CentOS stream will not work for most production environments.
- Question 2: What drawbacks and/or benefits might organizations encounter from CentOS shifting to an upstream build?
A lot of work goes into making RHEL very stable, and CentOS being a 1:1 fork of RHEL, benefited from that. This will unfortunately not be the case with the CentOS stream, as the changes will go into the stream before being adopted by RHEL, therefore, the stream will quickly become unstable. Before long, people will be scared to update their servers because they don’t want to risk breaking their workloads.
The approach of killing a stable, free Linux offering is not new to RedHat. In 2003, Red Hat Linux was discontinued, and instead, Red Hat Enterprise Linux (RHEL) was born. The main difference was that Red Hat Linux had a free version, and RHEL didn't. At that time, RedHat promised that Fedora Linux would be that free option for the community. Yet, it quickly became evident that Fedora was far too unstable for any production use. I believe that RedHat is using the same playbook this time around.
- Question 3: How deeply does CloudLinux rely on CentOS? How does this change affect your work? And, will CloudLinux be hiring more engineers as a result?
CloudLinux OS is a fork of RHEL. So, the production of a new version of CloudLinux OS 8 is not affected.
However, many of our customers use CentOS in places where CloudLinux OS doesn't make sense (or will until it makes sense). We make it very easy to switch from CentOS to CloudLinux, so it is essential for our company that the community has a free Linux distribution compatible with CloudLinux OS.
On the other hand, we see many CentOS users using our rebootless kernel update software (KernelCare). So, just having that community of users who use free Linux distribution is crucial to us.
Question 4: As a response to the recent changes in CentOS strategy, CloudLinux announced a 1:1 binary compatible, community-driven fork of RHEL called Project Lenix. Can you tell us more about the motivation behind this project?
There are several reasons why we have decided to step in. Since the inception of the company, we were the beneficiaries of open-source projects & open-source software. While we contribute where we can, I believe that is an excellent opportunity to step in and deliver a lot of value to the community.
It is also relatively easy for us. We have already built a fully functional fork of RHEL 8. We have a team of experts who maintain it every day and are getting paid for it. We can reuse about 95% of that.
Additionally, we believe it will be beneficial to us in the long run. I have already spoken about the need to have a free, stable, RHEL-based community support Linux to continue selling our products. We also believe that associating with this project will give us more credibility in the enterprise space.
Question 5: You are making a one million dollar annual investment toward Project Lenix. How do you plan on getting a return on your investment for this expenditure? Or, is this a sunk cost that is being considered a community investment?
Even though we will spend most of the money on engineering and operational teams, we are looking at it as part of our "marketing" budget. I don't envision a direct return on the investment. On the other hand, it will increase our credibility, and that is worth a lot.
Question 6: Is Project Lenix considering to develop any additional system features, or will it merely mirror an existing upstream vendor who is, in this case, Red Hat?
We are already thinking about collaborating with special interest groups, like universities, so they don't have to create another version of Scientific Linux. I’m sure other groups will want a somewhat modified list of packages on their default images, so we want to make that easy to generate. Yet, overall, we expect it to be community-driven. We are not making it for ourselves. Instead, we are making it for the community, and there are plenty of use-cases that can benefit the community if the option is free.
Question 7: Why "Lenix"? Can you tell us more about why you picked this name for the project?
Many people were anxious about the future after the RedHat announcement. We were getting many questions from customers and partners about the effect of the announcement. We needed to get the message out quickly, so we came up with ProjectLenix. It is a play on Linux/Fenix/Phoenix, with the benefit that we knew that we would not stick with Lenix OS in the long run. It is too close to Linux.
We are still working on the name and will announce it soon.
There are multiple things that we are planning to introduce:
- Community-driven governance board;
- A non-profit organization that will hold all the trademarks, funds, etc.;
- Easy to set up, fully functional build & test environment so that ANYONE can take and build their own Linux based off of RHEL.
We will listen to the community feedback and see if we need more safeguards to avoid conflicts of interest.
I don't look at RockyLinux as a competitor. I think they have a great team of people willing to freely give up their time and bring value to the open-source community. Will people have to consider which one to choose? Yes. Will it be a problem? I hope not, as switching should be easy, and both distributions should be very much alike.
We tried to join the RockyLinux effort, but at some point, we realized that it would not work at this stage. We already have a build environment that works, and we have been improving it for a decade. By the time I got in touch with RockyLinux, they already had plans for their build environment - one that would be very different from ours. My experience shows that only time can reconcile such differences.
Additionally, I worry about the future. I am sure that the RockyLinux team will create a CentOS 8 replacement in 2021. Yet, I am worried about 2023. Today, the project is new and exciting. However, maintaining a large number of packages takes a lot of work. In 2013-2014, I spoke with CentOS maintainers, before Redhat acquired CentOS. They were tired, not paid, and overwhelmed. They would go to their day job and then do maintenance of CentOS for free at night. The number of maintainers was dwindling each year because there was no one to pay for it. I haven't seen how RockyLinux plans to solve this problem.
At Cloud Linux, we plan to pay our maintainers, like we do today.
What this might mean for the end-users is that our distribution will get security patches a bit sooner than RockyLinux. We will be able to deliver minor and major updates faster. At least, that is what we have seen with CentOS for the past ten years.
Our customers know us, trust us, and realize that we will deliver. For them, the availability of a free RHEL fork is no longer questionable. We alleviated a lot of uncertainty and fears with our announcement.
We have already had so many people from the open-source community reach out to us and promise help and support. We believe vendor support will be one of the critical things to achieve for the distribution - and we already had several commitments from key players.
We want most of the day-to-day maintenance to be done by paid staff. Such work requires a lot of skill, effort, and diligence over a long period. We believe that delivering updates each day is the hardest part of the project, and we will be paying people to do that work.
For other things, like documentation, testing, support, future direction, special interests - those things we hope will come from the community.
Question 8: CentOS stream resulted in a community outcry and many criticisms. What steps has Project Lenix taken to ensure that it will remain a community initiative that will avoid any corporate conflicts of interest?
Question 9: It appears there is an additional fork named Rocky Linux. Is this looked upon as a competitor? Assuming it is, why should a client consider Project Lenix over Rocky Linux?
Question 10: Obviously, it is early, but what kind of reaction have you had from current and potential new customers since announcing Project Lenix? And, how has the FOSS community reacted to this initiative?
Question 11: Do you anticipate having heavy community involvement in this project, or will a lot of the work be done inhouse?
About Igor Seletskiy
Igor Seletskiy is a CEO and founder of CloudLinux - company behind KernelCare live patching. He is a Linux enthusiast by heart, and have been programming for Linux (or Unix) for about 25 years now.
About 7 years ago he decided to do KernelCare, partially because "we could", and partially because CloudLinux had so many people in a service provider space asking us for the product following Oracle acquisition of Ksplice. It has been 6 years and hundreds of thousands of servers since -- and it truly is a wild ride.