CPE Weekly Report: 2021-01-15

Friday , 15, January 2021 Aoife Moloney Uncategorized Leave a comment

Hi Everyone,

New Year, same CPE weekly(ish)

If you would like to see this report and toggle to the section you are
most interested in, I would suggest visiting this link
https://hackmd.io/8iV7PilARSG68Tqv8CzKOQ?view and use the header bar
on your left to skip to where you want to go!

General Project Updates

We are kicking off Q1 this year with some familiar project faces,
namely Noggin, the replacement of the current FAS system and
continuing our development of CentOS Stream.

Most of our initiatives live here
https://pagure.io/cpe/initiatives-proposal and you can use the new
issue button to submit your own proposal.

Our updated initative timetable can be viewed here for 2021
https://docs.fedoraproject.org/en-US/cpe/time_tables/ so you know when
I need it in by to review it.

We also have updated our docs section on the initiative process we
follow as we cannot accept everything so please do check it out if you
want to understand our process more
https://docs.fedoraproject.org/en-US/cpe/initiatives/

Misc

GitLab

Being very honest, I've found myself a little bit strapped for time to
give this project its due diligence over the last few months, but
please bear with us/me and expect a more concentrated effort on this
coming into Q2 (April, May, June) of this year. I apologise for the
time a resolution is taking and I really do appreciate all of your
patience.

Project Updates

*The below updates are pulled directly from our CPE team call we have
every week.*

CentOS Updates

CentOS

* Community newsletter can be read here
https://blog.centos.org/2021/01/centos-community-newsletter-january-2020-2101/

CentOS Stream

* Continuing to work on Stream 8 pushes and builds
* Investigating how to automate some module pushes
* Reviewing documentation that is available on Stream currently to
identify gaps and where needs improvement

Fedora

* OSBS is building for aarm64 & x86_64 in production since December!
* All of the projects under the fedora-infra and releng namespaces on
pagure have had their default branch migrated from “master” to “main”.
* F34 mass rebuild due to start next week

Noggin/AAA

* New sprint started focusing on testing correct access has been given
per user/account
* Last remaining apps being configured & tested with fasjson API
* Work will be tracked here https://github.com/fedora-infra/aaa-tracker/issues/4
* Our open issues board can be found here
https://github.com/orgs/fedora-infra/projects/6

Fedora Messaging Schemas

* We are working through supybot and greenwave applications currently
* There is a list of applications that require messaging schemas can
be found here https://hackmd.io/@nilsph/H1i8CAbkP/edit
* There is a readme which contains documentation on messaging schemas,
a cookie-cutter template to create the schema and a definition of Done
for writing a schemas
https://github.com/fedora-infra/fedora-messaging-schemas-issues
* The board they are working from can be viewed here
https://github.com/orgs/fedora-infra/projects/7

## Team Info

Background:

The Community Platform Engineering group, or CPE for short, is the Red
Hat team combining IT and release engineering from Fedora and CentOS.
Our goal is to keep core servers and services running and maintained,
build releases, and other strategic tasks that need more dedicated
time than volunteers can give.

See our wiki page here for more
information: https://docs.fedoraproject.org/en-US/cpe/

As always, feedback is welcome, and we will continue to look at ways
to improve the delivery and readability of this weekly report.

Have a great weekend!

Aoife

Source: https://hackmd.io/8iV7PilARSG68Tqv8CzKOQ?view

 

December updates

Friday , 15, January 2021 Rich Bowen Uncategorized Leave a comment

I usually include the below report in the monthly newsletter, and overlooked it this month. So, without further ado, here are the CentOS 7 updates that were pushed out in December:

Errata and Enhancements Advisories

We issued the following CEEA (CentOS Errata and Enhancements Advisories) during December:

Errata and Security Advisories

We issued the following CESA (CentOS Errata and Security Advisories) during December:

Errata and Bugfix Advisories

We issued the following CEBA (CentOS Errata and Bugfix Advisories) during December:

Other releases

The following releases also happened during December:

CentOS Community Newsletter, January 2021 (#2101)

Tuesday , 12, January 2021 Rich Bowen Newsletter 23 Comments

Dear CentOS Community,

As we enter the new year, I'm sure there's really only one thing on your mind, and so we'll start there.

As you are no doubt aware, the CentOS project has shifted focus from CentOS Linux - the RHEL rebuild - to CentOS Stream - the continuously delivered distribution that reflects what will be delivered in the next release of Red Hat Enterprise Linux (RHEL).

Many, many articles have been written about this, and I want to take an opportunity to call out some of the better ones, to help you understand what's happening, and where we go from here.

To those who claim that CentOS Stream will be somehow unstable, I would encourage you to read Brendan's article about how RHEL is made. Things that go into RHEL are not bleeding edge or continually shifting sands. They are small incremental changes which have been baked for a long time.

To those objecting to the term "rolling release", see Stef's article about continuous delivery, and how CentOS Linux and CentOS Stream related to RHEL.

And to those who are pre-judging CentOS Stream without the benefit of even trying it, you should read Jack's article about not knocking it until you try it. (Jack's an Ubuntu fan, but makes a lot of good points.)

Karsten has written an article about the various things that are kept in balance around the CentOS project, and some of the history that led to where we are.

Finally, Scott's article about ... well, all of it ... is definitely worth your time if you want to have a deeper understanding about why people are angry, and why they are right, and wrong, to be angry.

For those of you who are planning to move to Rocky, CloudLinux, or one of the other projects that has sprung up to take the place of CentOS Linux, we wish you - and these projects - all the best. But we caution you to understand that building an OS is a big project, and it's going to take a while for them to get where they're going. Please plan your migration accordingly.

There are other things happening in the CentOS community, but we understand that this one is pretty overshadowing right now.

Hyperscale SIG proposed

A group of developers has proposed a Hyperscale SIG, which will be voted on in Wednesday's board meeting. They propose to focus on solutions around large-scale infrastructures, such as those at organizations such as Facebook and Twitter.

If you are interested in this kind of SIG, and particularly if you are running a hyperscale infrastructure, we welcome your comments and participation.

CentOS Linux 8 (20-11) released

The fourth release of CentOS 8 is now available, as of December 7th. This release is labelled 8.2011 (ie, November 2020) and is based on the 8.3 release of RHEL.

Q1 CPE Priorities

In Q1, CPE will be working on the following priorities:

  • CentOS Stream
  • Noggin/AAA replacement
  • Fedora-Messaging Schemas 
  • Flatpak indexer code merge
  • Debuginfo-d
  • Datanomer & Datagrepper V.2

We'll be updating the centos-devel list as progress is made on these projects.

Happy New Year

We wish you a 2021 that is happy and productive, and hope to see you in person before the year is out. Thanks, as always, for being part of our community.

 

Balancing the needs around the CentOS platform

Saturday , 19, December 2020 Karsten Wade Community, distro 81 Comments

These past few weeks I’ve read through and listened to a lot people’s reactions and responses to our news about the future of the CentOS Project. I see a lot of surprise and disappointment, and I also see people worried about the future and how this is going to affect them, their livelihoods, and the ecosystem as a whole. I feel a strong sense of betrayal from people, I hear that.

I don’t know if my story here is going to help you or not, but I appreciate you reading it through and listening to what I have to say. The history I cover I think is necessary to understand where we are today. From here I’m going to be available on the CentOS devel list and Twitter if you want to talk further about why I think it’s going to turn out okay.

Read More

How RHEL is Made

Friday , 11, December 2020 Brendan Conoboy distro 20 Comments

This week Red Hat announced its plan to put all its energy into CentOS Stream 8, resulting in the discontinuation of CentOS Linux 8 in one year’s time.  CentOS Stream, originally announced in September of 2019, is a continuous release of RHEL which provides updates as soon as they are developed and verified.  Many people who use CentOS Linux today now wonder if CentOS Stream 8 will be a suitable distribution for their use: is it tested, will it be stable?  If you want to know what to expect from CentOS Stream, the best starting point is knowing how Red Hat Enterprise Linux is built.  Let’s get into it!

Red Hat has been making Linux releases for such a long time, its original development methodology predates the agile manifesto.  Historically, RHEL has been built behind closed doors, its plans held close enough that even the announcement of predictable 6-month minor / 3-year major releases seemed a monumental reveal during the RHEL 8 launch.  Fortunately, how Red Hat makes Linux distributions has evolved, not just since calendar years started with “19”, and there have been multiple process generations since RHEL 8 launched just 18 months ago.  While fundamentals like upstream first, copious quality engineering, ecosystem partnership, and customer care remain the same, we work continuously to improve how those fundamentals are implemented.  

Read More

Minutes for CentOS Board of Directors for 2020-11-11

Friday , 11, December 2020 Thomas Oulevey Board minutes 9 Comments

On 2020-11-11 the CentOS Board of Directors met to discuss ongoing business.

First, the board would like to thanks everybody involved in CentOS Linux 7.9 release.

The Board was in an Executive session, where Red Hat CTO, Chris Wright joined to present Red Hat plan around CentOS Linux and CentOS Stream. A Board discussion followed.

Following up the discussion around the different users' communities impacted by proposed changes, Chris Wright, mentioned to the Board that Red Hat is also reshaping and expanding the RHEL Developer program. The details will be communicated through standard Red Hat channels.

The following resolutions were approved by the majority of the Board :

  • CentOS Stream 8 will continue with contributions for the full-support phase of RHEL 8. APPROVED
  • CentOS Stream 9 will start on schedule with the RHEL 9 Beta. APPROVED
  • CentOS Linux 9 will not start. APPROVED
  • CentOS Linux 8 ends in December 2021. APPROVED

An announcement and detailed FAQ will be prepared in next weeks.

No other issue has been discussed this month, and updates will be amended to tickets if necessary.

Minutes for CentOS Board of Directors for 2020-09-09

Friday , 11, December 2020 Thomas Oulevey Board minutes 2 Comments

On 2020-09-09 the CentOS Board of Directors met to discuss ongoing business. The Board discussed the creation of the infrastructure SIG to streamline and foster contributions in this challenging area. 

The creation of the Infrastructure SIG was approved and Aoife Moloney will be the new chair in charge to gather requirements from all actors and define the SIG baseline contribution model. Board members insisted that administrative accesses to the CentOS infrastructure will need to be carefully granted and on the principle of least privilege. Rich Bowen kindly drafted a blog post that goes in deeper details for the next steps and the challenges ahead.

A lengthy exchange happened around the feedback loop from SIG to the CentOS Board. The board would like to invite SIG chairs (or their representative) for discussion when needed, but also hear from them on a regular basis. A communication will follow-up when the board agrees on the best format and frequency for these sessions.

Ralph Angenendt announced he decided to step down from the CentOS Board of Directors. The Board would like to thank Ralph for all his service and hard work over the years.

No other issue has been discussed this month, and updates will be amended to tickets if necessary.

CentOS Stream is Continuous Delivery

Friday , 11, December 2020 Stef Walter Uncategorized 63 Comments

Continuous Delivery 101: Do the hard things continuously,
so that they become easy.

From the outside, it may appear that the way we build RHEL (and thus the CentOS Linux content) hasn’t changed in a decade. But beneath the covers, we’re pulling off a monumental transformation of how we develop RHEL without impacting our customers.

I've told this story at various conferences, but the announcements about CentOS Linux 8 and CentOS Stream have provided the impetus to tell the story here.

Three years ago, several of us working in RHEL Engineering had an idea: what if we applied modern development practices to RHEL such as continuous integration, continuous delivery, predictable release cadence … paired with open source development practices like release early release often, pull requests, forking, and code review.

Obvious, no? … No.

The Linux distribution is the grand challenge of
continuous integration and delivery.

What drew me into open source has always been this integration challenge. There is an infinite sea of uncoordinated projects. It really is an amazing example of evolution. If you squint your eyes like so, you can just about see the strange organisms, the mutations, the microcosms, and the natural selection all happening before you.

Over the last 20 years, I’ve contributed to over a hundred different projects. My contributions focused on making projects function seamlessly together so the user would have a coherent experience.

The Cockpit project is the most visible example of this. We connected about 95 Linux APIs and components, each developed separately, and released on different schedules in over 10 different distros, into a coherent user experience, delivering stable releases every other week for six years and counting.

If Linux is the grand challenge of continuous integration and delivery, then I saw RHEL as the unparalleled absolute: take ten thousand uncoordinated projects, thousands of contributors, add additional structure (like kABI) and additional guarantees (like 10 + 3 years of hardware enablement), integrate them constantly, and deliver a stable release every single day.

With dreamy (well, watery) eyes, we called such an effort “Always Ready RHEL”.

The effort started painstakingly onboarding the thousands of packages into continuous integration. It shocked many that we didn’t already have CI for all RHEL components back in 2017. But if it was easy, it would have happened much earlier.

Today, any update that goes into RHEL has to pass continuous integration gating before landing in our nightly compose, which runs automated tests for that component. Then, each change needs to be explicitly verified to a RHEL quality (mostly by Quality Engineering) before it can land in the RHEL nightly builds.

The “Always Ready RHEL” effort now continues with continuous delivery, which you now know as CentOS Stream: the RHEL nightly composes are already delivered in CentOS Stream. The whole point of continuous delivery is to make each release as stable as the one before. We’re delivering daily.

Are we done? … No.

To the untrained eye,  CentOS Stream is
already as stable as RHEL.

But the challenge here is unparalleled, and RHEL engineers carry awareness of that. The way the different teams do their work integrating RHEL is as diverse as the upstream communities themselves. Yet because so many people are iterating together toward different aspects of this goal, I’m convinced we can make Continuous Delivery a reality..

Fedora, CentOS Stream and RHEL delivery

Diagram licensed CC-SA: https://creativecommons.org/licenses/by-sa/4.0/

Here’s how the flow of delivery looks for 8 and 9:You can see the Fedora releases on the left. And the chart illustrates how CentOS Stream is synonymous with the work on RHEL X.Y releases. Technically speaking, CentOS Stream and RHEL updates are two binary packages built from the same source. An update will be published to CentOS Stream if and only if it is published to the RHEL nightly builds. Thus the RHEL nightly builds are the CentOS Stream updates you get. Once we branch from Fedora, our development gets into a stride where each change is integrated cleanly on top of everything that went before. An update is pushed to CentOS Stream if and only if it is published to the unreleased minor update of RHEL. RHEL customers later see each of these as a RHEL Errata update.

Each of these changes, whether bug fixes or features, is tested via automated tests and verified by Quality Engineering processes before landing in CentOS Stream.

The only work not directly and immediately visible in Stream is the work we do on the already-released RHEL minor versions themselves (indicated as “errata” in the diagram). Often this work is done under NDA, are embargoed, or are backports of changes already in CentOS Stream.

CentOS Stream intends to be as stable as RHEL,
It’s fundamental to continuous delivery.

But hey, even the RHEL-released product is not completely stable. Back in July, a RHEL (andCentOS) fix for the “boot hole” vulnerabilities ended up being far worse than the CVE itself: it caused many systems not to boot. Oh, man.

As a result, we’re not only investing time in reworking upstream components, but also adapting our process to ensure that this cannot happen again. Rinse, repeat.

While I wasn’t part of the decision to EOL CentOS Linux 8, I’m committed to putting my effort toward pulling off CentOS Stream. Doubly so, because it makes RHEL be Open Source: Where we can work together with an entire ecosystem on this exciting continuous integration and delivery challenge.

Open sourcing a product is hard, yet we’ve made amazing progress. So far we’ve aligned the RHEL development process with Fedora, placed all the actual sources of RHEL in one readable place, enabled contributors to open a pull request against any part of RHEL, released early and often ...

And this is just the start. There are hundreds of people working toward this CentOS Stream change, all while not missing a beat delivering the RHEL releases you’ve come to expect.

CentOS Stream is the stable and reliable
continuous delivery of RHEL

CentOS Project shifts focus to CentOS Stream

Tuesday , 8, December 2020 Rich Bowen Uncategorized 655 Comments

The future of the CentOS Project is CentOS Stream, and over the next year we’ll be shifting focus from CentOS Linux, the rebuild of Red Hat Enterprise Linux (RHEL), to CentOS Stream, which tracks just ahead of a current RHEL release. CentOS Linux 8, as a rebuild of RHEL 8, will end at the end of 2021. CentOS Stream continues after that date, serving as the upstream (development) branch of Red Hat Enterprise Linux.

Meanwhile, we understand many of you are deeply invested in CentOS Linux 7, and we’ll continue to produce that version through the remainder of the RHEL 7 life cycle.

CentOS Stream will also be the centerpiece of a major shift in collaboration among the CentOS Special Interest Groups (SIGs). This ensures SIGs are developing and testing against what becomes the next version of RHEL. This also provides SIGs a clear single goal, rather than having to build and test for two releases. It gives the CentOS contributor community a great deal of influence in the future of RHEL. And it removes confusion around what “CentOS” means in the Linux distribution ecosystem.

When CentOS Linux 8 (the rebuild of RHEL8) ends, your best option will be to migrate to CentOS Stream 8, which is a small delta from CentOS Linux 8, and has regular updates like traditional CentOS Linux releases. If you are using CentOS Linux 8 in a production environment, and are concerned that CentOS Stream will not meet your needs, we encourage you to contact Red Hat about options.

We have an FAQ to help with your information and planning needs, as you figure out how this shift of project focus might affect you.

[See also: Red Hat's perspective on this.]

CPE Weekly: November 22 2020

Tuesday , 1, December 2020 Aoife Moloney Uncategorized Leave a comment

Hi Everyone,

Below is this week's CPE weekly for week ending 2020-11-22 for both
Fedora & CentOS, and if you want to visit the hackmd link
https://hackmd.io/8iV7PilARSG68Tqv8CzKOQ?view you can then use the
header bar on your left to skip to Fedora or CentOS updates that
interest you.

General Project Updates

Final project submission date for consideration in Q1 is Friday 27th
November. If you have an initiative that may take weeks/months and
multiple people to work on and want to request it to CPE, please
follow the steps outlined in our initiatives repo and create your
issue before 27th November https://pagure.io/cpe/initiatives-proposal
Our updated initative timetable can be viewed here for 2021
https://docs.fedoraproject.org/en-US/cpe/time_tables/

Below are the projects the CPE team are currently working on for the
months of October, November & December:
* CentOS Stream Phase 4 - Build system services
* Noggin Phase 4 - Data Migration of Fedora & CentOS Accounts, Community testing
* OSBS for aarch64 - this will begin in November
* Fedora Messaging Schemas - this work is continuing from Q3 and is
being worked on part-time

Misc

GitLab

New GitLab topic sent to devel-announce@lists.fedoraproject.org &
centos-devel@centos.org on Message Bus is out. See email in hackmd
here
https://hackmd.io/oZrDwbSeSWO-l_X65A1ndg?view

Project Updates

*The below updates are pulled directly from our CPE team call we have
every week.*

CentOS Updates

CentOS

* CentOS 6 is EOL 30th November
* CFP for FOSDEM Dojo - https://wiki.centos.org/Events/Dojo/FOSDEM2021
* Updated CentOS CI Openshift staging cluster to latest 4.6.4, Waiting
for stable release in the 4.6 branch before rolling out to production.
* CentOS 7.9.2009 is released! (for x86_64, i386, ppc64, ppc64le,
armhfp and aarch64 architectures)
* Lot of work done for Noggin/AAA

CentOS Stream

* Use centos-stream-release package to convert from CentOS 6 to CentOS
Stream before 30th November
* Working on integrating ODCS in Stream
* Curating out t_functional suite
https://github.com/centos/sig-core-t_functional
* Refining our testing for finding issues at distro-level

Fedora

Staging Environment

* Completed - any issues you find please report them in fedora infra
https://pagure.io/fedora-infrastructure/issues

Noggin/AAA

* Testing team owned apps in staging with Noggin
* We will be requesting community member testing before December so
keep an eye out for an announcement!
* The teams kanban board where they track their work can be found here
https://github.com/orgs/fedora-infra/projects/6
* And we have a project tracker available to be viewed here
https://github.com/fedora-infra/aaa-tracker

OSBS for aarch64

* Basic OKD 3.11 working on aarm64 with F31
* Working on repeating that install with F33
* Next step will be to

Fedora Messaging Schemas

* This project is worked on on a part time basis as we are
prioritizing completing Noggin first before fully committing to its
completion
* There is a list of applications that require messaging schemas can
be found here https://hackmd.io/@nilsph/H1i8CAbkP/edit
* There is a readme which contains documentation on messaging schemas,
a cookie-cutter template to create the schema and a definition of Done
for writing a schemas
https://github.com/fedora-infra/fedora-messaging-schemas-issues
* The board they are working from can be viewed here
https://github.com/orgs/fedora-infra/projects/7

 

Team Info

CPE Product Owner Office Hours

IRC office hours are now once per month.Below are the logs from the
most recent meetings and dates for the next ones.

#fedora-meeting-1

* Next Meeting: 2020-12-17 @ 1300 UTC on #fedora-meeting-1

#centos-meeting

* Next Meeting: 2020-12-15 @ 1500 UTC on #centos-meeting

Background:

The Community Platform Engineering group, or CPE for short, is the Red
Hat team combining IT and release engineering from Fedora and CentOS.
Our goal is to keep core servers and services running and maintained,
build releases, and other strategic tasks that need more dedicated
time than volunteers can give.

See our wiki page here for more
information: https://docs.fedoraproject.org/en-US/cpe/

As always, feedback is welcome, and we will continue to look at ways
to improve the delivery and readability of this weekly report.

Have a great week!

Aoife

Source: https://hackmd.io/8iV7PilARSG68Tqv8CzKOQ?view

--
Aoife Moloney
Product Owner
Community Platform Engineering Team
Red Hat EMEA
Communications House
Cork Road
Waterford
ReplyForward

_______________________________________________
CentOS-devel mailing list
CentOS-devel@centos.org
https://lists.centos.org/mailman/listinfo/centos-devel