Blog.CentOS.org

Hi everyone,

Welcome to September! Below are the most recent Community Platform
Engineering project updates, and if you want to know more about our
team, see our wiki page here for more information on who our team is:
https://docs.fedoraproject.org/en-US/cpe/

Here are some upcoming IRC meetings:

CPE Product Owner Office Hours

#fedora-meeting-1

* Weekly on Thursdays @ 1300 UTC on #fedora-meeting-1
* Next Meeting: 2020-09-03

#centos-meeting

* Every second Tuesday @ 1500 UTC on #centos-meeting
* Next Meeting: 2020-09-01

GitLab AMA Session

* September 10th @ 1330 UTC on #fedora-meeting-1

Below are the project & community updates this week:

GitLab

There will be an IRC based AMA session with GitLab on Thursday 10th
September @ 1330 UTC in place of the CPE PO office hours.
We are still talking to GitLab but we are deliberately taking our time
to make sure all of the technical blockers can be met and the move
will be worth it in the end.
There is very little to no updates in the tracker, but I will include
it nonetheless https://gitlab.com/gitlab-org/gitlab/-/issues/217350
I will also be sending a separate email on details of the AMA session
later this week, such as how to submit questions in advance so there
is content ready on the day.

Note: Since I'm posting this to the blog after the meeting, you may wish to read the

CentOS Updates

CentOS

* Updated ocp.stg to OCP v4.5.6.
* Added a number of users to the jump.ci host.
* Adding monitoring/alerting for NFS slowness to the ocp cluster.

CentOS Stream

* Module push tweaks.
* Exploring how to enable fedora messaging in Stream
* Reviewing documentation on contributor policies before publishing
them later this quarter.

Fedora Updates

Staging Environment

* Services will begin to be deployed this week
* Please be patient as some services will inevitably not work due to
networking errors that the team don't know until they deploy
* Thank you again for your patience and understanding during these
last few months!

AAA Replacement

* Deployment to staging for testing is delayed due to missing firewalls in IAD2
* This has just recently been unblocked so the team will begin some
deployment and testing of Noggin this week
* Wider community testing will be available, estimated next week
* In the meantime. Please feel free to check out the team kanban board
for more information on the features the team are working on and have
already completed here https://github.com/orgs/fedora-infra/projects/6

Fedora Messaging Schemas

* List of applications that require messaging schemas can be found
here https://hackmd.io/@nilsph/H1i8CAbkP/edit
* There is a readme which contains documentation on messaging schemas,
a cookie-cutter template to create the schema and a definition of Done
for writing a schemas
https://github.com/fedora-infra/fedora-messaging-schemas-issues
* The board they are working from can be viewed here
https://github.com/orgs/fedora-infra/projects/7

Packager Workflow Healthcare

* The team have been reviewing data on how packages are built in the
fedora infrastructure for the last 8 weeks and have gathered enough
information to create a report on their findings.
* This report is currently in draft format, and is going to be
reviewed by the team first, and then sent to the devel and infra lists
in the next 2 weeks est.
* The teams work is being tracked here
https://teams.fedoraproject.org/project/cpe-cicd/kanban

Here is a reminder of what our team has committed to work on in this
quarter of the year:

The CPE team are working on the following projects for Quarter 3,
which is the months of July, August & September:
* Data Centre Move - Final Works
* CentOS Stream Phase 3
* Noggin Phase 3
* Packager Workflow Healthcare
* Fedora Messaging Schemas

Details of the above projects, and of projects currently in progress,
done and what projects are in our backlog, can be found on our taiga
board per project card:
https://tree.taiga.io/project/amoloney1-cpe-team-projects/kanban?epic=null

We also have an updated initiative timetable for briefing in new
projects to our team & key dates
here: https://docs.fedoraproject.org/en-US/cpe/time_tables/
*Note: Initiatives are large pieces of work that require a team of
people and weeks/months to complete. Please continue to open tickets
in the normal way for bugs, issues, etc.

Background:

The Community Platform Engineering group, or CPE for short, is the Red
Hat team combining IT and release engineering from Fedora and CentOS.
Our goal is to keep core servers and services running and maintained,
build releases, and other strategic tasks that need more dedicated
time than volunteers can give.

As always, feedback is welcome, and we will continue to look at ways
to improve the delivery and readability of this weekly report.

Have a great week!

Aoife

Source: https://hackmd.io/8iV7PilARSG68Tqv8CzKOQ?view

As any open source project grows and matures, the people who have always done all the things can’t do everything any more. CentOS is at that point (really, we have been for a long time) and we’ve been struggling with those kind of growing pains for some years.

Recently, we made a small change in process, so that SIG leads can manage their own SIG membership and permissions associated with SIG membership. This removed some of the load on the people who have traditionally handled this on behalf of the SIGs. This change was made possible by updates in the authentication tooling, and has the Board looking at other areas where we can move more operational things out of the Board, which can then focus on governance and executive tasks.

Another area where this need for maturity has been clear for a while is our infrastructure, where a small number of people handle all of the infrastructure tasks. As our infrastructure grows (in both size and complexity) this has put us in situations where work was blocked because those people didn’t have time to get tasks done. And, sometimes, people like to take a holiday.

This week, the Board of Directors approved a plan to move infrastructure oversight into a SIG structure, so that different parts of the infra can be delegated to a larger group of people. We have already been in partnership with CPE (Red Hat’s Community Platform Engineering team) to do some of these tasks, and this will give greater liberty to spread the load around even further, but also greater transparency around what it takes to be granted that access.

We are still in the process of figuring out what is in scope, and collecting a community of interested volunteers willing to do the work. The CentOS infrastructure is very broad, including build machines, the mirror network, systems with donated hosting, and so on, and we want to be sure that we do this in a way that doesn’t interfere with the work that’s already being done by so many people, while at the same time working to identify more places where volunteers can make a difference.

To be clear, giving administrative access to any systems still requires a great deal of trust and assurance of competence. We’re not giving the keys away to anyone who asks. But people who have demonstrated that competence and trustworthiness will be able to do some of the tasks that have, thus far, been handled by 2 or 3 people. We’ll be working to develop and publish some clear guidelines around who can be trusted with this access, and how to demonstrate that competence.

The exact details of how this SIG will operate are still being worked out. But the Board has approved Aoife Moloney as the first Infrastructure SIG chair, since she has already been sending infrastructure updates to the centos-devel mailing list roughly weekly for some months now, and is a logical person to take on this organizational responsibility. So, thank you, and welcome, to Aoife, for her willingness to tackle this job.
We welcome participation and comment from anyone with the skills or interest to be helpful, particularly if you are already administering any of the infrastructure that the CentOS project relies on.

Background:

The Community Platform Engineering group, or CPE for short, is the Red
Hat team combining IT and release engineering from Fedora and CentOS.
Our goal is to keep core servers and services running and maintained,
build releases, and other strategic tasks that need more dedicated
time than volunteers can give.

See our wiki page here for more
information: https://docs.fedoraproject.org/en-US/cpe/

General Project Updates

The CPE team are working on the following projects for Quarter 3,
which is the months of July, August & September:
* Data Centre Move - Final Works
* CentOS Stream Phase 3
* Noggin Phase 3
* Packager Workflow Healthcare
* Fedora Messaging Schemas

Details of the above projects, and of projects currently in progress,
done and what projects are in our backlog, can be found on our taiga
board per project card:
https://tree.taiga.io/project/amoloney1-cpe-team-projects/kanban?epic=null

We also have an updated initiative timetable for briefing in new
projects to our team & key dates
here: https://docs.fedoraproject.org/en-US/cpe/time_tables/
*Note: Initiatives are large pieces of work that require a team of
people and weeks/months to complete. Please continue to open tickets
in the normal way for bugs, issues, etc.

CPE Product Owner Office Hours

#fedora-meeting-1

* Weekly on Thursdays @ 1300 UTC on #fedora-meeting-1
* Next Meeting: 2020-08-20

#centos-meeting

* Every second Tuesday @ 1500 UTC on #centos-meeting
* Next Meeting: 2020-08-18

Misc

Nest With Fedora Note

Thank you to everyone who attended Nest With Fedora over the weekend
and engaged with us during our talks and social sessions. It was a
fantastic event, very well put together and run by the Fedora team and
it was both my pleasure and honour to have been a part of what was a
fantastic schedule! I cant wait to catch up on talks I missed when
they are uploaded, and the CPE team, with thanks to Michal Konecny who
put the structure of our piece together, will be submitting a
collaborative blog post to the Community blog space in the coming
weeks, recapping our experience at Nest this year. Well done Marie
Nordin, Matthew Miller and the wider team, and our very own Vipul
Siddarth, on what was a very successful and enjoyable event!

Engagement Email Feedback

At the beginning of July I sent an email to devel-announce requesting
feedback from the community on changes I and the wider CPE team have
made when scheduling projects to work on, in an effort to find the
balance between work and life. We are still searching
However, I got some very good tips that I will definitely be
incorporating which I shared at Nest during the CPE AMA Session and in
reply to the original mail.

The link to the mails are here for full reading
https://lists.fedoraproject.org/archives/list/devel-announce@lists.fedoraproject.org/thread/UNWTLOI2U4VKAU4IKJZGZLZ2DTMPEFBW/

But the suggestions and actions are here for quicker reference:
Continue to communicate regularly on projects & updates
- Will do, weekly emails have been a bit more sporadic lately as I have had
some time off

Less acronyms & abbreviations in comms
- Sure, that's an easy fix on my end and makes sense

Publish team members timezone on docs.fpo/cpe to help define our ‘working
hours’
- Will do, I hope to get to this by end of August and they will be
reflected on docs.fpo/cpe

Publish the workflow diagram to docs.fpo/cpe and add filtered versions that
are user specific
- Same as above, publishing it on docs.fpo/cpe-initiatives

Office Hours on IRC are a useful way to contact team Product Owner
- Great to hear, please feel free to stop by when you can/want to. They are
on Thursdays @ 1300 UTC on #fedora-meeting-1 and every second Tuesday @
1500 UTC on #centos-meeting

Public tracker for bugs
- Our team meets twice a day, every day on IRC to review tickets and issues
to work on. They meet @ 0830 UTC on #centos-meeting and again @ 1800 UTC on
#fedora-admin. These are public meetings so please feel free to attend.

If you would like to offer any further suggestions or feedback on the
Community Platform Engineering team, please feel free to complete our
survey which is open until August 30th
https://fedoraproject.limequery.com/696793?lang=en

Fedora Updates

Data Centre Move

* Nearly done!!!!!!
* Firewalls for staging are going up this week
* Last of the hardware has been set up for networking changes
* The bringup of Communishift is being made into a dedicated project
for work as soon as the team have capacity to do so - this may be in
late September as the members of the colo team will (hopefully) take
some very well deserved time off work before tackling this one

AAA Replacement

* Some of the Noggin team have been enjoying some very well deserved
time off work over the last week or two so work has, naturally slowed.
* The code is currently being security audited by Patrick Uiterwijk,
thank you Patrick!
* Next steps will be to successfully deploy Noggin to staging when it
has been brought back up - we are estimating this to be late next week
* Once Noggin has been deployed to staging, we would love some
community feedback on the application and its performance. We will be
emailing the infra/devel lists on when to test and how to give us
feedback when we are fully deployed in stg.
* In the meantime. Please feel free to check out the team kanban board
for more information on the features the team are working on and have
already completed here https://github.com/orgs/fedora-infra/projects/6

Fedora Messaging Schemas

* This project is being worked on by the same team as Noggin, so there
has been some progress made but not a lot as team members enjoy some
holidays
* The team have already built a list of applications that require
messaging schemas, list can be found here
https://hackmd.io/@nilsph/H1i8CAbkP/edit
* They also have completed a readme which contains documentation on
messaging schemas, a cookie-cutter template to create the schema and a
definition of Done for writing a schemas
https://github.com/fedora-infra/fedora-messaging-schemas-issues
* The board they are working from can be viewed here
https://github.com/orgs/fedora-infra/projects/7

Packager Workflow Healthcare

* Project information: This is an investigative project that aims to
look at the entire packager workflow as a single piece of tooling to
identify where failures happen, and try to identify when and why
packages fail at different points within the workflow. We hope to have
two possible outcomes from this project at the end of the quarter
(September):
* The workflow breaks at X point and we will work on a solution to fix
* OR
* The workflow works fine, but we will need better monitoring on
the pipeline so we will work on a solution for this
* The team are using this Monitor-Script
https://pagure.io/fedora-ci/monitor-gating/ and are making
improvements to it on resiliency/reliability.
* They are finishing the investigation phase of the project and are
going to document the packager workflow (with graphs I have been
promised!) showcasing how the different systems interact with one
another
* And are working on an outline of the workflow steps (from packager
PoV) and systems involved (CPE team PoV), identifying metrics to be
measured
* The teams work is being tracked here
https://teams.fedoraproject.org/project/cpe-cicd/kanban

CentOS Updates

CentOS

* CentOS-infra team have created a ticket board, check it out here
https://pagure.io/centos-infra/boards/CentOS%20CI%20Infra
* CBS Koji x86 builders moved to new(er) infrastructure
(https://lists.centos.org/pipermail/centos-devel/2020-August/055988.html)
* The team also caught up C6, C7 and C8 Linux .. 2 outstanding
Bootstrap Modules for C8 Linux (eclipse and the latest rust-toolset).

CentOS Stream

* Not too much to report this week - The team are mostly working on
developing utility scripts that will ease the CentOS 8 and CentOS
Stream packaging workflow and business as usual updates to CentOS
Stream.

As always, feedback is welcome, and we will continue to look at ways
to improve the delivery and readability of this weekly report.

Have a great week!

Aoife

Source: https://hackmd.io/8iV7PilARSG68Tqv8CzKOQ?view

Hey folks,

CPE need your help 🙂

Over the last several months we've been trying to improve how we interact and share information with you all. From the blog posts, to mails and how we work on the tickets you send us.

Here is a link to a very short survey we've put together to learn how we can give you the best experience possible going forward by understanding the experiences you've had recently.

If you could take the time (5mins max) to complete it for us it would be hugely valuable as we work on this continuous improvement - https://fedoraproject.limequery.com/696793

Cheers,

Ant.

Dear CentOS enthusiasts,

Here's what's been happening over the past month.

News:

Boothole

Last week we were made aware of a security hole in grub2, and released a fix for that. You can see the details about this on the centos-announce mailing list. Unfortunately, the fix itself had problems that caused a small percentage of systems to be unbootable. We have since released fixes for that situation, too, which you can read about in the resolution of this bug - https://bugs.centos.org/view.php?id=17631 - and additional information may be found in the RHEL ticket, here: https://access.redhat.com/solutions/5272311

Further discussion of the issue may be found in recent threads on the centos@centos.org mailing list.

User Survey

Over the past 3 or 4 months, we have been running a survey about how you use CentOS. Many thanks to those who participated in this, to help us better understand how we can give you what you need.

I've written up the results of the survey here - https://blog.centos.org/2020/07/centos-community-user-survey/

While some of the results were expected - most of you use CentOS in small to medium shops, running services either for work or personal use - there were some eye-opening things in this. To me personally, the volunteerism question shows that a lot of you are looking for places to get involved, and that we haven't done a great job telling you where and how. We'll be working to fix that.

For those of you who missed the survey, we intend to do this again, perhaps as often as twice a year, and also provide opportunities to give more feedback beyond the simple 5-minute questionnaire.

Thank you all for participating, and for being part of our community.

New website: Contribution information

By now you've noticed the fresh look on the centos.org website. But it's not just a new look. It's also a new back-end, which, among other things, makes it easier for you to contribute changes and updates to the content. Please see this blog post for information about how you can help us improve the CentOS website.

CPE Updates

The Community Platform Engineering group is the Red Hat team combining IT and release engineering from Fedora and CentOS. Our goal is to keep core servers and services running and maintained, build releases, and other strategic tasks that need more dedicated time than volunteers can give.

• July 5th - https://blog.centos.org/2020/07/cpe-weekly-2020-06-218/
• July 25th - https://blog.centos.org/2020/07/cpe-weekly-2020-07-25/

Events:

Nest

This weekend is Nest - the Fedora conference. This conference is usually known as Flock, or "Flock To Fedora", but this year, since we cannot flock, we're going to nest at home.

Register at the event website (it's free!) and join us this weekend for 3 days of great Fedora (and CentOS!) content.

DevConf.US

DevConf is next month, and is another free, online event. DevConf targets developers - especially those developing on Linux. Event details and registration are available on the event website.

SIG reports:

Storage SIG:

Updates

Errata and Enhancements Advisories

We issued the following CEEA (CentOS Errata and Enhancements Advisories) during July:

Errata and Security Advisories

We issued the following CESA (CentOS Errata and Security Advisories) during July:

• Wed Jul 8 2020: CESA-2020:2827 Important CentOS 7 firefox Security - https://lists.centos.org/pipermail/centos-announce/2020-July/035774.html
• Wed Jul 8 2020: CESA-2020:2824 Important CentOS 6 firefox Security - https://lists.centos.org/pipermail/centos-announce/2020-July/035776.html
• Tue Jul 14 2020: CESA-2020:2894 Important CentOS 7 dbus Security - https://lists.centos.org/pipermail/centos-announce/2020-July/035777.html
• Thu Jul 30 2020: CESA-2020:3220 Important CentOS 7 kernel Security - https://lists.centos.org/pipermail/centos-announce/2020-July/035780.html
• Thu Jul 30 2020: CESA-2020:3217 Moderate CentOS 7 shim Security - https://lists.centos.org/pipermail/centos-announce/2020-July/035781.html
• Thu Jul 30 2020: CESA-2018:3140 Moderate CentOS 7 fwupdate Security - https://lists.centos.org/pipermail/centos-announce/2020-July/035782.html
• Thu Jul 30 2020: CESA-2020:3217 Moderate CentOS 7 shim-signed Security - https://lists.centos.org/pipermail/centos-announce/2020-July/035783.html
• Thu Jul 30 2020: CESA-2020:3217 Moderate CentOS 7 grub2 Security - https://lists.centos.org/pipermail/centos-announce/2020-July/035784.html

Errata and Bugfix Advisories

We issued the following CEBA (CentOS Errata and Bugfix Advisories) during July:

• Wed Jul 8 2020: CEBA-2020:2661 CentOS 7 389-ds-base BugFix - https://lists.centos.org/pipermail/centos-announce/2020-July/035775.html

Other releases

The following releases/updates also happened during July:

• Wed Jul 29 2020: CentOS Linux, CentOS Stream and the Boot Hole vulnerability - https://lists.centos.org/pipermail/centos-announce/2020-July/035778.html
• Wed Jul 29 2020: [Correction/Additions] CentOS Linux, CentOS Stream and the Boot Hole vulnerability - https://lists.centos.org/pipermail/centos-announce/2020-July/035779.html

I hope that by now you've seen the fresh new look on centos.org. If not, you should go look now.

Community member Alain Reguera Delgado, Fabian Arrotin, and a number of others proposed some style updates to the site, as well as a change to the way the site is generated and served. In addition to looking great, this also greatly simplifies the process of updating the content of the site.

If there are changes that you want to suggest for the CentOS website, the process for doing so is documented HERE, and the site content is there, too. You can send a pull request there, and we'll review them and publish once approved.

The CentOS website is *your* website, and we welcome your contributions to it.

Over the past several months we have been running a user survey to learn more about how CentOS is used. This was a very short survey - 4 questions - designed to give us a very high-level view of how the community breaks down in terms of usage. We hope that next year we can expand on this and get more detail.

There were 426 responses to the survey, with 96 of those individuals indicating that they would be open to us contacting them for more detail about their answers.

Without further introduction, here are the answers. (Note: we did an early report on results 2 months ago, and results have not changed significantly since then.)

Question 1: What we use it for.

Note: Percentages do not add up to 100% because multiple responses were allowed.

• Professional software development: 30.4% (129 responses)
• Personal/Hobby software development: 31.1% (132)
• Running services for a job/company: 62.6% (266)
• Running services for personal/home use: 62.8% (267)
• Desktop computer 27.5% (117)

Takeway: CentOS is still used primarily as the workhorse on which you run your network services, both at work and at home, but using it as a software development platform is a strong second. Lots of alternate answers entered under "Other" were almost all just rephrasings of the above categories.

Question 2: Volunteerism

This question asked where participants would be interested in volunteering. Approximately half of respondents skipped this question.

• Testing: 58.4% (132)
• Packaging: 37.6% (85)
• Design: 15.5% (35)
• Promotion: 22.1% (50)
• Event planning: 5.3% (12)

Takeaway: A lot of you want to help with testing and packaging. Why aren't you? This probably indicates that our onboarding documentation/process around the testing/qa effort needs a lot of improvement, and that we need to do a better job of pointing people to packaging opportunities like SIGs and EPEL. Those of you who want to help with design and promotion, please join us on the centos-promo mailing list!

Question 3: Size of installation

In this question we offered some very broad categories of installation sizes to get an idea of whether you're running this on a single machine or a cluster of thousands.

• 1-5 systems: 42.1% (179)
• 6-50 systems: 34.6% (147)
• 51-100 systems: 8.5% (36)
• 101-1000 systems: 10.4% (44)
• 1000+ systems: 4.5% (19)

Takeaway: While most of our fascinating stories tend to focus on the massive installations - the supercomputing clusters and massive social networks of the world - the overwhelming majority of users are smaller installations in the 1-50 systems range.

Question 4: Update frequency

With this question we were trying to get a clearer idea of how important frequent updates are - how often do you consume new packages with a dnf update

• Daily - 19.4% (82)
• Weekly - 36.3% (153)
• Monthly - 26.8% (113)
• New releases - 5.2% (22)
• Less frequently - 12.3% (52)

Takeaways: Almost all of you update at least monthly, with more than half of you updating at least once a week. We think this means that regular updates are pretty important to you. We're fascinated with the 12% of you who install a system and then leave it running with no updates.

What did we learn?

The biggest thing that I learned from this is that there's a lot of you out there that want to help and don't know what opportunities are available. We obviously need to do a much better job of telling you about those opportunities.

There's so much more that we wanted to ask, but we know that surveys are annoying, and you get a million of them every week. We do intend to do more of these in the future, but will try to do them in such a way that they are as uninavasive as possible, while giving the opportunity for those of you who want to tell us more, to do so.

Those of you who indicated that we can contact you to ask more - we're still trying to figure out what we want to ask you. Thanks for your willingness and your patience. You'll hear from us in the coming months.