Blog.CentOS.org

Hey everyone,

Back in March we published this blog where the CPE team came to a decision about a future Gitforge. The decision was made to opt for Gitlab as the Forge of choice. We can now announce that the service has been stood up successfully, with Gitlab running this as a SaaS offering on behalf of the community. We are still in the process of making configuration changes and starting to seed content. In the background the CPE team are working through their tooling and configurations to ensure when we launch that the process and experience is seamless for the community.

We expect CentOS Stream distribution repositories to be the first content that will show up in Gitlab. We plan on giving plenty of notice whenever we intend to relocate other things like SIG repos or Downstream source drops from Red Hat. Until then git.centos.org will remain up and running to support your work.

Hi Everyone,

Below is this week's CPE weekly for week ending 2020-10-25 for both
Fedora & CentOS, and if you want to visit the hackmd link
https://hackmd.io/8iV7PilARSG68Tqv8CzKOQ?view you can then use the
header bar on your left to skip to Fedora or CentOS updates that
interest you.

General Project Updates

We have a CPE Q3 Achievements blog out on the Fedora and CentOS websites
https://blog.centos.org/2020/10/cpe-q3-achievements-2020/
https://communityblog.fedoraproject.org/cpe-achievements-during-q3-2020/

Updated initative timetable can be viewed here
https://docs.fedoraproject.org/en-US/cpe/time_tables/

And below are the projects the CPE team are working on for the months
of October, November & December:
* CentOS Stream Phase 4 - Build system services
* Noggin Phase 4 - Data Migration of Fedora & CentOS Accounts, Community testing
* OSBS for aarch64 - this will begin in November
* Fedora Messaging Schemas - this work is continuing from Q3 and is
being worked on part-time

Misc

GitLab

Sent a mail to the devel lists for both Fedora & CentOS with questions
that had answers relating to the topic Accouns & Permissions. It has
been sent to devel-announce at fedoraproject.org &
centos-devel at centos.org. Here is the link to the hackmd doc I used to
write the email before copying it across to my email to send
https://hackmd.io/1pjX1cVnTjekOLVowj5UiQ?view

Project Updates

*The below updates are pulled directly from our CPE team call we have
every week.*

CentOS Updates

CentOS

* CentOS 7.9.2009 pkgs pushed to CR repo, next step is to install tree
and artifacts such as iso & cloud images.
* Working a lot with the AAA/Noggin team on CentOS account integration
and making good progress

CentOS Stream

* There are now cloud image updates available for CentOS Stream
* There are some documentation updates here on unshipped packages &
Stream feel free to read up!
https://wiki.centos.org/FAQ/CentOS8/UnshippedPackages

Fedora

General

* F33 will be released next week!!
* The team got an F33 nightly compose finished too!
* There is a dashboard available to view the performance of Anitya
https://monitor-dashboard-web-monitor-dashboard.app.os.fedoraproject.org/d/8Zi9LU5Mz/anitya?orgId=1
* And there is capability to do the same for the packager workflow
pipeline which we are still working on

Staging Environment

* Build system nearly done - waiting on a firewall change

Noggin/AAA

* New estimated deployment date for Noggin is 29th January 2021.
* The team are working on completing a full staging environment to
deploy Noggin in right now and will hope to have this in place in the
next few weeks
* We also have some members of CentOS working in this team now to help
with the work required for the migration of the CentOS accounts and
data to Noggin
* The teams kanban board where they track their work can be found here
https://github.com/orgs/fedora-infra/projects/6
* And we have a project tracker available to be viewed here
https://github.com/fedora-infra/aaa-tracker

Fedora Messaging Schemas

* This project is worked on on a part time basis as we are
prioritizing completing Noggin first before fully committing to its
completion
* There is a list of applications that require messaging schemas can
be found here https://hackmd.io/@nilsph/H1i8CAbkP/edit
* There is a readme which contains documentation on messaging schemas,
a cookie-cutter template to create the schema and a definition of Done
for writing a schemas
https://github.com/fedora-infra/fedora-messaging-schemas-issues
* The board they are working from can be viewed here
https://github.com/orgs/fedora-infra/projects/7

Team Info

CPE Product Owner Office Hours

IRC office hours are now once per month.Below are the logs from the
most recent meetings and dates for the next ones.

#fedora-meeting-1

* Next Meeting: 2020-11-12 @ 1300 UTC on #fedora-meeting-1 (On Freenode IRC)

#centos-meeting

* Next Meeting: 2020-11-10 @ 1500 UTC on #centos-meeting (On Freenode IRC)

Background:

The Community Platform Engineering group, or CPE for short, is the Red
Hat team combining IT and release engineering from Fedora and CentOS.
Our goal is to keep core servers and services running and maintained,
build releases, and other strategic tasks that need more dedicated
time than volunteers can give.

See our wiki page here for more
information: https://docs.fedoraproject.org/en-US/cpe/

As always, feedback is welcome, and we will continue to look at ways
to improve the delivery and readability of this weekly report.

Have a great week!

Aoife

Source: https://hackmd.io/8iV7PilARSG68Tqv8CzKOQ?view

Hi there,

I'm Aoife Moloney. You may remember me from such communications as the CPE office hours, Data Centre - what it means for you, and The Future of Communishift.

Over the last three months, the Community Platform Engineering team (or CPE for short as it's long to keep typing) have been working on a few projects, and generally surviving 2020 like everyone else. But we made it, and so did our projects! Mostly… 🙂

Over the last three months we worked on:

• The Great Fedora Data Centre Move of 2020
• Noggin
• CentOS Stream
• Packager Workflow Healthcare (Always check with your maintainer before taking this workflow. Side effects may be, but not limited to, frustration, tears, elation at successful builds)
• Fedora-Messaging Schemas

We also had our long standing (and long suffering) ‘sustaining team’ on the front lines who are daily maintaining and running both the Fedora and CentOS infrastructures and responding to issues, bugs, etc. And doing a damn fine job too.

And we attended and participated in a few conferences too, namely Nest with Fedora & DevConf US.

So, what did we as a team overall achieve in these last few months?

CPE Infra & Releng Team 

This team was led by Pingou, and its members in Q3 were Mark O’Brien, Michal Konecky, Fabian Arrotin, David Kirwan, Kevin Fenzi, Vipul Siddharth, Stephen John Smoogen & Tomas Hckra.

This team is a sub team of CPE and focuses on lights on work in both the Fedora and CentOS infrastructures. We will always have some of our team members working in this way each quarter as it is good to have a break from scheduled project workloads and take a foray into the (sometimes) chaotic world of infrastructure maintenance, aka FIRE!!! 🙂

What they did: 

• Changed their name. May we present: CPE Infra & Releng Team - oooooh, aaaaah!
• Vipul & David worked with Fabian in the CentOS infra and did something with openshift clusters & migrated the kojihub for https://cbs.centos.org to a new infra
• Kevin and Smooge moved all of the Fedora infrastructure. 117 servers.  Let that number sink in.
• Pingou & Michal did a ton of babysitting toddlers 🙂 They moved a lot of scripts over and things are working well
• Tomas helped bootstrap F33 - oh yeah!
• And Mark had (not biologically,  but in sentiment) a baby! And became an admin for some of the Fedora Infrastructure.

Why its good:

• The name change represents what this team works on and is easy to understand instantly. Plus, naming is hard so we wanted to keep it simple 🙂
• Helped release F33 beta
• We doubled down on toddlers allowing to build more automation around the infrastructure
• Assisted with the fedora datacentre move for minimal disruption to the fedorans day to day lives
• Helped keep CentOS CI operational - and then helped put out the flames when it caught fire 🙂
• Over 500 tickets across both Fedora & CentOS infra + releng resolved by this team - that is some seriously good firefighting!

Fedora Data Centre Move

This dynamic duo was Kevin Fenzi and Stephen Smoogen, with supporting cast members from both CPE and the community along the way. The goal of this project was to successfully move a (large) number of the Fedora infrastructure hardware from one datacentre to the other without too much chaos. And considering the world wide pandemic that happened right at the beginning, they did a pretty fine job succeeding. Some additional services are still being added to the infrastructure in its new home in IAD, so if you notice a few still missing, we are getting to them slowly but surely and thank you again for your patience and understanding during these last few months!

What they did: 

• Moved a ton of servers across the country of the United States
• Kept critical services in Fedora Infrastructure alive during the move
• Worked an uncountable amount of hours!

Why its good:

• We got some new hardware!
• The team carried out some resilience testing in the new data centre which means more reliability for the infrastructure should bad things happen 
• Updated records and warranties were a passive benefit of this move too

Noggin

This team was led by Aurelien Bompard, and its members in Q3 were Ryan Lerch, Nils Philippsen & James Richardson. The goal of this project is to replace the current FAS system with a newer one and migrate the CentOS accounts to the one FAS instance (Noggin), which will mean our team has one authentication system to maintain for two infrastructures long term. This team has been working to a November 2020 deadline, but unfortunately during Q3 the team faced a number of challenges such as a delayed staging environment to test in due to the data centre move, then when we got it, realized their plugin they spent time developing was not going to work long term and now have to redo a bit of work in Q4. There were also a lot of holidays and personal events for the team in Q3 because everyone is human and entitled to a life 🙂 They have re-scoped their work for Q4 to make sure what's delivered is sustainable and reliable long term, more people have joined the team including some sys-admin for support along the way, and are now looking at delivering Noggin in full by the end of January 2021. 

What they did: 

• A lot of ipsilon investigation
• Added a spam curtailer service to Noggin
• Added an agreements section for users to select their user preferences 
• Deployed Noggin to staging but found out the way they did it wont be good for the project long term
• Had a little cry about developing a plugin unnecessarily, hugged it out and then re-planned dates and the work we need to do in Quarter 4 to be able to deliver a better, more reliable and robust service in January 2021. Queue Noggin’  - Rise of the Phoenix Project

Why its good:

• We knew where we went wrong, learned a lot both technically and as a project team for it, and were able to call the mistakes out and get the support we need to get the project back on track. Just a little bit later than we wanted.
• We still created a solution that will meet the needs of both the CentOS and Fedora community users, and once we have the correct configurations in place and are ready to be tested we look forward to your feedback!

CentOS Stream:

This team was led by Brian Stinson, and its members in Q3 were Johnny Hughes, Carl George, Mohan Boddu, Leonardo Rosetti, James Antill & Siteshwar Vashisht

What they did: 

• A lot of darn package & module building
• Light hearted threats to their PO to teach her how to convert a CentOS Linux distro to CentOS Stream using the new release package - which she did! 🙂
• Kept CentOS Stream compose up to date with RHEL nightlies
• Launched the centos-stream-release package - Big deal. Like, huge.

Why its good:

• CentOS Stream is continuing to stand on its own and becoming a more robust distro
• There's lots more content in Stream for its users
• Users can now swap from CentOS Linux to Stream easily

Packager Workflow Healthcare:

This team was led by Will Woods and its team members were Adam Saleh and Stephen Coady with Pingou in a part-time consulting/reviewing role. The team took a look into the Fedora packager workflow and tried to identify weaker points in the chain, and spot times that are more prone to downtimes.  They are finalizing a report of their findings to send to the community lists with hopefully a ‘next steps’ section that they feel will help reduce the issues packagers face sometimes in Fedora. Its a work in progress, but to have some data to read and understand is a great launching point.

What they did:

• Refined the monitor-gating script that monitors the packager pipeline to enhance its performance
• Picked a certain date range and got a database dump to pull metrics from into graphana to chart uptimes of applications within the pipeline
• Created a diagram of the pipeline to help understand how packages flow through the fedora infra

Why its good:

• The diagram of the packager workflow process is a great resource for both packagers and new contributors of the fedora community to refer to and help understand how things work
• The team also have some recommendations they are working through with management and the wider CPE team to identify possible next steps and how we can improve the packager experience long term by adopting better monitoring.

Fedora-Messaging Schemas:

This project was also being worked on by the Noggin team part-time, so Aurelien Bompard, Nils Philippsen & Ryan Lerch. We needed to pause this work around the start of September and we hope to be able to return to it over the next quarter - October, November & December.

The guys have a github board here with a cookie-cutter schema available and a list of apps they were working on, so if you want to help out on this one, please feel free to visit the board and grab a card! 🙂 

What they did: 

• Created a board to track the work being done and whats left to do https://github.com/orgs/fedora-infra/projects/7
• Created a template schema 
• Created a list of applications that require a schema update
• Added some schemas to applications that need them

Why its good:

• This will help us progress the retirement of fed-msg in 2021
• It will also give applications, and application maintainers, access to new fedora messaging schemas for more faster & reliable notifications.

And that, my dear friends, is Quarter 3 for CPE.

Take care all, and see you around IRC! 🙂

Aoife

Hi Everyone,

Below is this week's CPE weekly for week ending 2020-10-17.
Im gone a little bi-weekly lately with this report. This has been more
circumstantial with our quarter 3 projects ending and launching
quarter 4 work, and will get back to a weekly report now that Q4 is
underway.
So, the updates for both Fedora & CentOS are below, and if you want to
visit the hackmd link https://hackmd.io/8iV7PilARSG68Tqv8CzKOQ?view
you can then use the header bar on your left to skip to Fedora or
CentOS updates that interest you.

General Project Updates

We have a CPE Q3 Achievements blog out on the Fedora and CentOS websites
https://blog.centos.org/2020/10/cpe-q3-achievements-2020/
https://communityblog.fedoraproject.org/cpe-achievements-during-q3-2020/

And below are the projects the CPE team are working on for the months
of October, November & December:
* CentOS Stream Phase 4 - Build system services
* Noggin Phase 4 - Data Migration of Fedora & CentOS Accounts, Community testing
* OSBS for aarch64 - this will begin in November
* Fedora Messaging Schemas - this work is continuing from Q3 and is
being worked on part-time

Misc

GitLab

There is now a blog post out on the Fedora community blog on the AMA,
you can read it here
https://communityblog.fedoraproject.org/gitlab-ama-follow-up/
Im going to be sending one mail per week to the CentOS and Fedora
devel lists on Fridays, dedicated to one topic.
The first topic will be Permission and Access in GitLab. I will pull
the questions and answers from the hackmd doc into an email body to
try to facilitate dedicated discussion around this topic. I would like
to just set expectations now that I don't consider myself to be
technical enough to weigh in on the discussion, but I am looking
forward to reading and learning from the conversation.

Project Updates

*The below updates are pulled directly from our CPE team call we have every week.*

Fedora

General

* Go NoGo meeting was on 15th Oct. F33 was No GO and new date is moved
to 2020-10-27

Staging Environment

* Working on getting OpenQA moved over
* Adding boxes to CentOS to test IPA

Noggin

* After the team reviewed the work still left to do to be able to
deploy a robust and secure solution, we discovered that there is more
development work to do and we are now working towards a release date
of January 29th 2021.
* Accounts & data will remain secure in the current solution until we
can cut over, we have established a monitoring plan to derisk this
extended time the team needs to complete their work.
* The team are working on completing a full staging environment to
deploy Noggin in right now and will hope to have this in place in the
next few weeks
* We also have some members of CentOS working in this team now to help
with the work required for the migration of the CentOS accounts and
data to Noggin
* The teams kanban board where they track their work can be found here
https://github.com/orgs/fedora-infra/projects/6

Fedora Messaging Schemas

* This project is worked on on a part time basis as we are
prioritizing completing Noggin first before fully committing to its
completion
* There is a list of applications that require messaging schemas can
be found here https://hackmd.io/@nilsph/H1i8CAbkP/edit
* There is a readme which contains documentation on messaging schemas,
a cookie-cutter template to create the schema and a definition of Done
for writing a schemas
https://github.com/fedora-infra/fedora-messaging-schemas-issues
* The board they are working from can be viewed here
https://github.com/orgs/fedora-infra/projects/7

CentOS Updates

CentOS

* New CI admin added - mobrien, welcome!
* Plumbing for duffy being worked on currently and will be deploying
to staging in the coming weeks

CentOS Stream

* Looking at ODCS services in the build system
* Also working on deploying mbbox operator to the system too
* centos-stream-release package is also out, go check it out!

Team Info

CPE Product Owner Office Hours

IRC office hours are now once per month.

#centos-meeting

* Next Meeting: 2020-11-10 @ 1500 UTC on #centos-meeting

Background:

The Community Platform Engineering group, or CPE for short, is the Red
Hat team combining IT and release engineering from Fedora and CentOS.
Our goal is to keep core servers and services running and maintained,
build releases, and other strategic tasks that need more dedicated
time than volunteers can give.

See our wiki page here for more
information: https://docs.fedoraproject.org/en-US/cpe/

As always, feedback is welcome, and we will continue to look at ways
to improve the delivery and readability of this weekly report.

Have a great week!

Aoife

Source: https://hackmd.io/8iV7PilARSG68Tqv8CzKOQ?view

Hi everyone,

Welcome to September! Below are the most recent Community Platform
Engineering project updates, and if you want to know more about our
team, see our wiki page here for more information on who our team is:
https://docs.fedoraproject.org/en-US/cpe/

Here are some upcoming IRC meetings:

CPE Product Owner Office Hours

#fedora-meeting-1

* Weekly on Thursdays @ 1300 UTC on #fedora-meeting-1
* Next Meeting: 2020-09-03

#centos-meeting

* Every second Tuesday @ 1500 UTC on #centos-meeting
* Next Meeting: 2020-09-01

GitLab AMA Session

* September 10th @ 1330 UTC on #fedora-meeting-1

Below are the project & community updates this week:

GitLab

There will be an IRC based AMA session with GitLab on Thursday 10th
September @ 1330 UTC in place of the CPE PO office hours.
We are still talking to GitLab but we are deliberately taking our time
to make sure all of the technical blockers can be met and the move
will be worth it in the end.
There is very little to no updates in the tracker, but I will include
it nonetheless https://gitlab.com/gitlab-org/gitlab/-/issues/217350
I will also be sending a separate email on details of the AMA session
later this week, such as how to submit questions in advance so there
is content ready on the day.

Note: Since I'm posting this to the blog after the meeting, you may wish to read the

CentOS Updates

CentOS

* Updated ocp.stg to OCP v4.5.6.
* Added a number of users to the jump.ci host.
* Adding monitoring/alerting for NFS slowness to the ocp cluster.

CentOS Stream

* Module push tweaks.
* Exploring how to enable fedora messaging in Stream
* Reviewing documentation on contributor policies before publishing
them later this quarter.

Fedora Updates

Staging Environment

* Services will begin to be deployed this week
* Please be patient as some services will inevitably not work due to
networking errors that the team don't know until they deploy
* Thank you again for your patience and understanding during these
last few months!

AAA Replacement

* Deployment to staging for testing is delayed due to missing firewalls in IAD2
* This has just recently been unblocked so the team will begin some
deployment and testing of Noggin this week
* Wider community testing will be available, estimated next week
* In the meantime. Please feel free to check out the team kanban board
for more information on the features the team are working on and have
already completed here https://github.com/orgs/fedora-infra/projects/6

Fedora Messaging Schemas

* List of applications that require messaging schemas can be found
here https://hackmd.io/@nilsph/H1i8CAbkP/edit
* There is a readme which contains documentation on messaging schemas,
a cookie-cutter template to create the schema and a definition of Done
for writing a schemas
https://github.com/fedora-infra/fedora-messaging-schemas-issues
* The board they are working from can be viewed here
https://github.com/orgs/fedora-infra/projects/7

Packager Workflow Healthcare

* The team have been reviewing data on how packages are built in the
fedora infrastructure for the last 8 weeks and have gathered enough
information to create a report on their findings.
* This report is currently in draft format, and is going to be
reviewed by the team first, and then sent to the devel and infra lists
in the next 2 weeks est.
* The teams work is being tracked here
https://teams.fedoraproject.org/project/cpe-cicd/kanban

Here is a reminder of what our team has committed to work on in this
quarter of the year:

The CPE team are working on the following projects for Quarter 3,
which is the months of July, August & September:
* Data Centre Move - Final Works
* CentOS Stream Phase 3
* Noggin Phase 3
* Packager Workflow Healthcare
* Fedora Messaging Schemas

Details of the above projects, and of projects currently in progress,
done and what projects are in our backlog, can be found on our taiga
board per project card:
https://tree.taiga.io/project/amoloney1-cpe-team-projects/kanban?epic=null

We also have an updated initiative timetable for briefing in new
projects to our team & key dates
here: https://docs.fedoraproject.org/en-US/cpe/time_tables/
*Note: Initiatives are large pieces of work that require a team of
people and weeks/months to complete. Please continue to open tickets
in the normal way for bugs, issues, etc.

Background:

The Community Platform Engineering group, or CPE for short, is the Red
Hat team combining IT and release engineering from Fedora and CentOS.
Our goal is to keep core servers and services running and maintained,
build releases, and other strategic tasks that need more dedicated
time than volunteers can give.

As always, feedback is welcome, and we will continue to look at ways
to improve the delivery and readability of this weekly report.

Have a great week!

Aoife

Source: https://hackmd.io/8iV7PilARSG68Tqv8CzKOQ?view

As any open source project grows and matures, the people who have always done all the things can’t do everything any more. CentOS is at that point (really, we have been for a long time) and we’ve been struggling with those kind of growing pains for some years.

Recently, we made a small change in process, so that SIG leads can manage their own SIG membership and permissions associated with SIG membership. This removed some of the load on the people who have traditionally handled this on behalf of the SIGs. This change was made possible by updates in the authentication tooling, and has the Board looking at other areas where we can move more operational things out of the Board, which can then focus on governance and executive tasks.

Another area where this need for maturity has been clear for a while is our infrastructure, where a small number of people handle all of the infrastructure tasks. As our infrastructure grows (in both size and complexity) this has put us in situations where work was blocked because those people didn’t have time to get tasks done. And, sometimes, people like to take a holiday.

This week, the Board of Directors approved a plan to move infrastructure oversight into a SIG structure, so that different parts of the infra can be delegated to a larger group of people. We have already been in partnership with CPE (Red Hat’s Community Platform Engineering team) to do some of these tasks, and this will give greater liberty to spread the load around even further, but also greater transparency around what it takes to be granted that access.

We are still in the process of figuring out what is in scope, and collecting a community of interested volunteers willing to do the work. The CentOS infrastructure is very broad, including build machines, the mirror network, systems with donated hosting, and so on, and we want to be sure that we do this in a way that doesn’t interfere with the work that’s already being done by so many people, while at the same time working to identify more places where volunteers can make a difference.

To be clear, giving administrative access to any systems still requires a great deal of trust and assurance of competence. We’re not giving the keys away to anyone who asks. But people who have demonstrated that competence and trustworthiness will be able to do some of the tasks that have, thus far, been handled by 2 or 3 people. We’ll be working to develop and publish some clear guidelines around who can be trusted with this access, and how to demonstrate that competence.

The exact details of how this SIG will operate are still being worked out. But the Board has approved Aoife Moloney as the first Infrastructure SIG chair, since she has already been sending infrastructure updates to the centos-devel mailing list roughly weekly for some months now, and is a logical person to take on this organizational responsibility. So, thank you, and welcome, to Aoife, for her willingness to tackle this job.
We welcome participation and comment from anyone with the skills or interest to be helpful, particularly if you are already administering any of the infrastructure that the CentOS project relies on.