After evaluating over 300 user stories from multiple stakeholders we have aligned on a decision for the Gitforge that CPE will operate for the coming years. We are opting for Gitlab for our dist git and project hosting and will continue to run pagure.io with community assistance.

Analysis and recommendation:

A lot of comments and concerns were raised about the suitability of Github as a forge of choice. The preference from all stakeholders (Fedora, CentOS, RHEL, CPE)  is that Github is not a contender and not a preference, with that in mind, we have decided to not analyse it as an option and respect the wider wishes of our stakeholders. Therefore the rest of this analysis focuses on Pagure versus Gitlab as our choice.

Looking at the user story list, we have a picture of a standard set of practices that users expect to have from a Gitforge. The basics of storing code, accessing it, merging, forking and the traditional git workflow are satisfied by both Forges under investigation. 

A key requirement coming to us is security. The need for HTTP/S pushes, the need for more stringent branch control via protected and private branches is a key operating requirement of the CentOS stakeholders. The need to interface with internal and external users in a private capacity whereby embargoed content can be worked on in private is a necessary requirement. 

Another key requirement is usability and accessibility. It is clear that our current forge solution is used as a mixture of ticket tracker, work planning, code repository and storage of documents and other artifacts. The barrier to usage needs to be low to attract drive by users and a strong representation was made for the need to have more accessible ways to interface with the system from a GUI to a command line client.

Developer centric needs came from multiple sources. Integrations with daily workflow, integrations within the IDE, integrations in an always ready and always on approach (SLA requirements were high) as well as the ability to use the forge as a means to improve the codebase (auto notifications of issues, interactive PR reviews etc.) and way of working by providing analytical output was also raised.

A big factor in a decision here needs to be both the immediate usability to meet stakeholder needs that includes an immovable deliverable for CentOS Stream which CPE must deliver by the end of the year. 

Another major factor is the stability, availability and responsiveness of the platform chosen. While no Forge meets the full suite of requirements, the issue of stability, availability and some of the richer features that were requested are currently not available in Pagure. Gitlab provides the most feature rich experience out of the box and the recommendation of the CPE Management is to opt for Gitlab as our chosen Forge for dist-git and general project hosting. For pagure.io we want to offer it to the community to maintain. CPE would provide power and ping and the rest of it will be up to the community willing to do the work. If no-one steps up to pick the maintenance of pagure.io, it will be a candidate application to sunset. Some top level requirements which helped us arrive at this decision:

• There is a need for CentOS Stream to integrate with a kernel workflow that is an automated bot driven merging solution (merge trains). This allows for richer CI capabilities and minimizes the need for human interaction
• Gitlab provides subgroups allowing for more granular permissions for repos
• Gitlab allows for project planning capability which could make multiple trackers such as Taiga redundant, allowing for the planning and tracking to reside within the repo. It would enrich the current ticket based solution that Pagure has evolved into for some groups
• 24/7 availability in an SLA model and not hosted by the CPE team freeing up resourcing and removing the need to staff a dedicated team for a Gitforge SLA which would necessitate a follow- the- sun Ops model and a heavy investment in stability and observability of the Pagure solution.

The opportunity cost to invest our finite resources into bringing Pagure up to the minimum standard that we require by the end of the year would mean feature starving both Fedora and CentOS for the next 18-24 months as we strive for the optimal standard. As a team, we spend 40% of our available resources on keeping the lights on day to day with a very small amount of that improving our technical debt situation. We are spending 30% of our team on delivering CentOS Stream. The available bandwidth for the team is not at a point that we could safely and with confidence deliver the required features to make Pagure work as our Forge of choice. It additionally would have a longer term impact with our lights on work needing to expand to move Pagure to an SLA, tilting our resourcing plan for that body of work towards 60% of our capacity. We feel this is not a responsible decision that we can make as the inward investment in a Forge is not something that we can do at the expense of planned initiatives that are on our backlog. Some of them include a better packager workflow, more investment in CI/CD to remove CPE from manual work and empower the community to do more things in our infrastructure, more observability and monitoring of our infra and services, movement of services towards the Cloud to make use of a modern tech stack and that's before we consider immovable service progression that we simply have to undertake, for example, the new Auth / AAA system.

However, we do not want to abandon Pagure and our plan going forward is thus.

5. Offer the maintenance of pagure.io to anyone in the community interested in leading it.
6. Engage with Gitlab on the possibility of a SaaS offering so that CPE can attain key requirements of uptime, availability and throughput as well as ensuring tooling integrations (such as Fedora Messaging among others) are preserved. Legal considerations with respect to control of code will be our first discussion point with them enabling us to make a SaaS Vs self hosted decision.
7. Keep Pagure running with our oversight while we analyse a sunset timeline which will give a minimum of 12 months notice once we have a plan firmed up. We will fix blocker bugs, address critical vulnerabilities and keep the lights on in the same manner that we have committed to over the last 14 months where Pagure has not been a staffed and supported initiative.
8. Where possible, when we have to update our tooling, we will attempt to refactor our tooling to be forge agnostic, allowing our Communities the choice of storing their code on Gitlab or continuing to use pagure.io
9. Watch closely for collaboration with other Communities on Pagure and provide them with guidance and oversight to help the Pagure Community grow. We recognise that this is a growing and unique ecosystem and we genuinely want to see it succeed and will do our best to support it in that capacity. To that end we will publish the roadmap difference between Pagure and Gitlab to allow the Community to focus on feature enhancements to bridge that gap.
10. Facilitate our Communities and assist them in standing up a version of Pagure that can be driven and maintained by the Community allowing a pure Open Source principles approach for those who seek it.

We recognize how difficult a decision this is and we empathize with the emotional attachment to Pagure. It is why we want to have a mutually beneficial approach to ultimately allow Pagure to grow and flourish and allow our community members to setup and work with any Forge they wish. This ultimately allows the CPE team to focus on adding value to a greater scale of initiatives  . This approach allows us to focus on value added services and initiatives that will benefit a large percentage of our communities instead of focusing on a singular foundational service which would ultimately consume our finite resourcing and limit our impact on both Communities.

-- Jim and Leigh

Over the past few weeks we've gotten questions on various forums - email, Twitter, IRC, and so on - about why there are no mentions of CentOS 8 updates, or CentOS Stream updates, on the centos-announce mailing list.

For those not familiar, centos-announce is were we tell you about security and bugfix updates that have been released. And, if you look at the archives, you'll notice that everything refers to CentOS 6 and CentOS 7.

This is not because nothing's happening with CentOS 8 or CentOS Stream. It has more to do with the tooling that generates those mailing list posts, which is all automated.

As was discussed in this blog post, many of the scripts that work fine with 6 and 7 don't work with the new 8 flow, and one of those is the script that produces the mailings that go to centos-announce. And with everything else that the team has been working on, it just hasn't (yet) been a priority to fix that.

This doesn't mean, however, that you have to fly blind. There is a service that lists all of the new packages that are flowing - what's in them, and what was changed. That service is feeds.centos.org and it provides RSS feeds of what's been updated.

A typical entry might look like:

Thu, 27 Feb 2020 16:44:39 GMT: ppp-2.4.7-26.el8_1.x86_64

ppp - The Point-to-Point Protocol daemon

The ppp package contains the PPP (Point-to-Point Protocol) daemon and
documentation for PPP support. The PPP protocol provides a method for
transmitting datagrams over serial point-to-point links. PPP is
usually used to dial in to an ISP (Internet Service Provider) or other
organization over a modem and phone line.

Change Log:

Tue, 25 Feb 2020 GMT - Jaroslav Škarvada <jskarvad@redhat.com> - 2.4.7-26
- Fixed buffer overflow in the eap_request and eap_response functions
  Resolves: CVE-2020-8597

Tue, 04 Dec 2018 GMT - Jaroslav Škarvada <jskarvad@redhat.com> - 2.4.7-25
- Fixed some issues found by coverity scan
  Resolves: rhbz#1602665

Tue, 20 Nov 2018 GMT - Jaroslav Škarvada <jskarvad@redhat.com> - 2.4.7-24
- Split out the network-scripts
  Resolves: rhbz#1608377

...

It shows what was updated, and a few of the most recent changes to that package.

Each repo that we're pushing content to has its own RSS feed.

For those of you who don't enjoy reading raw RSS files (and, really, who does?) I've written a little bit of python for my own convenience, which you're welcome to use. This script - https://github.com/rbowen/centos-community-tools/blob/master/scripts/rss_updates.py - parses all of those RSS files (comment out the ones you don't care about) and tells you what changed since the last time you looked at it. Output is captured in text and html formats for your perusal.

CPE - Community Platform Engineering - is the engineering group within Red Hat which does a lot of the behind-the-scenes work that makes the CentOS and Fedora projects possible.

CentOS Stream Project Update 2020-02-28

We would like to welcome you all to our first blog update on the CentOS Stream initiative. Over the course of this initiative, we will share regular updates on our plans, our progress and our deliverables. The CentOS Stream team is currently working within a Scrumban framework broken down into two week blocks. This allows the PO and team to plan and prioritise work for each block allowing stakeholders to gain updates on progress, plans and deliverables each fortnight. Each fortnight, stakeholders will review the block deliverables and provide feedback to be taken into consideration when planning the next block. Ensuring that CentOS Stream is delivered in line with changing requirements and expectations as the project build progresses.

The CentOS Stream team has made significant progress throughout February where their focus was on the cornerstone foundational build phase. Block 1 and 2 (2020-02-03 to 2020-02-21) delivered:

• Nightly composes:
  • Reports are being generated internally for now until a bug fix has been resolved
• CentOS QA can consume composes and the test suite can now run against both Stream and Linux
  • This has also caught three issues that would have made it to downstream otherwise
• The module for Stream this week is PHP:7.2
  • This is ahead of 8.1 in RHEL
• Accounts in git.centos.org are created

Block 3 deliverables were identified and are currently in progress. See below:

• Push to Git process being developed
• Scaffolding up to watch brew tags
• Python:3.8 is expected to be the next module for Stream*
  • This is dependant on a bootstrapping task relating to upstream updates that come from CentOS Linux to Stream
  • Information gathering with package maintainers on package lists around debranding/rebranding for future upstream & downstream use

AAA: FAS replacement project update 2020-02-28

The month of February was a very busy month for the CPE AAA team and community contributors working on this initiative. Great progress was made in the development phase of the AAA: FAS replacement build. Sprint 2 and 3 resulted in the completion of multiple user stories which added user functionality to join groups, change email address and password, disable account, database access along with putting a mapping solution in place for users moving from the current FAS to the new FAS (potential name incoming!). We also came to the end of developing our wireframes and mapping our user experience flow. Unit tests were carried out regarding password controller and the current codebase.

We received great support from the wider CPE team as well as Patrick Uiterwijk to allow us progress with user stories by gaining permissions and merging PR’s for the integration of CentOS CI. Christian Heimes assisted us greatly with sharing his knowledge regarding FREE IPA and answered numerous questions to allow us to move forward.

Sprint 4 began on Thursday the 20th of February. This sprint will focus on development tasks which will include working on FAS Json, Free IPA, API, Fedora Messaging integration, continuous deployment to stage environment, developing a secure coding tool to ensure code adheres to best practice, as well as continuing working on user functionality user stories. Please see our github board here to view current activity.

We also received some sad news since our last update, that we are losing a team member, Rick Elrod, as he moves on to pastures new with the Ansible team. Rick provided an excellent POC for AAA which is leaving us in good shape to continue on as planned. Thanks Rick and we will hopefully still see you around as a contributor going forward. We also welcomed a new team member Leonardo (Leo) Rossetti who joined at the start of Sprint 4 and has already hit the ground running. Leo is currently working on our FAS JSON user stories.

Regarding delivery of AAA, we may look at a phased release , this current phase focus is on the development of AAA to be delivered by 3/31/20. It is looking likely that the deployment of AAA will happen in a later phase due to requiring System Admin assistance. We are likely to gain this on the completion of the Colo Move (which is our planned data center move), approximately in mid April. We are inquiring to see if deploying to staging is possible within this phase to allow for a long testing period. I will provide an update on this in our next blog. The integration of CentOS will be worked on within an additional phase following the completion of AAA centric stories for Fedora.

On a final note, I would like to commend the CPE AAA team on their collaboration and productivity throughout this initiative even in the face of unknowns, team changes, cross team dependencies and other challenges, they continued to proactively work together and find solutions to keep this initiative moving forward.

We welcome all feedback, thoughts and contributions as we progress through this project. Please feel free to comment on any issue to log your thoughts.

• For more information regarding outstanding issues, please see here.
• To view our current scrum board, please see here.

Fedora Data Centre Move Project Update 2020-02-28

Hi Everyone,

As you may or may not be aware, last year Red Hat made the decision to move data centers in 2020.

The lease on the current data center in Phoenix was due to expire in 2020 and Red Hat negotiated a better lease with a provider in Northern Virginia.

This data centre is home to Fedora servers.

So, what does this mean for you as a Fedora user? Very little we hope!

The Community Platform Engineering team have been working closely with Red Hat IT to plan logistics, and other 'fun stuff' to make sure this move is successful and as undisruptive to everyone as possible.

During this planning phase, we identified a need to have a minimum viable fedora offering in place during some key dates to facilitate the move, and allow for the shipment of hardware that is integral to Fedora Infrastructure without halting development - or a whole infrastructure!

Here is the link to the discussion that was sent to the public lists in case you missed it on what a Minimum Viable Fedora would look like: https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org/thread/PN6RL7XT3V7DVC7MK46H3QDEJPL5FRI6/

The CPE team will be refocusing on this problem to begin technical development of this offering so we are ready to deploy it at the appropriate time.

But for now, here is a very high level view of the Data Centre move outline, and how it will impact you:

The Community Platform Engineering Team will move in two 'waves'

Wave 1

• Week beginning April 13th, the CPE team will move an initial batch of servers from Phoenix, Arizona -> Northern Virginia
• This will not affect the Fedora 32 release aimed for late April.
• However, during the period of 20th March - 3rd of July we would ask you to observe an 'infra-freeze', meaning no new applications deployments and all code changes reviewed by the sysadmin team before deployment.
• During April 20th - May 20th, the Community Platform Team will be working on bringing up a minimal viable Fedora solution for continuity of important services in Fedora for development
• Between May 20th - June 1st, the CPE team will bring the new temporary offering up and redirect services to this instance while the main servers for Fedora are brought down and ready to be shipped.

Wave 2:

• Between June 1st - 15th, The Community Platform Team will ship all remaining hardware from the Phoenix Data Centre to the new one in Washington.
• Between May 20th - July 3rd, Fedora services will run on the Minimum Viable Fedora offering to facilitate a successful move of equipment across country!
• As equipment arrives in the new datacenter, the CPE team will be bringing back services on the new networks and equipment. As with all major moves there will be delays and changes we will only see as we get the services back.
• We are hopefully, and quietly confident, that we will be able to resume Business As Usual (BAU) in Fedora Infrastructure, and have Fedora up and running in early July.

Expected Effects during Move:

There will be a very limited number of builders during this time frame.

• Builds will be slower.
• Composes will be slower.
• Services like koschei will be turned off.
• Services will be 'cramped' with less resources than usual.
• Searches in koji and other tools will be slower.
• Some applications like badges, voting and calendaring may not be available at all.
• Tickets will be slower to resolve. Most CPE engineers will be focusing on rebuilding services in the new center so other requests not involved with that will be put on the backlog.

Disclaimers:

As we move through this project, our dates may change, both for the better and sometimes for the worst so please take the above dates as a *fairly good* estimate for now.

We will be including as many real-time updates on the data center move in our weekly emails to the infra and devel lists.

And while we are planning for as little disruption as possible, there may be downtime during this move so we will endeavor to get ahead of it with messaging out to you all for awareness.

We would finally like to thank you all for your understanding and most of all your patience during the key dates of April 20th - July 3rd so that we can facilitate a successful move.

Please don't hesitate to reach out to us with your questions and we will do our best to answer all the ones we know, and follow up on the ones we don't!

At the end of each CentOS Dojo we have an attendee survey. While these never have the response rate I'd like, they do produce interesting data that help us improve future events.

Here's the results from the survey from the Dojo in Brussels, 2020

Q1: I use CentOS for ...

17 responses
76.5% Running services at work
41.2% Software development (professional)
41.2% Running services at home
11.8% My desktop computer
5.9% Software development (personal, hobby)

Q2: I came to this event from:

64.7% Elsewhere in Europe
23.5% Elsewhere in the world
11.8% Brussels

Q3: Talks were ...

88.2% About right
11.8% Not technical enough

Q4: I would like to see more content about:

• Specific CentOS topics, many talks were not very CentOS related and were then duplicated at the nearby FOSDEM
• Microsoft's activities to dominate the market
• CentOS internals
• id management. Integration with active directory
• Cloud and storage
• Containers
• How to get involved and where is help required
• How to contribute to CentOS
• New features in new releases, process of preparation new releases

Just one remark on this last item: We can only schedule talks that are submitted, and for this event, in particular, we had very few submissions. So take this last item as a hint of what kind of talks we'll be looking for next time.

Thank you to everyone that participated in the survey. Your feedback is very helpful!

We will be holding a CentOS Dojo at Facebook, Menlo Park, (San Francisco area) on April 24th. This is the Friday immediately before Red Hat Summit, so you can tack a few extra days on the front of your Summit trip and see how CentOS is used at Facebook.

Details of the event are available at https://wiki.centos.org/Events/Dojo/Facebook2020

The Call for Presentations is now open. We're looking for technical talks about stuff that is in and on CentOS. You can see examples of the content we have run in the past at https://www.youtube.com/thecentosproject

88% of our attendees in Brussels said that the content was about right, while 12% said it was not technical enough, if that helps set your expectations of what talks to submit.

The CFP closes on March 15th, and space is extremely limited, so don't wait. Get your talk submissions in now.