After evaluating over 300 user stories from multiple stakeholders we have aligned on a decision for the Gitforge that CPE will operate for the coming years. We are opting for Gitlab for our dist git and project hosting and will continue to run pagure.io with community assistance.
A lot of comments and concerns were raised about the suitability of Github as a forge of choice. The preference from all stakeholders (Fedora, CentOS, RHEL, CPE) is that Github is not a contender and not a preference, with that in mind, we have decided to not analyse it as an option and respect the wider wishes of our stakeholders. Therefore the rest of this analysis focuses on Pagure versus Gitlab as our choice.
Looking at the user story list, we have a picture of a standard set of practices that users expect to have from a Gitforge. The basics of storing code, accessing it, merging, forking and the traditional git workflow are satisfied by both Forges under investigation.
A key requirement coming to us is security. The need for HTTP/S pushes, the need for more stringent branch control via protected and private branches is a key operating requirement of the CentOS stakeholders. The need to interface with internal and external users in a private capacity whereby embargoed content can be worked on in private is a necessary requirement.
Another key requirement is usability and accessibility. It is clear that our current forge solution is used as a mixture of ticket tracker, work planning, code repository and storage of documents and other artifacts. The barrier to usage needs to be low to attract drive by users and a strong representation was made for the need to have more accessible ways to interface with the system from a GUI to a command line client.
Developer centric needs came from multiple sources. Integrations with daily workflow, integrations within the IDE, integrations in an always ready and always on approach (SLA requirements were high) as well as the ability to use the forge as a means to improve the codebase (auto notifications of issues, interactive PR reviews etc.) and way of working by providing analytical output was also raised.
A big factor in a decision here needs to be both the immediate usability to meet stakeholder needs that includes an immovable deliverable for CentOS Stream which CPE must deliver by the end of the year.
Another major factor is the stability, availability and responsiveness of the platform chosen. While no Forge meets the full suite of requirements, the issue of stability, availability and some of the richer features that were requested are currently not available in Pagure. Gitlab provides the most feature rich experience out of the box and the recommendation of the CPE Management is to opt for Gitlab as our chosen Forge for dist-git and general project hosting. For pagure.io we want to offer it to the community to maintain. CPE would provide power and ping and the rest of it will be up to the community willing to do the work. If no-one steps up to pick the maintenance of pagure.io, it will be a candidate application to sunset. Some top level requirements which helped us arrive at this decision:
The opportunity cost to invest our finite resources into bringing Pagure up to the minimum standard that we require by the end of the year would mean feature starving both Fedora and CentOS for the next 18-24 months as we strive for the optimal standard. As a team, we spend 40% of our available resources on keeping the lights on day to day with a very small amount of that improving our technical debt situation. We are spending 30% of our team on delivering CentOS Stream. The available bandwidth for the team is not at a point that we could safely and with confidence deliver the required features to make Pagure work as our Forge of choice. It additionally would have a longer term impact with our lights on work needing to expand to move Pagure to an SLA, tilting our resourcing plan for that body of work towards 60% of our capacity. We feel this is not a responsible decision that we can make as the inward investment in a Forge is not something that we can do at the expense of planned initiatives that are on our backlog. Some of them include a better packager workflow, more investment in CI/CD to remove CPE from manual work and empower the community to do more things in our infrastructure, more observability and monitoring of our infra and services, movement of services towards the Cloud to make use of a modern tech stack and that's before we consider immovable service progression that we simply have to undertake, for example, the new Auth / AAA system.
However, we do not want to abandon Pagure and our plan going forward is thus.
We recognize how difficult a decision this is and we empathize with the emotional attachment to Pagure. It is why we want to have a mutually beneficial approach to ultimately allow Pagure to grow and flourish and allow our community members to setup and work with any Forge they wish. This ultimately allows the CPE team to focus on adding value to a greater scale of initiatives . This approach allows us to focus on value added services and initiatives that will benefit a large percentage of our communities instead of focusing on a singular foundational service which would ultimately consume our finite resourcing and limit our impact on both Communities.
-- Jim and Leigh
Over the past few weeks we've gotten questions on various forums - email, Twitter, IRC, and so on - about why there are no mentions of CentOS 8 updates, or CentOS Stream updates, on the centos-announce mailing list.
For those not familiar, centos-announce is were we tell you about security and bugfix updates that have been released. And, if you look at the archives, you'll notice that everything refers to CentOS 6 and CentOS 7.
This is not because nothing's happening with CentOS 8 or CentOS Stream. It has more to do with the tooling that generates those mailing list posts, which is all automated.
As was discussed in this blog post, many of the scripts that work fine with 6 and 7 don't work with the new 8 flow, and one of those is the script that produces the mailings that go to centos-announce. And with everything else that the team has been working on, it just hasn't (yet) been a priority to fix that.
This doesn't mean, however, that you have to fly blind. There is a service that lists all of the new packages that are flowing - what's in them, and what was changed. That service is feeds.centos.org and it provides RSS feeds of what's been updated.
A typical entry might look like:
ppp - The Point-to-Point Protocol daemon
The ppp package contains the PPP (Point-to-Point Protocol) daemon and
documentation for PPP support. The PPP protocol provides a method for
transmitting datagrams over serial point-to-point links. PPP is
usually used to dial in to an ISP (Internet Service Provider) or other
organization over a modem and phone line.
Tue, 25 Feb 2020 GMT - Jaroslav Škarvada <firstname.lastname@example.org> - 2.4.7-26 - Fixed buffer overflow in the eap_request and eap_response functions Resolves: CVE-2020-8597 Tue, 04 Dec 2018 GMT - Jaroslav Škarvada <email@example.com> - 2.4.7-25 - Fixed some issues found by coverity scan Resolves: rhbz#1602665 Tue, 20 Nov 2018 GMT - Jaroslav Škarvada <firstname.lastname@example.org> - 2.4.7-24 - Split out the network-scripts Resolves: rhbz#1608377 ...
It shows what was updated, and a few of the most recent changes to that package.
Each repo that we're pushing content to has its own RSS feed.
For those of you who don't enjoy reading raw RSS files (and, really, who does?) I've written a little bit of python for my own convenience, which you're welcome to use. This script - https://github.com/rbowen/centos-community-tools/blob/master/scripts/rss_updates.py - parses all of those RSS files (comment out the ones you don't care about) and tells you what changed since the last time you looked at it. Output is captured in text and html formats for your perusal.
CPE - Community Platform Engineering - is the engineering group within Red Hat which does a lot of the behind-the-scenes work that makes the CentOS and Fedora projects possible.
We would like to welcome you all to our first blog update on the CentOS Stream initiative. Over the course of this initiative, we will share regular updates on our plans, our progress and our deliverables. The CentOS Stream team is currently working within a Scrumban framework broken down into two week blocks. This allows the PO and team to plan and prioritise work for each block allowing stakeholders to gain updates on progress, plans and deliverables each fortnight. Each fortnight, stakeholders will review the block deliverables and provide feedback to be taken into consideration when planning the next block. Ensuring that CentOS Stream is delivered in line with changing requirements and expectations as the project build progresses.
The CentOS Stream team has made significant progress throughout February where their focus was on the cornerstone foundational build phase. Block 1 and 2 (2020-02-03 to 2020-02-21) delivered:
Block 3 deliverables were identified and are currently in progress. See below:
The month of February was a very busy month for the CPE AAA team and community contributors working on this initiative. Great progress was made in the development phase of the AAA: FAS replacement build. Sprint 2 and 3 resulted in the completion of multiple user stories which added user functionality to join groups, change email address and password, disable account, database access along with putting a mapping solution in place for users moving from the current FAS to the new FAS (potential name incoming!). We also came to the end of developing our wireframes and mapping our user experience flow. Unit tests were carried out regarding password controller and the current codebase.
We received great support from the wider CPE team as well as Patrick Uiterwijk to allow us progress with user stories by gaining permissions and merging PR’s for the integration of CentOS CI. Christian Heimes assisted us greatly with sharing his knowledge regarding FREE IPA and answered numerous questions to allow us to move forward.
Sprint 4 began on Thursday the 20th of February. This sprint will focus on development tasks which will include working on FAS Json, Free IPA, API, Fedora Messaging integration, continuous deployment to stage environment, developing a secure coding tool to ensure code adheres to best practice, as well as continuing working on user functionality user stories. Please see our github board here to view current activity.
We also received some sad news since our last update, that we are losing a team member, Rick Elrod, as he moves on to pastures new with the Ansible team. Rick provided an excellent POC for AAA which is leaving us in good shape to continue on as planned. Thanks Rick and we will hopefully still see you around as a contributor going forward. We also welcomed a new team member Leonardo (Leo) Rossetti who joined at the start of Sprint 4 and has already hit the ground running. Leo is currently working on our FAS JSON user stories.
Regarding delivery of AAA, we may look at a phased release , this current phase focus is on the development of AAA to be delivered by 3/31/20. It is looking likely that the deployment of AAA will happen in a later phase due to requiring System Admin assistance. We are likely to gain this on the completion of the Colo Move (which is our planned data center move), approximately in mid April. We are inquiring to see if deploying to staging is possible within this phase to allow for a long testing period. I will provide an update on this in our next blog. The integration of CentOS will be worked on within an additional phase following the completion of AAA centric stories for Fedora.
On a final note, I would like to commend the CPE AAA team on their collaboration and productivity throughout this initiative even in the face of unknowns, team changes, cross team dependencies and other challenges, they continued to proactively work together and find solutions to keep this initiative moving forward.
We welcome all feedback, thoughts and contributions as we progress through this project. Please feel free to comment on any issue to log your thoughts.
As you may or may not be aware, last year Red Hat made the decision to move data centers in 2020.
The lease on the current data center in Phoenix was due to expire in 2020 and Red Hat negotiated a better lease with a provider in Northern Virginia.
This data centre is home to Fedora servers.
So, what does this mean for you as a Fedora user? Very little we hope!
The Community Platform Engineering team have been working closely with Red Hat IT to plan logistics, and other 'fun stuff' to make sure this move is successful and as undisruptive to everyone as possible.
During this planning phase, we identified a need to have a minimum viable fedora offering in place during some key dates to facilitate the move, and allow for the shipment of hardware that is integral to Fedora Infrastructure without halting development - or a whole infrastructure!
Here is the link to the discussion that was sent to the public lists in case you missed it on what a Minimum Viable Fedora would look like: https://email@example.com/thread/PN6RL7XT3V7DVC7MK46H3QDEJPL5FRI6/
The CPE team will be refocusing on this problem to begin technical development of this offering so we are ready to deploy it at the appropriate time.
But for now, here is a very high level view of the Data Centre move outline, and how it will impact you:
The Community Platform Engineering Team will move in two 'waves'
Expected Effects during Move:
There will be a very limited number of builders during this time frame.
As we move through this project, our dates may change, both for the better and sometimes for the worst so please take the above dates as a *fairly good* estimate for now.
We will be including as many real-time updates on the data center move in our weekly emails to the infra and devel lists.
And while we are planning for as little disruption as possible, there may be downtime during this move so we will endeavor to get ahead of it with messaging out to you all for awareness.
We would finally like to thank you all for your understanding and most of all your patience during the key dates of April 20th - July 3rd so that we can facilitate a successful move.
Please don't hesitate to reach out to us with your questions and we will do our best to answer all the ones we know, and follow up on the ones we don't!
On 2020-02-12 the CentOS Board of Directors met and discussed several ongoing efforts across the Project.
The opening discussion was around the new work to evolve the project logo and branding identity being conducted in open channels. Overall the Directors really liked the direction the effort is going and were quite pleased with the open nature of the process. What is needed to bring a conclusion and present a final design for the Board’s approval is the completion of the open design discussion and decision process to be conducted in centos-devel.
As the Board is working on adding new Directors and improving governance and transparency, there is an open discussion around the possibility of having a face-to-face meeting of the Board in 2020. This ideally would include an additional day of interactions with other project leadership. One idea floated was to conduct this prior to the CentOS Dojo being planned at CERN in October 2020. At the time of this writing, it is unknown if this Dojo will be affected by COVID-19 related or other travel restrictions.
On another topic, in addition to the focused resources of the Community Platform Engineering (CPE) team that supports the CentOS Project in technical ways, Karsten gave a brief explanation of how the Community Architects from the Red Hat Open Source Program Office (OSPO) are in support of the project, specifically Rich Bowen on the community side, Brian Exelbierd on the business interaction side, and Karsten Wade on the strategic and visionary side.
From the Brussels Dojo, Karsten gave a report out about how he had a meeting room for one-to-one discussions with community members. These discussions were an invitation to talk about what works and doesn’t work for users and contributors around the project; an open office hours to hear out anything. It also served to help get an idea of how and why people use CentOS as a platform. This work is to help inform the CentOS 2020 open goals discussion now underway.
In support of these efforts, the Board came to the following decisions, resolutions, and agreements:
Present at the meeting:
Due to the change in the status of Red Hat Summit, we have made the decision to postpone the CentOS Dojo at Facebook to a later date. Many of our potential speakers, as well as many of our attendees, had travel plans that were dependent on attendance at Red Hat Summit, and without that event happening they are no longer able to travel.
If you have already registered for the event, we encourage you to stay registered, so that we have an easy way to contact you about event updates through Eventbrite.
We do still plan to hold an event at Facebook, but, due to the current corona virus situation, we are holding off on making any firm plans until the danger has passed.
We thank you for your patience and understanding, and hope to see you when we reschedule.
At the end of each CentOS Dojo we have an attendee survey. While these never have the response rate I'd like, they do produce interesting data that help us improve future events.
Here's the results from the survey from the Dojo in Brussels, 2020
76.5% Running services at work
41.2% Software development (professional)
41.2% Running services at home
11.8% My desktop computer
5.9% Software development (personal, hobby)
64.7% Elsewhere in Europe
23.5% Elsewhere in the world
88.2% About right
11.8% Not technical enough
Q4: I would like to see more content about:
Just one remark on this last item: We can only schedule talks that are submitted, and for this event, in particular, we had very few submissions. So take this last item as a hint of what kind of talks we'll be looking for next time.
Thank you to everyone that participated in the survey. Your feedback is very helpful!
We will be holding a CentOS Dojo at Facebook, Menlo Park, (San Francisco area) on April 24th. This is the Friday immediately before Red Hat Summit, so you can tack a few extra days on the front of your Summit trip and see how CentOS is used at Facebook.
Details of the event are available at https://wiki.centos.org/Events/Dojo/Facebook2020
The Call for Presentations is now open. We're looking for technical talks about stuff that is in and on CentOS. You can see examples of the content we have run in the past at https://www.youtube.com/thecentosproject
88% of our attendees in Brussels said that the content was about right, while 12% said it was not technical enough, if that helps set your expectations of what talks to submit.
The CFP closes on March 15th, and space is extremely limited, so don't wait. Get your talk submissions in now.