Blog.CentOS.org

Background:
The Community Platform Engineering group is the Red Hat team combining
IT and release engineering from Fedora and CentOS. Our goal is to keep
core servers and services running and maintained, build releases, and
other strategic tasks that need more dedicated time than volunteers
can give.

See our wiki page here for more
information: https://docs.fedoraproject.org/en-US/cpe/

## General Project Updates

The CPE team have finished our Quarterly Planning for Q3, July -
September, and will begin work on the following projects starting from
Monday 5th July:
* Data Centre Move - Final Works
* CentOS Stream Phase 3
* Noggin Phase 3
* Packager Workflow Healthcare
* Fedora Messaging Schemas

Details of the above projects, and of projects currently in progress,
done and what projects are in our backlog, can be found on our taiga
board per project card:
https://tree.taiga.io/project/amoloney1-cpe-team-projects/kanban?epic=null

We also have an updated initiative timetable for briefing in new
projects to our team & key dates
here: https://docs.fedoraproject.org/en-US/cpe/time_tables/
*Note: Initiatives are large pieces of work that require a team of
people and weeks/months to complete. Please continue to open tickets
in the normal way for bugs, issues, etc.

CPE Product Owner Office Hours

#fedora-meeting-1

* Weekly on Thursdays @ 1300 UTC on #fedora-meeting-1
* Meeting log from 2020-06-25:
https://meetbot.fedoraproject.org/teams/cpe_project_owner_office_hours/cpe_project_owner_office_hours.2020-06-25-13.01.log.html
* Next Meeting: 2020-07-02

#centos-meeting

* Every second Tuesday @ 1500 UTC on #centos-meeting
* Meeting log from 2020-06-23:
https://www.centos.org/minutes/2020/June/centos-meeting.2020-06-23-15.00.log.html
* Next Meeting: 2020-07-07

Fedora Updates

Data Centre Move

* We are now officially operating under reduced Fedora services until
est 28th July to facilitate the final shipment of hardware to the new
data centre.
* Please View project card on taiga for builder - staging - remaining
app bringup dates here
https://tree.taiga.io/project/amoloney1-cpe-team-projects/us/42?kanban-status=2139945
* A list of affected services is available here
https://hackmd.io/hpYYJQRjQy-oHxUS7IonIA?view
* Details on what this move may mean for you can be found here
https://lists.fedoraproject.org/archives/list/devel-announce@lists.fedoraproject.org/thread/27U6YT73556KYW2RIFJO6J2HYMYVP22U/
* If an application is not working correctly at all, please check this
list https://hackmd.io/hpYYJQRjQy-oHxUS7IonIA?view before opening a
ticket to make sure its not listed as being moved. If it is being
moved, please wait a day or two, then try again.
* Similarly, please be patient when opening tickets for service issues
in general as we have now reached the critical point in this move and
all of our sys-admins and wider teams will be assisting in the
successful bringup of the reduced Fedora service and facilitation of
the final hardware shipment and move.
* Most recent update to devel-announce is here
https://lists.fedoraproject.org/archives/list/devel-announce@lists.fedoraproject.org/thread/5DNRZ4OUUNGSUJONQLEXXP3CKME43SCE/

AAA Replacement

* The team are finishing some work on applications including
* Codebase changes to applications
* Application maintainers can use the API to look up group and
account information
* And addressing how to add a spam blocking feature
* Please feel free to check out the team kanban board for more
information on the features the team are working on and have already
completed here https://github.com/orgs/fedora-infra/projects/6

Mbbox

* Project Dashboard here https://github.com/fedora-infra/mbbox/projects/1
* Tasks completed in the project currently
* MBBox handover to CentOS Stream
* Mbbox has been deployed to the staging environment
* Cleaning old code from the project was done
* Deployment guide - https://mbbox-operator.readthedocs.io/en/latest/
* Blog post published - check it out here
https://communityblog.fedoraproject.org/mbbox-module-building-in-a-box/

* Tasks expected to be complete by 3rd July
* MBox shared CRD - tests and polishing

Gitforge

We are still discussing technical aspects of the project and these are
tracked here:
https://gitlab.com/gitlab-org/gitlab/-/issues/217350
And are working on a date in August/September for an AMA session with
GitLab that will be run through IRC for any questions the Fedora and
CentOS communities may have to ask direct.
We will keep you up to date with the developments as and when we have
information to share and thank you for your patience.

CentOS Updates

CentOS

* CentOS Linux 8.2.2004 has been released
https://lists.centos.org/pipermail/centos-announce/2020-June/035756.html
* CentOS 7.x AMI images to land soon
* Infra Nodes are also moving
* The CentOS CPE team members are also working on a new website design
with the community, so this should be pushed soon too.

Centos Other

* Fedora CI and Noobaa tenants have been onboarded to CentOS CI infra
* The team are also looking at automation work within CI

CentOS Stream

* RealTime for Stream is built!
* The team have also finished laying out repos and are doing final
tests with the installer
* This will be pushed to the mirrors very soon so keep an eye out!

As always, feedback is welcome, and we will continue to look at ways
to improve the delivery and readability of this weekly report.

Have a great week ahead!

Aoife

Source: https://hackmd.io/8iV7PilARSG68Tqv8CzKOQ?view

Last week we had our first CPE/CentOS IRC office hours. This will become a regular thing, every second Thesday at 15:00 UTC. It's an opportunity to ask anything to the CPE - Community Platform Engineering - team about their work with the CentOS (and Fedora!) infrastructure, from the status of ongoing work, to requesting new work.

This meeting ended up mostly being me - Rich Bowen - asking Aoife Moloney questions that I've been hearing from the community over the past months, and so came across as a bit of a one-on-one interview. So I figured I'd just post it as such, for those who missed it.

We hope to see you on Tuesday, June 23, for the next edition of this meeting.

 <amoloney> #startmeeting

 <amoloney> #meetingname CPE PO Office Hours
 <centbot> The meeting name has been set to 'cpe_po_office_hours'
 <amoloney> #info About the CPE team: https://docs.fedoraproject.org/en-US/cpe/
 <amoloney> #topic Open floor and Introductions
 <amoloney> Hi Everyone!
 <amoloney> For those of you who dont know me, my name is Aoife Moloney and I am the Product Owner on the Community Platform Engineering Team
 <rbowen> Hi, amoloney. Thanks for hosting.
 <amoloney> I included a link to the fedora wiki site as I had it in my notes handy to share :)
 <amoloney> And hi Rich! You're welcome! Im happy to be here in the CentOS channel
 <rbowen> Questions, anybody?
 <rbowen> Don't be shy.
 <amoloney> I have not set an agenda or specific topic today so if there are people on this meeting who have them, please feel free to ask
 <amoloney> Even if its how to phonetically pronounce my first name  - there are a lot of vowels in there :)
 <rbowen> I think there is still some confusion about how you - CPE - work in conjunction with the CentOS infra group. Is there some way we can improve that situation?
 <amoloney> I can offer my understanding of how CPE includes CentOS infra if that gives any clarity? The CPE team as a whole is comprised of both Fedora and CentOS sys-admins, developers and release engineers. Both communities rely on our team to maitain and support both infrastructures. We also have projects that our team works on that are outside of the sustainment of the infrastructures and often the communities of the infras
 <amoloney> we support are unaware that the engineers that are doing this work in the infra are actually working across other projects too
 <amoloney> We have been working hard over the last few months to try to lessen these situations, or better yet to mitigate these situations altogether, but it has proven challenging at times!
 <amoloney> What we have done is try to diffretiate between our work request types to two categories: Large initiatives or projects that take a team and time to complete, and responsive, 'lights-on' work
 <amoloney> * What we have done is try to differentiate between our work request types to two categories: Large initiatives or projects that take a team and time to complete, and responsive, 'lights-on' work
 <rbowen> Is this tracked publicly somewhere that the community can inspect and comment on what you're prioritizing?
 <amoloney> This 'lights-on' work probably speaks to the relationship between CentOS infra and CPE as this, to me, would be support and maintenance of the distros infra
 <rbowen> I mean, other than your weekly emails to the centos-devel list, that is.
 <amoloney> Sure!  Not sure I have a CentOS infra tracker as I am not familiar with this side of the house (which is in part why Im here, to talk to the community first hand) :)
 <amoloney> but the initiatives can be found here https://tree.taiga.io/project/amoloney1-cpe-team-projects/kanban?epic=null
 <amoloney> This board and the cards are intended to be view only for drive-by viewing
 <rbowen> Oh, right, and I see that you linked to that in your weekly report email, too.
 <amoloney> This reflects what the CPE team are currently working on per quarter, what we are scoped and/or scoping, and the initiatives that are in our backlog
 <amoloney> Yep I've started to include it in there too
 <rbowen> Latest report is here: https://blog.centos.org/2020/06/cpe-weekly-2020-06-07/
 <amoloney> How are people finding those emails? Do you find them informative? Too long?
 <rbowen> I'm concerned by the fact that there has never been a response to one of those emails. I'm curious who is reading them. I read them every week, but it's kind of my job.
 <amoloney> Im glad to hear someone reads them :)
 <amoloney> No I am more than happy to provide the emails weekly, and I am assuming the format is suitable for people
 <amoloney> however, as I have learned today, 'assuming' makes an ass out of U and ME :)
 <amoloney> Hope Im ok to say that on this channel haha :)
 <amoloney> So please, if there are tips/stricks/suggestions you would like to make in relation to my weekly emails and/or how CPE can engage with CentOS Community & vice versa, please tell me
 <rbowen> Ok, so asking that question the other direction - if community members feel that your priorities should be different - that you should be working on something else - where do we make that request? How do we talk to you?
 <rbowen> I'm kind of asking a question I already know the answer to, but not everyone in the community can pick up the phone and call you or Leigh.
 <rbowen> What's the *public* API?
 <amoloney> You can drop me an email, that is the best way to contact me: amoloney@redhat.com
 <amoloney> Im happy to discuss priorities here in this meeting slot too on a weekly/bi/weekly basis if people want?
 <Evolution> there was a theory that once jira was migrated, some of this would be more open as well, yes?
 <amoloney> I have also started to send emails to the CentOS stakeholders, Rich as Community Architect and Karsten Wade as Board Secretary, leting them know whats coming down the line with CPE in the next quarter
 <Evolution> aside - I miss jira. ADO is...less ideal
 <amoloney> Oh hey Jim!
 <Evolution> Y HALLO THERE
 <amoloney> Yes and I hope it will be more than a theory :)
 <amoloney> With the launch of Red Hat One, the open  version of Jira, I would like to move all CPE initiatives from the taiga instance to a  centralized board there
 <Evolution> wait, Red Hat One?
 <Evolution> that's what they're calling it
 <amoloney> It would be an ideal scenario then that initiatives that CentOS & Fedora want worked on that require a team of people and a lot of time to complete will land on that board for me to pick up and scope with the requestor
 <Evolution> sounds like the name of Paul Cormier's plane.
 <amoloney> Yeah thats what Ive heard it being called but it could be different when released
 <amoloney> 'Launching  RED HAT ONE in T minus..' comes to mind ha
 <amoloney> but Im not really concerned what its called once it helps (selfishly) me interact with the community more directly when prioritizing initiatives, and equally gives visibility on WHY a project is either picked up and proiritized or rejected
 <amoloney> that information is important to share and Im very conscious of that
 <amoloney> Would a public instance of Jira work for the CentOS community to engage with me through?
 <rbowen> Yes, I think it would.
 <amoloney> I have added myself as a watcher to the boards issue tracker too to learn more about the types of requests you see
 <rbowen> It has already helped as a way to engage better with the board.
 <amoloney> oh thats good!
 <rbowen> But that's the difference between executive and operations, I suppose. The requests are going to be different, although many will inevitably be sent to the wrong place.
 <amoloney> In an ideal, and hopefully not too distant future, it would be wonderful to see both CentOS & Fedora project requests that are put forward by the Board and Council/FESCo land on the jira instance for me to pick up, and for operational requests like bug fixes, RAFs to land on a centralized Sustaining Team dashboard. We have some of our team working in this way, but it is currently more tilted to the Fedora side, but
 <amoloney> the end goal is to have CentOS infra requests operate in this way too and keep consistency throughout the team and to the communities we serve
 <rbowen> I think that many of the requests would come from our SIGs, rather than from the board, since they are the primary folks relying on this infrastructure.
 <amoloney> Not RAFS, RFE's :)
 <rbowen> Unfamilar with those TLAs.
 <amoloney> So was I haha - it means, and I did have to ask, its stands for Requested Feature Enhancements
 <amoloney> this is work thats minor to existing applications or services CPE maintain
 <amoloney> they could take a day or three, but would typically not need to have a team put around them
 <rbowen> Oh, this is the project/task distinction that you were making earlier. Got it.
 <amoloney> actually I have a Community Engagement Email drafted that I am waiting to add a graphic design of our initiative workflow to before sending, but Im wondering would this be better as a blog in the CentOS community?
 <rbowen> You're always welcome to post stuff to the blog, then we can promote it to all the various places our community lives.
 <amoloney> Thanks Rich, do you think it would also be equally beneficial to have our teams and the projects they work on listed on the wiki too?
 <rbowen> Having more ways that community members can participate in the work - whether that's packaging, infra, or whatever - would be great. Historically we haven't done much of that.
 <rbowen> That could be helpful, sure, so that people know who they can contact about what.
 <rbowen> Although I wouldn't want to circumvent your management process either.
 <amoloney> Im not sure people realize how much is asked of this team, from both communities, so having the work we do and the people who do that work on the wiki might help peoples awareness and might even offer opportunities to help us too when things get busy around release times
 <amoloney> and data centre moves!
 <rbowen> So, speaking of data center moves, I've heard some grumbling that the people who are most responsible for getting 8.2 out the door have been pulled onto other projects.
 <rbowen> Having a clearer understanding of how all of this interrelates is always going to help allay those kinds of concerns.
 <amoloney> So as the product owner of the entire team and all its projects, both CentOS and Fedora related, I can definitely help answer concerns on this
 <rbowen> (I suppose I should have brought that up earlier. We're almost out of time!)
 <amoloney> Fedoras hardware is moving currently across the USA. Its a well publicised  event and has been in my weekly emails for months (yet another reason to read them if you dont 🙂)
 <amoloney> I'll type fast so people know
 <amoloney> We have x2 sys-admins in Fedora and I think CentOS can emapthize here that that is not enough
 <amoloney> so, the bulk of this project has fallen on two sets of shoulders
 <amoloney> to move an entire distros stuff
 <amoloney> but we are fortunate to be in a collaborative team, CPE, and those sys-admins were able to reach ou to the CentOS sys-admin(s) to help because we are all part of the same team
 <amoloney> and thankfully, they are able to help their fellow teammates, because its the right thing to do
 <amoloney> and its very important this week to have the services we are maintaining for Fedora brought up in time to complete the move safely and on time for the F33 mass rebuild
 <rbowen> So, when people ask "Why is 8.2 taking so long" and we answer "The people working on that have other responsibilities", every single time they come back with "how can I help move that along." I, for one, would like to work towards a world where community members can help build the next release. Do we know what it would take to get there?
 <rbowen> (That's obviously a longer-term goal, but something that people ask me about pretty much every week.)
 <amoloney> Honestly, I have no idea how to get community members to build the next CentOS release, but I am more than willing to help facilitate those conversations if it helps my team feeling pressurized both internally and externally
 <amoloney> Im here to learn about the community and help find better ways of working for my team as their product owner, so if you have suggestions or ideas to make things more open and inclusive that I can help with, please reach out to me
 <rbowen> We're going to try to do this every 2 weeks, right?
 <rbowen> Perhaps next time we'll have more participation.
 <amoloney> Yes Im good with that
 <amoloney> Hopefully! :)
 <amoloney> Rich thanks for asking questions though, I hope i was able to give ok answers to them and anything I missed please let me know and I can follow up on
 <amoloney> Right were at time, thanks for having  me on here and Im looking forward to chatting to you all again!
 <amoloney> #endmeeting

Background:
The Community Platform Engineering group is the Red Hat team combining
IT and release engineering from Fedora and CentOS. Our goal is to keep
core servers and services running and maintained, build releases, and
other strategic tasks that need more dedicated time than volunteers
can give.

See our wiki page here for more
information: https://docs.fedoraproject.org/en-US/cpe/

General Project Updates

Please check out our updated initiative timetable for briefing in new
projects to our team
here: https://docs.fedoraproject.org/en-US/cpe/time_tables/
*Note: Initiatives are large pieces of work that require a team of
people and weeks/months to complete. Please continue to open tickets
in the normal way for bugs, issues, etc.

Don't forget to view our taiga board to see the projects we are
currently working on, what we have scoped and whats in our backlog
https://tree.taiga.io/project/amoloney1-cpe-team-projects/kanban?epic=null

CPE Product Owner Office Hours
* Weekly on Thursdays @ 1300 UTC on #fedora-meeting-1
* Every second Tuesday @ 1500 UTC on #centos-meeting (next meeting 23rd June)
* Meeting Logs
* Fedora: https://meetbot.fedoraproject.org/teams/cpe_product_owner_office_hours/cpe_product_owner_office_hours.2020-06-11-13.01.log.html
* CentOS: https://www.centos.org/minutes/2020/June/centos-meeting.2020-06-09-15.00.log.html

Fedora Updates

Data Centre Move

* The final Fedora hardware shipment is due to happen on Tuesday 16th June.
* We expect the shipment to arrive in the new data centre the week
beginning 22nd June and the team will begin bringing up services that
are affected by the move.
* A list of affected services is available here
https://hackmd.io/hpYYJQRjQy-oHxUS7IonIA?view
* Please read the below email sent by kfenzi if you have not already
done so: https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org/thread/E7HJULW2S76FZCAICURWXX223N5ZXXD7/
* Details on what this move may mean for you can be found here
https://lists.fedoraproject.org/archives/list/devel-announce@lists.fedoraproject.org/thread/27U6YT73556KYW2RIFJO6J2HYMYVP22U/
* If an application is not working correctly at all, please check this
list https://hackmd.io/hpYYJQRjQy-oHxUS7IonIA?view before opening a
ticket to make sure its not listed as being moved. If it is being
moved, please wait a day or two, then try again.
* Similarly, please be patient when opening tickets for service issues
in general as we have now reached the critical point in this move and
all of our sys-admins and wider teams will be assisting in the
successful bringup of the reduced Fedora service and facilitation of
the final hardware shipment and move.

AAA Replacement

* The team are working on the changing application code bases to use
the new solution
* They are also working on the CentOS account integration and the
solution will see users be able to select the contributor agreements
that are relative to their account.
* They are also testing a script to use in data migration from the
current FAS system to Noggin to allow for very little, if any, changes
to the user
* The team are also completing user documentation that includes a
how-to migration guide for maintainers and a guide to help users
understand the differences between Noggin and the current FAS option
(issue #246 on board)
* Please feel free to check out the team kanban board for more
information on the features the team are working on and have already
completed here https://github.com/orgs/fedora-infra/projects/6

Mbbox

* Project Dashboard here https://github.com/fedora-infra/mbbox/projects/1
* Tasks completed in the project currently
* MBS Backend CRD + documentation
* MBS Frontend CRD - configuration and certificates
* Draft of blog post about MBBox
* Tasks underway currently
* MBS Frontend CRD - deployment config and service
* Staging environment required
* MBox shared CRD

Gitforge

With the data centre move taking most of the teams focus this week,
discussions with gitlab have been quiet. We are still discussing
technical aspects of the project and these are tracked here:
https://gitlab.com/gitlab-org/gitlab/-/issues/217350
We will keep you up to date with the developments as and when we have
information to share and thank you for your patience.

CentOS Updates

CentOS

* OCP4 staging cluster in progress
* CentOS Linux 8.2.2004 content is in pre-release with more artifacts to come
* There was also a recent CentOS AMA on Reddit, logs are here so check
it out if you missed it
https://www.reddit.com/r/CentOS/comments/gwnvd2/centos_ama/

CentOS Stream

* The team have been working really hard to get package sources pushed
to keep caught up with RHEL
* We are also working on tooling for more push automation
* And tooling for RealTime and other Variants are coming in Stream soon

As always, feedback is welcome, and we will continue to look at ways
to improve the delivery and readability of this weekly report.

Have a great week ahead!

Aoife

Source: https://hackmd.io/8iV7PilARSG68Tqv8CzKOQ?view