At the beginning of 2020 (seems so long ago now!), Red Hat announced that it was making a conversion utility available to the public to convert RHEL-like systems in place to RHEL. What this means for customers is that they will be able to convert CentoS/Oracle Linux 6 & 7 to RHEL 6 or 7 on 64bit Intel systems. Reasons for this might include obtaining vendor support, taking advantage of supported major version upgrades, and consolidating ‘nix flavours.

A client approached us post security audit, and asked us if we could assist with one of their requirements: “We need to ensure our Linux fleet is supported by a vendor. We have a large fleet of CentOS servers that we need migrated to Red Hat Enterprise Linux”.

In this article, I will specifically talk about using Ansible & Red Hat Satellite to convert CentOS 7 in place to RHEL 7.

Customer Requirements

After some initial requirements gathering, the customer requires the following:

• The ability to revert to a VMWare snapshot
• An audit trail with visibility to stakeholders
• Notifications of puppet run errors pre- and post-conversion
• Change control for production systems
  

Technical Requirements

The following requirements and pre-requisites were met prior to undertaking conversion:

• CentOS patched to version 7.7 with no outstanding reboots
• CentOS servers joined to Red Hat Satellite (optional)
• Available Red Hat subscriptions on each hypervisor running a target vm
• An Activation Key using the Library Environment in Satellite using a Content View containing:
  • Red Hat Enterprise Linux 7 Server - Optional RPMs x86_64 7.7
  • Red Hat Enterprise Linux 7 Server RPMs x86_64 7.7
  • Red Hat Satellite Tools 6.6 for RHEL 7 Server RPMs x86_64
    
  
  

  FREE consultation: Understand how to build a modern cloud-based infrastructure with containers using the Red Hat OpenShift platform

  
  

Automating it all!

Ansible was the obvious choice to automate server snapshots, send MS Teams notifications, convert to RHEL, and re-join Satellite post-conversion.

But now, we need to automate to run on over 80 servers while meeting customer requirements and tailoring it to the environment.

I decided to split this into 4 re-usable ansible roles and use tags in order to include or exclude them. The playbook will run in this order:

1. Snapshot server
2. Send MS Teams notification after snapshots have completed
3. Run puppet agent and report failure to MS Teams before Migration
4. Convert 2 RHEL
5. Send MS Teams notification after conversion completed
6. Re-join Satellite and reinstall katello, gofer, qpid, insights and update all
7. Send Teams notification after post conversion completed
8. Run puppet agent and report failure to MS Teams

Automating CentOS to RHEL conversion

The epel repo is required in order to install the convert2rhel utility, in addition several rpm’s are required including the katello-ca-consumer-latest.noarch.rpm. These can be obtained from the Satellite server itself, the blog mentioned earlier has finer details around how to obtain these files. The activation key that was created in the Technical Requirements stage contains the required repositories in order to successfully replace all of the CentOS signed packages with the Red Hat signed equivalents.

The convert2rhel tool works as follows:
# convert2rhel -k convert2rhel_ak -a -v Server -o Organisation --no-rpm-va -y --enablerepo "*"

• ‘-k convert2rhel_ak’ refers to the activation key created within Satellite that contains the Content View required by the utility.
  
• ‘-a’ with auto-attach an available subscription.
• ‘-v Server’ specifies that the server is of Server variant.
• ‘-o Organisation’ refers to the Satellite organisation.
• ‘--no-rpm-va’ will not verify rpm’s on the server prior to conversion. When enabled this has been known to take up to an extra hour without providing much benefit.
• ‘-y’ is used to automatically proceed with the conversion (useful for automation).
• ‘--enablerepo "*"’ enables all repo’s available in the content view. This will ensure that the majority of packages are able to be re-installed.
  
  

The following Ansible role will do the following:

1. Copy a consistent yum.conf, making sure that the distropkg option isn’t set.
2. Stop and disable puppet to ensure that yum repos are not automatically created during a puppet run.
3. Remove all yum repos.
4. Copy the epel repo.
5. Install the convert2rhel utility.
6. Copy the subscription manager and katello-ca-consumer rpm’s to the convert2rhel/subscription-manager directory.
7. Copy and import the GPG key.
8. Copy the redhat-release-server rpm to the convert2rhel/redhat-release/Server/ directory.
9. Remove previously installed subscription-manager rpms.
10. Run convert2rhel.
11. Log results of the conversion to a variable for use in the main playbook.
12. Run grub2-mkconfig when the server is using EFI firmware. Otherwise it will fail to boot.
13. Show the results of the conversion.
14. Start and enable puppet.
15. Reboot the server.

---

- name: Copy yum.conf

copy:

   src: yum.conf

   dest: /etc/yum.conf

   owner: root

   group: root

   mode: 0644

- name: find files

find:

   path: /etc/yum.repos.d/

   recurse: no

   file_type: file

register: repos

- name: Stop and disable puppet

systemd:

   name: "{{item}}"

   state: stopped

   enabled: false

with_items:

   - puppet

ignore_errors: yes

- name: Remove all repos

file:

   path: "{{item.path}}"

   state: absent

with_items:

   - ""

- name: Copy epel repo

copy:

   src: epel.repo

   dest: /etc/yum.repos.d/epel.repo

- name: Install convert2rhel

yum:

   name: convert2rhel

   state: present

- name: Copy subscription manager rpms

copy:

   dest: /usr/share/convert2rhel/subscription-manager/

   src: "{{item}}"

with_items:

   - subscription-manager-1.24.13-1.el7.x86_64.rpm

   - subscription-manager-rhsm-1.24.13-1.el7.x86_64.rpm

   - subscription-manager-rhsm-certificates-1.24.13-1.el7.x86_64.rpm

   - katello-ca-consumer-latest.noarch.rpm

- name: Copy GPG Key

copy:

   dest: /tmp/

   src: "{{item}}"

with_items:

   - RPM-GPG-KEY-redhat-release

- name: Import GPG key

command: rpm --import /tmp/RPM-GPG-KEY-redhat-release

- name: Copy subscription manager rpms

copy:

   dest: /usr/share/convert2rhel/redhat-release/Server/

   src: "{{item}}"

with_items:

   - redhat-release-server-7.7-10.el7.x86_64.rpm

- name: Remove previously installed subscription-manager rpms

yum:

   name: "{{item}}"

   state: absent

with_items:

   - subscription-manager

   - subscription-manager-rhsm

   - subscription-manager-rhsm-certificates

- name: Covert to RHEL using Satellite

command: convert2rhel -k convert2rhel_ak -a -v Server -o Organisation --no-rpm-va -y --enablerepo "*"

register: conv_result

failed_when: conv_result.rc == 1

- name: Build conversion result value

set_fact:

   conv_teams: ""

- name: Check if /sys/firmware/efi dir exists

stat:

   path=/sys/firmware/efi

register: efi

- name: run grub2-mkconfig when uefi

command: grub2-mkconfig -o /boot/efi/EFI/redhat/grub.cfg

when: efi.stat.exists

- name:

debug:

   var: conv_result.stdout_lines

- name: Start and enable puppet

systemd:

   name: "{{item}}"

   state: stopped

   enabled: true

with_items:

   - puppet

ignore_errors: yes

- name: Reboot server

reboot:

register: reboot_result

tags:

   - reboot

Where all the magic happens (main playbook)

Extensive use of blocks and tags ensures that logical tasks can remain separated and re-useable.

The following playbook was used to automate and drive the entire conversion process:

---

- name: Migrate CentOS to RHEL

hosts: centos_migration_round_1

become: true

gather_facts: true

vars_files:

   - vars/vault.yml

vars:

   patching_desc: CentOS to RHEL Migration

   vcenter_datacenter: Turino

   vcenter_hostname: vc1.example.org

   teams_webhook_url: "https://outlook.office.com/webhook/738/IncomingWebhook/738”

tasks:

   - name: Snapshot server

     block:

       - name: Set Datacenter fact

         set_fact:

           vcenter_datacenter: Alberta

           vcenter_hostname: vc2.example.org

          when: "'alb' in inventory_hostname"

       - include_role:

           name: vmware_snapshot

         vars:

           vcenter_username: svcSnapShots

           vcenter_password: ""

           vcenter_snapshot_name: ""

           vcenter_snapshot_description: " from Ansible"

       - name:

         run_once: true

         set_fact:

           facts_list: ""

         loop: ""

     become: false

     delegate_to: localhost

     tags:

       - snapshot

   - name: Send Teams notification after snapshots have completed

     block:

       - include_role:

           name: teams

         vars:

           webhook_url: ""

           title: CentOS to RHEL Migration notification from Ansible

           text: "CentOS to RHEL VMWare snapshot summary"

           color: E81123

           section_title: VMWare snapshot summary

           section_text: "Snapshots completed with the following results:"

           section_facts: ""

     become: false

     delegate_to: localhost

     run_once: true

     tags:

       - snapshot

       - O365

   - name: Run puppet agent and report failure to Teams before Migration

     block:

       - name: Run puppet agent

         puppet:

         register: puppetrc

         ignore_errors: True

       - name: Show puppet results

         debug:

            var: puppetrc

       - include_role:

           name: teams

         vars:

           webhook_url: ""

           title: Pre migration - Puppet failure notification from Ansible

           text: "Pre migration - CentOS to RHEL Conversion summary"

           color: E81123

           section_title: " - Puppet run failed with the following results:"

           section_text: ""

         when:

           - puppetrc.failed

    tags:

       - puppet1

   - name: Convert 2 RHEL

     block:

       - include_role:

           name: convert2rhel

       - name:

         run_once: true

         set_fact:

           facts_list: ""

         loop: ""

     tags:

       - convert

   - name: Send Teams notification after conversion completed

     block:

       - include_role:

           name: teams

         vars:

           webhook_url: ""

           title: CentOS to RHEL Migration notification from Ansible

           text: "CentOS to RHEL Conversion summary"

           color: E81123

           section_title: Conversion results summary

           section_text: "Conversion completed with the following results:"

           section_facts: ""

     become: false

     delegate_to: localhost

     run_once: true

     tags:

       - convert

       - O365

   - name: Re-join Satellite and reinstall katello, gofer, qpid, insights and update all

     block:

       - include_role:

           name: postconvert2rhel

       - name:

         run_once: true

         set_fact:

           facts_list: ""

         loop: ""

     tags:

       - rejoin

   - name: Send Teams notification after post conversion completed

     block:

       - include_role:

           name: teams

         vars:

           webhook_url: ""

           title: CentOS to RHEL Migration notification from Ansible

           text: "CentOS to RHEL Post Conversion summary"

           color: E81123

           section_title: Post conversion activity summary

           section_text: "Conversion completed with the following results:"

           section_facts: ""

     become: false

     delegate_to: localhost

     run_once: true

     tags:

       - rejoin

       - O365

   - name: Run puppet agent and report failure to Teams

     block:

       - pause:

           seconds: 45

       - name: Run puppet agent

         puppet:

         register: puppetrc

         ignore_errors: True

       - name: Show puppet results

         debug:

           var: puppetrc

       - include_role:

           name: teams

         vars:

           webhook_url: ""

           title: Puppet failure notification from Ansible

           text: "CentOS to RHEL Conversion summary"

           color: E81123

           section_title: " - Puppet run failed with the following results:"

           section_text: ""

         when:

           - puppetrc.failed

     tags:

       - puppet2

Final outcome: Up to 160 hours saved!

I was able to re-use two of the roles in this conversion and greatly reduce the amount of time required to build automation. The conversion of 88 CentOS servers took 76 hours in total with the majority of time being spent on building automation and staggering after-hours conversions in small numbers.

I would estimate that the amount of time saved would be around 100-160 hours by using Ansible to automate and Red Hat Satellite to manage.

Learn more: "OSS Group harnesses the power of Red Hat Satellite"

Interesting articles to read on the subject :

• Convert2RHEL: How to update RHEL-like systems in place to subscribe to RHEL

• Converting from CentOS to RHEL with Convert2RHEL and Satellite

Read more about Red Hat