New devts

dev/README.md

@@ -2,8 +2,25 @@
 
 ## Ubuntu Host Requirements
 
+### Check kvm & virtualbox
+
+```bash
+sudo kvm-ok
+sudo apt install -y qemu qemu-kvm libvirt-daemon libvirt-clients bridge-utils virt-manager
+
+sudo apt install virtualbox
+```
+
+> Or install Virtualbox from https://www.virtualbox.org/wiki/Downloads
+
+### Install vagrant & ansible
+
 ```bash
-sudo apt install vagrant ansible virtualbox
+sudo apt install ansible
+
+wget -O- https://apt.releases.hashicorp.com/gpg | gpg --dearmor | sudo tee /usr/share/keyrings/hashicorp-archive-keyring.gpg
+echo "deb [signed-by=/usr/share/keyrings/hashicorp-archive-keyring.gpg] https://apt.releases.hashicorp.com $(lsb_release -cs) main" | sudo tee /etc/apt/sources.list.d/hashicorp.list
+sudo apt update && sudo apt install vagrant
 
 vagrant plugin install vagrant-hostmanager
 
@@ -12,6 +29,8 @@ ansible-galaxy collection install ansible.posix
 ansible-galaxy collection install community.crypto
 
 ansible-galaxy collection install community.general
+
+ansible-galaxy collection install community.mysql
 ```
 
 ## Deploy
@@ -19,3 +38,106 @@ ansible-galaxy collection install community.general
 ```bash
 vagrant up
 ```
+
+> In case of "'/var/run/libvirt/libvirt-sock': Permission denied", try to login/logout or restart the machine
+
+## Apply Ansible
+
+Once all VMs are up (see 'Deploy'), you can lauch Ansible without Vagrant. Examples:
+
+```bash
+ansible -v -i '192.168.56.41,' --key-file .vagrant/machines/lb/virtualbox/private_key -u vagrant -b -m setup all
+
+ansible-playbook -v -i provisioning/ansible/hosts -u vagrant -b provisioning/ansible/playbook.yml
+```
+
+## Result
+
+In your `/etc/hosts` file, add a line to match the dev nextcloud domain (defined in your Vagrantfile), eg "nextcloud.test", to the choosen IP, eg "192.168.56.51".
+
+Open a browser with "https://nextcloud.test"
+
+## Validation
+
+### 1 DB - 1 Web - 1 LB
+
+Done with:
+
+- Apache & mod_php
+- KeyDB instead of Redis
+- KeepAlived & HAProxy present but not used
+
+OK
+
+> TODO: add a switch to choose mod_php or php-FPM for Apache
+
+```Ruby
+HOSTS = [ 
+  { :hostname => "db1",         :ip => NETWORK+"11",  :ram => 1024,  :cpu => 1,  :box => "centos/7",       :group => "db_servers"       }, 
+  { :hostname => "web.test",    :ip => NETWORK+"41",  :ram => 1024,  :cpu => 1,  :box => "centos/7",       :group => "web_servers",     :ipdb => NETWORK+"20", :redis_vip => NETWORK+"40", :priority => 101 },
+  { :hostname => "lb.test",     :ip => NETWORK+"51",  :ram => 1024,  :cpu => 1,  :box => "ubuntu/focal64", :group => "lbal_servers",    :state => "MASTER",  :priority => 101 },
+]
+```
+
+### 3 DBs galera cluster - 1 Web - 1 LB
+
+OK
+
+> TODO: install KeepAlived in DB nodes
+
+```Ruby
+HOSTS = [ 
+  { :hostname => "db1",         :ip => NETWORK+"11",  :ram => 1024,  :cpu => 1,  :box => "centos/7",       :group => "db_servers"       },
+  { :hostname => "db2",         :ip => NETWORK+"12",  :ram => 1024,  :cpu => 1,  :box => "centos/7",       :group => "db_servers"       },
+  { :hostname => "db3",         :ip => NETWORK+"13",  :ram => 1024,  :cpu => 1,  :box => "centos/7",       :group => "db_servers"       },
+  { :hostname => "web.test",    :ip => NETWORK+"41",  :ram => 1024,  :cpu => 1,  :box => "centos/7",       :group => "web_servers",     :ipdb => NETWORK+"20", :redis_vip => NETWORK+"40", :priority => 101 },
+  { :hostname => "lb.test",     :ip => NETWORK+"51",  :ram => 1024,  :cpu => 1,  :box => "ubuntu/focal64", :group => "lbal_servers",    :state => "MASTER",  :priority => 101 },
+]
+```
+
+### 3 DBs galera cluster - 2 ProxySQL - 1 Web - 1 LB
+
+OK
+
+```Ruby
+HOSTS = [
+  { :hostname => "db1",         :ip => NETWORK+"11",  :ram => 1024,  :cpu => 1,  :box => "centos/7",       :group => "db_servers",      },
+  { :hostname => "db2",         :ip => NETWORK+"12",  :ram => 1024,  :cpu => 1,  :box => "centos/7",       :group => "db_servers",      },
+  { :hostname => "db3",         :ip => NETWORK+"13",  :ram => 1024,  :cpu => 1,  :box => "centos/7",       :group => "db_servers",      },
+  { :hostname => "lbsql1",      :ip => NETWORK+"19",  :ram => 1024,  :cpu => 1,  :box => "centos/7",       :group => "db_lbal_servers", :state => "MASTER",  :priority => 101, :vip => NETWORK+"20" },
+  { :hostname => "lbsql2",      :ip => NETWORK+"18",  :ram => 1024,  :cpu => 1,  :box => "centos/7",       :group => "db_lbal_servers", :state => "BACKUP",  :priority => 100, :vip => NETWORK+"20" },
+  { :hostname => "web.test",    :ip => NETWORK+"41",  :ram => 1024,  :cpu => 1,  :box => "centos/7",       :group => "web_servers",     :ipdb => NETWORK+"20", :redis_vip => NETWORK+"40", :priority => 101 },
+  { :hostname => "lb.test",     :ip => NETWORK+"51",  :ram => 1024,  :cpu => 1,  :box => "ubuntu/focal64", :group => "lbal_servers",    :state => "MASTER",  :priority => 101 },
+]
+```
+
+### 1 DB - 2 GL - 2 Web - 2 LB
+
+TODO
+
+Done with:
+
+- Apache & mod_php
+- KeyDB instead of Redis
+- KeepAlived & HAProxy present but not used
+
+```Ruby
+HOSTS = [
+  { :hostname => "db1",         :ip => NETWORK+"11",  :ram => 1024,  :cpu => 1,  :box => "centos/7",       :group => "db_servers",      }, 
+  { :hostname => "gl1",         :ip => NETWORK+"31",  :ram => 1024,  :cpu => 1,  :box => "centos/7",       :group => "gluster_servers"  },
+  { :hostname => "gl2",         :ip => NETWORK+"32",  :ram => 1024,  :cpu => 1,  :box => "centos/7",       :group => "gluster_servers"  },
+  { :hostname => "web.test",    :ip => NETWORK+"41",  :ram => 1024,  :cpu => 1,  :box => "centos/7",       :group => "web_servers",     :ipdb => NETWORK+"20", :redis_vip => NETWORK+"40", :priority => 101 },
+  { :hostname => "web2.test",   :ip => NETWORK+"42",  :ram => 1024,  :cpu => 1,  :box => "centos/7",       :group => "web_servers",     :ipdb => NETWORK+"20", :redis_vip => NETWORK+"40", :priority => 100 },
+  { :hostname => "lb.test",     :ip => NETWORK+"51",  :ram => 1024,  :cpu => 1,  :box => "ubuntu/focal64", :group => "lbal_servers",    :state => "MASTER",  :priority => 101 },
+  { :hostname => "lb2.test",    :ip => NETWORK+"52",  :ram =>  512,  :cpu => 1,  :box => "ubuntu/focal64", :group => "lbal_servers",    :state => "BACKUP",  :priority => 100 },
+]
+```
+
+### 3 DB galera cluster - 2 ProxySQL -  2 GL - 2 Web - 2 LB
+
+TODO
+
+```Ruby
+HOSTS = [
+]
+```

dev/Vagrantfile

@@ -11,19 +11,32 @@ MAIN = NETWORK+"10"
 NCDOM = "nextcloud.test"
 
 # VM machines configuration
-# ip address of the vm is NETWORK plus the last part of the IP
 HOSTS = [
-     #VM_NAME              IP_ADDRESS            RAM(mb)        CPU         BOX                  GROUP
-  { :hostname => "db",     :ip => NETWORK+"11",  :ram => 1024,  :cpu => 1,  :box => "centos/7",  :group => "database_servers" },
-  { :hostname => "redis",  :ip => NETWORK+"21",  :ram =>  512,  :cpu => 1,  :box => "centos/7",  :group => "redis_servers" }, #:folder_guest => "/srv/website", :folder_host => "src/" },
-  #{ :hostname => "redis2", :ip => NETWORK+"22",  :ram =>  512,  :cpu => 1,  :box => "centos/7",  :group => "redis_servers" }, #:port_guest => 80, :port_host => 8080 },
-  { :hostname => "web",    :ip => NETWORK+"31",  :ram => 1024,  :cpu => 1,  :box => "centos/7",  :group => "web_servers" },
-  #{ :hostname => "web2",   :ip => NETWORK+"32",  :ram => 1024,  :cpu => 1,  :box => "centos/7",  :group => "web_servers" },
-  { :hostname => "lb",     :ip => NETWORK+"41",  :ram =>  512,  :cpu => 1,  :box => "ubuntu/focal64",  :group => "loadbalancer_servers" },
-  #{ :hostname => "lb2",    :ip => NETWORK+"42",  :ram =>  512,  :cpu => 1,  :box => "ubuntu/focal64",  :group => "loadbalancer_servers" },
+    #VM_NAME                    IP_ADDRESS            RAM(mb)        CPU         BOX                       GROUP
+  { :hostname => "db1",         :ip => NETWORK+"11",  :ram => 1024,  :cpu => 1,  :box => "centos/7",       :group => "db_servers",      }, #:guestport => 3306, :hostport => 13306 }, #:user => ""  :pass => ""  },
+  #{ :hostname => "db2",         :ip => NETWORK+"12",  :ram => 1024,  :cpu => 1,  :box => "centos/7",       :group => "db_servers",      }, #:guestport => 3306, :hostport => 23306 }, #:user => ""  :pass => ""  },
+  #{ :hostname => "db3",         :ip => NETWORK+"13",  :ram => 1024,  :cpu => 1,  :box => "centos/7",       :group => "db_servers",      }, #:guestport => 3306, :hostport => 33306 }, #:user => ""  :pass => ""  },
+  #{ :hostname => "lbsql1",      :ip => NETWORK+"19",  :ram => 1024,  :cpu => 1,  :box => "centos/7",       :group => "db_lbal_servers", :state => "MASTER",  :priority => 101, :vip => NETWORK+"20" },
+  #{ :hostname => "lbsql2",      :ip => NETWORK+"18",  :ram => 1024,  :cpu => 1,  :box => "centos/7",       :group => "db_lbal_servers", :state => "BACKUP",  :priority => 100, :vip => NETWORK+"20" },
+  #{ :hostname => "redis",       :ip => NETWORK+"21",  :ram =>  512,  :cpu => 1,  :box => "centos/7",       :group => "redis_servers"    },
+  #{ :hostname => "redis2",      :ip => NETWORK+"22",  :ram =>  512,  :cpu => 1,  :box => "centos/7",       :group => "redis_servers"    },
+  #{ :hostname => "gl",          :ip => NETWORK+"31",  :ram => 1024,  :cpu => 1,  :box => "centos/7",       :group => "gluster_servers"  },
+  #{ :hostname => "gl2",         :ip => NETWORK+"32",  :ram => 1024,  :cpu => 1,  :box => "centos/7",       :group => "gluster_servers"  },
+  { :hostname => "web.test",    :ip => NETWORK+"41",  :ram => 1024,  :cpu => 1,  :box => "centos/7",       :group => "web_servers",     :ipdb => NETWORK+"20", :redis_vip => NETWORK+"40", :priority => 101 },
+  #{ :hostname => "web2.test",   :ip => NETWORK+"42",  :ram => 1024,  :cpu => 1,  :box => "centos/7",       :group => "web_servers",     :ipdb => NETWORK+"20", :redis_vip => NETWORK+"40", :priority => 100 },
+  { :hostname => "lb.test",     :ip => NETWORK+"51",  :ram => 1024,  :cpu => 1,  :box => "ubuntu/focal64", :group => "lbal_servers",    :state => "MASTER",  :priority => 101 },
+  #{ :hostname => "lb2.test",    :ip => NETWORK+"52",  :ram =>  512,  :cpu => 1,  :box => "ubuntu/focal64", :group => "lbal_servers",    :state => "BACKUP",  :priority => 100 },
+  #{ :hostname => "prome",       :ip => NETWORK+"61",  :ram =>  512,  :cpu => 1,  :box => "centos/7",       :group => "monitor_servers"  },
+  #{ :hostname => "node",        :ip => NETWORK+"62",  :ram =>  512,  :cpu => 1,  :box => "centos/7",       :group => "monitor_servers"  },
+  #{ :hostname => "grafa",       :ip => NETWORK+"63",  :ram =>  512,  :cpu => 1,  :box => "centos/7",       :group => "monitor_servers"  },
+  #{ :hostname => "essai1.test", :ip => NETWORK+"98",  :ram => 1024,  :cpu => 1,  :box => "centos/7",       :group => "test_servers",    :priority => 101, :vip => NETWORK+"100" },
+  #{ :hostname => "essai2.test", :ip => NETWORK+"99",  :ram => 1024,  :cpu => 1,  :box => "centos/7",       :group => "test_servers",    :priority => 100, :vip => NETWORK+"100" },
+  #{ :hostname => "web.test",    :ip => NETWORK+"41",  :ram => 1024,  :cpu => 1,  :box => "centos/7",       :group => "web_servers"      },
+  #{ :hostname => "web2.test",   :ip => NETWORK+"42",  :ram => 1024,  :cpu => 1,  :box => "centos/7",       :group => "web_servers"      },
 ]
 
 # Defined ansible playbook
+vagrant_root = File.expand_path(File.dirname(__FILE__))
 # If empty, will skip the ansible provisioner block
 ansible_playbook = "provisioning/ansible/playbook.yml"
 # Ansible inventory. The path supports nested directories or a single file
@@ -50,12 +63,8 @@ Vagrant.configure(VAGRANTFILE_API_VERSION) do |config|
     if ! groups.has_key?(cfg[:group])
       groups[cfg[:group]] = [cfg[:hostname]]
     else
-      #combi = cfg[:ip]+" server_name="+cfg[:hostname]
-      #groups[cfg[:group]].push(combi)
       groups[cfg[:group]].push(cfg[:hostname])
     end
-    #combi = cfg[:ip]+" server_name="+cfg[:hostname]
-    #groups["all"].push(combi)
     groups["all"].push(cfg[:hostname])
   end
 
@@ -64,15 +73,62 @@ Vagrant.configure(VAGRANTFILE_API_VERSION) do |config|
   if File.dirname(ansible_inventory_path) != "."
     Dir.mkdir(File.dirname(ansible_inventory_path)) unless Dir.exist?(File.dirname(ansible_inventory_path))
   end
+  File.delete(ansible_inventory_path) if File.exist?(ansible_inventory_path)
   File.open(ansible_inventory_path, 'w') do |f|
     HOSTS.each do |cfg|
-      f.write "#{cfg[:hostname]} ansible_host=#{cfg[:ip]}\n"
+      ssh_key = vagrant_root+"/.vagrant/machines/"+cfg[:hostname]+"/virtualbox/private_key"
+      f.write "#{cfg[:hostname]} ansible_host=#{cfg[:ip]} ansible_ssh_private_key_file=#{ssh_key}\n"
     end
-    groups.keys.each do |g|
+    groups.keys.each_with_index do |g, index|
       f.write "\n"
       f.write "[#{g}]\n"
       groups[g].each do |h| 
-        f.write "#{h}\n"
+        f.write "#{h}"
+        network = NETWORK+"0/24"
+        f.write " network_allowed="+network
+        if g == "lbal_servers"
+          HOSTS.each do |cfg|
+            if cfg[:hostname] == h
+              priority = (cfg[:priority].to_s || "UNKNOWN" )
+              state = (cfg[:state].to_s || "UNKNOWN" )           
+              f.write " ssl_name="+NCDOM+" keepalived_vip="+MAIN+" keepalived_priority="+priority+" keepalived_state="+state
+            end
+          end
+        end
+        if g == "web_servers"
+          HOSTS.each do |cfg|
+            if cfg[:hostname] == h
+              ipdb = (cfg[:ipdb].to_s || "" )
+              priority = (cfg[:priority].to_s || "UNKNOWN" )
+              redis = (cfg[:redis_vip].to_s || "UNKNOWN" )
+              if groups.count("web_servers") > 2
+                f.write " nc_global_name="+NCDOM+" db_host="+ipdb+" redis_host="+redis+" keepalived_vip="+redis+" keepalived_priority="+priority
+              else
+                f.write " nc_global_name="+NCDOM+" db_host="+ipdb
+              end
+            end
+          end
+        end
+        if g == "db_lbal_servers"
+          HOSTS.each do |cfg|
+            if cfg[:hostname] == h
+              priority = (cfg[:priority].to_s || "UNKNOWN" )
+              state = (cfg[:state].to_s || "UNKNOWN" )
+              vip = (cfg[:vip].to_s || "UNKNOWN" )
+              f.write " keepalived_vip="+vip+" keepalived_priority="+priority+" keepalived_state="+state
+            end
+          end
+        end
+        if g == "test_servers"
+          HOSTS.each do |cfg|
+            if cfg[:hostname] == h
+              priority = (cfg[:priority].to_s || "UNKNOWN" )
+              vip = (cfg[:vip].to_s || "UNKNOWN" )
+              f.write " keepalived_vip="+vip+" keepalived_priority="+priority
+            end
+          end
+        end
+        f.write "\n"
       end
     end
   end
@@ -112,148 +168,23 @@ Vagrant.configure(VAGRANTFILE_API_VERSION) do |config|
       # Provision nodes with Ansible.
       # The index used here in order to execute the provision just after all
       # the servers are up and running.
-      #if index == HOSTS.size - 1
-      #  if ansible_playbook != ""
-      #    conf.vm.provision :ansible do |ansible|
-      #      ansible.limit = "all"
-      #      ansible.compatibility_mode = "2.0"
-      #      ansible.become = true
-      #      ansible.inventory_path = ansible_inventory_path
-      #      ansible.playbook = ansible_playbook
-      #      #ansible.verbose = "vvvv"
-      #  end
-      #end      
+      if index == HOSTS.size - 1
+        if ansible_playbook != ""
+          conf.vm.provision :ansible do |ansible|
+            ansible.limit = "all"
+            ansible.compatibility_mode = "2.0"
+            ansible.inventory_path = ansible_inventory_path
+            ansible.playbook = ansible_playbook
+            ansible.become = true
+            #ansible.verbose = "vvvv"
+          end
+        end
+      end      
     end
   end
 
-  # VM PROVISIONING
+# VM PROVISIONING
 
-  #Database Server 
-  config.vm.define "db" do |db|
-    # Temp NFS stuff waiting Ceph
-    db.vm.provision "shell", path: "provisioning/bash/nfs-server.sh"
-    #
-    db.vm.provision "ansible" do |ansible|
-      ansible.compatibility_mode = "2.0"
-      ansible.playbook="provisioning/ansible/mariadb.yml"
-      ansible.inventory_path = ansible_inventory_path
-      ansible.become = true
-      ansible.extra_vars = {
-        db_users: [
-          { name: 'web', password: 'secret', host: 'web' },
-          { name: 'web', password: 'secret', host: 'web2' }
-        ]
-      }
-    end    
-  end
-
-  #Redis Server 
-  config.vm.define "redis" do |redis|
-    redis.vm.provision :ansible do |ansible|
-      ansible.compatibility_mode = "2.0"
-      ansible.playbook="provisioning/ansible/redis.yml"
-      ansible.inventory_path = ansible_inventory_path
-      ansible.become = true
-      #ansible.extra_vars = {
-      #  #redis_bind_interface: "192.168.56.14", #bug Centos
-      #}
-    end
-  end
-
-  #  #Web Server
-  config.vm.define "web" do |web|
-    web.vm.provision "shell", path: "provisioning/bash/Centos_7.sh"
-    web.vm.provision "ansible" do |ansible|
-       ansible.compatibility_mode = "2.0"
-       ansible.playbook = "provisioning/ansible/nextcloud.yml"
-       ansible.inventory_path = ansible_inventory_path
-       ansible.become = true
-       ansible.extra_vars = { 
-         ssl_name: NCDOM,
-         nc_trusted_domain: "web",
-         db_host: "db",
-         nc_db_user: "web",
-         nc_db_password: "secret",
-         use_redis_server: "true",
-         redis_host: "redis",
-         #nc_multiple: "nfs",
-         #nfs_server: "db",
-       }
-       #ansible.verbose = "vvvv"
-    end
-  end
-
-  #  #LoadBalancer (master)
-  config.vm.define "lb" do |lb|
-    lb.vm.provision "ansible" do |ansible|
-      ansible.compatibility_mode = "2.0"
-      ansible.playbook="provisioning/ansible/haproxy.yml"
-      ansible.inventory_path = ansible_inventory_path
-      ansible.become = true
-      ansible.extra_vars = {
-        ssl_name: NCDOM,
-        network_allowed: NETWORK+"0/24",
-        keepalived_vip: MAIN,
-        keepalived_priority: 101,
-        keepalived_state: "MASTER",
-        haproxy_backend_servers: [
-          { name: 'web', ip: 'web:8000' },
-          #{ name: 'web2', ip: 'web2:8000' }
-        ]
-      }
-    end
-  end
-
-#  #LoadBalancer (backup)
-#  config.vm.define "lb2" do |lb2|
-#    lb2.vm.provision "shell", inline: "apt-get install -y haproxy keepalived"
-#    lb2.vm.provision "ansible" do |ansible|
-#       ansible.compatibility_mode = "2.0"
-#       ansible.playbook="provisioning/ansible/haproxy.yml"
-#       ansible.become = true
-#       ansible.extra_vars = { 
-#         ansible_python_interpreter: "/usr/bin/python3",
-#         ssl_name: NCDOM,
-#         network_allowed: NETWORK+"0/24",
-#         keepalived_vip: MAIN,
-#         keepalived_priority: 100,
-#         keepalived_state: "BACKUP",
-#         haproxy_backend_servers: [
-#           { name: 'web', ip: 'web:8000' },
-#           #{ name: 'web2', ip: 'web2:8000' }
-#         ]
-#       }
-#    end 
-#  end
-#
-#  #Web Server 2
-#  config.vm.define "web2" do |web2|
-#    web2.vm.hostname = "nextcloud"
-#    web2.vm.box = "centos/7"
-#    web2.vm.network "private_network", ip: "192.168.56.15"
-#
-#    web2.vm.provision "shell", path: "provisioning/install/Centos_7.sh"
-#    web2.vm.provision "ansible" do |ansible|
-#       ansible.compatibility_mode = "2.0"
-#       ansible.playbook = "provisioning/ansible/nextcloud.yml"
-#       ansible.become = true
-#       ansible.extra_vars = { 
-#         ansible_python_interpreter: "/usr/bin/python2",
-#         ssl_name: "nextcloud.test",
-#         nc_trusted_domain: "192.168.56.15",
-#         db_host: "192.168.56.21",
-#         nc_db_user: "web",
-#         nc_db_password: "secret",
-#         use_redis_server: "true",
-#         redis_host: "192.168.56.13",
-#         #nc_multiple: "nfs",
-#         #nfs_server: "192.168.56.21",
-#       }
-#       #ansible.inventory_path = "provisioning/apache.inventory"
-#       #ansible.verbose = "vvvv"
-#    end
-#  end 
-#
 #  #Prometheus
 #  config.vm.define "prometheus" do |prometheus|
 #    prometheus.vm.box = 'centos/7'

dev/provisioning/ansible/mariadb.yml → dev/provisioning/ansible/database.yml

@@ -1,11 +1,10 @@
 ---
-- name: apply mariadb configuration
-  hosts: database_servers
+- name: apply database configuration
+  collections:
+    - community.mysql
+  hosts: db_servers
   vars:
     ansible_python_interpreter: /usr/bin/python3
-  #  ansible_user: vagrant
-  #  ansible_password: vagrant
-  #  app_bind_address: 192.168.56.14
   pre_tasks:
     - name: define ansible_python_interpreter group // linux distribution
       set_fact:

dev/provisioning/ansible/essai.yml

@@ -0,0 +1,29 @@
+---
+- name: apply a test role
+  collections:
+    - community.general
+    - ansible.posix
+  hosts: test_servers
+  vars:
+    ansible_python_interpreter: /usr/bin/python3
+  pre_tasks:
+    - name: define ansible_python_interpreter group // linux distribution
+      set_fact:
+        ansible_python_interpreter: /usr/bin/python2
+      when: ansible_distribution == 'CentOS' and ansible_distribution_major_version == '7'
+  roles:
+    - role: redis
+      vars:
+        redis_daemon: "keydb"
+        redis_port: "6380"
+    - role: keepalived
+    - role: haproxy
+      vars:
+        hatarget: "keydb"
+        redis_vip_port: "6379"
+        redis_port: "6380"
+        ssl_self: false
+    - role: web_php
+      vars:
+        redis_daemon: "keydb"
+        redis_port: "6380"

dev/provisioning/ansible/haproxy.yml → dev/provisioning/ansible/loadbalancer.yml

@@ -1,18 +1,15 @@
 ---
-- name: apply haproxy role
-  hosts: loadbalancer_servers
+- name: apply loadbalancer role
+  hosts: lbal_servers
   vars:
     ansible_python_interpreter: /usr/bin/python3
-  #  ansible_user: vagrant
-  #  ansible_password: vagrant
-  #  ssl_name: "nextcloud.test"
-  #  network_allowed: "192.168.56.0/24"
-  #  haproxy_backend_servers:
-  #    { name: 'web', ip: '192.168.56.14:8000' }
   pre_tasks:
     - name: define ansible_python_interpreter group // linux distribution
       set_fact:
         ansible_python_interpreter: /usr/bin/python2
       when: ansible_distribution == 'CentOS' and ansible_distribution_major_version == '7'
   roles:
-    - { role: haproxy }
+    - role: haproxy
+      vars:
+        hatarget: "http"
+        ssl_self: true

dev/provisioning/ansible/nextcloud.yml

@@ -6,13 +6,29 @@
   hosts: web_servers
   vars:
     ansible_python_interpreter: /usr/bin/python3
-  #  ansible_user: vagrant
-  #  ansible_password: vagrant
-  #  db_host: 192.168.56.13
   pre_tasks:
     - name: define ansible_python_interpreter group // linux distribution
       set_fact:
         ansible_python_interpreter: /usr/bin/python2
       when: ansible_distribution == 'CentOS' and ansible_distribution_major_version == '7'
   roles:
-    - { role: nextcloud }
+    - role: redis
+      vars:
+        redis_daemon: "keydb"
+        redis_port: "6380"
+    - role: keepalived
+    - role: haproxy
+      vars:
+        hatarget: "keydb"
+        redis_vip_port: "6380"
+        redis_port: "6380"
+        ssl_self: false
+    - role: web_php
+      vars:
+        enable_php_fpm: false
+        redis_daemon: "keydb"
+        redis_port: "6380"
+    - role: nextcloud
+      vars:
+        NEXTCLOUD_VERSION: "24.0.8"
+        redis_port: "6380"

dev/provisioning/ansible/playbook.yml

@@ -1,8 +1,11 @@
 ---
-- import_playbook: mariadb.yml
-- import_playbook: redis.yml
+- import_playbook: database.yml
+- import_playbook: sql_loadbalancer.yml
+- import_playbook: storage.yml
 - import_playbook: nextcloud.yml
-- import_playbook: haproxy.yml
+- import_playbook: loadbalancer.yml
+
+  #- import_playbook: essai.yml
 
   #- import_playbook: node_exporter.yml
   #- import_playbook: prometheus.yml

dev/provisioning/ansible/roles/gluster/tasks/main.yml

@@ -0,0 +1,74 @@
+- include_tasks: "prep_os/{{ ansible_os_family }}.yml"
+
+### Configuration
+
+- name: Create gluster share directory
+  file:
+    path: "{{ item }}"
+    state: directory
+    owner: root
+    group: root
+    mode: '0750'
+  with_items: ["/mnt/data", "/nextcloud"]
+  changed_when: false
+
+- name: Show all the gluster_servers in the inventory
+  debug:
+    msg: "{{ groups['gluster_servers'] | list }}"
+
+- name: Create gluster 1
+  gluster_volume:
+    state: present
+    force: yes
+    name: ncgluster1
+    bricks: /mnt/data/g1
+    #rebalance: yes
+    host: "{{ inventory_hostname }}"
+    replicas: 2
+    cluster: "{{ groups['gluster_servers'] | list }}"
+  run_once: true
+  ignore_errors: yes
+
+- name: Create gluster 2
+  gluster_volume:
+    state: present
+    force: yes
+    name: ncgluster2
+    bricks: /nextcloud/g1
+    host: "{{ inventory_hostname }}"
+    replicas: 2
+    cluster: "{{ groups['gluster_servers'] | list }}"
+  run_once: true
+  ignore_errors: yes
+
+- name: Set tuning quorum-count on GlusterFS volumes
+  shell: "gluster volume set {{ item }}"
+  loop:
+    - "ncgluster1 cluster.quorum-count 1"
+    - "ncgluster1 cluster.quorum-reads false"
+    - "ncgluster2 cluster.quorum-count 1"
+    - "ncgluster2 cluster.quorum-reads false"
+  loop_control:
+    pause: 2
+  run_once: true
+  delegate_to: "{{ groups['gluster_servers'][0] }}"
+
+- name: Heal GlusterFS volumes
+  shell: "gluster volume heal {{ item }} enable"
+  loop:
+    - ncgluster1
+    - ncgluster2
+  loop_control:
+    pause: 2
+  run_once: true
+  delegate_to: "{{ groups['gluster_servers'][0] }}"
+
+#- name: start gluster volume
+#  gluster_volume:
+#    state: started
+#    name: ncgluster
+
+- name: GlusterFS reStarting
+  service:
+    name: glusterd
+    state: restarted

dev/provisioning/ansible/roles/gluster/tasks/prep_os/CentOS.yml

@@ -0,0 +1,57 @@
+---
+# CentOS related tasks
+
+- name: Selinux... enable seboolean settings
+  seboolean:
+    name: "{{ item }}"
+    state: yes 
+    persistent: yes 
+  with_items:
+    - virt_sandbox_use_fusefs
+    - virt_use_fusefs
+
+- name: Install glusterfs repo
+  yum:
+    name: centos-release-gluster41
+    state: latest
+
+- name: Install glusterfs packages and requirements
+  yum:
+    #name: ['glusterfs-server', 'samba', 'nfs-utils']
+    name: glusterfs-server
+    state: latest
+
+    #- name: permit traffic in default zone for services
+    #  firewalld:
+    #    service: "{{ item }}"
+    #    permanent: yes
+    #    state: enabled
+    #  with_items:
+    #    - nfs
+    #    - samba
+    #    - samba-client
+    #
+    #- name: traffic in default zone on ports
+    #  firewalld:
+    #    port: "{{ item }}"
+    #    permanent: yes
+    #    state: enabled
+    #  with_items:
+    #    - "111/tcp"
+    #    - "139/tcp"
+    #    - "445/tcp"
+    #    - "965/tcp"
+    #    - "2049/tcp"
+    #    - "38465-38469/tcp"
+    #    - "631/tcp"
+    #    - "24007-24008/tcp"
+    #    - "24009/tcp"
+    #    - "111/udp"
+    #    - "963/udp"
+    #    - "49152-49251/tcp"
+
+- name: GlusterFS Installed Starting
+  service:
+    name: glusterd
+    state: started
+    enabled: yes 

dev/provisioning/ansible/roles/gluster/tasks/prep_os/RedHat.yml

@@ -0,0 +1,3 @@
+---
+
+- include_tasks: "{{ ansible_facts['distribution'] }}.yml"

dev/provisioning/ansible/roles/haproxy/defaults/main.yml

@@ -1,21 +1,24 @@
 ---
-# Frontend settings.
+hatarget: "keydb"
+
+# Common settings
+backend_balance_method: 'roundrobin' # leastconn | roundrobin
+
+# KeyDB settings
+redis_port: 6379
+
+# HTTP settings
+# frontend
 frontend_mode: 'http'
 ssl_name: 'nextcloud.test'
 ssl_crt_path: '/etc/ssl/private'
 ssl_self: true
-
-# Backend settings.
+# Backend
 backend_mode: 'http'
-backend_balance_method: 'roundrobin' # leastconn | roundrobin
-
-# Specific nextcloud settings.
+# Specific nextcloud settings
 nc_settings: true
 network_allowed: '192.168.56.0/24'
 
 # List of backend servers.
 haproxy_backend_servers:
   - { name: 'web', ip: '192.168.56.88:8000' }
-
-keepalived_priority: 101
-keepalived_vip: "192.168.56.10"

dev/provisioning/ansible/roles/haproxy/tasks/main.yml

@@ -1,4 +1,14 @@
 ---
+- name: Set haproxy_backend_servers variable
+  block:
+    - name: Try group web_servers
+      set_fact:
+        haproxy_backend_servers: "{{ groups['web_servers'] | list }}"
+  rescue:
+    - name: Try group test_servers
+      set_fact:
+        haproxy_backend_servers: "{{ groups['test_servers'] | list }}"
+
 - name: Include OS specific variables.
   include_vars: "{{ ansible_os_family }}.yml"
 
@@ -26,10 +36,6 @@
   shell: "ip -4 addr show | grep {{ result.stdout }} | rev | cut -d ' ' -f 1 | rev"
   register: itfn
 
-- name: Set keepalived_bind_interface.
-  set_fact:
-    keepalived_bind_interface: "{{ itfn.stdout }}"
-
 - name: Integration net.ipv4
   blockinfile:
     dest: /etc/sysctl.conf
@@ -37,16 +43,8 @@
       net.ipv4.ip_forward = 1 
       net.ipv4.ip_nonlocal_bind = 1 
 
-- name: Ensure keepalived is started and enabled on boot.
-  service: name=keepalived state=started enabled=yes
-
-- name: Ensure keepalived conf is set 
-  template: >
-    src=templates/keepalived.conf.j2
-    dest=/etc/keepalived/keepalived.conf
-
-- name: Ensure HAProxy is started and enabled on boot.
-  service: name=haproxy state=started enabled=yes
+- name: Ensure HAProxy is enabled on boot
+  service: name=haproxy enabled=yes
 
 - name: Create private key (RSA, 4096 bits)
   community.crypto.openssl_privatekey:
@@ -80,6 +78,7 @@
   community.crypto.openssl_dhparam:
     path: /etc/haproxy/dhparams.pem
     size: 2048
+  when: ssl_self
 
     #- name: Add ssl dhparam file
     #  lineinfile:
@@ -99,13 +98,14 @@
     #    content: '{{ cfg_content }}'
     #    state: present
 
-- name: Ensure HAProxy conf is set 
-  template: >
-    src=templates/haproxy.cfg.j2
-    dest=/etc/haproxy/haproxy.cfg
+- name: Ensure HAProxy conf is set
+  template:
+    src: "haproxy_{{ hatarget }}.cfg.j2"
+    dest: /etc/haproxy/haproxy.cfg
+    mode: 0640
 
-- name: keepalived restart
-  service: name=keepalived state=restarted
+      #- name: HAProxy start
+      #  service: name=haproxy state=started
 
-- name: HAProxy restart
+- name: HAProxy reload
   service: name=haproxy state=restarted

dev/provisioning/ansible/roles/haproxy/tasks/setup/Debian.yml

@@ -1,4 +1,8 @@
 ---
+- name: Update the {{ ansible_distribution }} repos
+  apt:
+    update_cache: yes
+
 - name: Install all the {{ ansible_distribution }} packages
   apt:
     name: "{{ specific_packages }}"

dev/provisioning/ansible/roles/haproxy/tasks/setup/RedHat.yml

@@ -11,9 +11,13 @@
     owner: root
     group: root
     mode: 0750
+  when: ssl_self
 
-    #- name: Mariadb service
-    #  service:
-    #    name: "{{ mariadb_service }}"
-    #    state: started
-    #    enabled: yes
+- name: Selinux... enable seboolean settings
+  seboolean:
+    name: "{{ item }}"
+    state: yes
+    persistent: yes
+  with_items:
+    - haproxy_connect_any
+  when: ansible_selinux.status == "enabled"

dev/provisioning/ansible/roles/haproxy/templates/haproxy.cfg.j2 → dev/provisioning/ansible/roles/haproxy/templates/haproxy.old

@@ -87,7 +87,7 @@ backend http_servers
 {% endif %}
 
 {% if haproxy_backend_servers != '' %}
-{% for dict_item in haproxy_backend_servers %}
-    server {{ dict_item.name }} {{ dict_item.ip }}
+{% for item in haproxy_backend_servers %}
+    server {{ item }} {{ item }}:8000
 {% endfor %}
 {% endif %}

dev/provisioning/ansible/roles/haproxy/templates/haproxy_http.cfg.j2

@@ -0,0 +1,101 @@
+global
+  log 127.0.0.1 local2
+  chroot      /var/lib/haproxy
+  pidfile     /var/run/haproxy.pid
+  user haproxy
+  group haproxy
+  maxconn 5000
+  nbproc          2
+  cpu-map         1 0
+  cpu-map         2 1
+  daemon
+  tune.ssl.default-dh-param 2048
+  stats socket /var/lib/haproxy/stats mode 660 level admin
+
+listen stats
+  bind :9000
+  mode http
+  stats enable
+  stats hide-version
+  stats show-node
+  stats realm Haproxy\ Statistics
+  stats uri /haproxy_stats
+  stats auth admin:admin  # Authentication credentials
+  timeout connect         86400
+  timeout client          86400
+  timeout server          86400
+
+defaults
+  mode                    tcp
+  log                     global
+  option                  tcplog
+  option                  dontlognull
+  option http-server-close
+  option forwardfor       except 127.0.0.0/8
+  option                  redispatch
+  retries                 3
+  timeout http-request    10
+  timeout queue           1m
+  timeout connect         1m
+  timeout client          1m
+  timeout server          1m
+  timeout http-keep-alive 10
+  timeout check           10
+
+frontend ft_http
+  bind *:80
+  mode http
+  timeout client          1m
+  default_backend bk_http
+
+frontend ft_https
+  bind *:443
+  mode tcp
+  timeout client          1m
+  http-request set-header X-Forwarded-Proto: https
+  default_backend bk_https
+
+#frontend ft_mix
+#  mode http
+#  bind *:80
+#  bind *:443 ssl crt {{ ssl_crt_path }}/{{ ssl_name }}.pem
+#  http-request redirect scheme https code 301 if !{ ssl_fc }
+#  http-response set-header Strict-Transport-Security max-age=63072000
+#  default_backend bk_mix
+#
+#backend bk_mix
+#  mode http
+#  balance {{ backend_balance_method }}
+#  default-server inter 1s
+#  timeout connect         10s 
+#  timeout server          1m  
+#  {% if haproxy_backend_servers != '' %}
+#  {% for item in haproxy_backend_servers %}
+#  server {{ item }} {{ hostvars[item]['ansible_host'] }}:80 maxconn 250 check id {{ loop.index }}
+#  {% endfor %}
+#  {% endif %}
+
+backend bk_http
+  mode http
+  balance {{ backend_balance_method }}
+  default-server inter 1s
+  timeout connect         10s
+  timeout server          1m
+  {% if haproxy_backend_servers != '' %}
+  {% for item in haproxy_backend_servers %}
+  server {{ item }} {{ hostvars[item]['ansible_host'] }}:80 maxconn 250 check id {{ loop.index }}
+  {% endfor %}
+  {% endif %}
+
+backend bk_https
+  mode tcp
+  balance {{ backend_balance_method }}
+  option ssl-hello-chk
+  default-server inter 1s
+  timeout connect         10s
+  timeout server          1m
+  {% if haproxy_backend_servers != '' %}
+  {% for item in haproxy_backend_servers %}
+  server {{ item }} {{ hostvars[item]['ansible_host'] }}:443 maxconn 250 check id {{ loop.index }}
+  {% endfor %}
+  {% endif %}

dev/provisioning/ansible/roles/haproxy/templates/haproxy_keydb.cfg.j2

@@ -0,0 +1,27 @@
+global 
+  user haproxy 
+  group haproxy 
+ 
+defaults KEYDB
+  mode tcp 
+  timeout connect 3s 
+  timeout server 6s 
+  timeout client 6s 
+ 
+frontend ft_keydb 
+  bind {{ keepalived_vip }}:{{ redis_vip_port }} name redis 
+  default_backend bk_keydb 
+ 
+backend bk_keydb 
+  option tcp-check 
+  #tcp-check send *1\r\n$4\r\nping\r\n 
+  #tcp-check expect string +PONG
+  tcp-check send PING\r\n
+  tcp-check expect string +PONG
+  tcp-check send info\ replication\r\n
+  tcp-check expect string role: active-replica
+  tcp-check send QUIT\r\n
+  tcp-check expect string +OK 
+  {% for item in haproxy_backend_servers %}
+  server {{ item }} {{ hostvars[item]['ansible_host'] }}:{{ redis_port }} check inter 100ms
+  {% endfor %}

dev/provisioning/ansible/roles/haproxy/templates/haproxy_keydb.new

@@ -0,0 +1,52 @@
+global
+    log /dev/log    local0
+    log /dev/log    local1 notice
+    chroot /var/lib/haproxy
+    #stats socket /run/haproxy/admin.sock mode 660 level admin expose-fd listeners
+    stats timeout 30s
+    user haproxy
+    group haproxy
+    daemon
+
+    # The lines below enable multithreading. This should correlate to number of threads available you want to use.
+    nbproc 1
+    #nbthread 4
+    #cpu-map auto:1/1-4 0-3
+
+    # Default SSL material locations
+    #ca-base /etc/ssl/certs
+    #crt-base /etc/ssl/private
+
+    # Default ciphers to use on SSL-enabled listening sockets.
+    #ssl-default-bind-ciphers ECDH+AESGCM:DH+AESGCM:ECDH+AES256:DH+AES256:ECDH+AES128:DH+AES:RSA+AESGCM:RSA+AES:!aNULL:!MD5:!DSS
+    #ssl-default-bind-options no-sslv3
+    maxconn 40000
+
+defaults
+    log global
+    mode    http
+    option  httplog
+    option  dontlognull
+    timeout connect 5000
+    timeout client  50000
+    timeout server  50000
+
+listen mykeydb 
+    bind *:{{ redis_vip_port }}
+    maxconn 40000 
+    mode tcp
+    balance first
+    option tcplog
+    option tcp-check
+    #uncomment these lines if you have basic auth
+    #tcp-check send AUTH\ yourpassword\r\n
+    #tcp-check expect string +OK
+    tcp-check send PING\r\n
+    tcp-check expect string +PONG
+    tcp-check send info\ replication\r\n
+    tcp-check expect string role: active-replica
+    tcp-check send QUIT\r\n
+    tcp-check expect string +OK
+    {% for item in haproxy_backend_servers %}
+    server {{ item }} {{ hostvars[item]['ansible_host'] }}:{{ redis_port }} maxconn 20000 check inter 1s
+    {% endfor %}

dev/provisioning/ansible/roles/haproxy/vars/RedHat.yml

@@ -1,6 +1,5 @@
 specific_packages:
   - haproxy 
-  - keepalived
   - python2-cryptography
 
 log_0: "log         127.0.0.1 local2"
@@ -9,6 +8,7 @@ log_2: "#"
 
 haproxy_stats_socket: "/var/lib/haproxy/stats"
 haproxy_pid: "/var/run/haproxy.pid"
+
 ssl_ciphersuites: "#"
 ssl_options: "#"
 ssl_dh: "tune.ssl.default-dh-param 2048"

dev/provisioning/ansible/roles/mariadb/defaults/main.yml

@@ -27,15 +27,14 @@ mariadb_database:
     encoding: utf8
     state: present
     target: omit
+    priv: '*:ALL,GRANT'
+    pass: 'secret'
+    hosts: "{{ groups['web_servers'] | default('localhost') }}"
 
-# Add mariabd users
-# (replaced with global vars when calling role)
-db_users:
-  - { name: 'web', password: 'secret', host: '192.168.56.15' }
-
-mariadb_user_priv: 'nextcloudb.*:ALL,GRANT'
 mariadb_user_encrypted: false
 
 # Specify slow query log
 mariadb_slow_query_log_enabled: false
 mariadb_slow_query_time: "2"
+
+galera_cluster_name: "gaclunc"

dev/provisioning/ansible/roles/mariadb/tasks/config/secure-installation.yml

@@ -61,7 +61,14 @@
     host: "{{ item }}"
     state: absent
   with_items: "{{ mysql_anonymous_hosts.stdout_lines|default([]) }}"
-  no_log: true
+
+- name: Remove mysql users.
+  mysql_user:
+    name: "{{ item }}"
+    host: "localhost"
+    state: absent
+  with_items:
+    - mysql
 
 - name: Remove MySQL test database.
   mysql_db: "name='test' state=absent"

dev/provisioning/ansible/roles/mariadb/tasks/config/template.yml

@@ -5,12 +5,19 @@
   changed_when: false
   check_mode: false
 
-- name: setup Mariadb config file
+- name: Setup Mariadb config file
   template:
     src: server.j2
     dest: "{{ mariadb_config_file }}"
     owner: "{{ mariadb_config_file_owner }}"
     group: "{{ mariadb_config_file_group }}"
     mode: 0644
-  notify: 
-  - Restart mariadb
+
+- name: Setup Galera config file if needed
+  template:
+    src: galera.j2
+    dest: "{{ galera_config_file }}"
+    owner: "{{ mariadb_config_file_owner }}"
+    group: "{{ mariadb_config_file_group }}"
+    mode: 0644
+  when: groups['db_servers'] | length > 1

dev/provisioning/ansible/roles/mariadb/tasks/database/databases.yml

@@ -9,3 +9,5 @@
     target: "{{ item.target | default(omit) }}"
     login_unix_socket: "{{ mariadb_socket }}"
   with_items: "{{ mariadb_database }}"
+  run_once: true
+  no_log: true

dev/provisioning/ansible/roles/mariadb/tasks/database/users.yml

@@ -1,14 +1,71 @@
 ---
 
-- name: Ensure Mariadb users are present.
+- name: Ensure Mariadb users are present
   mysql_user:
-    name: "{{ item.name }}"
-    host: "{{ item.host | default('localhost') }}"
-    password: "{{ item.password }}"
-    priv: "{{ mariadb_user_priv | default('*.*:USAGE') }}"
+    name: "{{ item.0.name }}"
+    #host: "{{ item.1 }}"
+    host: "%"
+    password: "{{ item.0.pass }}"
+    priv: "{{ item.0.name }}.{{ item.0.priv }}"
     state: "present"
     append_privs: "no"
     encrypted: "{{ mariadb_user_encrypted | default('no') }}"
     login_unix_socket: "{{ mariadb_socket }}"
-  with_items: "{{ db_users }}"
+  with_subelements:
+    - "{{ mariadb_database }}"
+    - hosts
+  run_once: true
   no_log: true
+
+    #- name: Allow previous user to ProxySQL
+    #  mysql_user:
+    #    name: "{{ item.0.name }}"
+    #    host: "{{ groups['db_lbal_servers'][0] | default('localhost') }}"
+    #    password: "{{ item.0.pass | default('sbpass') }}"
+    #    priv: "*.*:ALL,GRANT"
+    #    state: "present"
+    #    append_privs: "no"
+    #    encrypted: "{{ mariadb_user_encrypted | default('no') }}"
+    #    login_unix_socket: "{{ mariadb_socket }}"
+    #  with_subelements:
+    #    - "{{ mariadb_database }}"
+    #  run_once: true
+    #  no_log: true
+    #    #when: "'db_lbal_servers' in groups.keys()"
+
+- name: Create a new MySQL user called monitor
+  mysql_user:
+    name: "monitor"
+    host: "%"
+    password: "{{ db_monitor_passwd | default('MonitoringPassword') }}"
+    priv: "*.*:USAGE"
+    state: "present"
+    append_privs: "no"
+    encrypted: "{{ mariadb_user_encrypted | default('no') }}"
+    login_unix_socket: "{{ mariadb_socket }}"
+  run_once: true
+  no_log: true
+    #when: "'db_lbal_servers' in groups.keys()" 
+
+  #- name: Read SQL configuration
+  #  set_fact:
+  #    sql_content: "{{ lookup('template', '{{ role_path }}/templates/grant_all_hosts.sql.j2') }}"
+  #  with_items:
+  #    - user: monitor
+  #      host: localhost
+  #
+  #- name: Integration SQL configuration
+  #  copy:
+  #    dest: /tmp/file.sql
+  #    content: '{{ sql_content }}'
+  #
+  #- name: Grant appdb for all servers
+  #  shell: "mysql -u {{ mysql_root_username }} -p{{ mysql_root_password }} -h 127.0.0.1 < /tmp/file.sql"
+  #  run_once: true
+  #  no_log: true
+  #  when: "'db_lbal_servers' in groups.keys()" 
+  #
+  #- name: Remove file
+  #  file:
+  #    path: /tmp/file.sql
+  #    state: absent

dev/provisioning/ansible/roles/mariadb/tasks/main.yml

@@ -1,20 +1,63 @@
 ---
 # tasks file for ansible-role-mariadb
 
+- name: Set db_main host variable
+  set_fact:
+    db_main: "{{ groups['db_servers'][0] }}"
+    my_service: {"name": "mariadb.service", "source": "systemd", "state": "unknown", "status": "disabled"}
+
 - name: Include OS specific variables.
   include_vars: "{{ ansible_os_family }}.yml"
 
+- name: collect facts about system services
+  service_facts:
+  register: services_state
+
+- name: Set db_main host variable
+  set_fact:
+    my_service: "{{ ansible_facts.services['mariadb.service'] }}"
+  when: "'mariadb.service' in ansible_facts.services.keys()"
+
+- name: Check Mariadb status
+  debug:
+    var: my_service
+
+      #- name: Ending if Mariadb is already up and running
+      #  meta: end_play
+      #  when: my_service.state != "unknown" and my_service.status != "disabled"
+
 - name: Install Mariadb
   include_tasks: "setup/{{ ansible_os_family }}.yml"
+  when: my_service.state == "unknown" and my_service.status == "disabled"
 
 - name: Ensure Mariadb configfile is present
   include_tasks: "config/template.yml"
+  when: my_service.state == "unknown" and my_service.status == "disabled"
 
 - name: Ensure Mariadb is secure
   include_tasks: "config/secure-installation.yml"
 
-- name: Ensure Mariadb databases are present
-  include_tasks: "database/databases.yml"
+- name: Creating a SELinux Policy
+  include_tasks: "selinux.yml"
+  when: groups['db_servers'] | length > 1 and ( my_service.state == "unknown" and my_service.status == "disabled" )
+
+- name: Bring Up the First Node
+  shell: galera_new_cluster
+  delegate_to: "{{ groups['db_servers'][0] }}"
+  run_once: true
+  when: groups['db_servers'] | length > 1 and ( my_service.state == "unknown" and my_service.status == "disabled" )
+
+- name: Start the others Nodes
+  service:
+    name: "{{ mariadb_service }}"
+    state: started
+  when: inventory_hostname == item and ( my_service.state == "unknown" and my_service.status == "disabled" )
+  with_items: "{{ groups['db_servers'] | reject('equalto', groups['db_servers'][0]) | list }}"
+  loop_control:
+    pause: 5
+
+      #- name: Ensure Mariadb databases are present
+      #  include_tasks: "database/databases.yml"
 
 - name: Ensure Mariadb users are present
   include_tasks: "database/users.yml"

dev/provisioning/ansible/roles/mariadb/tasks/selinux.yml

@@ -0,0 +1,85 @@
+---
+
+- name: Allow mysql to listen on tcp port 4567,4568,4444
+  seport:
+    ports: 4567,4568,4444
+    proto: tcp
+    setype: mysqld_port_t
+    state: present
+
+- name: Allow mysql to listen on udp port 4567
+  seport:
+    ports: 4567
+    proto: udp
+    setype: mysqld_port_t
+    state: present
+
+- name: Sets the MySQL SELinux domain to permissive mode temporarily
+  command: semanage permissive -a mysqld_t
+
+- name: Mariadb service
+  service:
+    name: "{{ mariadb_service }}"
+    state: stopped
+
+- name: Bootstrap the cluster
+  command: galera_new_cluster
+  delegate_to: "{{ groups['db_servers'][0] }}"
+
+- name: Create a database for the specific purpose of logging SST events
+  mysql_db:
+    name: selinux
+    state: present
+  delegate_to: "{{ groups['db_servers'][0] }}"
+  run_once: true
+
+- name: Create a table for the specific purpose of logging SST events
+  community.mysql.mysql_query:
+    login_db: selinux
+    query: "CREATE TABLE selinux.selinux_policy (id INT NOT NULL AUTO_INCREMENT, PRIMARY KEY(id));"
+  delegate_to: "{{ groups['db_servers'][0] }}"
+  run_once: true
+
+- name: Run insert queries against db selinux in single transaction
+  community.mysql.mysql_query:
+    login_db: selinux
+    query: "INSERT INTO selinux.selinux_policy VALUES ();"
+    single_transaction: yes
+  delegate_to: "{{ groups['db_servers'][0] }}"
+  run_once: true
+
+- name: Mariadb service
+  service:
+    name: "{{ mariadb_service }}"
+    state: started
+
+- name: Generate IST events
+  community.mysql.mysql_query:
+    login_db: selinux
+    query: "INSERT INTO selinux.selinux_policy VALUES ();"
+    single_transaction: yes 
+
+- name: Create and enable the SELinux policy
+  shell: "grep mysql /var/log/audit/audit.log | audit2allow -M Galera"
+  ignore_errors: yes
+
+- name: Build and install Galera.pp
+  shell: semodule -i Galera.pp
+
+- name: Disable permissive mode
+  shell: semanage permissive -d mysqld_t
+
+- name: Mariadb stop service
+  service:
+    name: "{{ mariadb_service }}"
+    state: stopped
+
+- name: Make Bootstrap safe
+  lineinfile:
+    dest: /var/lib/mysql/grastate.dat
+    regexp: "^{{ item.property | regex_escape() }}.*"
+    line: "{{ item.value }}"
+  with_items:
+    - { property: 'safe_to_bootstrap:', value: 'safe_to_bootstrap: 1' }
+  delegate_to: "{{ groups['db_servers'][0] }}"
+  run_once: true

dev/provisioning/ansible/roles/mariadb/tasks/setup/RedHat.yml

@@ -10,6 +10,16 @@
     name: "{{ mariadb_packages }}"
     state: present
 
+- name: Install rsync in case of Galera
+  dnf: 
+    name: rsync
+    state: present
+  when: groups['db_servers'] | length > 1
+
+- name: Set galera_wsrep_provider variable
+  set_fact:
+    galera_wsrep_provider: "/usr/lib64/galera-4/libgalera_smm.so"
+
 - name: Mariadb service
   service:
     name: "{{ mariadb_service }}"

dev/provisioning/ansible/roles/mariadb/templates/galera.j2

@@ -1,33 +1,22 @@
+# Galera-related settings
 #
-# These groups are read by MariaDB server.
-# Use it for options that only the server (but not clients) should see
-
-# this is read by the standalone daemon and embedded servers
-[server]
-
 [mysqld]
-bind-address = {{ mariadb_bind_address }}
-port = {{ mariadb_port }}
-
-{% if mariadb_slow_query_log_enabled == true %}
-slow_query_log = 1
-long_query_time = {{ mariadb_slow_query_time }}
-{% endif %}
-
-default-storage-engine = innodb
-
-# Galera-related settings
-[galera]
-wsrep_provider = {{ galera_wsrep_provider }}
-wsrep_cluster_address = "gcomm://{% for host in groups['galera_cluster'] %}{{ hostvars[host]['ansible_default_ipv4']['address'] }},{% endfor %}"
-wsrep_node_name = {{ ansible_hostname }}
-wsrep_cluster_name = {{ galera_cluster_name }}
-wsrep_node_address = {{ ansible_default_ipv4.address }}
 binlog_format = row
 default_storage_engine = InnoDB
 innodb_autoinc_lock_mode = 2
+bind-address = {{ mariadb_bind_address }}
+
+# Galera Provider Configuration
 wsrep_on = ON
+wsrep_provider = {{ galera_wsrep_provider }}
+
+# Galera Cluster Configuration 
+wsrep_cluster_name = {{ galera_cluster_name }}
+wsrep_cluster_address = "gcomm://{% for host in groups['db_servers'] %}{{ hostvars[host]['ansible_host'] }}{{ ',' if not loop.last }}{% endfor %}"
 
-[embedded]
+# Galera Synchronization Configuration 
+wsrep_sst_method = rsync
 
-[mariadb]
+# Galera Node Configuration 
+wsrep_node_address = {{ hostvars[ansible_hostname]['ansible_host'] }}
+wsrep_node_name = {{ ansible_hostname }}

dev/provisioning/ansible/roles/mariadb/templates/grant_all_hosts.sql.j2

@@ -0,0 +1,3 @@
+CREATE USER '{{ item.user }}'@'{{ item.host }}' IDENTIFIED BY '{{ item.user }}';
+GRANT USAGE ON *.* TO '{{ item.user }}'@'{{ item.host }}';
+FLUSH PRIVILEGES;

dev/provisioning/ansible/roles/mariadb/vars/RedHat.yml

@@ -7,6 +7,7 @@ mariadb_packages:
 
 mariadb_service: mariadb
 mariadb_config_file: /etc/my.cnf.d/server.cnf
+galera_config_file: /etc/my.cnf.d/galera.cnf
 mariadb_socket: /var/lib/mysql/mysql.sock
 mariadb_config_file_owner: root
 mariadb_config_file_group: root

dev/provisioning/ansible/roles/nextcloud/defaults/main.yml

@@ -1,44 +1,21 @@
 ---
 # [NEXTCLOUD CONFIG]
-ssl_name: "nextcloud.test"
-nc_trusted_domain: "127.0.0.1"
+nc_global_name: "nc.test"
 nextcloud_ipv6: false
-nc_multiple: ""
 
 # defaults file for nextcloud
-NEXTCLOUD_VERSION: 24.0.6 # overload from the role
+NEXTCLOUD_VERSION: "25.0.2"
 NEXTCLOUD_TARBALL: "nextcloud-{{ NEXTCLOUD_VERSION }}.tar.bz2"
 NEXTCLOUD_URL: "https://download.nextcloud.com/server/releases/{{ NEXTCLOUD_TARBALL }}"
 NEXTCLOUD_GPG: "https://nextcloud.com/nextcloud.asc"
 GPG_FINGERPRINT: "28806A878AE423A28372792ED75899B9A724937A"
 
-# [PHP CONFIG AND EXTENSIONS]
-php_version: 8.1
-PHP_POST_LIMIT: 50G
-PHP_UPLOAD_LIMIT: 25G
-PHP_MAX_FILE: 200
-PHP_MAX_TIME: 3600
-PHP_MEMORY_LIMIT: 512M
-
-APC_SHM_SIZE: 128M
-OPCACHE_MEM_SIZE: 128M
-
-add_php_fpm: true
-nc_pm: "ondemand"
-nc_pm_max_children: 80
-nc_pm_start_servers: 2
-nc_pm_min_spare_servers: 1
-nc_pm_max_spare_servers: 3
-
 # [REDIS CONFIG]
-use_redis_server: false    # overload from the role
 redis_host: "127.0.0.1"    # overload from the role
+redis_port: "6379"
 
 # [WEB CONFIG]
-nextcloud_install_websrv: true
-nextcloud_websrv: "apache2"  # "apache2" | "nginx"
-nextcloud_disable_websrv_default_site: false
-nextcloud_websrv_template: "templates/{{ nextcloud_websrv }}_nc.j2"
+ssl_path: "/etc/{{ ansible_fqdn }}/ssl"
 nc_data_dir: "/srv/data"
 nc_admin_name: "pedro"
 nc_admin_pwd: "pedro"
@@ -63,16 +40,17 @@ nextcloud_config_settings:
   - { name: 'mail_domain', value: 'uclouvain.be' }
   - { name: 'mail_smtphost', value: 'smtp.sgsi.ucl.ac.be' }
   - { name: 'mail_smtpauthtype', value: 'LOGIN' }
-  - { name: 'overwrite.cli.url', value: 'https://{{ ssl_name }}' }
-  - { name: 'overwritehost', value: '{{ ssl_name }}' }
+    #- { name: 'overwrite.cli.url', value: 'https://{{ nc_global_name }}' }
+    #- { name: 'overwritehost', value: '{{ nc_global_name }}' }
   - { name: 'overwriteprotocol', value: 'https' }
 
 #php /var/www/html/occ config:system:set share_folder --value="/Shared"
 
 # [DATABASE]
 db_host: "127.0.0.1" # overload from the role
+db_port: "3306"
 nc_db_name: "nextcloudb"
-nc_db_user: "web" # overload from the role
+nc_db_user: "nextcloudb" # overload from the role
 nc_db_password: "secret" # overload from the role
 
 # [APPS]

dev/provisioning/ansible/roles/nextcloud/tasks/main.yml

@@ -1,43 +1,86 @@
 ---
 # tasks file for nextcloud
 
-- include_tasks: "prep_os/{{ ansible_os_family }}.yml"
+- name: Main... Set Nextcloud db_port to ProxySQL port instead of default MySQL
+  set_fact:
+    db_port: "6033"
+  when: "'db_lbal_servers' in groups.keys()"
+
+- name: Main... Ovewrite Nextcloud db_host if no SQL_loadBalancer
+  set_fact:
+    db_host: "{{ groups['db_servers'][0] }}"
+  when: "'db_lbal_servers' not in groups.keys()"
+
+    #- name: Main... Set Nextcloud redis_host variable
+    #  set_fact:
+    #    redis_host: "{{ groups['redis_servers'][0] }}"
+    #  when: "'redis_servers' in groups.keys()"
+
+  #- name: Main... Check if Nexctloud's gonna deployed on multiple web servers
+  #debug: var=nc_multiple
+  #"{{ groups['all'] | length }}"
 
-- include_tasks: "prep_php/{{ ansible_os_family }}.yml"
+- name: Main... Show Selinux variable
+  debug: var=ansible_selinux
 
-- name: Main... Create shared directories
+- include_tasks: "prep_os/{{ ansible_os_family }}.yml"
+
+- name: Main... Create shared directories (web & data)
   file:
     path: "{{ item }}"
     state: directory
     owner: "{{ nextcloud_websrv_user }}"
     group: "{{ nextcloud_websrv_group }}"
-    mode: 0770
+    mode: 0750
   with_items:
     - "{{ nc_data_dir }}"
-    - "{{ http_webroot }}/nextcloud"
-  when: nc_multiple != ""
+    - "{{ http_webroot }}"
+  ignore_errors: yes
 
 - name: Main... Mount shared directories
-  mount:
-    src: "{{ nfs_server }}:{{ item.src }}"
-    path: "{{ item.path }}"
-    state: mounted
-    fstype: nfs 
-    opts: nosharecache,context="system_u:object_r:httpd_sys_rw_content_t:s0"
-  with_items:
-    - { src: "{{ nfs_data_path }}", path: "{{ nc_data_dir }}" }
-    - { src: "{{ nfs_web_path }}",  path: "{{ http_webroot }}/nextcloud" }
-  when: nc_multiple != ""
-
-- name: Main... Check if Nextcloud is downloaded
+  include_tasks: "nc_storage.yml"
+  #when: nc_multiple != ""
+  when: (groups['web_servers'] | length) > 1
+
+  #- name: Main... Check web directory
+  #  stat:
+  #    path: "{{ http_webroot }}"
+  #  register: is_http_webroot
+
+  #- name: Main... Check Selinux
+  #  include_tasks: "prep_os/selinux.yml"
+  #  when:
+  #    - (ansible_os_family == "RedHat")
+  #    - (ansible_selinux.status == "enabled")
+
+- name: Main... Check if Nextcloud is already in the web repo
   stat:
     path: "{{ http_webroot }}/nextcloud/index.php"
   register: nc_nextcloud_downloaded
+  delegate_to: "{{ groups['web_servers'][0] }}"
+  run_once: true
+
+    #- name: Main... Check is Set Nextcloud redis_host variable
+    #  set_fact:
+    #    nc_multiple: true
+    #  when: redis_host != ""
+    #
+    #- name: Main... is multiple nextcloud web servers ?
+    #  debug: var=nc_multiple
 
 - name: Main... Download and Install Nextcloud
   include_tasks: "nc_download.yml"
+  args:
+    apply:
+      delegate_to: "{{ groups['web_servers'][0] }}"
+  run_once: true
   when: (nc_nextcloud_downloaded.stat.isreg is undefined) or (not nc_nextcloud_downloaded.stat.isreg)
 
+- name: Main... Start {{ http_service_name }} service
+  service:
+    name: "{{ http_service_name }}"
+    state: started
+
 - name: Main... Check Nextcloud status
   become_user: "{{ nextcloud_websrv_user }}"
   become: true
@@ -45,34 +88,80 @@
   args:
     chdir: "{{ http_webroot }}/nextcloud"
   register: jsoncontent
+  delegate_to: "{{ groups['web_servers'][0] }}"
+  run_once: true
 
 - name: Main... Set Nextcloud variables status
   set_fact:
     nc_status: "{{ jsoncontent.stdout | from_json }}"
+  delegate_to: "{{ groups['web_servers'][0] }}"
+  run_once: true
 
-- name: Main... The nc_status content
+- name: Main... Check the nc_status content
   debug: var=nc_status
+  delegate_to: "{{ groups['web_servers'][0] }}"
+  run_once: true
 
-- name: Main... Install nextcloud
+- name: Main... Install and deploy Nextcloud if needed
   include_tasks: nc_install.yml
+  args:
+    apply:
+      delegate_to: "{{ groups['web_servers'][0] }}"
+  run_once: true
   when: nc_status.installed|bool == false
 
-- name: Main... Check Selinux
-  include_tasks: "selinux.yml"
-  when:
-    - (ansible_os_family == "RedHat")
-    - (ansible_selinux.status == "enabled")
-
 - name: Main... Setup nextcloud
   include_tasks: nc_setup.yml
-  when: nc_status.installed|bool == false
+  args:
+    apply:
+      delegate_to: "{{ groups['web_servers'][0] }}"
+  run_once: true
+  #when: nc_status.installed|bool == false
 
-    #- name: Setup... Set Trusted Local Domain
-    #  become_user: "{{ nextcloud_websrv_user }}"
-    #  become: true
-    #  shell: "{{ php_bin }} occ config:system:set trusted_domains 0 --value={{ nc_trusted_domain }}"
-    #  args:
-    #    chdir: "{{ http_webroot }}/nextcloud"
+- name: Main... Set Trusted Local Domain lists
+  set_fact:
+    list_zero:
+      - "{{ nc_global_name | default() }}"
+    list_one: "{{ groups['lbal_servers'] | default([]) }}"
+    list_two: "{{ groups['web_servers'] }}"
+
+- name: Main... Merge the Trusted lists
+  set_fact:
+    trusted_dom: "{{ list_zero + list_one + list_two }}"
+
+- name: Main... Set a local redirect name
+  set_fact:
+    redirect_name: "{{ groups['web_servers'][0] }}"
+
+- name: Main... Set a global redirect name if loadbalancing
+  set_fact:
+    redirect_name: "{{ nc_global_name | default(nc.test) }}"
+  when: groups['lbal_servers'] is defined and (groups['lbal_servers']|length>0)
+
+- name: Main... Set Trusted Local Domain
+  become_user: "{{ nextcloud_websrv_user }}"
+  become: true
+  shell: "{{ php_bin }} occ config:system:set trusted_domains {{ occ_idx }} --value={{ item }}"
+  args:
+    chdir: "{{ http_webroot }}/nextcloud"
+  loop: "{{ trusted_dom }}"
+  loop_control:
+    index_var: occ_idx
+    pause: 2
+  delegate_to: "{{ groups['web_servers'][0] }}"
+  run_once: true
+
+- name: Main... Set Nextcloud overwrite url and host
+  become_user: "{{ nextcloud_websrv_user }}"
+  become: true
+  shell: "{{ php_bin }} occ config:system:set {{ item.name }} --value={{ item.value }}"
+  args:
+    chdir: "{{ http_webroot }}/nextcloud"
+  with_items:
+    - { name: 'overwritehost', value: '{{ redirect_name }}' }
+    - { name: 'overwrite.cli.url', value: 'https://{{ redirect_name }}' }
+  delegate_to: "{{ groups['web_servers'][0] }}"
+  run_once: true
 
 - name: Setup... Configure Cron
   cron:
@@ -82,17 +171,14 @@
     job: "{{ php_bin }} -f {{ http_webroot }}/nextcloud/cron.php"
     cron_file: "nextcloud"
   when: (nc_background_cron | bool)
+  delegate_to: "{{ groups['web_servers'][0] }}"
+  run_once: true
 
-  #- name: Main... Setting stronger directories ownership
-  #  file:
-  #    path: "{{ item }}"
-  #    state: directory
-  #    owner: "{{ nextcloud_websrv_user }}"
-  #    group: "{{ nextcloud_websrv_group }}"
-  #    mode: 0770
-  #  with_items:
-  #    - "{{ nc_data_dir }}"
-  #    - "{{ http_webroot }}/nextcloud"
+- name: Main... Check Selinux
+  include_tasks: "prep_os/selinux.yml"
+  when:
+    - (ansible_os_family == "RedHat")
+    - (ansible_selinux.status == "enabled")
 
 - name: Main... Restart {{ http_service_name }} service
   service:
@@ -105,4 +191,5 @@
   shell: "{{ php_bin }} -f cron.php"
   args:
     chdir: "{{ http_webroot }}/nextcloud"
-
+  delegate_to: "{{ groups['web_servers'][0] }}"
+  run_once: true

dev/provisioning/ansible/roles/nextcloud/tasks/nc_install.yml

@@ -16,31 +16,33 @@
     group: "{{ nextcloud_websrv_group }}"
     mode: 0770
 
-- name: Install... Create data directory
-  file:
-    path: "{{ nc_data_dir }}"
-    state: directory
-    owner: "{{ nextcloud_websrv_user }}"
-    group: "{{ nextcloud_websrv_group }}"
-    mode: 0770
-  when: nc_multiple == ""
+      #- name: Install... Create data directory
+      #  file:
+      #    path: "{{ nc_data_dir }}"
+      #    state: directory
+      #    owner: "{{ nextcloud_websrv_user }}"
+      #    group: "{{ nextcloud_websrv_group }}"
+      #    mode: 0770
+      #  when: nc_multiple == ""
 
 - name: Install... Check if a mysql/mariadb database is available
-  shell: mysql --host={{ db_host }} --user={{ nc_db_user }} --password={{ nc_db_password }} -e 'SHOW DATABASES;' | grep -cx {{ nc_db_name }}
-  register: dbstatus
-  failed_when: "dbstatus.stdout|int != 1"
+  shell: mysql --host={{ db_host }} --user={{ nc_db_user }} --password={{ nc_db_password }} --port={{ db_port }} -e "show status like 'Connections%';"
   no_log: true
 
 - name: Install... First setup Nextcloud
   become_user: "{{ nextcloud_websrv_user }}"
   become: true
-  shell: "{{ php_bin }} occ maintenance:install --database=mysql --database-host={{ db_host }} --database-name={{ nc_db_name }} --database-user={{ nc_db_user }} --database-pass={{ nc_db_password }} --admin-user={{ nc_admin_name }} --admin-pass={{ nc_admin_pwd }} --data-dir={{ nc_data_dir }}"
+  shell: "{{ php_bin }} occ maintenance:install --database=mysql --database-host={{ db_host }} --database-port={{ db_port }} --database-name={{ nc_db_name }} --database-user={{ nc_db_user }} --database-pass={{ nc_db_password }} --admin-user={{ nc_admin_name }} --admin-pass={{ nc_admin_pwd }} --data-dir={{ nc_data_dir }}"
   args:
     chdir: "{{ http_webroot }}/nextcloud"
     creates: "{{ http_webroot }}/nextcloud/config/config.php"
   register: setup_nc
+  #no_log: true
 
 - name: Install... Removing possibly sample config
   file:
-    path: "{{ http_webroot }}/nextcloud/config/config.sample.php"
+    path: "{{ item }}"
     state: absent
+  with_items:
+    - "{{ http_webroot }}/nextcloud/config/config.sample.php"
+    - "{{ http_webroot }}/nextcloud/config/CAN_INSTALL"

dev/provisioning/ansible/roles/nextcloud/tasks/nc_setup.yml

@@ -16,20 +16,6 @@
     group: "{{ nextcloud_websrv_group }}"
     mode: 0640
 
-    #- name: Setup... Set Trusted Local Domain
-    #  become_user: "{{ nextcloud_websrv_user }}"
-    #  become: true
-    #  shell: "{{ php_bin }} occ config:system:set trusted_domains 0 --value={{ nc_trusted_domain }}"
-    #  args:
-    #    chdir: "{{ http_webroot }}/nextcloud"
-
-- name: Setup... Set Master SSL Nextcloud Domain
-  become_user: "{{ nextcloud_websrv_user }}"
-  become: true
-  shell: "{{ php_bin }} occ config:system:set trusted_domains 1 --value={{ ssl_name }}"
-  args:
-    chdir: "{{ http_webroot }}/nextcloud"
-
 - name: Setup... Check disabled apps list
   shell: "{{ php_bin }} occ app:list --no-warnings | grep -A30 'Disabled' | grep -v 'Disabled' | cut -d'-' -f2 | cut -d':' -f1 | grep -v 'encryption'"
   args:
@@ -111,14 +97,13 @@
   with_items:
     - "{{ nextcloud_config_settings }}"
 
-- name: Setup... Set Redis Server
+- name: Setup... Set Redis setup
   template:
     dest: "{{ http_webroot }}/nextcloud/config/redis.config.php" 
     src: redis.config.php.j2
     owner: "{{ nextcloud_websrv_user }}"
     group: "{{ nextcloud_websrv_group }}"
     mode: 0640
-  when: (use_redis_server | bool)
 
 - name: Setup... Install Nextcloud Apps
   become_user: "{{ nextcloud_websrv_user }}"
@@ -157,15 +142,27 @@
     - /etc/coolwsd
   when: nc_collabora
 
-- name: Setup... Ensure Nextcloud directories are 0750
-  shell: find {{ http_webroot }}/nextcloud -type d -exec chmod -c 0750 {} \;
-  register: nc_installation_chmod_result
-  changed_when: "nc_installation_chmod_result.stdout != \"\""
-
-- name: Setup... Ensure Nextcloud files are 0640
-  shell: find {{ http_webroot }}/nextcloud -type f -exec chmod -c 0640 {} \;
-  register: nc_installation_chmod_result
-  changed_when: "nc_installation_chmod_result.stdout != \"\""
+  #- name: Main... Setting stronger directories ownership
+  #  file:
+  #    path: "{{ item }}"
+  #    state: directory
+  #    owner: "{{ nextcloud_websrv_user }}"
+  #    group: "{{ nextcloud_websrv_group }}"
+  #    recurse: yes
+  #    mode: 0750
+  #  with_items:
+  #    - "{{ nc_data_dir }}"
+  #    - "{{ http_webroot }}"
+
+      #- name: Setup... Ensure Nextcloud directories are 0750
+      #  shell: find {{ http_webroot }}/nextcloud -type d -exec chmod -c 0750 {} \;
+      #  register: nc_installation_chmod_result
+      #  changed_when: "nc_installation_chmod_result.stdout != \"\""
+      #
+      #- name: Setup... Ensure Nextcloud files are 0640
+      #  shell: find {{ http_webroot }}/nextcloud -type f -exec chmod -c 0640 {} \;
+      #  register: nc_installation_chmod_result
+      #  changed_when: "nc_installation_chmod_result.stdout != \"\""
 
       ###- name: Setup... "[NC] Setting stronger directory ownership"
       ###  ansible.builtin.file:

dev/provisioning/ansible/roles/nextcloud/tasks/nc_storage.yml

@@ -0,0 +1,85 @@
+- name: Storage... mount glusterFS_1
+  mount:
+    src: "{{ groups['gluster_servers'][0] }}:ncgluster1"
+    path: "{{ nc_data_dir }}"
+    fstype: glusterfs
+    opts: acl
+    state: mounted
+
+#- name: Main... Mount shared directories
+#  mount:
+#    src: "{{ nfs_server }}:{{ item.src }}"
+#    path: "{{ item.path }}"
+#    state: mounted
+#    fstype: nfs 
+#    opts: nosharecache,context="system_u:object_r:httpd_sys_rw_content_t:s0"
+#  with_items:
+#    - { src: "{{ nfs_data_path }}", path: "{{ nc_data_dir }}" }
+#    - { src: "{{ nfs_web_path }}",  path: "{{ http_webroot }}/nextcloud" }
+#  when: nc_multiple != ""
+
+- name: Storage... acl glusterFS_1
+  acl:
+    path: "{{ nc_data_dir }}"
+    entry: "user:{{ nextcloud_websrv_user }}:rwX"
+    state: present
+
+- name: Storage... acl default glusterFS_1
+  acl:
+    path: "{{ nc_data_dir }}"
+    entry: "default:user:{{ nextcloud_websrv_user }}:rwX"
+    state: present
+
+- name: Storage... permission glusterFS_1
+  file:
+    owner: "{{ nextcloud_websrv_user }}"
+    group: "{{ nextcloud_websrv_group }}"
+    path: "{{ nc_data_dir }}"
+
+- name: Storage... mount glusterFS_2
+  mount:
+    src: "{{ groups['gluster_servers'][0] }}:ncgluster2"
+    path: "{{ http_webroot }}"
+    fstype: glusterfs
+    opts: acl
+    state: mounted
+
+- name: Storage... acl glusterFS_2
+  acl:
+    path: "{{ http_webroot }}"
+    entry: "user:{{ nextcloud_websrv_user }}:rwX"
+    state: present
+
+- name: Storage... acl default glusterFS_2
+  acl:
+    path: "{{ http_webroot }}"
+    entry: "default:user:{{ nextcloud_websrv_user }}:rwX"
+    state: present
+
+- name: Storage... permission glusterFS_2
+  file:
+    owner: "{{ nextcloud_websrv_user }}"
+    group: "{{ nextcloud_websrv_group }}"
+    path: "{{ http_webroot }}"
+
+      #- name: Storage... Ensure Nextcloud directories are 0750
+      #  shell: find {{ http_webroot }}/nextcloud -type d -exec chmod -c 0750 {} \;
+      #  register: nc_installation_chmod_result
+      #  changed_when: "nc_installation_chmod_result.stdout != \"\""
+      #
+      #- name: Storage... Ensure Nextcloud files are 0640
+      #  shell: find {{ http_webroot }}/nextcloud -type f -exec chmod -c 0640 {} \;
+      #  register: nc_installation_chmod_result
+      #  changed_when: "nc_installation_chmod_result.stdout != \"\""
+
+      #- name: Storage... Setting stronger directories ownership
+      #  file:
+      #    path: "{{ item }}"
+      #    state: directory
+      #    owner: "{{ nextcloud_websrv_user }}"
+      #    group: "{{ nextcloud_websrv_group }}"
+      #    recurse: yes
+      #    mode: 0750
+      #  with_items:
+      #    - "{{ nc_data_dir }}"
+      #    - "{{ http_webroot }}"

dev/provisioning/ansible/roles/nextcloud/tasks/prep_os/CentOS.yml

@@ -1,6 +1,29 @@
 ---
 # CentOS related tasks
 
+- name: Prep OS... Check web server installed
+  shell: "httpd -v"
+  register: web_apache
+  ignore_errors: yes
+
+- name: Prep OS... Check nginx installed
+  shell: "nginx -v"
+  register: web_nginx
+  ignore_errors: yes
+  when: web_apache.stderr | length > 0
+
+- name: Prep OS... Set nextcloud_websrv variable
+  set_fact:
+    nextcloud_websrv: "apache"
+
+- name: Prep OS... Set nextcloud_websrv variable
+  set_fact:
+    nextcloud_websrv: "nginx"
+  when: web_apache.stderr | length > 0
+
+- name: Prep OS... display web server
+  debug: var=nextcloud_websrv
+
 #- name: Prep OS... Create tmp directory
 #  file:
 #    path: "{{ item }}"
@@ -34,6 +57,12 @@
     baseurl: https://www.collaboraoffice.com/repos/CollaboraOnline/CODE-centos{{ ansible_distribution_major_version|int }}
   when: nc_collabora
 
+- name: Prep OS... update os
+  dnf:
+    name: '*'
+    update_cache: true
+    state: latest
+
 - name: Prep OS... install Collabora packages
   dnf:
     name:
@@ -45,27 +74,14 @@
     state: latest
   when: nc_collabora
 
-- name: Prep OS... update os
-  dnf:
-    name: '*'
-    update_cache: true
-    state: latest
-
 - name: Prep OS... install needed packages
   dnf:
     name:
       - libreoffice
       - ffmpeg
-      - mariadb
     state: latest
     enablerepo: epel
 
-- name: Prep OS... Ensure Apache is installed on {{ ansible_facts['distribution'] }}
-  dnf:
-    name: httpd
-    state: present
-  when: nextcloud_websrv in ["apache", "apache2"]
-
 - name: Prep OS... Set http env on {{ ansible_facts['distribution'] }}
   set_fact:
     http_service_name: httpd
@@ -74,26 +90,38 @@
     nextcloud_websrv_group: apache
   when: nextcloud_websrv in ["apache", "apache2"]
 
+- name: Prep OS... Get httpd version
+  shell: "httpd -v | head -n1 | cut -d'/' -f2 | cut -d' ' -f1"
+  register: web_version
+  when: nextcloud_websrv in ["apache", "apache2"]
+
+- name: Prep OS... Get SSL version
+  shell: "openssl version | cut -d' ' -f2 | cut -d'-' -f1"
+  register: ssl_version
+
+- name: Prep OS... Set web and SSL status
+  set_fact:
+    web_status: "{{ web_version.stdout|string is version('2.4.8', '>=') }}"
+    ssl_status: "{{ ssl_version.stdout|string is version('1.1.1', '>') }}"
+
+- name: Prep OS... Set php env for {{ ansible_facts['distribution'] }}
+  set_fact: 
+    php_bin: /usr/bin/php
+    #php_dir: "/etc/opt/remi/php{{ php_version | replace('.','') }}/php.d"
+
+- include_tasks: "ssl.yml"
+
 - name: Prep OS... Generate Nextcloud configuration for apache
   template:
     dest: /etc/httpd/conf.d/nextcloud.conf
-    src: nextcloud_apache2.j2
+    src: apache2_nc_conf.j2
+    #src: apache2_nc_conf_old.j2
     mode: 0640
   when: nextcloud_websrv in ["apache", "apache2"]
 
-- name: Prep OS... Allow http to listen on tcp port 8000
-  seport:
-    ports: 8000
-    proto: tcp
-    setype: http_port_t
-    state: present
-
-    #- name: Prep OS... semanage port
-    #  command: semanage port -m -t http_port_t -p tcp {{ item }}
-    #  loop:
-    #    - "8000"
+- name: Prep OS... Install glusterFS client
+  dnf:
+    name: glusterfs-fuse
+    state: latest
+  when: (groups['web_servers'] | length) > 1
 
-- name: Prep OS... Start {{ http_service_name }} service
-  service:
-    name: "{{ http_service_name }}"
-    state: started

dev/provisioning/ansible/roles/nextcloud/tasks/prep_os/RedHat.yml

@@ -1,3 +1,4 @@
 ---
 
 - include_tasks: "{{ ansible_facts['distribution'] }}.yml"
+

dev/provisioning/ansible/roles/nextcloud/tasks/selinux.yml → dev/provisioning/ansible/roles/nextcloud/tasks/prep_os/selinux.yml

@@ -22,14 +22,18 @@
     state: yes
     persistent: yes
   with_items:
-    - httpd_can_sendmail
     - httpd_unified
     - httpd_graceful_shutdown
     - httpd_can_network_relay
     - httpd_can_network_connect
     - httpd_can_network_connect_db
     - daemons_enable_cluster_mode
-    #- httpd_execmem
+    - httpd_use_fusefs
+    - httpd_use_cifs
+    - httpd_use_gpg
+    - httpd_use_nfs
+    - httpd_execmem
+    - httpd_can_sendmail
 
       ###- name: Selinux... enable seboolean settings
       ###  command: semodule -i {{ role_path }}/files/{{ item }}
@@ -38,6 +42,12 @@
       ###    - httpd-to-redis-socket.pp
       ###    - httpd-to-upload-tmp.pp
 
+# if you have trouble with php-fpm and selinux in this nextcloud configuration :
+# # ausearch -c 'php-fpm' --raw | audit2allow -M my-phpfpm
+# # semodule -X 300 -i my-phpfpm.pp
+# # ausearch -c 'df' --raw | audit2allow -M my-df
+# # semodule -X 300 -i my-df.pp
+
 - name: Selinux... Run restore context to reload selinux
   shell: restorecon -R -v {{ item.target }}
   when: filecontext.results[item.index] is changed

dev/provisioning/ansible/roles/nextcloud/tasks/prep_os/ssl.yml

@@ -0,0 +1,50 @@
+---
+- name: SSL... Create ssl certificates Directory
+  file:
+    dest: "{{ ssl_path }}"
+    owner: root
+    group: root
+    state: directory
+    recurse: yes
+
+- name: SSL... Create private key (RSA, 4096 bits)
+  openssl_privatekey:
+    path: "{{ ssl_path }}/{{ ansible_fqdn }}.key"
+
+- name: SSL... Check if CRT exists
+  stat:
+    path: "{{ ssl_path }}/{{ ansible_fqdn }}.crt"
+  register: result_crt
+
+- name: SSL... Create certificate signing request (CSR) for self-signed certificate
+  community.crypto.openssl_csr_pipe:
+    privatekey_path: "{{ ssl_path }}/{{ ansible_fqdn }}.key"
+    country_name: BE
+    locality_name: Louvain-la-Neuve
+    common_name: "{{ ansible_fqdn }}"
+    organization_name: UCLouvain
+    organizational_unit_name: ELIC
+  register: csr
+  when: (result_crt.stat.isreg is undefined) or (not result_crt.stat.isreg)
+
+- name: SSL... Generate a Self Signed OpenSSL certificate
+  community.crypto.x509_certificate:
+    path: "{{ ssl_path }}/{{ ansible_fqdn }}.crt"
+    csr_content: "{{ csr.csr }}"
+    privatekey_path: "{{ ssl_path }}/{{ ansible_fqdn }}.key"
+    provider: selfsigned
+  when: (result_crt.stat.isreg is undefined) or (not result_crt.stat.isreg)
+
+    #- name: SSL... Remove previous PEM file
+    #  file:
+    #    path: "{{ ssl_path }}/{{ ansible_fqdn }}.pem"
+    #    state: absent
+
+- name: SSL... Merge KEY and CRT to generate PEM
+  shell: "echo '' > {{ ssl_path }}/{{ ansible_fqdn }}.pem; cat {{ ssl_path }}/{{ ansible_fqdn }}.key {{ ssl_path }}/{{ ansible_fqdn }}.crt >> {{ ssl_path }}/{{ ansible_fqdn }}.pem"
+
+- name: SSL... Generate DH Parameters with a different size (2048 bits)
+  community.crypto.openssl_dhparam:
+    path: "{{ ssl_path }}/dhparams.pem"
+    size: 2048
+  register: dhparam

dev/provisioning/ansible/roles/nextcloud/templates/apache2_nc.j2

@@ -1,103 +0,0 @@
-################################################################################
-# This file was generated by Ansible for {{ansible_fqdn}}
-# Do NOT modify this file by hand!
-################################################################################
-
-{% if nextcloud_install_tls and nextcloud_tls_enforce %}
-{% for domain in nextcloud_trusted_domain %}
-<VirtualHost *:80>
-  ServerName {{ domain }}
-  Redirect permanent / https://{{ domain | ansible.utils.ipwrap }}/
-</VirtualHost>
-{% endfor %}
-{% else %}
-<VirtualHost *:80>
-  ServerName {{ nextcloud_trusted_domain[0] }}
-{% for index in range(1, nextcloud_trusted_domain|length) %}
-  ServerAlias {{ nextcloud_trusted_domain[index]}}
-{% endfor %}
-  DocumentRoot {{ nextcloud_webroot }}
-  {% if (nextcloud_max_upload_size_in_bytes|int) <= 2147483647-%}
-  LimitRequestBody {{ nextcloud_max_upload_size_in_bytes }}
-  {% endif -%}
-  <Directory {{ nextcloud_webroot }}>
-    Allow from all
-    Satisfy Any
-    Options +FollowSymlinks
-    AllowOverride All
-
-   <IfModule mod_dav.c>
-    Dav off
-   </IfModule>
-
-   SetEnv HOME {{ nextcloud_webroot }}
-   SetEnv HTTP_HOME {{ nextcloud_webroot }}
-
-  </Directory>
-</VirtualHost>
-{% endif %}
-
-{% if nextcloud_install_tls %}
-<VirtualHost *:443>
-  ServerName {{ nextcloud_trusted_domain[0] }}
-{% for index in range(1, nextcloud_trusted_domain|length) %}
-  ServerAlias {{ nextcloud_trusted_domain[index]}}
-{% endfor %}
-  DocumentRoot {{ nextcloud_webroot }}
-  {% if (nextcloud_max_upload_size_in_bytes|int) <= 2147483647-%}
-  LimitRequestBody {{ nextcloud_max_upload_size_in_bytes }}
-  {% endif -%}
-  SSLEngine on
-  SSLCertificateFile      {{ nextcloud_tls_cert_file }}
-  SSLCertificateKeyFile   {{ nextcloud_tls_cert_key_file }}
-{% if nextcloud_tls_cert_chain_file is defined %}
-  SSLCertificateChainFile {{ nextcloud_tls_cert_chain_file }}
-{% endif %}
-
-  # enable HTTP/2, if available
-  Protocols h2 http/1.1
-
-{% if nextcloud_hsts is string %}
-  <IfModule mod_headers.c>
-    Header always set Strict-Transport-Security "{{ nextcloud_hsts }}"
-  </IfModule>
-{% endif %}
-
-  <Directory {{ nextcloud_webroot }}>
-    Allow from all
-    Satisfy Any
-    Options +FollowSymlinks
-    AllowOverride All
-
-   <IfModule mod_dav.c>
-    Dav off
-   </IfModule>
-
-   SetEnv HOME {{ nextcloud_webroot }}
-   SetEnv HTTP_HOME {{ nextcloud_webroot }}
-
-  </Directory>
-</VirtualHost>
-{% endif %}
-
-{% if nextcloud_install_tls %}
-{% if nextcloud_mozilla_modern_ssl_profile %}
-# modern configuration, tweak to your needs
-SSLProtocol             all -SSLv3 -TLSv1 -TLSv1.1 -TLSv1.2
-{% else %}
-# intermediate configuration, tweak to your needs
-SSLProtocol             all -SSLv3 -TLSv1 -TLSv1.1
-SSLCipherSuite          ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384
-{% endif %}
-
-SSLHonorCipherOrder     off
-# SSLSessionTickets       off
-
-SSLCompression          off
-
-# OCSP stapling
-SSLUseStapling          on
-SSLStaplingResponderTimeout 5
-SSLStaplingReturnResponderErrors off
-SSLStaplingCache        shmcb:/var/run/ocsp(128000)
-{% endif %}

dev/provisioning/ansible/roles/nextcloud/templates/apache2_nc_conf.j2

@@ -0,0 +1,66 @@
+<VirtualHost *:80>
+  DocumentRoot {{ http_webroot }}/nextcloud
+  RewriteEngine On
+  RewriteCond %{SERVER_NAME} ={{ ansible_fqdn }}
+  RewriteRule ^ https://%{SERVER_NAME}%{REQUEST_URI} [END,NE,R=permanent]
+</VirtualHost>
+
+<IfModule mod_ssl.c>
+  <VirtualHost *:443>
+    SSLEngine on
+    SSLOptions +StrictRequire
+    LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-agent}i\"" vhost_combined
+    LogFormat "%v %h %l %u %t \"%r\" %>s %b" vhost_common
+    ServerName {{ ansible_fqdn }}
+    ServerAdmin admin@{{ ansible_fqdn }}
+    DocumentRoot {{ http_webroot }}/nextcloud
+    SSLCertificateFile {{ ssl_path }}/{{ ansible_fqdn }}.crt
+    SSLCertificateKeyFile {{ ssl_path }}/{{ ansible_fqdn }}.key
+    <Directory {{ http_webroot }}/nextcloud/>
+      Options +FollowSymlinks
+      AllowOverride All
+      <IfModule mod_dav.c>
+        Dav off
+      </IfModule>
+      <Files ".ht*">
+        Require all denied
+      </Files>
+      SetEnv HOME {{ http_webroot }}/nextcloud
+      SetEnv HTTP_HOME {{ http_webroot }}/nextcloud
+
+      # Fix zero file sizes
+      # See https://github.com/nextcloud/server/issues/3056#issuecomment-954209565
+      SetEnv proxy-sendcl 1
+
+      # See https://httpd.apache.org/docs/current/en/mod/core.html#limitrequestbody
+      LimitRequestBody 0
+    </Directory>
+    <IfModule mod_headers.c>
+      Header always set Strict-Transport-Security "max-age=15768000; preload"
+      #Header set Referrer-Policy "strict-origin-when-cross-origin"
+      #Header set X-Content-Type-Options "nosniff"
+      #Header always set X-Frame-Options "SAMEORIGIN"
+    </IfModule>
+  </VirtualHost>
+
+  {% if ssl_status is sameas true %}
+  SSLProtocol -all +TLSv1.3 +TLSv1.2
+  {% else %}
+  SSLProtocol -all +TLSv1.2
+  {% endif %}
+  SSLCipherSuite TLS-CHACHA20-POLY1305-SHA256:TLS-AES-256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA512:DHE-RSA-AES256-GCM-SHA512:ECDHE-RSA-AES256-GCM-SHA384:DHE-RSA-AES256-GCM-SHA384
+  SSLHonorCipherOrder on
+  SSLCompression off
+  SSLSessionTickets off
+  SSLUseStapling off
+  SSLStaplingResponderTimeout 5
+  SSLStaplingReturnResponderErrors off
+  SSLStaplingCache shmcb:/var/run/ocsp(128000)
+  {% if web_status is sameas true %}
+  SSLOpenSSLConfCmd Curves X448:secp521r1:secp384r1:prime256v1
+  SSLOpenSSLConfCmd ECDHParameters secp384r1
+  SSLOpenSSLConfCmd DHParameters "{{ ssl_path }}/dhparams.pem"
+  {% endif %}  
+  SSLRandomSeed startup file:/dev/urandom 1024
+  SSLRandomSeed connect file:/dev/urandom 1024
+</IfModule>

dev/provisioning/ansible/roles/nextcloud/templates/apache2_nc_conf_old.j2

@@ -0,0 +1,26 @@
+<VirtualHost *:80>
+  ServerName {{ ansible_fqdn }}
+  ServerAdmin admin@{{ ansible_fqdn }}
+  DocumentRoot {{ http_webroot }}/nextcloud
+  <Directory {{ http_webroot }}/nextcloud/>
+    Options Indexes FollowSymLinks
+    Require all granted
+    AllowOverride All
+    Options FollowSymLinks MultiViews
+      
+  <IfModule mod_dav.c>
+    Dav off
+  </IfModule>
+  </Directory>
+  # Deny access to .ht files
+  <Files ".ht*">
+    Require all denied
+  </Files>
+
+  # Fix zero file sizes 
+  # See https://github.com/nextcloud/server/issues/3056#issuecomment-954209565
+  SetEnv proxy-sendcl 1
+
+  # See https://httpd.apache.org/docs/current/en/mod/core.html#limitrequestbody
+  LimitRequestBody 0
+</VirtualHost>

dev/provisioning/ansible/roles/nextcloud/templates/nc_config.php.j2

@@ -1,14 +0,0 @@
-<?php
-$CONFIG = array (
-  'datadirectory' => '{{ nc_data_dir }}',
-  'dbtype' => 'mysql',
-  'version' => '{{ nc_status.version }}',
-  'dbname' => '{{ nc_db_name }}',
-  'dbhost' => '{{ db_host }}',
-  'dbport' => '',
-  'dbtableprefix' => 'oc_',
-  'mysql.utf8mb4' => true,
-  'dbuser' => '{{ nc_db_user }}',
-  'dbpassword' => '{{ nc_db_password }}',
-  'installed' => true,
-);

dev/provisioning/ansible/roles/nextcloud/templates/nextcloud_apache2.j2

@@ -1,26 +0,0 @@
-Listen 8000
-<VirtualHost *:8000>
-    # Nextcloud dir
-    DocumentRoot {{ http_webroot }}/nextcloud/
-    <Directory {{ http_webroot }}/nextcloud/>
-        Options Indexes FollowSymLinks
-        Require all granted
-        AllowOverride All
-        Options FollowSymLinks MultiViews
-        
-        <IfModule mod_dav.c>
-            Dav off
-        </IfModule>
-    </Directory>
-    # Deny access to .ht files
-    <Files ".ht*">
-        Require all denied
-    </Files>
-
-    # Fix zero file sizes 
-    # See https://github.com/nextcloud/server/issues/3056#issuecomment-954209565
-    SetEnv proxy-sendcl 1
-
-    # See https://httpd.apache.org/docs/current/en/mod/core.html#limitrequestbody
-    LimitRequestBody 0
-</VirtualHost>

dev/provisioning/ansible/roles/nextcloud/templates/redis.config.php.j2

@@ -1,9 +1,14 @@
 <?php
 $CONFIG = array (
-  'memcache.distributed' => '\OC\Memcache\Redis',
+  'filelocking.enabled' => true,
   'memcache.locking' => '\OC\Memcache\Redis',
   'redis' => array (
     'host' => '{{ redis_host }}',
-    'port' => 6379,
+    'port' => {{ redis_port }},
+    'dbindex'  => 0,
+    'timeout' => 1.5,
   ),
+{% if (groups['web_servers'] | length) > 10 %}
+  'memcache.distributed' => '\OC\Memcache\Redis',
+{% endif %}
 );

dev/provisioning/ansible/roles/nextcloud/templates/root-my.cnf.j2

@@ -1,5 +0,0 @@
-{{ ansible_managed | comment }}
-
-[client]
-user="root"
-password="{{ nextcloud_mysql_root_pwd }}"

dev/provisioning/ansible/roles/proxysql/defaults/main.yml

@@ -0,0 +1,19 @@
+---
+mysql_user_name: admin
+mysql_user_password: pedro
+
+mysql_root_username: root
+mysql_root_password: pedro
+
+# Specify address to listen
+mariadb_bind_address: '0.0.0.0'
+mariadb_port: 3306
+
+# Add mariabd users
+# (replaced with global vars when calling role)
+db_user_name: 'web'
+db_user_passwd: 'secret'
+
+keepalived_priority: 101
+keepalived_state: "MASTER"
+keepalived_vip: "192.168.56.19"

dev/provisioning/ansible/roles/proxysql/files/galera_check.pl

@@ -0,0 +1,2765 @@
+#!/usr/bin/perl
+# This tool is "fat-packed": most of its dependent modules are embedded
+# in this file.
+# https://github.com/Tusamarco/proxy_sql_tools/blob/master/galera_check.pl
+#######################################
+#
+# ProxySQL galera check v1 
+#
+# Author Marco Tusa 
+# Copyright (C) (2016 - 2020)
+# 
+#
+#THIS PROGRAM IS PROVIDED "AS IS" AND WITHOUT ANY EXPRESS OR IMPLIED
+#WARRANTIES, INCLUDING, WITHOUT LIMITATION, THE IMPLIED WARRANTIES OF
+#MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE.
+#
+#This program is free software; you can redistribute it and/or modify it under
+#the terms of the GNU General Public License as published by the Free Software
+#Foundation, version 2; OR the Perl Artistic License.  On UNIX and similar
+#systems, you can issue `man perlgpl' or `man perlartistic' to read these
+#licenses.
+#
+#You should have received a copy of the GNU General Public License along with
+#this program; if not, write to the Free Software Foundation, Inc., 59 Temple
+#Place, Suite 330, Boston, MA  02111-1307  USA.
+
+#######################################
+
+package galera_check ;
+use Time::HiRes qw(gettimeofday);
+use strict;
+use DBI;
+use Getopt::Long;
+use Pod::Usage;
+
+
+$Getopt::Long::ignorecase = 0;
+my $Param = {};
+my $user = "admin";
+my $pass = "admin";
+my $help = '';
+my $host = '' ;
+my $debug = 0 ;
+my %hostgroups;
+my $mysql_connect_timeout=6;
+
+my %processState;
+my %processCommand;
+my @HGIds;
+
+
+
+
+######################################################################
+#Local functions
+######################################################################
+
+sub URLDecode {
+    my $theURL = $_[0];
+    $theURL =~ tr/+/ /;
+    $theURL =~ s/%([a-fA-F0-9]{2,2})/chr(hex($1))/eg;
+    $theURL =~ s/<!--(.|\n)*-->//g;
+    return $theURL;
+}
+sub URLEncode {
+    my $theURL = $_[0];
+   $theURL =~ s/([\W])/"%" . uc(sprintf("%2.2x",ord($1)))/eg;
+   return $theURL;
+}
+
+# return a proxy object
+sub get_proxy($$$$){
+    my $dns = shift;
+    my $user = shift;
+    my $pass = shift;
+    my $debug = shift;
+    my $proxynode = ProxySqlNode->new();
+    $proxynode->dns($dns);
+    $proxynode->user($user);
+    $proxynode->password($pass);
+    $proxynode->debug($debug);
+    
+    return $proxynode;
+    
+}
+
+ sub main{
+    # ============================================================================
+    #+++++ INITIALIZATION
+    # ============================================================================
+    if($#ARGV < 0){
+        pod2usage(-verbose => 2) ;
+       	exit 1;
+    }
+    
+    if($#ARGV < 3){
+      #given a ProxySQL scheduler
+      #limitation we will pass the whole set of params as one
+      # and will split after
+      @ARGV = split('\ ',$ARGV[0]);
+    }
+    
+    $Param->{user}       = '';
+    $Param->{log}       = undef ;
+    $Param->{password}   = '';
+    $Param->{host}       = '';
+    $Param->{port}       = 3306;
+    $Param->{debug}      = 0; 
+    $Param->{processlist} = 0;
+    $Param->{OS} = $^O;
+    $Param->{main_segment} = 1;
+    $Param->{retry_up} = 0;
+    $Param->{retry_down} = 0;
+    $Param->{print_execution} = 1;
+    $Param->{development} = 0;
+    $Param->{development_time} = 2;
+    $Param->{active_failover} = 0;
+    $Param->{single_writer} = 1;
+    $Param->{writer_is_reader} = 1;
+    $Param->{check_timeout} = 800;
+    $Param->{ssl_certs_path} = undef;
+    
+    my $run_pid_dir = "/tmp" ;
+    
+    #if (
+    GetOptions(
+        'user|u:s'       => \$Param->{user},
+        'password|p:s'   => \$Param->{password},
+        'host|h:s'       => \$host,
+        'port|P:i'       => \$Param->{port},
+        'debug|d:i'      => \$Param->{debug},
+        'log:s'      => \$Param->{log},
+        'hostgroups|H:s'=> \$Param->{hostgroups},
+        'main_segment|S:s'=> \$Param->{main_segment},
+        'retry_up:i' =>	\$Param->{retry_up},
+        'retry_down:i' =>	\$Param->{retry_down},
+        'execution_time:i' => \$Param->{print_execution},
+        'development:i' => \$Param->{development},
+        'development_time:i' => \$Param->{development_time},
+        'single_writer:i' => \$Param->{single_writer},
+        'writer_is_also_reader:i' => \$Param->{writer_is_reader},        
+        'active_failover:i' => \$Param->{active_failover},
+        'check_timeout:i' => \$Param->{check_timeout},
+        'ssl_certs_path:s' => \$Param->{ssl_certs_path},
+        
+        'help|?'       => \$Param->{help}
+       
+    ) or pod2usage(2);
+    pod2usage(-verbose => 2) if $Param->{help};
+
+    die print Utils->print_log(1,"Option --hostgroups not specified.\n") unless defined($Param->{hostgroups});
+    die print Utils->print_log(1,"Option --host not specified.\n") unless defined $Param->{host};
+    die print Utils->print_log(1,"Option --user not specified.\n") unless defined $Param->{user};
+    die print Utils->print_log(1,"Option --port not specified.\n") unless defined $Param->{port};
+    die print Utils->print_log(1,"Option --active_failover has an invalid value ($Param->{active_failover}).\n"
+                               ."Valid values are:\n"
+                               ." 0 [default] do not make failover\n"
+                               ." 1 make failover only if HG 8000 is specified in ProxySQL mysl_servers\n"
+                               ." 2 use PXC_CLUSTER_VIEW to identify a server in the same segment\n"
+                               ." 3 do whatever to keep service up also failover to another segment (use PXC_CLUSTER_VIEW) ") unless $Param->{active_failover} < 4;
+    #die "Option --log not specified. We need a place to log what is going on, don't we?\n" unless defined $Param->{log};
+    print Utils->print_log(2,"Option --log not specified. We need a place to log what is going on, don't we?\n") unless defined $Param->{log};
+    
+    if($Param->{debug}){
+       Utils::debugEnv();
+    }
+    
+    $Param->{host} = URLDecode($host);    
+    my $dsn  = "DBI:mysql:host=$Param->{host};port=$Param->{port}";
+    
+    if(defined $Param->{user}){
+      $user = "$Param->{user}";
+    }
+    if(defined $Param->{password}){
+	    $pass = "$Param->{password}";
+    }
+    my $hg =$Param->{hostgroups};
+    $hg =~ s/[\:,\,]/_/g;
+    my $base_path = "${run_pid_dir}/proxysql_galera_check_${hg}.pid";
+    
+    
+    
+    #============================================================================
+    # Execution
+    #============================================================================
+    if(defined $Param->{log}){
+       open(FH, '>>', $Param->{log}."_".$hg.".log") or die Utils->print_log(1,"cannot open file");
+       FH->autoflush if $Param->{development} < 2;
+       select FH;
+    }
+    #checks for ssl cert path and identify if accessible.
+    #If defined and not accessible exit with an error     
+    if(defined $Param->{ssl_certs_path}){
+       my $ssl_path = $Param->{ssl_certs_path};
+       if (-d $ssl_path) {
+          # directory called cgi-bin exists
+          if(-e $ssl_path."/client-key.pem"
+             && -e $ssl_path."/client-cert.pem"
+             && -e $ssl_path."/ca.pem"){
+              print Utils->print_log(4," SSL Directory exists and all the files are there $ssl_path")
+          }
+          else{
+              print Utils->print_log(1,"SSL Path (ssl_certs_path) declared and accessible [$ssl_path]. But certification files must have specific names:\n \t\t client-key.pem \n \t\t client-cert.pem \n \t\t ca.pem \n");
+              exit 1
+          }
+       } 
+       else{
+          # ssl path declared but not exists, exit with error
+          print Utils->print_log(1,"SSL Path (ssl_certs_path) declared but not existing \n \t\t $ssl_path \n \t\t Please create directory and assign the right to access it to the ProxySQL user \n");
+          exit 1;
+       }
+    }
+    
+    if($Param->{development} < 2){
+      if(!-e $base_path){
+         `echo "$$" > $base_path`
+      }
+      else{
+          my $existing_pid=`cat $base_path`;
+          my $exists = kill 0, $existing_pid;
+          if($exists > 0){
+             print STDOUT "Another process is running using the same HostGroup and settings,\n Or orphan pid file. check in $base_path \n";
+             print Utils->print_log(1,"Another process is running using the same HostGroup and settings,\n Or orphan pid file. check in $base_path \n");
+             exit 1;
+          }
+          else{
+               `echo "$$" > $base_path`;
+          }
+      }    
+    }
+     
+    # for test only purpose comment for prod
+
+    my $xx =1;
+    my $y =0;
+    $xx=2000000000 if($Param->{development} > 0);
+ 	
+    while($y < $xx){
+       ++$y ;
+	
+       my $start = gettimeofday();    
+       if($Param->{debug} >= 1){
+          print Utils->print_log(3,"START EXECUTION\n");
+       }
+       
+   
+       
+       my $proxy_sql_node = get_proxy($dsn, $user, $pass ,$Param->{debug}) ;
+       $proxy_sql_node->retry_up($Param->{retry_up});
+       $proxy_sql_node->retry_down($Param->{retry_down});
+       $proxy_sql_node->hostgroups($Param->{hostgroups}) ;
+       $proxy_sql_node->require_failover($Param->{active_failover});
+       $proxy_sql_node->check_timeout($Param->{check_timeout});
+       
+       $proxy_sql_node->connect();
+       
+       # create basic galera cluster object and fill info
+       $proxy_sql_node->set_galera_cluster();
+       my $galera_cluster = $proxy_sql_node->get_galera_cluster();
+       
+       if( defined $galera_cluster){
+           $galera_cluster->main_segment($Param->{main_segment});
+           $galera_cluster->cluster_identifier($hg);
+           $galera_cluster->get_nodes();
+       }
+       
+       # Retrive the nodes state
+       if(defined $galera_cluster->nodes){
+          $galera_cluster->process_nodes();
+    
+       }
+       
+       #Analyze nodes state from ProxySQL prospective;
+       if(defined  $galera_cluster->nodes){
+          my %action_node = $proxy_sql_node->evaluate_nodes($galera_cluster);
+    
+       }
+       
+       if(defined $proxy_sql_node->action_nodes){
+          $proxy_sql_node->push_changes;
+       }
+       
+       my $end = gettimeofday();
+       print Utils->print_log(3,"END EXECUTION Total Time(ms):".($end - $start) * 1000 ."\n") if $Param->{print_execution} >0; 
+       if($Param->{debug} >= 1){
+          print Utils->print_log(3,"\n");
+       }
+       FH->flush(); 
+       
+       $proxy_sql_node->disconnect();
+    
+       #debug braket 	
+        sleep $Param->{development_time} if($Param->{development} > 0);
+
+    }
+    
+    if(defined $Param->{log}){
+       close FH;  # in the end
+    }
+    
+    `rm -f $base_path`;
+    
+      
+    exit(0);
+    
+    
+ }
+
+# ############################################################################
+# Run the program.
+# ############################################################################
+    exit main(@ARGV);
+
+{
+    package Galeracluster;
+    use threads;
+    use threads::shared;
+    use strict;
+    use warnings;
+    use Time::HiRes qw(gettimeofday usleep);
+    
+    sub new {
+        my $class = shift;
+        my $SQL_get_mysql_servers=" SELECT a.* FROM runtime_mysql_servers a join stats_mysql_connection_pool b on a.hostname=b.srv_host and a.port=b.srv_port and a.hostgroup_id=b.hostgroup  WHERE b.status not in ('OFFLINE_HARD','SHUNNED') ";
+        
+        # Variable section for  looping values
+        #Generalize object for now I have conceptualize as:
+        # Cluster (generic container)
+        # Cluster->{name}     This is the cluster name
+        # Cluster->{nodes}       the nodes in the cluster Map by node name
+        # Cluster->{status}     cluster status [Primary|not Primary]
+        # Cluster->{size}     cluster status [Primary|not Primary]
+        # Cluster->{singlenode}=0;  0 if false 1 if true meaning only one ACTIVE node in the cluster 
+        # Cluster->{haswriter}=0;  0 if false 1 if true at least a node is fully active as writer
+        # Cluster->{singlewriter}=1;  0 if false 1 if true this cluster can have ONLY one writer a time [true default]
+        
+        my $self = {
+            _name      => undef,
+            _hosts  => {},
+            _status    => undef,
+            _size  => {},
+            _singlenode  => 0,
+            _haswriter => 0,
+            _singlewriter => 1,
+            _main_segment => 0,
+            _SQL_get_mysql_servers => $SQL_get_mysql_servers,
+            _hostgroups => undef,
+            _dbh_proxy => undef,
+            _debug => 0,
+            _monitor_user => undef,
+            _monitor_password => undef,
+            _nodes => {},
+            _nodes_maint => {},
+            _check_timeout => 100, #timeout in ms
+            _cluster_identifier => undef,
+            _hg_writer_id => 0,
+            _hg_reader_id => 0,
+            _ssl_certificates_path => undef,
+            _writer_is_reader => 0,
+            _reader_nodes => [] ,
+            _writer_nodes => [] ,
+            _has_failover_node =>0,
+            _writers =>0,
+            #_hg => undef,
+        };
+        bless $self, $class;
+        return $self;
+        
+    }
+
+    sub ssl_certificates_path{
+        my ( $self, $in ) = @_;
+        $self->{_ssl_certificates_path} = $in if defined($in);
+        return $self->{_ssl_certificates_path};
+    }
+    
+   sub has_failover_node{
+        my ( $self, $in ) = @_;
+        $self->{_has_failover_node} = $in if defined($in);
+        return $self->{_has_failover_node};
+    }
+   
+   sub writer_nodes{
+        my ( $self, $in ) = @_;
+        $self->{_writer_nodes} = $in if defined($in);
+        return $self->{_writer_nodes};
+    }
+
+    sub reader_nodes{
+        my ( $self, $in ) = @_;
+        $self->{_reader_nodes} = $in if defined($in);
+        return $self->{_reader_nodes};
+    }
+
+    sub cluster_identifier{
+        my ( $self, $in ) = @_;
+        $self->{_cluster_identifier} = $in if defined($in);
+        return $self->{_cluster_identifier};
+    }
+
+    sub main_segment{
+        my ( $self, $main_segment ) = @_;
+        $self->{_main_segment} = $main_segment if defined($main_segment);
+        return $self->{_main_segment};
+    }
+
+    sub check_timeout{
+        my ( $self, $check_timeout ) = @_;
+        $self->{_check_timeout} = $check_timeout if defined($check_timeout);
+        return $self->{_check_timeout};
+    }
+
+    sub debug{
+        my ( $self, $debug ) = @_;
+        $self->{_debug} = $debug if defined($debug);
+        return $self->{_debug};
+    }
+
+    
+    sub dbh_proxy{
+        my ( $self, $dbh_proxy ) = @_;
+        $self->{_dbh_proxy} = $dbh_proxy if defined($dbh_proxy);
+        return $self->{_dbh_proxy};
+    }
+    
+    sub name {
+        my ( $self, $name ) = @_;
+        $self->{_name} = $name if defined($name);
+        return $self->{_name};
+    }
+
+    sub nodes {
+        my ( $self, $nodes ) = @_;
+        $self->{_nodes} = $nodes if defined($nodes);
+        return $self->{_nodes};
+    }
+
+    sub nodes_maint {
+        my ( $self, $nodes ) = @_;
+        $self->{_nodes_maint} = $nodes if defined($nodes);
+        return $self->{_nodes_maint};
+    }
+
+    sub status {
+        my ( $self, $status ) = @_;
+        $self->{_status} = $status if defined($status);
+        return $self->{_status};
+    }
+
+    sub size {
+        my ( $self, $size ) = @_;
+        $self->{_size} = $size if defined($size);
+        return $size->{_size};
+    }
+
+    sub singlenode {
+        my ( $self, $singlenode ) = @_;
+        $self->{_singlenode} = $singlenode if defined($singlenode);
+        return $self->{_singlenode};
+    }
+
+    sub haswriter {
+        my ( $self, $haswriter ) = @_;
+        $self->{_haswriter} = $haswriter if defined($haswriter);
+        return $self->{_haswriter};
+    }
+    
+    sub singlewriter {
+        my ( $self, $singlewriter ) = @_;
+        $self->{_singlewriter} = $singlewriter if defined($singlewriter);
+        return $self->{_singlewriter};
+    }
+
+    sub writer_is_reader {
+        my ( $self, $writer_is_reader ) = @_;
+        $self->{_writer_is_reader} = $writer_is_reader if defined($writer_is_reader);
+        return $self->{_writer_is_reader};
+    }
+    sub writers {
+        my ( $self, $in ) = @_;
+        $self->{_writers} = $in if defined($in);
+        return $self->{_writers};
+    }
+
+    sub hostgroups {
+        my ( $self, $hostgroups ) = @_;
+        $self->{_hostgroups} = $hostgroups if defined($hostgroups);
+        return $self->{_hostgroups};
+    }
+    sub hg_writer_id {
+        my ( $self, $hostgroups ) = @_;
+        $self->{_hg_writer_id} = $hostgroups if defined($hostgroups);
+        return $self->{_hg_writer_id};
+    }
+
+    sub hg_reader_id {
+        my ( $self, $hostgroups ) = @_;
+        $self->{_hg_reader_id} = $hostgroups if defined($hostgroups);
+        return $self->{_hg_reader_id};
+    }
+    
+    
+    sub monitor_user{
+        my ( $self, $monitor_user ) = @_;
+        $self->{_monitor_user} = $monitor_user if defined($monitor_user);
+        return $self->{_monitor_user};
+    }
+    sub monitor_password {
+        my ( $self, $monitor_password ) = @_;
+        $self->{_monitor_password} = $monitor_password if defined($monitor_password);
+        return $self->{_monitor_password};
+    }
+    # this function is used to identify the nodes in the cluster
+    # using the HG as reference
+    sub get_nodes{
+        my ( $self) = @_;
+        
+        my $dbh = $self->{_dbh_proxy};
+        my $cmd =$self->{_SQL_get_mysql_servers}." AND hostgroup_id IN (".join(",",sort keys(%{$self->hostgroups})).") order by hostgroup_id, hostname";
+        my $sth = $dbh->prepare($cmd);
+        $sth->execute();
+        my $i = 1;
+        my $locHg = $self->{_hostgroups};
+        my $ssl_certificates = "";
+        #if a ssl certificate path is defined, will create the path for each certificate and add to the dns string
+        if(defined $self->{_ssl_certificates_path}){
+           $ssl_certificates = ";mysql_ssl_client_key=".$self->{_ssl_certificates_path}."/client-key.pem"
+           .";mysql_ssl_client_cert=".$self->{_ssl_certificates_path}."/client-cert.pem"
+           .";mysql_ssl_ca_file=".$self->{_ssl_certificates_path}."/ca.pem"
+        }
+        
+       while (my $ref = $sth->fetchrow_hashref()) {
+            my $ssl_options="" ;
+            my $node = GaleraNode->new();
+            $node->debug($self->debug);
+            $node->use_ssl($ref->{use_ssl});
+            $node->hostgroups($ref->{hostgroup_id});
+            if($node->{_hostgroups} > 8000
+               && exists $locHg->{$node->{_hostgroups}}){
+                  $self->{_has_failover_node} = 1;
+                
+            }
+            $node->ip($ref->{hostname});
+            $node->port($ref->{port});
+
+            if($node->use_ssl gt 0 ){
+               $ssl_options = ";mysql_ssl=1";
+               if($self->debug){print Utils->print_log(4," Galera cluster node   " . $node->ip.":". $node->port.":HG=".$node->hostgroups." Using SSL ($ssl_options)\n" ) }
+               if(defined $self->{_ssl_certificates_path}){
+                 $ssl_options = $ssl_options . $ssl_certificates;
+                 if($self->debug){print Utils->print_log(4," Certificates also in use ($self->{_ssl_certificates_path})\n")}
+               }
+            }
+            
+            $node->dns("DBI:mysql:host=".$node->ip.";port=".$node->port.";mysql_connect_timeout=$mysql_connect_timeout".$ssl_options);
+            
+            $node->weight($ref->{weight});
+            $node->connections($ref->{max_connections});
+            $node->user($self->{_monitor_user});
+            $node->password($self->{_monitor_password});
+            $node->proxy_status($ref->{status});
+            $node->comment($ref->{comment});
+            $node->set_retry_up_down($self->{_cluster_identifier});
+            
+            $node->gtid_port($ref->{gtid_port});
+            $node->compression($ref->{compression});
+            $node->max_latency($ref->{max_latency_ms});
+            $node->max_replication_lag($ref->{max_replication_lag});
+	    
+     
+            $self->{_nodes}->{$i++}=$node;
+            $node->debug($self->debug);
+            
+            if($self->debug){print Utils->print_log(3," Galera cluster node   " . $node->ip.":". $node->port.":HG=".$node->hostgroups."\n" ) }
+        }
+	
+       if($self->debug){print Utils->print_log(3," Galera cluster nodes loaded \n") ; }
+    }
+    #Processing the nodes in the cluster and identify which node is active and which is to remove
+    
+    sub process_nodes{
+        my ( $self ) = @_;
+
+        my $nodes = $self->{_nodes} ;
+        my $start = gettimeofday();
+        my $run_milliseconds=0;
+        my $init =0;
+        my $irun = 1;
+        my %Threads;
+        my $new_nodes ={} ;
+        my $processed_nodes ={} ;
+        
+        #using multiple threads to connect if a node is present in more than one HG it will have 2 threads
+        while($irun){
+            $irun = 0;
+            foreach my $key (sort keys %{$self->{_nodes}}){
+                if(!exists $Threads{$key}){
+                   if($self->debug){print Utils->print_log(3, " Creating new thread to manage server check:".
+                      $self->{_nodes}->{$key}->ip.":".
+                      $self->{_nodes}->{$key}->port.":HG".$self->{_nodes}->{$key}->hostgroups."\n" ) }
+                    $new_nodes->{$key} =  $self->{_nodes}->{$key};
+                    $new_nodes->{$key}->{_process_status} = -1;
+                    $new_nodes->{$key}->{_ssl_certificates_path} = $self->ssl_certificates_path;
+
+                    #  debug senza threads comment next line
+                    $Threads{$key}=threads->create(sub  {return get_node_info($self,$key)});
+                    
+                    #DEBUG Without threads uncomment from here
+                    #next unless  $new_nodes->{$key} = get_node_info($self,$key);
+                    #evaluate_joined_node($self, $key, $new_nodes, $processed_nodes) ;
+    
+             # to here
+		    
+                }
+            }
+            ##DEBUG SENZA THREADS commenta da qui
+            foreach my $thr (sort keys %Threads) {
+                if($new_nodes->{$thr}->{_process_status} eq -100){
+                    next;
+                }
+                
+                if ($Threads{$thr}->is_running()) {
+                    my $tid = $Threads{$thr}->tid;
+                    #print "  - Thread $tid running\n";
+                   
+                    if($run_milliseconds >  $self->{_check_timeout} ){
+                       if($self->debug >=0){
+                         my $timeout = ($run_milliseconds -  $self->{_check_timeout});  
+                         print print Utils->print_log(2,"Check timeout Node ip : $new_nodes->{$thr}->{_ip} ,  THID " . $tid." (taken: $run_milliseconds   max_allowed: $self->{_check_timeout}  over for ms: $timeout  \n")
+                        }	
+                          $irun = 0 ; 
+                    }
+                    else{
+                         $irun = 1;
+                    }
+                } 
+                elsif ( $Threads{$thr}->is_joinable()) {
+                 
+                       my $tid = $Threads{$thr}->tid;
+                       ( $new_nodes->{$thr} ) = $Threads{$thr}->join;
+                       #$processed_nodes =
+                       evaluate_joined_node($self, $thr, $new_nodes, $processed_nodes) ;
+                     
+                     if($self->debug){print Utils->print_log(3," Thread joined :   " . $tid."\n" ) }	
+                        #print "  - Results for thread $tid:\n";
+                        #print "  - Thread $tid has been joined\n";
+                }
+                #print ".";
+            }
+            ## a qui
+            if($self->debug){$run_milliseconds = (gettimeofday() -$start ) *1000};
+            #sleep for a time equal to the half of the timeout to save cpu cicle
+            #usleep(($self->{_check_timeout} * 1000)/2);
+        }
+
+        $self->{_nodes} = $new_nodes;
+        if($self->debug){$run_milliseconds = (gettimeofday() -$start ) *1000};
+	
+        if($debug>=3){
+            foreach my $key (sort keys %{$new_nodes}){
+              if($new_nodes->{$key}->{_process_status} == 1){
+                  print Utils->print_log(4,$new_nodes->{$key}->{_ip}.":".$new_nodes->{$key}->{_hostgroups}." Processed \n");
+              }
+              else{
+                  print Utils->print_log(4,$new_nodes->{$key}->{_ip}.":".$new_nodes->{$key}->{_hostgroups}." NOT Processed\n");
+              }
+           }
+        }
+       	if($self->debug){print Utils->print_log(3," Multi Thread execution done in :   " . $run_milliseconds. "(ms) \n"  )}	
+    
+    }
+    
+    sub evaluate_joined_node($$$$){
+        my $self = shift;
+        my $thr = shift;
+        my $new_nodes = shift;
+        my $processed_nodes = shift;
+        
+            #count the number of nodes by segment
+         if($new_nodes->{$thr}->{_proxy_status} ne "OFFLINE_SOFT"
+            && $new_nodes->{$thr}->{_proxy_status} ne "SHUNNED"
+            && ($new_nodes->{$thr}->{_process_status} < 0 ||
+             !exists $processed_nodes->{$new_nodes->{$thr}->{_ip}})
+             && defined $new_nodes->{$thr}->{_wsrep_segment} 
+           ){
+                $self->{_size}->{$new_nodes->{$thr}->{_wsrep_segment}} = (($self->{_size}->{$new_nodes->{$thr}->{_wsrep_segment}}|| 0) +1);
+                $processed_nodes->{$new_nodes->{$thr}->{_ip}}=$self->{_size}->{$new_nodes->{$thr}->{_wsrep_segment}};
+         }
+      
+      #assign size to HG
+        if($new_nodes->{$thr}->{_proxy_status} ne "OFFLINE_SOFT"
+           && defined $new_nodes->{$thr}->{_wsrep_segment} 
+           ){
+            $self->{_hostgroups}->{$new_nodes->{$thr}->{_hostgroups}}->{_size} = ($self->{_hostgroups}->{$new_nodes->{$thr}->{_hostgroups}}->{_size}) + 1;
+        }
+        
+     #checks for ONLINE writer(s)
+
+       if(defined $new_nodes->{$thr}->{_read_only}
+          && $new_nodes->{$thr}->{_read_only} eq "OFF"
+          && ($new_nodes->{$thr}->{_proxy_status} eq "ONLINE" || $new_nodes->{$thr}->{_proxy_status} eq "OFFLINE_SOFT")
+          && ($new_nodes->{$thr}->{_hostgroups} == $self->hg_writer_id || $new_nodes->{$thr}->{_hostgroups} == ($self->hg_writer_id +9000))
+         ){
+           if($new_nodes->{$thr}->{_hostgroups} == $self->hg_writer_id
+              && $new_nodes->{$thr}->{_proxy_status} eq "ONLINE"
+             ){
+              $self->{_haswriter} = 1 ;
+              $self->{_writers} = $self->{_writers} +1;
+           }
+             push (@{$self->{_writer_nodes}}, "$new_nodes->{$thr}->{_ip}:$new_nodes->{$thr}->{_port}");  
+        }
+       elsif(($new_nodes->{$thr}->{_proxy_status} eq "ONLINE" || $new_nodes->{$thr}->{_proxy_status} eq "OFFLINE_SOFT")
+          && ($new_nodes->{$thr}->{_hostgroups} == $self->hg_reader_id || $new_nodes->{$thr}->{_hostgroups} == ($self->hg_reader_id +9000))
+       ){
+            push (@{$self->{_reader_nodes}}, "$new_nodes->{$thr}->{_ip}:$new_nodes->{$thr}->{_port}");  
+       }
+       else{
+             if($self->debug
+                && $new_nodes->{$thr}->{_hostgroups} == $self->hg_writer_id){
+                   print Utils->print_log(3," Not a writer :" .$new_nodes->{$thr}->{_ip} . " HG: $new_nodes->{$thr}->{_hostgroups}  \n" )
+             }	
+       }
+      # check if under maintenance
+       if($new_nodes->{$thr}->{_proxy_status} eq "OFFLINE_SOFT"
+          && $new_nodes->{$thr}->{_pxc_maint_mode} eq "MAINTENANCE"){
+             $self->{_nodes_maint}->{$thr} =  $new_nodes->{$thr};
+        }
+       #return $processed_nodes;
+     
+    }
+    
+    sub get_node_info($$){
+        my $self = shift;
+        my $key = shift;
+        my $nodes =shift;
+        my ( $node ) = $self->{_nodes}->{$key};
+        if(!defined $node->get_node_info()){
+           $node->{_process_status}=-100;
+         }
+	
+        return $node;
+        
+    }
+    
+}
+
+{
+    package GaleraNode;
+    #Node Proxy States
+    sub new {
+        my $class = shift;
+        my $SQL_get_variables="SHOW GLOBAL VARIABLES LIKE 'wsrep%";
+        my $SQL_get_status="SHOW GLOBAL STATUS LIKE 'wsrep%";
+        my $SQL_get_read_only="SHOW GLOBAL VARIABLES LIKE 'read_only'";  
+
+        # Variable section for  looping values
+        #Generalize object for now I have conceptualize as:
+        # Node (generic container)
+        # Node->{name}     This is the cluster name
+        # Node->{IP}
+        # Node->{hostgroups}
+        # Node->{clustername} This is the cluster name
+        # Node->{read_only} Read only node
+        # Node->{wsrep_status}     node status (OPEN 0,Primary 1,Joiner 2,Joined 3,Synced 4,Donor 5)
+        # Node->{wsrep_rejectqueries} (NON, ALL,ALL_KILL)
+        # Node->{wsrep_donorrejectqueries} If true the node when donor 
+        # Node->{wsrep_connected}=0;   if false 1 if true meaning only one ACTIVE node in the cluster 
+        # Node->{wsrep_desinccount}=0;  0 if false 1 if true at least a node is fully active as writer
+        # Node->{wsrep_ready}  ON -OFF 
+        
+        my $self = {
+            _name      => undef,
+            _ip  => undef,
+            _port => 3306,
+            _hostgroups => undef,
+            _clustername    => undef,
+            _read_only    => undef,
+            _wsrep_status  => -1,
+            _wsrep_rejectqueries => undef,
+            _wsrep_donorrejectqueries => undef,
+            _wsrep_connected => undef,
+            _wsrep_desinccount => undef,
+            _wsrep_ready => undef,
+            _wsrep_provider => [],
+            _wsrep_segment => 1000,
+            _wsrep_pc_weight => 1,
+            _SQL_get_variables => $SQL_get_variables,
+            _SQL_get_status=> $SQL_get_status,
+            _SQL_get_read_only=> $SQL_get_read_only,
+            _dns  => undef,
+            _user => undef,
+            _password => undef,
+            _debug => 0,
+            _port => undef,
+            _proxy_status    => undef,
+            _weight => 1,
+            _connections => 2000,
+            _cluster_status    => undef,
+            _cluster_size  => 0,
+            _process_status => -1, 
+            _MOVE_UP_OFFLINE => 1000, #move a node from OFFLINE_SOFT 
+            _MOVE_UP_HG_CHANGE => 1010, #move a node from HG 9000 (plus hg id) to reader HG 
+            _MOVE_DOWN_HG_CHANGE => 3001, #move a node from original HG to maintenance HG (HG 9000 (plus hg id) ) kill all existing connections
+            _MOVE_DOWN_OFFLINE => 3010 , # move node to OFFLINE_soft keep existing connections, no new connections.
+            _MOVE_TO_MAINTENANCE => 3020 , # move node to OFFLINE_soft keep existing connections, no new connections because maintenance.
+            _MOVE_OUT_MAINTENANCE => 3030 , # move node to OFFLINE_soft keep existing connections, no new connections because maintenance.
+            _INSERT_READ => 4010, # Insert a node in the reader host group
+            _INSERT_WRITE => 4020, # Insert a node in the writer host group
+            _DELETE_NODE => 5000, # this remove the node from the hostgroup
+            _SAVE_RETRY => 9999, # this reset the retry counter in the comment 
+            #_MOVE_SWAP_READER_TO_WRITER => 5001, #Future use
+            #_MOVE_SWAP_WRITER_TO_READER => 5010, #Future use
+            _retry_down_saved => 0, # number of retry on a node before declaring it as failed.
+            _retry_up_saved => 0, # number of retry on a node before declaring it OK.
+            _comment => undef,
+            _gtid_port => 0,
+            _compression => 0,
+            _use_ssl => 0,
+            _ssl_certificates_path => undef,
+            _max_latency => 0,
+            _max_replication_lag => 0,
+            _wsrep_gcomm_uuid => undef,
+            _wsrep_local_index => 0,
+            _pxc_maint_mode => undef,
+
+        };
+        bless $self, $class;
+        return $self;
+        
+    }
+
+    sub ssl_certificates_path{
+        my ( $self, $in ) = @_;
+        $self->{_ssl_certificates_path} = $in if defined($in);
+        return $self->{_ssl_certificates_path};
+    }
+ 
+    sub max_replication_lag{
+        my ( $self, $in ) = @_;
+        $self->{_max_replication_lag} = $in if defined($in);
+        return $self->{_max_replication_lag};
+    }
+
+    sub max_latency{
+        my ( $self, $in ) = @_;
+        $self->{_max_latency} = $in if defined($in);
+        return $self->{_max_latency};
+    }
+    
+    sub use_ssl{
+        my ( $self, $in ) = @_;
+        $self->{_use_ssl} = $in if defined($in);
+        return $self->{_use_ssl};
+    }
+
+    sub compression{
+        my ( $self, $in ) = @_;
+        $self->{_compression} = $in if defined($in);
+        return $self->{_compression};
+    }
+
+    sub gtid_port{
+        my ( $self, $in ) = @_;
+        $self->{_gtid_port} = $in if defined($in);
+        return $self->{_gtid_port};
+    }
+
+    sub pxc_maint_mode{
+        my ( $self, $in ) = @_;
+        $self->{_pxc_maint_mode} = $in if defined($in);
+        return $self->{_pxc_maint_mode};
+    }
+    sub wsrep_local_index{
+        my ( $self, $in ) = @_;
+        $self->{_wsrep_local_index} = $in if defined($in);
+        return $self->{_wsrep_local_index};
+    }
+
+    sub comment{
+        my ( $self, $in ) = @_;
+        $self->{_comment} = $in if defined($in);
+        return $self->{_comment};
+    }
+
+    sub wsrep_gcomm_uuid{
+        my ( $self, $in ) = @_;
+        $self->{_wsrep_gcomm_uuid} = $in if defined($in);
+        return $self->{_wsrep_gcomm_uuid};
+    }
+    
+    sub retry_down_saved{
+        my ( $self, $in ) = @_;
+        $self->{_retry_down_saved} = $in if defined($in);
+        return $self->{_retry_down_saved};
+    }
+
+    sub retry_up_saved{
+        my ( $self, $in ) = @_;
+        $self->{_retry_up_saved} = $in if defined($in);
+        return $self->{_retry_up_saved};
+    }
+    
+    sub process_status {
+        my ( $self, $process_status ) = @_;
+        $self->{_process_status} = $process_status if defined($process_status);
+        return $self->{_process_status};
+    }
+
+    sub debug{
+        my ( $self, $debug ) = @_;
+        $self->{_debug} = $debug if defined($debug);
+        return $self->{_debug};
+    }
+
+    sub SAVE_RETRY {
+        my ( $self) = @_;
+        return $self->{_SAVE_RETRY};
+    }
+
+    sub MOVE_UP_OFFLINE {
+        my ( $self) = @_;
+        return $self->{_MOVE_UP_OFFLINE};
+    }
+
+    sub MOVE_UP_HG_CHANGE {
+        my ( $self) = @_;
+        return $self->{_MOVE_UP_HG_CHANGE};
+    }
+    
+    sub MOVE_DOWN_OFFLINE {
+        my ( $self) = @_;
+        return $self->{_MOVE_DOWN_OFFLINE};
+    }
+    
+    
+    sub MOVE_TO_MAINTENANCE {
+        my ( $self) = @_;
+        return $self->{_MOVE_TO_MAINTENANCE};
+    }
+    
+    sub MOVE_OUT_MAINTENANCE {
+        my ( $self) = @_;
+        return $self->{_MOVE_OUT_MAINTENANCE};
+    }
+
+    sub MOVE_DOWN_HG_CHANGE {
+        my ( $self) = @_;
+        return $self->{_MOVE_DOWN_HG_CHANGE};
+    }
+    sub DELETE_NODE {
+        my ( $self) = @_;
+        return $self->{_DELETE_NODE};
+    }
+    sub INSERT_READ {
+        my ( $self) = @_;
+        return $self->{_INSERT_READ};
+    }
+    sub INSERT_WRITE {
+        my ( $self) = @_;
+        return $self->{_INSERT_WRITE};
+    }
+
+    sub cluster_status {
+        my ( $self, $status ) = @_;
+        $self->{_cluster_status} = $status if defined($status);
+        return $self->{_cluster_status};
+    }
+
+    sub cluster_size {
+        my ( $self, $size ) = @_;
+        $self->{_cluster_size} = $size if defined($size);
+        return $size->{_cluster_size};
+    }
+    
+    sub weight {
+        my ( $self, $weight ) = @_;
+        $self->{_weight} = $weight if defined($weight);
+        return $self->{_weight};
+    }
+
+    sub connections {
+        my ( $self, $connections ) = @_;
+        $self->{_connections} = $connections if defined($connections);
+        return $self->{_connections};
+    }
+    
+    sub proxy_status {
+        my ( $self, $status ) = @_;
+        $self->{_proxy_status} = $status if defined($status);
+        return $self->{_proxy_status};
+    }
+    
+    sub dns {
+        my ( $self, $dns ) = @_;
+        $self->{_dns} = $dns if defined($dns);
+        return $self->{_dns};
+    }
+    
+    sub user{
+        my ( $self, $user ) = @_;
+        $self->{_user} = $user if defined($user);
+        return $self->{_user};
+    }
+    sub password {
+        my ( $self, $password ) = @_;
+        $self->{_password} = $password if defined($password);
+        return $self->{_password};
+    }
+    sub name {
+        my ( $self, $name ) = @_;
+        $self->{_name} = $name if defined($name);
+        return $self->{_name};
+    }
+
+    sub ip {
+        my ( $self, $ip ) = @_;
+        $self->{_ip} = $ip if defined($ip);
+        return $self->{_ip};
+    }
+    sub port {
+        my ( $self, $port ) = @_;
+        $self->{_port} = $port if defined($port);
+        return $self->{_port};
+    }
+
+
+    sub hostgroups {
+        my ( $self, $hostgroups ) = @_;
+        $self->{_hostgroups} = $hostgroups if defined($hostgroups);
+        return $self->{_hostgroups};
+    }
+    
+    sub clustername {
+        my ( $self, $clustername ) = @_;
+        $self->{_clustername} = $clustername if defined($clustername);
+        return $self->{_clustername};
+    }
+
+    sub read_only {
+        my ( $self, $read_only ) = @_;
+        $self->{_read_only} = $read_only if defined($read_only);
+        return $self->{_read_only};
+    }
+
+    sub wsrep_status {
+        my ( $self, $wsrep_status ) = @_;
+        $self->{_wsrep_status} = $wsrep_status if defined($wsrep_status);
+        return $self->{_wsrep_status};
+    }
+
+    sub wsrep_rejectqueries {
+        my ( $self, $wsrep_rejectqueries ) = @_;
+        $self->{_wsrep_rejectqueries} = $wsrep_rejectqueries if defined($wsrep_rejectqueries);
+        return $self->{_wsrep_rejectqueries};
+    }
+
+    sub wsrep_donorrejectqueries {
+        my ( $self, $wsrep_donorrejectqueries ) = @_;
+        $self->{_wsrep_donorrejectqueries} = $wsrep_donorrejectqueries if defined($wsrep_donorrejectqueries);
+        return $self->{_wsrep_donorrejectqueries};
+    }
+
+    sub wsrep_connected {
+        my ( $self, $wsrep_connected ) = @_;
+        $self->{_wsrep_connected} = $wsrep_connected if defined($wsrep_connected);
+        return $self->{_wsrep_connected};
+    }
+
+    sub wsrep_desinccount {
+        my ( $self, $wsrep_desinccount ) = @_;
+        $self->{_wsrep_desinccount} = $wsrep_desinccount if defined($wsrep_desinccount);
+        return $self->{_wsrep_desinccount};
+    }
+
+
+    sub wsrep_ready {
+        my ( $self, $wsrep_ready ) = @_;
+        $self->{_wsrep_ready} = $wsrep_ready if defined($wsrep_ready);
+        return $self->{_wsrep_ready};
+    }
+
+    sub wsrep_segment {
+        my ( $self, $wsrep_segment ) = @_;
+        $self->{_wsrep_segment} = $wsrep_segment if defined($wsrep_segment);
+        return $self->{_wsrep_segment};
+    }
+
+    sub wsrep_pc_weight {
+        my ( $self, $wsrep_pc_weight ) = @_;
+        $self->{_wsrep_pc_weight} = $wsrep_pc_weight if defined($wsrep_pc_weight);
+        return $self->{_wsrep_pc_weight};
+    }
+
+    sub wsrep_provider {
+        my ( $self, $wsrep_provider ) = @_;
+        my ( @array)= @{$wsrep_provider} ;
+        my %provider_map ;
+        foreach my $item (@array){
+          my @items = split('\=', $item);
+          $provider_map{Utils::trim($items[0])}=$items[1];
+        }
+        ($self->{_wsrep_provider}) = {%provider_map} ;
+        $self->wsrep_segment($provider_map{"gmcast.segment"});
+        $self->wsrep_pc_weight($provider_map{"pc.weight"});
+        return $self->{_wsrep_provider};
+    }
+
+    sub get_node_info($$){
+        my ( $self ) = @_;
+        
+        if($self->debug >=1){
+           print Utils->print_log(4," Node check START "
+            .$self->{_ip}
+            .":".$self->{_port}
+            .":HG".$self->{_hostgroups}
+            ."\n"  );
+        }	
+	       if($self->debug >=1){
+           print Utils->print_log(4," Getting connection START "
+            .$self->{_ip}
+            .":".$self->{_port}
+            .":HG".$self->{_hostgroups}." \n"  );
+        }	
+        my $dbh = Utils::get_connection($self->{_dns},$self->{_user},$self->{_password},' ');
+        if(!defined $dbh){
+           print Utils->print_log(1," Node is not responding setting it as SHUNNED (internally) (ProxySQL bug - #2658)"
+             .$self->{_ip}
+            .":".$self->{_port}
+            .":HG".$self->{_hostgroups}." \n"  );
+           $self->{_proxy_status} = "SHUNNED";
+           return $self ; 
+           
+        }
+	       if($self->debug >=1){
+           print Utils->print_log(4," Getting connection END "
+             .$self->{_ip}
+            .":".$self->{_port}
+            .":HG".$self->{_hostgroups}." \n"  );
+
+        }	
+
+	       if($self->debug >=1){
+           print Utils->print_log(4," Getting NODE info START "
+             .$self->{_ip}
+            .":".$self->{_port}
+            .":HG".$self->{_hostgroups}." \n"  );
+        }	
+
+        my $variables = Utils::get_variables($dbh,0);
+        my $status = Utils::get_status_by_name($dbh,0,"wsrep_%");
+        my $pxc_view = Utils::get_pxc_clusterview($dbh, $status->{wsrep_gcomm_uuid} );
+
+	       if($self->debug >=1){
+           print Utils->print_log(4," Getting NODE info END "
+             .$self->{_ip}
+            .":".$self->{_port}
+            .":HG".$self->{_hostgroups}." \n"  );
+        }	
+        
+        $self->{_name} = $variables->{wsrep_node_name};
+        $self->{_clustername} = $variables->{wsrep_cluster_name};
+        $self->{_read_only} = $variables->{read_only};
+        $self->{_wsrep_rejectqueries} = $variables->{wsrep_reject_queries};
+        #print "AAAAAAAAAAAAAAAAAAAAA  $self->{_ip}  $self->{_wsrep_rejectqueries} \n";
+        $self->{_wsrep_donorrejectqueries} = $variables->{wsrep_sst_donor_rejects_queries};
+        my ( @provider ) =  split('\;', $variables->{wsrep_provider_options});
+        $self->{_pxc_maint_mode} = $variables->{pxc_maint_mode};
+        
+        $self->wsrep_provider( [ @provider]) ;
+        $self->{_wsrep_status} = $status->{wsrep_local_state};
+        $self->{_wsrep_connected} = $status->{wsrep_connected};
+        $self->{_wsrep_desinccount} = $status->{wsrep_desync_count};
+        $self->{_wsrep_ready} = $status->{wsrep_ready};
+        $self->{_cluster_status} = $status->{wsrep_cluster_status};
+        $self->{_cluster_size} = $status->{wsrep_cluster_size};
+        $self->{_wsrep_gcomm_uuid} = $status->{wsrep_gcomm_uuid};
+        $self->{wsrep_segment} = ($self->{_wsrep_provider}->{"gmcast.segment"} );
+        $self->{wsrep_segment} =~ s/^\s+|\s+$//g;
+        $self->{_wsrep_local_index} = $pxc_view->{local_index};
+        if($self->{wsrep_segment} == 0){
+           $self->{_wsrep_segment} = $pxc_view->{segment};
+        }
+                
+        $dbh->disconnect if (defined $dbh);
+	#sleep 5;
+	
+        $self->{_process_status} = 1;
+        if($self->debug>=1){
+            print Utils->print_log(4," Node check END "
+            .$self->{_ip}
+            .":".$self->{_port}
+            .":HG".$self->{_hostgroups}
+            ."\n"  );}	
+
+        return $self;
+        
+    }
+    
+    sub set_retry_up_down(){
+      my ( $self, $hg ) = @_;
+      if($self->debug >=1){print Utils->print_log(4,"Calculate retry from comment Node:".$self->ip." port:".$self->port . " hg:".$self->hostgroups ." Time IN \n");}
+	
+     	my %comments = split /[;=]/, $self->{_comment};
+      if(exists $comments{$hg."_retry_up"}){
+          $self->{_retry_up_saved} = $comments{$hg."_retry_up"};
+      }
+      else{
+          $self->{_retry_up_saved} = 0;
+      }
+      if(exists $comments{$hg."_retry_down"}){
+          $self->{_retry_down_saved} = $comments{$hg."_retry_down"};
+      }
+      else{
+          $self->{_retry_down_saved} = 0;
+      }
+      my $removeUp=$hg."_retry_up=".$self->{_retry_up_saved}.";";
+      my $removeDown=$hg."_retry_down=".$self->{_retry_down_saved}.";";
+      $self->{_comment} =~ s/$removeDown//ig ;
+      $self->{_comment} =~ s/$removeUp//ig ;
+      
+      if($self->debug >=1){print Utils->print_log(4,"Calculate retry from comment Node:".$self->ip." port:".$self->port . " hg:".$self->hostgroups ." Time OUT \n");}
+    }
+    
+    sub get_retry_up(){
+      my ( $self,$in) = @_;
+      $self->{_retry_up_saved} = $in if defined($in);
+      return $self->{_retry_up_saved};
+    }
+
+    sub get_retry_down(){
+      my ( $self,$in) = @_;
+      $self->{_retry_down_saved} = $in if defined($in);
+      return $self->{_retry_down_saved};
+    }
+    
+    sub promote_writer(){
+      my ( $self,$proxynode,$Galera_cluster,$exclude_delete ) = @_;
+      if($self->{_hostgroups} > 8000){
+         print Utils->print_log(3,"Special Backup - Group found! I am electing a node to writer following the indications\n This Node Try to become the new"
+                                  ." WRITER for HG $proxynode->{_hg_writer_id} Server details: " 
+             .$self->{_ip}
+             .":".$self->{_port}
+             .":HG".$self->{_hostgroups}
+             ."\n"  );	
+       
+      }
+      print Utils->print_log(3,"This Node Try to become a WRITER promoting to HG $proxynode->{_hg_writer_id} " 
+          .$self->{_ip}
+          .":".$self->{_port}
+          .":HG ".$self->{_hostgroups}
+          ."\n"  );	
+	
+      #my $dbh = Utils::get_connection($self->{_dns},$self->{_user},$self->{_password},' ');
+      #if(!defined $dbh){
+      #    return undef;
+      #}
+      #(9000 + $proxynode->{_hg_writer_id})
+      
+      my $proxy_sql_command= "INSERT INTO mysql_servers (hostname,hostgroup_id,port,weight,max_connections,use_ssl,compression,max_latency_ms) VALUES ('$self->{_ip}',$proxynode->{_hg_writer_id},$self->{_port},$self->{_weight},$self->{_connections},$self->{_use_ssl},$self->{_compression},$self->{_max_latency});";
+      if($Galera_cluster->{_singlewriter} > 0){
+         my $delete = "DELETE from mysql_servers where hostgroup_id in ($proxynode->{_hg_writer_id},".(9000 + $proxynode->{_hg_writer_id}).") AND STATUS = 'ONLINE'".$exclude_delete;
+        print Utils->print_log(2," DELETE from writer group as: " 
+            ." SQL:" .$delete
+            ."\n" );			    
+         
+         $proxynode->{_dbh_proxy}->do($delete) or die "Couldn't execute statement: " .  $proxynode->{_dbh_proxy}->errstr;
+      }
+      #if the writer is NOT a reader by default remove it from reader groups also the 
+      if($Galera_cluster->{_writer_is_reader} < 1 ){
+        my $delete = "DELETE from mysql_servers where hostgroup_id in ($proxynode->{_hg_reader_id},".(9000 + $proxynode->{_hg_reader_id}).") and hostname = '$self->{_ip}' and port=$self->{_port}  ";
+        $proxynode->{_dbh_proxy}->do($delete) or die "Couldn't execute statement: " .  $proxynode->{_dbh_proxy}->errstr; 
+        
+      }  
+      $proxynode->{_dbh_proxy}->do($proxy_sql_command) or die "Couldn't execute statement: " .  $proxynode->{_dbh_proxy}->errstr;
+      $proxynode->{_dbh_proxy}->do("LOAD MYSQL SERVERS TO RUNTIME") or die "Couldn't execute statement: " .  $proxynode->{_dbh_proxy}->errstr;
+      $proxynode->{_dbh_proxy}->do("SAVE MYSQL SERVERS TO DISK") or die "Couldn't execute statement: " .  $proxynode->{_dbh_proxy}->errstr;
+        print Utils->print_log(2," Move node:"
+            .$self->{_ip}.":"
+            .$self->{_port}
+            .$self->{_weight}
+            .$self->{_connections}
+            .$proxynode->{hg_writer_id}
+            ." SQL:" .$proxy_sql_command
+            ."\n" );			    
+
+      
+      
+      return 1;
+    }
+    
+    sub move_to_writer_hg(){
+      my ( $self ) = @_;
+        
+      print Utils->print_log(3,"This Node Try to become a WRITER set READ_ONLY to 0 " 
+          .$self->{_ip}
+          .":".$self->{_port}
+          .":HG".$self->{_hostgroups}
+          ."\n"  );	
+	
+      my $dbh = Utils::get_connection($self->{_dns},$self->{_user},$self->{_password},' ');
+      if(!defined $dbh){
+          return undef;
+      }
+      if($dbh->do("SET GLOBAL READ_ONLY=0")){
+         print Utils->print_log(3,"This Node NOW HAS READ_ONLY = 0 "
+             .$self->{_ip}
+             .":".$self->{_port}
+             .":HG".$self->{_hostgroups}
+             ."\n"  );	
+      }
+      else{
+          die "Couldn't execute statement: " .  $dbh->errstr;
+      }
+     	$dbh->disconnect if (defined $dbh);
+      #or die "Couldn't execute statement: " .  $dbh->errstr;
+      return 1;
+    }
+}
+
+{
+    package ProxySqlNode;
+    sub new {
+        my $class = shift;
+
+        my $SQL_get_monitor = "select variable_name name,variable_value value from global_variables where variable_name in( 'mysql-monitor_username','mysql-monitor_password','mysql-monitor_read_only_timeout' ) order by 1";
+        my $SQL_get_hostgroups = "select distinct hostgroup_id hg_isd from runtime_mysql_servers order by 1;";
+        my $SQL_get_rep_hg = "select writer_hostgroup,reader_hostgroup from mysql_replication_hostgroups order by 1;";
+        my $SQL_get_pxc_cluster_view = "select * from performance_schema.pxc_cluster_view order by  SEGMENT, LOCAL_INDEX;";
+
+        # Variable section for  looping values
+        #Generalize object for now I have conceptualize as:
+        # Proxy (generic container)
+        # Proxy->{DNS} conenction reference
+        # Proxy->{PID} processes pid (angel and real)
+        # Proxy->{hostgroups}
+        # Proxy->{user} This is the user name
+        # Proxy->{password} 
+        # Proxy->{port}     node status (OPEN 0,Primary 1,Joiner 2,Joined 3,Synced 4,Donor 5)
+        
+        my $self = {
+            _dns  => undef,
+            _pid  => undef,
+            _hostgroups => undef,
+            _hg_writer_id => 0,
+            _hg_reader_id => 0,
+            _user => undef,
+            _password => undef,
+            _port => undef,
+            _monitor_user => undef,
+            _monitor_password => undef,
+            _SQL_get_monitor => $SQL_get_monitor,
+            _SQL_get_hg=> $SQL_get_hostgroups,
+            _SQL_get_replication_hg=> $SQL_get_rep_hg,
+            _dbh_proxy => undef,
+            _check_timeout => 800, #timeout in ms
+            _action_nodes => {},
+            _retry_down => 0, # number of retry on a node before declaring it as failed.
+            _retry_up => 0, # number of retry on a node before declaring it OK.
+            _status_changed => 0, #if 1 something had happen and a node had be modify
+            _require_failover => 0, # Valid values are:
+                                      # 0 [default] do not make failover
+                                      # 1 make failover only if HG 8000 is specified in ProxySQL mysl_servers
+                                      # 2 use PXC_CLUSTER_VIEW to identify a server in the same segment
+                                      # 3 do whatever to keep service up also failover to another segment (use PXC_CLUSTER_VIEW) 
+                                                 
+
+        };
+        bless $self, $class;
+        return $self;
+        
+    }
+    sub require_failover{
+        my ( $self, $in ) = @_;
+        $self->{_require_failover} = $in if defined($in);
+        return $self->{_require_failover};
+    }
+
+    sub hg_reader_id{
+        my ( $self, $in ) = @_;
+        $self->{_hg_reader_id} = $in if defined($in);
+        return $self->{_hg_reader_id};
+    }
+    
+    sub status_changed{
+        my ( $self, $in ) = @_;
+        $self->{_status_changed} = $in if defined($in);
+        return $self->{_status_changed};
+    }
+
+    sub retry_down{
+        my ( $self, $in ) = @_;
+        $self->{_retry_down} = $in if defined($in);
+        return $self->{_retry_down};
+    }
+
+    sub retry_up{
+        my ( $self, $in ) = @_;
+        $self->{_retry_up} = $in if defined($in);
+        return $self->{_retry_up};
+    }
+
+    
+    sub debug{
+        my ( $self, $debug ) = @_;
+        $self->{_debug} = $debug if defined($debug);
+        return $self->{_debug};
+    }
+
+    sub action_nodes {
+        my ( $self, $action_nodes ) = @_;
+        $self->{_action_nodes} = $action_nodes if defined($action_nodes);
+        return $self->{_action_nodes};
+    }
+    
+    sub dns {
+        my ( $self, $dns ) = @_;
+        $self->{_dns} = $dns if defined($dns);
+        return $self->{_dns};
+    }
+
+    sub dbh_proxy{
+        my ( $self, $dbh_proxy ) = @_;
+        $self->{_dbh_proxy} = $dbh_proxy if defined($dbh_proxy);
+        return $self->{_dbh_proxy};
+    }
+
+    sub pid {
+        my ( $self, $pid ) = @_;
+        $self->{_pid} = $pid if defined($pid);
+        return $self->{_pid};
+    }
+
+    sub hg_writer_id {
+        my ( $self, $pid ) = @_;
+        $self->{_hg_writer_id} = $pid if defined($pid);
+        return $self->{_hg_writer_id};
+    }
+    
+    sub hostgroups {
+        my ( $self, $hostgroups ) = @_;
+        if (defined $hostgroups){
+            my @HGIds=split('\,', $Param->{hostgroups});
+            
+            foreach my $hg (@HGIds){
+               my $proxy_hg = ProxySqlHG->new();
+               my $proxy_hgM = ProxySqlHG->new();
+               my $proxy_hgB = ProxySqlHG->new();
+               my  ($id,$type) = split /:/, $hg;
+               $proxy_hg->id($id);
+               $proxy_hg->type(lc($type));
+               if(lc($type) eq 'w'){
+                  $self->hg_writer_id($id); 
+               }
+               if(lc($type) eq 'r'){
+                  $self->hg_reader_id($id); 
+               }
+
+               $self->{_hostgroups}->{$id}=($proxy_hg);
+               $proxy_hgM->id(($id + 9000));
+               $proxy_hgM->type("m".lc($type));
+               $self->{_hostgroups}->{$proxy_hgM->id(($id + 9000))}=($proxy_hgM);
+               #add a special group in case of back server for failover
+               if(lc($type) eq "w"){
+                  $proxy_hgM->id(($id + 8000));
+                  $proxy_hgM->type("b".lc($type));
+                  $self->{_hostgroups}->{$proxy_hgM->id(($id + 8000))}=($proxy_hgM);
+               }
+               if(lc($type) eq "r"){
+                  $proxy_hgM->id(($id + 8000));
+                  $proxy_hgM->type("b".lc($type));
+                  $self->{_hostgroups}->{$proxy_hgM->id(($id + 8000))}=($proxy_hgM);
+               }
+
+               if($self->debug >=1){print Utils->print_log(3," Inizializing hostgroup " . $proxy_hg->id ." ".$proxy_hg->type . "with maintenance HG ". $proxy_hgM->id ." ".$proxy_hgM->type."\n") ; }
+            }
+        }
+        return $self->{_hostgroups};
+    }
+    
+    sub user{
+        my ( $self, $user ) = @_;
+        $self->{_user} = $user if defined($user);
+        return $self->{_user};
+    }
+    sub password {
+        my ( $self, $password ) = @_;
+        $self->{_password} = $password if defined($password);
+        return $self->{_password};
+    }
+    
+        sub monitor_user{
+        my ( $self, $monitor_user ) = @_;
+        $self->{_monitor_user} = $monitor_user if defined($monitor_user);
+        return $self->{_monitor_user};
+    }
+    sub monitor_password {
+        my ( $self, $monitor_password ) = @_;
+        $self->{_monitor_password} = $monitor_password if defined($monitor_password);
+        return $self->{_monitor_password};
+    }
+
+    sub port {
+        my ( $self, $port ) = @_;
+        $self->{_port} = $port if defined($port);
+        return $self->{_port};
+    }
+
+    sub check_timeout{
+        my ( $self, $check_timeout ) = @_;
+        $self->{_check_timeout} = $check_timeout if defined($check_timeout);
+        return $self->{_check_timeout};
+    }
+    
+    #Connect method connect an populate the cluster returns the Galera cluster
+    sub connect{
+        my ( $self, $port ) = @_;
+        my $dbh = Utils::get_connection($self->{_dns}, $self->{_user}, $self->{_password},' ');
+        $self->{_dbh_proxy} = $dbh;
+        
+        # get monitor user/pw                
+        my $cmd = $self->{_SQL_get_monitor};
+
+
+        my $sth = $dbh->prepare($cmd);
+        $sth->execute();
+        while (my $ref = $sth->fetchrow_hashref()) {
+            if($ref->{'name'} eq 'mysql-monitor_password' ){$self->{_monitor_password} = $ref->{'value'};}
+            if($ref->{'name'} eq 'mysql-monitor_username' ) {$self->{_monitor_user} = $ref->{'value'};}
+            #This is for now comment out.
+            # this is related to issue #10, where the node is not answering in time to the check.
+            # The timeout cannot be the same of the the ProxySQL read_only check
+            #if($ref->{'name'} eq 'mysql-monitor_read_only_timeout' ) {$self->{_check_timeout} = $ref->{'value'};}
+            
+        }
+        if($self->debug >=1){print Utils->print_log(3," Connecting to ProxySQL " . $self->{_dns}. "\n" ); }
+        
+    }
+    
+    sub disconnect{
+        my ( $self, $port ) = @_;
+        $self->{_dbh_proxy}->disconnect;
+	
+        
+    }
+    sub get_galera_cluster{
+        my ( $self, $in ) = @_;
+        $self->{_galera_cluster} = $in if defined($in);
+        return $self->{_galera_cluster};
+    }
+
+    sub set_galera_cluster(){
+        my ( $self, $port ) = @_;
+        my $galera_cluster = Galeracluster->new();
+        
+        $galera_cluster->hostgroups($self->hostgroups);
+        $galera_cluster->dbh_proxy($self->dbh_proxy);
+        $galera_cluster->check_timeout($self->check_timeout);
+        $galera_cluster->monitor_user($self->monitor_user);
+        $galera_cluster->monitor_password($self->monitor_password);
+        $galera_cluster->debug($self->debug);
+        $galera_cluster->hg_writer_id($self->hg_writer_id);
+        $galera_cluster->hg_reader_id($self->hg_reader_id);
+        $galera_cluster->singlewriter($Param->{single_writer});
+        $galera_cluster->writer_is_reader($Param->{writer_is_reader});
+        $galera_cluster->ssl_certificates_path($Param->{ssl_certs_path});
+        
+        $self->get_galera_cluster($galera_cluster);
+       if($self->debug >=1){print Utils->print_log(3," Galera cluster object created  " . caller(3). "\n" ); }
+    }
+    
+    sub evaluate_nodes{
+      my ($proxynode,$GGalera_cluster)  = @_ ;
+      my ( $nodes ) = $GGalera_cluster->{_nodes};
+      my $action_nodes = undef;
+
+
+     #Rules:
+	    #see rules in the doc
+	    
+     #do the checks
+     if($proxynode->debug >=1){print Utils->print_log(3," Evaluate nodes state \n" ) }	
+       foreach my $key (sort keys %{$nodes}){
+         if(defined $nodes->{$key} ){
+	
+            #only if node has HG that is not maintenance it can evaluate to be put down in some way
+            if($nodes->{$key}->{_hostgroups} < 8000
+               && $nodes->{$key}->{_process_status} > 0){
+                #Check major exclusions
+                # 1) wsrep state
+                # 2) Node is not read only
+                # 3) at least another node in the HG 
+          
+                if( $nodes->{$key}->wsrep_status == 2
+                  && $nodes->{$key}->read_only eq "OFF"
+                  #&& $GGalera_cluster->{_main_segment} != $nodes->{$key}->wsrep_segment
+                  && $nodes->{$key}->proxy_status ne "OFFLINE_SOFT"
+                  ){
+                     if($GGalera_cluster->{_hostgroups}->{$nodes->{$key}->{_hostgroups}}->{_size} <= 1){
+                        print Utils->print_log(3," Node ".$nodes->{$key}->ip.";".$nodes->{$key}->port.";".$nodes->{$key}->hostgroups."; Is in state ".$nodes->{$key}->wsrep_status
+                                               .". But I will not move to OFFLINE_SOFT given last node left in the Host group \n");
+                        next;
+                     }
+                     $action_nodes->{$nodes->{$key}->ip.";".$nodes->{$key}->port.";".$nodes->{$key}->hostgroups.";".$nodes->{$key}->{_MOVE_DOWN_OFFLINE}}= $nodes->{$key};
+                     #if retry is > 0 then it's managed
+                     if($proxynode->retry_down > 0){
+                         $nodes->{$key}->get_retry_down($nodes->{$key}->get_retry_down + 1);
+                     }
+                     if($proxynode->debug >=1){print Utils->print_log(3," Evaluate nodes state "
+                         .$nodes->{$key}->ip.";".$nodes->{$key}->port.";".$nodes->{$key}->hostgroups.";".$nodes->{$key}->{_MOVE_DOWN_OFFLINE}
+                         ." Retry #".$nodes->{$key}->get_retry_down."\n" ) }	
+                     next;
+                   }
+             
+               if( $nodes->{$key}->wsrep_status ne 4
+                && $nodes->{$key}->wsrep_status ne 2){
+                     $action_nodes->{$nodes->{$key}->ip.";".$nodes->{$key}->port.";".$nodes->{$key}->hostgroups.";".$nodes->{$key}->{_MOVE_DOWN_HG_CHANGE}}= $nodes->{$key};
+             
+                     #if retry is > 0 then it's managed
+                     if($proxynode->retry_down > 0){
+                         $nodes->{$key}->get_retry_down($nodes->{$key}->get_retry_down + 1);
+                     }
+             
+                    if($proxynode->debug >=1){print Utils->print_log(3," Evaluate nodes state "
+                        .$nodes->{$key}->ip.";".$nodes->{$key}->port.";".$nodes->{$key}->hostgroups.";".$nodes->{$key}->{_MOVE_DOWN_HG_CHANGE}
+                        ." Retry #".$nodes->{$key}->get_retry_down."\n" )   }	
+             
+                   next;
+               }
+              
+               #3) Node/cluster in non primary
+               if($nodes->{$key}->cluster_status ne "Primary"){
+                  $action_nodes->{$nodes->{$key}->ip.";".$nodes->{$key}->port.";".$nodes->{$key}->hostgroups.";".$nodes->{$key}->{_MOVE_DOWN_HG_CHANGE}}= $nodes->{$key};
+             
+                  #if retry is > 0 then it's managed
+                  if($proxynode->retry_down > 0){
+                      $nodes->{$key}->get_retry_down($nodes->{$key}->get_retry_down + 1);
+                  }
+              
+                  if($proxynode->debug >=1){print Utils->print_log(3," Evaluate nodes state "
+                      .$nodes->{$key}->ip.";".$nodes->{$key}->port.";".$nodes->{$key}->hostgroups.";".$nodes->{$key}->{_MOVE_DOWN_HG_CHANGE}
+                      ." Retry #".$nodes->{$key}->get_retry_down."\n" )   }	
+                  next;
+               }		
+    
+               # 4) wsrep_reject_queries=NONE
+               if($nodes->{$key}->wsrep_rejectqueries ne "NONE" && $nodes->{$key}->proxy_status ne "OFFLINE_SOFT"){
+                  my $inc =0;
+                  if($nodes->{$key}->wsrep_rejectqueries eq "ALL"){
+                      $action_nodes->{$nodes->{$key}->ip.";".$nodes->{$key}->port.";".$nodes->{$key}->hostgroups.";".$nodes->{$key}->{_MOVE_DOWN_HG_CHANGE}}= $nodes->{$key};
+                      $inc=1;
+                  }else{
+                      $action_nodes->{$nodes->{$key}->ip.";".$nodes->{$key}->port.";".$nodes->{$key}->hostgroups.";".$nodes->{$key}->{_MOVE_DOWN_HG_CHANGE}}= $nodes->{$key};
+                      $inc=1;
+                  }
+                  #if retry is > 0 then it's managed
+                  if($proxynode->retry_down > 0 && $inc > 0){
+                      $nodes->{$key}->get_retry_down($nodes->{$key}->get_retry_down + 1);
+                  }
+               
+                  if($proxynode->debug >=1){
+                     print Utils->print_log(3," Evaluate nodes state "
+                        .$nodes->{$key}->ip.";".$nodes->{$key}->port.";".$nodes->{$key}->hostgroups.";".$nodes->{$key}->{_MOVE_DOWN_HG_CHANGE}
+                        ." Retry #".$nodes->{$key}->get_retry_down."\n" )   }	
+                  next;
+               }
+               
+               #5) Donor, node reject queries =1 size of cluster > 2 of nodes in the same segments
+               if($nodes->{$key}->wsrep_status eq 2
+                  && $nodes->{$key}->wsrep_donorrejectqueries eq "ON"){
+                  $action_nodes->{$nodes->{$key}->ip.";".$nodes->{$key}->port.";".$nodes->{$key}->hostgroups.";".$nodes->{$key}->{_MOVE_DOWN_HG_CHANGE}}= $nodes->{$key};
+                  #if retry is > 0 then it's managed
+                  if($proxynode->retry_down > 0){
+                      $nodes->{$key}->get_retry_down($nodes->{$key}->get_retry_down + 1);
+                  }
+               
+                  if($proxynode->debug >=1){
+                     print Utils->print_log(3," Evaluate nodes state "
+                        .$nodes->{$key}->ip.";".$nodes->{$key}->port.";".$nodes->{$key}->hostgroups.";".$nodes->{$key}->{_MOVE_DOWN_HG_CHANGE}
+                        ." Retry #".$nodes->{$key}->get_retry_down."\n" )   }	
+               
+                  next;
+               }
+               
+               #Set OFFLINE_SOFT a writer: 
+               #1) donor node reject queries - 0
+               #2)size of cluster > 2 of nodes in the same segments
+               #3) more then one writer in the same HG
+               #4) Node had pxc_maint_mode set to anything except DISABLED, not matter what it will go in OFFLINE_SOFT
+               
+               if(
+                  $nodes->{$key}->read_only eq "ON"
+                  && $nodes->{$key}->{_hostgroups} ==  $GGalera_cluster->{_hg_writer_id}
+                  && $nodes->{$key}->wsrep_donorrejectqueries eq "OFF"
+                  && $nodes->{$key}->proxy_status ne "OFFLINE_SOFT"
+                  ){
+                    ## In case READ_ONLY is OFF and we have only a node left but desync do not put it down
+                    #if( $GGalera_cluster->{_size}->{$nodes->{$key}->{_wsrep_segment}} == 1
+                    #    &&$nodes->{$key}->read_only eq "OFF"){
+                    #    next; 
+                    #}
+                    $action_nodes->{$nodes->{$key}->ip.";".$nodes->{$key}->port.";".$nodes->{$key}->hostgroups.";".$nodes->{$key}->{_MOVE_DOWN_OFFLINE}}= $nodes->{$key};
+                    #if retry is > 0 then it's managed
+                    if($proxynode->retry_down > 0){
+                        $nodes->{$key}->get_retry_down($nodes->{$key}->get_retry_down + 1);
+                    }
+                 
+                    if($proxynode->debug >=1){print Utils->print_log(3," Evaluate nodes state "
+                        .$nodes->{$key}->ip.";".$nodes->{$key}->port.";".$nodes->{$key}->hostgroups.";".$nodes->{$key}->{_MOVE_DOWN_OFFLINE}
+                        ." Retry #".$nodes->{$key}->get_retry_down."\n" ) }	
+                 
+                    next;
+               }
+               
+               #4) Node had pxc_maint_mode set to anything except DISABLED, not matter what it will go in OFFLINE_SOFT
+               if( defined $nodes->{$key}->pxc_maint_mode 
+                && $nodes->{$key}->pxc_maint_mode ne "DISABLED"
+                && $nodes->{$key}->proxy_status ne "OFFLINE_SOFT"){
+                     $action_nodes->{$nodes->{$key}->ip.";".$nodes->{$key}->port.";".$nodes->{$key}->hostgroups.";".$nodes->{$key}->{_MOVE_TO_MAINTENANCE}}= $nodes->{$key};
+             
+                     #if retry is > 0 then it's managed
+                     if($proxynode->retry_down > 0){
+                         $nodes->{$key}->get_retry_down($proxynode->retry_down ); # this is a known state and we do not want any delay set the retry to his max
+                     }
+                    
+                    if(
+                      $nodes->{$key}->{_hostgroups} ==  $GGalera_cluster->{_hg_writer_id}
+                      && $GGalera_cluster->{_singlewriter} > 0
+                      ){
+                       $GGalera_cluster->{_haswriter} = $GGalera_cluster->{_haswriter} -1;
+                    }
+                    
+                    
+                    if($proxynode->debug >=1){print Utils->print_log(3," Evaluate nodes state "
+                        .$nodes->{$key}->ip.";".$nodes->{$key}->port.";".$nodes->{$key}->hostgroups.";".$nodes->{$key}->{_MOVE_TO_MAINTENANCE}
+                        ." Retry #".$nodes->{$key}->get_retry_down."\n" )   }	
+             
+                   next;
+               }
+       
+               # Node must be removed if writer is reader is disable and node is in writer group
+               if($nodes->{$key}->wsrep_status eq 4
+                    && $nodes->{$key}->wsrep_rejectqueries eq "NONE"
+                    && $nodes->{$key}->read_only eq "OFF"
+                    && $nodes->{$key}->cluster_status eq "Primary"
+                    && $nodes->{$key}->hostgroups == $proxynode->{_hg_reader_id}
+                    && $GGalera_cluster->{_writer_is_reader} < 1
+                    && $nodes->{$key}->proxy_status eq "ONLINE"
+                    ){
+                      #my $nodes_read_ips = join(',', @{$GGalera_cluster->{_reader_nodes}});
+                      my $nodes_write_ips = join(',', @{$GGalera_cluster->{_writer_nodes}});
+                      
+                      my $ip = "$nodes->{$key}->{_ip}:$nodes->{$key}->{_port}";
+                      
+                      if($nodes_write_ips =~ m/$ip/ ){
+                        $action_nodes->{$nodes->{$key}->ip.";".$nodes->{$key}->port.";".$nodes->{$key}->hostgroups.";".$nodes->{$key}->{_DELETE_NODE}}= $nodes->{$key}; 
+                         #if retry is > 0 then it's managed
+                         if($proxynode->retry_up > 0){
+                             $nodes->{$key}->get_retry_up($nodes->{$key}->get_retry_up +1);
+                         }
+                         print Utils->print_log(3," Writer is also reader disabled removing node from reader Hostgroup "
+                               .$nodes->{$key}->ip.";".$nodes->{$key}->port.";".$nodes->{$key}->hostgroups.";".$nodes->{$key}->{_DELETE_NODE}
+                               ." Retry #".$nodes->{$key}->get_retry_up."\n" );
+                         
+                        next;
+                      }
+
+                 }
+      
+               
+            }    
+            #Node comes back from offline_soft when (all of them):
+            # 1) Node state is 4
+            # 3) wsrep_reject_queries = none
+            # 4) Primary state
+            # 5) pxc_maint_mode is DISABLED or undef
+   
+            if($nodes->{$key}->wsrep_status eq 4
+               && $nodes->{$key}->proxy_status eq "OFFLINE_SOFT"
+               && $nodes->{$key}->wsrep_rejectqueries eq "NONE"
+               && $nodes->{$key}->read_only eq "OFF"
+               &&$nodes->{$key}->cluster_status eq "Primary"
+               &&(!defined $nodes->{$key}->pxc_maint_mode || $nodes->{$key}->pxc_maint_mode eq "DISABLED") 
+               && $nodes->{$key}->hostgroups < 8000
+               ){
+                  if($GGalera_cluster->haswriter > 0
+                     && $GGalera_cluster->singlewriter > 0
+                     && $nodes->{$key}->hostgroups == $GGalera_cluster->hg_writer_id
+                     ){
+                     $action_nodes->{$nodes->{$key}->ip.";".$nodes->{$key}->port.";".$nodes->{$key}->hostgroups.";".$nodes->{$key}->{_DELETE_NODE}}= $nodes->{$key};
+                     #if retry is > 0 then it's managed
+                     if($proxynode->retry_up > 0){
+                         $nodes->{$key}->get_retry_up($nodes->{$key}->get_retry_up +1);
+                     }
+                     if($proxynode->debug <=1){
+                        print Utils->print_log(3, " Evaluate nodes state "
+                            .$nodes->{$key}->ip.";".$nodes->{$key}->port.";".$nodes->{$key}->hostgroups.";".$nodes->{$key}->{_DELETE_NODE}
+                            ." Retry #".$nodes->{$key}->get_retry_up."\n" ) }			    
+                     next;
+                  }
+                 else{
+                     $action_nodes->{$nodes->{$key}->ip.";".$nodes->{$key}->port.";".$nodes->{$key}->hostgroups.";".$nodes->{$key}->{_MOVE_UP_OFFLINE}}= $nodes->{$key};
+                     #if retry is > 0 then it's managed
+                     if($proxynode->retry_up > 0){
+                         $nodes->{$key}->get_retry_up($nodes->{$key}->get_retry_up +1);
+                     }
+                     if($proxynode->debug <=1){
+                        print Utils->print_log(3, " Evaluate nodes state "
+                            .$nodes->{$key}->ip.";".$nodes->{$key}->port.";".$nodes->{$key}->hostgroups.";".$nodes->{$key}->{_MOVE_UP_OFFLINE}
+                            ." Retry #".$nodes->{$key}->get_retry_up."\n" ) }			    
+                     next;
+                 }
+            }
+            
+            # Node comes back from maintenance HG when (all of them):
+            # 1) node state is 4
+            # 3) wsrep_reject_queries = none
+            # 4) Primary state
+            if($nodes->{$key}->wsrep_status eq 4
+               && $nodes->{$key}->wsrep_rejectqueries eq "NONE"
+               && $nodes->{$key}->cluster_status eq "Primary"
+               && $nodes->{$key}->hostgroups >= 9000
+               ){
+                 $action_nodes->{$nodes->{$key}->ip.";".$nodes->{$key}->port.";".$nodes->{$key}->hostgroups.";".$nodes->{$key}->{_MOVE_UP_HG_CHANGE}}= $nodes->{$key};
+                 #if retry is > 0 then it's managed
+                 if($proxynode->retry_up > 0){
+                     $nodes->{$key}->get_retry_up($nodes->{$key}->get_retry_up +1);
+                 }
+                 if($proxynode->debug >=1){
+                    print Utils->print_log(3," Evaluate nodes state "
+                       .$nodes->{$key}->ip.";".$nodes->{$key}->port.";".$nodes->{$key}->hostgroups.";".$nodes->{$key}->{_MOVE_UP_HG_CHANGE}
+                       ." Retry #".$nodes->{$key}->get_retry_up."\n" ) }			    
+                 next;
+            }
+   
+           #Special case when a node goes down it goes through several state and the check disable it moving form original group
+           #This is to remove it to his original HG when is not reachable
+           if($nodes->{$key}->{_process_status} < 0 
+              && $nodes->{$key}->hostgroups >= 9000
+             ){
+                 #if retry is > 0 then it's managed
+                 if($proxynode->retry_up > 0){
+                     $nodes->{$key}->get_retry_up($nodes->{$key}->get_retry_up +1);
+                 }
+              
+                 $action_nodes->{$nodes->{$key}->ip.";".$nodes->{$key}->port.";".$nodes->{$key}->hostgroups.";".$nodes->{$key}->{_MOVE_UP_HG_CHANGE}}= $nodes->{$key};
+                 if($proxynode->debug >=1){
+                    print Utils->print_log(3," Evaluate nodes state "
+                       .$nodes->{$key}->ip.";".$nodes->{$key}->port.";".$nodes->{$key}->hostgroups.";".$nodes->{$key}->{_MOVE_UP_HG_CHANGE}
+                       ." Retry #".$nodes->{$key}->get_retry_up."\n" ) }			    
+                 next;
+           }
+           
+           #Check if any node that is in the read backup host group is not present in the readhostgroup while it should.
+           #If identify it will add to the read HG
+           
+           if($nodes->{$key}->wsrep_status eq 4
+               && $nodes->{$key}->wsrep_rejectqueries eq "NONE"
+               && $nodes->{$key}->cluster_status eq "Primary"
+               && $nodes->{$key}->hostgroups == (8000 + $proxynode->{_hg_reader_id})
+               ){
+                 my $nodes_read_ips = join(',', @{$GGalera_cluster->{_reader_nodes}});
+                 my $nodes_write_ips = join(',', @{$GGalera_cluster->{_writer_nodes}});
+                 
+                 my $ip = "$nodes->{$key}->{_ip}:$nodes->{$key}->{_port}";
+                 
+                 if($nodes_read_ips =~ m/$ip/
+                    || ( $nodes_write_ips =~ m/$ip/ 
+                        && $GGalera_cluster->{_writer_is_reader} < 1)){
+                  if($proxynode->debug >=1){
+                    print Utils->print_log(3," Node already ONLINE in read hg "
+                       .$nodes->{$key}->ip.";".$nodes->{$key}->port.";\n" ) }			 
+                 }else{
+                    $action_nodes->{$nodes->{$key}->ip.";".$nodes->{$key}->port.";".$nodes->{$key}->hostgroups.";".$nodes->{$key}->{_INSERT_READ}}= $nodes->{$key}; 
+                    #if retry is > 0 then it's managed
+                    if($proxynode->retry_up > 0){
+                        $nodes->{$key}->get_retry_up($nodes->{$key}->get_retry_up +1);
+                    }
+                    if($proxynode->debug >=1){
+                       print Utils->print_log(3," Evaluate nodes state "
+                          .$nodes->{$key}->ip.";".$nodes->{$key}->port.";".$nodes->{$key}->hostgroups.";".$nodes->{$key}->{_INSERT_READ}
+                          ." Retry #".$nodes->{$key}->get_retry_up."\n" )
+                    }
+                 }
+                 next;
+            }
+
+           #Check if any node that is in the write backup host group is not present in the WRITE hostgroup while it should WHEN MULTIPLE WRITERS.
+           #If identify it will add to the read HG
+           
+           if($nodes->{$key}->wsrep_status eq 4
+               && $nodes->{$key}->wsrep_rejectqueries eq "NONE"
+               && $nodes->{$key}->read_only eq "OFF"
+               && $nodes->{$key}->cluster_status eq "Primary"
+               && $nodes->{$key}->hostgroups == (8000 + $proxynode->{_hg_writer_id})
+               && $GGalera_cluster->{_singlewriter} < 1
+               ){
+                 #my $nodes_read_ips = join(',', @{$GGalera_cluster->{_reader_nodes}});
+                 my $nodes_write_ips = join(',', @{$GGalera_cluster->{_writer_nodes}});
+                 
+                 my $ip = "$nodes->{$key}->{_ip}:$nodes->{$key}->{_port}";
+                 
+                 if($nodes_write_ips =~ m/$ip/ 
+                     && $GGalera_cluster->{_single_writer} < 1){
+                  if($proxynode->debug >=1){
+                    print Utils->print_log(3," Node already ONLINE in write hg "
+                       .$nodes->{$key}->ip.";".$nodes->{$key}->port.";\n" ) }			 
+                 }else{
+                    $action_nodes->{$nodes->{$key}->ip.";".$nodes->{$key}->port.";".$nodes->{$key}->hostgroups.";".$nodes->{$key}->{_INSERT_WRITE}}= $nodes->{$key}; 
+                    #if retry is > 0 then it's managed
+                    if($proxynode->retry_up > 0){
+                        $nodes->{$key}->get_retry_up($nodes->{$key}->get_retry_up +1);
+                    }
+                    if($proxynode->debug >=1){
+                       print Utils->print_log(3," Evaluate nodes state "
+                          .$nodes->{$key}->ip.";".$nodes->{$key}->port.";".$nodes->{$key}->hostgroups.";".$nodes->{$key}->{_INSERT_WRITE}
+                          ." Retry #".$nodes->{$key}->get_retry_up."\n" )
+                    }
+                 }
+                 next;
+            }
+           
+           
+           # in the case node is not in one of the declared state
+           # BUT it has the counter retry set THEN I reset it to 0 whatever it was because
+           # I assume it is ok now
+           if($proxynode->retry_up > 0
+              && $nodes->{$key}->get_retry_up > 0){
+               $nodes->{$key}->get_retry_up(0);
+               $action_nodes->{$nodes->{$key}->ip.";".$nodes->{$key}->port.";".$nodes->{$key}->hostgroups.";".$nodes->{$key}->{_SAVE_RETRY}}= $nodes->{$key};
+           }
+           if($proxynode->retry_down > 0
+              && $nodes->{$key}->get_retry_down > 0){
+               $nodes->{$key}->get_retry_down(0);
+               $action_nodes->{$nodes->{$key}->ip.";".$nodes->{$key}->port.";".$nodes->{$key}->hostgroups.";".$nodes->{$key}->{_SAVE_RETRY}}= $nodes->{$key};
+           }
+		
+         }
+       }
+       $proxynode->action_nodes($action_nodes);
+       
+      #failover has higher priority BUT if it happens before other action will interfere with recovery and move nodes that may prevent the failover to happen
+      # and it will ends the script work (if successful)
+      #if($GGalera_cluster->{_writers} > 1
+      #   && $GGalera_cluster->{_singlewriter} > 0)
+      #{
+      #  $proxynode->{_require_failover} = 1;
+      #}
+      
+      
+      
+      if($proxynode->require_failover > 0
+         && !defined $action_nodes
+         ){
+        if($GGalera_cluster->haswriter < 1
+           || ( $GGalera_cluster->{_writers} > 1 && $GGalera_cluster->{_singlewriter} > 0)
+           ){
+          print Utils->print_log(2,"Fail-over in action Using Method = $proxynode->{_require_failover}\n" );
+          if($proxynode->initiate_failover($GGalera_cluster) >0){
+             #if($proxynode->debug >=1){
+               print Utils->print_log(2,"!!!! FAILOVER !!!!! \n Cluster was without WRITER I have try to restore service promoting a node\n" );
+               #exit 0;
+             #}
+          }
+        }
+      }
+      elsif($proxynode->require_failover > 0
+            && $GGalera_cluster->haswriter < 1){
+       print Utils->print_log(2,"PXC maintenance on single writer, is asking for failover. Fail-over in action Using Method = $proxynode->{_require_failover}\n" );
+       $proxynode->push_changes;
+       if($proxynode->initiate_failover($GGalera_cluster) >0){
+             #if($proxynode->debug >=1){
+               print Utils->print_log(2,"!!!! FAILOVER !!!!! \n Cluster was without WRITER I have try to restore service promoting a node\n" );
+               #exit 0;
+             #}
+        }
+      }
+
+    }
+    
+    sub push_changes{
+        my ($proxynode)  = @_ ;
+        my $node = GaleraNode->new();
+        my $SQL_command="";
+        
+        
+        foreach my $key (sort keys %{$proxynode->{_action_nodes}}){
+            my ($host,  $port, $hg, $action) = split /s*;\s*/, $key;
+           
+            SWITCH: {
+                     if ($action == $node->MOVE_DOWN_OFFLINE) { if($proxynode->{_action_nodes}->{$key}->get_retry_down >= $proxynode->retry_down){$proxynode->move_node_offline($key,$proxynode->{_action_nodes}->{$key})}; last SWITCH; }
+                     if ($action == $node->MOVE_DOWN_HG_CHANGE) { if($proxynode->{_action_nodes}->{$key}->get_retry_down >= $proxynode->retry_down){ $proxynode->move_node_down_hg_change($key,$proxynode->{_action_nodes}->{$key})}; last SWITCH; }
+                     if ($action == $node->MOVE_UP_OFFLINE) { if($proxynode->{_action_nodes}->{$key}->get_retry_up >= $proxynode->retry_up){ $proxynode->move_node_up_from_offline($key,$proxynode->{_action_nodes}->{$key})}; last SWITCH; }
+                     if ($action == $node->MOVE_UP_HG_CHANGE) { if($proxynode->{_action_nodes}->{$key}->get_retry_up >= $proxynode->retry_up){$proxynode->move_node_up_from_hg_change($key,$proxynode->{_action_nodes}->{$key})}; last SWITCH; }
+                     if ($action == $node->MOVE_TO_MAINTENANCE) { if($proxynode->{_action_nodes}->{$key}->get_retry_down >= $proxynode->retry_down){$proxynode->move_node_to_maintenance($key,$proxynode->{_action_nodes}->{$key})}; last SWITCH; }
+                     if ($action == $node->DELETE_NODE) {
+                         if($proxynode->{_action_nodes}->{$key}->get_retry_up >= $proxynode->retry_up){
+                          $proxynode->delete_node_from_hostgroup($key,$proxynode->{_action_nodes}->{$key})}; last SWITCH; }
+                     if($action == $node->INSERT_READ){if($proxynode->{_action_nodes}->{$key}->get_retry_up >= $proxynode->retry_up){
+                          $proxynode->insert_reader($key,$proxynode->{_action_nodes}->{$key})
+                          }; last SWITCH;                      
+                     }
+                                          
+                     if($action == $node->INSERT_WRITE){if($proxynode->{_action_nodes}->{$key}->get_retry_up >= $proxynode->retry_up){
+                          $proxynode->insert_writer($key,$proxynode->{_action_nodes}->{$key})
+                          }; last SWITCH;                      
+                     }
+
+            }
+            if($proxynode->retry_up > 0 || $proxynode->retry_down > 0){
+               save_retry($proxynode,$key,$proxynode->{_action_nodes}->{$key});
+            }      
+        }   
+        $proxynode->{_action_nodes} = undef;
+    }
+    
+    sub save_retry{
+        #this action will take place only if retry is active
+        my ($self,$key,$node) = @_;
+        my ($host,  $port, $hg,$action) = split /s*;\s*/, $key;
+        
+        if($self->debug >=1){print Utils->print_log(4,"Check retry Node:".$host." port:".$port . " hg:".$hg ." Time IN \n");}
+       
+        my $sql_string = "UPDATE mysql_servers SET comment='"
+            .$node->{_comment}
+            .$self->get_galera_cluster->cluster_identifier."_retry_up=".$node->get_retry_up
+            .";".$self->get_galera_cluster->cluster_identifier."_retry_down=".$node->get_retry_down
+            .";' WHERE hostgroup_id=$hg AND hostname='$host' AND port='$port'";
+       
+        $self->{_dbh_proxy}->do($sql_string) or die "Couldn't execute statement: " .  $self->{_dbh_proxy}->errstr;
+        $self->{_dbh_proxy}->do("LOAD MYSQL SERVERS TO RUNTIME") or die "Couldn't execute statement: " .  $self->{_dbh_proxy}->errstr;
+        
+        if($self->debug >=1){print Utils->print_log(2," Reset retry to UP:".$node->get_retry_up." Down:".$node->get_retry_down."for node:" .$key
+              ." SQL:" .$sql_string
+              ."\n")}  ;			    
+        if($self->debug >=1){print Utils->print_log(4,"Check retry Node:".$host." port:".$port . " hg:".$hg ." Time OUT \n");}
+    }
+ 
+    sub move_node_offline{
+        #this action involve only the proxy so we will 
+        my ($proxynode, $key,$node) = @_;
+        
+        my ($host,  $port, $hg,$action) = split /s*;\s*/, $key;
+        my $proxy_sql_command= " UPDATE mysql_servers SET status='OFFLINE_SOFT' WHERE hostgroup_id=$hg AND hostname='$host' AND port='$port'";
+        $proxynode->{_dbh_proxy}->do($proxy_sql_command) or die "Couldn't execute statement: " .  $proxynode->{_dbh_proxy}->errstr;
+        $proxynode->{_dbh_proxy}->do("LOAD MYSQL SERVERS TO RUNTIME") or die "Couldn't execute statement: " .  $proxynode->{_dbh_proxy}->errstr;
+        
+        print Utils->print_log(2," Move node:" .$key
+              ." SQL:" .$proxy_sql_command
+              ."\n")  ;			    
+    }
+    sub move_node_to_maintenance{
+        #this action involve only the proxy so we will 
+        my ($proxynode, $key,$node) = @_;
+        
+        my ($host,  $port, $hg,$action) = split /s*;\s*/, $key;
+        my $proxy_sql_command= " UPDATE mysql_servers SET status='OFFLINE_SOFT' WHERE hostgroup_id=$hg AND hostname='$host' AND port='$port'";
+        $proxynode->{_dbh_proxy}->do($proxy_sql_command) or die "Couldn't execute statement: " .  $proxynode->{_dbh_proxy}->errstr;
+        $proxynode->{_dbh_proxy}->do("LOAD MYSQL SERVERS TO RUNTIME") or die "Couldn't execute statement: " .  $proxynode->{_dbh_proxy}->errstr;
+        
+        print Utils->print_log(2," Move node:" .$key
+              ." SQL:" .$proxy_sql_command
+              ."\n")  ;			    
+    }
+
+    #remove a node from an hostgroup
+    sub delete_node_from_hostgroup{
+        #this action involve only the proxy so we will 
+        my ($proxynode, $key,$node) = @_;
+        
+        my ($host,  $port, $hg,$action) = split /s*;\s*/, $key;
+        my $proxy_sql_command= " DELETE from mysql_servers WHERE hostgroup_id=$hg AND hostname='$host' AND port='$port'";
+        $proxynode->{_dbh_proxy}->do($proxy_sql_command) or die "Couldn't execute statement: " .  $proxynode->{_dbh_proxy}->errstr;
+        $proxynode->{_dbh_proxy}->do("LOAD MYSQL SERVERS TO RUNTIME") or die "Couldn't execute statement: " .  $proxynode->{_dbh_proxy}->errstr;
+        
+        print Utils->print_log(2," DELETE node:" .$key
+              ." SQL:" .$proxy_sql_command
+              ."\n")  ;			    
+    }
+
+
+
+
+    #move a node to a maintenance HG ((9000 + HG id))
+    sub move_node_down_hg_change{
+        my ($proxynode, $key,$node) = @_;
+        
+        my ($host,  $port, $hg,$action) = split /s*;\s*/, $key;
+        if($hg > 9000) {return 1;}
+        
+        my $node_sql_command = "SET GLOBAL READ_ONLY=1;";
+        my $proxy_sql_command =" UPDATE mysql_servers SET hostgroup_id=".(9000 + $hg)." WHERE hostgroup_id=$hg AND hostname='$host' AND port='$port'";
+        $proxynode->{_dbh_proxy}->do($proxy_sql_command) or die "Couldn't execute statement: " .  $proxynode->{_dbh_proxy}->errstr;
+        $proxynode->{_dbh_proxy}->do("LOAD MYSQL SERVERS TO RUNTIME") or die "Couldn't execute statement: " .  $proxynode->{_dbh_proxy}->errstr;
+        print Utils->print_log(2," Move node:" .$key
+            ." SQL:" .$proxy_sql_command
+            ."\n" );			    
+    }
+
+    #Bring back a node that is just offline
+    sub move_node_up_from_offline{
+        my ($proxynode, $key,$node) = @_;
+        my ($host,  $port, $hg,$action) = split /s*;\s*/, $key;
+        my $proxy_sql_command= " UPDATE mysql_servers SET status='ONLINE' WHERE hostgroup_id=$hg AND hostname='$host' AND port='$port'";
+        $proxynode->{_dbh_proxy}->do($proxy_sql_command) or die "Couldn't execute statement: " .  $proxynode->{_dbh_proxy}->errstr;
+        my $error_code = $proxynode->{_dbh_proxy}->err();
+        $proxynode->{_dbh_proxy}->do("LOAD MYSQL SERVERS TO RUNTIME") or die "Couldn't execute statement: " .  $proxynode->{_dbh_proxy}->errstr;
+        print Utils->print_log(2," Move node:" .$key
+            ." SQL:" .$proxy_sql_command
+            ."\n" );			    
+    }
+
+    #move a node back to his original HG ((HG id - 9000))
+    sub move_node_up_from_hg_change{
+        my ($proxynode, $key,$node) = @_;
+        
+        my ($host,  $port, $hg,$action) = split /s*;\s*/, $key;
+        #my $node_sql_command = "SET GLOBAL READ_ONLY=1;";
+        my $proxy_sql_command =" UPDATE mysql_servers SET hostgroup_id=".($hg - 9000)." WHERE hostgroup_id=$hg AND hostname='$host' AND port='$port'";
+        $proxynode->{_dbh_proxy}->do($proxy_sql_command) or warn "Couldn't execute statement: " .  $proxynode->{_dbh_proxy}->errstr;
+        my $error_code = $proxynode->{_dbh_proxy}->err();
+        $proxynode->{_dbh_proxy}->do("LOAD MYSQL SERVERS TO RUNTIME") or die "Couldn't execute statement: " .  $proxynode->{_dbh_proxy}->errstr;
+        print Utils->print_log(2," Move node:" .$key
+            ." SQL:" .$proxy_sql_command
+            ."\n" ) ;			    
+    }
+    
+    #move a node back to his original HG ((HG id - 9000))
+    sub add_node_to_readers{
+        my ($proxynode, $key,$node) = @_;
+        
+        my ($host,  $port, $hg,$action) = split /s*;\s*/, $key;
+        my $node_sql_command = "SET GLOBAL READ_ONLY=1;";
+        my $proxy_sql_command =" UPDATE mysql_servers SET hostgroup_id=".($hg - 9000)." WHERE hostgroup_id=$hg AND hostname='$host' AND port='$port'";
+        $proxynode->{_dbh_proxy}->do($proxy_sql_command) or die "Couldn't execute statement: " .  $proxynode->{_dbh_proxy}->errstr;
+        my $error_code = $proxynode->{_dbh_proxy}->err();
+        $proxynode->{_dbh_proxy}->do("LOAD MYSQL SERVERS TO RUNTIME") or die "Couldn't execute statement: " .  $proxynode->{_dbh_proxy}->errstr;
+        print Utils->print_log(2," Move node:" .$key
+            ." SQL:" .$proxy_sql_command
+            ."\n" ) ;			    
+    }
+
+   sub insert_reader{
+         my ($proxynode, $key,$node) = @_;
+         my ($host,  $port, $hg,$action) = split /s*;\s*/, $key;
+         my $proxy_sql_command ="INSERT INTO mysql_servers (hostgroup_id, hostname,port,gtid_port,status,weight,compression,max_connections,max_replication_lag,use_ssl,max_latency_ms,comment) ".
+            " VALUES($proxynode->{_hg_reader_id}" .
+            ",'$node->{_ip}'" .
+            ",$node->{_port} " .
+            ",$node->{_gtid_port} " .
+            ",'$node->{_proxy_status}' " .
+            ",$node->{_weight}" .
+            ",$node->{_compression}" .
+            ",$node->{_connections}" .
+            ",$node->{_max_replication_lag}" .
+            ",$node->{_use_ssl}" .
+            ",$node->{_max_latency}" .
+            ",'$node->{_comments}')" ;
+            
+         #my $proxy_sql_command =" UPDATE mysql_servers SET hostgroup_id=".($hg - 9000)." WHERE hostgroup_id=$hg AND hostname='$host' AND port='$port'";
+         $proxynode->{_dbh_proxy}->do($proxy_sql_command) or warn "Couldn't execute statement: $proxy_sql_command" .  $proxynode->{_dbh_proxy}->errstr;
+         my $error_code = $proxynode->{_dbh_proxy}->err();
+         $proxynode->{_dbh_proxy}->do("LOAD MYSQL SERVERS TO RUNTIME") or warn "Couldn't execute statement: " .  $proxynode->{_dbh_proxy}->errstr;
+         print Utils->print_log(2," Move node:" .$key
+             ." SQL:" .$proxy_sql_command
+             ."\n" ) ;			    
+    }
+   
+     sub insert_writer{
+         my ($proxynode, $key,$node) = @_;
+         my ($host,  $port, $hg,$action) = split /s*;\s*/, $key;
+         my $proxy_sql_command ="INSERT INTO mysql_servers (hostgroup_id, hostname,port,gtid_port,status,weight,compression,max_connections,max_replication_lag,use_ssl,max_latency_ms,comment) ".
+            " VALUES($proxynode->{_hg_writer_id}" .
+            ",'$node->{_ip}'" .
+            ",$node->{_port} " .
+            ",$node->{_gtid_port} " .
+            ",'$node->{_proxy_status}' " .
+            ",$node->{_weight}" .
+            ",$node->{_compression}" .
+            ",$node->{_connections}" .
+            ",$node->{_max_replication_lag}" .
+            ",$node->{_use_ssl}" .
+            ",$node->{_max_latency}" .
+            ",'$node->{_comments}')" ;
+            
+         #my $proxy_sql_command =" UPDATE mysql_servers SET hostgroup_id=".($hg - 9000)." WHERE hostgroup_id=$hg AND hostname='$host' AND port='$port'";
+         $proxynode->{_dbh_proxy}->do($proxy_sql_command) or warn "Couldn't execute statement: $proxy_sql_command" .  $proxynode->{_dbh_proxy}->errstr;
+         my $error_code = $proxynode->{_dbh_proxy}->err();
+         $proxynode->{_dbh_proxy}->do("LOAD MYSQL SERVERS TO RUNTIME") or warn "Couldn't execute statement: " .  $proxynode->{_dbh_proxy}->errstr;
+         print Utils->print_log(2," Move node:" .$key
+             ." SQL:" .$proxy_sql_command
+             ."\n" ) ;			    
+    }
+    sub initiate_failover{
+        my ($proxynode,$Galera_cluster)  = @_ ;
+        my ( $nodes ) = $Galera_cluster->{_nodes};
+        my ( $nodes_maint ) = $Galera_cluster->{_nodes_maint};
+        my $failover_node;
+        my $candidate_failover_node;
+        my $min_index = 100;
+        my $max_weight=0;
+        my $cand_min_index = 100;
+        my $cand_max_weight=0;
+
+        my $local_node;
+        my $hg_writer_id=0;
+        my $exclude_delete="";
+        
+        #Valid values are:
+        #    0 [default] do not make failover
+        #    1 make failover only if HG 8000 is specified in ProxySQL mysl_servers
+        #    2 use PXC_CLUSTER_VIEW to identify a server in the same segment
+        #    3 do whatever to keep service up also failover to another segment (use PXC_CLUSTER_VIEW) 
+        #           
+        foreach my $key (sort keys %{$nodes}){
+          if(defined $nodes->{$key} ){
+         
+             #only if node has HG that is not maintenance it can be evaluated to be put down in some way
+             #Look for the node with the lowest weight in the same segment
+             if($nodes->{$key}->{_hostgroups} < 9000
+                && $nodes->{$key}->{_proxy_status} eq "ONLINE"
+                && $nodes->{$key}->{_process_status} > 0
+                && $nodes->{$key}->{_wsrep_status} == 4
+                && $nodes->{$key}->{_wsrep_rejectqueries} eq "NONE"
+                && $nodes->{$key}->{_wsrep_donorrejectqueries} eq "OFF"
+                && $nodes->{$key}->{_pxc_maint_mode} eq "DISABLED"
+                && $nodes->{$key}->{_read_only} eq "OFF" 
+                ){
+              
+              #IN case failover option is 1 we need to have:
+              #The failover group defined in Proxysql (8xxx + id of the HG)
+              #Node must be of HG 8XXXX id
+              #And must be in the same segment of the writer 
+                 if(
+                    $proxynode->{_require_failover} == 1
+                   # && $nodes->{$key}->{_wsrep_segment} == $Galera_cluster->{_main_segment}
+                    && $Galera_cluster->{_has_failover_node} >0
+                    && $nodes->{$key}->{_hostgroups} == (8000 + $Galera_cluster->{_hg_writer_id})
+                    ){
+
+                       if($nodes->{$key}->{_weight} > $max_weight
+                          && $nodes->{$key}->{_wsrep_segment} == $Galera_cluster->{_main_segment}
+                          ){
+                          $max_weight= $nodes->{$key}->{_weight};
+                          #$min_index =  $nodes->{$key}->{_wsrep_local_index};
+                          $failover_node = $nodes->{$key};
+                       }
+                       elsif($nodes->{$key}->{_wsrep_segment} != $Galera_cluster->{_main_segment}
+                          && !defined $failover_node
+                          && $nodes->{$key}->{_weight} > $cand_max_weight){
+                            $cand_max_weight= $nodes->{$key}->{_weight};
+                           # $cand_min_index =  $nodes->{$key}->{_wsrep_local_index};
+                            $candidate_failover_node = $nodes->{$key};
+                       }
+
+                       #if($nodes->{$key}->{_weight} > $max_weight){
+                       #     $max_weight= $nodes->{$key}->{_weight};
+                       #     $failover_node = $nodes->{$key};
+                       #}
+                 }
+              #IN case failover option is 2 we need to have:
+              # must be in the same segment of the writer
+              #and be in the PXC_CLUSTER_VIEW
+
+                 elsif(
+                    $proxynode->{_require_failover} == 2
+                    && $nodes->{$key}->{_wsrep_segment} == $Galera_cluster->{_main_segment}                  
+                 ){
+                      if($nodes->{$key}->{_wsrep_local_index} < $min_index
+                         ){
+                           $min_index =  $nodes->{$key}->{_wsrep_local_index};
+                           $failover_node = $nodes->{$key};
+                      }
+                 }
+                 elsif($proxynode->{_require_failover} == 3){
+                       if($nodes->{$key}->{_wsrep_segment} == $Galera_cluster->{_main_segment}
+                          && $nodes->{$key}->{_wsrep_local_index} < $min_index){
+                          $min_index =  $nodes->{$key}->{_wsrep_local_index};
+                          $failover_node = $nodes->{$key};
+                       }
+                       elsif($nodes->{$key}->{_wsrep_segment} != $Galera_cluster->{_main_segment}
+                          && !defined $failover_node
+                          && $nodes->{$key}->{_wsrep_local_index} < $cand_min_index){
+                            $cand_min_index =  $nodes->{$key}->{_wsrep_local_index};
+                            $candidate_failover_node = $nodes->{$key};
+                       }
+                 }
+             }
+          }
+        }
+        if(defined $nodes_maint ){
+           my $exclude_id = "";
+           my $exclude_port = "";
+           foreach my $key (sort keys %{$nodes_maint}){
+             if(defined $nodes_maint->{$key}){ 
+                if(length($exclude_id) > 1){
+                   $exclude_id = $exclude_id . ",";
+                   $exclude_port = $exclude_port . ",";
+                }
+                   
+                $exclude_id = $exclude_id ."'". $nodes_maint->{$key}->{_ip} ."'";
+                $exclude_port = $exclude_port.$nodes_maint->{$key}->{_port} ;
+             }
+           }
+           if(length($exclude_id) > 1){
+              $exclude_delete = $exclude_delete . " AND (hostname not in (".$exclude_id.") AND port not in (".$exclude_port."))" ;
+           }
+        }
+        
+        #if a node was found, try to do the failover removing the READ_ONLY
+        if(defined $candidate_failover_node && $failover_node){
+            return $failover_node->promote_writer($proxynode,$Galera_cluster,$exclude_delete);
+        }
+        elsif(defined $candidate_failover_node && !defined $failover_node){
+            return $candidate_failover_node->promote_writer($proxynode,$Galera_cluster,$exclude_delete);
+        }
+        else{
+            if(!defined $failover_node){
+                SWITCH: {
+                  if ($proxynode->{_require_failover} == 1) { print Utils->print_log(1,"!!!! No node for failover found , try to use active_failover=2 OR add a valid node to the 8000 HG pool \n" ) ; last SWITCH; }
+                  if ($proxynode->{_require_failover} == 2) { print Utils->print_log(1,"!!!! No node for failover found , try to use active_failover=3 But that may move production to the other segment.\n" ); last SWITCH; }
+                  if ($proxynode->{_require_failover} == 3) { print Utils->print_log(1,"!!!! No node for failover found also in the other segments, I cannot continue you need to act manually \n" ); last SWITCH; }
+                }
+            }
+         
+            if(defined $failover_node){
+               return $failover_node->promote_writer($proxynode,$Galera_cluster,$exclude_delete);
+            }
+        }
+       
+        
+        return 0;
+    }
+
+}
+
+{
+    package ProxySqlHG;
+    sub new {
+        my $class = shift;
+        
+        my $self = {
+            _id  => undef, # 
+            _type  => undef, # available types: w writer; r reader ; mw maintance writer; mr maintenance reader
+            _size => 0,
+        };
+        bless $self, $class;
+        return $self;
+    }
+    
+    sub id {
+        my ( $self, $id ) = @_;
+        $self->{_id} = $id if defined($id);
+        return $self->{_id};
+    }
+
+    sub type {
+        my ( $self, $type ) = @_;
+        $self->{_type} = $type if defined($type);
+        return $self->{_type};
+    }
+    sub size {
+        my ( $self, $size ) = @_;
+        $self->{_size} = $size if defined($size);
+        return $self->{_size};
+    }
+
+}
+
+{
+    package Utils;
+    use Time::HiRes qw(gettimeofday);
+    #============================================================================
+    ## get_connection -- return a valid database connection handle (or die)
+    ## $dsn  -- a perl DSN, e.g. "DBI:mysql:host=ltsdbwm1;port=3311"
+    ## $user -- a valid username, e.g. "check"
+    ## $pass -- a matching password, e.g. "g33k!"
+    
+    sub get_connection($$$$) {
+      my $dsn  = shift;
+      my $user = shift;
+      my $pass = shift;
+      my $SPACER = shift;
+      my $dbh = DBI->connect($dsn, $user, $pass ,  {
+        PrintError => 0,
+        PrintWarn  => 0,
+        RaiseError => 0});
+      
+      if (!defined($dbh)) {
+        #die
+        print Utils->print_log(1, "Cannot connect to $dsn as $user\n");
+        # Should not die and instead return undef so we can handle this shit 
+        #die();
+        return undef;
+      }
+      
+      return $dbh;
+    }
+    
+    
+    ######################################################################
+    ## collection functions -- fetch status data from db
+    ## get_status -- return a hash ref to SHOW GLOBAL STATUS output
+    ## $dbh -- a non-null database handle, as returned from get_connection()
+    ##
+    
+    
+    sub get_status($$) {
+      my $dbh = shift;
+      my $debug = shift;
+      my %v;
+      my $cmd = "show /*!50000 global */ status";
+    
+      my $sth = $dbh->prepare($cmd);
+      $sth->execute() or warn "Couldn't execute statement: $cmd" .  $dbh->errstr ." \n";
+      while (my $ref = $sth->fetchrow_hashref()) {
+        my $n = $ref->{'Variable_name'};
+        $v{"\L$n\E"} = $ref->{'Value'};
+        if ($debug>0){print "MySQL status = ".$n."\n";}
+      }
+    
+      return \%v;
+    }
+    ######################################################################
+    ## collection functions -- fetch status data from db
+    ## get_status -- return a hash ref to SHOW GLOBAL STATUS output
+    ## $dbh -- a non-null database handle, as returned from get_connection()
+    ##
+    
+    sub get_status_by_name($$) {
+      my $dbh = shift;
+      my $debug = shift;
+      my $name  = shift ; 
+      my %v;
+      my $cmd = "show /*!50000 global */ status like '$name'";
+    
+      my $sth = $dbh->prepare($cmd);
+      $sth->execute();
+      while (my $ref = $sth->fetchrow_hashref()) {
+        my $n = $ref->{'Variable_name'};
+        $v{"\L$n\E"} = $ref->{'Value'};
+        if ($debug>0){print "MySQL status = ".$n."\n";}
+      }
+    
+      return \%v;
+    }
+    ##
+    ## get_variables -- return a hash ref to SHOW GLOBAL VARIABLES output
+    ##
+    ## $dbh -- a non-null database handle, as returned from get_connection()
+    ##
+    sub get_variables($$) {
+      my $dbh = shift;
+      my $debug = shift;
+      my %v;
+      my $cmd = "select * from performance_schema.global_variables";
+      $dbh->{LongReadLen} = 0;
+      $dbh->{LongTruncOk} = 0;
+      
+      my $sth = $dbh->prepare($cmd);
+      $sth->execute() or warn "Couldn't execute statement: $cmd" .  $dbh->errstr ." \n";
+      while (my $ref = $sth->fetchrow_hashref()) {
+        my $n = $ref->{'VARIABLE_NAME'};
+        $v{"\L$n\E"} = $ref->{'VARIABLE_VALUE'};
+        # print STDERR "$n  :  ".$v{$n}. " ZZZZZZZZZZZZZZZZZZ ". $ref->{'Value'} ."\n";
+      }
+      
+     
+      return \%v;
+    }
+    ##
+    ## get_variables -- return a hash ref to SHOW GLOBAL VARIABLES output
+    ##
+    ## $dbh -- a non-null database handle, as returned from get_connection()
+    ##
+    sub get_variablesByName($$) {
+        my $dbh = shift;
+        my $variableName = shift;
+        #my $debug = shift;
+        my %v;
+        my $cmd = "show variables like '$variableName'";
+      
+        my $sth = $dbh->prepare($cmd);
+        $sth->execute() or warn "Couldn't execute statement: $cmd" .  $dbh->errstr ." \n";
+        while (my $ref = $sth->fetchrow_hashref()) {
+          my $n = $ref->{'Variable_name'};
+          $v{"\L$n\E"} = $ref->{'Value'};
+        }
+        return \%v;
+    }
+    ##
+    ## get_variables -- return a hash ref to SHOW GLOBAL VARIABLES output
+    ##
+    ## $dbh -- a non-null database handle, as returned from get_connection()
+    ##
+    sub get_pxc_clusterview($$) {
+        my $dbh = shift;
+        my $variableName = shift;
+        #my $debug = shift;
+       
+        my %v;
+        my $cmd = "select * from performance_schema.pxc_cluster_view where UUID = '$variableName'";
+      
+        my $sth = $dbh->prepare($cmd);
+        $sth->execute() or warn "Couldn't execute statement: $cmd" .  $dbh->errstr ." \n";
+      my $ref;  
+      while ( $ref = $sth->fetchrow_hashref()) {
+          foreach my $name ('HOST_NAME', 'UUID','STATUS','LOCAL_INDEX','SEGMENT'){  
+               my $n = lc $name;
+              $v{$n} = $ref->{$name};
+          }
+        }        
+        return \%v;
+    }    
+    #Print time from invocation with milliseconds
+    sub get_current_time{
+        use POSIX qw(strftime);
+        my $t = gettimeofday();
+        my $date = strftime "%Y/%m/%d %H:%M:%S", localtime $t;
+        $date .= sprintf ".%03d", ($t-int($t))*1000; # without rounding
+        
+        return $date;
+    }
+
+    #prit all environmnt variables    
+    sub debugEnv{
+        my $key = keys %ENV;
+        foreach $key (sort(keys %ENV)) {
+           print $key, '=', $ENV{$key}, "\n";
+        }
+    
+    }
+    
+    
+    #Print a log entry
+    sub print_log($$){
+        my $log_level = $_[1];
+        my $text = $_[2];
+        my $log_text = "[ - ] ";
+	
+        SWITCH: {
+              if ($log_level == 1) { $log_text= "[ERROR] "; last SWITCH; }
+              if ($log_level == 2) { $log_text= "[WARN] "; last SWITCH; }
+              if ($log_level == 3) { $log_text= "[INFO] "; last SWITCH; }
+              if ($log_level == 4) { $log_text= "[DEBUG] "; last SWITCH; }
+       }
+       return Utils::get_current_time.":".$log_text.$text;
+	
+    }
+    
+    
+    #trim a string
+    sub  trim {
+        my $s = shift;
+        $s =~ s/^\s+|\s+$//g;
+        return $s
+    };
+
+
+}
+
+
+# ############################################################################
+# Documentation
+# #################
+=pod
+
+=head1 NAME
+galera_check.pl
+
+=head1 OPTIONS
+
+=over
+
+galera_check.pl -u=admin -p=admin -h=192.168.1.50 -H=500:W,501:R -P=3310 --main_segment=1 --debug=0  --log <full_path_to_file> --help
+sample [options] [file ...]
+ Options:
+   -u|user            user to connect to the proxy
+   -p|password        Password for the proxy
+   -h|host            Proxy host
+   -H                 Hostgroups with role definition. List comma separated.
+		      Definition R = reader; W = writer [500:W,501:R]
+   --main_segment     If segments are in use which one is the leading at the moment
+   --retry_up         The number of loop/test the check has to do before moving a node up (default 0)
+   --retry_down       The number of loop/test the check has to do before moving a node Down (default 0)
+   --log	      Full path to the log file ie (/var/log/proxysql/galera_check_) the check will add
+		      the identifier for the specific HG.
+   --active_failover  A value from 0 to 3, indicating what level/kind of fail-over the script must perform.
+                       active_failover
+                       Valid values are:
+                          0 [default] do not make failover
+                          1 make failover only if HG 8000 is specified in ProxySQL mysl_servers
+                          2 use PXC_CLUSTER_VIEW to identify a server in the same segment
+                          3 do whatever to keep service up also failover to another segment (use PXC_CLUSTER_VIEW)
+   --single_writer    Active by default [single_writer = 1 ] if disable will allow to have multiple writers       
+   
+   
+   Performance parameters 
+   --check_timeout    This parameter set in ms then time the script can alow a thread connecting to a MySQL node to wait, before forcing a returnn.
+                      In short if a node will take longer then check_timeout its entry will be not filled and it will eventually ignored in the evaluation.
+                      Setting the debug option  =1 and look for [WARN] Check timeout Node ip : Information will tell you how much your nodes are exceeding the allowed limit.
+                      You can use the difference to correctly set the check_timeout 
+                      Default is 800 ms
+   
+   --help              help message
+   --debug             When active the log will have a lot of information about the execution. Parse it for ERRORS if you have problems
+   --print_execution   Active by default, it will print the execution time the check is taking in the log. This can be used to tune properly the scheduler time, and also the --check_timeout
+   
+   --development      When set to 1 you can run the script in a loop from bash directly and test what is going to happen
+   --development_time Time in seconds that the loop wait to execute when in development mode (default 2 seconds)
+   
+   SSL support
+   Now the script identify if the node in the ProxySQL table mysql_servers has use_ssl = 1 and will set SSL to be used for that specific entry.
+   This means that SSL connection is by ProxySQL mysql_server entry NOT by IP:port combination.
+   
+   --ssl_certs_path This parameter allow you to specify a DIRECTORY to use to assign specific certificates.
+                    At the moment is NOT possible to change the files names and ALL these 3 files must be there and named as follow:
+                     -  client-key.pem
+                     -  client-cert.pem
+                     -  ca.pem
+                     Script will exit with an error if ssl_certs_pathis declared but not filled properly
+                     OR if the user running the script doesn't have acces.
+   !!NOTE!! SSL connection requires more time to be established. This script is a check that needs to run very fast and constantly.
+            force it to use ssl WILL impact in the performance of the check. Tune properly the check_timeout parameter.
+
+=back    
+   
+=head1 DESCRIPTION
+
+Galera check is a script to manage integration between ProxySQL and Galera (from Codership).
+Galera and its implementations like Percona Cluster (PCX), use the data-centric concept, as such the status of a node is relvant in relation to a cluster.
+
+In ProxySQL is possible to represent a cluster and its segments using HostGroups.
+Galera check is design to manage a X number of nodes that belong to a given Hostgroup (HG). 
+In Galera_check it is also important to qualify the HG in case of use of Replication HG.
+
+galera_check works by HG and as such it will perform isolated actions/checks by HG. 
+It is not possible to have more than one check running on the same HG. The check will create a lock file {proxysql_galera_check_${hg}.pid} that will be used by the check to prevent duplicates.
+
+Galera_check will connect to the ProxySQL node and retrieve all the information regarding the Nodes/proxysql configuration. 
+It will then check in parallel each node and will retrieve the status and configuration.
+
+At the moment galera_check analyze and manage the following:
+
+Node states: 
+  read_only 
+  wsrep_status 
+  wsrep_rejectqueries 
+  wsrep_donorrejectqueries 
+  wsrep_connected 
+  wsrep_desinccount 
+  wsrep_ready 
+  wsrep_provider 
+  wsrep_segment 
+  Number of nodes in by segment
+  Retry loop
+  
+- Number of nodes in by segment
+If a node is the only one in a segment, the check will behave accordingly. 
+IE if a node is the only one in the MAIN segment, it will not put the node in OFFLINE_SOFT when the node become donor to prevent the cluster to become unavailable for the applications. 
+As mention is possible to declare a segment as MAIN, quite useful when managing prod and DR site.
+
+-The check can be configure to perform retry in a X interval. 
+Where X is the time define in the ProxySQL scheduler. 
+As such if the check is set to have 2 retry for UP and 4 for down, it will loop that number before doing anything. Given that Galera does some action behind the hood.
+This feature is very useful in some not well known cases where Galera bhave weird.
+IE whenever a node is set to READ_ONLY=1, galera desync and resync the node. 
+A check not taking this into account will cause a node to be set OFFLINE and back for no reason.
+
+Another important differentiation for this check is that it use special HGs for maintenance, all in range of 9000. 
+So if a node belong to HG 10 and the check needs to put it in maintenance mode, the node will be moved to HG 9010. 
+Once all is normal again, the Node will be put back on his original HG.
+
+This check does NOT modify any state of the Nodes. 
+Meaning It will NOT modify any variables or settings in the original node. It will ONLY change states in ProxySQL. 
+    
+The check is still a prototype and is not suppose to go to production (yet).
+
+
+=over
+
+=item 1
+
+Note that galera_check is also Segment aware, as such the checks on the presence of Writer /reader is done by segment, respecting the MainSegment as primary.
+
+=back
+
+=head1 Configure in ProxySQL
+
+
+INSERT  INTO scheduler (id,active,interval_ms,filename,arg1) values (10,0,2000,"/var/lib/proxysql/galera_check.pl","-u=remoteUser -p=remotePW -h=192.168.1.50 -H=500:W,501:R -P=6032 --retry_down=2 --retry_up=1 --main_segment=1 --debug=0 --active_failover=1 --single_writer=1 --log=/var/lib/proxysql/galeraLog");
+LOAD SCHEDULER TO RUNTIME;SAVE SCHEDULER TO DISK;
+
+To activate it
+update scheduler set active=1 where id=10;
+LOAD SCHEDULER TO RUNTIME;SAVE SCHEDULER TO DISK;
+
+To update the parameters you must pass all of them not only the ones you want to change(IE enabling debug)
+update scheduler set arg1="-u=remoteUser -p=remotePW -h=192.168.1.50 -H=500:W,501:R -P=6032 --retry_down=2 --retry_up=1 --main_segment=1 --debug=1 --active_failover=1 --single_writer=1 --log=/var/lib/proxysql/galeraLog" where id =10;  
+LOAD SCHEDULER TO RUNTIME;SAVE SCHEDULER TO DISK;
+
+
+delete from scheduler where id=10;
+LOAD SCHEDULER TO RUNTIME;SAVE SCHEDULER TO DISK;
+
+
+
+=head1 Rules:
+
+=over
+
+=item 1
+
+Set to offline_soft :
+    
+    any non 4 or 2 state, read only =ON
+    donor node reject queries - 0 size of cluster > 2 of nodes in the same segments more then one writer, node is NOT read_only.
+    
+    Changes to pxc_maint_mode to anything else DISABLED
+
+=item 2
+
+change HG t maintenance HG:
+    
+    Node/cluster in non primary
+    wsrep_reject_queries different from NONE
+    Donor, node reject queries =1 size of cluster 
+
+=item 3
+
+Node comes back from offline_soft when (all of them):
+
+     1) Node state is 4
+     3) wsrep_reject_queries = none
+     4) Primary state
+
+
+=item 4
+
+ Node comes back from maintenance HG when (all of them):
+
+     1) node state is 4
+     3) wsrep_reject_queries = none
+     4) Primary state
+
+
+=item 5
+
+ active_failover
+      Valid values are:
+          0 [default] do not make failover
+          1 make failover only if HG 8000 is specified in ProxySQL mysl_servers
+          2 use PXC_CLUSTER_VIEW to identify a server in the same segment
+          3 do whatever to keep service up also failover to another segment (use PXC_CLUSTER_VIEW) 
+
+=item 6
+ PXC_MAIN_MODE is fully supported.
+ Any node in a state different from pxc_maint_mode=disabled will be set in OFFLINE_SOFT for all the HostGroup.
+ 
+=item 7
+ internally shunning node.
+ While I am trying to rely as much as possible on ProxySQL, given few inefficiencies there are cases when I have to set a node to SHUNNED because ProxySQL doesn't recognize it correctly.  
+          
+=back
+=cut	
+

dev/provisioning/ansible/roles/proxysql/files/proxysql_galera_checker

@@ -0,0 +1,2589 @@
+#!/bin/bash
+## inspired by Percona clustercheck.sh
+# https://github.com/percona/proxysql-admin-tool/blob/master/proxysql_galera_checker
+
+#-------------------------------------------------------------------------------
+#
+# Step 1 : Bash internal configuration
+#
+
+set -o nounset    # no undefined variables
+
+#-------------------------------------------------------------------------------
+#
+# Step 2 : Global variables
+#
+
+#
+# Script parameters/constants
+#
+declare  -i DEBUG=0
+readonly    PROXYSQL_ADMIN_VERSION="1.4.14"
+
+#Timeout exists for instances where mysqld may be hung
+declare  -i TIMEOUT=10
+
+declare     RED=""
+declare     NRED=""
+
+
+#
+# Global variables used by the script
+#
+declare     ERR_FILE="/dev/stderr"
+declare     NODE_MONITOR_LOG_FILE=""
+declare     CONFIG_FILE="/etc/proxysql-admin.cnf"
+declare     HOST_PRIORITY_FILE=""
+declare     CHECKER_PIDFILE=""
+
+# Set to send output here when DEBUG is set
+declare     DEBUG_ERR_FILE="/dev/null"
+
+
+declare -i  HOSTGROUP_WRITER_ID=1
+declare -i  HOSTGROUP_READER_ID=-1
+declare -i  HOSTGROUP_SLAVEREADER_ID=-1
+declare -i  NUMBER_WRITERS=0
+
+declare     WRITER_IS_READER="ondemand"
+
+declare     SLAVE_IS_WRITER="yes"
+
+declare     P_MODE=""
+declare     P_PRIORITY=""
+
+declare     MYSQL_USERNAME
+declare     MYSQL_PASSWORD
+declare     PROXYSQL_USERNAME
+declare     PROXYSQL_PASSWORD
+declare     PROXYSQL_HOSTNAME
+declare     PROXYSQL_PORT
+
+declare     PROXYSQL_DATADIR='/var/lib/proxysql'
+
+# Set to 1 if slave readers are being used
+declare -i  HAVE_SLAVEREADERS=0
+declare -i  HAVE_SLAVEWRITERS=0
+
+# How far behind can a slave be before its put into OFFLINE_SOFT state
+declare     SLAVE_SECONDS_BEHIND=3600
+
+# Some extra text that will be logged
+# (useful for debugging)
+declare     LOG_TEXT=""
+
+# Default value for max_connections in mysql_servers
+declare     MAX_CONNECTIONS=1000
+
+
+#-------------------------------------------------------------------------------
+#
+# Step 3 : Helper functions
+#
+
+function log() {
+  local lineno=$1
+  shift
+
+  if [[ -n $ERR_FILE ]]; then
+    if [[ -n $lineno && $DEBUG -ne 0 ]]; then
+      echo -e "[$(date +%Y-%m-%d\ %H:%M:%S)] $$ (line $lineno) $*" >> $ERR_FILE
+    else
+      echo -e "[$(date +%Y-%m-%d\ %H:%M:%S)] $$ $*" >> $ERR_FILE
+    fi
+  fi
+}
+
+# Checks the return value of the most recent command
+#
+# Globals:
+#   None
+#
+# Arguments:
+#   1: the lineno where the error occurred
+#   2: the error code of the most recent command
+#   3: the error message if the error code is non-zero
+#
+function check_cmd() {
+  local lineno=$1
+  local retcode=$2
+  local errmsg=$3
+  shift 3
+
+  if [[ ${retcode} -ne 0 ]]; then
+    error "$lineno" $errmsg
+    if [[ -n "$*" ]]; then
+      log "$lineno" $*
+    fi
+  fi
+  return $retcode
+}
+
+# Checks the return value of the most recent command
+# Exits the program if the command fails (non-zero return codes)
+#
+# This should not be used with commands that can fail for legitimate
+# reasons.
+#
+# Globals:
+#   None
+#
+# Arguments:
+#   1: the lineno where the error occurred
+#   2: the error code of the most recent command
+#   3: the error message if the error code is non-zero
+#
+function check_cmd_and_exit() {
+  check_cmd "$@"
+  local retcode=$?
+  if [[ $retcode -ne 0 ]]; then
+    exit 1
+  fi
+  return $retcode
+}
+
+function log_if_success() {
+  local lineno=$1
+  local rc=$2
+  shift 2
+
+  if [[ $rc -eq 0 ]]; then
+    log "$lineno" "$*"
+  fi
+  return $rc
+}
+
+function error() {
+  local lineno=$1
+  shift
+
+  log "$lineno" "proxysql_galera_checker : Error ($lineno): $*"
+}
+
+function warning() {
+  local lineno=$1
+  shift
+
+  log "$lineno" "Warning: $*"
+}
+
+function debug() {
+  if [[ $DEBUG -eq 0 ]]; then
+    return
+  fi
+
+  local lineno=$1
+  shift
+
+  log "$lineno" "${RED}debug: $*${NRED}"
+}
+
+function usage() {
+  local path=$0
+  cat << EOF
+Usage: ${path##*/} --write-hg=10 --read-hg=11 --config-file=/etc/proxysql-admin.cnf --log=/var/lib/proxysql/pxc_test_proxysql_galera_check.log
+
+Options:
+  -w, --write-hg=<NUMBER>             Specify ProxySQL write hostgroup.
+  -r, --read-hg=<NUMBER>              Specify ProxySQL read hostgroup.
+  -c, --config-file=PATH              Specify ProxySQL-admin configuration file.
+  -l, --log=PATH                      Specify proxysql_galera_checker log file.
+  --log-text=TEXT                     This is text that will be written to the log file
+                                      whenever this script is run (useful for debugging).
+  --node-monitor-log=PATH             Specify proxysql_node_monitor log file.
+  -n, --writer-count=<NUMBER>         Maximum number of write hostgroup_id nodes
+                                      that can be marked ONLINE
+                                      When 0 (default), all nodes can be marked ONLINE
+  -p, --priority=<HOST_LIST>          Can accept comma delimited list of write nodes priority
+  -m, --mode=[loadbal|singlewrite]    ProxySQL read/write configuration mode,
+                                      currently supporting: 'loadbal' and 'singlewrite'
+  --writer-is-reader=<value>          Defines if the writer node also accepts writes.
+                                      Possible values are 'always', 'never', and 'ondemand'.
+                                      'ondemand' means that the writer node only accepts reads
+                                      if there are no other readers.
+                                      (default: 'never')
+  --use-slave-as-writer=<yes/no>      If this is 'yes' then slave nodes may
+                                      be added to the write hostgroup if all other
+                                      cluster nodes are down.
+                                      (default: 'yes')
+  --max-connections=<NUMBER>          Value for max_connections in the mysql_servers table.
+                                      This is the maximum number of connections that
+                                      ProxySQL will open to the backend servers.
+                                      (default: 1000)
+  --debug                             Enables additional debug logging.
+  -h, --help                          Display script usage information
+  -v, --version                       Print version info
+
+Notes about the mysql_servers in ProxySQL:
+
+- NODE STATUS   * Nodes that are in status OFFLINE_HARD will not be checked
+                  nor will their status be changed
+                * SHUNNED nodes are not to be used with Galera based systems,
+                  they will be checked and their status will be changed
+                  to either ONLINE or OFFLINE_SOFT.
+
+
+When no nodes were found to be in wsrep_local_state=4 (SYNCED) for either
+read or write nodes, then the script will try 5 times for each node to try
+to find nodes wsrep_local_state=4 (SYNCED) or wsrep_local_state=2 (DONOR/DESYNC)
+EOF
+}
+
+
+# Check the permissions for a file or directory
+#
+# Globals:
+#   None
+#
+# Arguments:
+#   1: the bash test to be applied to the file
+#   2: the lineno where this call is invoked (used for errors)
+#   3: the path to the file
+#   4: (optional) description of the path (mostly used for existence checks)
+#
+# Exits the script if the permissions test fails.
+#
+function check_permission() {
+  local permission=$1
+  local lineno=$2
+  local path_to_check=$3
+  local description=""
+  if [[ $# -gt 3 ]]; then
+    description="$4"
+  fi
+
+  if [ ! $permission "$path_to_check" ] ; then
+    if [[ $permission == "-r" ]]; then
+      error $lineno "You do not have READ permission for: $path_to_check"
+    elif [[ $permission == "-w" ]]; then
+      error $lineno "You do not have WRITE permission for: $path_to_check"
+    elif [[ $permission == "-x" ]]; then
+      error $lineno "You do not have EXECUTE permission for: $path_to_check"
+    elif [[ $permission == "-e" ]]; then
+      if [[ -n $description ]]; then
+        error $lineno "Could not find the $description: $path_to_check"
+      else
+        error $lineno "Could not find: $path_to_check"
+      fi
+    elif [[ $permission == "-d" ]]; then
+      if [[ -n $description ]]; then
+        error $lineno "Could not find the $description: $path_to_check"
+      else
+        error $lineno "Could not find the directory: $path_to_check"
+      fi
+    elif [[ $permission == "-f" ]]; then
+      if [[ -n $description ]]; then
+        error $lineno "Could not find the $description: $path_to_check"
+      else
+        error $lineno "Could not find the file: $path_to_check"
+      fi
+    else
+      error $lineno "You do not have the correct permissions for: $path_to_check"
+    fi
+    exit 1
+  fi
+}
+
+# Executes a SQL query with the (fully) specified server
+#
+# Globals:
+#   None
+#
+# Arguments:
+#   1: lineno
+#   2: the name of the user
+#   3: the user's password
+#   4: the hostname of the server
+#   5: the port used to connect to the server
+#   6: the query to be run
+#   7: arguments to the mysql client
+#   8: timeout in secs
+#   9: additional options, space separated
+#      Available options:
+#       "hide_output"
+#         This will not show the output of the query when DEBUG is set.
+#         Used to stop the display of sensitve information (such as passwords)
+#         from being displayed when debugging.
+#
+function exec_sql() {
+  local lineno=$1
+  local user=$2
+  local password=$3
+  local hostname=$4
+  local port=$5
+  local query=$6
+  local args=$7
+  local timeout_secs=$8
+  local more_options=$9
+  local retvalue
+  local retoutput
+
+  debug "$lineno" "exec_sql : $user@$hostname:$port ($args) ==> $query"
+
+  retoutput=$(printf "[client]\nuser=${user}\npassword=\"${password}\"\nhost=${hostname}\nport=${port}"  \
+      | timeout ${timeout_secs} mysql --defaults-file=/dev/stdin --protocol=tcp \
+              ${args} -e "$query")
+  retvalue=$?
+
+  if [[ $DEBUG -eq 1 ]]; then
+    local number_of_newlines=0
+    local dbgoutput=$retoutput
+
+    if [[ " $more_options " =~ [[:space:]]hide_output[[:space:]] ]]; then
+      dbgoutput="**** data hidden ****"
+    fi
+
+    if [[ -n $dbgoutput ]]; then
+      number_of_newlines=$(printf "%s" "${dbgoutput}" | wc -l)
+    fi
+
+    if [[  $retvalue -ne 0 ]]; then
+      debug "" "--> query failed $retvalue"
+    elif [[ -z $dbgoutput ]]; then
+      debug "" "--> query returned $retvalue : <query returned no data>"
+    elif [[ ${number_of_newlines} -eq 0 ]]; then
+      debug "" "--> query returned $retvalue : ${dbgoutput}"
+    else
+      debug "" "--> query returned $retvalue : <data follows>"
+      printf "${dbgoutput//%/%%}\n" | while IFS= read -r line; do
+        debug "" "----> $line"
+      done
+    fi
+  fi
+  printf "${retoutput//%/%%}"
+  return $retvalue
+}
+
+
+# Executes a SQL query on proxysql (with a timeout of $TIMEOUT seconds)
+#
+# Globals:
+#   PROXYSQL_USERNAME
+#   PROXYSQL_PASSWORD
+#   PROXYSQL_HOSTNAME
+#   PROXYSQL_PORT
+#
+# Arguments:
+#   1: lineno (used for debugging/output, may be blank)
+#   2: Additional arguments to the mysql client for the query
+#   3: The SQL query
+#   4: (optional) see the additional options for exec_sql
+#
+function proxysql_exec() {
+  local lineno=$1
+  local args=$2
+  local query="$3"
+  local more_options=""
+  local retoutput
+
+  if [[ $# -ge 4 ]]; then
+    more_options=$4
+  fi
+
+  exec_sql "$lineno" "$PROXYSQL_USERNAME" "$PROXYSQL_PASSWORD" \
+           "$PROXYSQL_HOSTNAME" "$PROXYSQL_PORT" \
+           "$query" "$args" "$TIMEOUT" "$more_options"
+  retoutput=$?
+  if [[ $retoutput -eq 124 ]]; then
+    error $lineno "TIMEOUT: SQL query ($PROXYSQL_HOSTNAME:$PROXYSQL_PORT) : $query"
+  fi
+  return $retoutput
+}
+
+# Executes a SQL query on mysql (with a timeout of $TIMEOUT secs)
+#
+# Globals:
+#   MYSQL_USERNAME
+#   MYSQL_PASSWORD
+#
+# Arguments:
+#   1: lineno (used for debugging/output, may be blank)
+#   2: the hostname of the server
+#   3: the port used to connect to the server
+#   4: arguments to the mysql client
+#   5: the query to be run
+#   6: (optional) more options see exec_sql
+#
+function mysql_exec() {
+  local lineno=$1
+  local hostname=$2
+  local port=$3
+  local args=$4
+  local query=$5
+  local more_options=""
+  local retoutput
+
+  if [[ $# -ge 6 ]]; then
+    more_options=$6
+  fi
+
+  exec_sql "$lineno" "$MYSQL_USERNAME" "$MYSQL_PASSWORD" \
+           "$hostname" "$port" \
+           "$query" "$args" "$TIMEOUT" "$more_options"
+  retoutput=$?
+  if [[ $retoutput -eq 124 ]]; then
+    error $lineno "TIMEOUT: SQL query ($hostname:$$port) : $query"
+  fi
+  return $retoutput
+}
+
+
+# Separates the IP address from the port in a network address
+# Works for IPv4 and IPv6
+#
+# Globals:
+#   None
+#
+# Params:
+#   1. The network address to be parsed
+#
+# Outputs:
+#   A string with a space separating the IP address from the port
+#
+function separate_ip_port_from_address()
+{
+  #
+  # Break address string into host:port/path parts
+  #
+  local address=$1
+
+  # Has to have at least one ':' to separate the port from the ip address
+  if [[ $address =~ : ]]; then
+    ip_addr=${address%:*}
+    port=${address##*:}
+  else
+    ip_addr=$address
+    port=""
+  fi
+
+  # Remove any braces that surround the ip address portion
+  ip_addr=${ip_addr#\[}
+  ip_addr=${ip_addr%\]}
+
+  echo "${ip_addr} ${port}"
+}
+
+# Combines the IP address and port into a network address
+# Works for IPv4 and IPv6
+# (If the IP address is IPv6, the IP portion will have brackets)
+#
+# Globals:
+#   None
+#
+# Params:
+#   1: The IP address portion
+#   2: The port
+#
+# Outputs:
+#   A string containing the full network address
+#
+function combine_ip_port_into_address()
+{
+  local ip_addr=$1
+  local port=$2
+  local addr
+
+  if [[ ! $ip_addr =~ \[.*\] && $ip_addr =~ .*:.* ]] ; then
+    # If there are no brackets and it does have a ':', then add the brackets
+    # because this is an unbracketed IPv6 address
+    addr="[${ip_addr}]:${port}"
+  else
+    addr="${ip_addr}:${port}"
+  fi
+  echo $addr
+}
+
+
+# upgrade scheduler from old layout to new layout
+#
+# Globals:
+#   PROXYSQL_DATADIR
+#   TIMEOUT
+#   HOST_PRIORITY_FILE
+#
+# Arguments:
+#   None
+function upgrade_scheduler(){
+  log $LINENO "**** Scheduler upgrade started ****"
+  if [[ -f /etc/proxysql-admin.cnf ]]; then
+    source /etc/proxysql-admin.cnf
+  else
+    error $LINENO "Assert! proxysql-admin configuration file : /etc/proxysql-admin.cnf does not exist, Terminating!"
+    exit 1
+  fi
+
+  # For this function, use a shorter timeout than normal
+  TIMEOUT=2
+
+  local scheduler_rows
+  local -i rows_found=0
+  local -i rows_modified=0
+
+  scheduler_rows=$(proxysql_exec $LINENO "-Ns" "SELECT * FROM scheduler")
+  check_cmd_and_exit $LINENO $? "Could not retreive rows from scheduler (query failed). Exiting"
+
+  while read i; do
+    if [[ -z $i ]]; then continue; fi
+
+    rows_found+=1
+
+    # Extract fields from the line
+    local id=$(echo "$i" | awk '{print $1}')
+    local s_write_hg=$(echo "$i" | awk '{print $5}')
+    local s_read_hg=$(echo "$i" | awk '{print $6}')
+    local s_number_of_writes=$(echo "$i" | awk '{print $7}')
+    local s_log=$(echo "$i" | awk '{print $9}')
+    local s_cluster_name=$(echo "$i" | awk '{print $10}')
+    local s_mode=""
+
+    log $LINENO "Modifying the scheduler for write hostgroup: $s_write_hg"
+
+    # Get the mode for this cluster
+    local proxysql_mode_file
+    if [[ -z $s_cluster_name ]]; then
+      proxysql_mode_file="${PROXYSQL_DATADIR}/mode"
+    else
+      proxysql_mode_file="${PROXYSQL_DATADIR}/${s_cluster_name}_mode"
+    fi
+
+    if [[ -f ${proxysql_mode_file} && -r ${proxysql_mode_file} ]] ; then
+      s_mode=$(cat ${proxysql_mode_file})
+    else
+      log $LINENO ".. Cannot find the ${proxysql_mode_file} file"
+      if [[ $s_read_hg == "-1" ]]; then
+        log $LINENO ".. Assuming mode='loadbal'"
+        s_mode="loadbal"
+      else
+        log $LINENO ".. Assuming mode='singlewrite'"
+        s_mode="singlewrite"
+      fi
+    fi
+
+    # TODO: kennt
+    # This will fail in the multi-cluster case, but may be a non-issue
+    # since those nodes should appear in one cluster (may cause extra work
+    # for the othre clusters though).
+
+    # Get the host priority file
+    local s_host_priority=''
+    if [[ -n $HOST_PRIORITY_FILE && -f $HOST_PRIORITY_FILE ]]; then
+      debug $LINENO "Found a host priority file: $HOST_PRIORITY_FILE"
+      local p_priority_hosts=""
+
+      # Get the list of hosts from the host_priority file ignoring blanks
+      # and any lines that start with '#'
+      p_priority_hosts=$(cat $HOST_PRIORITY_FILE | grep '^[^#]' | sed ':a;N;$!ba;s/\n/,/g')
+      if [[ ! -z $p_priority_hosts ]] ; then
+        s_host_priority="--priority=$p_priority_hosts"
+      fi
+    fi
+
+    # Make the changes
+    if [[ ! -z $s_write_hg ]] && [[ ! -z $s_read_hg ]] && [[ ! -z $s_number_of_writes ]] && [[ ! -z $s_log ]]; then
+      proxysql_exec $LINENO -Ns "UPDATE scheduler SET arg1='--config-file=/etc/proxysql-admin.cnf --writer-is-reader=ondemand --write-hg=$s_write_hg --read-hg=$s_read_hg --writer-count=$s_number_of_writes $s_host_priority --mode=$s_mode --log=$s_log', arg2=NULL, arg3=NULL, arg4=NULL, arg5=NULL WHERE id=$id"
+      check_cmd_and_exit $LINENO $? "Could not update the scheduler (query failed). Exiting."
+      rows_modified+=1
+    fi
+  done< <(printf "${scheduler_rows}\n")
+
+  if [[ $rows_modified -gt 0 ]]; then
+    proxysql_exec $LINENO -Ns "LOAD SCHEDULER TO RUNTIME; SAVE SCHEDULER TO DISK;"
+    check_cmd_and_exit $LINENO $? "Could not save scheduler changes to runtime or disk (query failed). Exiting."
+    log_if_success $LINENO $? "Scheduler changes saved to runtime and disk"
+  fi
+
+  log $LINENO "$rows_modified row(s) modified / $rows_found row(s) found"
+  log $LINENO "**** Scheduler upgrade finished ****"
+}
+
+
+# Upgrade the status of a node
+# This may also perform an INSERT (depending on the reader_status)
+#
+# Globals:
+#   TIMEOUT
+#
+# Arguments:
+#   1: lineno
+#   2: hostgroup
+#   3: server address
+#   4: port
+#   5: new status
+#   6: reader_status
+#   7: comment
+#
+function change_server_status() {
+  local lineno=$1
+  local hostgroup=$2
+  local server=$3
+  local port=$4
+  local status=$5
+  local reader_status=$6
+  local comment=$7
+  local address
+
+  address=$(combine_ip_port_into_address "$server" "$port")
+
+  # If we have a PRIORITY_NODE, then we don't have a WRITER entry
+  # to upgrade, but we do have a READER entry, so upgrade that
+  if [[ $reader_status == "PRIORITY_NODE" ]]; then
+    proxysql_exec $lineno -Ns "INSERT INTO mysql_servers
+          (hostname,hostgroup_id,port,weight,status,comment,max_connections)
+          VALUES ('$server',$hostgroup,$port,1000000,'$status','WRITE',$MAX_CONNECTIONS);"
+    check_cmd_and_exit $LINENO $? "Could not create new mysql_servers row (query failed). Exiting."
+    log "$lineno" "Adding server $hostgroup:$address with status $status. Reason: $comment"
+  else
+    proxysql_exec $lineno -Ns "UPDATE mysql_servers
+          set status = '$status' WHERE hostgroup_id = $hostgroup AND hostname = '$server' AND port = $port;" 2>>${ERR_FILE}
+    check_cmd_and_exit $LINENO $? "Could not update new mysql_servers row (query failed). Exiting."
+    log "$lineno" "Changing server $hostgroup:$address to status $status. Reason: $comment"
+  fi
+}
+
+
+# Arguments:
+#   The arguments to the script.
+#
+function parse_args() {
+  local go_out=""
+
+  # TODO: kennt, what happens if we don't have a functional getopt()?
+  # Check if we have a functional getopt(1)
+  if ! getopt --test; then
+    go_out="$(getopt --options=w:r:c:l:n:m:p:vh --longoptions=write-hg:,read-hg:,config-file:,log:,node-monitor-log:,writer-count:,mode:,priority:,writer-is-reader:,use-slave-as-writer:,log-text:,max-connections:,version,debug,help \
+    --name="$(basename "$0")" -- "$@")"
+    if [[ $? -ne 0 ]]; then
+      # no place to send output
+      echo "proxysql_galera_checker : Script error: getopt() failed" >&2
+      exit 1
+    fi
+    eval set -- "$go_out"
+  fi
+
+  if [[ $go_out == " --" ]];then
+    usage
+    exit 1
+  fi
+
+  #
+  # We iterate through the command-line options twice
+  # (1) to handle options that don't need permissions (such as --help)
+  # (2) to handle options that need to be done before other
+  #     options, such as loading the config file
+  #
+  for arg
+  do
+    case "$arg" in
+      -- ) shift; break;;
+      --config-file )
+        CONFIG_FILE="$2"
+        check_permission -e $LINENO "$CONFIG_FILE" "proxysql-admin configuration file"
+        debug $LINENO  "--config-file specified, using : $CONFIG_FILE"
+        shift 2
+        ;;
+      --help)
+        usage
+        exit 0
+        ;;
+      -v | --version)
+        echo "proxysql_galera_checker version $PROXYSQL_ADMIN_VERSION"
+        exit 0
+        ;;
+      --debug)
+        DEBUG=1
+        shift
+        ;;
+      *)
+        shift
+        ;;
+    esac
+  done
+
+  #
+  # Load the config file before reading in the command-line options
+  #
+  readonly CONFIG_FILE
+  if [ ! -e "$CONFIG_FILE" ]; then
+      warning "" "Could not locate the configuration file: $CONFIG_FILE"
+  else
+      check_permission -r $LINENO "$CONFIG_FILE"
+      debug $LINENO "Loading $CONFIG_FILE"
+      source "$CONFIG_FILE"
+  fi
+
+  if [[ $DEBUG -ne 0 ]]; then
+    # For now
+    if [[ -t 1 ]]; then
+      ERR_FILE=/dev/stderr
+    fi
+  fi
+
+  # Reset the command line for the next invocation
+  eval set -- "$go_out"
+
+  for arg
+  do
+    case "$arg" in
+      -- ) shift; break;;
+      -w | --write-hg )
+        HOSTGROUP_WRITER_ID=$2
+        shift 2
+      ;;
+      -r | --read-hg )
+        HOSTGROUP_READER_ID=$2
+        shift 2
+      ;;
+      --config-file )
+        # Do no processing of config-file here, it is processed
+        # before this loop (see above)
+        shift 2
+      ;;
+      -l | --log )
+        ERR_FILE="$2"
+        shift 2
+        # Test if stderr is open to a terminal
+        # We cannot use stdout as the log output, since it is used
+        # to return values.
+        if [[ $ERR_FILE == "/dev/stderr" ]]; then
+          RED=$(tput setaf 1)
+          NRED=$(tput sgr0)
+        else
+          RED=""
+          NRED=""
+        fi
+      ;;
+      --node-monitor-log )
+        NODE_MONITOR_LOG_FILE="$2"
+        shift 2
+      ;;
+      -n | --writer-count )
+        NUMBER_WRITERS="$2"
+        shift 2
+      ;;
+      -p | --priority )
+        P_PRIORITY="$2"
+        shift 2
+      ;;
+      -m | --mode )
+        P_MODE="$2"
+        shift 2
+        if [ "$P_MODE" != "loadbal" ] && [ "$P_MODE" != "singlewrite" ]; then
+          echo "ERROR: Invalid --mode passed:"
+          echo "  Please choose one of these modes: loadbal, singlewrite"
+          exit 1
+        fi
+      ;;
+      --writer-is-reader )
+        WRITER_IS_READER="$2"
+        shift 2
+      ;;
+      --use-slave-as-writer )
+        SLAVE_IS_WRITER="$2"
+        shift 2
+      ;;
+      --max-connections )
+        MAX_CONNECTIONS="$2"
+        shift 2
+      ;;
+      --debug )
+        # Not handled here, see above
+        shift
+      ;;
+      --log-text )
+        LOG_TEXT="$2"
+        shift 2
+      ;;
+      -v | --version )
+        # Not handled here, see above
+        shift
+      ;;
+      -h | --help )
+        # Not handled here, see above
+        shift
+      ;;
+    esac
+  done
+
+  if [[ $DEBUG -eq 1 ]]; then
+    DEBUG_ERR_FILE=$ERR_FILE
+  fi
+
+  #
+  # Argument validation
+  #
+  test $HOSTGROUP_WRITER_ID -ge 0 &>/dev/null
+  if [[ $? -ne 0 ]]; then
+    echo "ERROR: writer hostgroup_id is not an integer"
+    usage
+    exit 1
+  fi
+
+  test $HOSTGROUP_READER_ID -ge -1 &>/dev/null
+  if [[ $? -ne 0 ]]; then
+    echo "ERROR: reader hostgroup_id is not an integer"
+    usage
+    exit 1
+  fi
+
+  HOSTGROUP_SLAVEREADER_ID=$HOSTGROUP_READER_ID
+  if [ $HOSTGROUP_SLAVEREADER_ID -eq $HOSTGROUP_WRITER_ID ];then
+    let HOSTGROUP_SLAVEREADER_ID+=1
+  fi
+
+  if [[ $NUMBER_WRITERS -lt 0 ]]; then
+    echo "ERROR: The number of writers should either be 0 to enable all possible nodes ONLINE"
+    echo "       or be larger than 0 to limit the number of writers"
+    usage
+    exit 1
+  fi
+
+  if [[ ! $WRITER_IS_READER =~ ^(always|never|ondemand)$ ]]; then
+    error "" "Invalid --writer-is-reader option: '$WRITER_IS_READER'"
+    echo "Please choose one of these values: always, never, or ondemand"
+    exit 1
+  fi
+
+  if [[ ! $SLAVE_IS_WRITER =~ ^(yes|YES|no|NO)$ ]]; then
+    error "" "Invalid --use-slave-as-writer option: '$SLAVE_IS_WRITER'"
+    echo "Please choose either yes or no"
+    exit 1
+  fi
+  if [[ $SLAVE_IS_WRITER =~ ^(yes|YES)$ ]]; then
+    SLAVE_IS_WRITER="yes"
+  else
+    SLAVE_IS_WRITER="no"
+  fi
+
+  # These may get set in the config file, but they are not used
+  # by this script.  So to avoid confusion and problems, remove
+  # these variables explicitly.
+  unset WRITE_HOSTGROUP_ID
+  unset READ_HOSTGROUP_ID
+  unset SLAVEREAD_HOSTGROUP_ID
+
+
+  # Verify that we have an integer
+  if [[ -n $MAX_CONNECTIONS ]]; then
+    if ! [ "$MAX_CONNECTIONS" -eq "$MAX_CONNECTIONS" ] 2>/dev/null
+    then
+      error "" "option '--max-connections' parameter (must be a number) : $MAX_CONNECTIONS"
+      exit 1
+    fi
+  fi
+
+  readonly ERR_FILE
+  readonly CONFIG_FILE
+  readonly DEBUG_ERR_FILE
+
+  readonly HOSTGROUP_WRITER_ID
+  readonly HOSTGROUP_READER_ID
+  readonly HOSTGROUP_SLAVEREADER_ID
+  readonly NUMBER_WRITERS
+
+  readonly WRITER_IS_READER
+  readonly SLAVE_IS_WRITER
+
+  readonly P_PRIORITY
+  readonly P_MODE
+
+  readonly MAX_CONNECTIONS
+}
+
+
+# Checks to see if another instance of the script is running.
+# We want only one instance of this script to be running at a time.
+#
+# Globals:
+#   PROXYSQL_DATADIR
+#   ERR_FILE
+#
+# Arguments:
+#   1: the cluster name
+#
+# This function will exit if another instance of the script is running.
+#
+# With thanks, http://bencane.com/2015/09/22/preventing-duplicate-cron-job-executions/
+#
+function check_is_galera_checker_running() {
+  local cluster_name=$1
+
+  if [[ -z $cluster_name ]]; then
+    CHECKER_PIDFILE=${PROXYSQL_DATADIR}/galera_checker.pid
+  else
+    CHECKER_PIDFILE=${PROXYSQL_DATADIR}/${cluster_name}_galera_checker.pid
+  fi
+
+  if [[ -f $CHECKER_PIDFILE && -r $CHECKER_PIDFILE ]] ; then
+    local GPID
+    GPID=$(cat "$CHECKER_PIDFILE")
+    if ps -p $GPID -o args=ARGS | grep $ERR_FILE | grep -o proxysql_galera_check >/dev/null 2>&1 ; then
+      ps -p $GPID > /dev/null 2>&1
+      if [[ $? -eq 0 ]]; then
+        log "$LINENO" "ProxySQL galera checker process already running. (pid:$GPID  this pid:$$)"
+
+        # We don't want to remove this file on cleanup
+        CHECKER_PIDFILE=""
+        exit 1
+      else
+        echo $$ > $CHECKER_PIDFILE
+        if [[ $? -ne 0 ]] ; then
+          warning "$LINENO" "Could not create galera checker PID file"
+          exit 1
+        fi
+        debug "$LINENO" "Created PID file at $CHECKER_PIDFILE"
+      fi
+    else
+      warning "$LINENO" "Existing PID($GPID) belongs to some other process. Creating new PID file."
+      echo $$ > $CHECKER_PIDFILE
+      if [[ $? -ne 0 ]] ; then
+        warning "$LINENO" "Could not create galera checker PID file"
+        exit 1
+      fi
+      debug "$LINENO" "Created PID file at $CHECKER_PIDFILE"
+    fi
+  else
+    echo "$$" > "$CHECKER_PIDFILE"
+    if [[ $? -ne 0 ]]; then
+      warning "$LINENO" "Could not create galera checker PID file"
+      exit 1
+    fi
+    debug "$LINENO" "Created PID file at $CHECKER_PIDFILE"
+  fi
+}
+
+function cleanup_handler() {
+  if [[ -n $CHECKER_PIDFILE ]]; then
+    rm -f $CHECKER_PIDFILE
+  fi
+}
+
+
+# Checks to see that the READ hostgroup entries have been created/deleted
+# This function has no meaning if mode="loadbal".
+#
+# Globals:
+#   HOSTGROUP_READER_ID
+#   HOSTGROUP_WRITER_ID
+#   WRITER_IS_READER
+#   MODE
+#
+# Arguments:
+#   1: the name of the reload_check_file
+#   2: the number of online-readers
+#
+function writer_is_reader_check() {
+  local reload_check_file=$1
+  local number_readers_online=$2
+  local servers
+
+  if [[ $MODE == "loadbal" ]]; then
+    return
+  fi
+
+  servers=$(proxysql_exec $LINENO -Ns "SELECT hostgroup_id, hostname, port, status
+                                       FROM mysql_servers
+                                       WHERE hostgroup_id IN ($HOSTGROUP_READER_ID,$HOSTGROUP_WRITER_ID) AND
+                                       status <> 'OFFLINE_HARD'
+                                        AND comment <> 'SLAVEREAD'
+                                       ORDER BY hostname, port, hostgroup_id")
+  check_cmd_and_exit $LINENO $? "Could not retreive data from mysql_servers (query failed). Exiting."
+
+  debug $LINENO "writer_is_reader_check : number_readers_online:$number_readers_online"
+  # The extra test at the end is to ensure that the very last line read in
+  # is also handled (the read may be returning EOF)
+  printf "${servers}" | while read hostgroup server port stat || [ -n "$stat" ]
+  do
+    debug $LINENO "Examining $hostgroup:$server:$port"
+
+    # Only look at writer nodes, so skip non-writers
+    if [[ $hostgroup -ne $HOSTGROUP_WRITER_ID ]]; then
+      continue
+    fi
+
+    local regex="(^|[[:space:]])${HOSTGROUP_READER_ID}[[:blank:]]${server}[[:blank:]]${port}[[:blank:]]"
+    local address
+    address=$(combine_ip_port_into_address "$server" "$port")
+
+    if [[ $WRITER_IS_READER == "always" || $WRITER_IS_READER == "ondemand" ]]; then
+      #
+      # Ensure that there is a corresponding entry in mysql_servers
+      #
+      local reader_row=$(echo "$servers" | grep -E "$regex")
+      if [[ -z $reader_row ]]; then
+        # We did not find a matching row, insert the corresponding READER entry
+
+        # If we are to always have a reader, or if there are no other
+        # readers, add it as an ONLINE reader
+        local comment
+        if [[ $WRITER_IS_READER == "always" || $number_readers_online -eq 0 ]]; then
+          comment="ONLINE"
+        else
+          comment="OFFLINE_SOFT"
+        fi
+        proxysql_exec $LINENO -Ns "INSERT INTO mysql_servers (hostname,hostgroup_id,port,weight,status,comment,max_connections) VALUES ('$server',$HOSTGROUP_READER_ID,$port,1000,'$comment','READ',$MAX_CONNECTIONS);"
+        check_cmd_and_exit $LINENO $? "writer-is-reader($WRITER_IS_READER): Failed to add $HOSTGROUP_READER_ID:$address (query failed). Exiting."
+        log_if_success $LINENO $? "writer-is-reader($WRITER_IS_READER): Adding reader $HOSTGROUP_READER_ID:$address with status OFFLINE_SOFT"
+        echo "1" > ${reload_check_file}
+      elif [[ $WRITER_IS_READER == "ondemand" && $stat == "ONLINE" && $number_readers_online -gt 1 ]]; then
+        # If the reader node is ONLINE and there is another reader
+        # Then we can deactivate this reader node (if ondemand)
+
+        # This should only do this if there is a corresponding READ entry that is NOT "OFFLINE_SOFT"
+        # Search through the $servers for an entry corresponding to the READ data
+        local reader_status=$(echo -e "$reader_row" | cut -f4)
+        if [[ $reader_status != "OFFLINE_SOFT" ]]; then
+          proxysql_exec $LINENO -Ns "UPDATE mysql_servers SET status='OFFLINE_SOFT' WHERE hostgroup_id=$HOSTGROUP_READER_ID AND hostname='$server' AND port=$port"
+          check_cmd_and_exit $LINENO $? "Could not update mysql_servers (query failed). Exiting."
+          log_if_success $LINENO $? "Updating $hostgroup:$address to OFFLINE_SOFT because writers prefer to not be readers (writer-is-reader:ondemand)"
+          echo "1" > ${reload_check_file}
+        fi
+      fi
+    elif [[ $WRITER_IS_READER == "never" ]]; then
+      #
+      # Ensure that there is NO corresponding entry in mysql_servers
+      #
+      if [[ $servers =~ $regex ]]; then
+        # Delete the corresponding READER entry
+        proxysql_exec $LINENO -Ns "DELETE FROM mysql_servers WHERE hostgroup_id=$HOSTGROUP_READER_ID and hostname='$server' and port=$port"
+        check_cmd_and_exit $LINENO $? "writer-is-reader(never): Failed to remove $HOSTGROUP_READER_ID:$address (query failed). Exiting."
+        log_if_success $LINENO $? "writer-is-reader(never): Removing reader $HOSTGROUP_READER_ID:$address"
+        echo "1" > ${reload_check_file}
+      fi
+    fi
+  done
+}
+
+
+# Returns the address of an available (online) cluster host
+#
+# Globals:
+#   None
+#
+# Arguments:
+#   None
+#
+# Returns:
+#   0 : if the function succeeds (this means that no errors occurred in the SQL queries)
+#     The function can return nothing and still return success
+#   1 : if an error occurs
+#
+function find_online_cluster_host() {
+  # Query the proxysql database for hosts,ports in use
+  # Then just go through the list until we reach one that responds
+  local hosts
+  hosts=$(proxysql_exec $LINENO -Ns "SELECT hostname,port FROM mysql_servers where hostgroup_id in ($HOSTGROUP_WRITER_ID, $HOSTGROUP_READER_ID) and comment <> 'SLAVEREAD' and status='ONLINE'")
+  if [[ $? -ne 0 ]]; then
+    return 1
+  fi
+  printf "$hosts" | while read server port || [[ -n $port ]]
+  do
+    debug $LINENO "Trying to contact $server:$port..."
+    mysql_exec "$LINENO" "$server" "$port" -Bs "select @@port" 1>/dev/null 2>>${DEBUG_ERR_FILE}
+    if [[ $? -eq 0 ]]; then
+      printf "$server $port"
+      return 0
+    fi
+  done
+
+  # No cluster host available (cannot contact any)
+  return 0
+}
+
+# Checks the number of online writers and promotes a reader to a writer
+# if needed
+#
+# Globals:
+#   HOSTGROUP_WRITER_ID
+#   CHECK_STATUS
+#
+# Arguments:
+#   1: The path to the reload_check_file
+#
+function ensure_one_writer_node() {
+  local reload_check_file=$1
+
+  log $LINENO "No ONLINE writers found, looking for readers to promote"
+
+  # We are trying to find a reader node that can be promoted to
+  # a writer.
+  # So what we do is we ORDER BY writer.status ASC because
+  # by accident ONLINE is last in the line
+  local reader_proxysql_query="SELECT
+           reader.hostname,
+           reader.port,
+           writer.status
+    FROM mysql_servers as reader
+    LEFT JOIN mysql_servers as writer
+      ON writer.hostgroup_id = $HOSTGROUP_WRITER_ID
+      AND writer.hostname = reader.hostname
+      AND writer.port = reader.port
+    WHERE reader.hostgroup_id = $HOSTGROUP_READER_ID
+      AND reader.status = 'ONLINE'
+      AND reader.comment = 'READ'
+    ORDER BY writer.status ASC,
+             reader.weight DESC,
+             reader.hostname,
+             reader.port"
+
+  local possible_hosts
+  possible_hosts=$(proxysql_exec $LINENO -Ns "$reader_proxysql_query")
+  check_cmd_and_exit $LINENO $? "Could not get data from mysql_servers (query failed). Exiting."
+  if [[ -z $possible_hosts ]]; then
+    log $LINENO "Cannot find a reader that can be promoted to a writer"
+    return
+  fi
+
+  while read line; do
+    if [[ -z $line ]]; then
+      continue
+    fi
+
+    local host=$(echo $line | awk '{ print $1 }')
+    local port=$(echo $line | awk '{ print $2 }')
+
+    # Have to check that we can actually access the node
+    local wsrep_status
+    local pxc_main_mode
+    local result
+    local address
+    address=$(combine_ip_port_into_address "$host" "$port")
+    result=$(mysql_exec $LINENO "$host" "$port" -Nns "SHOW STATUS LIKE 'wsrep_local_state'; SHOW VARIABLES LIKE 'pxc_maint_mode';" 2>>${DEBUG_ERR_FILE})
+    wsrep_status=$(echo "$result" | grep "wsrep_local_state" | awk '{ print $2 }')
+    pxc_main_mode=$(echo "$result" | grep "pxc_maint_mode" | awk '{ print $2 }')
+
+    if [[ -z $pxc_main_mode ]]; then
+      pxc_main_mode="DISABLED"
+    fi
+
+    if [[ $wsrep_status -ne 4 || $pxc_main_mode != "DISABLED" ]]; then
+      continue
+    fi
+
+    local write_stat=$(echo "$line" | awk '{print $3}')
+    debug $LINENO "Looking at $address write_status:$write_stat"
+
+    log $LINENO "Promoting $address as writer node..."
+
+    # We have an ONLINE reader node
+    # if ondemand or always keep the reader node (may need to move it to OFFLINE_SOFT)
+    #   if we have a writer, move to ONLINE
+    #   if no writer, create it
+
+    if [[ $WRITER_IS_READER == "ondemand" || $WRITER_IS_READER == "always" ]]; then
+      if [[ -z $write_stat || $write_stat == "NULL" ]]; then
+        # Writer does not exist, add an ONLINE writer
+        proxysql_exec $LINENO -Ns \
+          "INSERT INTO mysql_servers (hostname,hostgroup_id,port,weight,status,comment,max_connections)
+           VALUES ('$host',$HOSTGROUP_WRITER_ID,$port,1000000,'ONLINE','WRITE',$MAX_CONNECTIONS);"
+        check_cmd_and_exit $LINENO $? "Cannot add a PXC writer node to ProxySQL (query failed). Exiting."
+        log_if_success $LINENO $? "Added $HOSTGROUP_WRITER_ID:$address as a writer."
+      else
+        # Writer exists, move writer to ONLINE
+        proxysql_exec $LINENO -Ns "UPDATE mysql_servers SET status='ONLINE',comment='WRITE',weight=1000000 WHERE hostgroup_id=$HOSTGROUP_WRITER_ID AND hostname='$host' AND port=$port"
+        check_cmd_and_exit $LINENO $? "Cannot update the PXC node $HOSTGROUP_WRITER_ID:$address (query failed). Exiting."
+        log_if_success $LINENO $? "Updated ${HOSTGROUP_WRITER_ID}:${host}:${port} node in ProxySQL."
+      fi
+    else
+      if [[ -z $write_stat || $write_stat == "NULL" ]]; then
+        # Writer does not exist, move reader to writer
+        proxysql_exec $$LINENO -Ns\
+            "UPDATE mysql_servers set status='ONLINE',hostgroup_id=$HOSTGROUP_WRITER_ID, comment='WRITE', weight=1000000 WHERE hostgroup_id=$HOSTGROUP_READER_ID and hostname='$host' and port=$port"
+        check_cmd_and_exit $LINENO $? "Cannot update PXC writer in ProxySQL (query failed). Exiting."
+        log_if_success $LINENO $? "Added $HOSTGROUP_WRITER_ID:$address as a writer node."
+      else
+        # Writer exists, move writer to ONLINE, remove reader
+        proxysql_exec $LINENO -Ns "UPDATE mysql_servers SET status='ONLINE',comment='WRITE',weight=1000000 WHERE hostgroup_id=$HOSTGROUP_WRITER_ID AND hostname='$host' AND port=$port"
+        check_cmd_and_exit $LINENO $? "Cannot update the PXC node $HOSTGROUP_WRITER_ID:$address (query failed). Exiting."
+        log_if_success $LINENO $? "Updated ${HOSTGROUP_WRITER_ID}:${host}:${port} node in ProxySQL."
+
+        proxysql_exec $LINENO -Ns "DELETE FROM mysql_servers WHERE hostgroup_id=$HOSTGROUP_READER_ID and hostname='$host' and port=$port"
+        check_cmd_and_exit $LINENO $? "writer-is-reader(never): Failed to remove $HOSTGROUP_READER_ID:$server:$port (query failed). Exiting."
+        log_if_success $LINENO $? "writer-is-reader(never): Removing reader $HOSTGROUP_READER_ID:$server:$port"
+      fi
+    fi
+
+    echo '1' > ${reload_check_file}
+    break
+
+  done< <(printf "${possible_hosts}\n")
+}
+
+# Returns the host priority list defined by the user
+#
+# Globals:
+#   HOST_PRIORITY_FILE
+#   P_PRIORITY
+#   PROXYSQL_DATADIR
+#   CLUSTER_NAME
+#
+# Arguments:
+#   1: cluster name
+#
+function get_host_priority_list() {
+  local cluster_name=$1
+  local priority_hosts=""
+
+  if [[ -z $HOST_PRIORITY_FILE || ! -f $HOST_PRIORITY_FILE ]]; then
+    HOST_PRIORITY_FILE=${PROXYSQL_DATADIR}/${cluster_name}_host_priority
+  fi
+
+  if [[ ! -z "$P_PRIORITY" ]] ; then
+    IFS=',' read -r -a priority_hosts <<< "$P_PRIORITY"
+    debug $LINENO "host_priority = ${priority_hosts[@]}"
+  elif [[ -f $HOST_PRIORITY_FILE ]];then
+    # Get the list of hosts from the host_priority file ignoring blanks and
+    # any lines that start with '#'
+    debug $LINENO "Found a host priority file: $HOST_PRIORITY_FILE"
+    priority_hosts=$(cat "$HOST_PRIORITY_FILE" | grep '^[^#]')
+    if [[ -n $priority_hosts ]]; then
+      priority_hosts=($(echo $priority_hosts))
+    fi
+  fi
+  #   File sample:
+  #   10.11.12.21:3306
+  #   10.21.12.21:3306
+  #   10.31.12.21:3306
+  #
+  echo "${priority_hosts[@]}"
+}
+
+
+# Builds the proxysql host list and merges it with the priority list.
+# The entries from the priority list are put at the top in order.
+#
+# If they have an entry in the proxysql_list, the data is copied over.
+# If it does not exist in the proxysql_list, then it is added
+# with status=OFFLINE_SOFT.
+#
+# This ensures that the entry will not be added as
+# a writer if the node is really offline.
+#
+# Globals:
+#   None
+#
+# Argument:
+#   1: field separator
+#   2: the proxysql list (array of nodes from proxysql)
+#   3: the priority list
+#
+function build_proxysql_list_with_priority() {
+  local sep=$1
+  local proxysql_list=$(echo "$2" | tr ' ' '\n')
+  local priority_list=($3)
+  local new_proxysql_list=()
+  local reader_list=""
+  local reader_query=0
+
+  for prio in "${priority_list[@]}"; do
+    local psql_entry
+    local prio_entry
+    local prio_ip
+    local prio_port
+    prio_entry=$(separate_ip_port_from_address "$prio")
+    prio_ip=$(echo "$prio_entry" | cut -d' ' -f1)
+    prio_port=$(echo "$prio_entry" | cut -d' ' -f2)
+    prio_entry="$prio_ip$sep$prio_port"
+    # properly escape the characters for grep
+    prio_entry_re="$(printf '%s' "$prio_entry" | sed 's/[.[\*^$]/\\&/g')"
+
+    psql_entry=$(echo "$proxysql_list" | grep "^${prio_entry_re}[^[[:space:]]]*")
+    if [[ $? -eq 0 && -n ${psql_entry} ]]; then
+      # Add entries that are in the priority list and in the proxysql list
+      new_proxysql_list+=($psql_entry)
+    else
+      # Add entries that are in the priority list but not in proxysql
+      # (but only if there is a reader entry for the address)
+      if [[ $reader_query -eq 0 ]]; then
+        reader_list=$(proxysql_exec $LINENO -Ns "SELECT hostname, port 
+                          FROM mysql_servers
+                          WHERE hostgroup_id IN ($HOSTGROUP_READER_ID)
+                            AND status <> 'OFFLINE_HARD'
+                            AND comment <> 'SLAVEREAD'
+                            ORDER BY status DESC, hostgroup_id, weight DESC, hostname, port")
+        check_cmd_and_exit $LINENO $? "Unable to obtain list of nodes from ProxySQL (query failed). Exiting."
+        reader_list=$(echo "$reader_list" | tr '\t' "$sep" | tr '\n' ' ')
+        reader_query=1
+      fi
+      if [[ -n $reader_list ]]; then
+        if echo $reader_list | grep -E -q "(^|[[:space:]])${prio_entry_re}($|[[:space:]])*"; then
+          # If this address is a reader, add a fake writer entry
+          # If detected to be online, the writer check algorithm
+          # will insert an entry
+          new_proxysql_list+=("${prio_entry}${sep}${HOSTGROUP_WRITER_ID}${sep}OFFLINE_SOFT${sep}WRITE${sep}PRIORITY_NODE")
+        fi
+      fi
+    fi
+  done
+
+  # Add the rest of the entries (not in the priority list but in proxysql)
+
+  # For this magic, see:
+  # https://stackoverflow.com/questions/7577052/bash-empty-array-expansion-with-set-u
+  if [[ -n ${priority_list[@]+"${priority_list[@]}"} ]]; then
+    local priority_string="${priority_list[@]}"
+    for row in ${proxysql_list}; do
+      local host=$(echo $row | cut -d "$sep" -f 1)
+      local port=$(echo $row | cut -d "$sep" -f 2)
+      local addr=$(combine_ip_port_into_address "$host" "$port")
+
+      # properly escape the characters for grep -E
+      addr_re="$(printf '%s' "$addr" | sed 's/[.[\*^$()+?{|]/\\&/g')"
+
+      if ! echo $priority_string | grep -E -q "(^|[[:space:]])${addr_re}($|[[:space:]])"; then
+        new_proxysql_list+=($row)
+      fi
+    done
+  fi
+
+  if [[ -n ${new_proxysql_list[@]+"${new_proxysql_list[@]}"} ]]; then
+    debug $LINENO "new priority list = ${new_proxysql_list[@]}"
+    echo "${new_proxysql_list[@]}"
+  fi
+}
+
+
+function update_writers() {
+  local reload_check_file=$1
+  local proxysql_list=""
+  local priority_list=""
+
+  # Nodes are orderd by status DESC first, this allows ONLINE nodes to always
+  # be processed first.
+  writer_proxysql_query="SELECT
+            writer.hostname,
+            writer.port,
+            writer.hostgroup_id,
+            writer.status,
+            writer.comment,
+            reader.status
+    FROM mysql_servers as writer
+    LEFT JOIN mysql_servers as reader
+      ON reader.hostgroup_id = $HOSTGROUP_READER_ID
+      AND reader.hostname = writer.hostname
+      AND reader.port = writer.port
+    WHERE writer.hostgroup_id = $HOSTGROUP_WRITER_ID
+      AND writer.status <> 'OFFLINE_HARD'
+    ORDER BY writer.status DESC"
+
+  proxysql_list=$(proxysql_exec $LINENO -Ns "$writer_proxysql_query")
+  check_cmd_and_exit $LINENO $? "Unable to obtain list of nodes from ProxySQL (query failed). Exiting."
+
+  if [[ $proxysql_list =~ [[:space:]]ONLINE[[:space:]]SLAVEREAD[[:space:]] ]]; then
+    HAVE_SLAVEWRITERS=1
+  fi
+
+  # Now that we have the proxysql list, order it by the priority list
+  # (with unprioritized nodes at the end)
+
+  # Using the priority list only makes sense for non-load balancing
+  if [[ $MODE != "loadbal" ]]; then
+    priority_list=$(get_host_priority_list "$cluster_name")
+  fi
+  if [[ -n $priority_list ]]; then
+    #
+    # Need to do some processing of the lists so that we can pass them down
+    # Assume that the fields cannot contain the ';' character'
+    # Can't use "local -n" becaus of Centos6 (needs newer bash version)
+    #
+    local prio_list=${priority_list}
+    local pxsql_list=$(echo "${proxysql_list[@]}" | tr '\t' ';' | tr '\n' ' ')
+
+    proxysql_list=$(build_proxysql_list_with_priority ";" "$pxsql_list" "$prio_list")
+    proxysql_list=$(echo "${proxysql_list}" | tr ' ' '\n' | tr ';' '\t')
+  fi
+
+  if [[ -z ${proxysql_list[@]+"${proxysql_list[@]}"} ]]; then
+    log $LINENO "No writers found."
+    return
+  fi
+
+  # Go through the list
+  #
+  while read line; do
+    if [[ -z $line ]]; then
+      continue
+    fi
+
+    server=$(echo -e "$line" | cut -f1)
+    port=$(echo -e "$line" | cut -f2)
+    hostgroup=$(echo -e "$line" | cut -f3)
+    stat=$(echo -e "$line" | cut -f4)
+    comment=$(echo -e "$line" | cut -f5)
+    rdstat=$(echo -e "$line" | cut -f6)
+
+    if [[ $comment == "SLAVEREAD" ]]; then
+      set_slave_status "${reload_check_file}" $hostgroup $server $port $stat
+      continue
+    fi
+
+    local wsrep_status
+    local pxc_main_mode
+    local result
+    local address
+    address=$(combine_ip_port_into_address "$server" "$port")
+    result=$(mysql_exec $LINENO "$server" "$port" -Nns "SHOW STATUS LIKE 'wsrep_local_state'; SHOW VARIABLES LIKE 'pxc_maint_mode';" 2>>${DEBUG_ERR_FILE})
+    wsrep_status=$(echo "$result" | grep "wsrep_local_state" | awk '{ print $2 }')
+    pxc_main_mode=$(echo "$result" | grep "pxc_maint_mode" | awk '{ print $2 }')
+
+    if [[ -z $wsrep_status ]]; then
+      wsrep_status="<unknown:query failed>"
+    fi
+
+    # For PXC 5.6 there is no pxc_maint_mode, so assume DISABLED
+    if [[ -z $pxc_main_mode ]]; then
+      pxc_main_mode="DISABLED"
+    fi
+
+    # If this node was added because of the priority list and the
+    # node is not reporting SYNCED, then just skip it
+    if [[ $rdstat == "PRIORITY_NODE" && $wsrep_status != "4" ]]; then
+      continue
+    fi
+
+    log "" "--> Checking WRITE server $hostgroup:$address, current status $stat, wsrep_local_state $wsrep_status"
+
+    # we have to limit amount of writers, WSREP status OK, AND node is marked ONLINE
+    # PXC and ProxySQL agreee on the status
+    # wsrep:ok  pxc:--  status:online
+    if [ $NUMBER_WRITERS -gt 0 -a "${wsrep_status}" = "4" -a "$stat" == "ONLINE" -a "${pxc_main_mode}" == "DISABLED" ] ; then
+      if [[ $number_writers_online -lt $NUMBER_WRITERS ]]; then
+        number_writers_online=$(( $number_writers_online + 1 ))
+        log "" "server $hostgroup:$address is already ONLINE: ${number_writers_online} of ${NUMBER_WRITERS} write nodes"
+      else
+        number_writers_online=$(( $number_writers_online + 1 ))
+        change_server_status $LINENO $HOSTGROUP_WRITER_ID "$server" "$port" "OFFLINE_SOFT" "$rdstat" \
+                             "max write nodes reached (${NUMBER_WRITERS})"
+        echo "1" > ${reload_check_file}
+      fi
+    fi
+
+    # WSREP status OK, but node is not marked ONLINE
+    # Make the node ONLINE if possible
+    # wsrep:ok  pxc:ok  status:not online
+    if [ "${wsrep_status}" = "4" -a "$stat" != "ONLINE" -a "${pxc_main_mode}" == "DISABLED" ] ; then
+      # we have to limit amount of writers
+      if [[ $NUMBER_WRITERS -gt 0 ]] ; then
+        if [[ $number_writers_online -lt $NUMBER_WRITERS ]]; then
+          number_writers_online=$(( $number_writers_online + 1 ))
+          change_server_status $LINENO $HOSTGROUP_WRITER_ID "$server" "$port" "ONLINE" "$rdstat" \
+                               "${number_writers_online} of ${NUMBER_WRITERS} write nodes"
+          echo "1" > ${reload_check_file}
+        else
+          number_writers_online=$(( $number_writers_online + 1 ))
+          if [ "$stat" != "OFFLINE_SOFT" ]; then
+            change_server_status $LINENO $HOSTGROUP_WRITER_ID "$server" "$port" "OFFLINE_SOFT" "" \
+                                 "max write nodes reached (${NUMBER_WRITERS})"
+            echo "1" > ${reload_check_file}
+          elif [[ $rdstat != "PRIORITY_NODE" ]]; then
+             log $LINENO "server $hostgroup:$address is already OFFLINE_SOFT, max write nodes reached (${NUMBER_WRITERS})"
+          fi
+        fi
+      # we do not have to limit
+      elif [[ $NUMBER_WRITERS -eq 0 ]] ; then
+        # TODO: kennt, What if node is SHUNNED?
+        change_server_status $LINENO $HOSTGROUP_WRITER_ID "$server" "$port" "ONLINE" "$rdstat"\
+                             "Changed state, marking write node ONLINE"
+        echo "1" > ${reload_check_file}
+      fi
+    fi
+
+    # WSREP status is not ok, but the node is marked online, we should put it offline
+    # wsrep:not ok  pxc:--  status:online
+    if [ "${wsrep_status}" != "4" -a "$stat" = "ONLINE" ]; then
+      change_server_status $LINENO $HOSTGROUP_WRITER_ID "$server" "$port" "OFFLINE_SOFT" "" \
+                           "WSREP status is ${wsrep_status} which is not ok"
+      echo "1" > ${reload_check_file}
+    # wsrep:--  pxc:not ok  status:online
+    elif [ "${pxc_main_mode}" != "DISABLED" -a "$stat" = "ONLINE" ]; then
+      change_server_status $LINENO $HOSTGROUP_WRITER_ID "$server" "$port" "OFFLINE_SOFT" "" \
+                           "pxc_maint_mode is $pxc_main_mode" 2>>${ERR_FILE}
+      echo "1" > ${reload_check_file}
+    # wsrep:not ok  pxc:--  status:offline soft
+    elif [ "${wsrep_status}" != "4" -a "$stat" = "OFFLINE_SOFT" -a "$rdstat" != "PRIORITY_NODE" ]; then
+      log "" "server $hostgroup:$address is already OFFLINE_SOFT, WSREP status is ${wsrep_status} which is not ok"
+    # wsrep:--  pxc:not ok  status:offline soft
+    elif [ "${pxc_main_mode}" != "DISABLED" -a "$stat" = "OFFLINE_SOFT" -a "$rdstat" != "PRIORITY_NODE" ]; then
+      log "" "server $hostgroup:$address is already OFFLINE_SOFT, pxc_maint_mode is ${pxc_main_mode} which is not ok"
+    fi
+  done< <(printf "${proxysql_list[@]}\n")
+}
+
+#
+# This function checks the status of slave machines and sets their status field
+#
+# Globals:
+#   PROXSQL_DATADIR
+#   SLAVE_SECONDS_BEHIND
+#   HOSTGROUP_SLAVEREADER_ID
+#
+# Arguments:
+#   1: Path to the reload_check_file
+#   2: Slave hostgroup
+#   3: Slave IP address
+#   4: Slave port
+#   5: Slave status in ProxySQL
+#
+function set_slave_status() {
+  debug $LINENO "START set_slave_status"
+  local reload_check_file=$1
+  local ws_hg_id=$2
+  local ws_ip=$3
+  local ws_port=$4
+  local ws_status=$5
+  local ws_address
+  ws_address=$(combine_ip_port_into_address "$ws_ip" "$ws_port")
+  local node_id="${ws_hg_id}:${ws_address}"
+
+  # This function will get and return a status of a slave node, 4=GOOD, 2=BEHIND, 0=OTHER
+  local slave_status
+
+  log $LINENO "--> Checking SLAVE server ${node_id}"
+
+  slave_status=$(mysql_exec $LINENO "$ws_ip" "$ws_port" -nsE "SHOW SLAVE STATUS")
+  check_cmd $LINENO $? "Cannot get status from the slave $ws_address, Please check cluster login credentials"
+
+  slave_status=$(echo "$slave_status" | sed 's/ //g')
+  echo "$slave_status" | grep "^Master_Host:" >/dev/null
+  if [ $? -ne 0 ];then
+    #
+    # No status was found, this is not replicating
+    # Only changing the status here as another node might be in the writer hostgroup
+    #
+    proxysql_exec $LINENO -Ns "UPDATE mysql_servers SET status = 'OFFLINE_HARD'  WHERE hostname='$ws_ip' and port=$ws_port;"
+    check_cmd_and_exit $LINENO $? "Cannot update Galera Cluster node $ws_address to ProxySQL database (query failed). Exiting"
+    log_if_success $LINENO $? "slave server ${ws_address} set to OFFLINE_HARD status in ProxySQL (cannot determine slave status)."
+    echo "1" > ${reload_check_file}
+  else
+    local slave_master_host slave_io_running slave_sql_running seconds_behind
+    slave_master_host=$(echo "$slave_status" | grep "^Master_Host:" | cut -d: -f2)
+    slave_io_running=$(echo "$slave_status" | grep "^Slave_IO_Running:" | cut -d: -f2)
+    slave_sql_running=$(echo "$slave_status" | grep "^Slave_SQL_Running:" | cut -d: -f2)
+    seconds_behind=$(echo "$slave_status" | grep "^Seconds_Behind_Master:" | cut -d: -f2)
+
+    if [ "$seconds_behind" == "NULL" ];then
+      #
+      # When slave_io is not working, the seconds behind value will read 'NULL',
+      # convert this to a number higher than the max
+      #
+      let seconds_behind=SLAVE_SECONDS_BEHIND+1
+    fi
+
+    if [ "$slave_sql_running" != "Yes" ];then
+      #
+      # Slave is not replicating, so set to OFFLINE_HARD
+      #
+      if [ "$ws_status" != "OFFLINE_HARD" ];then
+        proxysql_exec $LINENO -Ns "UPDATE mysql_servers SET status = 'OFFLINE_HARD' WHERE hostname='$ws_ip' and port=$ws_port;"
+        check_cmd_and_exit $LINENO $? "Cannot update Galera Cluster node $ws_address to ProxySQL database (query failed). Exiting."
+        log_if_success $LINENO $? "slave server ${ws_address} set to OFFLINE_HARD status in ProxySQL (io:$slave_io_running sql:$slave_sql_running)."
+        echo "1" > ${reload_check_file}
+      else
+        log $LINENO "slave server (${ws_hg_id}:${ws_address}) current status '$ws_status' in ProxySQL. (io:$slave_io_running sql:$slave_sql_running)"
+      fi
+    elif [[ $slave_io_running == "Yes" ]]; then
+      #
+      # The slave is replicating (and the cluster is up)
+      # So set the status accordingly
+      #
+      if [ $seconds_behind -gt $SLAVE_SECONDS_BEHIND ];then
+        # Slave is more than the set number of seconds behind, return status 2
+        if [ "$ws_status" != "OFFLINE_SOFT" ];then
+          proxysql_exec $LINENO -Ns "UPDATE mysql_servers SET status = 'OFFLINE_SOFT' WHERE hostname='$ws_ip' and port=$ws_port;"
+          check_cmd_and_exit $LINENO $? "Cannot update Galera Cluster node $ws_address to ProxySQL database (query failed). Exiting."
+          log_if_success $LINENO $? "slave server ${ws_address} set to OFFLINE_SOFT status in ProxySQL (slave is too far behind:$seconds_behind). (io:$slave_io_running sql:$slave_sql_running)"
+          echo "1" > ${reload_check_file}
+        else
+          log $LINENO "slave server (${node_id}) current status '$ws_status' in ProxySQL. (io:$slave_io_running sql:$slave_sql_running)"
+        fi
+      else
+        #
+        # The slave is replicating and is caught up (relatively)
+        # So it is ok for READs
+        #
+        if [ "$ws_status" != "ONLINE" ];then
+          proxysql_exec $LINENO -Ns "UPDATE mysql_servers SET status = 'ONLINE' WHERE hostgroup_id=$HOSTGROUP_SLAVEREADER_ID AND hostname='$ws_ip' and port=$ws_port;"
+          check_cmd_and_exit $LINENO $? "Cannot update Galera Cluster node $ws_address in ProxySQL (query failed). Exiting."
+          log_if_success $LINENO $? "slave server $HOSTGROUP_SLAVEREADER_ID:${ws_address} set to ONLINE status in ProxySQL. (io:$slave_io_running sql:$slave_sql_running)"
+          if [[ $ws_hg_id -ne $HOSTGROUP_SLAVEREADER_ID ]]; then
+            log $LINENO "slave server (${node_id}) current status '$ws_status' in ProxySQL. (io:$slave_io_running sql:$slave_sql_running)"
+          fi
+          echo "1" > ${reload_check_file}
+        else
+          log $LINENO "slave server (${node_id}) current status '$ws_status' in ProxySQL. (io:$slave_io_running sql:$slave_sql_running)"
+        fi
+      fi
+    else
+      #
+      # Note: if slave_sql_running is YES and slave_io_running is NO
+      # This may indicate that the cluster is down, so we may move a
+      # slave to the WRITE hostgroup (but that is not done here).
+      # So leave it ONLINE in this state.
+      #
+      if [[ $ws_status == "ONLINE" ]]; then
+        log $LINENO "slave server ${ws_address} status:'${ws_status}' maintained in case the cluster is down. (io:$slave_io_running sql:$slave_sql_running)"
+      else
+        proxysql_exec $LINENO -Ns "UPDATE mysql_servers SET status = 'OFFLINE_SOFT' WHERE hostname='$ws_ip' and port=$ws_port;"
+        check_cmd_and_exit $LINENO $? "Unable to update mysql_servers (query failed). Exiting."
+        log $LINENO "slave server ${ws_address} status:'OFFLINE_SOFT'. (io:$slave_io_running sql:$slave_sql_running)"
+        echo "1" > ${reload_check_file}
+      fi
+    fi
+  fi
+  debug $LINENO "END set_slave_status"
+}
+
+
+function update_readers() {
+  local reload_check_file=$1
+  if [[ $WRITER_IS_READER != "ondemand" ]]; then
+    reader_proxysql_query="SELECT hostgroup_id,
+                                  hostname,
+                                  port,
+                                  status,
+                                  comment,
+                                  'NULL'
+                            FROM mysql_servers
+                            WHERE hostgroup_id IN ($HOSTGROUP_READER_ID)
+                              AND status <> 'OFFLINE_HARD'
+                            ORDER BY weight DESC, hostname, port"
+  elif [[ $WRITER_IS_READER == "ondemand" ]]; then
+    # We will not try to change reader state of nodes that are writer ONLINE,
+    # so what we do is we ORDER BY writer.status ASC because by accident ONLINE
+    # is last in the line
+    reader_proxysql_query="SELECT reader.hostgroup_id,
+           reader.hostname,
+           reader.port,
+           reader.status,
+           reader.comment,
+           writer.status
+    FROM mysql_servers as reader
+    LEFT JOIN mysql_servers as writer
+      ON writer.hostgroup_id = $HOSTGROUP_WRITER_ID
+      AND writer.hostname = reader.hostname
+      AND writer.port = reader.port
+    WHERE reader.hostgroup_id = $HOSTGROUP_READER_ID
+    ORDER BY writer.status ASC,
+             reader.weight DESC,
+             reader.hostname,
+             reader.port"
+  fi
+
+  # This is the count of nodes that have readers with ONLINE status
+  # and writers that are not ONLINE (either no entry or !ONLINE)
+  local online_readonly_nodes_found=0
+  local query_result
+
+  query_result=$(proxysql_exec $LINENO -Ns "$reader_proxysql_query")
+  check_cmd_and_exit $LINENO $? "Unable to obtain list of nodes from ProxySQL (query failed). Exiting."
+
+  if [[ $query_result =~ [[:space:]]SLAVEREAD[[:space:]] ]]; then
+    HAVE_SLAVEREADERS=1
+  fi
+
+  while read line; do
+    if [[ -z $line ]]; then
+      continue
+    fi
+
+    hostgroup=$(echo "$line" | cut -f1)
+    server=$(echo "$line" | cut -f2)
+    port=$(echo "$line" | cut -f3)
+    stat=$(echo "$line" | cut -f4)
+    comment=$(echo "$line" | cut -f5)
+    writer_stat=$(echo "$line" | cut -f6)
+
+    if [[ $comment == "SLAVEREAD" ]]; then
+      set_slave_status "${reload_check_file}" $hostgroup $server $port $stat
+      continue
+    fi
+    if [[ $stat == "OFFLINE_HARD" ]]; then
+      continue
+    fi
+
+    local wsrep_status
+    local pxc_main_mode
+    local result
+    local address
+    address=$(combine_ip_port_into_address "$server" "$port")
+    result=$(mysql_exec $LINENO "$server" "$port" -Nns "SHOW STATUS LIKE 'wsrep_local_state'; SHOW VARIABLES LIKE 'pxc_maint_mode';" 2>>${DEBUG_ERR_FILE})
+    wsrep_status=$(echo "$result" | grep "wsrep_local_state" | awk '{ print $2 }')
+    pxc_main_mode=$(echo "$result" | grep "pxc_maint_mode" | awk '{ print $2 }')
+
+    if [[ -z $wsrep_status ]]; then
+      wsrep_status="<unknown:query failed>"
+    fi
+
+    # For PXC 5.6 there is no pxc_maint_mode, so assume DISABLED
+    if [[ -z $pxc_main_mode ]]; then
+      pxc_main_mode="DISABLED"
+    fi
+
+    log "" "--> Checking READ server $hostgroup:$address, current status $stat, wsrep_local_state $wsrep_status"
+
+    if [[ $WRITER_IS_READER == "ondemand" && $writer_stat == "ONLINE" ]] ; then
+      if [ $online_readonly_nodes_found -eq 0 ] ; then
+        # WSREP:ok  PXC:ok  STATUS:online
+        if [ "${wsrep_status}" = "4" -a "$stat" == "ONLINE" -a "${pxc_main_mode}" == "DISABLED" ] ; then
+          log "" "server $hostgroup:$address is already ONLINE, is also write node in ONLINE state, not enough non-ONLINE readers found"
+        fi
+
+        # WSREP:ok  PXC:ok  STATUS:not online
+        if [ "${wsrep_status}" = "4" -a "$stat" != "ONLINE"  -a "${pxc_main_mode}" == "DISABLED" ] ; then
+          #
+          # Enable the first one found as ONLINE
+          # (when writer-is-reader=ondemand)
+          #
+          change_server_status $LINENO $HOSTGROUP_READER_ID "$server" "$port" "ONLINE" ""\
+                               "marking ONLINE write node as read ONLINE state, not enough non-ONLINE readers found"
+          echo "1" > ${reload_check_file}
+        fi
+      else
+        # WSREP:ok  PXC:ok  STATUS:online
+        if [ "${wsrep_status}" = "4" -a "$stat" == "ONLINE" -a "${pxc_main_mode}" == "DISABLED" ] ; then
+          # Else disable the other READ nodes
+          change_server_status $LINENO $HOSTGROUP_READER_ID "$server" "$port" "OFFLINE_SOFT" ""\
+                               "making ONLINE writer node as read OFFLINE_SOFT as well because writers should not be readers"
+          echo "1" > ${reload_check_file}
+        fi
+        # WSREP:ok  PXC:ok  STATUS:not online
+        if [ "${wsrep_status}" = "4" -a "$stat" != "ONLINE" -a "${pxc_main_mode}" == "DISABLED" ] ; then
+          log "" "server $hostgroup:$address is $stat, keeping node as $stat,as it is an ONLINE writer and we prefer not to have writers as readers (writer-is-reader:ondemand)"
+        fi
+      fi
+    else
+      # WSREP:ok  PXC:ok  STATUS:online
+      if [ "${wsrep_status}" = "4" -a "$stat" == "ONLINE" -a "${pxc_main_mode}" == "DISABLED" ] ; then
+        log "" "server $hostgroup:$address is already ONLINE"
+        online_readonly_nodes_found=$(( $online_readonly_nodes_found + 1 ))
+      # WSREP:ok  PXC:not ok  STATUS:not online
+      elif [ "${wsrep_status}" = "4" -a "$stat" != "ONLINE" -a "${pxc_main_mode}" != "DISABLED" ] ; then
+        log "" "server $hostgroup:$address is $stat"
+      fi
+
+      # WSREP status OK, but node is not marked ONLINE
+      # WSREP:ok  PXC:ok  STATUS:not online
+      if [ "${wsrep_status}" = "4" -a "$stat" != "ONLINE" -a "${pxc_main_mode}" == "DISABLED" ] ; then
+        change_server_status $LINENO $HOSTGROUP_READER_ID "$server" "$port" "ONLINE" ""\
+                              "changed state"
+        echo "1" > ${reload_check_file}
+        online_readonly_nodes_found=$(( $online_readonly_nodes_found + 1 ))
+      fi
+    fi
+
+    # WSREP status is not ok, but the node is marked online, we should put it offline
+    # WSREP:not ok  STATUS:online
+    if [ "${wsrep_status}" != "4" -a "$stat" = "ONLINE" ]; then
+      change_server_status $LINENO $HOSTGROUP_READER_ID "$server" "$port" "OFFLINE_SOFT" ""\
+                           "WSREP status is ${wsrep_status} which is not ok"
+      echo "1" > ${reload_check_file}
+    # PXC:not ok  STATUS:online
+    elif [ "${pxc_main_mode}" != "DISABLED" -a "$stat" = "ONLINE" ];then
+      change_server_status $LINENO $HOSTGROUP_READER_ID "$server" "$port" "OFFLINE_SOFT" ""\
+                           "pxc_maint_mode is $pxc_main_mode" 2>>${ERR_FILE}
+      echo "1" > ${reload_check_file}
+    # WSREP:not ok  STATUS:offline soft
+    elif [ "${wsrep_status}" != "4" -a "$stat" = "OFFLINE_SOFT" ]; then
+      log "" "server $hostgroup:$address is already OFFLINE_SOFT, WSREP status is ${wsrep_status} which is not ok"
+    fi
+  done< <(printf "$query_result\n")
+}
+
+
+# Looks specifically for nodes that are in the DONOR/DESYNCED(2) state
+#
+# Globals:
+#   NUMBER_WRITERS
+#   HOSTGROUP_WRITER_ID
+#
+# Arguments:
+#   None
+#
+function search_for_desynced_writers() {
+  local reload_check_file=$1
+  local cnt=0
+  local sort_order="ASC"
+
+  # We want the writers to come before the readers
+  if [[ $HOSTGROUP_WRITER_ID -gt $HOSTGROUP_READER_ID ]]; then
+    sort_order="DESC"
+  fi
+
+  local writer_query="SELECT hostgroup_id, hostname, port, status
+                      FROM mysql_servers
+                      WHERE hostgroup_id IN ($HOSTGROUP_WRITER_ID,$HOSTGROUP_READER_ID)
+                        AND status <> 'OFFLINE_HARD'
+                        AND comment <> 'SLAVEREAD'
+                      ORDER BY hostgroup_id ${sort_order}"
+
+  query_result=$(proxysql_exec $LINENO -Ns "$writer_query")
+  check_cmd_and_exit $LINENO $? "Could not get the list of nodes (query failed). Exiting."
+
+  while read line; do
+    if [[ -z $line ]]; then
+      continue
+    fi
+
+    hostgroup=$(echo "$line" | cut -f1)
+    server=$(echo "$line" | cut -f2)
+    port=$(echo "$line" | cut -f3)
+    stat=$(echo "$line" | cut -f4)
+
+    safety_cnt=0
+
+    while [ ${cnt} -lt $NUMBER_WRITERS -a ${safety_cnt} -lt 5 ]
+      do
+        local wsrep_status
+        local pxc_main_mode
+        local result
+        local address
+        address=$(combine_ip_port_into_address "$server" "$port")
+        result=$(mysql_exec $LINENO "$server" "$port" -Nns "SHOW STATUS LIKE 'wsrep_local_state'; SHOW VARIABLES LIKE 'pxc_maint_mode';" 2>>${DEBUG_ERR_FILE})
+        wsrep_status=$(echo "$result" | grep "wsrep_local_state" | awk '{ print $2 }')
+        pxc_main_mode=$(echo "$result" | grep "pxc_maint_mode" | awk '{ print $2 }')
+
+        if [[ -z $wsrep_status ]]; then
+          wsrep_status="<unknown:query failed>"
+        fi
+
+        # PXC 5.6 does not have this, so default to DISABLED
+        if [[ -z $pxc_main_mode ]]; then
+          pxc_main_mode="DISABLED"
+        fi
+
+        # Nodes in maintenance are not allowed
+        if [[ $pxc_main_mode != "DISABLED" ]]; then
+          log "" "Skipping $hostgroup:$address node is in pxc_maint_mode:$pxc_main_mode"
+          break
+        fi
+
+        log "" "Checking $hostgroup:$address for node in DONOR state, status $stat , wsrep_local_state $wsrep_status"
+        if [ "${wsrep_status}" = "2" -a "$stat" != "ONLINE" ]; then
+          # if we are on Donor/Desync and not online in mysql_servers -> proceed
+
+          # If we do not have a writer row, we have to add it
+          local writer_count=$(proxysql_exec $LINENO -Ns "SELECT count(*) FROM mysql_servers WHERE hostgroup_id=$HOSTGROUP_WRITER_ID AND hostname='$server' AND port=$port")
+          check_cmd_and_exit $LINENO $? "Could not get writer count (query failed). Exiting."
+
+          if [[ $writer_count -eq 0 ]]; then
+            proxysql_exec $LINENO -Ns \
+              "INSERT INTO mysql_servers (hostname,hostgroup_id,port,weight,comment,max_connections)
+                  VALUES ('$server',$HOSTGROUP_WRITER_ID,$port,1000000,'WRITE',$MAX_CONNECTIONS);"
+            check_cmd_and_exit $LINENO $? "Could not add writer node (query failed). Exiting."
+            log $LINENO "Adding server $HOSTGROUP_WRITER_ID:$address with status ONLINE. Reason: WSREP status is DESYNC/DONOR, as this is the only node we will put this one online"
+          else
+            change_server_status $LINENO $HOSTGROUP_WRITER_ID "$server" "$port" "ONLINE" ""\
+                                 "WSREP status is DESYNC/DONOR, as this is the only node we will put this one online"
+          fi
+          cnt=$(( $cnt + 1 ))
+
+          local proxy_runtime_status
+          proxy_runtime_status=$(proxysql_exec $LINENO -Ns "SELECT status FROM runtime_mysql_servers WHERE hostname='${server}' AND port='${port}' AND hostgroup_id='${hostgroup}'")
+          check_cmd_and_exit $LINENO $? "Could not get writer node status (query failed). Exiting."
+          if [ "${proxy_runtime_status}" != "ONLINE" ]; then
+            # if we are not online in runtime_mysql_servers, proceed to change
+            # the server status and reload mysql_servers
+            echo "1" > ${reload_check_file}
+          fi
+
+          # TODO: kennt, this doesn't work. We will go through the upper
+          # loop and move the writer to OFFLINE_SOFT (since it's desynced).
+          # We would have to detect that state (then we could move it to 0).
+          # Or we could only force the update if the tables are the same.
+          # (But only for the descyned node case).
+          # Maybe we can compare the table to itself, no updates needed if it
+          # hasn't changed.
+
+          # Note: If the node in the runtime is already ONLINE, then we don't
+          # need to change the reload_check_file to 1
+          # So if there was no change to the state, we will skip uploading
+          # the in-memory tables and nothing will change.  (This only makes
+          # sense if all the nodes are already down).  If any of the nodes are
+          # up, this node will be moved to OFFLINE_SOFT since it is not in
+          # the SYNCED(4) state.
+        fi
+        safety_cnt=$(( $safety_cnt + 1 ))
+    done
+  done< <(printf "$query_result\n")
+}
+
+
+#
+#
+# Globals:
+#   HOSTGROUP_READER_ID
+#
+# Arguments:
+#   None
+#
+function search_for_desynced_readers() {
+  local reload_check_file=$1
+  local cnt=0
+
+  query_result=$(proxysql_exec $LINENO -Ns "SELECT hostgroup_id, hostname, port, status FROM mysql_servers WHERE hostgroup_id IN ($HOSTGROUP_READER_ID) AND status <> 'OFFLINE_HARD' AND comment <> 'SLAVEREAD'")
+  check_cmd_and_exit $LINENO $? "Could not get the list of nodes (query failed). Exiting."
+
+  while read line; do
+    if [[ -z $line ]]; then
+      continue
+    fi
+
+    hostgroup=$(echo "$line" | cut -f1)
+    server=$(echo "$line" | cut -f2)
+    port=$(echo "$line" | cut -f3)
+    stat=$(echo "$line" | cut -f4)
+
+    local safety_cnt=0
+    while [[ ${cnt} -eq 0 && ${safety_cnt} -lt 5 ]]
+      do
+        local wsrep_status
+        local pxc_main_mode
+        local result
+        local address
+        address=$(combine_ip_port_into_address "$server" "$port")
+        result=$(mysql_exec $LINENO "$server" "$port" -Nns "SHOW STATUS LIKE 'wsrep_local_state'; SHOW VARIABLES LIKE 'pxc_maint_mode';" 2>>${DEBUG_ERR_FILE})
+        wsrep_status=$(echo "$result" | grep "wsrep_local_state" | awk '{ print $2 }')
+        pxc_main_mode=$(echo "$result" | grep "pxc_maint_mode" | awk '{ print $2 }')
+
+        if [[ -z $wsrep_status ]]; then
+          wsrep_status="<unknown:query failed>"
+        fi
+
+        # PXC 5.6 does not have this, so default to DISABLED
+        if [[ -z $pxc_main_mode ]]; then
+          pxc_main_mode="DISABLED"
+        fi
+
+        # Nodes in maintenance are not allowed
+        if [[ $pxc_main_mode != "DISABLED" ]]; then
+          log "" "Skipping $hostgroup:$address node is in pxc_maint_mode:$pxc_main_mode"
+          break
+        fi
+
+        log "" "Checking $hostgroup:$address for node in DONOR state, status $stat , wsrep_local_state $wsrep_status"
+        if [ "${wsrep_status}" = "2" -a "$stat" != "ONLINE" ];then
+          # if we are on Donor/Desync an not online in mysql_servers -> proceed
+          change_server_status $LINENO $HOSTGROUP_READER_ID "$server" "$port" "ONLINE" ""\
+                               "WSREP status is DESYNC/DONOR, as this is the only node we will put this one online"
+          cnt=$(( $cnt + 1 ))
+
+          local proxy_runtime_status
+          proxy_runtime_status=$(proxysql_exec $LINENO -Ns "SELECT status FROM runtime_mysql_servers WHERE hostname='${server}' AND port='${port}' AND hostgroup_id='${hostgroup}'")
+          check_cmd_and_exit $LINENO $? "Could not get the runtime server status (query failed). Exiting."
+          if [ "${proxy_runtime_status}" != "ONLINE" ]; then
+            # if we are not online in runtime_mysql_servers,
+            # proceed to change the server status and reload mysql_servers
+            echo "1" > ${reload_check_file}
+          fi
+          # Note: If the node in the runtime is already ONLINE, then we don't
+          # need to change the reload_check_file to 1
+          # So if there was no change to the state, we will skip uploading
+          # the in-memory tables and nothing will change.  (This only makes
+          # sense if all the nodes are already down).  If any of the nodes are
+          # up, this node will be moved to OFFLINE_SOFT since it is not in
+          # the SYNCED(4) state.
+          break 2
+        fi
+        safety_cnt=$(( $safety_cnt + 1 ))
+    done
+  done< <(printf "$query_result\n")
+}
+
+
+# Saves the reload_check_file value and resets it to 0
+#
+# Globals:
+#   None
+#
+# Arguments:
+#   1: path to the reload_check_file
+#
+function save_reload_check() {
+  local reload_check_file=$1
+  local save_state
+
+  save_state=$(cat "${reload_check_file}")
+  echo 0 > "${reload_check_file}"
+  echo "${save_state}"
+}
+
+# Resets the restore_reload_check value if needed
+# If nothing has changed, restores the value
+#
+# Globals:
+#   None
+#
+# Arguments:
+#   1: Path to the reload_check_file
+#   2: The saved (previous) value
+#
+# Returns:
+#   Returns 0 (success) if something changed
+#   Else returns 1
+#
+function restore_reload_check() {
+  local reload_check_file=$1
+  local save_state=$2
+  # Default return value is failure (non-zero)
+  local rc=1
+
+  if [[ $(cat ${reload_check_file}) -ne 0 ]] ; then
+    save_state=1
+
+    # A success means that something changed
+    rc=0
+  fi
+  echo "${save_state}" > "${reload_check_file}"
+  return $rc
+}
+
+
+# Adds slaves to the write hostgroup if needed
+#
+# Globals:
+#   HOSTGROUP_WRITER_ID
+#   HOSTGROUP_SLAVEREADER_ID
+#
+# Arguments:
+#   None
+#
+function add_slave_to_write_hostgroup() {
+  # General outline:
+  #   Delete all non-ONLINE slaves to the read hostgroup
+  #   Check for an ONLINE slave in the write hostgroup
+  #   If there is none, add a random slave to the write hostgroup
+  #   If cannot find an ONLINE, look for OFFLINE_SOFT (emergency only)
+  local reload_check_file=$1
+  local -i online_count=0
+  local -i offline_count=0
+  local query_result=""
+
+  query_result=$(proxysql_exec $LINENO -Ns "SELECT status FROM mysql_servers
+                                            WHERE hostgroup_id=$HOSTGROUP_WRITER_ID
+                                            AND comment = 'SLAVEREAD'")
+  check_cmd_and_exit $LINENO $? "Could not get the list of slave writers (query failed). Exiting."
+
+  # Count # of online vs offline slave nodes
+  local status
+  while read line; do
+    if [[ -z $line ]]; then
+      continue
+    fi
+
+    status=$(echo $line | awk '{ print $1 }')
+    if [[ $status = 'ONLINE' ]]; then
+      online_count+=1
+    else
+      offline_count+=1
+    fi
+  done< <(printf "$query_result\n")
+
+  # Remove any nodes that have been moved OFFLINE
+  if [[ $offline_count -gt 0 ]]; then
+    proxysql_exec $LINENO -Ns "DELETE FROM mysql_servers
+                                WHERE hostgroup_id=$HOSTGROUP_WRITER_ID
+                                  AND status != 'ONLINE'
+                                  AND comment = 'SLAVEREAD'"
+    check_cmd_and_exit $LINENO $? "Failed to remove all non-ONLINE slaves acting as writers (query failed). Exiting."
+    log_if_success $LINENO $? "Removed all non-ONLINE slaves acting as writers"
+    echo 1 > "${reload_check_file}"
+  fi
+
+  # If there is an active slave writer node, no need to add another node
+  if [[ $online_count -eq 1 ]]; then
+    log $LINENO "Async-slave already ONLINE"
+    return
+  fi
+
+  if [[ $online_count -eq 0 ]]; then
+    # There are no ONLINE slaves, so add a random one from the
+    # list of ONLINE reader slaves
+    local next_host  host  port
+    next_host=$(proxysql_exec $LINENO -Ns "select hostname,port FROM mysql_servers
+                                           WHERE status='ONLINE'
+                                           AND comment = 'SLAVEREAD'
+                                           AND hostgroup_id='$HOSTGROUP_SLAVEREADER_ID'
+                                           ORDER BY random() LIMIT 1")
+    check_cmd_and_exit $LINENO $? "Could not get info for a slave node (query failed). Exiting."
+
+    # Emergency situation, if there are no ONLINE hosts,
+    # look for OFFLINE_SOFT hosts
+    if [[ -z $next_host ]]; then
+      next_host=$(proxysql_exec $LINENO -Ns "select hostname,port FROM mysql_servers
+                                           WHERE status='OFFLINE_SOFT'
+                                           AND comment = 'SLAVEREAD'
+                                           AND hostgroup_id='$HOSTGROUP_SLAVEREADER_ID'
+                                           ORDER BY random() LIMIT 1")
+      check_cmd_and_exit $LINENO $? "Could not get info for a slave node (query failed). Exiting."
+      if [[ -n $next_host ]]; then
+        local host=$(echo "$next_host" | awk '{ print $1 }')
+        local port=$(echo "$next_host" | awk '{ print $2 }')
+        local address=$(combine_ip_port_into_address "$host" "$port")
+
+        # Found a node, update the READER to ONLINE
+        proxysql_exec $LINENO -Ns "UPDATE mysql_servers
+                                    SET status='ONLINE'
+                                    WHERE hostgroup_id=$HOSTGROUP_SLAVEREADER_ID
+                                      AND hostname='$host'
+                                      AND port=$port"
+        check_cmd_and_exit $LINENO $? "Could not update the info for a slave node (query failed). Exiting."
+        log_if_success $LINENO $? "slave server ${HOSTGROUP_SLAVEREADER_ID}:$address set to ONLINE status in ProxySQL."
+        echo 1 > "${reload_check_file}"
+      fi
+    fi
+
+    if [[ -n $next_host ]]; then
+      local host=$(echo "$next_host" | awk '{ print $1 }')
+      local port=$(echo "$next_host" | awk '{ print $2 }')
+      local address=$(combine_ip_port_into_address "$host" "$port")
+
+      proxysql_exec $LINENO -Ns "INSERT INTO mysql_servers
+            (hostname,hostgroup_id,port,weight,status,comment,max_connections)
+            VALUES ('$host',$HOSTGROUP_WRITER_ID,$port,1000000,'ONLINE','SLAVEREAD',$MAX_CONNECTIONS);"
+      check_cmd_and_exit $LINENO $? "Cannot add Galera Cluster node $address to ProxySQL (query failed). Exiting."
+      log_if_success $LINENO $? "slave server ${HOSTGROUP_WRITER_ID}:$address set to ONLINE status in ProxySQL."
+      echo 1 > "${reload_check_file}"
+    else
+      log $LINENO "Could not find any slave readers to promote to a writer"
+    fi
+  fi
+}
+
+# Removes slaves from the write hostgroup
+# (Actually moves the writers to OFFLINE_SOFT)
+#
+# Globals:
+#   HOSTGROUP_WRITER_ID
+#
+# Arguments:
+#   None
+#
+function remove_slave_from_write_hostgroup() {
+  local reload_check_file=$1
+  log $LINENO "Removing async-slaves from the writegroup"
+  # Move all writer slaves to OFFLINE_SOFT
+  proxysql_exec $LINENO -Ns "UPDATE mysql_servers
+                             SET status = 'OFFLINE_SOFT'
+                             WHERE hostgroup_id = $HOSTGROUP_WRITER_ID
+                             AND comment = 'SLAVEREAD'"
+  check_cmd_and_exit $LINENO $? "Failed to move slave writers to OFFLINE_SOFT (query failed). Exiting."
+  log_if_success $LINENO $? "Moved slave writers to OFFLINE_SOFT"
+   echo 1 > "${reload_check_file}"
+}
+
+
+#
+# Globals:
+#   TIMEOUT
+#   MYSQL_USERNAME MYSQL_PASSWORD MYSQL_HOSTNAME MYSQL_PORT
+#   PROXYSQL_DATADIR
+#   HOSTGROUP_READER_ID HOSTGROUP_WRITER_ID
+#   CONFIG_FILE
+#   NUMBER_WRITERS
+#
+function main() {
+  local cluster_name=""
+  local mysql_credentials
+  local scheduler_id
+
+  # If this call fails, exit out.  We can't do anything without the monitor
+  # credentials.  (Or if we can't connect to proxysql).
+  mysql_credentials=$(proxysql_exec $LINENO -Ns "SELECT variable_value FROM global_variables WHERE variable_name IN ('mysql-monitor_username','mysql-monitor_password') ORDER BY variable_name DESC" "hide_output")
+  check_cmd_and_exit $LINENO $? "Unable to obtain MySQL credentials from ProxySQL (query failed). Exiting."
+
+  MYSQL_USERNAME=$(echo $mysql_credentials | awk '{print $1}')
+  MYSQL_PASSWORD=$(echo $mysql_credentials | awk '{print $2}')
+
+  # Search for the scheduler id that corresponds to our write hostgroup
+  scheduler_id=$(proxysql_exec $LINENO -Ns "SELECT id FROM scheduler where arg1 LIKE '%--write-hg=$HOSTGROUP_WRITER_ID %' OR arg1 LIKE '%-w $HOSTGROUP_WRITER_ID %'")
+  check_cmd_and_exit $LINENO $? "Could not retreive scheduler row (query failed). Exiting."
+  if [[ -z $scheduler_id ]]; then
+    error $LINENO "Cannot find the scheduler row with write hostgroup : $HOSTGROUP_WRITER_ID"
+  else
+    cluster_name=$(proxysql_exec $LINENO -Ns "SELECT comment FROM scheduler WHERE id=${scheduler_id}" 2>>$ERR_FILE)
+    check_cmd_and_exit $LINENO $? "Cannot get the cluster name from ProxySQL (query failed). Exiting."
+  fi
+
+  if [[ -z $cluster_name ]]; then
+    #
+    # Could not find the cluster name from the scheduler,
+    # contact the cluster directly
+    #
+    local available_host=""
+    local server port
+
+    # Find a PXC host that's we can connect to
+    available_host=$(find_online_cluster_host)
+    check_cmd_and_exit $LINENO $? "Cannot get the list of nodes in the cluster from ProxySQL (query failed). Exiting"
+
+    if [[ -z $available_host ]]; then
+      error $LINENO "Cannot contact a PXC host in hostgroups($HOSTGROUP_WRITER_ID, $HOSTGROUP_READER_ID)"
+    else
+      server=$(echo $available_host | awk '{print $1}')
+      port=$(echo $available_host | awk '{print $2}')
+
+      cluster_name=$(mysql_exec $LINENO "$server" "$port" -Nn \
+          "SELECT @@wsrep_cluster_name" 2>>${ERR_FILE} | tail -1)
+    fi
+
+    if [[ ! -z $cluster_name ]]; then
+      #
+      # We've found the cluster name so update the scheduler args
+      #
+      if [[ -z $scheduler_id ]]; then
+        warning "$LINENO" "Cannot update scheduler due to missing scheduler_id"
+      else
+        local arg1 my_path
+        log "$LINENO" "Updating scheduler for cluster:${cluster_name} write_hg:$HOSTGROUP_WRITER_ID"
+        arg1=$(proxysql_exec $LINENO -Ns "SELECT arg1 from scheduler where id=${scheduler_id}")
+        check_cmd_and_exit $LINENO $? "Could not get arg1 from scheduler from (query failed)" "Please check ProxySQL credentials and status."
+        if [[ -z $arg1 ]]; then
+          warning $LINENO "Cannot update scheduler due to missing arguments (arg1 is empty)"
+        else
+          my_path=$(echo ${PROXYSQL_DATADIR}/${cluster_name}_proxysql_galera_check.log | sed  's#\/#\\\/#g')
+          arg1=$(echo $arg1 | sed "s/--log=.*/--log=$my_path/g")
+
+          proxysql_exec $LINENO -Ns "UPDATE scheduler set comment='$cluster_name',arg1='$arg1' where id=${scheduler_id};load scheduler to runtime;save scheduler to disk"
+          check_cmd_and_exit $LINENO $? "Could not update scheduler from (query failed). Exiting."
+        fi
+      fi
+    fi
+  fi
+
+  # Check to see if there are other proxysql_galera_checkers running
+  # Before we can check, we need the cluster name
+  check_is_galera_checker_running "$cluster_name"
+
+  local mode=$MODE
+  if [[ ! -z $P_MODE ]] ; then
+    mode=$P_MODE
+  else
+    local proxysql_mode_file
+    if [[ -z $cluster_name ]]; then
+      proxysql_mode_file="${PROXYSQL_DATADIR}/mode"
+    else
+      proxysql_mode_file="${PROXYSQL_DATADIR}/${cluster_name}_mode"
+    fi
+
+    if [[ ! -f ${proxysql_mode_file} ]]; then
+      local mode_check
+      mode_check=$(proxysql_exec $LINENO -Ns "SELECT comment from mysql_servers where comment='WRITE' and hostgroup_id in ($HOSTGROUP_WRITER_ID, $HOSTGROUP_READER_ID)")
+      if [[ "$mode_check" == "WRITE" ]]; then
+        echo "singlewrite" > ${proxysql_mode_file}
+      else
+        echo "loadbal" > ${proxysql_mode_file}
+      fi
+    fi
+
+    if [[ -r $proxysql_mode_file ]]; then
+      mode=$(cat ${proxysql_mode_file})
+    fi
+  fi
+  MODE=$mode
+
+  # Running proxysql_node_monitor script.
+  # First try the same directory as this script
+  local monitor_dir
+  monitor_dir=$(cd $(dirname $0) && pwd)
+
+  if [[ ! -f $monitor_dir/proxysql_node_monitor ]]; then
+    # Cannot find it in same directory, try default location
+    monitor_dir="/usr/bin"
+    if [[ ! -f $monitor_dir/proxysql_node_monitor ]]; then
+      log "" "ERROR! Could not find proxysql_node_monitor. Terminating"
+      exit 1
+    fi
+  fi
+
+  # If the reload_check_file contains 1, then there was a change
+  # made to the state and we need to upload the MYSQL SERVERS to runtime.
+  # This is used because changes are made to the MYSQL SERVERS table
+  # in other scripts/subshells.
+  local reload_check_file
+  if [[ -z $cluster_name ]]; then
+    reload_check_file="${PROXYSQL_DATADIR}/reload"
+  else
+    reload_check_file="${PROXYSQL_DATADIR}/${cluster_name}_reload"
+  fi
+  echo "0" > ${reload_check_file}
+
+  # Run the monitor script
+  local proxysql_monitor_log
+  if [[ -n $NODE_MONITOR_LOG_FILE ]]; then
+    proxysql_monitor_log=$NODE_MONITOR_LOG_FILE
+  else
+    if [[ -z $cluster_name ]]; then
+      proxysql_monitor_log="${PROXYSQL_DATADIR}/proxysql_node_monitor.log"
+    else
+      proxysql_monitor_log="${PROXYSQL_DATADIR}/${cluster_name}_proxysql_node_monitor.log"
+    fi
+  fi
+
+  local more_monitor_options=""
+  if [[ $DEBUG -ne 0 ]]; then
+    more_monitor_options+=" --debug "
+  fi
+
+  # TODO: kennt, do we need to check the return code?  do we care?
+  $monitor_dir/proxysql_node_monitor --config-file=$CONFIG_FILE \
+                                     --write-hg=$HOSTGROUP_WRITER_ID \
+                                     --read-hg=$HOSTGROUP_READER_ID \
+                                     --mode=$mode \
+                                     --reload-check-file="$reload_check_file" \
+                                     --log-text="$LOG_TEXT" \
+                                     --max-connections="$MAX_CONNECTIONS" \
+                                     --log="$proxysql_monitor_log" $more_monitor_options
+
+  # print information prior to a run if ${ERR_FILE} is defined
+  log "" "###### proxysql_galera_checker.sh SUMMARY ######"
+  log "" "Hostgroup writers $HOSTGROUP_WRITER_ID"
+  log "" "Hostgroup readers $HOSTGROUP_READER_ID"
+  log "" "Number of writers $NUMBER_WRITERS"
+  log "" "Writers are readers $WRITER_IS_READER"
+  log "" "Log file          $ERR_FILE"
+  log "" "Mode              $MODE"
+  if [[ -n $P_PRIORITY ]]; then
+    log "" "Priority          $P_PRIORITY"
+  fi
+  if [[ -n $LOG_TEXT ]]; then
+    log "" "Extra notes       $LOG_TEXT"
+  fi
+
+  local number_readers_online=0
+  local number_writers_online=0
+  local save_reload_state=0
+
+
+  log "" "###### HANDLE WRITER NODES ######"
+  update_writers "${reload_check_file}"
+  number_writers_online=$(proxysql_exec $LINENO -Ns "SELECT count(*) FROM mysql_servers WHERE hostgroup_id=$HOSTGROUP_WRITER_ID AND status = 'ONLINE' AND comment <> 'SLAVEREAD'" 2>>${ERR_FILE})
+  check_cmd_and_exit $LINENO $? "Could not get node count (query failed). Exiting."
+
+  # Check to see if we need to add readers for any writer nodes
+  # (depends on the writer-is-reader setting)
+  number_readers_online=$(proxysql_exec $LINENO -Ns "SELECT count(*) FROM mysql_servers WHERE hostgroup_id IN ($HOSTGROUP_READER_ID) AND status = 'ONLINE' AND comment <> 'SLAVEREAD'")
+  check_cmd_and_exit $LINENO $? "Could not get node count (query failed). Exiting."
+
+  save_reload_state=$(save_reload_check "${reload_check_file}")
+
+  writer_is_reader_check "$reload_check_file" "$number_readers_online"
+
+  # Something changed
+  if restore_reload_check "${reload_check_file}" $save_reload_state; then
+    number_readers_online=$(proxysql_exec $LINENO -Ns "SELECT count(*) FROM mysql_servers WHERE hostgroup_id IN ($HOSTGROUP_READER_ID) AND status = 'ONLINE' AND comment <> 'SLAVEREAD'")
+    check_cmd_and_exit $LINENO $? "Could not get node count (query failed). Exiting."
+  fi
+
+
+  if [ ${HOSTGROUP_READER_ID} -ne -1 ]; then
+    log "" "###### HANDLE READER NODES ######"
+    save_reload_state=$(save_reload_check "${reload_check_file}")
+
+    update_readers "${reload_check_file}"
+
+    # Something changed
+    if restore_reload_check "${reload_check_file}" $save_reload_state; then
+      number_readers_online=$(proxysql_exec $LINENO -Ns "SELECT count(*) FROM mysql_servers WHERE hostgroup_id IN ($HOSTGROUP_READER_ID) AND status = 'ONLINE' AND comment <> 'SLAVEREAD'")
+      check_cmd_and_exit $LINENO $? "Could not get node count (query failed). Exiting."
+    fi
+  fi
+
+  if [[ $MODE != "loadbal" ]]; then
+    # If we have no writers, check if we can create one (just one)
+    if [[ $number_writers_online -eq 0 ]]; then
+      save_reload_state=$(save_reload_check "${reload_check_file}")
+
+      ensure_one_writer_node "${reload_check_file}"
+
+      # Something changed
+      if restore_reload_check "${reload_check_file}" $save_reload_state; then
+        # Check to see if we need to add readers for any writer nodes
+        # (depends on the writer-is-reader setting)
+        writer_is_reader_check "$reload_check_file" "$number_readers_online"
+        echo 1 > "${reload_check_file}"
+        number_readers_online=$(proxysql_exec $LINENO -Ns "SELECT count(*) FROM mysql_servers WHERE hostgroup_id IN ($HOSTGROUP_READER_ID) AND status = 'ONLINE' AND comment <> 'SLAVEREAD'")
+        check_cmd_and_exit $LINENO $? "Could not get node count (query failed). Exiting."
+        number_writers_online=$(proxysql_exec $LINENO -Ns "SELECT count(*) FROM mysql_servers WHERE hostgroup_id=$HOSTGROUP_WRITER_ID AND status = 'ONLINE' AND comment <> 'SLAVEREAD'" 2>>${ERR_FILE})
+        check_cmd_and_exit $LINENO $? "Could not get node count (query failed). Exiting."
+      fi
+    else
+      debug $LINENO "writer nodes found: $number_writers_online, no need to add more"
+    fi
+  fi
+
+
+  log "" "###### SUMMARY ######"
+  log "" "--> Number of writers that are 'ONLINE': ${number_writers_online} : hostgroup: ${HOSTGROUP_WRITER_ID}"
+  [[ ${HOSTGROUP_READER_ID} -ne -1 ]] && log "" "--> Number of readers that are 'ONLINE': ${number_readers_online} : hostgroup: ${HOSTGROUP_READER_ID}"
+
+
+  # We don't have any writers... alert, try to bring some online!
+  # This includes bringing a DONOR online
+  if [[ ${number_writers_online} -eq 0 ]]; then
+    log "" "###### TRYING TO FIX MISSING WRITERS ######"
+    log "" "No writers found, Trying to enable last available node of the cluster (in Donor/Desync state)"
+    save_reload_state=$(save_reload_check "${reload_check_file}")
+
+    search_for_desynced_writers "${reload_check_file}"
+
+    # Something changed
+    if restore_reload_check "${reload_check_file}" $save_reload_state; then
+      number_writers_online=$(proxysql_exec $LINENO -Ns "SELECT count(*) FROM mysql_servers WHERE hostgroup_id=$HOSTGROUP_WRITER_ID AND status = 'ONLINE' AND comment <> 'SLAVEREAD'" 2>>${ERR_FILE})
+      check_cmd_and_exit $LINENO $? "Could not get node count (query failed). Exiting."
+    fi
+  fi
+
+
+  # We don't have any readers... alert, try to bring some online!
+  if [[  ${HOSTGROUP_READER_ID} -ne -1 && ${number_readers_online} -eq 0 ]]; then
+    log "" "###### TRYING TO FIX MISSING READERS ######"
+    log "" "--> No readers found, Trying to enable last available node of the cluster (in Donor/Desync state) or pick the master"
+    save_reload_state=$(save_reload_check "${reload_check_file}")
+
+    search_for_desynced_readers "${reload_check_file}"
+
+    # Something changed
+    if restore_reload_check "${reload_check_file}" $save_reload_state; then
+      number_readers_online=$(proxysql_exec $LINENO -Ns "SELECT count(*) FROM mysql_servers WHERE hostgroup_id IN ($HOSTGROUP_READER_ID) AND status = 'ONLINE' AND comment <> 'SLAVEREAD'")
+      check_cmd_and_exit $LINENO $? "Could not get node count (query failed). Exiting."
+    fi
+  fi
+
+  # Check to see if we need to enable the slaves as readers/writers
+  if [[ $HAVE_SLAVEREADERS -eq 1 ]]; then
+    save_reload_state=$(save_reload_check "${reload_check_file}")
+
+    log $LINENO "###### ASYNC-SLAVE ACTIVITY ######"
+
+    # If use-slave-as-writer is set, then we do not allow slaves to
+    # be added, however we still call the remove slave writers
+    # in case the option was just changed.
+
+    if [[ $SLAVE_IS_WRITER == "yes" && $number_readers_online -eq 0 && $number_writers_online -eq 0 ]]; then
+      # No cluster nodes active, add a slave to the writer hostgroup
+      debug $LINENO "Nothing in the cluster, checking slavereaders"
+      add_slave_to_write_hostgroup "${reload_check_file}"
+    elif [[ $HAVE_SLAVEWRITERS -eq 1 ]]; then
+      # Active cluster nodes discovered, remove all write nodes
+      debug $LINENO "Found a cluster, removing slavereaders (acting as writers)"
+      remove_slave_from_write_hostgroup "${reload_check_file}"
+    fi
+
+    local query_result
+    local -i num_slave_writers=0
+    local -i num_slave_readers=0
+
+    query_result=$(proxysql_exec $LINENO -Ns "SELECT hostgroup_id FROM mysql_servers
+                                            WHERE hostgroup_id IN ($HOSTGROUP_WRITER_ID, $HOSTGROUP_READER_ID, $HOSTGROUP_SLAVEREADER_ID)
+                                            AND comment = 'SLAVEREAD'
+                                            AND status = 'ONLINE'")
+    check_cmd_and_exit $LINENO $? "Could not get the list of slave nodes (query failed). Exiting."
+    while read line; do
+      if [[ -z $line ]]; then
+        continue
+      fi
+
+      if [[ $line = "$HOSTGROUP_WRITER_ID" ]]; then
+        num_slave_writers+=1
+      elif [[ $line = "$HOSTGROUP_READER_ID" ]]; then
+        num_slave_readers+=1
+      fi
+    done< <(printf "$query_result\n")
+
+    log "" "--> Number of slave writers that are 'ONLINE': ${num_slave_writers} : hostgroup: ${HOSTGROUP_WRITER_ID}"
+    [[ ${HOSTGROUP_READER_ID} -ne -1 ]] && log "" "--> Number of slave readers that are 'ONLINE': ${num_slave_readers} : hostgroup: ${HOSTGROUP_READER_ID}"
+
+    restore_reload_check "${reload_check_file}" $save_reload_state
+  fi
+
+
+  if [[ $(cat ${reload_check_file}) -ne 0 ]] ; then
+      log "" "###### Loading mysql_servers config into runtime ######"
+      proxysql_exec $LINENO -Ns "LOAD MYSQL SERVERS TO RUNTIME;" 2>>${ERR_FILE}
+      check_cmd_and_exit $LINENO $? "Could not update the mysql_servers table in ProxySQL. Exiting."
+  else
+      log "" "###### Not loading mysql_servers, no change needed ######"
+  fi
+}
+
+
+#-------------------------------------------------------------------------------
+#
+# Step 4 : Begin script execution
+#
+
+#
+# In the initial version, ProxySQL has 5 slots for command-line arguments
+# and would send them separately (each enclosed in double quotes,
+# such as "arg1" "arg2" etc..).
+#
+# We now configure all parameters in the arg1 field (since we need more
+# than 5 arguments).  This means that we now receive all the arguments
+# in one parameter "arg1 arg2 arg3 arg4 arg5"
+#
+# This means that anytime this script is called with > 1 argument, we
+# assume that the old method is in use and we need to upgrade the script
+# to the new method (in upgrade_scheduler).
+#
+
+if [ "$#" -eq 1 ]; then
+  # Below set will reshuffle parameters.
+  # example arg1=" --one=1 --two=2" will result in:
+  # $1 = --one=1
+  # $2 = --two=2
+  #
+  # The eval is needed here to preserve whitespace in the arguments
+  eval set -- $1
+else
+  # Parse the arguments
+  declare param value
+
+  # We don't need all the options here, just the log/debug options
+  while [[ $# -gt 0 && "$1" != "" ]]; do
+      param=`echo $1 | awk -F= '{print $1}'`
+      value=`echo $1 | awk -F= '{print $2}'`
+
+      # Assume that all options start with a '-'
+      # otherwise treat as a positional parameter
+      if [[ ! $param =~ ^- ]]; then
+        continue
+      fi
+      case $param in
+        -h | --help)
+          usage
+          exit 0
+          ;;
+        --debug)
+          DEBUG=1
+          ;;
+        -v | --version)
+          echo "proxysql_galera_checker version $PROXYSQL_ADMIN_VERSION"
+          exit 0
+          ;;
+        --log)
+          if [[ -n $value ]]; then
+            ERR_FILE=$value
+          fi
+          ;;
+      esac
+      shift
+  done
+
+  echo "Old config detected...trying to upgrade More than one parameter"
+
+  if [[ $DEBUG -eq 1 ]]; then
+    # For now
+    if [[ -z $ERR_FILE && -t 1 ]]; then
+      ERR_FILE=/dev/stderr
+    fi
+  fi
+
+  upgrade_scheduler
+  exit 1
+fi
+
+trap cleanup_handler EXIT
+
+parse_args "$@"
+main
+
+exit 0

dev/provisioning/ansible/roles/proxysql/files/proxysql_galera_checker.sh

@@ -0,0 +1,321 @@
+#!/bin/bash
+## inspired by Percona clustercheck.sh
+# https://github.com/sysown/proxysql/blob/v2.x/tools/proxysql_galera_checker.sh
+
+# CHANGE THOSE
+PROXYSQL_USERNAME="admin"
+PROXYSQL_PASSWORD="admin"
+PROXYSQL_HOSTNAME="localhost"
+PROXYSQL_PORT="6032"
+#
+
+function usage()
+{
+  cat << EOF
+
+Usage: $0 <hostgroup_id write> [hostgroup_id read] [number writers] [writers are readers 0|1] [log_file]
+
+- HOSTGROUP WRITERS   (required)  (0..)   The hostgroup_id that contains nodes that will server 'writes'
+- HOSTGROUP READERS   (optional)  (0..)   The hostgroup_id that contains nodes that will server 'reads'
+- NUMBER WRITERS      (optional)  (0..)   Maximum number of write hostgroup_id node that can be marked ONLINE
+                                          When 0 (default), all nodes can be marked ONLINE
+- WRITERS ARE READERS (optional)  (0|1)   When 1 (default), ONLINE nodes in write hostgroup_id will prefer not
+                                          to be ONLINE in read hostgroup_id
+- LOG_FILE            (optional)  file    logfile where node state checks & changes are written to (verbose)
+
+
+Notes about the mysql_servers in ProxySQL:
+
+- WEIGHT           Hosts with a higher weight will be prefered to be put ONLINE
+- NODE STATUS      * Nodes that are in status OFFLINE_HARD will not be checked nor will their status be changed
+                   * SHUNNED nodes are not to be used with Galera based systems, they will be checked and status
+                     will be changed to either ONLINE or OFFLINE_SOFT.
+
+
+When no nodes were found to be in wsrep_local_state=4 (SYNCED) for either 
+read or write nodes, then the script will try 5 times for each node to try 
+to find nodes wsrep_local_state=4 (SYNCED) or wsrep_local_state=2 (DONOR/DESYNC)
+
+This is to avoid $0 to mark all nodes as OFFLINE_SOFT
+
+EOF
+}
+
+
+# DEFAULTS
+HOSTGROUP_WRITER_ID="${1}"
+HOSTGROUP_READER_ID="${2:--1}"
+NUMBER_WRITERS="${3:-0}"
+WRITER_IS_READER="${4:-1}"
+ERR_FILE="${5:-/dev/null}"
+RELOAD_CHECK_FILE="/var/lib/proxysql/reload"
+
+echo "0" > ${RELOAD_CHECK_FILE}
+
+if [ "$1" = '-h' -o "$1" = '--help'  -o -z "$1" ]
+then
+  usage
+  exit 0
+fi
+
+test $HOSTGROUP_WRITER_ID -ge 0 &> /dev/null
+if [ $? -ne 0 ]; then
+  echo "ERROR: writer hostgroup_id is not an integer"
+  usage
+  exit 1
+fi
+
+test $HOSTGROUP_READER_ID -ge -1 &> /dev/null
+if [ $? -ne 0 ]; then
+  echo "ERROR: reader hostgroup_id is not an integer"
+  usage
+  exit 1
+fi
+
+if [ $# -lt 1 -o $# -gt 5 ]; then
+  echo "ERROR: Invalid number of arguments"
+  usage
+  exit 1
+fi
+
+if [ $NUMBER_WRITERS -lt 0 ]; then
+  echo "ERROR: The number of writers should either be 0 to enable all possible nodes ONLINE"
+  echo "       or be larger than 0 to limit the number of writers"
+  usage
+  exit 1
+fi
+
+if [ $WRITER_IS_READER -ne 0 -a $WRITER_IS_READER -ne 1 ]; then
+  echo "ERROR: Writers are readers requires a boolean argument (0|1)"
+  usage
+  exit 1
+fi
+
+
+# print information prior to a run if ${ERR_FILE} is defined 
+echo "`date` ###### proxysql_galera_checker.sh SUMMARY ######" >> ${ERR_FILE}
+echo "`date` Hostgroup writers $HOSTGROUP_WRITER_ID" >> ${ERR_FILE}
+echo "`date` Hostgroup readers $HOSTGROUP_READER_ID" >> ${ERR_FILE}
+echo "`date` Number of writers $NUMBER_WRITERS" >> ${ERR_FILE}
+echo "`date` Writers are readers $WRITER_IS_READER" >> ${ERR_FILE}
+echo "`date` log file $ERR_FILE" >> ${ERR_FILE}
+
+#Timeout exists for instances where mysqld may be hung
+TIMEOUT=10
+
+PROXYSQL_CMDLINE="env MYSQL_PWD=$PROXYSQL_PASSWORD mysql -u$PROXYSQL_USERNAME -h $PROXYSQL_HOSTNAME -P $PROXYSQL_PORT --protocol=tcp -Nse"
+MYSQL_CREDENTIALS=$($PROXYSQL_CMDLINE "SELECT variable_value FROM global_variables WHERE variable_name IN ('mysql-monitor_username','mysql-monitor_password') ORDER BY variable_name DESC")
+MYSQL_USERNAME=$(echo $MYSQL_CREDENTIALS | awk '{print $1}')
+MYSQL_PASSWORD=$(echo $MYSQL_CREDENTIALS | awk '{print $2}')
+MYSQL_CMDLINE="env MYSQL_PWD=$MYSQL_PASSWORD timeout $TIMEOUT mysql -nNE -u$MYSQL_USERNAME"
+
+
+function change_server_status() {
+  echo "`date` Changing server $1:$2:$3 to status $4. Reason: $5" >> ${ERR_FILE}
+  $PROXYSQL_CMDLINE "UPDATE mysql_servers set status = '$4' WHERE hostgroup_id = $1 AND hostname = '$2' AND port = $3;" 2>> ${ERR_FILE}
+}
+
+
+echo "`date` ###### HANDLE WRITER NODES ######" >> ${ERR_FILE}
+NUMBER_WRITERS_ONLINE=0
+$PROXYSQL_CMDLINE "SELECT hostgroup_id, hostname, port, status FROM mysql_servers WHERE hostgroup_id IN ($HOSTGROUP_WRITER_ID) AND status <> 'OFFLINE_HARD' ORDER BY hostgroup_id, weight DESC, hostname, port" | while read hostgroup server port stat
+do
+  WSREP_STATUS=$($MYSQL_CMDLINE -h $server -P $port -e "SHOW STATUS LIKE 'wsrep_local_state'" 2>>${ERR_FILE}| tail -1 2>>${ERR_FILE})
+
+  echo "`date` --> Checking WRITE server $hostgroup:$server:$port, current status $stat, wsrep_local_state $WSREP_STATUS" >> ${ERR_FILE}
+
+  # we have to limit amount of writers, WSREP status OK, AND node is not marked ONLINE
+  if [ $NUMBER_WRITERS -gt 0 -a "${WSREP_STATUS}" = "4" -a "$stat" == "ONLINE" ] ; then
+      if [ $NUMBER_WRITERS_ONLINE -lt $NUMBER_WRITERS ]; then
+        NUMBER_WRITERS_ONLINE=$(( $NUMBER_WRITERS_ONLINE + 1 ))
+        echo "`date` server $hostgroup:$server:$port is already ONLINE: ${NUMBER_WRITERS_ONLINE} of ${NUMBER_WRITERS} write nodes" >> ${ERR_FILE}
+      else
+        NUMBER_WRITERS_ONLINE=$(( $NUMBER_WRITERS_ONLINE + 1 ))
+        change_server_status $HOSTGROUP_WRITER_ID "$server" $port "OFFLINE_SOFT" "max write nodes reached (${NUMBER_WRITERS})"
+        echo "1" > ${RELOAD_CHECK_FILE}
+
+      fi
+  fi
+
+  # WSREP status OK, but node is not marked ONLINE
+  if [ "${WSREP_STATUS}" = "4" -a "$stat" != "ONLINE" ] ; then
+    # we have to limit amount of writers
+    if [ $NUMBER_WRITERS -gt 0 ] ; then
+      if [ $NUMBER_WRITERS_ONLINE -lt $NUMBER_WRITERS ]; then
+        NUMBER_WRITERS_ONLINE=$(( $NUMBER_WRITERS_ONLINE + 1 ))
+        change_server_status $HOSTGROUP_WRITER_ID "$server" $port "ONLINE" "{NUMBER_WRITERS_ONLINE} of ${NUMBER_WRITERS} write nodes"
+        echo "1" > ${RELOAD_CHECK_FILE}
+      else
+        NUMBER_WRITERS_ONLINE=$(( $NUMBER_WRITERS_ONLINE + 1 ))
+        if [ "$stat" != "OFFLINE_SOFT" ]; then
+          change_server_status $HOSTGROUP_WRITER_ID "$server" $port "OFFLINE_SOFT" "max write nodes reached (${NUMBER_WRITERS})"
+          echo "1" > ${RELOAD_CHECK_FILE}
+        else
+           echo "`date` server $hostgroup:$server:$port is already OFFLINE_SOFT, max write nodes reached (${NUMBER_WRITERS})" >> ${ERR_FILE}
+
+        fi
+      fi
+    # we do not have to limit
+    elif [ $NUMBER_WRITERS -eq 0 ] ; then
+      change_server_status $HOSTGROUP_WRITER_ID "$server" $port "ONLINE" "Changed state, marking write node ONLINE"
+      echo "1" > ${RELOAD_CHECK_FILE}
+    fi
+  fi
+
+  # WSREP status is not ok, but the node is marked online, we should put it offline
+  if [ "${WSREP_STATUS}" != "4" -a "$stat" = "ONLINE" ]; then
+    change_server_status $HOSTGROUP_WRITER_ID "$server" $port "OFFLINE_SOFT" "WSREP status is ${WSREP_STATUS} which is not ok"
+    echo "1" > ${RELOAD_CHECK_FILE}
+  elif [ "${WSREP_STATUS}" != "4" -a "$stat" = "OFFLINE_SOFT" ]; then
+    echo "`date` server $hostgroup:$server:$port is already OFFLINE_SOFT, WSREP status is ${WSREP_STATUS} which is not ok" >> ${ERR_FILE}
+  fi
+
+done
+
+# NUMBER_WRITERS_ONLINE is lost after loop
+NUMBER_WRITERS_ONLINE=$($PROXYSQL_CMDLINE "SELECT count(*) FROM mysql_servers WHERE hostgroup_id IN ($HOSTGROUP_WRITER_ID) AND status = 'ONLINE'" 2>>${ERR_FILE}| tail -1 2>>${ERR_FILE})
+
+
+NUMBER_READERS_ONLINE=0
+if [ ${HOSTGROUP_READER_ID} -ne -1 ]; then
+
+  echo "`date` ###### HANDLE READER NODES ######" >> ${ERR_FILE}
+  if [ $WRITER_IS_READER -eq 1 ]; then
+    READER_PROXYSQL_QUERY="SELECT hostgroup_id, hostname, port, status, 'NULL' FROM mysql_servers WHERE hostgroup_id IN ($HOSTGROUP_READER_ID) AND status <> 'OFFLINE_HARD' ORDER BY weight DESC, hostname, port"
+  elif [ $WRITER_IS_READER -eq 0 ]; then
+    # We will not try to change reader state of nodes that are writer ONLINE, so what we do is we ORDER BY writer.status ASC because by accident ONLINE is last in the line
+    READER_PROXYSQL_QUERY="SELECT reader.hostgroup_id,
+         reader.hostname,
+         reader.port,
+         reader.status,
+         writer.status
+  FROM mysql_servers as reader
+  LEFT JOIN mysql_servers as writer 
+    ON writer.hostgroup_id = $HOSTGROUP_WRITER_ID 
+    AND writer.hostname = reader.hostname 
+    AND writer.port = reader.port
+  WHERE reader.hostgroup_id = $HOSTGROUP_READER_ID
+    AND reader.status <> 'OFFLINE_HARD'
+  ORDER BY writer.status ASC,
+           reader.weight DESC,
+           reader.hostname,
+           reader.port"
+  fi
+
+  OFFLINE_READERS_FOUND=0
+  $PROXYSQL_CMDLINE "$READER_PROXYSQL_QUERY" | while read hostgroup server port stat writer_stat
+  do
+    WSREP_STATUS=$($MYSQL_CMDLINE -h $server -P $port -e "SHOW STATUS LIKE 'wsrep_local_state'" 2>>${ERR_FILE}| tail -1 2>>${ERR_FILE})
+
+    echo "`date` --> Checking READ server $hostgroup:$server:$port, current status $stat, wsrep_local_state $WSREP_STATUS" >> ${ERR_FILE}
+
+    if [ $WRITER_IS_READER -eq 0 -a "$writer_stat" == "ONLINE" ] ; then
+
+      if [ $OFFLINE_READERS_FOUND -eq 0 ] ; then
+        if [ "${WSREP_STATUS}" = "4" -a "$stat" == "ONLINE" ] ; then
+          echo "`date` server $hostgroup:$server:$port is already ONLINE, is also write node in ONLINE state, not enough non-ONLINE readers found" >> ${ERR_FILE}
+        fi
+
+        if [ "${WSREP_STATUS}" = "4" -a "$stat" != "ONLINE" ] ; then
+          change_server_status $HOSTGROUP_READER_ID "$server" $port "ONLINE" "marking ONLINE write node as read ONLINE state, not enough non-ONLINE readers found"
+          echo "1" > ${RELOAD_CHECK_FILE}
+        fi
+      else
+        if [ "${WSREP_STATUS}" = "4" -a "$stat" == "ONLINE" ] ; then
+          change_server_status $HOSTGROUP_READER_ID "$server" $port "OFFLINE_SOFT" "making ONLINE writer node as read OFFLINE_SOFT as well because writers should not be readers"
+          echo "1" > ${RELOAD_CHECK_FILE}
+        fi
+
+        if [ "${WSREP_STATUS}" = "4" -a "$stat" != "ONLINE" ] ; then
+          echo "`date` server $hostgroup:$server:$port is $stat, keeping node in $stat is a writer ONLINE and it's preferred not to have writers as readers" >> ${ERR_FILE}
+        fi
+      fi
+    else
+      if [ "${WSREP_STATUS}" = "4" -a "$stat" == "ONLINE" ] ; then
+        echo "`date` server $hostgroup:$server:$port is already ONLINE" >> ${ERR_FILE}
+        OFFLINE_READERS_FOUND=$(( $OFFLINE_READERS_FOUND + 1 ))
+      fi
+
+      # WSREP status OK, but node is not marked ONLINE
+      if [ "${WSREP_STATUS}" = "4" -a "$stat" != "ONLINE" ] ; then
+        change_server_status $HOSTGROUP_READER_ID "$server" $port "ONLINE" "changed state, making read node ONLINE"
+        echo "1" > ${RELOAD_CHECK_FILE}
+        OFFLINE_READERS_FOUND=$(( $OFFLINE_READERS_FOUND + 1 ))
+      fi
+    fi
+
+    # WSREP status is not ok, but the node is marked online, we should put it offline
+    if [ "${WSREP_STATUS}" != "4" -a "$stat" = "ONLINE" ]; then
+      change_server_status $HOSTGROUP_READER_ID "$server" $port "OFFLINE_SOFT" "WSREP status is ${WSREP_STATUS} which is not ok"
+      echo "1" > ${RELOAD_CHECK_FILE}
+    elif [ "${WSREP_STATUS}" != "4" -a "$stat" = "OFFLINE_SOFT" ]; then
+      echo "`date` server $hostgroup:$server:$port is already OFFLINE_SOFT, WSREP status is ${WSREP_STATUS} which is not ok" >> ${ERR_FILE}
+    fi
+  done
+
+  NUMBER_READERS_ONLINE=$($PROXYSQL_CMDLINE "SELECT count(*) FROM mysql_servers WHERE hostgroup_id IN ($HOSTGROUP_READER_ID) AND status = 'ONLINE'" 2>>${ERR_FILE}| tail -1 2>>${ERR_FILE})
+fi
+
+echo "`date` ###### SUMMARY ######" >> ${ERR_FILE}
+echo "`date` --> Number of writers that are 'ONLINE': ${NUMBER_WRITERS_ONLINE} : hostgroup: ${HOSTGROUP_WRITER_ID}" >> ${ERR_FILE}
+[ ${HOSTGROUP_READER_ID} -ne -1 ] && echo "`date` --> Number of readers that are 'ONLINE': ${NUMBER_READERS_ONLINE} : hostgroup: ${HOSTGROUP_READER_ID}" >> ${ERR_FILE}
+
+
+cnt=0
+# We don't have any writers... alert, try to bring some online!
+# This includes bringing a DONOR online
+if [ ${NUMBER_WRITERS_ONLINE} -eq 0 ]; then
+  echo "`date` ###### TRYING TO FIX MISSING WRITERS ######"
+  echo "`date` No writers found, Trying to enable last available node of the cluster (in Donor/Desync state)" >> ${ERR_FILE}
+  $PROXYSQL_CMDLINE "SELECT hostgroup_id, hostname, port, status FROM mysql_servers WHERE hostgroup_id IN ($HOSTGROUP_WRITER_ID) AND status <> 'OFFLINE_HARD'" | while read hostgroup server port stat
+  do
+    safety_cnt=0
+      while [ ${cnt} -le $NUMBER_WRITERS -a ${safety_cnt} -lt 5 ]
+      do
+        WSREP_STATUS=$($MYSQL_CMDLINE -h $server -P $port -e "SHOW STATUS LIKE 'wsrep_local_state'" 2>>${ERR_FILE} | tail -1 2>>${ERR_FILE})
+        echo "`date` Check server $hostgroup:$server:$port for only available node in DONOR state, status $stat , wsrep_local_state $WSREP_STATUS" >> ${ERR_FILE}
+        if [ "${WSREP_STATUS}" = "2" -a "$stat" != "ONLINE" ]
+        then
+          change_server_status $HOSTGROUP_WRITER_ID "$server" $port "ONLINE" "WSREP status is DESYNC/DONOR, as this is the only node we will put this one online"
+          echo "1" > ${RELOAD_CHECK_FILE}
+          cnt=$(( $cnt + 1 ))
+        fi
+        safety_cnt=$(( $safety_cnt + 1 ))
+    done
+  done
+fi
+
+
+cnt=0
+# We don't have any readers... alert, try to bring some online!
+if [  ${HOSTGROUP_READER_ID} -ne -1 -a ${NUMBER_READERS_ONLINE} -eq 0 ]; then
+  echo "`date` ###### TRYING TO FIX MISSING READERS ######"
+  echo "`date` --> No readers found, Trying to enable last available node of the cluster (in Donor/Desync state) or pick the master" >> ${ERR_FILE}
+  $PROXYSQL_CMDLINE "SELECT hostgroup_id, hostname, port, status FROM mysql_servers WHERE hostgroup_id IN ($HOSTGROUP_READER_ID) AND status <> 'OFFLINE_HARD'" | while read hostgroup server port stat
+  do
+    safety_cnt=0
+      while [ ${cnt} -eq 0 -a ${safety_cnt} -lt 5 ]
+      do
+        WSREP_STATUS=$($MYSQL_CMDLINE -h $server -P $port -e "SHOW STATUS LIKE 'wsrep_local_state'" 2>>${ERR_FILE} | tail -1 2>>${ERR_FILE})
+        echo "`date` Check server $hostgroup:$server:$port for only available node in DONOR state, status $stat , wsrep_local_state $WSREP_STATUS" >> ${ERR_FILE}
+        if [ "${WSREP_STATUS}" = "2" -a "$stat" != "ONLINE" ]
+        then
+          change_server_status $HOSTGROUP_READER_ID "$server" $port "ONLINE" "WSREP status is DESYNC/DONOR, as this is the only node we will put this one online"
+          echo "1" > ${RELOAD_CHECK_FILE}
+          cnt=$(( $cnt + 1 ))
+        fi
+        safety_cnt=$(( $safety_cnt + 1 ))
+    done
+  done
+fi
+
+
+if [ $(cat ${RELOAD_CHECK_FILE})  -ne 0 ] ; then
+    echo "`date` ###### Loading mysql_servers config into runtime ######" >> ${ERR_FILE}
+    $PROXYSQL_CMDLINE "LOAD MYSQL SERVERS TO RUNTIME;" 2>> ${ERR_FILE}
+else
+    echo "`date` ###### Not loading mysql_servers, no change needed ######" >> ${ERR_FILE}
+fi
+
+
+exit 0

dev/provisioning/ansible/roles/proxysql/files/proxysql_node_monitor

@@ -0,0 +1,1086 @@
+#!/bin/bash -u
+# This script will assist to setup Galera cluster ProxySQL monitoring script.
+#####################################################################################
+
+
+#-------------------------------------------------------------------------------
+#
+# Step 1 : Bash internal configuration
+#
+
+set -o nounset    # no undefined variables
+set -o pipefail   # internal pipe failures cause an exit
+
+#bash prompt internal configuration
+declare RED=""
+declare NRED=""
+
+#-------------------------------------------------------------------------------
+#
+# Step 2 : Global variables
+#
+
+declare -i  DEBUG=0
+readonly    PROXYSQL_ADMIN_VERSION="2.0.16"
+
+declare     CONFIG_FILE="/etc/proxysql-admin.cnf"
+declare     ERR_FILE="/dev/null"
+declare     RELOAD_CHECK_FILE=""
+
+# Set to send output here when DEBUG is set
+declare     DEBUG_ERR_FILE="/dev/null"
+
+declare -i  WRITE_HOSTGROUP_ID=1
+declare -i  READ_HOSTGROUP_ID=-1
+declare -i  SLAVEREAD_HOSTGROUP_ID
+
+# This is the hostgroup that new nodes will be added to
+declare -i  DEFAULT_HOSTGROUP_ID
+
+declare     MODE
+
+declare     CHECK_STATUS=0
+
+declare     PROXYSQL_DATADIR='/var/lib/proxysql'
+
+declare -i  TIMEOUT=10
+
+# Maximum time to wait for cluster status
+declare -i  CLUSTER_TIMEOUT=3
+
+# Extra text that will be logged with the output
+# (useful for debugging/testing)
+declare     LOG_TEXT=""
+
+# Default value for max_connections in mysql_servers
+declare     MAX_CONNECTIONS="1000"
+
+
+#we need to be able to disable autodiscovery
+#This because in presence of multiple IPs on the PXC nodes the script choose to use the use the one
+#Nodes are registered for internal traffic which is normally different from the one
+#used for application traffic.
+#Result is that the AUTODICOVERY will cause ProxySQL to use wrong IP set
+declare PXC_AUTODISCOVERY=1
+
+#-------------------------------------------------------------------------------
+#
+# Step 3 : Helper functions
+#
+
+function log() {
+  local lineno=$1
+  shift
+
+  if [[ -n $ERR_FILE ]]; then
+    if [[ -n $lineno && $DEBUG -ne 0 ]]; then
+      echo "[$(date +%Y-%m-%d\ %H:%M:%S)] (line $lineno) $*" >> $ERR_FILE
+    else
+      echo "[$(date +%Y-%m-%d\ %H:%M:%S)] $*" >> $ERR_FILE
+    fi
+  fi
+}
+
+function log_if_success() {
+  local lineno=$1
+  local rc=$2
+  shift 2
+
+  if [[ $rc -eq 0 ]]; then
+    log "$lineno" "$*"
+  fi
+}
+
+function error() {
+  local lineno=$1
+  shift
+
+  log "$lineno" "ERROR: $*"
+}
+
+function warning() {
+  local lineno=$1
+  shift
+
+  log "$lineno" "WARNING: $*"
+}
+
+function debug() {
+  if [[ $DEBUG -eq 0 ]]; then
+    return
+  fi
+
+  local lineno=$1
+  shift
+
+  log "$lineno" "${RED}debug: $*${NRED}"
+}
+
+
+function usage () {
+  local path=$0
+  cat << EOF
+Usage: ${path##*/} [ options ]
+
+Example:
+  proxysql_node_monitor --write-hg=10 --read-hg=11 --config-file=/etc/proxysql-admin.cnf --log=/var/lib/proxysql/pxc_test_proxysql_galera_check.log
+
+Options:
+  -w, --write-hg=<NUMBER>             Specify ProxySQL write hostgroup.
+  -r, --read-hg=<NUMBER>              Specify ProxySQL read hostgroup.
+  -m, --mode=[loadbal|singlewrite]    ProxySQL read/write configuration mode, currently supporting: 'loadbal' and 'singlewrite' (the default) modes
+  -p, --priority=<HOST_LIST>          Can accept comma delimited list of write nodes priority
+  -c, --config-file=PATH              Specify ProxySQL-admin configuration file.
+  -l, --log=PATH                      Specify proxysql_node_monitor log file.
+  --log-text=TEXT                     This is text that will be written to the log file
+                                      whenever this script is run (useful for debugging).
+  --reload-check-file=PATH            Specify file used to notify proysql_galera_checker
+                                      of a change in server configuration
+  --max-connections=<NUMBER>          Value for max_connections in the mysql_servers table.
+                                      This is the maximum number of connections that
+                                      ProxySQL will open to the backend servers.
+                                      (default: 1000)
+  --debug                             Enables additional debug logging.
+  -h, --help                          Display script usage information
+  -v, --version                       Print version info
+EOF
+}
+
+
+# Check the permissions for a file or directory
+#
+# Globals:
+#   None
+#
+# Arguments:
+#   1: the bash test to be applied to the file
+#   2: the lineno where this call is invoked (used for errors)
+#   3: the path to the file
+#   4: (optional) description of the path (mostly used for existence checks)
+#
+# Exits the script if the permissions test fails.
+#
+function check_permission() {
+  local permission=$1
+  local lineno=$2
+  local path_to_check=$3
+  local description=""
+  if [[ $# -gt 3 ]]; then
+    description="$4"
+  fi
+
+  if [ ! $permission "$path_to_check" ] ; then
+    if [[ $permission == "-r" ]]; then
+      error $lineno "You do not have READ permission for: $path_to_check"
+    elif [[ $permission == "-w" ]]; then
+      error $lineno "You do not have WRITE permission for: $path_to_check"
+    elif [[ $permission == "-x" ]]; then
+      error $lineno "You do not have EXECUTE permission for: $path_to_check"
+    elif [[ $permission == "-e" ]]; then
+      if [[ -n $description ]]; then
+        error $lineno "Could not find the $description: $path_to_check"
+      else
+        error $lineno "Could not find: $path_to_check"
+      fi
+    elif [[ $permission == "-d" ]]; then
+      if [[ -n $description ]]; then
+        error $lineno "Could not find the $description: $path_to_check"
+      else
+        error $lineno "Could not find the directory: $path_to_check"
+      fi
+    elif [[ $permission == "-f" ]]; then
+      if [[ -n $description ]]; then
+        error $lineno "Could not find the $description: $path_to_check"
+      else
+        error $lineno "Could not find the file: $path_to_check"
+      fi
+    else
+      error $lineno "You do not have the correct permissions for: $path_to_check"
+    fi
+    exit 1
+  fi
+}
+
+
+#
+#
+# Globals:
+#   None
+#
+# Arguments:
+#   1: the lineno
+#   2: the return value that is being checked
+#   3: the error message
+#   4: Additional information (only used if an error occurred) (optional)
+#
+# Returns:
+#   Returns the return value that is passed in.
+#   This allows the code that follows to check the return value.
+#
+# Note that this will NOT exit the script.
+#
+function check_cmd() {
+  local lineno=$1
+  local mpid=$2
+  local error_msg=$3
+  local error_info=""
+
+  if [[ $# -ge 4 ]]; then
+    error_info=$4
+  fi
+
+  if [ "$mpid" == "124" ]; then
+    error $lineno "TIMEOUT: Connection terminated due to timeout."
+  fi
+  if [ ${mpid} -ne 0 ]; then
+    warning $lineno "$error_msg."
+    if [[ ! -z  $error_info ]]; then
+      log $lineno "$error_info."
+    fi
+  fi
+  return $mpid
+}
+
+# Executes a SQL query with the (fully) specified server
+#
+# Globals:
+#   None
+#
+# Arguments:
+#   1: lineno
+#   2: the name of the user
+#   3: the user's password
+#   4: the hostname of the server
+#   5: the port used to connect to the server
+#   6: timeout in secs
+#   7: arguments to the mysql client
+#   8: additional options to the [client] config
+#   9: the query to be run
+#   10: additional options, space separated
+#      Available options:
+#       "hide_output"
+#         This will not show the output of the query when DEBUG is set.
+#         Used to stop the display of sensitve information (such as passwords)
+#         from being displayed when debugging.
+#
+function exec_sql() {
+  local lineno=$1
+  local user=$2
+  local password=$3
+  local hostname=$4
+  local port=$5
+  local timeout_secs=$6
+  local args=$7
+  local client_options=$8
+  local query="$9"
+  local more_options="${10}"
+  local retvalue
+  local retoutput
+
+  debug "$lineno" "exec_sql : $user@$hostname:$port ==> $query"
+
+  retoutput=$(printf "[client]\n${client_options}\nuser=${user}\npassword=\"${password}\"\nhost=${hostname}\nport=${port}"  \
+      | timeout ${timeout_secs} mysql --defaults-file=/dev/stdin --protocol=tcp \
+              ${args} -e "$query")
+  retvalue=$?
+
+  if [[ $DEBUG -eq 1 ]]; then
+    local number_of_newlines=0
+    local dbgoutput=$retoutput
+
+    if [[ " $more_options " =~ [[:space:]]hide_output[[:space:]] ]]; then
+      dbgoutput="**** data hidden ****"
+    fi
+
+    if [[ -n $dbgoutput ]]; then
+      number_of_newlines=$(printf "%s" "${dbgoutput}" | wc -l)
+    fi
+
+    if [[  $retvalue -ne 0 ]]; then
+      debug "" "--> query failed $retvalue"
+    elif [[ -z $dbgoutput ]]; then
+      debug "" "--> query returned $retvalue : <query returned no data>"
+    elif [[ ${number_of_newlines} -eq 0 ]]; then
+      debug "" "--> query returned $retvalue : ${dbgoutput}"
+    else
+      debug "" "--> query returned $retvalue : <data follows>"
+      printf "${dbgoutput//%/%%}\n" | while IFS= read -r line; do
+        debug "" "----> $line"
+      done
+    fi
+  fi
+
+  printf "${retoutput//%/%%}"
+  return $retvalue
+}
+
+
+# Executes a SQL query on proxysql (with a timeout of $TIMEOUT seconds)
+#
+# Globals:
+#   PROXYSQL_USERNAME
+#   PROXYSQL_PASSWORD
+#   PROXYSQL_HOSTNAME
+#   PROXYSQL_PORT
+#   TIMEOUT
+#
+# Arguments:
+#   1: lineno (used for debugging/output, may be blank)
+#   2: The SQL query
+#   3: (optional) more options, see exec_sql
+#
+function proxysql_exec() {
+  local lineno=$1
+  local query="$2"
+  local more_options=""
+  local retoutput
+
+  if [[ $# -ge 3 ]]; then
+    more_options=$3
+  fi
+
+  exec_sql "$lineno" "$PROXYSQL_USERNAME" "$PROXYSQL_PASSWORD" \
+           "$PROXYSQL_HOSTNAME" "$PROXYSQL_PORT" \
+           "$TIMEOUT" "-Bs" "" "$query" "$more_options"
+  retoutput=$?
+  return $retoutput
+}
+
+# Executes a SQL query on mysql (with a timeout of $TIMEOUT secs)
+#
+# Globals:
+#   CLUSTER_USERNAME
+#   CLUSTER_PASSWORD
+#   CLUSTER_HOSTNAME
+#   CLUSTER_PORT
+#   CLUSTER_TIMEOUT
+#
+# Arguments:
+#   1: lineno (used for debugging/output, may be blank)
+#   2: the query to be run
+#   3: (optional) more options, see exec_sql
+#
+function mysql_exec() {
+  local lineno=$1
+  local query=$2
+  local more_options=""
+  local retoutput
+
+  if [[ $# -ge 3 ]]; then
+    more_options=$3
+  fi
+
+  exec_sql "$lineno" "$CLUSTER_USERNAME" "$CLUSTER_PASSWORD" \
+           "$CLUSTER_HOSTNAME" "$CLUSTER_PORT" \
+           "$TIMEOUT" "-Bs" "connect-timeout=${CLUSTER_TIMEOUT}" "$query" "$more_options"
+  retoutput=$?
+  return $retoutput
+}
+
+
+# Executes a SQL query on mysql (with a timeout of $TIMEOUT secs)
+#
+# Globals:
+#   CLUSTER_USERNAME
+#   CLUSTER_PASSWORD
+#   CLUSTER_TIMEOUT
+#
+# Arguments:
+#   1: lineno (used for debugging/output, may be blank)
+#   2: the hostname of the server
+#   3: the port used to connect to the server
+#   4: the query to be run
+#   5: (optional) more options, see exec_sql
+#
+function slave_exec() {
+  local lineno=$1
+  local hostname=$2
+  local port=$3
+  local query=$4
+  local more_options=""
+  local timeout_secs=$TIMEOUT
+  local retoutput
+
+  if [[ $# -ge 5 ]]; then
+    more_options=$5
+  fi
+
+
+  exec_sql "$lineno" "$CLUSTER_USERNAME" "$CLUSTER_PASSWORD" \
+           "$hostname" "$port" \
+           "$timeout_secs" "-Bs" "" "$query" "$more_options"
+  retoutput=$?
+  return $retoutput
+}
+
+# Separates the IP address from the port in a network address
+# Works for IPv4 and IPv6
+#
+# Globals:
+#   None
+#
+# Params:
+#   1. The network address to be parsed
+#
+# Outputs:
+#   A string with a space separating the IP address from the port
+#
+function separate_ip_port_from_address()
+{
+  #
+  # Break address string into host:port/path parts
+  #
+  local address=$1
+
+  # Has to have at least one ':' to separate the port from the ip address
+  if [[ $address =~ : ]]; then
+    ip_addr=${address%:*}
+    port=${address##*:}
+  else
+    ip_addr=$address
+    port=""
+  fi
+
+  # Remove any braces that surround the ip address portion
+  ip_addr=${ip_addr#\[}
+  ip_addr=${ip_addr%\]}
+
+  echo "${ip_addr} ${port}"
+}
+
+# Combines the IP address and port into a network address
+# Works for IPv4 and IPv6
+# (If the IP address is IPv6, the IP portion will have brackets)
+#
+# Globals:
+#   None
+#
+# Params:
+#   1: The IP address portion
+#   2: The port
+#
+# Outputs:
+#   A string containing the full network address
+#
+function combine_ip_port_into_address()
+{
+  local ip_addr=$1
+  local port=$2
+  local addr
+
+  if [[ ! $ip_addr =~ \[.*\] && $ip_addr =~ .*:.* ]] ; then
+    # If there are no brackets and it does have a ':', then add the brackets
+    # because this is an unbracketed IPv6 address
+    addr="[${ip_addr}]:${port}"
+  else
+    addr="${ip_addr}:${port}"
+  fi
+  echo $addr
+}
+
+
+# Update Galera Cluster nodes in ProxySQL database
+# This will take care of nodes that have gone up or gone down
+# (i.e. if the ProxySQL and PXC memberships differ).
+#
+# This does not take care of the policy issues, it does not
+# ensure there is a writer.
+#
+# Globals:
+#   WRITE_HOSTGROUP_ID
+#   READ_HOSTGROUP_ID
+#   SLAVEREAD_HOSTGROUP_ID
+#   MODE
+#   MODE_COMMENT
+#   CHECK_STATUS
+#
+# Arguments:
+#   1: active cluster host (may be empty if cluster is offline)
+#   1: active cluster port (may be empty if cluster is offline)
+#
+function update_cluster() {
+  debug $LINENO "START update_cluster"
+  local cluster_host=$1
+  local cluster_port=$2
+  local host_info=""
+  local current_hosts=""
+  local is_current_hosts_empty=0
+  local wsrep_address=""
+  local ws_address
+  local ws_ip
+  local ws_port
+  local ws_hg_status
+  local ws_hg_id
+  local ws_status
+  local ws_comment
+
+  # get all nodes from ProxySQL in use by hostgroups
+  host_info=$(proxysql_exec $LINENO "SELECT DISTINCT hostname || ':' || port,hostgroup_id,status FROM mysql_servers where status != 'OFFLINE_HARD' and hostgroup_id in ( $WRITE_HOSTGROUP_ID, $READ_HOSTGROUP_ID, $SLAVEREAD_HOSTGROUP_ID )" | tr '\t' ' ')
+  if [[ -n host_info ]]; then
+    # Extract the hostname and port from the rows
+    # Creates a string of "host:port" separated by spaces
+    current_hosts=""
+
+    while read line; do
+      if [[ -z $line ]]; then
+        continue
+      fi
+      net_address=$(echo $line | cut -d' ' -f1)
+      net_address=$(separate_ip_port_from_address $net_address)
+      local ip_addr=$(echo "$net_address" | cut -d' ' -f1)
+      local port=$(echo "$net_address" | cut -d' ' -f2)
+      net_address=$(combine_ip_port_into_address "$ip_addr" "$port")
+      current_hosts+="$net_address "
+    done< <(printf "$host_info\n")
+
+    current_hosts=${current_hosts% }
+  fi
+
+  if [[ -n $cluster_host && -n $cluster_port ]]; then
+    # First, find a host that is online from ProxySQL
+    ws_ip=$cluster_host
+    ws_port=$cluster_port
+
+    # Second, get the wsrep_incoming_addresses from the cluster
+    wsrep_address=$(slave_exec $LINENO "${ws_ip}" "${ws_port}" \
+          "SHOW STATUS LIKE 'wsrep_incoming_addresses'" | awk '{print $2}' | sed 's|,| |g')
+  fi
+
+  if [[ -z $wsrep_address && -z $current_hosts ]]; then
+    debug $LINENO "Returning from update_cluster(), both PXC and ProxySQL have no active nodes"
+    return
+  fi
+
+  #
+  # Given the WSREP members, compare to ProxySQL
+  # If missing from ProxySQL, add to ProxySQL as a reader.
+  #
+  debug $LINENO "Looking for PXC nodes not in ProxySQL"
+  for i in ${wsrep_address}; do
+    # if we have a match, the the PXC node is in ProxySQL and we can skip
+    if [[ -n $current_hosts && " ${current_hosts} " =~ " ${i} " ]]; then
+      continue
+    fi
+
+    log $LINENO "Cluster node (${i}) does not exist in ProxySQL, adding as a $MODE_COMMENT node"
+    ws_address=$(separate_ip_port_from_address "$i")
+    ws_ip=$(echo "$ws_address" | cut -d' ' -f1)
+    ws_port=$(echo "$ws_address" | cut -d' ' -f2)
+
+    # Add the node as a reader
+    local hostgroup
+
+    # Before inserting, check if a previous READ entry exists (it may be in OFFLINE_HARD state)
+    hostgroup=$(proxysql_exec $LINENO "SELECT hostgroup_id FROM mysql_servers WHERE hostgroup_id=${DEFAULT_HOSTGROUP_ID} AND hostname='${ws_ip}' AND port=${ws_port}")
+
+    if [[ -n $hostgroup ]]; then
+      # Update reader to OFFLINE_SOFT if new PXC node in ProxySQL
+      proxysql_exec $LINENO "UPDATE mysql_servers SET status='OFFLINE_SOFT',weight=1000,comment='$MODE_COMMENT' WHERE hostname='${ws_ip}' AND port=${ws_port} AND hostgroup_id=${hostgroup}"
+      check_cmd $LINENO $? "Cannot update Galera Cluster node $ws_address (hostgroup $hostgroup) to ProxySQL database, Please check ProxySQL login credentials"
+      log_if_success $LINENO $? "Updated ${hostgroup}:${i} node in the ProxySQL database."
+    else
+      if [[ $PXC_AUTODISCOVERY -eq 1 ]]; then
+          # Insert a reader if new PXC node not in ProxySQL
+          proxysql_exec $LINENO "INSERT INTO mysql_servers (hostname,hostgroup_id,port,weight,comment,max_connections) VALUES ('$ws_ip',$DEFAULT_HOSTGROUP_ID,$ws_port,1000,'$MODE_COMMENT',$MAX_CONNECTIONS);"
+          check_cmd $LINENO $? "Cannot add Galera Cluster node $ws_address (hostgroup $DEFAULT_HOSTGROUP_ID) to ProxySQL database, Please check ProxySQL login credentials"
+          log_if_success $LINENO $? "Added ${DEFAULT_HOSTGROUP_ID}:${i} node into ProxySQL database."
+      else
+          log_if_success $LINENO $? "Node identified by AUTODISCOVERY but will not add it because insert by AUTODISCOVERY for PXC is disable. Candidate node: ${DEFAULT_HOSTGROUP_ID}:${i} for ProxySQL database."
+      fi
+    fi
+
+    CHECK_STATUS=1
+  done
+
+  #
+  # Given the ProxySQL members, compare to WSREP
+  # If not in WSREP, mark as OFFLINE_HARD
+  #
+  debug $LINENO "Looking for ProxySQL nodes not in PXC"
+  for i in $current_hosts; do
+    # if we have a match, then the proxysql node is in PXC
+    # so we can skip it
+    if [[ -n ${wsrep_address} && " ${wsrep_address} " =~ " ${i} " ]]; then
+      continue
+    fi
+
+    debug $LINENO "ProxySQL host $i not found in cluster membership"
+    #
+    # The current host in current_hosts was not found in cluster membership,
+    # set it OFFLINE_SOFT unless its a slave node
+    #
+    ws_address=$(separate_ip_port_from_address "$i")
+    ws_ip=$(echo "$ws_address" | cut -d' ' -f1)
+    ws_port=$(echo "$ws_address" | cut -d' ' -f2)
+
+    # This is supported by status, so OFFLINE should come before ONLINE
+    # Note that the status is in DESC order, so "ONLINE : OFFLINE_SOFT : OFFLINE_HARD"
+    # This is needed because there may be multiple entries
+    ws_hg_status=$(proxysql_exec $LINENO "SELECT hostgroup_id,status,comment from mysql_servers WHERE hostname='$ws_ip' and port=$ws_port ORDER BY status DESC LIMIT 1")
+    ws_hg_id=$(echo -e "$ws_hg_status" | cut -f1)
+    ws_status=$(echo -e "$ws_hg_status" | cut -f2)
+    ws_comment=$(echo -e "$ws_hg_status" | cut -f3)
+
+    if [ "$ws_comment" == "SLAVEREAD" ]; then
+      # This update now happens in proxysql_galera_checker
+      continue
+    fi
+
+    if [ "$ws_status" == "OFFLINE_SOFT" ]; then
+      #
+      # If OFFLINE_SOFT, move to OFFLINE_HARD
+      #
+
+      log $LINENO "Cluster node ${ws_hg_id}:${i} does not exist in PXC! Changing status from OFFLINE_SOFT to OFFLINE_HARD"
+      proxysql_exec $LINENO "UPDATE mysql_servers set status='OFFLINE_HARD' WHERE hostname='$ws_ip' and port=$ws_port"
+      check_cmd $LINENO $? "Cannot update Galera Cluster writer node in ProxySQL database, Please check ProxySQL login credentials"
+      CHECK_STATUS=1
+    elif [[ $ws_status == "ONLINE" ]]; then
+      #
+      # else if ONLINE, move to OFFLINE_SOFT
+      # It will take another iteration to get it to OFFLINE_HARD
+      #
+      log $LINENO "Cluster node ${ws_hg_id}:${i} does not exist in PXC! Changing status to OFFLINE_SOFT"
+      # Set all entries to OFFLINE_SOFT
+      proxysql_exec $LINENO "UPDATE mysql_servers set status='OFFLINE_SOFT' WHERE hostname='$ws_ip' and port=$ws_port"
+      check_cmd $LINENO $? "Cannot update Galera Cluster writer node in ProxySQL database, Please check ProxySQL login credentials"
+      CHECK_STATUS=1
+    fi
+
+    node_status=$(proxysql_exec $LINENO "SELECT status from mysql_servers WHERE hostname='$ws_ip' and port=$ws_port ORDER BY status LIMIT 1")
+    log $LINENO "Non-PXC node (${i}) current status '$node_status' in ProxySQL."
+  done
+
+  # Update the ProxySQL status for the new nodes
+  for i in ${wsrep_address}; do
+    if [[ -n $current_hosts && " ${current_hosts} " =~ " ${i} " ]]; then
+      # Lookup the status in the host_info
+      local host
+
+      ws_address=$(separate_ip_port_from_address "$i")
+      ws_ip=$(echo "$ws_address" | cut -d' ' -f1)
+      ws_port=$(echo "$ws_address" | cut -d' ' -f2)
+
+      # properly escape the characters for grep
+      local re_i="$(printf '%s' "$ws_ip:$ws_port" | sed 's/[.[\*^$]/\\&/g')"
+      host=$(echo "$host_info" | grep "${re_i}" | head -1)
+
+      ws_hg_id=$(echo $host | cut -d' ' -f2)
+      ws_status=$(echo $host | cut -d' ' -f3)
+      log "" "Cluster node (${ws_hg_id}:${i}) current status '$ws_status' in ProxySQL."
+    else
+      ws_address=$(separate_ip_port_from_address "$i")
+      ws_ip=$(echo "$ws_address" | cut -d' ' -f1)
+      ws_port=$(echo "$ws_address" | cut -d' ' -f2)
+      ws_hg_status=$(proxysql_exec $LINENO "SELECT hostgroup_id,status from mysql_servers WHERE hostname='$ws_ip' and port=$ws_port")
+      ws_hg_id=$(echo $ws_hg_status | cut -d' ' -f1)
+      ws_status=$(echo $ws_hg_status | cut -d' ' -f2)
+
+      log $LINENO "Cluster node (${ws_hg_id}:${i}) current status '$ws_status' in ProxySQL database!"
+      if [ "$ws_status" == "OFFLINE_HARD" ]; then
+        # The node was OFFLINE_HARD, but its now in the cluster list
+        # so lets make it OFFLINE_SOFT
+        proxysql_exec $LINENO "UPDATE mysql_servers set status = 'OFFLINE_SOFT' WHERE hostname='$ws_ip' and port=$ws_port;"
+        check_cmd $LINENO $? "Cannot update Galera Cluster node $i in the ProxySQL database, Please check the ProxySQL login credentials"
+        log_if_success $LINENO $? "${ws_hg_id}:${i} node set to OFFLINE_SOFT status to ProxySQL database."
+        CHECK_STATUS=1
+      fi
+    fi
+  done
+  debug $LINENO "END update_cluster"
+}
+
+
+# Move the entries in the list from writers to readers
+#
+# Globals:
+#   READ_HOSTGROUP_ID
+#   WRITE_HOSTGROUP_ID
+#   CHECK_STATUS
+#
+# Arguments:
+#   1: A list of nodes to move to readers (entries are 'server port hostgroup')
+#
+function move_writers_to_readers() {
+  debug $LINENO "START move_writers_to_readers($*)"
+  local offline_writers=$1
+
+  debug $LINENO "$offline_writers"
+  printf "$offline_writers" | while read host port hostgroup status || [ -n "$hostgroup" ]
+  do
+    local read_count
+
+    debug $LINENO "mode_change_check: Found OFFLINE_SOFT writer, changing to READ status and hostgroup $READ_HOSTGROUP_ID"
+
+    read_count=$(proxysql_exec $LINENO "SELECT COUNT(*) FROM mysql_servers WHERE hostgroup_id=$READ_HOSTGROUP_ID AND hostname='$host' AND port=$port")
+    if [[ $read_count -ne 0 ]]; then
+      # If node is already a READER, update the READER
+      if [[ $status == 'ONLINE' ]]; then
+        proxysql_exec $LINENO "UPDATE mysql_servers SET status='OFFLINE_SOFT',hostgroup_id=$READ_HOSTGROUP_ID, comment='READ', weight=1000 WHERE hostgroup_id=$READ_HOSTGROUP_ID AND hostname='$host' AND port=$port"
+        check_cmd $LINENO $? "Cannot update Galera Cluster writer node in ProxySQL database, Please check ProxySQL login credentials"
+        log_if_success $LINENO $? "Changed OFFLINE_SOFT writer to a reader ($READ_HOSTGROUP_ID:$host:$port)"
+        CHECK_STATUS=1
+      fi
+    else
+      if [[ $status == "ONLINE" ]]; then
+        local address
+        address=$(combine_ip_port_into_address "$host" "$port")
+
+        # If node is not a reader, add node as a reader
+        debug $LINENO "Node is not a reader, adding node as a reader"
+
+        proxysql_exec $LINENO \
+              "INSERT INTO mysql_servers (hostname,hostgroup_id,port,status,weight,comment,max_connections)
+                  VALUES ('$host',$READ_HOSTGROUP_ID,$port,'OFFLINE_SOFT',1000,'READ',$MAX_CONNECTIONS);"
+        check_cmd $LINENO $? "Cannot add Galera Cluster reader node in ProxySQL database, Please check ProxySQL login credentials"
+        log $LINENO "Adding server $READ_HOSTGROUP_ID:$address with status OFFLINE_SOFT."
+        CHECK_STATUS=1
+      fi
+    fi
+  done
+}
+
+
+#
+# Globals:
+#   PROXYSQL_DATADIR
+#   CLUSTER_NAME
+#   WRITE_HOSTGROUP_ID  READ_HOSTGROUP_ID
+#   MODE
+#
+# Arguments:
+#   None
+#
+function mode_change_check(){
+  debug $LINENO "START mode_change_check"
+
+  # Check if the current writer is in an OFFLINE_SOFT state
+  local offline_writers
+  offline_writers=$(proxysql_exec $LINENO "SELECT hostname,port,hostgroup_id,status from mysql_servers where comment in ('WRITE', 'READWRITE') and status <> 'ONLINE' and hostgroup_id in ($WRITE_HOSTGROUP_ID)")
+  if [[ -n $offline_writers ]]; then
+    #
+    # Found a writer node that was in 'OFFLINE_SOFT' state,
+    # move it to the READ hostgroup unless the MODE is 'loadbal'
+    #
+    if [ "$MODE" != "loadbal" ]; then
+      move_writers_to_readers "$offline_writers"
+    fi
+  fi
+
+  debug $LINENO "END mode_change_check"
+}
+
+
+#
+# Globals:
+#   DEBUG
+#   CONFIG_FILE
+#   WRITE_HOSTGROUP_ID  READ_HOSTGROUP_ID
+#   DEFAULT_HOSTGROUP_ID
+#   MODE
+#   ERR_FILE
+#   PROXYSQL_ADMIN_VERSION
+#   MODE_COMMENT
+#   WRITE_WEIGHT
+#
+# Arguments:
+#
+function parse_args() {
+  # Check if we have a functional getopt(1)
+  if ! getopt --test; then
+    go_out="$(getopt --options=w:r:c:l:m:p:vh --longoptions=write-hg:,read-hg:,mode:,priority:,config-file:,log:,reload-check-file:,log-text:,max-connections:,debug,version,help \
+    --name="$(basename "$0")" -- "$@")"
+    if [[ $? -ne 0 ]]; then
+      # no place to send output
+      echo "Script error: getopt() failed" >&2
+      exit 1
+    fi
+    eval set -- "$go_out"
+  fi
+
+  if [[ $go_out == " --" ]];then
+    usage
+    exit 1
+  fi
+
+  #
+  # We iterate through the command-line options twice
+  # (1) to handle options that don't need permissions (such as --help)
+  # (2) to handle options that need to be done before other
+  #     options, such as loading the config file
+  #
+  for arg
+  do
+    case "$arg" in
+      -- ) shift; break;;
+      --config-file )
+        CONFIG_FILE="$2"
+        check_permission -e $LINENO "$CONFIG_FILE" "proxysql-admin configuration file"
+        debug $LINENO  "--config-file specified, using : $CONFIG_FILE"
+        shift 2
+        ;;
+      --help)
+        usage
+        exit 0
+        ;;
+      -v | --version)
+        echo "proxysql_node_monitor version $PROXYSQL_ADMIN_VERSION"
+        exit 0
+        ;;
+      --debug)
+        DEBUG=1
+        shift
+        ;;
+      *)
+        shift
+        ;;
+    esac
+  done
+
+  #
+  # Load the config file before reading in the command-line options
+  #
+  readonly CONFIG_FILE
+  if [ ! -e "$CONFIG_FILE" ]; then
+      warning "" "Could not locate the configuration file: $CONFIG_FILE"
+  else
+      check_permission -r $LINENO "$CONFIG_FILE"
+      debug $LINENO "Loading $CONFIG_FILE"
+      source "$CONFIG_FILE"
+  fi
+
+
+  if [[ $DEBUG -ne 0 ]]; then
+    # For now
+    if [[ -t 1 ]]; then
+      ERR_FILE=/dev/stdout
+    fi
+  fi
+
+  local p_mode=""
+
+  # Reset the command line for the next invocation
+  eval set -- "$go_out"
+
+  for arg
+  do
+    case "$arg" in
+      -- ) shift; break;;
+      -w | --write-hg )
+        WRITE_HOSTGROUP_ID=$2
+        shift 2
+      ;;
+      -r | --read-hg )
+        READ_HOSTGROUP_ID=$2
+        shift 2
+      ;;
+      -m | --mode )
+        p_mode="$2"
+        shift 2
+        if [ "$p_mode" != "loadbal" ] && [ "$p_mode" != "singlewrite" ]; then
+          echo "ERROR: Invalid --mode passed:"
+          echo "  Please choose any of these modes: loadbal, singlewrite"
+          exit 1
+        fi
+      ;;
+      -p | --priority )
+        # old parameter
+        shift 2
+      ;;
+      --config-file )
+        shift 2
+        # The config-file is loaded before the command-line
+        # arguments are handled.
+      ;;
+      -l | --log )
+        ERR_FILE="$2"
+        shift 2
+
+        # Test if stdout and stderr are open to a terminal
+        if [[ $ERR_FILE == "/dev/stdout" || $ERR_FILE == "/dev/stderr" ]]; then
+          RED=$(tput setaf 1)
+          NRED=$(tput sgr0)
+        fi
+      ;;
+      --reload-check-file )
+        RELOAD_CHECK_FILE="$2"
+        shift 2
+      ;;
+      --log-text )
+        LOG_TEXT="$2"
+        shift 2
+      ;;
+      --max-connections )
+        MAX_CONNECTIONS="$2"
+        shift 2
+      ;;
+      --debug )
+        shift;
+      ;;
+      -v | --version )
+        shift;
+      ;;
+      -h | --help )
+        shift;
+      ;;
+    esac
+  done
+
+  if [[ $DEBUG -eq 1 ]]; then
+    DEBUG_ERR_FILE=$ERR_FILE
+  fi
+
+  #Timeout exists for instances where mysqld/proxysql may be hung
+  TIMEOUT=5
+
+  SLAVEREAD_HOSTGROUP_ID=$READ_HOSTGROUP_ID
+  if [ $SLAVEREAD_HOSTGROUP_ID -eq $WRITE_HOSTGROUP_ID ];then
+    let SLAVEREAD_HOSTGROUP_ID+=1
+  fi
+
+  DEFAULT_HOSTGROUP_ID=$READ_HOSTGROUP_ID
+  if [[ $DEFAULT_HOSTGROUP_ID -eq -1 ]]; then
+    DEFAULT_HOSTGROUP_ID=$WRITE_HOSTGROUP_ID
+  fi
+
+  CHECK_STATUS=0
+
+  debug $LINENO "#### PROXYSQL NODE MONITOR ARGUMENT CHECKING"
+  debug $LINENO "MODE: $MODE"
+  debug $LINENO "check mode name from proxysql data directory "
+  CLUSTER_NAME=$(proxysql_exec $LINENO "SELECT comment from scheduler where arg1 LIKE '%--write-hg=$WRITE_HOSTGROUP_ID %' OR arg1 LIKE '%-w $WRITE_HOSTGROUP_ID %'")
+  check_cmd $LINENO $? "Cannot connect to ProxySQL at $PROXYSQL_HOSTNAME:$PROXYSQL_PORT"
+  if [[ ! -z $p_mode ]] ; then
+    MODE=$p_mode
+    debug $LINENO "command-line: setting MODE to $MODE"
+  else
+    # Get the name of the mode file
+    local proxysql_mode_file
+    if [[ -z $CLUSTER_NAME ]]; then
+      proxysql_mode_file="${PROXYSQL_DATADIR}/mode"
+    else
+      proxysql_mode_file="${PROXYSQL_DATADIR}/${CLUSTER_NAME}_mode"
+    fi
+
+    if [[ -f "$proxysql_mode_file" && -r "$proxysql_mode_file" ]]; then
+      MODE=$(cat ${proxysql_mode_file})
+      debug $LINENO "file: $proxysql_mode_file: setting MODE to $MODE"
+    fi
+  fi
+
+
+  if [ "$MODE" == "loadbal" ]; then
+    MODE_COMMENT="READWRITE"
+    WRITE_WEIGHT="1000"
+  else
+    MODE_COMMENT="READ"
+    WRITE_WEIGHT="1000000"
+  fi
+
+  if [[ -z $RELOAD_CHECK_FILE ]]; then
+    error $LINENO "The --reload-check-file option is required."
+    exit 1
+  fi
+  check_permission -r $LINENO "$RELOAD_CHECK_FILE"
+
+  # Verify that we have an integer
+  if ! [ "$MAX_CONNECTIONS" -eq "$MAX_CONNECTIONS" ] 2>/dev/null
+  then
+    error $LINENO "Invalid --max-connections value (must be a number) : $MAX_CONNECTIONS"
+    exit 1
+  fi
+
+  readonly WRITE_HOSTGROUP_ID
+  readonly READ_HOSTGROUP_ID
+  readonly SLAVEREAD_HOSTGROUP_ID
+  readonly MODE
+  readonly MODE_COMMENT
+  readonly WRITE_WEIGHT
+  readonly CLUSTER_NAME
+  readonly RELOAD_CHECK_FILE
+  readonly MAX_CONNECTIONS
+}
+
+# Returns the address of an available (online) cluster host
+#
+# Globals:
+#   WRITE_HOSTGROUP_ID
+#   READ_HOSTGROUP_ID
+#
+# Arguments:
+#   None
+#
+function find_online_cluster_host() {
+  # Query the proxysql database for hosts,ports in use
+  # Then just go through the list until we reach one that responds
+  local hosts
+  hosts=$(proxysql_exec $LINENO "SELECT DISTINCT hostname,port FROM mysql_servers WHERE comment<>'SLAVEREAD' AND hostgroup_id in ($WRITE_HOSTGROUP_ID, $READ_HOSTGROUP_ID)")
+  printf "$hosts" | while read server port || [[ -n $port ]]
+  do
+    debug $LINENO "Trying to contact $server:$port..."
+    slave_exec "$LINENO" "$server" "$port" "select @@port" 1>/dev/null 2>>${DEBUG_ERR_FILE}
+    if [[ $? -eq 0 ]]; then
+      printf "$server $port"
+      return 0
+    fi
+  done
+
+  # No cluster host available (cannot contact any)
+  return 1
+}
+
+function main() {
+  # Monitoring user needs 'REPLICATION CLIENT' privilege
+  log $LINENO "###### Galera Cluster status ######"
+  if [[ -n $LOG_TEXT ]]; then
+    log $LINENO "Extra notes        : $LOG_TEXT"
+  fi
+  debug $LINENO "write hostgroup id : $WRITE_HOSTGROUP_ID"
+  debug $LINENO "read hostgroup id  : $READ_HOSTGROUP_ID"
+  debug $LINENO "mode               : $MODE"
+
+  CLUSTER_USERNAME=$(proxysql_exec $LINENO "SELECT variable_value FROM global_variables WHERE variable_name='mysql-monitor_username'")
+  check_cmd $LINENO $? "Could not retrieve cluster login info from ProxySQL. Please check ProxySQL login credentials"
+
+  CLUSTER_PASSWORD=$(proxysql_exec $LINENO "SELECT variable_value FROM global_variables WHERE variable_name='mysql-monitor_password'" "hide_output")
+  check_cmd $LINENO $? "Could not retrieve cluster login info from ProxySQL. Please check ProxySQL login credentials"
+
+  CLUSTER_TIMEOUT=$(proxysql_exec $LINENO "SELECT MAX(MAX(interval_ms / 1000 - 1, 1)) FROM scheduler")
+
+  local cluster_host_info
+  cluster_host_info=$(find_online_cluster_host)
+
+  local host=""
+  local port=""
+  if [[ -n $cluster_host_info ]]; then
+    host=$(echo $cluster_host_info | awk '{ print $1 }')
+    port=$(echo $cluster_host_info | awk '{ print $2 }')
+  fi
+
+  update_cluster "$host" "$port"
+  mode_change_check
+
+  if [ $CHECK_STATUS -eq 0 ]; then
+    if [[ -n $cluster_host_info ]]; then
+      log $LINENO "Galera Cluster membership looks good"
+    else
+      log $LINENO "Galera Cluster is offline!"
+    fi
+  else
+    echo "1" > ${RELOAD_CHECK_FILE}
+    log $LINENO "###### MYSQL SERVERS was updated ######"
+  fi
+}
+
+
+#-------------------------------------------------------------------------------
+#
+# Step 4 : Begin script execution
+#
+
+parse_args "$@"
+debug $LINENO "#### START PROXYSQL NODE MONITOR"
+main
+debug $LINENO "#### END PROXYSQL NODE MONITOR"
+
+exit 0

dev/provisioning/ansible/roles/proxysql/handlers/main.yml

@@ -0,0 +1,6 @@
+---
+# handlers file for ansible-role-mariadb
+- name: Restart proxySQL
+  service:
+    name: proxysql
+    state: restarted

dev/provisioning/ansible/roles/proxysql/tasks/config.yml

@@ -0,0 +1,25 @@
+---
+- name: Get MySQL version.
+  command: 'mysql --version'
+  register: mysql_cli_version
+  changed_when: false
+  check_mode: false
+
+- name: Read SQL configuration
+  set_fact:
+    sql_content: "{{ lookup('template', '{{ role_path }}/templates/sql.j2') }}"
+  no_log: true
+
+- name: Integration SQL configuration
+  copy:
+    dest: /tmp/file.sql
+    content: '{{ sql_content }}'
+
+- name: Set ProxySQL admin config
+  shell: "mysql -u admin -padmin -h 127.0.0.1 -P 6032 < /tmp/file.sql"
+  no_log: true
+
+- name: Remove file
+  file:
+    path: /tmp/file.sql
+    state: absent

dev/provisioning/ansible/roles/proxysql/tasks/keepalived.yml

@@ -0,0 +1,27 @@
+- name: Get IP range.
+  shell: "echo {{ network_allowed }} | cut -d'.' --fields=1,2,3"
+  register: result
+
+- name: Get interface name.
+  shell: "ip -4 addr show | grep {{ result.stdout }} | rev | cut -d ' ' -f 1 | rev"
+  register: itfn
+
+- name: Set keepalived_bind_interface.
+  set_fact:
+    keepalived_bind_interface: "{{ itfn.stdout }}"
+
+- name: Integration net.ipv4
+  blockinfile:
+    dest: /etc/sysctl.conf
+    block: |
+      net.ipv4.ip_forward = 1
+      net.ipv4.ip_nonlocal_bind = 1
+
+- name: Ensure keepalived is started and enabled on boot.
+  service: name=keepalived state=started enabled=yes
+
+- name: Ensure keepalived conf is set 
+  template: >
+    src=templates/keepalived.conf.j2
+    dest=/etc/keepalived/keepalived.conf
+

dev/provisioning/ansible/roles/proxysql/tasks/main.yml

@@ -0,0 +1,67 @@
+---
+# tasks file for ansible-role-mariadb
+
+- name: Set db_main host variable
+  set_fact:
+    lbsql_main: "{{ groups['db_lbal_servers'][0] }}"
+    my_service: {"name": "proxysql.service", "source": "systemd", "state": "unknown", "status": "disabled"}
+
+- name: Include OS specific variables.
+  include_vars: "{{ ansible_os_family }}.yml"
+
+- name: collect facts about system services
+  service_facts:
+  register: services_state
+
+- name: Set db_main host variable
+  set_fact:
+    my_service: "{{ ansible_facts.services['proxysql.service'] }}"
+  when: "'proxysql.service' in ansible_facts.services.keys()"
+
+- name: Check proxySQL status
+  debug:
+    var: my_service
+
+- name: Install proxySQL
+  include_tasks: "setup/{{ ansible_os_family }}.yml"
+
+- name: Install proxysql scripts
+  copy:
+    src: "{{ item }}"
+    dest: /var/lib/proxysql/
+    owner: proxysql
+    group: proxysql
+    mode: 0744
+  loop:
+    - "{{ role_path }}/files/proxysql_galera_checker"
+    - "{{ role_path }}/files/proxysql_node_monitor"
+
+- name: Install proxysql admin config
+  template:
+    dest: /etc/proxysql-admin.cnf
+    src: proxysql-admin.cnf.j2
+    mode: 0644
+
+- name: Ensure proxysql_galera_checker.log exists
+  copy:
+    content: ""
+    dest: /var/lib/proxysql/proxysql_galera_checker.log
+    force: no
+    group: proxysql
+    owner: proxysql
+    mode: 0644
+
+- name: Ending if proxySQL is already up and running
+  meta: end_play
+  when: my_service.state != "unknown" and my_service.status != "disabled"
+
+# Configuring ProxySQL through the admin interface
+
+- name: Ensure proxySQL configfile is present
+  include_tasks: "config.yml"
+
+# Or Configuring ProxySQL through the config file
+
+    #- name: Ensure proxySQL configfile is present
+    #include_tasks: "config_file.yml"
+

dev/provisioning/ansible/roles/proxysql/tasks/setup/RedHat.yml

@@ -0,0 +1,24 @@
+---
+- name: Install all the {{ ansible_distribution }} mariadb packages
+  dnf:
+    name: "{{ mariadb_packages }}"
+    state: present
+
+- name: Download ProxySQL
+  get_url:
+    url:  "{{ proxySQL_url }}"
+    dest: /tmp/{{ proxySQL_rpm }}
+    checksum: "sha256:{{ proxySQL_sha256 }}"
+
+- name: Install ProxySQL 
+  yum:
+    name: 
+      - /tmp/{{ proxySQL_rpm }}
+      - sysbench
+    state: present
+
+- name: ProxySQL service
+  service:
+    name: proxysql
+    state: started
+    enabled: yes 

dev/provisioning/ansible/roles/proxysql/tasks/setup/Suse.yml

@@ -0,0 +1,13 @@
+---
+# Install mariadb
+
+- name: Install all the Suse mariadb packages
+  zypper:
+    name: "{{ mariadb_packages }}"
+    state: present
+
+- name: Mariadb service
+  service:
+    name: "{{ mariadb_service }}"
+    state: started
+    enabled: yes

dev/provisioning/ansible/roles/proxysql/templates/proxysql-admin.cnf.j2

@@ -0,0 +1,21 @@
+# proxysql admin interface credentials.
+export PROXYSQL_USERNAME='admin'
+export PROXYSQL_PASSWORD='AdminStrongPassword'
+export PROXYSQL_HOSTNAME='localhost'
+export PROXYSQL_PORT='6032'
+export PROXYSQL_DATADIR='/var/lib/proxysql'
+
+
+# ProxySQL read/write hostgroup
+export HOSTGROUP_WRITER_ID=10
+export HOSTGROUP_READER_ID=11
+
+# Writer-is-reader configuration
+export WRITER_IS_READER="ondemand"
+
+# ProxySQL read/write configuration mode : loadbal|singlewrite
+export MODE="singlewrite"
+
+# max_connections default (used only when INSERTing a new mysql_servers entry)
+export MAX_CONNECTIONS=1000
+

dev/provisioning/ansible/roles/proxysql/templates/sql.j2

@@ -0,0 +1,25 @@
+UPDATE global_variables SET variable_value='admin:AdminStrongPassword' WHERE variable_name='admin-admin_credentials';
+LOAD ADMIN VARIABLES TO RUNTIME;
+SAVE ADMIN VARIABLES TO DISK;
+{%for host in groups['db_servers']%}
+  {% if loop.index == 1 %}
+INSERT INTO mysql_servers(hostgroup_id,hostname,port,weight,comment) VALUES (10,'{{ hostvars[host]['ansible_host'] }}',3306,1000000,'WRITE');
+  {% else %}
+INSERT INTO mysql_servers(hostgroup_id,hostname,port,weight,comment) VALUES (11,'{{ hostvars[host]['ansible_host'] }}',3306,1000,'READ');
+  {% endif %}
+{% endfor %}
+LOAD mysql servers TO RUNTIME;
+SAVE mysql servers TO DISK;
+INSERT INTO mysql_query_rules(rule_id,active,match_pattern,destination_hostgroup,apply) VALUES (1,1,'^SELECT.*FOR UPDATE$',10,1), (2,1,'^SELECT',11,1);
+LOAD MYSQL query rules TO RUNTIME;
+SAVE MYSQL query rules TO DISK;
+INSERT INTO mysql_users(username,password,default_hostgroup) VALUES ('nextcloudb','secret',10);
+LOAD mysql users TO RUNTIME;
+SAVE mysql users TO DISK;
+UPDATE global_variables SET variable_value='monitor' WHERE variable_name='mysql-monitor_username';
+UPDATE global_variables SET variable_value='MonitoringPassword' WHERE variable_name='mysql-monitor_password';
+LOAD MYSQL VARIABLES TO RUNTIME;
+SAVE MYSQL VARIABLES TO DISK;
+INSERT INTO scheduler(active,interval_ms,filename,arg1,comment) VALUES (1,10000,'/var/lib/proxysql/proxysql_galera_checker','--config-file=/etc/proxysql-admin.cnf --write-hg=10 --read-hg=11 --writer-count=1 --mode=singlewrite --log=/var/lib/proxysql/proxysql_galera_checker.log', 'gaclunc');
+LOAD SCHEDULER TO RUNTIME;
+SAVE SCHEDULER TO DISK;

dev/provisioning/ansible/roles/proxysql/templates/sql_old.j2

@@ -0,0 +1,26 @@
+UPDATE global_variables SET variable_value='admin:AdminStrongPassword' WHERE variable_name='admin-admin_credentials';
+LOAD ADMIN VARIABLES TO RUNTIME;
+SAVE ADMIN VARIABLES TO DISK;
+{%for host in groups['db_servers']%}
+  {% if loop.index == 1 %}
+INSERT INTO mysql_servers(hostgroup_id,hostname,port,weight,comment) VALUES (10,'{{ hostvars[host]['ansible_host'] }}',3306,1000000,'WRITE');
+  {% else %}
+INSERT INTO mysql_servers(hostgroup_id,hostname,port,weight,comment) VALUES (11,'{{ hostvars[host]['ansible_host'] }}',3306,1000,'READ');
+  {% endif %}
+{% endfor %}
+LOAD mysql servers TO RUNTIME;
+SAVE mysql servers TO DISK;
+INSERT INTO mysql_query_rules(rule_id,active,match_pattern,destination_hostgroup,apply) VALUES (1,1,'^SELECT.*FOR UPDATE$',10,1), (2,1,'^SELECT',11,1);
+LOAD MYSQL query rules TO RUNTIME;
+SAVE MYSQL query rules TO DISK;
+INSERT INTO mysql_users(username,password,default_hostgroup) VALUES ('sbuser','sbpass',10);
+LOAD mysql users TO RUNTIME;
+SAVE mysql users TO DISK;
+UPDATE global_variables SET variable_value='monitor' WHERE variable_name='mysql-monitor_username';
+UPDATE global_variables SET variable_value='MonitoringPassword' WHERE variable_name='mysql-monitor_password';
+UPDATE global_variables SET variable_value='2000' WHERE variable_name IN ('mysql-monitor_connect_interval','mysql-monitor_ping_interval','mysql-monitor_read_only_interval');
+LOAD MYSQL VARIABLES TO RUNTIME;
+SAVE MYSQL VARIABLES TO DISK;
+INSERT INTO scheduler(active,interval_ms,filename,arg1,comment) VALUES (1,5000,'/var/lib/proxysql/proxysql_galera_checker','--config-file=/etc/proxysql-admin.cnf --write-hg=10 --read-hg=11 --writer-count=1 --mode=singlewrite --log=/var/lib/proxysql/proxysql_galera_checker.log', 'gaclunc');
+LOAD SCHEDULER TO RUNTIME;
+SAVE SCHEDULER TO DISK;

dev/provisioning/ansible/roles/proxysql/vars/RedHat.yml

@@ -0,0 +1,10 @@
+mariadb_packages:
+  - mariadb
+  - mariadb-libs
+  - MySQL-python
+  - perl-DBD-MySQL
+
+proxySQL_version: 2.4.4
+proxySQL_rpm: "proxysql-{{ proxySQL_version }}-1-centos7.x86_64.rpm"
+proxySQL_url: "https://github.com/sysown/proxysql/releases/download/v{{ proxySQL_version }}/{{ proxySQL_rpm }}"
+proxySQL_sha256: "d739e89ebad2f2e2f614875f1b3bd5fcc0276406dd01cd580c5aa80e35a04600"

dev/provisioning/ansible/roles/proxysql/vars/Suse.yml

@@ -0,0 +1,9 @@
+mariadb_packages:
+  - mariadb
+  - python3-PyMySQL
+
+mariadb_service: mariadb
+mariadb_config_file: /etc/my.cnf.d/server.cnf
+mariadb_socket: /run/mysql/mysql.sock
+mariadb_config_file_owner: root
+mariadb_config_file_group: root

dev/provisioning/ansible/roles/redis/defaults/main.yml

@@ -1,12 +1,20 @@
 ---
-redis_port: 6379
+redis_daemon: "redis"
+redis_port: "6379"
 redis_bind_interface: "0.0.0.0" # overload from role
-redis_unixsocket: ''
-redis_timeout: 300
+
+redis_conf_path: "/etc/{{ redis_daemon }}/{{ redis_daemon }}.conf"
+redis_conf_mode: 0644
+redis_daemonize: "yes"
+redis_protected_mode: "yes"
 
 redis_loglevel: "notice"
-redis_logfile: /var/log/redis/redis-server.log
+redis_logfile: "/var/log/{{ redis_daemon }}/{{ redis_daemon }}-server.log"
+redis_pidfile: "/var/run/{{ redis_daemon }}/{{ redis_daemon }}-server.pid"
 
+#redis_unixsocket: "/var/run/redis/redis.sock"
+redis_unixsocket: ""
+redis_timeout: 0
 redis_databases: 16
 
 # Set to an empty set to disable persistence (saving the DB to disk).
@@ -16,35 +24,19 @@ redis_save:
   - 60 10000
 
 redis_rdbcompression: "yes"
-redis_dbfilename: dump.rdb
-redis_dbdir: /var/lib/redis
-
-redis_maxmemory: 0
-redis_maxmemory_policy: "noeviction"
-redis_maxmemory_samples: 5
-
-redis_appendonly: "no"
-redis_appendfsync: "everysec"
-
-# Add extra include files for local configuration/overrides.
-redis_includes: []
-
-# Require authentication to Redis with a password.
-redis_requirepass: ""
-
-# Disable certain Redis commands for security reasons.
-redis_disabled_commands: []
-#  - FLUSHDB
-#  - FLUSHALL
-#  - KEYS
-#  - PEXPIRE
-#  - DEL
-#  - CONFIG
-#  - SHUTDOWN
-#  - BGREWRITEAOF
-#  - BGSAVE
-#  - SAVE
-#  - SPOP
-#  - SREM
-#  - RENAME
-#  - DEBUG
+redis_dbfilename: "dump.rdb"
+redis_dbdir: "/var/lib/{{ redis_daemon }}"
+
+need_sentinel: false
+
+redis_sentinel_daemon: "redis-sentinel"
+redis_sentinel_port: 26379
+redis_sentinel_bind_interface: "0.0.0.0" # overload from role
+
+redis_sentinel_conf_path: "/etc/redis-sentinel.conf"
+redis_sentinel_conf_mode: 0644
+
+redis_sentinel_pidfile: "/var/run/redis-sentinel.pid"
+redis_sentinel_logfile: "/var/log/redis/sentinel.log"
+
+redis_sentinel_down_after_milliseconds: 10000

dev/provisioning/ansible/roles/redis/files/redis_ha.te

@@ -0,0 +1,11 @@
+# create new
+module redis_ha 1.0;
+
+require {
+        type etc_t;
+        type redis_t;
+        class file write;
+}
+
+#============= redis_t ==============
+allow redis_t etc_t:file write;

dev/provisioning/ansible/roles/redis/handlers/main.yml

@@ -1,11 +1,21 @@
 ---
 
-- name: start redis
+- name: "start {{ redis_daemon }}"
   ansible.builtin.service:
     name: "{{ redis_daemon }}"
     state: started
 
-- name: restart redis
+- name: "restart {{ redis_daemon }}"
   ansible.builtin.service:
     name: "{{ redis_daemon }}"
     state: restarted
+
+- name: start redis-sentinel
+  ansible.builtin.service:
+    name: "{{ redis_sentinel_daemon }}"
+    state: started
+
+- name: restart redis-sentinel
+  ansible.builtin.service:
+    name: "{{ redis_sentinel_daemon }}"
+    state: restarted

dev/provisioning/ansible/roles/redis/tasks/main.yml

@@ -1,4 +1,14 @@
 ---
+- name: Set redis_list_servers variable
+  block:
+    - name: Try group web_servers
+      set_fact:
+        redis_list_servers: "{{ groups['web_servers'] | list }}"
+  rescue:
+    - name: Try group test_servers
+      set_fact:
+        redis_list_servers: "{{ groups['test_servers'] | list }}"
+
 # include os specific tasks
 - include_tasks: "setup/{{ ansible_os_family }}.yml"
 
@@ -8,12 +18,58 @@
     state: directory
     mode: 0755
 
-- name: Ensure Redis is configured.
+      #- name: Get the IP adress
+      #  set_fact:
+      #    curip: "{{ ansible_all_ipv4_addresses | select('search', private_network) }}"
+
+- name: Ensure Redis is configured
   template:
-    src: redis.conf.j2
+    src: "{{ redis_daemon }}.conf.j2"
     dest: "{{ redis_conf_path }}"
     mode: "{{ redis_conf_mode }}"
-  notify: restart redis
+  notify: restart {{ redis_daemon }}
+
+- name: Checking overcommit memory
+  shell: 'cat /proc/sys/vm/overcommit_memory'
+  register: overcommit_memory
+  changed_when: false
+  failed_when: false
+
+- name: Overcommit memory
+  sysctl:
+    name: vm.overcommit_memory
+    value: '1'
+    state: present
+  when: '"1" not in overcommit_memory.stdout'
+
+- name: Increase max connections
+  sysctl:
+    name: net.core.somaxconn
+    value: '511'
+    state: present
+
+- name: Disable THP
+  shell: echo never > /sys/kernel/mm/transparent_hugepage/enabled
+
+- name: Allow HAProxy to start
+  sysctl:
+    name: net.ipv4.ip_nonlocal_bind
+    value: '1'
+    sysctl_set: yes
+    state: present
+    reload: yes
 
 - name: Ensure Redis is running and enabled on boot.
   service: "name={{ redis_daemon }} state=started enabled=yes"
+
+- name: Ensure Redis Sentinel is configured
+  template:
+    src: redis_sentinel.conf.j2
+    dest: "{{ redis_sentinel_conf_path }}"
+    mode: "{{ redis_sentinel_conf_mode }}"
+  notify: restart {{ redis_sentinel_daemon }}
+  when: need_sentinel
+
+- name: Ensure Redis Sentinel is running and enabled on boot.
+  service: "name={{ redis_sentinel_daemon }} state=started enabled=yes"
+  when: need_sentinel

dev/provisioning/ansible/roles/redis/tasks/setup/CentOS.yml

@@ -1,21 +1,50 @@
 ---
 # CentOS related tasks
 
-#- name: update os
-#  dnf:
-#    name: '*'
-#    update_cache: true
-#    state: latest
-#  when: not debug_speed_check
+- name: Add Remi repo
+  dnf:
+    name: https://rpms.remirepo.net/enterprise/remi-release-{{ ansible_distribution_major_version|int }}.rpm
+    state: latest
+    disable_gpg_check: yes
+    validate_certs: no
+  when: redis_daemon == "redis"
 
 - name: Ensure Redis is installed.
   dnf:
     name: redis
     state: present
-    enablerepo: epel
+    enablerepo: remi
+  when: redis_daemon == "redis"
+
+- name: Add keydb repo
+  dnf:
+    name: https://download.keydb.dev/pkg/open_source/rpm/centos7/x86_64/keydb-latest-1.el7.x86_64.rpm
+    disable_gpg_check: yes 
+    validate_certs: no
+  when: redis_daemon == "keydb"
+
+- name: Ensure KeyDB is installed.
+  dnf:
+    name: keydb
+    state: present
+  when: redis_daemon == "keydb"
+
+- name: Install redis_ha.te script
+  copy:
+    src: "{{ role_path }}/files/redis_ha.te"
+    dest: /tmp/redis_ha.te
+    mode: 0644
+
+- name: Selinux... checkmodule redis_ha.te
+  shell: checkmodule -m -M -o /tmp/redis_ha.mod /tmp/redis_ha.te
+
+- name: Selinux... semodule_package redis_ha.mod
+  shell: semodule_package --outfile /tmp/redis_ha.pp --module /tmp/redis_ha.mod
+
+- name: Selinux... semodule redis_ha.pp
+  shell: semodule -i /tmp/redis_ha.pp
 
-- name: Define Redis vars.
-  set_fact:
-    redis_conf_path: "/etc/redis.conf"
-    redis_conf_mode: 0644
-    redis_daemon: "redis" 
+- name: Selinux... add custom port
+  shell: semanage port -a -t redis_port_t -p tcp {{ redis_port }}
+  when: redis_port != "6379"
+  ignore_errors: yes

dev/provisioning/ansible/roles/redis/templates/keydb.conf.j2

@@ -0,0 +1,88 @@
+# {{ ansible_managed }}
+
+bind 0.0.0.0
+protected-mode {{ redis_protected_mode }}
+port {{ redis_port }}
+
+timeout {{ redis_timeout }}
+daemonize {{ redis_daemonize }}
+pidfile {{ redis_pidfile }}
+loglevel {{ redis_loglevel }}
+logfile {{ redis_logfile }}
+databases {{ redis_databases }}
+{% for save in redis_save %}
+save {{ save }}
+{% endfor %}
+rdbcompression {{ redis_rdbcompression }}
+dbfilename {{ redis_dbfilename }}
+dir {{ redis_dbdir }}
+tcp-backlog 511
+tcp-keepalive 300
+supervised no
+always-show-logo yes
+set-proc-title yes
+proc-title-template "{title} {listen-addr} {server-mode}"
+stop-writes-on-bgsave-error yes
+rdbchecksum yes
+rdb-del-sync-files no
+replica-serve-stale-data yes
+replica-read-only yes
+repl-diskless-sync no
+repl-diskless-sync-delay 5
+repl-diskless-load disabled
+repl-disable-tcp-nodelay no
+replica-priority 100
+acllog-max-len 128
+lazyfree-lazy-eviction no
+lazyfree-lazy-expire no
+lazyfree-lazy-server-del no
+replica-lazy-flush no
+lazyfree-lazy-user-del no
+lazyfree-lazy-user-flush no
+oom-score-adj no
+oom-score-adj-values 0 200 800
+disable-thp yes
+appendonly no
+appendfilename "appendonly.aof"
+appendfsync everysec
+no-appendfsync-on-rewrite no
+auto-aof-rewrite-percentage 100
+auto-aof-rewrite-min-size 64mb
+aof-load-truncated yes
+aof-use-rdb-preamble yes
+lua-time-limit 5000
+slowlog-log-slower-than 10000
+slowlog-max-len 128
+latency-monitor-threshold 0
+notify-keyspace-events ""
+hash-max-ziplist-entries 512
+hash-max-ziplist-value 64
+list-max-ziplist-size -2
+list-compress-depth 0
+set-max-intset-entries 512
+zset-max-ziplist-entries 128
+zset-max-ziplist-value 64
+hll-sparse-max-bytes 3000
+stream-node-max-bytes 4096
+stream-node-max-entries 100
+activerehashing yes
+client-output-buffer-limit normal 0 0 0
+client-output-buffer-limit replica 256mb 64mb 60
+client-output-buffer-limit pubsub 32mb 8mb 60
+hz 10
+dynamic-hz yes
+aof-rewrite-incremental-fsync yes
+rdb-save-incremental-fsync yes
+jemalloc-bg-thread yes
+server-threads 2
+replica-weighting-factor 2
+
+{% if redis_list_servers|length == 2 %}
+multi-master yes
+active-replica yes
+{% if ansible_fqdn != redis_list_servers[0] %}
+replicaof {{ hostvars[redis_list_servers[0]]['ansible_host'] }} {{ redis_port }}
+{% else %}
+replicaof {{ hostvars[redis_list_servers[1]]['ansible_host'] }} {{ redis_port }}
+{% endif %}
+{% endif %}

dev/provisioning/ansible/roles/redis/templates/redis.conf.j2

@@ -1,12 +1,19 @@
 # {{ ansible_managed }}
 
-daemonize yes
-pidfile /var/run/redis/{{ redis_daemon }}.pid
+pidfile /var/run/{{ redis_daemon }}.pid
 port {{ redis_port }}
-bind {{ redis_bind_interface }}
+
+bind 0.0.0.0
+
+{% if groups['redis_servers']|length > 1 %}
+{% if ansible_fqdn != groups['redis_servers'][0] %}
+slaveof {{ groups['redis_servers'][0] }} {{ redis_port }}
+{% endif %}
+{% endif %}
 
 {% if redis_unixsocket %}
 unixsocket {{ redis_unixsocket }}
+unixsocketperm 775
 {% endif %}
 
 timeout {{ redis_timeout }}
@@ -14,12 +21,6 @@ timeout {{ redis_timeout }}
 loglevel {{ redis_loglevel }}
 logfile {{ redis_logfile }}
 
-# To enable logging to the system logger, just set 'syslog-enabled' to yes,
-# and optionally update the other syslog parameters to suit your needs.
-# syslog-enabled no
-# syslog-ident redis
-# syslog-facility local0
-
 databases {{ redis_databases }}
 
 {% for save in redis_save %}
@@ -30,26 +31,53 @@ rdbcompression {{ redis_rdbcompression }}
 dbfilename {{ redis_dbfilename }}
 dir {{ redis_dbdir }}
 
-# maxclients 128
-
-{% if redis_maxmemory %}
-maxmemory {{ redis_maxmemory }}
-maxmemory-policy {{ redis_maxmemory_policy }}
-maxmemory-samples {{ redis_maxmemory_samples }}
-{% endif %}
-
-appendonly {{ redis_appendonly }}
-appendfsync {{ redis_appendfsync }}
+protected-mode {{ redis_protected_mode }}
+tcp-backlog 511
+tcp-keepalive 300
+daemonize {{ redis_daemonize }}
+supervised no
+always-show-logo yes
+stop-writes-on-bgsave-error yes
+rdbchecksum yes
+replica-serve-stale-data yes
+replica-read-only yes
+repl-diskless-sync no
+repl-diskless-sync-delay 5
+repl-disable-tcp-nodelay no
+replica-priority 100
+lazyfree-lazy-eviction no
+lazyfree-lazy-expire no
+lazyfree-lazy-server-del no
+replica-lazy-flush no
+appendonly no
+appendfilename "appendonly.aof"
+appendfsync everysec
 no-appendfsync-on-rewrite no
-
-{% for include in redis_includes %}
-include {{ include }}
-{% endfor %}
-
-{% if redis_requirepass %}
-requirepass {{ redis_requirepass }}
-{% endif %}
-
-{% for redis_disabled_command in redis_disabled_commands %}
-rename-command {{ redis_disabled_command }} ""
-{% endfor %}
+auto-aof-rewrite-percentage 100
+auto-aof-rewrite-min-size 64mb
+aof-load-truncated yes
+aof-use-rdb-preamble yes
+lua-time-limit 5000
+slowlog-log-slower-than 10000
+slowlog-max-len 128
+latency-monitor-threshold 0
+notify-keyspace-events ""
+hash-max-ziplist-entries 512
+hash-max-ziplist-value 64
+list-max-ziplist-size -2
+list-compress-depth 0
+set-max-intset-entries 512
+zset-max-ziplist-entries 128
+zset-max-ziplist-value 64
+hll-sparse-max-bytes 3000
+stream-node-max-bytes 4096
+stream-node-max-entries 100
+activerehashing yes
+client-output-buffer-limit normal 0 0 0
+client-output-buffer-limit replica 256mb 64mb 60
+client-output-buffer-limit pubsub 32mb 8mb 60
+hz 10
+dynamic-hz yes
+aof-rewrite-incremental-fsync yes
+rdb-save-incremental-fsync yes
+maxclients 512

dev/provisioning/ansible/roles/redis/templates/redis.conf.j2_new

@@ -0,0 +1,108 @@
+daemonize yes
+pidfile /var/run/redis/{{ redis_daemon }}.pid
+port {{ redis_port }}
+
+# {% if ansible_eth0.ipv4.address == hostvars[groups['lan_rd'][0]]['ansible_host'] %}
+# bind 127.0.0.1 {{ ansible_default_ipv4.address }}
+# {% else %}
+# bind 127.0.0.1 {{ ansible_default_ipv4.address }}
+# slaveof {{ hostvars[groups['lan_rd'][0]]['ansible_host'] }} 6379
+# {% endif %}
+bind {{ redis_bind_interface }}
+
+{% if redis_unixsocket %}
+unixsocket {{ redis_unixsocket }}
+{% endif %}
+
+timeout {{ redis_timeout }}
+
+loglevel {{ redis_loglevel }}
+logfile {{ redis_logfile }}
+
+# To enable logging to the system logger, just set 'syslog-enabled' to yes,
+# and optionally update the other syslog parameters to suit your needs.
+# syslog-enabled no
+# syslog-ident redis
+# syslog-facility local0
+
+databases {{ redis_databases }}
+
+{% for save in redis_save %}
+save {{ save }}
+{% endfor %}
+
+rdbcompression {{ redis_rdbcompression }}
+dbfilename {{ redis_dbfilename }}
+dir {{ redis_dbdir }}
+
+# maxclients 128
+
+{% if redis_maxmemory %}
+maxmemory {{ redis_maxmemory }}
+maxmemory-policy {{ redis_maxmemory_policy }}
+maxmemory-samples {{ redis_maxmemory_samples }}
+{% endif %}
+
+appendonly {{ redis_appendonly }}
+appendfsync {{ redis_appendfsync }}
+no-appendfsync-on-rewrite no
+
+{% for include in redis_includes %}
+include {{ include }}
+{% endfor %}
+
+{% if redis_requirepass %}
+requirepass {{ redis_requirepass }}
+{% endif %}
+
+{% for redis_disabled_command in redis_disabled_commands %}
+rename-command {{ redis_disabled_command }} ""
+{% endfor %}
+
+protected-mode yes
+unixsocketperm 775
+maxclients 512
+tcp-backlog 511
+tcp-keepalive 300
+supervised no
+always-show-logo yes
+stop-writes-on-bgsave-error yes
+rdbchecksum yes
+replica-serve-stale-data yes
+replica-read-only yes
+repl-diskless-sync no
+repl-diskless-sync-delay 5
+repl-disable-tcp-nodelay no
+replica-priority 100
+lazyfree-lazy-eviction no
+lazyfree-lazy-expire no
+lazyfree-lazy-server-del no
+replica-lazy-flush no
+appendfilename "appendonly.aof"
+auto-aof-rewrite-percentage 100
+auto-aof-rewrite-min-size 64mb
+aof-load-truncated yes
+aof-use-rdb-preamble yes
+lua-time-limit 5000
+slowlog-log-slower-than 10000
+slowlog-max-len 128
+latency-monitor-threshold 0
+notify-keyspace-events ""
+hash-max-ziplist-entries 512
+hash-max-ziplist-value 64
+list-max-ziplist-size -2
+list-compress-depth 0
+set-max-intset-entries 512
+zset-max-ziplist-entries 128
+zset-max-ziplist-value 64
+hll-sparse-max-bytes 3000
+stream-node-max-bytes 4096
+stream-node-max-entries 100
+activerehashing yes
+client-output-buffer-limit normal 0 0 0
+client-output-buffer-limit replica 256mb 64mb 60
+client-output-buffer-limit pubsub 32mb 8mb 60
+hz 10
+dynamic-hz yes
+aof-rewrite-incremental-fsync yes
+rdb-save-incremental-fsync yes

dev/provisioning/ansible/roles/redis/templates/redis.orig

@@ -0,0 +1,35 @@
+# Ansible managed
+
+daemonize yes
+pidfile /var/run/redis/redis.pid
+port 6379
+bind 0.0.0.0
+
+
+timeout 300
+
+loglevel notice
+logfile /var/log/redis/redis-server.log
+
+# To enable logging to the system logger, just set 'syslog-enabled' to yes,
+# and optionally update the other syslog parameters to suit your needs.
+# syslog-enabled no
+# syslog-ident redis
+# syslog-facility local0
+
+databases 16
+
+save 900 1
+save 300 10
+save 60 10000
+
+rdbcompression yes
+dbfilename dump.rdb
+dir /var/lib/redis
+
+# maxclients 128
+
+
+appendonly no
+appendfsync everysec
+no-appendfsync-on-rewrite no

dev/provisioning/ansible/roles/redis/templates/redis_sentinel.conf.j2

@@ -0,0 +1,348 @@
+# Example sentinel.conf
+
+bind {{ redis_sentinel_bind_interface }}
+
+# By default protected mode is disabled in sentinel mode. Sentinel is reachable
+# from interfaces different than localhost. Make sure the sentinel instance is
+# protected from the outside world via firewalling or other means.
+protected-mode no
+
+# port <sentinel-port>
+# The port that this sentinel instance will run on
+port {{ redis_sentinel_port }}
+
+# By default Redis Sentinel does not run as a daemon. Use 'yes' if you need it.
+# Note that Redis will write a pid file in /var/run/redis-sentinel.pid when
+# daemonized.
+daemonize no
+
+# When running daemonized, Redis Sentinel writes a pid file in
+# /var/run/redis-sentinel.pid by default. You can specify a custom pid file
+# location here.
+pidfile {{ redis_sentinel_pidfile }}
+
+# Specify the log file name. Also the empty string can be used to force
+# Sentinel to log on the standard output. Note that if you use standard
+# output for logging but daemonize, logs will be sent to /dev/null
+logfile {{ redis_sentinel_logfile }}
+
+# sentinel announce-ip <ip>
+# sentinel announce-port <port>
+#
+# The above two configuration directives are useful in environments where,
+# because of NAT, Sentinel is reachable from outside via a non-local address.
+#
+# When announce-ip is provided, the Sentinel will claim the specified IP address
+# in HELLO messages used to gossip its presence, instead of auto-detecting the
+# local address as it usually does.
+#
+# Similarly when announce-port is provided and is valid and non-zero, Sentinel
+# will announce the specified TCP port.
+#
+# The two options don't need to be used together, if only announce-ip is
+# provided, the Sentinel will announce the specified IP and the server port
+# as specified by the "port" option. If only announce-port is provided, the
+# Sentinel will announce the auto-detected local IP and the specified port.
+#
+# Example:
+#
+# sentinel announce-ip 1.2.3.4
+
+# dir <working-directory>
+# Every long running process should have a well-defined working directory.
+# For Redis Sentinel to chdir to /tmp at startup is the simplest thing
+# for the process to don't interfere with administrative tasks such as
+# unmounting filesystems.
+dir /tmp
+
+# sentinel monitor <master-name> <ip> <redis-port> <quorum>
+#
+# Tells Sentinel to monitor this master, and to consider it in O_DOWN
+# (Objectively Down) state only if at least <quorum> sentinels agree.
+#
+# Note that whatever is the ODOWN quorum, a Sentinel will require to
+# be elected by the majority of the known Sentinels in order to
+# start a failover, so no failover can be performed in minority.
+#
+# Replicas are auto-discovered, so you don't need to specify replicas in
+# any way. Sentinel itself will rewrite this configuration file adding
+# the replicas using additional configuration options.
+# Also note that the configuration file is rewritten when a
+# replica is promoted to master.
+#
+# Note: master name should not include special characters or spaces.
+# The valid charset is A-z 0-9 and the three characters ".-_".
+{% if ansible_fqdn != groups['redis_servers'][0] %}
+sentinel monitor mymaster {{ groups['redis_servers'][0] }} {{ redis_port }} 2
+{% else %}
+sentinel monitor mymaster 127.0.0.1 {{ redis_port }} 2
+{% endif %}
+
+# sentinel auth-pass <master-name> <password>
+#
+# Set the password to use to authenticate with the master and replicas.
+# Useful if there is a password set in the Redis instances to monitor.
+#
+# Note that the master password is also used for replicas, so it is not
+# possible to set a different password in masters and replicas instances
+# if you want to be able to monitor these instances with Sentinel.
+#
+# However you can have Redis instances without the authentication enabled
+# mixed with Redis instances requiring the authentication (as long as the
+# password set is the same for all the instances requiring the password) as
+# the AUTH command will have no effect in Redis instances with authentication
+# switched off.
+#
+# Example:
+#
+# sentinel auth-pass mymaster MySUPER--secret-0123passw0rd
+
+# sentinel auth-user <master-name> <username>
+#
+# This is useful in order to authenticate to instances having ACL capabilities,
+# that is, running Redis 6.0 or greater. When just auth-pass is provided the
+# Sentinel instance will authenticate to Redis using the old "AUTH <pass>"
+# method. When also an username is provided, it will use "AUTH <user> <pass>".
+# In the Redis servers side, the ACL to provide just minimal access to
+# Sentinel instances, should be configured along the following lines:
+#
+#     user sentinel-user >somepassword +client +subscribe +publish \
+#                        +ping +info +multi +slaveof +config +client +exec on
+
+# sentinel down-after-milliseconds <master-name> <milliseconds>
+#
+# Number of milliseconds the master (or any attached replica or sentinel) should
+# be unreachable (as in, not acceptable reply to PING, continuously, for the
+# specified period) in order to consider it in S_DOWN state (Subjectively
+# Down).
+#
+# Default is 30 seconds.
+sentinel down-after-milliseconds mymaster {{ redis_sentinel_down_after_milliseconds }}
+
+# IMPORTANT NOTE: starting with Redis 6.2 ACL capability is supported for
+# Sentinel mode, please refer to the Redis website https://redis.io/topics/acl
+# for more details.
+
+# Sentinel's ACL users are defined in the following format:
+#
+#   user <username> ... acl rules ...
+#
+# For example:
+#
+#   user worker +@admin +@connection ~* on >ffa9203c493aa99
+#
+# For more information about ACL configuration please refer to the Redis
+# website at https://redis.io/topics/acl and redis server configuration 
+# template redis.conf.
+
+# ACL LOG
+#
+# The ACL Log tracks failed commands and authentication events associated
+# with ACLs. The ACL Log is useful to troubleshoot failed commands blocked 
+# by ACLs. The ACL Log is stored in memory. You can reclaim memory with 
+# ACL LOG RESET. Define the maximum entry length of the ACL Log below.
+acllog-max-len 128
+
+# Using an external ACL file
+#
+# Instead of configuring users here in this file, it is possible to use
+# a stand-alone file just listing users. The two methods cannot be mixed:
+# if you configure users here and at the same time you activate the external
+# ACL file, the server will refuse to start.
+#
+# The format of the external ACL user file is exactly the same as the
+# format that is used inside redis.conf to describe users.
+#
+# aclfile /etc/redis/sentinel-users.acl
+
+# requirepass <password>
+#
+# You can configure Sentinel itself to require a password, however when doing
+# so Sentinel will try to authenticate with the same password to all the
+# other Sentinels. So you need to configure all your Sentinels in a given
+# group with the same "requirepass" password. Check the following documentation
+# for more info: https://redis.io/topics/sentinel
+#
+# IMPORTANT NOTE: starting with Redis 6.2 "requirepass" is a compatibility
+# layer on top of the ACL system. The option effect will be just setting
+# the password for the default user. Clients will still authenticate using
+# AUTH <password> as usually, or more explicitly with AUTH default <password>
+# if they follow the new protocol: both will work.
+#
+# New config files are advised to use separate authentication control for
+# incoming connections (via ACL), and for outgoing connections (via
+# sentinel-user and sentinel-pass) 
+#
+# The requirepass is not compatible with aclfile option and the ACL LOAD
+# command, these will cause requirepass to be ignored.
+
+# sentinel sentinel-user <username>
+#
+# You can configure Sentinel to authenticate with other Sentinels with specific
+# user name. 
+
+# sentinel sentinel-pass <password>
+#
+# The password for Sentinel to authenticate with other Sentinels. If sentinel-user
+# is not configured, Sentinel will use 'default' user with sentinel-pass to authenticate.
+
+# sentinel parallel-syncs <master-name> <numreplicas>
+#
+# How many replicas we can reconfigure to point to the new replica simultaneously
+# during the failover. Use a low number if you use the replicas to serve query
+# to avoid that all the replicas will be unreachable at about the same
+# time while performing the synchronization with the master.
+sentinel parallel-syncs mymaster 1
+
+# sentinel failover-timeout <master-name> <milliseconds>
+#
+# Specifies the failover timeout in milliseconds. It is used in many ways:
+#
+# - The time needed to re-start a failover after a previous failover was
+#   already tried against the same master by a given Sentinel, is two
+#   times the failover timeout.
+#
+# - The time needed for a replica replicating to a wrong master according
+#   to a Sentinel current configuration, to be forced to replicate
+#   with the right master, is exactly the failover timeout (counting since
+#   the moment a Sentinel detected the misconfiguration).
+#
+# - The time needed to cancel a failover that is already in progress but
+#   did not produced any configuration change (SLAVEOF NO ONE yet not
+#   acknowledged by the promoted replica).
+#
+# - The maximum time a failover in progress waits for all the replicas to be
+#   reconfigured as replicas of the new master. However even after this time
+#   the replicas will be reconfigured by the Sentinels anyway, but not with
+#   the exact parallel-syncs progression as specified.
+#
+# Default is 3 minutes.
+sentinel failover-timeout mymaster 180000
+
+# SCRIPTS EXECUTION
+#
+# sentinel notification-script and sentinel reconfig-script are used in order
+# to configure scripts that are called to notify the system administrator
+# or to reconfigure clients after a failover. The scripts are executed
+# with the following rules for error handling:
+#
+# If script exits with "1" the execution is retried later (up to a maximum
+# number of times currently set to 10).
+#
+# If script exits with "2" (or an higher value) the script execution is
+# not retried.
+#
+# If script terminates because it receives a signal the behavior is the same
+# as exit code 1.
+#
+# A script has a maximum running time of 60 seconds. After this limit is
+# reached the script is terminated with a SIGKILL and the execution retried.
+
+# NOTIFICATION SCRIPT
+#
+# sentinel notification-script <master-name> <script-path>
+# 
+# Call the specified notification script for any sentinel event that is
+# generated in the WARNING level (for instance -sdown, -odown, and so forth).
+# This script should notify the system administrator via email, SMS, or any
+# other messaging system, that there is something wrong with the monitored
+# Redis systems.
+#
+# The script is called with just two arguments: the first is the event type
+# and the second the event description.
+#
+# The script must exist and be executable in order for sentinel to start if
+# this option is provided.
+#
+# Example:
+#
+# sentinel notification-script mymaster /var/redis/notify.sh
+
+# CLIENTS RECONFIGURATION SCRIPT
+#
+# sentinel client-reconfig-script <master-name> <script-path>
+#
+# When the master changed because of a failover a script can be called in
+# order to perform application-specific tasks to notify the clients that the
+# configuration has changed and the master is at a different address.
+# 
+# The following arguments are passed to the script:
+#
+# <master-name> <role> <state> <from-ip> <from-port> <to-ip> <to-port>
+#
+# <state> is currently always "start"
+# <role> is either "leader" or "observer"
+# 
+# The arguments from-ip, from-port, to-ip, to-port are used to communicate
+# the old address of the master and the new address of the elected replica
+# (now a master).
+#
+# This script should be resistant to multiple invocations.
+#
+# Example:
+#
+# sentinel client-reconfig-script mymaster /var/redis/reconfig.sh
+
+# SECURITY
+#
+# By default SENTINEL SET will not be able to change the notification-script
+# and client-reconfig-script at runtime. This avoids a trivial security issue
+# where clients can set the script to anything and trigger a failover in order
+# to get the program executed.
+
+sentinel deny-scripts-reconfig yes
+
+# REDIS COMMANDS RENAMING (DEPRECATED)
+#
+# WARNING: avoid using this option if possible, instead use ACLs.
+#
+# Sometimes the Redis server has certain commands, that are needed for Sentinel
+# to work correctly, renamed to unguessable strings. This is often the case
+# of CONFIG and SLAVEOF in the context of providers that provide Redis as
+# a service, and don't want the customers to reconfigure the instances outside
+# of the administration console.
+#
+# In such case it is possible to tell Sentinel to use different command names
+# instead of the normal ones. For example if the master "mymaster", and the
+# associated replicas, have "CONFIG" all renamed to "GUESSME", I could use:
+#
+# SENTINEL rename-command mymaster CONFIG GUESSME
+#
+# After such configuration is set, every time Sentinel would use CONFIG it will
+# use GUESSME instead. Note that there is no actual need to respect the command
+# case, so writing "config guessme" is the same in the example above.
+#
+# SENTINEL SET can also be used in order to perform this configuration at runtime.
+#
+# In order to set a command back to its original name (undo the renaming), it
+# is possible to just rename a command to itself:
+#
+# SENTINEL rename-command mymaster CONFIG CONFIG
+
+# HOSTNAMES SUPPORT
+#
+# Normally Sentinel uses only IP addresses and requires SENTINEL MONITOR
+# to specify an IP address. Also, it requires the Redis replica-announce-ip
+# keyword to specify only IP addresses.
+#
+# You may enable hostnames support by enabling resolve-hostnames. Note
+# that you must make sure your DNS is configured properly and that DNS
+# resolution does not introduce very long delays.
+#
+SENTINEL resolve-hostnames no
+
+# When resolve-hostnames is enabled, Sentinel still uses IP addresses
+# when exposing instances to users, configuration files, etc. If you want
+# to retain the hostnames when announced, enable announce-hostnames below.
+#
+SENTINEL announce-hostnames no
+
+# When master_reboot_down_after_period is set to 0, Sentinel does not fail over
+# when receiving a -LOADING response from a master. This was the only supported
+# behavior before version 7.0.
+#
+# Otherwise, Sentinel will use this value as the time (in ms) it is willing to
+# accept a -LOADING response after a master has been rebooted, before failing
+# over.
+
+SENTINEL master-reboot-down-after-period mymaster 0

dev/provisioning/ansible/roles/web_php/defaults/main.yml

@@ -0,0 +1,31 @@
+---
+
+# [WEB CONFIG]
+websrv: "apache2"  # "apache" | "nginx"
+ssl_path: "/etc/{{ ansible_fqdn }}/ssl"
+
+# [PHP CONFIG AND EXTENSIONS]
+php_version: 8.1
+PHP_POST_LIMIT: 10240M
+PHP_UPLOAD_LIMIT: 10240M
+PHP_MAX_FILE: 100
+PHP_MAX_TIME: 1800
+PHP_MEMORY_LIMIT: 512M
+
+redis_daemon: "redis"
+redis_host: "localhost"
+redis_port: "6379"
+
+OPCACHE_MEM_SIZE: 128
+
+APC_SHM_SIZE: 128M
+
+# [PHP-FPM]
+add_php_fpm: true
+enable_php_fpm: false
+pm: "dynamic"
+pm_max_children: 240
+pm_start_servers: 20
+pm_min_spare_servers: 10
+pm_max_spare_servers: 20
+pm_max_requests: 500

dev/provisioning/ansible/roles/web_php/handlers/main.yml

@@ -0,0 +1,25 @@
+---
+- name: start http
+  service:
+    name: "{{ http_service_name }}"
+    state: started
+
+- name: restart http
+  service:
+    name: "{{ http_service_name }}"
+    state: restarted
+
+- name: reload http
+  service:
+    name: "{{ http_service_name }}"
+    state: reloaded
+
+- name: start php-fpm
+  service:
+    name: php{{ php_ver }}-fpm
+    state: started
+
+- name: reload php-fpm
+  service:
+    name: php{{ php_ver }}-fpm
+    state: reloaded

dev/provisioning/ansible/roles/web_php/tasks/main.yml

@@ -0,0 +1,35 @@
+---
+# tasks file
+
+- name: Main... Show Selinux variable
+  debug: var=ansible_selinux
+
+- name: Main... collect facts about system services
+  service_facts:
+  register: services_state
+
+- name: Set my_redis_service host variable
+  set_fact:
+    my_redis_service: "{{ ansible_facts.services['%s.service' % (redis_daemon)] }}"
+  when: ('%s.service' % (redis_daemon)) in ansible_facts.services.keys()
+
+- name: Check Redis status
+  debug:
+    var: my_redis_service
+
+- include_tasks: "web/{{ ansible_os_family }}.yml"
+
+- include_tasks: "php/{{ ansible_os_family }}.yml"
+
+  #- include_tasks: "ssl.yml"
+
+- name: Main... Check Selinux
+  include_tasks: "selinux.yml"
+  when:
+    - (ansible_os_family == "RedHat")
+    - (ansible_selinux.status == "enabled")
+
+- name: Main... Restart {{ http_service_name }} service
+  service:
+    name: "{{ http_service_name }}"
+    state: restarted

dev/provisioning/ansible/roles/nextcloud/tasks/prep_php/CentOS.yml → dev/provisioning/ansible/roles/web_php/tasks/php/CentOS.yml

@@ -8,9 +8,11 @@
 
 - name: Prep php... disable all the php repositories
   shell: yum-config-manager --disable 'remi-php*'
+  no_log: true
 
 - name: Prep php... enable the repo php{{ php_version | replace(".","") }}
   shell: yum-config-manager --enable remi-php{{ php_version | replace(".","") }}
+  no_log: true
 
 - name: Prep php... update os
   dnf:
@@ -53,43 +55,39 @@
       - php{{ php_version | replace(".","") }}-php-opcache
     state: latest
 
+- name: Prep php... Set php in web server conf 
+  template:
+    dest: /etc/httpd/conf.d/php{{ php_version | replace(".","") }}-php.conf
+    src: CentOS/httpd_php.conf.j2
+    mode: 0640
+  when: websrv in ["apache", "apache2"]
+
 - name: Prep php... Set php env for {{ ansible_facts['distribution'] }}
   set_fact: 
     php_bin: "php{{ php_version | replace('.','') }}"
     php_dir: "/etc/opt/remi/php{{ php_version | replace('.','') }}/php.d"
-    #php_pkg_apcu: "{{ php_config_ref[php_ver|replace('.','_')].php_pkg_apcu | d(php_config_ref.defaults.php_pkg_apcu) }}"
-    #php_pkg_spe: "{{ php_config_ref[php_ver|replace('.','_')].php_pkg_spe | d(php_config_ref.defaults.php_pkg_spe) }}"
-    #php_socket: "{{ php_config_ref[php_ver|replace('.','_')].php_socket | d(php_config_ref.defaults.php_socket) }}"
 
-- name: Prep php... Read Nextcloud configuration for PHP
+- name: Prep php... Read configuration for PHP
   set_fact:
-    php_content: "{{ lookup('template', '{{ role_path }}/templates/php_nc_ini.j2') }}"
+    php_content: "{{ lookup('template', '{{ role_path }}/templates/CentOS/php.ini.j2') }}"
 
-- name: Prep php... Integration Nextcloud configuration for PHP
+- name: Prep php... Integration configuration for PHP
   blockinfile:
     dest: /etc/opt/remi/php{{ php_version | replace(".","") }}/php.ini
     content: '{{ php_content }}'
     state: present
-
-- name: Prep php... Read APCU configuration for PHP 
-  set_fact:
-    php_content: "{{ lookup('template', '{{ role_path }}/templates/apcu_nc_ini.j2') }}"
-
-- name: Prep php... Integration APCU configuration for PHP 
-  blockinfile:
-    dest: /etc/opt/remi/php{{ php_version | replace(".","") }}/php.d/40-apcu.ini
-    content: '{{ php_content }}'
-    state: present
+    marker: "; {mark} ANSIBLE MANAGED BLOCK"
 
 - name: Prep php... Read OPCACHE configuration for PHP 
   set_fact:
-    php_content: "{{ lookup('template', '{{ role_path }}/templates/opcache_nc_ini.j2') }}"
+    php_content: "{{ lookup('template', '{{ role_path }}/templates/CentOS/10-opcache.ini.j2') }}"
 
 - name: Prep php... Integration OPCACHE configuration for PHP 
   blockinfile:
     dest: /etc/opt/remi/php{{ php_version | replace(".","") }}/php.d/10-opcache.ini
     content: '{{ php_content }}'
     state: present
+    marker: "; {mark} ANSIBLE MANAGED BLOCK"
 
 - name: Prep php... Install PHP-FPM
   dnf:
@@ -104,9 +102,63 @@
     regexp: "^{{ item.property | regex_escape() }}.*"
     line: "{{ item.value }}"
   with_items:
-    - { property: 'pm = dynamic', value: 'pm = {{ nc_pm }}' }
-    - { property: 'pm.max_children =', value: 'pm.max_children = {{ nc_pm_max_children }}' }
-    - { property: 'pm.start_servers =', value: 'pm.start_servers = {{ nc_pm_start_servers }}' }
-    - { property: 'pm.min_spare_servers =', value: 'pm.min_spare_servers = {{ nc_pm_min_spare_servers }}' }
-    - { property: 'pm.max_spare_servers =', value: 'pm.max_spare_servers = {{ nc_pm_max_spare_servers }}' }
+    - { property: 'listen = 127.0.0.1:9000', value: 'listen = /var/opt/remi/php81/run/php-fpm/www.sock' }
+    - { property: 'pm = dynamic',            value: 'pm = {{ pm }}' }
+    - { property: 'pm.max_children =',       value: 'pm.max_children = {{ pm_max_children }}' }
+    - { property: 'pm.start_servers =',      value: 'pm.start_servers = {{ pm_start_servers }}' }
+    - { property: 'pm.min_spare_servers =',  value: 'pm.min_spare_servers = {{ pm_min_spare_servers }}' }
+    - { property: 'pm.max_spare_servers =',  value: 'pm.max_spare_servers = {{ pm_max_spare_servers }}' }
+    - { property: ';pm.max_requests =',      value: 'pm.max_requests = {{ pm_max_requests }}' }
+    - { property: 'php_value[session.save_handler]', value: ';php_value[session.save_handler] = files' }
+    - { property: 'php_value[session.save_path]',    value: ';php_value[session.save_path]    = /var/opt/remi/php81/lib/php/session' }
+    - { property: 'php_value[soap.wsdl_cache_dir]',  value: ';php_value[soap.wsdl_cache_dir]  = /var/opt/remi/php81/lib/php/wsdlcache' }
   when: add_php_fpm
+
+- name: Prep php... Read PHP-FPM configuration
+  set_fact:
+    phpfpm_content: "{{ lookup('template', '{{ role_path }}/templates/CentOS/www.conf.j2') }}"
+
+- name: Prep php... Integration PHP-FPM configuration
+  blockinfile:
+    dest: /etc/opt/remi/php{{ php_version | replace(".","") }}/php-fpm.d/www.conf
+    content: '{{ phpfpm_content }}'
+    state: present
+    marker: "; {mark} ANSIBLE MANAGED BLOCK"
+
+- name: Prep php... Read APCU configuration for PHP 
+  set_fact:
+    php_content: "{{ lookup('template', '{{ role_path }}/templates/CentOS/apcu_nc_ini.j2') }}"
+  
+- name: Prep php... Integration APCU configuration for PHP 
+  blockinfile:
+    dest: /etc/opt/remi/php{{ php_version | replace(".","") }}/php.d/40-apcu.ini
+    content: '{{ php_content }}'
+    state: present
+    marker: "; {mark} ANSIBLE MANAGED BLOCK"
+
+- name: Prep php... Integration Redis configuration for PHP
+  template:
+    dest: /etc/opt/remi/php{{ php_version | replace(".","") }}/php.d/50-redis.ini
+    src: CentOS/50-redis.ini.j2
+    mode: 0640
+  when: my_redis_service.status == "enabled"
+
+- name: Prep php... Set alias php
+  file:
+    src: /opt/remi/php{{ php_version | replace(".","") }}/root/usr/bin/php
+    dest: /usr/bin/php
+    state: link
+
+- name: PHP Installed Starting
+  service:
+    name: php{{ php_version | replace(".","") }}-php-fpm
+    state: started
+    enabled: yes
+
+- name: Prep php... Set php-fpm instead of mod_php in web server conf 
+  template:
+    dest: /etc/httpd/conf.d/php{{ php_version | replace(".","") }}-php.conf
+    src: CentOS/httpd_php_fpm.conf.j2
+    mode: 0640
+  when: (websrv in ["apache", "apache2"]) and (enable_php_fpm)
+

dev/provisioning/ansible/roles/web_php/tasks/selinux.yml

@@ -0,0 +1,26 @@
+---
+
+- name: Selinux... enable seboolean settings
+  seboolean:
+    name: "{{ item }}"
+    state: yes
+    persistent: yes
+  with_items:
+    - httpd_unified
+    - httpd_graceful_shutdown
+    - httpd_can_network_relay
+    - httpd_can_network_connect
+    - httpd_can_network_connect_db
+    - daemons_enable_cluster_mode
+    - httpd_use_fusefs
+    - httpd_use_cifs
+    - httpd_use_gpg
+    - httpd_use_nfs
+    - httpd_execmem
+    - httpd_can_sendmail
+
+# if you have trouble with php-fpm and selinux in this nextcloud configuration :
+# # ausearch -c 'php-fpm' --raw | audit2allow -M my-phpfpm
+# # semodule -X 300 -i my-phpfpm.pp
+# # ausearch -c 'df' --raw | audit2allow -M my-df
+# # semodule -X 300 -i my-df.pp

dev/provisioning/ansible/roles/web_php/tasks/ssl.yml

@@ -0,0 +1,50 @@
+---
+- name: SSL... Create ssl certificates Directory
+  file:
+    dest: "{{ ssl_path }}"
+    owner: root
+    group: root
+    state: directory
+    recurse: yes
+
+- name: SSL... Create private key (RSA, 4096 bits)
+  openssl_privatekey:
+    path: "{{ ssl_path }}/{{ ansible_fqdn }}.key"
+
+- name: SSL... Check if CRT exists
+  stat:
+    path: "{{ ssl_path }}/{{ ansible_fqdn }}.crt"
+  register: result_crt
+
+- name: SSL... Create certificate signing request (CSR) for self-signed certificate
+  community.crypto.openssl_csr_pipe:
+    privatekey_path: "{{ ssl_path }}/{{ ansible_fqdn }}.key"
+    country_name: BE
+    locality_name: Louvain-la-Neuve
+    common_name: "{{ ansible_fqdn }}"
+    organization_name: UCLouvain
+    organizational_unit_name: ELIC
+  register: csr
+  when: (result_crt.stat.isreg is undefined) or (not result_crt.stat.isreg)
+
+- name: SSL... Generate a Self Signed OpenSSL certificate
+  community.crypto.x509_certificate:
+    path: "{{ ssl_path }}/{{ ansible_fqdn }}.crt"
+    csr_content: "{{ csr.csr }}"
+    privatekey_path: "{{ ssl_path }}/{{ ansible_fqdn }}.key"
+    provider: selfsigned
+  when: (result_crt.stat.isreg is undefined) or (not result_crt.stat.isreg)
+
+    #- name: SSL... Remove previous PEM file
+    #  file:
+    #    path: "{{ ssl_path }}/{{ ansible_fqdn }}.pem"
+    #    state: absent
+
+- name: SSL... Merge KEY and CRT to generate PEM
+  shell: "echo '' > {{ ssl_path }}/{{ ansible_fqdn }}.pem; cat {{ ssl_path }}/{{ ansible_fqdn }}.key {{ ssl_path }}/{{ ansible_fqdn }}.crt >> {{ ssl_path }}/{{ ansible_fqdn }}.pem"
+
+- name: SSL... Generate DH Parameters with a different size (2048 bits)
+  community.crypto.openssl_dhparam:
+    path: "{{ ssl_path }}/dhparams.pem"
+    size: 2048
+  register: dhparam

dev/provisioning/ansible/roles/web_php/tasks/web/CentOS.yml

@@ -0,0 +1,61 @@
+---
+# CentOS related tasks
+
+- name: install ancillaries packages
+  dnf:
+    name:
+      - mariadb
+      - python2-cryptography
+      - mod_ssl
+      - openssl
+    state: latest
+    enablerepo: epel
+
+- name: Ensure Apache is installed on {{ ansible_facts['distribution'] }}
+  dnf:
+    name: httpd
+    state: present
+  when: websrv in ["apache", "apache2"]
+
+- name: Set http env on {{ ansible_facts['distribution'] }}
+  set_fact:
+    http_service_name: httpd
+    http_webroot: /var/www/html
+    websrv_user: apache
+    websrv_group: apache
+  when: websrv in ["apache", "apache2"]
+
+- name: Get httpd version
+  shell: "httpd -v | head -n1 | cut -d'/' -f2 | cut -d' ' -f1"
+  register: web_version
+  when: websrv in ["apache", "apache2"]
+
+- name: Get SSL version
+  shell: "openssl version | cut -d' ' -f2 | cut -d'-' -f1"
+  register: ssl_version
+
+- name: Set web and SSL status
+  set_fact:
+    web_status: "{{ web_version.stdout|string is version('2.4.8', '>=') }}"
+    ssl_status: "{{ ssl_version.stdout|string is version('1.1.1', '>') }}"
+
+- name: Change apache configuration
+  template:
+    src: CentOS/httpd.conf.j2
+    dest: /etc/httpd/conf.d/tune.conf
+    owner: root
+    group: root
+    mode: '0644'
+  when: websrv in ["apache", "apache2"]
+
+- name: Allow http to listen on tcp port 80
+  seport:
+    ports: 80
+    proto: tcp
+    setype: http_port_t
+    state: present
+
+- name: Start {{ http_service_name }} service
+  service:
+    name: "{{ http_service_name }}"
+    state: started

dev/provisioning/ansible/roles/web_php/tasks/web/RedHat.yml

@@ -0,0 +1,4 @@
+---
+
+- include_tasks: "{{ ansible_facts['distribution'] }}.yml"
+

dev/provisioning/ansible/roles/web_php/tasks/web/Suse.yml

@@ -0,0 +1,57 @@
+---
+# Suse related tasks
+
+- name: add rpmfusion-free-release {{ ansible_distribution_major_version|int }} repo
+  dnf:
+    name: https://download1.rpmfusion.org/free/el/rpmfusion-free-release-{{ ansible_distribution_major_version|int }}.noarch.rpm
+    disable_gpg_check: yes
+    validate_certs: no
+    state: latest
+  when: not {{ debug_speed_check }}
+
+- name: update os
+  dnf:
+    name: '*'
+    update_cache: true
+    state: latest
+  when: not {{ debug_speed_check }}
+
+- name: install needed packages
+  dnf:
+    name:
+      - epel-release
+      - yum-utils
+      - curl
+      - bash-completion
+      - mlocate
+      - bzip2
+      - wget
+      - libreoffice
+      - ffmpeg
+      - mariadb
+    state: latest
+    enablerepo: epel
+  when: not {{ debug_speed_check }}
+
+- name: Ensure Apache is installed on {{ ansible_facts['distribution'] }}
+  dnf:
+    name: httpd
+    state: present
+  when: nextcloud_websrv in ["apache", "apache2"]
+  notify: start http
+
+- name: Set http env on {{ ansible_facts['distribution'] }}
+  set_fact:
+    http_service_name: httpd
+    http_webroot: /var/www/html
+    nextcloud_websrv_user: apache
+    nextcloud_websrv_group: apache
+  when: nextcloud_websrv in ["apache", "apache2"]
+
+- name: Generate Nextcloud configuration for apache
+  template:
+    dest: /etc/httpd/conf.d/nextcloud.conf
+    src: nextcloud_apache2.j2
+    mode: 0640
+  when: nextcloud_websrv in ["apache", "apache2"]
+  notify: restart http

dev/provisioning/ansible/roles/nextcloud/templates/opcache_nc_ini.j2 → dev/provisioning/ansible/roles/web_php/templates/CentOS/10-opcache.ini.j2

@@ -1,13 +1,10 @@
-
 opcache.enable=1
 opcache.enable_cli=1
-opcache.memory_consumption={{ OPCACHE_MEM_SIZE }}
 opcache.interned_strings_buffer=32
 opcache.max_accelerated_files=10000
+opcache.memory_consumption={{ OPCACHE_MEM_SIZE }}
 opcache.save_comments=1
-opcache.revalidate_freq=60
-
+opcache.revalidate_freq=1
 opcache.fast_shutdown=1
 opcache.jit=disable
 opcache.jit_buffer_size=0
-

dev/provisioning/ansible/roles/web_php/templates/CentOS/50-redis.ini.j2

@@ -0,0 +1,50 @@
+; Enable redis extension module
+extension = redis.so
+
+; phpredis can be used to store PHP sessions.
+; To do this, uncomment and configure below
+
+; RPM note : save_handler and save_path are defined
+; for mod_php, in /etc/httpd/conf.d/php.conf
+; for php-fpm, in /etc/opt/remi/php81/php-fpm.d/*conf
+
+session.save_handler = redis
+{% if (groups['web_servers'] | length) > 1 %}
+session.save_path = "tcp://127.0.0.1:{{ redis_port }}?weight=2&timeout=2.5, tcp://{{ keepalived_vip }}:{{ redis_port }}?weight=1"
+{% else %}
+session.save_path = "tcp://{{ redis_host }}:{{ redis_port }}"
+{% endif %}
+soap.wsdl_cache_dir = "tcp://{{ redis_host }}:{{ redis_port }}"
+
+; Configuration
+;redis.arrays.algorithm = ''
+;redis.arrays.auth = ''
+;redis.arrays.autorehash = 0
+;redis.arrays.connecttimeout = 0
+;redis.arrays.consistent = 0
+;redis.arrays.distributor = ''
+;redis.arrays.functions = ''
+;redis.arrays.hosts = ''
+;redis.arrays.index = 0
+;redis.arrays.lazyconnect = 0
+;redis.arrays.names = ''
+;redis.arrays.pconnect = 0
+;redis.arrays.previous = ''
+;redis.arrays.readtimeout = 0
+;redis.arrays.retryinterval = 0
+;redis.clusters.auth = 0
+;redis.clusters.cache_slots = 0
+;redis.clusters.persistent = 0
+;redis.clusters.read_timeout = 0
+;redis.clusters.seeds = ''
+;redis.clusters.timeout = 0
+;redis.pconnect.pooling_enabled = 1
+;redis.pconnect.connection_limit = 0
+;redis.pconnect.echo_check_liveness = 1
+;redis.pconnect.pool_detect_dirty = 0
+;redis.pconnect.pool_poll_timeout = 0
+;redis.pconnect.pool_pattern => ''
+redis.session.locking_enabled = 1
+redis.session.lock_retries = -1
+redis.session.lock_wait_time = 10000
+;redis.session.lock_expire = 60

dev/provisioning/ansible/roles/nextcloud/templates/apcu_nc_ini.j2 → dev/provisioning/ansible/roles/web_php/templates/CentOS/apcu_nc_ini.j2

@@ -1,4 +1,2 @@
-
 apc.enable_cli=1
 apc.shm_size={{ APC_SHM_SIZE }}
-

dev/provisioning/ansible/roles/web_php/templates/CentOS/httpd.conf.j2

@@ -0,0 +1,35 @@
+ServerName {{ ansible_fqdn }}
+ServerAdmin admin@{{ ansible_fqdn }}
+<Directory "/var/www/html">
+    Options Indexes FollowSymLinks
+    AllowOverride All
+    Require all granted
+</Directory>
+<IfModule http2_module>
+    Protocols h2 h2c http/1.1
+    H2Direct on
+    H2StreamMaxMemSize 5120000000
+</IfModule>
+<IfModule prefork.c>
+    StartServers 2
+    {% if web_status is sameas true %}
+    MinSpareThreads 25
+    MaxSpareThreads 75
+    ThreadLimit 64
+    ThreadsPerChild 25
+    {% endif %}
+    MaxRequestWorkers 150 
+    MaxConnectionsPerChild 1000
+</IfModule>
+<IfModule mpm_event_module>
+    StartServers 2
+    MinSpareThreads 25
+    MaxSpareThreads 75
+    ThreadLimit 64
+    ThreadsPerChild 25
+    MaxRequestWorkers 150
+    MaxConnectionsPerChild 1000
+</IfModule>
+ServerTokens Prod
+ServerSignature Off
+TraceEnable Off

dev/provisioning/ansible/roles/web_php/templates/CentOS/httpd_php.conf.j2

@@ -0,0 +1,52 @@
+#
+# The following lines prevent .user.ini files from being viewed by Web clients.
+#
+<Files ".user.ini">
+    <IfModule mod_authz_core.c>
+        Require all denied
+    </IfModule>
+    <IfModule !mod_authz_core.c>
+        Order allow,deny
+        Deny from all
+        Satisfy All
+    </IfModule>
+</Files>
+
+#
+# Allow php to handle Multiviews
+#
+AddType text/html .php
+
+#
+# Add index.php to the list of files that will be served as directory
+# indexes.
+#
+DirectoryIndex index.php
+
+# mod_php options
+<IfModule  mod_php.c>
+    #
+    # Cause the PHP interpreter to handle files with a .php extension.
+    #
+    <FilesMatch \.(php|phar)$>
+        SetHandler application/x-httpd-php
+    </FilesMatch>
+
+    #
+    # Uncomment the following lines to allow PHP to pretty-print .phps
+    # files as PHP source code:
+    #
+    #<FilesMatch \.phps$>
+    #    SetHandler application/x-httpd-php-source
+    #</FilesMatch>
+
+    #
+    # Apache specific PHP configuration options
+    # those can be override in each configured vhost
+    #
+    #php_value session.save_handler "files"
+    #php_value session.save_path    "/var/opt/remi/php81/lib/php/session"
+    #php_value soap.wsdl_cache_dir  "/var/opt/remi/php81/lib/php/wsdlcache"
+
+    php_value opcache.file_cache   "/var/opt/remi/php81/lib/php/opcache"
+</IfModule>

dev/provisioning/ansible/roles/web_php/templates/CentOS/httpd_php_fpm.conf.j2

@@ -0,0 +1,50 @@
+#
+# The following lines prevent .user.ini files from being viewed by Web clients.
+#
+<Files ".user.ini">
+    <IfModule mod_authz_core.c>
+        Require all denied
+    </IfModule>
+    <IfModule !mod_authz_core.c>
+        Order allow,deny
+        Deny from all
+        Satisfy All
+    </IfModule>
+</Files>
+
+#
+# Allow php to handle Multiviews
+#
+AddType text/html .php
+
+#
+# Add index.php to the list of files that will be served as directory
+# indexes.
+#
+DirectoryIndex index.php
+
+# php-fpm options
+<IfModule  mod_proxy_fcgi.c>
+
+    SuexecUserGroup apache apache
+
+    <Proxy "unix:/var/opt/remi/php81/run/php-fpm/www.sock|fcgi://php-fpm">
+            ProxySet disablereuse=off
+    </Proxy>
+    #
+    # Cause the PHP interpreter to handle files with a .php extension.
+    #
+    <FilesMatch \.(php|phar)$>
+        SetHandler proxy:fcgi://php-fpm
+    </FilesMatch>
+
+    #
+    # Apache specific PHP configuration options
+    # those can be override in each configured vhost
+    #
+    #php_value session.save_handler "files"
+    #php_value session.save_path    "/var/opt/remi/php81/lib/php/session"
+    #php_value soap.wsdl_cache_dir  "/var/opt/remi/php81/lib/php/wsdlcache"
+
+    php_value opcache.file_cache   "/var/opt/remi/php81/lib/php/opcache"
+</IfModule>

dev/provisioning/ansible/roles/web_php/templates/CentOS/httpd_ssl.conf.j2

@@ -0,0 +1,224 @@
+#
+# When we also provide SSL we have to listen to the 
+# the HTTPS port in addition.
+#
+Listen 443 https
+
+##
+##  SSL Global Context
+##
+##  All SSL configuration in this context applies both to
+##  the main server and all SSL-enabled virtual hosts.
+##
+
+#   Pass Phrase Dialog:
+#   Configure the pass phrase gathering process.
+#   The filtering dialog program (`builtin' is a internal
+#   terminal dialog) has to provide the pass phrase on stdout.
+SSLPassPhraseDialog exec:/usr/libexec/httpd-ssl-pass-dialog
+
+#   Inter-Process Session Cache:
+#   Configure the SSL Session Cache: First the mechanism 
+#   to use and second the expiring timeout (in seconds).
+SSLSessionCache         shmcb:/run/httpd/sslcache(512000)
+SSLSessionCacheTimeout  300
+
+#   Pseudo Random Number Generator (PRNG):
+#   Configure one or more sources to seed the PRNG of the 
+#   SSL library. The seed data should be of good random quality.
+#   WARNING! On some platforms /dev/random blocks if not enough entropy
+#   is available. This means you then cannot use the /dev/random device
+#   because it would lead to very long connection times (as long as
+#   it requires to make more entropy available). But usually those
+#   platforms additionally provide a /dev/urandom device which doesn't
+#   block. So, if available, use this one instead. Read the mod_ssl User
+#   Manual for more details.
+SSLRandomSeed startup file:/dev/urandom  256
+SSLRandomSeed connect builtin
+#SSLRandomSeed startup file:/dev/random  512
+#SSLRandomSeed connect file:/dev/random  512
+#SSLRandomSeed connect file:/dev/urandom 512
+
+#
+# Use "SSLCryptoDevice" to enable any supported hardware
+# accelerators. Use "openssl engine -v" to list supported
+# engine names.  NOTE: If you enable an accelerator and the
+# server does not start, consult the error logs and ensure
+# your accelerator is functioning properly. 
+#
+SSLCryptoDevice builtin
+#SSLCryptoDevice ubsec
+
+##
+## SSL Virtual Host Context
+##
+
+<VirtualHost _default_:443>
+
+# General setup for the virtual host, inherited from global configuration
+#DocumentRoot "/var/www/html"
+#ServerName www.example.com:443
+
+# Use separate log files for the SSL virtual host; note that LogLevel
+# is not inherited from httpd.conf.
+ErrorLog logs/ssl_error_log
+TransferLog logs/ssl_access_log
+LogLevel warn
+
+#   SSL Engine Switch:
+#   Enable/Disable SSL for this virtual host.
+SSLEngine on
+
+#   SSL Protocol support:
+# List the enable protocol levels with which clients will be able to
+# connect.  Disable SSLv2 access by default:
+SSLProtocol all -SSLv2 -SSLv3
+
+#   SSL Cipher Suite:
+#   List the ciphers that the client is permitted to negotiate.
+#   See the mod_ssl documentation for a complete list.
+SSLCipherSuite HIGH:3DES:!aNULL:!MD5:!SEED:!IDEA
+
+#   Speed-optimized SSL Cipher configuration:
+#   If speed is your main concern (on busy HTTPS servers e.g.),
+#   you might want to force clients to specific, performance
+#   optimized ciphers. In this case, prepend those ciphers
+#   to the SSLCipherSuite list, and enable SSLHonorCipherOrder.
+#   Caveat: by giving precedence to RC4-SHA and AES128-SHA
+#   (as in the example below), most connections will no longer
+#   have perfect forward secrecy - if the server's key is
+#   compromised, captures of past or future traffic must be
+#   considered compromised, too.
+#SSLCipherSuite RC4-SHA:AES128-SHA:HIGH:MEDIUM:!aNULL:!MD5
+#SSLHonorCipherOrder on 
+
+#   Server Certificate:
+# Point SSLCertificateFile at a PEM encoded certificate.  If
+# the certificate is encrypted, then you will be prompted for a
+# pass phrase.  Note that a kill -HUP will prompt again.  A new
+# certificate can be generated using the genkey(1) command.
+SSLCertificateFile /etc/pki/tls/certs/localhost.crt
+
+#   Server Private Key:
+#   If the key is not combined with the certificate, use this
+#   directive to point at the key file.  Keep in mind that if
+#   you've both a RSA and a DSA private key you can configure
+#   both in parallel (to also allow the use of DSA ciphers, etc.)
+SSLCertificateKeyFile /etc/pki/tls/private/localhost.key
+
+#   Server Certificate Chain:
+#   Point SSLCertificateChainFile at a file containing the
+#   concatenation of PEM encoded CA certificates which form the
+#   certificate chain for the server certificate. Alternatively
+#   the referenced file can be the same as SSLCertificateFile
+#   when the CA certificates are directly appended to the server
+#   certificate for convinience.
+#SSLCertificateChainFile /etc/pki/tls/certs/server-chain.crt
+
+#   Certificate Authority (CA):
+#   Set the CA certificate verification path where to find CA
+#   certificates for client authentication or alternatively one
+#   huge file containing all of them (file must be PEM encoded)
+#SSLCACertificateFile /etc/pki/tls/certs/ca-bundle.crt
+
+#   Client Authentication (Type):
+#   Client certificate verification type and depth.  Types are
+#   none, optional, require and optional_no_ca.  Depth is a
+#   number which specifies how deeply to verify the certificate
+#   issuer chain before deciding the certificate is not valid.
+#SSLVerifyClient require
+#SSLVerifyDepth  10
+
+#   Access Control:
+#   With SSLRequire you can do per-directory access control based
+#   on arbitrary complex boolean expressions containing server
+#   variable checks and other lookup directives.  The syntax is a
+#   mixture between C and Perl.  See the mod_ssl documentation
+#   for more details.
+#<Location />
+#SSLRequire (    %{SSL_CIPHER} !~ m/^(EXP|NULL)/ \
+#            and %{SSL_CLIENT_S_DN_O} eq "Snake Oil, Ltd." \
+#            and %{SSL_CLIENT_S_DN_OU} in {"Staff", "CA", "Dev"} \
+#            and %{TIME_WDAY} >= 1 and %{TIME_WDAY} <= 5 \
+#            and %{TIME_HOUR} >= 8 and %{TIME_HOUR} <= 20       ) \
+#           or %{REMOTE_ADDR} =~ m/^192\.76\.162\.[0-9]+$/
+#</Location>
+
+#   SSL Engine Options:
+#   Set various options for the SSL engine.
+#   o FakeBasicAuth:
+#     Translate the client X.509 into a Basic Authorisation.  This means that
+#     the standard Auth/DBMAuth methods can be used for access control.  The
+#     user name is the `one line' version of the client's X.509 certificate.
+#     Note that no password is obtained from the user. Every entry in the user
+#     file needs this password: `xxj31ZMTZzkVA'.
+#   o ExportCertData:
+#     This exports two additional environment variables: SSL_CLIENT_CERT and
+#     SSL_SERVER_CERT. These contain the PEM-encoded certificates of the
+#     server (always existing) and the client (only existing when client
+#     authentication is used). This can be used to import the certificates
+#     into CGI scripts.
+#   o StdEnvVars:
+#     This exports the standard SSL/TLS related `SSL_*' environment variables.
+#     Per default this exportation is switched off for performance reasons,
+#     because the extraction step is an expensive operation and is usually
+#     useless for serving static content. So one usually enables the
+#     exportation for CGI and SSI requests only.
+#   o StrictRequire:
+#     This denies access when "SSLRequireSSL" or "SSLRequire" applied even
+#     under a "Satisfy any" situation, i.e. when it applies access is denied
+#     and no other module can change it.
+#   o OptRenegotiate:
+#     This enables optimized SSL connection renegotiation handling when SSL
+#     directives are used in per-directory context. 
+#SSLOptions +FakeBasicAuth +ExportCertData +StrictRequire
+<Files ~ "\.(cgi|shtml|phtml|php3?)$">
+    SSLOptions +StdEnvVars
+</Files>
+<Directory "/var/www/cgi-bin">
+    SSLOptions +StdEnvVars
+</Directory>
+
+#   SSL Protocol Adjustments:
+#   The safe and default but still SSL/TLS standard compliant shutdown
+#   approach is that mod_ssl sends the close notify alert but doesn't wait for
+#   the close notify alert from client. When you need a different shutdown
+#   approach you can use one of the following variables:
+#   o ssl-unclean-shutdown:
+#     This forces an unclean shutdown when the connection is closed, i.e. no
+#     SSL close notify alert is send or allowed to received.  This violates
+#     the SSL/TLS standard but is needed for some brain-dead browsers. Use
+#     this when you receive I/O errors because of the standard approach where
+#     mod_ssl sends the close notify alert.
+#   o ssl-accurate-shutdown:
+#     This forces an accurate shutdown when the connection is closed, i.e. a
+#     SSL close notify alert is send and mod_ssl waits for the close notify
+#     alert of the client. This is 100% SSL/TLS standard compliant, but in
+#     practice often causes hanging connections with brain-dead browsers. Use
+#     this only for browsers where you know that their SSL implementation
+#     works correctly. 
+#   Notice: Most problems of broken clients are also related to the HTTP
+#   keep-alive facility, so you usually additionally want to disable
+#   keep-alive for those clients, too. Use variable "nokeepalive" for this.
+#   Similarly, one has to force some clients to use HTTP/1.0 to workaround
+#   their broken HTTP/1.1 implementation. Use variables "downgrade-1.0" and
+#   "force-response-1.0" for this.
+BrowserMatch "MSIE [2-5]" \
+         nokeepalive ssl-unclean-shutdown \
+         downgrade-1.0 force-response-1.0
+
+#   Per-Server Logging:
+#   The home of a custom SSL log file. Use this when you want a
+#   compact non-error SSL logfile on a virtual host basis.
+CustomLog logs/ssl_request_log \
+          "%t %h %{SSL_PROTOCOL}x %{SSL_CIPHER}x \"%r\" %b"
+
+SuexecUserGroup apache apache 
+<Proxy "unix:/var/opt/remi/php81/run/php-fpm/www.sock|fcgi://php-fpm"> 
+  ProxySet disablereuse=off 
+</Proxy> 
+<FilesMatch \.php$> 
+  SetHandler proxy:fcgi://php-fpm 
+</FilesMatch> 
+
+</VirtualHost> 

dev/provisioning/ansible/roles/nextcloud/templates/php_nc_ini.j2 → dev/provisioning/ansible/roles/web_php/templates/CentOS/php.ini.j2

@@ -1,11 +1,15 @@
-
+[PHP]
 memory_limit = {{ PHP_MEMORY_LIMIT }}
-upload_max_filesize = {{ PHP_UPLOAD_LIMIT }}
-post_max_size = {{ PHP_POST_LIMIT }}
-max_file_uploads = {{ PHP_MAX_FILE }}
+output_buffering = 'Off'
 max_execution_time = {{ PHP_MAX_TIME }}
 max_input_time = {{ PHP_MAX_TIME }}
-session.gc_maxlifetime = {{ PHP_MAX_TIME }}
-
-output_buffering = Off
+post_max_size = {{ PHP_POST_LIMIT }}
+upload_max_filesize = {{ PHP_UPLOAD_LIMIT }}
+max_file_uploads = {{ PHP_MAX_FILE }}
+log_errors_max_len = 1024
+html_errors = On
 
+[Interbase]
+ibase.allow_persistent = 1
+ibase.max_persistent = -1
+ibase.max_links = -1

dev/provisioning/ansible/roles/web_php/templates/CentOS/www.conf.j2

@@ -0,0 +1,20 @@
+
+{% if websrv == "nginx" %}
+;listen.allowed_clients = 127.0.0.1
+listen.acl_users = nginx
+{% else %}
+listen.acl_users = apache
+{% endif %}
+env[HOSTNAME] = $HOSTNAME
+env[PATH] = /usr/local/bin:/usr/bin:/bin
+env[TMP] = /tmp
+env[TMPDIR] = /tmp
+env[TEMP] = /tmp
+
+php_value[session.save_handler] = redis
+{% if (groups['web_servers'] | length) > 1 %}
+php_value[session.save_path]    = "tcp://127.0.0.1:{{ redis_port }}?weight=2&timeout=2.5, tcp://{{ keepalived_vip }}:{{ redis_port }}?weight=1"
+{% else %}
+php_value[session.save_path]    = "tcp://{{ redis_host }}:{{ redis_port }}"
+{% endif %}
+php_value[soap.wsdl_cache_dir]  = "tcp://{{ redis_host }}:{{ redis_port }}"

dev/provisioning/ansible/roles/web_php/templates/nginx_nc.j2

@@ -0,0 +1,207 @@
+################################################################################
+# This file was generated by Ansible for {{ansible_fqdn}}
+# Do NOT modify this file by hand!
+################################################################################
+
+{% if nextcloud_install_tls and nextcloud_tls_enforce %}
+server {
+    listen 80;
+{% if nextcloud_ipv6 %}
+    listen [::]:80;
+{% endif %}
+    server_name {{ nextcloud_trusted_domain | ansible.utils.ipwrap | join(' ') }};
+
+    # Prevent nginx HTTP Server Detection
+    server_tokens off;
+
+    # Enforce HTTPS
+    return 301 https://$server_name$request_uri;
+}
+{% endif %}
+
+server {
+    server_name {{ nextcloud_trusted_domain | ansible.utils.ipwrap | join(' ') }};
+
+{% if not nextcloud_install_tls or not nextcloud_tls_enforce %}
+    listen 80;
+{% if nextcloud_ipv6 %}
+    listen [::]:80;
+{% endif %}
+{% endif %}
+
+{% if nextcloud_install_tls %}
+    listen 443 ssl http2;
+{% if nextcloud_ipv6 %}
+    listen [::]:443 ssl http2;
+{% endif %}
+    ssl_certificate {{ nextcloud_tls_cert_file }};
+    ssl_certificate_key {{ nextcloud_tls_cert_key_file }};
+
+    # Prevent nginx HTTP Server Detection
+    server_tokens off;
+
+    ssl_session_timeout 1d;
+    ssl_session_cache shared:SSL:{{ nextcloud_tls_session_cache_size }};
+    # ssl_session_tickets off;
+
+    # OCSP stapling
+    ssl_stapling on;
+    ssl_stapling_verify on;
+
+    # Diffie-Hellman parameter for DHE ciphersuites, recommended 2048 bits
+    ssl_dhparam {{ nextcloud_tls_dhparam }};
+
+    # Use Mozilla's guidelines for SSL/TLS settings
+    # https://mozilla.github.io/server-side-tls/ssl-config-generator/
+{% if nextcloud_mozilla_modern_ssl_profile %}
+    # modern configuration. tweak to your needs.
+    ssl_protocols TLSv1.3;
+{% else %}
+    # intermediate configuration. tweak to your needs.
+    ssl_protocols TLSv1.2 TLSv1.3;
+    ssl_ciphers ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384;
+{% endif %}
+    ssl_prefer_server_ciphers off;
+
+    # HSTS settings
+    # WARNING: Only add the preload option once you read about
+    # the consequences in https://hstspreload.org/. This option
+    # will add the domain to a hardcoded list that is shipped
+    # in all major browsers and getting removed from this list
+    # could take several months.
+{% if nextcloud_hsts is string %}
+    add_header Strict-Transport-Security "{{ nextcloud_hsts }}";
+{% endif %}
+{% endif %}
+
+    # set max upload size and increase upload timeout:
+    client_max_body_size {{ nextcloud_max_upload_size }};
+    client_body_timeout 300s;
+    fastcgi_buffers 64 4K;
+
+    # Enable gzip but do not remove ETag headers
+    gzip on;
+    gzip_vary on;
+    gzip_comp_level 4;
+    gzip_min_length 256;
+    gzip_proxied expired no-cache no-store private no_last_modified no_etag auth;
+    gzip_types application/atom+xml application/javascript application/json application/ld+json application/manifest+json application/rss+xml application/vnd.geo+json application/vnd.ms-fontobject application/wasm application/x-font-ttf application/x-web-app-manifest+json application/xhtml+xml application/xml font/opentype image/bmp image/svg+xml image/x-icon text/cache-manifest text/css text/plain text/vcard text/vnd.rim.location.xloc text/vtt text/x-component text/x-cross-domain-policy;
+
+    # Pagespeed is not supported by Nextcloud, so if your server is built
+    # with the `ngx_pagespeed` module, uncomment this line to disable it.
+    # pagespeed off;
+
+    # HTTP response headers borrowed from Nextcloud `.htaccess`
+    add_header Referrer-Policy                      "no-referrer"   always;
+    add_header X-Content-Type-Options               "nosniff"       always;
+    add_header X-Download-Options                   "noopen"        always;
+    add_header X-Frame-Options                      "SAMEORIGIN"    always;
+    add_header X-Permitted-Cross-Domain-Policies    "none"          always;
+    add_header X-Robots-Tag                         "none"          always;
+    add_header X-XSS-Protection                     "1; mode=block" always;
+
+    # Remove X-Powered-By, which is an information leak
+    fastcgi_hide_header X-Powered-By;
+
+    # Path to the root of your installation
+    root {{ nextcloud_webroot }};
+
+    # Specify how to handle directories -- specifying `/index.php$request_uri`
+    # here as the fallback means that Nginx always exhibits the desired behaviour
+    # when a client requests a path that corresponds to a directory that exists
+    # on the server. In particular, if that directory contains an index.php file,
+    # that file is correctly served; if it doesn't, then the request is passed to
+    # the front-end controller. This consistent behaviour means that we don't need
+    # to specify custom rules for certain paths (e.g. images and other assets,
+    # `/updater`, `/ocm-provider`, `/ocs-provider`), and thus
+    # `try_files $uri $uri/ /index.php$request_uri`
+    # always provides the desired behaviour.
+    index index.php index.html /index.php$request_uri;
+
+    # Rule borrowed from `.htaccess` to handle Microsoft DAV clients
+    location = / {
+        if ( $http_user_agent ~ ^DavClnt ) {
+            return 302 /remote.php/webdav/$is_args$args;
+        }
+    }
+
+    location = /robots.txt {
+        allow all;
+        log_not_found off;
+        access_log off;
+    }
+
+    # Make a regex exception for `/.well-known` so that clients can still
+    # access it despite the existence of the regex rule
+    # `location ~ /(\.|autotest|...)` which would otherwise handle requests
+    # for `/.well-known`.
+    location ^~ /.well-known {
+        # The rules in this block are an adaptation of the rules
+        # in `.htaccess` that concern `/.well-known`.
+
+        location = /.well-known/carddav { return 301 /remote.php/dav/; }
+        location = /.well-known/caldav  { return 301 /remote.php/dav/; }
+
+        location /.well-known/acme-challenge    { try_files $uri $uri/ =404; }
+        location /.well-known/pki-validation    { try_files $uri $uri/ =404; }
+
+        # Let Nextcloud's API for `/.well-known` URIs handle all other
+        # requests by passing them to the front-end controller.
+        return 301 /index.php$request_uri;
+    }
+
+    # Rules borrowed from `.htaccess` to hide certain paths from clients
+    location ~ ^/(?:build|tests|config|lib|3rdparty|templates|data)(?:$|/)  { return 404; }
+    location ~ ^/(?:\.|autotest|occ|issue|indie|db_|console)                { return 404; }
+
+    # Ensure this block, which passes PHP files to the PHP process, is above the blocks
+    # which handle static assets (as seen below). If this block is not declared first,
+    # then Nginx will encounter an infinite rewriting loop when it prepends `/index.php`
+    # to the URI, resulting in a HTTP 500 error response.
+    location ~ \.php(?:$|/) {
+        # Required for legacy support
+        rewrite ^/(?!index|remote|public|cron|core\/ajax\/update|status|ocs\/v[12]|updater\/.+|oc[ms]-provider\/.+|.+\/richdocumentscode\/proxy) /index.php$request_uri;
+
+        fastcgi_split_path_info ^(.+?\.php)(/.*)$;
+        set $path_info $fastcgi_path_info;
+
+        try_files $fastcgi_script_name =404;
+
+        include fastcgi_params;
+        fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;
+        fastcgi_param PATH_INFO $path_info;
+        fastcgi_param HTTPS on;
+
+        fastcgi_param modHeadersAvailable true;         # Avoid sending the security headers twice
+        fastcgi_param front_controller_active true;     # Enable pretty urls
+        fastcgi_pass php-handler;
+
+        fastcgi_intercept_errors on;
+        fastcgi_request_buffering off;
+    }
+
+    location ~ \.(?:css|js|svg|gif|png|jpg|ico|wasm|tflite)$ {
+        try_files $uri /index.php$request_uri;
+        expires 6M;         # Cache-Control policy borrowed from `.htaccess`
+        access_log off;     # Optional: Don't log access to assets
+
+        location ~ \.wasm$ {
+            default_type application/wasm;
+        }
+    }
+
+    location ~ \.woff2?$ {
+        try_files $uri /index.php$request_uri;
+        expires 7d;         # Cache-Control policy borrowed from `.htaccess`
+        access_log off;     # Optional: Don't log access to assets
+    }
+
+    # Rule borrowed from `.htaccess`
+    location /remote {
+        return 301 /remote.php$request_uri;
+    }
+
+    location / {
+        try_files $uri $uri/ /index.php$request_uri;
+    }
+}

dev/provisioning/ansible/roles/web_php/templates/nginx_php_handler.j2

@@ -0,0 +1,9 @@
+################################################################################
+# This file was generated by Ansible for {{ansible_fqdn}}
+# Do NOT modify this file by hand!
+################################################################################
+
+upstream php-handler {
+    # server 127.0.0.1:9000;
+    server unix:{{ php_socket }};
+}

dev/provisioning/ansible/sql_loadbalancer.yml

@@ -0,0 +1,15 @@
+---
+- name: apply SQL load balancer configuration
+  collections:
+    - community.mysql
+  hosts: db_lbal_servers
+  vars:
+    ansible_python_interpreter: /usr/bin/python3
+  pre_tasks:
+    - name: define ansible_python_interpreter group // linux distribution
+      set_fact:
+        ansible_python_interpreter: /usr/bin/python2
+      when: ansible_distribution == 'CentOS' and ansible_distribution_major_version == '7'
+  roles:
+    - role: keepalived
+    - role: proxysql

dev/provisioning/ansible/storage.yml

@@ -0,0 +1,15 @@
+---
+- name: apply storage role
+  collections:
+    - community.general
+    - ansible.posix
+  hosts: gluster_servers
+  vars:
+    ansible_python_interpreter: /usr/bin/python3
+  pre_tasks:
+    - name: define ansible_python_interpreter group // linux distribution
+      set_fact:
+        ansible_python_interpreter: /usr/bin/python2
+      when: ansible_distribution == 'CentOS' and ansible_distribution_major_version == '7'
+  roles:
+    - { role: gluster }

dev/provisioning/bash/common.sh

@@ -6,7 +6,9 @@ if [[ -z "$1" ]]; then
 fi
 if [[ $1 == centos* ]]; then
    pkgmanager="yum"
-   yum install -y python3 dnf epel-release
+   yum install -y epel-release
+   yum install -y python3 dnf policycoreutils-python python2-cryptography libselinux-python3  
+   yum install -y epel-release
 elif [[ $1 == ubuntu* ]]; then
   pkgmanager="apt-get"
 else

report/Projet_brevet.md

@@ -512,6 +512,25 @@ Dans un second temps, les rôles Ansible ci-dessus seront enrichis:
 
 ![Environnement de développement avancé](./assets/dia_nc_dev_improved.png){width=80%}
 
+Nextcloud:
+
+- https://www.slideshare.net/Severalnines/tips-to-drive-maria-db-cluster-performance-for-nextcloud
+
+Redis:
+
+- https://redis.io/docs/management/sentinel/
+- https://severalnines.com/blog/redis-high-availability-architecture-sentinel/
+- https://nicoll.io/posts/ha-cache/
+
+Galera & ProxySQL:
+
+- https://proxysql.com/blog/galera-awareness-and-proxysql-scheduler/?highlight=proxysql_galera_checker
+- https://www.digitalocean.com/community/tutorials/how-to-optimize-mysql-queries-with-proxysql-caching-on-ubuntu-16-04-fr
+- https://mydbops.wordpress.com/2018/08/20/proxysql-series-percona-cluster-mariadb-cluster-galera-read-write-split/
+- https://mysqldb-info.blogspot.com/2019/05/load-balancing-pxc-with-proxysql.html
+- https://mysqldb-info.blogspot.com/2019/08/make-proxysql-for-high-availability.html
+- https://mydbops.wordpress.com/2018/02/19/proxysql-series-mysql-replication-read-write-split-up/
+
 ### Environnement de test
 
 ... la même chose mais dans **OpenStack** avec :