Make ceph use storage network

With current "Ceph production example" the difference between ceph's
public and storage network is not clear.

We assign Storage Network to compute nodes, but it's not used there.
We also asign Storage Network to ceph monitors, but it's not used there
as well.

Same problems apply to AIO environment.

As Dmitriy suggested in [1], ceph should not use mgmt network for
storage traffic.

This change makes ceph use storage network for:
- OSD<>OSD communication
- client<>OSD communication
- client<>MON communication

I think it's the most common scenario where all ceph-related traffic
uses dedicated(storage) network and do not depend on mgmt network.

This change affects both "Ceph production example" docs and AIO
environments.

[1] https://review.opendev.org/c/openstack/openstack-ansible/+/856566

Change-Id: I74387a2e961e2b8355ea6a0c889b2f5674233ebf

etc/openstack_deploy/openstack_user_config.yml.aio.j2

@@ -206,6 +206,8 @@ global_overrides:
           - nova_compute
           - manila_share
           - swift_proxy
+          - ceph-mon
+          - ceph-osd
 
 # keystone
 identity_hosts:

etc/openstack_deploy/openstack_user_config.yml.prod-ceph.example

@@ -73,6 +73,7 @@ global_overrides:
           - manila_share
           - nova_compute
           - ceph-mon
+          - ceph-osd
 
 ###
 ### Infrastructure

etc/openstack_deploy/user_variables.yml.prod-ceph.example

@@ -15,8 +15,8 @@ fsid: 116f14c4-7fe1-40e4-94eb-9240b63de5c1 # Replace with your generated UUID
 ## ceph-ansible settings
 ## See https://github.com/ceph/ceph-ansible/tree/master/group_vars for
 ## additional configuration options available.
-monitor_address_block: "{{ cidr_networks.container }}"
-public_network: "{{ cidr_networks.container }}"
+monitor_address_block: "{{ cidr_networks.storage }}"
+public_network: "{{ cidr_networks.storage }}"
 cluster_network: "{{ cidr_networks.storage }}"
 journal_size: 10240 # size in MB
 # ceph-ansible automatically creates pools & keys for OpenStack services

tests/roles/bootstrap-host/templates/user_variables_ceph.yml.j2

@@ -16,8 +16,8 @@
 ## ceph-ansible AIO settings
 is_hci: true
 common_single_host_mode: true
-monitor_interface: "{{ ('metal' in bootstrap_host_scenarios_expanded) | ternary('br-mgmt', 'eth1') }}" # Management network in the AIO
-public_network: "{{ (mgmt_range ~ '.0/' ~ netmask) | ansible.utils.ipaddr('net') }}"
+monitor_interface: "{{ ('metal' in bootstrap_host_scenarios_expanded) | ternary('br-storage', 'eth2') }}" # Storage network in the AIO
+public_network: "{{ (storage_range ~ '.0/' ~ netmask) | ansible.utils.ipaddr('net') }}"
 journal_size: 100
 osd_scenario: collocated
 ceph_conf_overrides_custom: