...
...
| Table of Contents |
|---|
Implementation of LB Cluster solution deployment is based on ansible playbooks. Parameters that are important for deployment are defined in configuration files, playbook commands are set in makefile. To start deployment user runs command "make deploy" (see user rights requirements), "make" is a command which executes command from makefile, "deploy" is deployFE" and "deployFEJ" are the main deployment commandcommands. "Make" utility finds "deployFE" or "deploydeployFEJ" label in makefile and runs script for ansible playbook.
- Run "make deployFE" to deploy cluster with FIXEdge C++
- Run "make deployFEJ" to deploy cluster with FIXEdge Java
Distribution Package
Part of needed files included in distribution package
| File description | Example | |
|---|---|---|
| FIXEdge distribution archive | FIXEdge-6.6.1-lb-cluster.138-FA-2.25.0.138-Linux-3.10.0-gcc-4.8.5-x86_64.tar.gz | |
| Configuration Service distribution archive | configuration-service-1.0.0-SNAPSHOT*.tar | |
| REST API key file | rapi.key | |
| REST API certificate | rapi.crtScheduler service distribution archive | scheduler-service*.tar |
| Playbook files | yml configuration files, readme files, etc. |
files Files not included in distribution package - license files, TLS certificates, Consul distribution packages.
Playbook
Playbook and all deployment-dependent files are saved on deployment workstation - Linux-based machine that may be a separate machine or one of the hosts configured for cluster components.
Playbook requires
- ansible installed on deployment workstation with all dependent packages (ansible version 2.5.1 and higher)
- passwordless sudo configured on all hosts
- python installed on all hosts (version 2.7 and higher)
- ssh access to all hosts
- "make" utility available on deployment workstation
User runs ansible playbook using "make" utility on deployment workstation.
example of makefile:
| Code Block | ||||
|---|---|---|---|---|
| ||||
APB=ansible-playbook INVENTORY=hosts.yml PB=deploy.yml SSH_ARGS=-o StrictHostKeyChecking=no # Default ANSIBLE_FLAGS_FIXEDGE=--skip-tags # Default target all: deploy # Target to deploy the services to the configured machines deploydeployFE: ${INVENTORY} ${PB} ${APB} -i ${INVENTORY} ${ANSIBLE_FLAGS_FIXEDGE} java ${PB} --ssh-common-args="${SSH_ARGS}" -vv deployFEJ: ${INVENTORY} ${PB} ${APB} -i ${INVENTORY} ${ANSIBLE_FLAGS_FIXEDGE} c++ ${PB} --ssh-common-args="${SSH_ARGS}" |
User rights
Deployment process require superuser rights on all hosts for user that will run playbook.
Parameters Configuration
Directory structure
| Name | Description |
|---|---|
| PLAYBOOK_ROOT_DIR | root directory for deployment files Contains: ansible.cfg - ansible configuration file deploy.yml - steps of deployment hosts.yml - file for hardware units configuration makefile - file for "make" command arguments Readme.MD - root readme document |
PLAYBOOK_ROOT_DIR/doc PLAYBOOK_ROOT_DIR/doc/quickstart | directories for documentation files Contains: readme files |
| PLAYBOOK_ROOT_DIR/files | directory for distribution packages, license, key files |
| PLAYBOOK_ROOT_DIR/group_vars | directory for files containing global variables |
| PLAYBOOK_ROOT_DIR/roles | directory for deployment-related files for roles |
Roles
For convenience all deployment jobs are divided into parts according to functionality that is deployed. These parts are called "roles". Parameters for roles are defined in PLAYBOOK_ROOT_DIR/roles/<role_name>/defaults/main.yml. These parameters can be overrided by global parameters with the same name. For example user can define "fe_destdir" parameter in all.yml file and it will be taken by ansible playbook instead of parameter defined in PLAYBOOK_ROOT_DIR/roles/fixedge/defaults/main.yml
Hosts
Hardware unit parameters are stored in hosts.yml
| Parameter | Description |
|---|---|
| ansible_host | IP address for hw unit |
| ansible_port | IP port used for ssh access on hw units |
| ansible_user | user login used by ansible for changing configuration on hw unit |
| ansible_ssh_pass | user ssh pass used by ansible for changing configuration on hw unit |
| ansible_connection: | used for docker implementation of Oracle DB in test configuration |
| ansible_connection_args: | used for docker implementation of Oracle DB in test configuration |
| Code Block | ||||||
|---|---|---|---|---|---|---|
| ||||||
---
all:
children:
# Host names or IP addresses where FIXEdge daemon with Consul Agent
# will be deployed.
fixedge:
hosts:
fixedge_1:
ansible_host: 10.6.223.22
ansible_port: 22
ansible_user: user
ansible_ssh_pass: epm-bfix
fixedge_2:
ansible_host: 10.6.220.172
ansible_port: 22
ansible_user: user
ansible_ssh_pass: epm-bfix
# Host names or IP addresses where Configuration Service daemon with
# Consul Agent will be deployed.
confservice:
hosts:
cs_1:
ansible_host: 10.6.221.249
ansible_port: 22
ansible_user: user
ansible_ssh_pass: epm-bfix
# Host names or IP addresses where HAProxy with Consul Agent and
# Consul Template will be deployed/configured.
proxy:
hosts:
haproxy:
ansible_host: 10.6.223.32
ansible_port: 22
ansible_user: user
ansible_ssh_pass: epm-bfix
# Host name or IP address of the Oracle Database
database:
hosts:
oracle:
ansible_host: user_oracle-database_1
ansible_connection: docker
ansible_connection_args: '-H=tcp://10.6.221.187:2375'
# Host name or IP address of the Oracle Database
consul_cluster:
hosts:
consul_server_1:
ansible_host: 10.6.221.187
ansible_port: 22
ansible_user: user
ansible_ssh_pass: epm-bfix
splunk:
hosts:
splunk_1:
ansible_host: 10.6.221.187
ansible_port: 22
ansible_user: user
ansible_ssh_pass: epm-bfix |
Global variables
| variable | component | description | required for FIXEdge C++ | required | description | |
|---|---|---|---|---|---|---|
| dba_user | Oracle DB | + | + | DB Administrator login. Database user with administrator rights is needed for adding user, create and configure database for the solution needs. | ||
| dba_password | Oracle DB | + | + | DB Administrator password | ||
| db_user | Oracle DB | + | + | DB user login | ||
| db_password | Oracle DB | + | + | DB user password | ||
| db_address | Oracle DB | + | + | IP address for connection to DB | ||
| db_port | Oracle DB | + | + | IP port for connection to DB | ||
| db_sid | Oracle DB | + | + | DB SIDfe | ||
| file_ | cluster_idconsul | Cluster | unique identity of LB Cluster | |||
| fe_lic_dnl | FIXEdge nodes | path to license file | ||||
| fe_archive_dest | FIXEdge nodes | path to FIXEdgeConsul | + | + | path to Consul Agent distribution archive | |
| file_ | csConfiguration Service | path to Configuration Service distribution archive | ||||
| file_consul | Consul | path to Consul Agent distribution archive | ||||
| file_ctemplate | Consul | path to Consul template software package (ctemplate | Consul | + | + | path to Consul template software package (ZIP archive). |
| fe_rapicluster_keyid | FIXEdge nodes | path to REST-API key file | ||||
| fe_rapi.crt | FIXEdge nodes | path to REST-API certificate file | ||||
| fe_rapi_port | FIXEdge nodes | IP port for REST-APICluster | + | + | unique identity of FIXEdge C++ LB Cluster | |
| file_cs | Configuration Service | + | + | path to Configuration Service distribution archive | ||
| fe_splunk_host | Splunk | + | + | IP address for connection to Splunk system | ||
| fe_splunk_port | Splunk | + | + | IP port for connection to Splunk system | ||
| consul_deploy_dir | Consul | + | + | path for Consul deployment |
Global variables are stored in group_vars/all.yml file
...
| title | example for all.yml |
|---|---|
| collapse | true |
...
| fe_archive_dest_scheduler | Scheduler Service | + | + | path to Scheduler Service distribution archive |
| fe_lic_dnl | FIXEdge C++ nodes | + | path to license file for FIXEdge C++ | |
| fe_archive_dest | FIXEdge C++ nodes | + | path to FIXEdge C++ distribution archive | |
| fe_rapi_key | FIXEdge C++ nodes | + | path to REST-API key file | |
| fe_rapi.crt | FIXEdge C++ nodes | + | path to REST-API certificate file | |
| fe_rapi_port | FIXEdge C++ nodes | + | IP port for REST-API | |
fe_lic_dnl_java | FIXEdge Java nodes | + | path to license file for FIXEdge Java | |
| fe_archive_dest_java | FIXEdge Java nodes | + | path to FIXEdge Java distribution archive | |
| fe_archive_dest_fo | FIXEdge Java nodes | + | path to FO Java distribution archive |
Global variables are stored in group_vars/all.yml file
| Code Block | ||||||
|---|---|---|---|---|---|---|
| ||||||
--- lbc_dedicated_network_configuration_service_address: 10.3.0.11 # Oracle dba_user: system dba_password: oracle db_user: fix db_password: fix db_addr: '10.3.0.5' db_port: 1521 db_sid: XE # Clusters identifier fe_cluster_id: 'FixEdge1' fej_cluster_id: 'FIXEdge-Java-1' fe_lic_dnl: '{{ playbook_dir }}/files/engine.license' fe_archive_dest: '{{ playbook_dir }}/files/FIXEdgeLBC-6.8.1_141.tar.gz' fe_rapi_key: '{{ playbook_dir }}/files/rapi.key' fe_rapi_crt: '{{ playbook_dir }}/files/rapi.crt' fe_rapilic_dnl_portjava: '8903{{ playbook_dir }}/files/fixaj2-license.bin' fe_splunkarchive_dest_hostjava: '10.6.221.187{{ playbook_dir }}/files/fixedgej-lbc-1.3.1.zip' fe_archive_splunkdest_portfo: '1514' consul_deploy{{ playbook_dir: '}}/srv/consul-agent' consul_server_addresses: | {%- set addresses = [] -%} {%- for item in groups['consul_cluster'] -%} {%- if addresses.append(hostvars[item]['ansible_default_ipv4']['address']) -%} {%- endif -%} {%- endfor -%} {{ addresses }} |
Troubleshooting
Deployment process will display messages on the terminal output indicating results for every step. When all the steps finished successfully user will get the output similar to:
Incorrect deployment process ending will be similar to:
Host is unreachable
Problem: user gets error message similar to:
| Warning |
|---|
LAY [Debug variables] *******************************************************files/fo-storage-server-1.3.21-bin.tar'
fe_archive_dest_scheduler: '{{ playbook_dir }}/files/scheduler-service-1.0.0.tar'
file_cs: '{{ playbook_dir }}/files/configuration-service-1.1.0.tar'
file_consul: '{{ playbook_dir }}/files/consul_1.5.2_linux_amd64.zip'
file_ctemplate: '{{ playbook_dir }}/files/consul-template_0.20.0_linux_amd64.zip'
fe_rapi_port: '8903'
fe_splunk_host: '10.3.0.10'
fe_splunk_port: '1514'
fe_fix_port: '8901'
fe_fixssl_port: '8905'
fe_fix_java_port: '8911'
consul_deploy_dir: '/srv/consul-agent'
consul_server_addresses: |
{%- set addresses = [] -%}
{%- for item in groups['consul_cluster'] -%}
{%- if addresses.append(hostvars[item]['ansible_default_ipv4']['address']) -%}
{%- endif -%}
{%- endfor -%}
{{ addresses }}
|
Troubleshooting
Deployment process will display messages on the terminal output indicating results for every step. When all the steps finished successfully user will get the output similar to:
Incorrect deployment process ending will be similar to:
Host is unreachable
Problem: user gets error message similar to:
| Warning |
|---|
LAY [Debug variables] **************************************************************************************************************************************** TASK [Gathering Facts] ***************** ***************************************************************************************************************************************************** **********************task path: /home/egor/work/ansible-play-lbc/deploy.yml:3 ok: [cs_1] ok: [haproxy] ok: [fixedge_2] ok: [consul_server_1] fatal: [fixedge_1]: UNREACHABLE! => {"changed": false, "msg": "Failed to connect to the host via ssh: ssh: connect to host 10.6.223.22 port 22: Connection timed out\r\n", "unreachable": true} |
Description: possible reason may be that host is unreachable via network
Solution:
- check network connectivity to troubled host (ip address is present in error message). Restore network connectivity if it was lost.
Role not found
Problem: user gets error message similar to:
| Warning |
|---|
| env DOCKER_HOST="tcp://10.6.221.187:2375" ansible-playbook -i hosts.yml deploy.yml --ssh-common-args="-o StrictHostKeyChecking=no" -vv ansible-playbook 2.5.1 config file = /home/user/work/ansible-play-lbc/ansible.cfg configured module search path = [u'/home/user/.ansible/plugins/modules', u'/usr/share/ansible/plugins/modules'] ansible python module location = /usr/lib/python2.7/dist-packages/ansible executable location = /usr/bin/ansible-playbook python version = 2.7.15rc1 (default, Nov 12 2018, 14:31:15) [GCC 7.3.0] Using /home/user/work/ansible-play-lbc/ansible.cfg as config file ERROR! the role 'ha-proxy' was not found in /home/user/work/ansible-play-lbc/roles:/home/user/.ansible/roles:/usr/share/ansible/roles:/etc/ansible/roles:/home/user/work/ansible-play-lbc The error appears to have been in '/home/user **** be elsewhere in the file depending on the exact syntax problem. The offending line appears to be: - consul-agent - ha-proxy ^ here makefile:11: recipe for target 'deploy' failed make: *** [deploy] Error 1 user@lbc:~/work/ansible-play-lbc$ :3 |
Description: possible reason may be that role name defined in PLAYBOOK_ROOT_DIR/deploy.yml is wrong - due to human mistake it differs from role name in PLAYBOOK_ROOT_DIR/roles
Solution:
...
host is unreachable via network
Solution:
- check network connectivity to troubled host (ip address is present in error message). Restore network connectivity if it was lost.
Wrong oracle access parameters
Problem: user gets error message similar to:
| Warning |
|---|
TASK [setup-oracle-db : Cleanup Oracle Database contents] ********************************************************************************************************************************************************* PLAY RECAP ******************************************
********************************************************************************************************************************************************* *****
|
Description: possible reason may be wrong access parameters
...
- correct database access parameters in PLAYBOOK_ROOT_DIR/group_vars/all.yml
- check that access to database works using database client software
Database (oracle) unavailable
Problem: user gets error message similar to:
| Warning |
|---|
TASK [setup-oracle-db : Cleanup Oracle Database contents] ********************************************************************************************************************************************************* ********************************************************************************************************************************************************* ***********************************************
|
Description: possible reason may be in database. It may be unreachable or may work not properly
Solution:
- check network connectivity. Restore it if needed.
- try to access database with database client using credentials configured in deploy config files.
- check database health.
Lack of user rights
Problem: user gets error message similar to:
| Warning |
|---|
TASK [consul-agent : Create the Consul Agent deployment directory] ********************************************************************************************************************************************************* ********************************************************************************************************************************************************* *
TASK [consul-agent : set_fact] **************
********************************************************************************************************************************************************* *************
|
Description: the reason is that passwordless sudo is not configured for user on one of the hosts.
...
- Configure passwordless sudo on host (ip address is present in the error message)
File is not found
Problem: user gets error message similar to:
| Warning |
|---|
TASK [cs : Create the Configuration Service destination directory] ********************************************************************************************************************************************************* task path: /home/user/work/ansible-play-lbc/roles/cs/tasks/main.yml:17 ********************************************************************************************************************************************************* **task path: /home/user/work/ansible-play-lbc/roles/cs/tasks/main.yml:23 ********************************************************************************************************************************************************* |
Description: deployment script can not find one of archive files for some reasons
...