Change deprecated include to include_tasks

* Add doc on 8.x support
* Updating top level readme

.kitchen.yml

@@ -22,7 +22,7 @@ provisioner:
     extra_vars:
       es_major_version: "<%= ENV['VERSION'] %>"
       <% if ENV['VERSION'] == '6.x' %>
-      es_version: '6.8.18'
+      es_version: '6.8.23'
       <% end %>
   <% end %>
 

CHANGELOG.md

@@ -1,5 +1,39 @@
 # Changelog
 
+## 7.17.0
+
+* 7.17.0 as default version.
+
+
+| PR | Author | Title |
+| --- | --- | --- |
+| [#845](https://github.com/elastic/ansible-elasticsearch/pull/845) | [@jmlrt](https://github.com/jmlrt) | Remove CentOS 8 tests  |
+
+
+## 7.16.3
+
+* 7.16.3 as default version.
+* 6.8.23 as 6.x tested version
+
+## 7.16.2
+
+* 7.16.2 as default version.
+* 6.8.22 as 6.x tested version
+
+## 7.16.1
+
+* 7.16.1 as default version.
+* 6.8.21 as 6.x tested version
+
+
+| PR | Author | Title |
+| --- | --- | --- |
+| [#838](https://github.com/elastic/ansible-elasticsearch/pull/838) | [@mgreau](https://github.com/mgreau) | Add warning message about 8.x versions  |
+| [#830](https://github.com/elastic/ansible-elasticsearch/pull/830) | [@bfontaine](https://github.com/bfontaine) | README: fix link to ES licensing change blog post  |
+| [#831](https://github.com/elastic/ansible-elasticsearch/pull/831) | [@gaima8](https://github.com/gaima8) | command necessarily causes a change on each invocation breaking idempotency, wait_for accomplishes the same goal without the change  |
+| [#833](https://github.com/elastic/ansible-elasticsearch/pull/833) | [@Pablohn26](https://github.com/Pablohn26) | Fix broken link  |
+
+
 ## 7.16.0
 
 * 7.16.0 as default version.

README.md

@@ -1,10 +1,23 @@
+# ARCHIVED
+
+This project is no longer maintained.
+
+You are welcomed to keep using it and adapting it to work for your own needs, including with Elasticsearch [8.x](docs/8x-support.md).
+
+For alternative getting started experiences, you may want to try one of these options:
+
+- Start a [free trial on Elastic Cloud](https://www.elastic.co/cloud/elasticsearch-service/signup), our hosted service.
+- Take a look at [Elastic Cloud on Kubernetes (ECK)](https://elastic.co/guide/en/cloud-on-k8s/current/k8s-quickstart.html) for launching the stack via Kubernetes.
+- Read our [Running the Elastic Stack on Docker](https://www.elastic.co/guide/en/elastic-stack-get-started/current/get-started-docker.html) guide.
+- Take a look at the [Elastic Stack Terraform provider.](https://github.com/elastic/terraform-provider-elasticstack)
+
 # ansible-elasticsearch
-[![Build Status](https://img.shields.io/jenkins/s/https/devops-ci.elastic.co/job/elastic+ansible-elasticsearch+main.svg)](https://devops-ci.elastic.co/job/elastic+ansible-elasticsearch+main/)
 [![Ansible Galaxy](https://img.shields.io/badge/ansible--galaxy-elastic.elasticsearch-blue.svg)](https://galaxy.ansible.com/elastic/elasticsearch/)
 
-**THIS ROLE IS FOR 7.x & 6.x**
 
-Ansible role for 7.x/6.x Elasticsearch.  Currently this works on Debian and RedHat based linux systems. Tested platforms are:
+**THIS ROLE IS FOR 7.x & 6.x**, but should still work with 8.x (see [note](docs/8x-support.md)).
+
+Ansible role for 7.x/6.x Elasticsearch - tests used to run and pass on the below platforms:
 
 * Ubuntu 16.04
 * Ubuntu 18.04
@@ -13,11 +26,8 @@ Ansible role for 7.x/6.x Elasticsearch.  Currently this works on Debian and RedH
 * Debian 9
 * Debian 10
 * CentOS 7
-* CentOS 8
 * Amazon Linux 2
 
-The latest Elasticsearch versions of 7.x & 6.x are actively tested.
-
 ## BREAKING CHANGES
 
 ### Notice about multi-instance support
@@ -49,7 +59,7 @@ Starting from Elasticsearch 7.11.0, OSS distributions will no longer be provided
 This Ansible role will fail if `oss_version` is set to `true` and `es_version` is greater than 
 `7.11.0`.
 
-See [Doubling down on open, Part II](https://www.elastic.co/blog/licensing-change for more details)
+See [Doubling down on open, Part II](https://www.elastic.co/blog/licensing-change)
 blog post for more details.
 
 #### How to override configuration files provided by ansible-elasticsearch?
@@ -68,7 +78,7 @@ This role uses the json_query filter which [requires jmespath](https://github.co
 Create your Ansible playbook with your own tasks, and include the role elasticsearch. You will have to have this repository accessible within the context of playbook.
 
 ```sh
-ansible-galaxy install elastic.elasticsearch,v7.16.0
+ansible-galaxy install elastic.elasticsearch,v7.17.0
 ```
 
 Then create your playbook yaml adding the role elasticsearch.
@@ -82,14 +92,14 @@ The simplest configuration therefore consists of:
   roles:
     - role: elastic.elasticsearch
   vars:
-    es_version: 7.16.0
+    es_version: 7.17.0
 ```
 
-The above installs Elasticsearch 7.16.0 in a single node 'node1' on the hosts 'localhost'.
+The above installs Elasticsearch 7.17.0 in a single node 'node1' on the hosts 'localhost'.
 
 **Note**:
 Elasticsearch default version is described in [`es_version`](https://github.com/elastic/ansible-elasticsearch/blob/main/defaults/main.yml#L2). You can override this variable in your playbook to install another version.
-While we are testing this role only with one 7.x and one 6.x version (respectively [7.16.0](https://github.com/elastic/ansible-elasticsearch/blob/main/defaults/main.yml#L2) and [6.8.18](https://github.com/elastic/ansible-elasticsearch/blob/main/.kitchen.yml#L22) at the time of writing), this role should work with other versions also in most cases.
+While we are testing this role only with one 7.x and one 6.x version (respectively [7.17.0](https://github.com/elastic/ansible-elasticsearch/blob/main/defaults/main.yml#L2) and [6.8.23](https://github.com/elastic/ansible-elasticsearch/blob/main/.kitchen.yml#L22) at the time of writing), this role should work with other versions also in most cases.
 
 This role also uses [Ansible tags](https://docs.ansible.com/ansible/2.9/user_guide/playbooks_tags.html). Run your playbook with the `--list-tasks` flag for more information.
 
@@ -421,7 +431,7 @@ In addition to es_config, the following parameters allow the customization of th
 
 * ```oss_version```  Default `false`. Setting this to `true` will install the oss release of Elasticsearch (for version <7.11.0 only).
 * `es_xpack_trial` Default `false`. Setting this to `true` will start the 30-day trail once the cluster starts.
-* ```es_version``` (e.g. "7.16.0").
+* ```es_version``` (e.g. "7.17.0").
 * ```es_api_host``` The host name used for actions requiring HTTP e.g. installing templates. Defaults to "localhost".
 * ```es_api_port``` The port used for actions requiring HTTP e.g. installing templates. Defaults to 9200. **CHANGE IF THE HTTP PORT IS NOT 9200**
 * ```es_api_basic_auth_username``` The Elasticsearch username for making admin changing actions. Used if Security is enabled. Ensure this user is admin.

defaults/main.yml

@@ -1,5 +1,5 @@
 ---
-es_version: "7.16.0"
+es_version: "7.17.0"
 es_use_snapshot_release: false
 oss_version: false
 es_package_name: "elasticsearch"

docs/8x-support.md

@@ -0,0 +1,116 @@
+# 8.x support
+
+In [December 2021](https://github.com/elastic/ansible-elasticsearch/pull/838), we made the hard decision to deprecate this Ansible playbook without adding support for Elasticsearch 8.X. 
+We acknowledge the impact this has had on many developers and organizations, and while we are not reverting the decision, we decided to share some guidelines around how to proceed from here, for folks wanting to keep using this playbook with Elasticsearch 8.X.
+
+---
+
+At a high level, this role is expected to work in most cases for fresh installs and upgrades from 7.17+ by only overriding the `es_version` variable as long as the security is enforced properly using the [SSL/TLS doc](ssl-tls-setup.md).
+
+1. install the last released version of the role from galaxy: `ansible-galaxy install elastic.elasticsearch,v7.17.0`
+
+2. copy the TLS PKCS12 keystore and truststore (https://www.elastic.co/guide/en/elasticsearch/reference/current/security-settings.html#security-http-pkcs12-files) 
+
+3. write a minimal playbook to deploy 8.2.3 on localhost:
+```yaml
+- hosts: localhost
+  roles:
+    - elastic.elasticsearch
+  vars:
+    es_version: 8.2.3
+    es_api_basic_auth_username: elastic
+    es_api_basic_auth_password: changeme
+    es_enable_http_ssl: true
+    es_enable_transport_ssl: true
+    es_ssl_keystore: "certs/keystore-password.p12"
+    es_ssl_truststore: "certs/truststore-password.p12"
+    es_ssl_keystore_password: password1
+    es_ssl_truststore_password: password2
+    es_validate_certs: no
+```
+
+4. deploy locally: `ansible-playbook es.yml`
+
+## Context for the below experiment
+
+The intent is to assess if the current playbook can still work with ES 8.X and what modifications may be needed. The testing was done on Ubuntu 20.04 and CentOS7 GCP VMs.
+
+The only code change done in the Ansible playbook was the override of the `es_version` variable.
+
+### What is working
+
+- ✅ Deploying a standalone Elasticsearch cluster in 8.2.3 with the security example playbook from 7.x:
+- ✅ managing Elasticsearch users
+- ✅ upgrading a 7.17.0 standalone cluster **with security already enabled** to 8.2.3
+- ✅ managing Elasticsearch license
+
+The below configuration was used in the tests
+
+```yaml
+- hosts: localhost
+  roles:
+  - elastic.elasticsearch
+  vars:
+    es_config:
+      xpack.security.authc.realms.file.file1.order: 0
+    es_api_basic_auth_username: elastic
+    es_api_basic_auth_password: changeme
+    es_api_sleep: 5
+    es_enable_http_ssl: true
+    es_enable_transport_ssl: true
+    es_ssl_keystore: "test/integration/files/certs/keystore-password.p12"
+    es_ssl_truststore: "test/integration/files/certs/truststore-password.p12"
+    es_ssl_keystore_password: password1
+    es_ssl_truststore_password: password2
+    es_validate_certs: no
+    es_users:
+      file:
+        es_admin:
+          password: changeMe
+          roles:
+          - admin
+        testUser:
+          password: changeMeAlso!
+          roles:
+          - power_user
+          - user
+    es_roles:
+      file:
+        admin:
+          cluster:
+          - all
+          indices:
+          - names: '*'
+             privileges:
+              - all
+      power_user:
+        cluster:
+          - monitor
+        indices:
+          - names: '*'
+            privileges:
+              - all
+      user:
+        indices:
+          - names: '*'
+            privileges:
+              - read
+```             
+
+### What is not working
+
+**Deploying an 8.X cluster with the default Ansible configuration (no security) will not work.**
+
+When runnin Elasticsearch 8.x outside of Ansible without any security configuration, Elasticsearch will autogenerate a security configuration and still activate security.
+However, when you run Elasticsearch 8.x as part of the Ansible role without any security configuration, this will fail because the Ansible role will not be able to retrieve and use the autogenerated security configuration.
+
+To tackle this, you always have to specify your own security configuration based on the [SSL/TLS doc](ssl-tls-setup.md).
+
+
+### What has not been tested
+
+**Deploying a 3 nodes cluster**
+
+When trying to deploy a 3 nodes clusters, the nodes seem to be configured successfully but they aren't able to communicate together with the test certificates (the ones used in automated standalone tests). It's highly likely that the problem lies with the tests certs themselves and not with the role.
+
+Should you be able to deploy a multi-node clusters, you will most likely have to change the configuration to use the new `node.roles` parameter ([example](https://github.com/elastic/ansible-elasticsearch/pull/772)) instead of the `node.master` and `node.data` (which got deprecated in 7.9, but the role never got [fixed](https://github.com/elastic/ansible-elasticsearch/issues/731).

tasks/elasticsearch-Debian.yml

@@ -65,7 +65,7 @@
 
 - name: Include optional user and group creation.
   when: (es_user_id is defined) and (es_group_id is defined)
-  include: elasticsearch-optional-user.yml
+  include_tasks: elasticsearch-optional-user.yml
 
 - name: Debian - Get installed elasticsearch version
   command: dpkg-query --showformat='${Version}' --show {{ es_package_name }}

tasks/elasticsearch-RedHat.yml

@@ -26,7 +26,7 @@
   when: es_use_repository
 
 - name: RedHat - include versionlock
-  include: elasticsearch-RedHat-version-lock.yml
+  include_tasks: elasticsearch-RedHat-version-lock.yml
 
 - name: RedHat - Remove the other elasticsearch package if switching between OSS and standard
   become: yes
@@ -36,7 +36,7 @@
 
 - name: Include optional user and group creation.
   when: (es_user_id is defined) and (es_group_id is defined)
-  include: elasticsearch-optional-user.yml
+  include_tasks: elasticsearch-optional-user.yml
 
 - name: RedHat - Install Elasticsearch
   become: yes

tasks/elasticsearch.yml

@@ -1,9 +1,9 @@
 ---
 
 - name: Include specific Elasticsearch
-  include: elasticsearch-Debian.yml
+  include_tasks: elasticsearch-Debian.yml
   when: ansible_os_family == 'Debian'
 
 - name: Include specific Elasticsearch
-  include: elasticsearch-RedHat.yml
+  include_tasks: elasticsearch-RedHat.yml
   when: ansible_os_family == 'RedHat'

tasks/main.yml

@@ -12,49 +12,49 @@
       - always
 
 - name: set compatibility variables
-  include: compatibility-variables.yml
+  include_tasks: compatibility-variables.yml
   tags:
       - always
 
 - name: check-set-parameters
-  include: elasticsearch-parameters.yml
+  include_tasks: elasticsearch-parameters.yml
   tags:
       - always
 
 - name: use snapshot release
-  include: snapshot-release.yml
+  include_tasks: snapshot-release.yml
   when: es_use_snapshot_release
 
 - name: include java.yml
-  include: java.yml
+  include_tasks: java.yml
   when: es_java_install
   tags:
       - java
 
 - name: include elasticsearch.yml
-  include: elasticsearch.yml
+  include_tasks: elasticsearch.yml
   tags:
       - install
 
 - name: include elasticsearch-config.yml
-  include: elasticsearch-config.yml
+  include_tasks: elasticsearch-config.yml
   tags:
       - config
 
 - name: include elasticsearch-plugins.yml
-  include: elasticsearch-plugins.yml
+  include_tasks: elasticsearch-plugins.yml
   when: es_plugins is defined or es_plugins_reinstall
   tags:
       - plugins
 
   #We always execute xpack as we may need to remove features
 - name: include xpack/elasticsearch-xpack.yml
-  include: xpack/elasticsearch-xpack.yml
+  include_tasks: xpack/elasticsearch-xpack.yml
   tags:
       - xpack
 
 - name: include elasticsearch-ssl.yml
-  include: elasticsearch-ssl.yml
+  include_tasks: elasticsearch-ssl.yml
   when: es_enable_http_ssl or es_enable_transport_ssl
   tags:
       - xpack
@@ -88,25 +88,25 @@
   when: manage_native_realm | bool
 
 - name: activate-license
-  include: ./xpack/security/elasticsearch-xpack-activation.yml
+  include_tasks: ./xpack/security/elasticsearch-xpack-activation.yml
   when: es_start_service and not oss_version and es_xpack_license is defined and es_xpack_license != ''
   run_once: True
 
 - name: activate-trial
-  include: ./xpack/security/elasticsearch-xpack-trial-activation.yml
+  include_tasks: ./xpack/security/elasticsearch-xpack-trial-activation.yml
   when: es_start_service and not oss_version and es_xpack_trial
   run_once: True
 
 #perform security actions here now elasticsearch is started
 - name: include xpack/security/elasticsearch-security-native.yml
-  include: ./xpack/security/elasticsearch-security-native.yml
+  include_tasks: ./xpack/security/elasticsearch-security-native.yml
   when: manage_native_realm | bool
   run_once: True
 
 #Templates done after restart - handled by flushing the handlers. e.g. suppose user removes security on a running node and doesn't specify es_api_basic_auth_username and es_api_basic_auth_password.  The templates will subsequently not be removed if we don't wait for the node to restart.
 #We also do after the native realm to ensure any changes are applied here first and its denf up.
 - name: include elasticsearch-template.yml
-  include: elasticsearch-template.yml
+  include_tasks: elasticsearch-template.yml
   when: es_templates | bool
   tags:
       - templates

tasks/xpack/elasticsearch-xpack.yml

@@ -2,7 +2,7 @@
 
 #Security configuration
 - name: include security/elasticsearch-security.yml
-  include: security/elasticsearch-security.yml
+  include_tasks: security/elasticsearch-security.yml
   when: not oss_version
 
 #Make sure elasticsearch.keystore has correct Permissions

tasks/xpack/security/elasticsearch-security.yml

@@ -73,7 +73,7 @@
 
 #-----------------------------FILE BASED REALM----------------------------------------
 
-- include: elasticsearch-security-file.yml
+- include_tasks: elasticsearch-security-file.yml
   when: (es_users is defined and es_users.file is defined) or (es_roles is defined and es_roles.file is defined)
 
 #-----------------------------ROLE MAPPING ----------------------------------------

test/integration/custom-config.yml

@@ -2,7 +2,7 @@
 - name: Setup Elasticsearch with custom config files
   hosts: localhost
   post_tasks:
-    - include: elasticsearch/test/integration/debug.yml
+    - include_tasks: elasticsearch/test/integration/debug.yml
   roles:
     - elasticsearch
   vars:

test/integration/default.yml

@@ -2,6 +2,6 @@
 - name: Setup Elasticsearch using default configuration
   hosts: localhost
   post_tasks:
-    - include: elasticsearch/test/integration/debug.yml
+    - include_tasks: elasticsearch/test/integration/debug.yml
   roles:
     - elasticsearch

test/integration/license.yml

@@ -2,7 +2,7 @@
 - name: Elasticsearch Xpack HTTP different keystore and truststore with password
   hosts: localhost
   post_tasks:
-    - include: elasticsearch/test/integration/debug.yml
+    - include_tasks: elasticsearch/test/integration/debug.yml
   roles:
     - elasticsearch
   vars:

test/integration/oss-to-default-upgrade.yml

@@ -2,7 +2,7 @@
 - name: Deploy Elasticsearch OSS version
   hosts: localhost
   post_tasks:
-    - include: elasticsearch/test/integration/debug.yml
+    - include_tasks: elasticsearch/test/integration/debug.yml
   roles:
     - elasticsearch
   vars:
@@ -12,7 +12,7 @@
 - name: Upgrade to Elasticsearch default version
   hosts: localhost
   post_tasks:
-    - include: elasticsearch/test/integration/debug.yml
+    - include_tasks: elasticsearch/test/integration/debug.yml
   roles:
     - elasticsearch
   vars:

test/integration/oss-upgrade.yml

@@ -2,7 +2,7 @@
 - name: Deploy Elasticsearch OSS previous version
   hosts: localhost
   post_tasks:
-    - include: elasticsearch/test/integration/debug.yml
+    - include_tasks: elasticsearch/test/integration/debug.yml
   roles:
     - elasticsearch
   vars:
@@ -12,7 +12,7 @@
 - name: Deploy Elasticsearch OSS latest version
   hosts: localhost
   post_tasks:
-    - include: elasticsearch/test/integration/debug.yml
+    - include_tasks: elasticsearch/test/integration/debug.yml
   roles:
     - elasticsearch
   vars:

test/integration/oss.yml

@@ -2,7 +2,7 @@
 - name: Setup Elasticsearch OSS 6.x
   hosts: localhost
   post_tasks:
-    - include: elasticsearch/test/integration/debug.yml
+    - include_tasks: elasticsearch/test/integration/debug.yml
   roles:
     - elasticsearch
   vars:

test/integration/security.yml

@@ -2,7 +2,7 @@
 - name: Elasticsearch Xpack HTTP different keystore and truststore with password
   hosts: localhost
   post_tasks:
-    - include: elasticsearch/test/integration/debug.yml
+    - include_tasks: elasticsearch/test/integration/debug.yml
   roles:
     - elasticsearch
   vars:

test/integration/trial.yml

@@ -2,7 +2,7 @@
 - name: Setup Elasticsearch with security enabled and a trial license
   hosts: localhost
   post_tasks:
-    - include: elasticsearch/test/integration/debug.yml
+    - include_tasks: elasticsearch/test/integration/debug.yml
   roles:
     - elasticsearch
   vars:

test/integration/upgrade.yml

@@ -2,7 +2,7 @@
 - name: Deploy Elasticsearch previous version
   hosts: localhost
   post_tasks:
-    - include: elasticsearch/test/integration/debug.yml
+    - include_tasks: elasticsearch/test/integration/debug.yml
   roles:
     - elasticsearch
   vars:
@@ -11,6 +11,6 @@
 - name: Deploy Elasticsearch latest version
   hosts: localhost
   post_tasks:
-    - include: elasticsearch/test/integration/debug.yml
+    - include_tasks: elasticsearch/test/integration/debug.yml
   roles:
     - elasticsearch

test/matrix-6x.yml

@@ -6,7 +6,6 @@ OS:
   - debian-9
   - debian-10
   - centos-7
-  - centos-8
   - amazonlinux-2
 TEST_TYPE:
   - custom-config

test/matrix.yml

@@ -6,7 +6,6 @@ OS:
   - debian-9
   - debian-10
   - centos-7
-  - centos-8
   - amazonlinux-2
 TEST_TYPE:
   - custom-config