Update ElasticSearch
ElasticSearch update
1. Backup elasticsearch data directory
sudo service elasticsearch stop sudo tar cfz var_lib_elasticsearch_$(date +%y%m%d).tgz /var/lib/elasticsearch sudo service elasticsearch start
2. Check that ES is up and running, and note the results:
curl -X GET "localhost:9200/_cat/indices?v" health status index pri rep docs.count docs.deleted store.size pri.store.size yellow open aips 5 1 24816 1 175.4mb 175.4mb yellow open transfers 5 1 0 0 720b 720b
Check also if there are any configured snapshot:
curl -XGET "localhost:9200/_snapshot/_all?pretty"
3. Remove Snapshots
If there are any snapshot (step 1), please remove it:
curl -XDELETE "localhost:9200/_snapshot/backup-repo?pretty"
4. Stop elasticsearch service
sudo service elasticsearch stop
5. Check wich elasticsearch version is installed
Redhat/Centos: rpm -q elasticsearch Ubuntu: dpkg -l elasticsearch
6. Download a temporary copy of elasticsearch (same version than intalled)
wget https://download.elastic.co/elasticsearch/elasticsearch/elasticsearch-1.7.5.tar.gz
7. Uncompress the downloaded tar.gz and go into it
tar zxvf elasticsearch-1.7.5.tar.gz cd elasticsearch-1.7.5
8. Copy data from system wide installed elasticsearch, ot the temporary copy
sudo cp /var/lib/elasticsearch data -rf
9. Adjust file permissions
sudo chown <your user>:<your group> data -R
10. Launch the temporary elasticsearch instance
ES_JAVA_OPTS="-Xms2g -Xmx2g" ./bin/elasticsearch -d -p elastic-tmp.pid -Des.http.port=9500 -Des.discovery.zen.ping.multicast.enabled=false
11. Verify that the indexes were properly copied. Use the port specified in the previous command:
curl -X GET "localhost:9500/_cat/indices?v"
12. Remove elasticsearch 1.7.5 and it's files
Redhat: sudo yum remove elasticsearch Ubuntu: sudo apt-get remove --purge elasticsearch sudo mv /var/lib/elasticsearch /var/lib/elasticsearch-1.7.5 sudo mv /etc/elasticsearch /etc/elasticsearch-1.7.5
13. Upgrade archivematica and install elasticsearch 6
ansible-playbook -i hosts singlenode.yml --tags=elasticsearch,archivematica-src
14. Configure ElasticSearch 6 to handle reindex from the temporary es
echo 'reindex.remote.whitelist: localhost:9500' | sudo tee -a /etc/elasticsearch/elasticsearch.yml sudo service elasticsearch restart
15. Migrate the indexes
sudo -u archivematica bash -c " \
set -a -e -x
source /etc/default/archivematica-dashboard || \
source /etc/sysconfig/archivematica-dashboard \
|| (echo 'Environment file not found'; exit 1)
cd /usr/share/archivematica/dashboard
/usr/share/archivematica/virtualenvs/archivematica-dashboard/bin/python manage.py reindex_from_remote_cluster -t 60 -s 10 http://localhost:9500
";
The command should finish with
All reindex requests ended successfully!
Else, you might need to tweak the timeout (-t) and size (-s) parameters.
16. Verify that the new indexes were created and populated:
curl -X GET "localhost:9200/_cat/indices?v" health status index uuid pri rep docs.count docs.deleted store.size pri.store.size yellow open transferfiles jlu2d2yZQpWwpKKT1ZzMAA 5 1 0 0 1.1kb 1.1kb yellow open aips VAvONAByRBuVcWBYhlhUmw 5 1 116 0 66.9mb 66.9mb yellow open transfers OhfLVBzuRqCNUYRCSSTidg 5 1 0 0 1.1kb 1.1kb yellow open aipfiles clX1gcqnT9CTdwTm8jeatw 5 1 24700 0 70.6mb 70.6mb
You should have 4 indexes now, and the sum of the aips and aipfiles docs.count column should be equal to the doc.count for the aip index in ElasticSearch 1.7.5. The same happens with the transfers/transferfiles indexes.
17. Stop the temporary elasticsearch service
kill $(cat ~/elasticsearch-1.7.5/elastic-tmp.pid)
18. Restart all archivematica services
sudo service archivematica-mcp-server restart sudo service archivematica-mcp-client restart sudo service archivematica-dashboard restart sudo service archivematica-storage-service restart