Difference between revisions of "Update ElasticSearch"

From Archivematica
Jump to navigation Jump to search
Line 15: Line 15:
 
   sudo service elasticsearch stop
 
   sudo service elasticsearch stop
  
 +
3. Check wich elasticsearch version is installed
  
−
3. Download a temporary copy of elasticsearch (same version than intalled)
+
  Redhat/Centos: rpm -qa | grep elasticsearch
 +
  Ubuntu: dpkg -l | grep elasticsearch
 +
 
 +
4. Download a temporary copy of elasticsearch (same version than intalled)
  
 
   wget https://download.elastic.co/elasticsearch/elasticsearch/elasticsearch-1.7.5.tar.gz
 
   wget https://download.elastic.co/elasticsearch/elasticsearch/elasticsearch-1.7.5.tar.gz
  
  
−
4. Uncompress the downloaded tar.gz and go into it
+
5. Uncompress the downloaded tar.gz and go into it
  
 
   tar zxvf elasticsearch-1.7.5.tar.gz
 
   tar zxvf elasticsearch-1.7.5.tar.gz
Line 27: Line 31:
  
  
−
5. Copy data from system wide installed elasticsearch, ot the temporary copy
+
6. Copy data from system wide installed elasticsearch, ot the temporary copy
  
 
   cp /var/lib/elasticsearch data -rf
 
   cp /var/lib/elasticsearch data -rf
  
−
6. Adjust file permissions
+
7. Adjust file permissions
  
 
   chown <your user>:<your group> data -R
 
   chown <your user>:<your group> data -R
  
−
7. Launch the temporary elasticsearch instance
+
8. Launch the temporary elasticsearch instance
  
 
   ./bin/elasticsearch -d -p elastic-tmp.pid -Des.http.port=9500
 
   ./bin/elasticsearch -d -p elastic-tmp.pid -Des.http.port=9500
  
−
8. Verify that the indexes were properly copied
+
9. Verify that the indexes were properly copied
  
 
   curl -X GET "localhost:9500/_cat/indices?v"
 
   curl -X GET "localhost:9500/_cat/indices?v"
  
−
9. Remove elasticsearch 1.7.5 and it's files
+
10. Remove elasticsearch 1.7.5 and it's files
  
 
   Redhat: sudo yum remove elasticsearch
 
   Redhat: sudo yum remove elasticsearch
Line 49: Line 53:
 
   rm -rf /var/lib/elasticsearch
 
   rm -rf /var/lib/elasticsearch
  
−
10. Upgrade archivematica and install elasticsearch 6
+
11. Upgrade archivematica and install elasticsearch 6
  
 
   ansible-playbook -i hosts singlenode.yml --tags=elasticsearch,archivematica-src
 
   ansible-playbook -i hosts singlenode.yml --tags=elasticsearch,archivematica-src
  
−
11. Configure ElasticSearch 6 to handle reindex from the temporary es  
+
12. Configure ElasticSearch 6 to handle reindex from the temporary es  
  
 
   echo 'reindex.remote.whitelist: localhost:9500' | sudo tee -a /etc/elasticsearch/elasticsearch.yml
 
   echo 'reindex.remote.whitelist: localhost:9500' | sudo tee -a /etc/elasticsearch/elasticsearch.yml
 
   sudo service elasticsearch restart
 
   sudo service elasticsearch restart
  
−
12. Migrate the indexes
+
13. Migrate the indexes
  
 
   sudo -u archivematica bash -c " \
 
   sudo -u archivematica bash -c " \
Line 74: Line 78:
  
  
−
13. Verify that the new indexes were created and populated:
+
14. Verify that the new indexes were created and populated:
 
    
 
    
 
   curl -X GET "localhost:9200/_cat/indices?v"  
 
   curl -X GET "localhost:9200/_cat/indices?v"  
Line 85: Line 89:
 
You should have 4 indexes now, and the sum of the aips and aipfiles docs.count column should be equal to the doc.count for the aip index in ElasticSearch 1.7.5. The same happens with the transfers/transferfiles indexes.
 
You should have 4 indexes now, and the sum of the aips and aipfiles docs.count column should be equal to the doc.count for the aip index in ElasticSearch 1.7.5. The same happens with the transfers/transferfiles indexes.
  
−
14. Stop the temporary elasticsearch service
+
15. Stop the temporary elasticsearch service
  
 
   kill $(cat ~/elasticsearch-1.7.5/elastic-tmp.pid)
 
   kill $(cat ~/elasticsearch-1.7.5/elastic-tmp.pid)
  
−
15. Restart all archivematica services
+
16. Restart all archivematica services
 
    
 
    
 
   service archivematica-dashboard restart
 
   service archivematica-dashboard restart

Revision as of 11:41, 19 February 2019

ElasticSearch update

1. Check that ES is up and running, and note the results: 

 curl -X GET "localhost:9200/_cat/indices?v"
 health status index     pri rep docs.count docs.deleted store.size pri.store.size 
 yellow open   aips        5   1         11            0    267.9kb        267.9kb 
 yellow open   transfers   5   1         24            0     35.2kb         35.2kb

2. Stop elasticsearch service 

 sudo service elasticsearch stop

3. Check wich elasticsearch version is installed 

 Redhat/Centos: rpm -qa | grep elasticsearch
 Ubuntu: dpkg -l | grep elasticsearch

4. Download a temporary copy of elasticsearch (same version than intalled) 

 wget https://download.elastic.co/elasticsearch/elasticsearch/elasticsearch-1.7.5.tar.gz


5. Uncompress the downloaded tar.gz and go into it 

 tar zxvf elasticsearch-1.7.5.tar.gz
 cd elasticsearch-1.7.5


6. Copy data from system wide installed elasticsearch, ot the temporary copy 

 cp /var/lib/elasticsearch data -rf

7. Adjust file permissions 

 chown <your user>:<your group> data -R

8. Launch the temporary elasticsearch instance 

 ./bin/elasticsearch -d -p elastic-tmp.pid -Des.http.port=9500

9. Verify that the indexes were properly copied 

 curl -X GET "localhost:9500/_cat/indices?v"

10. Remove elasticsearch 1.7.5 and it's files 

 Redhat: sudo yum remove elasticsearch
 Ubuntu: sudo apt-get remove elasticsearch
 rm -rf /var/lib/elasticsearch

11. Upgrade archivematica and install elasticsearch 6 

 ansible-playbook -i hosts singlenode.yml --tags=elasticsearch,archivematica-src

12. Configure ElasticSearch 6 to handle reindex from the temporary es 

 echo 'reindex.remote.whitelist: localhost:9500' | sudo tee -a /etc/elasticsearch/elasticsearch.yml
 sudo service elasticsearch restart

13. Migrate the indexes 

 sudo -u archivematica bash -c " \
 set -a -e -x
 source /etc/default/archivematica-dashboard || \
 source /etc/sysconfig/archivematica-dashboard \
       || (echo 'Environment file not found'; exit 1)
 cd /usr/share/archivematica/dashboard
 /usr/share/archivematica/virtualenvs/archivematica-dashboard/bin/python manage.py reindex_from_remote_cluster http://localhost:9500
 ";

The command should finish with 

 All reindex requests ended successfully!


14. Verify that the new indexes were created and populated: 

 curl -X GET "localhost:9200/_cat/indices?v" 
 health status index         uuid                   pri rep docs.count docs.deleted store.size pri.store.size
 yellow open   aips          IL7lbAZFR9adKg-GpgepTw   5   1          1            0    135.8kb        135.8kb
 yellow open   transferfiles hlG1GBwIRNCi1rI74LhQpQ   5   1         23            0     49.1kb         49.1kb
 yellow open   transfers     ebRiOvutR_e7N4dNGoNq4w   5   1          1            0        6kb            6kb
 yellow open   aipfiles      2QFnLdr7Txi2qgG2C7-eJw   5   1         10            0    158.7kb        158.7kb

You should have 4 indexes now, and the sum of the aips and aipfiles docs.count column should be equal to the doc.count for the aip index in ElasticSearch 1.7.5. The same happens with the transfers/transferfiles indexes.

15. Stop the temporary elasticsearch service 

 kill $(cat ~/elasticsearch-1.7.5/elastic-tmp.pid)

16. Restart all archivematica services 

 service archivematica-dashboard restart
 service archivematica-mcp-server restart
 service archivematica-mcp-client restart
 service archivematica-storage-service restart
Retrieved from "https://wiki.archivematica.org/index.php?title=Update_ElasticSearch&oldid=12885"