Learn about Centmin Mod LEMP Stack today
Become a Member

Xenforo Elasticsearch Install for Xenforo

Discussion in 'Forum software usage' started by Sunka, Nov 27, 2015.

  1. Sunka

    Sunka Well-Known Member

    1,140
    312
    83
    Oct 31, 2015
    Pula, Croatia
    Ratings:
    +508
    Local Time:
    1:08 PM
    Nginx 1.17.9
    MariaDB 10.3.22
    It is easy to say for you. :D
     
  2. Xon

    Xon Active Member

    170
    61
    28
    Nov 16, 2015
    Ratings:
    +223
    Local Time:
    7:08 PM
    1.15.x
    MariaDB 10.3.x
    This is a bit late, but no :p

    I'll be working on a fix this weekend, as I need to check Enhanced Search Improvements add-on too.
     
  3. Sunka

    Sunka Well-Known Member

    1,140
    312
    83
    Oct 31, 2015
    Pula, Croatia
    Ratings:
    +508
    Local Time:
    1:08 PM
    Nginx 1.17.9
    MariaDB 10.3.22
    Please @Xon after you update your addon, write a notice should we rebuild search after upgrade your addon or we do not have to (y)
     
  4. Xon

    Xon Active Member

    170
    61
    28
    Nov 16, 2015
    Ratings:
    +223
    Local Time:
    7:08 PM
    1.15.x
    MariaDB 10.3.x
    I think you'll need to rebuild the index.

    Note; With elasticsearch 5 you can no longer set "index." settings in elasticsearch.yml file.

    This means the following will break:
    Code:
    index.number_of_shards: 1
    index.number_of_replicas: 0
    
    To "Fix" this you must use indices templates
    Code:
    curl -XPUT 'http://localhost:9200/_template/default' -d'{
      "template": "*",
      "settings": {
        "number_of_shards": "1",
        "number_of_replicas": "0"
      }
    }'
    
    Note; you must quote the "0" for number_of_replicas of it fails to apply.

    You can change live number of replicas with:
    Code:
    curl -XPUT 'http://localhost:9200/_settings?preserve_existing=true' -d '{
      "index.number_of_replicas" : "0"
    }'
    
     
  5. eva2000

    eva2000 Administrator Staff Member

    44,750
    10,204
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +15,815
    Local Time:
    9:08 PM
    Nginx 1.17.x
    MariaDB 5.5/10.x
    wow Elasticsearch surely loves to changes things up :)
     
    Last edited: Nov 4, 2016
  6. Xon

    Xon Active Member

    170
    61
    28
    Nov 16, 2015
    Ratings:
    +223
    Local Time:
    7:08 PM
    1.15.x
    MariaDB 10.3.x
    I believed I've finished updating Elastic Search Essentials to work with ES5, but needs testing which I'll finish after work.
     
  7. Sunka

    Sunka Well-Known Member

    1,140
    312
    83
    Oct 31, 2015
    Pula, Croatia
    Ratings:
    +508
    Local Time:
    1:08 PM
    Nginx 1.17.9
    MariaDB 10.3.22



    So, just to c/p that in cli and execute?
    Code:
    curl -XPUT 'http://localhost:9200/_template/default' -d'{
      "template": "*",
      "settings": {
        "number_of_shards": "1",
        "number_of_replicas": "0"
      }
    }'
    
     
  8. Xon

    Xon Active Member

    170
    61
    28
    Nov 16, 2015
    Ratings:
    +223
    Local Time:
    7:08 PM
    1.15.x
    MariaDB 10.3.x
    Yup. BUt this will only apply the next time you delete your index.

    Use the following to change live indexes:
    Code:
    curl -XPUT 'http://localhost:9200/_settings?preserve_existing=true' -d '{
      "index.number_of_replicas" : "0"
    }'
    
     
  9. Sunka

    Sunka Well-Known Member

    1,140
    312
    83
    Oct 31, 2015
    Pula, Croatia
    Ratings:
    +508
    Local Time:
    1:08 PM
    Nginx 1.17.9
    MariaDB 10.3.22
    I rebuilded several times search index from than (y)

    This is only for replicas, what about shards?
    Maybe this?
    Code:
    curl -XPUT 'http://localhost:9200/_settings?preserve_existing=true' -d '{
    "number_of_shards": "1"}'
     
  10. Sunka

    Sunka Well-Known Member

    1,140
    312
    83
    Oct 31, 2015
    Pula, Croatia
    Ratings:
    +508
    Local Time:
    1:08 PM
    Nginx 1.17.9
    MariaDB 10.3.22
    Just one quick question.
    When copy/paste to cli, is this three commands will execute OK? It is same command, but last two are without new rows and spaces (very last is without every space).

    Code:
    curl -XPUT 'http://localhost:9200/_template/default' -d'{
      "template": "*",
      "settings": {
        "number_of_shards": "1",
        "number_of_replicas": "0"
      }
    }'
    Code:
    curl -XPUT 'http://localhost:9200/_template/default' -d'{  "template": "*",  "settings": {    "number_of_shards": "1",    "number_of_replicas": "0"  }}'
    Code:
    curl -XPUT 'http://localhost:9200/_template/default' -d'{"template": "*","settings": {"number_of_shards": "1","number_of_replicas": "0"}}'
     
  11. Sunka

    Sunka Well-Known Member

    1,140
    312
    83
    Oct 31, 2015
    Pula, Croatia
    Ratings:
    +508
    Local Time:
    1:08 PM
    Nginx 1.17.9
    MariaDB 10.3.22
    This is tutorial for installing (not upgrading) elasticsearch 5 on centos.
    If I am missing something, please post here.
    Without help from @eva2000 and @Xon this tutorial could not be alive.

    I tried to upgrade first from 2.4 version, but ended up with strange errors, so I delete (yum remove elasticsearch) and install fresh new one.
    Because of something, if you remove old one and than install new one elasticsearch, change cluster name to new one, do not use old one (you have to change it also in ACP)

    Code:
    rpm --import https://artifacts.elastic.co/GPG-KEY-elasticsearch
    
    Code:
    nano /etc/yum.repos.d/elasticsearch.repo
    
    insert this:
    Code:
    [elasticsearch-5.x]
    name=Elasticsearch repository for 5.x packages
    baseurl=https://artifacts.elastic.co/packages/5.x/yum
    gpgcheck=1
    gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch
    enabled=1
    autorefresh=1
    type=rpm-md
    
    Code:
    yum -y install java-1.8.0-openjdk
    
    Code:
    yum -y install elasticsearch
    
    Code:
    curl -XPUT 'http://localhost:9200/_template/default' -d'{
      "template": "*",
      "settings": {
        "number_of_shards": "1",
        "number_of_replicas": "0"
      }
    }'
    
    Code:
    nano /etc/elasticsearch/elasticsearch.yml
    
    at the end of file add this:
    Code:
    cluster.name: CUSTOM NAME OF YOUR CLUSTER (same name should be inserted into your XenForo field in Admin ontrol Panel)
    network.host: 127.0.0.1
    http.port: 9200
    
    set how much ram you will add for elasticsearch - 512 megabytes is set in example (botx Xms and Xmx must have same value)
    Code:
    nano /etc/elasticsearch/jvm.options
    
    Code:
    -Xms512m
    -Xmx512m
    
    Code:
    mkdir /etc/elasticsearch/scripts/
    
    Code:
    cp -a /path/to/xenforoinstall/library/XenES/_scripts/*.groovy /etc/elasticsearch/scripts/
    
    Code:
    service elasticsearch start
    
    Code:
    systemctl daemon-reload
    
    Code:
    systemctl enable elasticsearch.service
    
    Code:
    systemctl start elasticsearch.service
    
    test if everything OK (give 1-2 minute after restarting elasticsearch)
    Code:
    curl -XGET '127.0.0.1:9200'
    
     
    Last edited by a moderator: Mar 20, 2017
  12. Sunka

    Sunka Well-Known Member

    1,140
    312
    83
    Oct 31, 2015
    Pula, Croatia
    Ratings:
    +508
    Local Time:
    1:08 PM
    Nginx 1.17.9
    MariaDB 10.3.22
    Elasticsearch 5.0.1 is out there

    Code:
    [root@upcloud ~]# yum update
    Loaded plugins: fastestmirror, priorities
    base                                                     | 3.6 kB     00:00     
    centos-sclo-rh                                           | 2.9 kB     00:00     
    centos-sclo-sclo                                         | 2.9 kB     00:00     
    elasticsearch-5.x                                        | 1.3 kB     00:00     
    epel/x86_64/metalink                                     |  20 kB     00:00     
    epel                                                     | 4.3 kB     00:00     
    extras                                                   | 3.4 kB     00:00     
    mariadb                                                  | 2.9 kB     00:00     
    rpmforge                                                 | 1.9 kB     00:00     
    updates                                                  | 3.4 kB     00:00     
    varnish-4.1                                              |  951 B     00:00     
    elasticsearch-5.x/primary                                  | 9.2 kB   00:01     
    Loading mirror speeds from cached hostfile
     * base: ftp.hosteurope.de
     * epel: mirror.de.leaseweb.net
     * extras: ftp.hosteurope.de
     * rpmforge: mirror.de.leaseweb.net
     * updates: ftp.hosteurope.de
    elasticsearch-5.x                                                         20/20
    233 packages excluded due to repository priority protections
    Resolving Dependencies
    --> Running transaction check
    ---> Package elasticsearch.noarch 0:5.0.0-1 will be updated
    ---> Package elasticsearch.noarch 0:5.0.1-1 will be an update
    --> Finished Dependency Resolution
    
    Dependencies Resolved
    
    ================================================================================
     Package             Arch         Version         Repository               Size
    ================================================================================
    Updating:
     elasticsearch       noarch       5.0.1-1         elasticsearch-5.x        31 M
    
    Transaction Summary
    ================================================================================
    Upgrade  1 Package
    
    Total download size: 31 M
    Is this ok [y/d/N]: y
    Downloading packages:
    No Presto metadata available for elasticsearch-5.x
    elasticsearch-5.0.1.rpm                                    |  31 MB   00:02     
    Running transaction check
    Running transaction test
    Transaction test succeeded
    Running transaction
      Updating   : elasticsearch-5.0.1-1.noarch                                 1/2
    warning: /etc/elasticsearch/elasticsearch.yml created as /etc/elasticsearch/elasticsearch.yml.rpmnew
      Cleanup    : elasticsearch-5.0.0-1.noarch                                 2/2
      Verifying  : elasticsearch-5.0.1-1.noarch                                 1/2
      Verifying  : elasticsearch-5.0.0-1.noarch                                 2/2
    
    Updated:
      elasticsearch.noarch 0:5.0.1-1                                               
    
    Complete!
    [root@upcloud ~]# systemctl daemon-reload
    [root@upcloud ~]# service elasticsearch restart
    Restarting elasticsearch (via systemctl):                  [  OK  ]
    [root@upcloud ~]# curl -XGET '127.0.0.1:9200'
    {
      "name" : "-QxWHO2",
      "cluster_name" : "xxx",
      "cluster_uuid" : "xxxxxxxx",
      "version" : {
        "number" : "5.0.1",
        "build_hash" : "080bb47",
        "build_date" : "2016-11-11T22:08:49.812Z",
        "build_snapshot" : false,
        "lucene_version" : "6.2.1"
      },
      "tagline" : "You Know, for Search"
    }
     
  13. Sunka

    Sunka Well-Known Member

    1,140
    312
    83
    Oct 31, 2015
    Pula, Croatia
    Ratings:
    +508
    Local Time:
    1:08 PM
    Nginx 1.17.9
    MariaDB 10.3.22
    Elasticsearch 5.0.2 is out there
    Release notes

    Code:
    [root@upcloud /]# yum update
    Loaded plugins: fastestmirror, priorities
    base                                                                      | 3.6 kB  00:00:00     
    centos-sclo-rh                                                            | 2.9 kB  00:00:00     
    centos-sclo-sclo                                                          | 2.9 kB  00:00:00     
    elasticsearch-5.x                                                         | 1.3 kB  00:00:00     
    epel/x86_64/metalink                                                      |  24 kB  00:00:00     
    epel                                                                      | 4.3 kB  00:00:00     
    extras                                                                    | 3.4 kB  00:00:00     
    mariadb                                                                   | 2.9 kB  00:00:00     
    rpmforge                                                                  | 1.9 kB  00:00:00     
    updates                                                                   | 3.4 kB  00:00:00     
    varnish-4.1                                                               |  951 B  00:00:00     
    (1/11): base/7/x86_64/group_gz                                            | 155 kB  00:00:00     
    (2/11): epel/x86_64/group_gz                                              | 170 kB  00:00:00     
    (3/11): epel/x86_64/updateinfo                                            | 677 kB  00:00:00     
    (4/11): epel/x86_64/primary_db                                            | 4.4 MB  00:00:00     
    (5/11): centos-sclo-sclo/x86_64/primary_db                                | 117 kB  00:00:00     
    (6/11): extras/7/x86_64/primary_db                                        | 166 kB  00:00:00     
    (7/11): updates/7/x86_64/primary_db                                       | 9.1 MB  00:00:00     
    (8/11): centos-sclo-rh/x86_64/primary_db                                  | 1.8 MB  00:00:00     
    (9/11): mariadb/primary_db                                                |  18 kB  00:00:00     
    (10/11): elasticsearch-5.x/primary                                        |  13 kB  00:00:00     
    (11/11): base/7/x86_64/primary_db                                         | 5.3 MB  00:00:00     
    (1/2): rpmforge/primary_db                                                | 125 kB  00:00:00     
    (2/2): varnish-4.1/x86_64/primary                                         | 9.0 kB  00:00:00     
    Determining fastest mirrors
     * base: ftp.hosteurope.de
     * epel: mirror.de.leaseweb.net
     * extras: ftp.hosteurope.de
     * rpmforge: mirror.de.leaseweb.net
     * updates: ftp.hosteurope.de
    elasticsearch-5.x                                                                          30/30
    varnish-4.1                                                                                29/29
    230 packages excluded due to repository priority protections
    Resolving Dependencies
    --> Running transaction check
    ---> Package elasticsearch.noarch 0:5.0.1-1 will be updated
    ---> Package elasticsearch.noarch 0:5.0.2-1 will be an update
    --> Finished Dependency Resolution
    
    Dependencies Resolved
    
    =================================================================================================
     Package                 Arch             Version              Repository                   Size
    =================================================================================================
    Updating:
     elasticsearch           noarch           5.0.2-1              elasticsearch-5.x            31 M
    
    Transaction Summary
    =================================================================================================
    Upgrade  1 Package
    
    Total download size: 31 M
    Is this ok [y/d/N]: y
    Downloading packages:
    No Presto metadata available for elasticsearch-5.x
    elasticsearch-5.0.2.rpm                                                   |  31 MB  00:00:02     
    Running transaction check
    Running transaction test
    Transaction test succeeded
    Running transaction
      Updating   : elasticsearch-5.0.2-1.noarch                                                  1/2
    warning: /etc/elasticsearch/jvm.options created as /etc/elasticsearch/jvm.options.rpmnew
      Cleanup    : elasticsearch-5.0.1-1.noarch                                                  2/2
      Verifying  : elasticsearch-5.0.2-1.noarch                                                  1/2
      Verifying  : elasticsearch-5.0.1-1.noarch                                                  2/2
    
    Updated:
      elasticsearch.noarch 0:5.0.2-1                                                                 
    
    Complete!
    
    [root@upcloud /]# systemctl daemon-reload
    
    [root@upcloud /]# service elasticsearch restart
    Restarting elasticsearch (via systemctl):                  [  OK  ]
    
    [root@upcloud /]# curl -XGET '127.0.0.1:9200'
    {
      "name" : "-QxWHO2",
      "cluster_name" : "xxx",
      "cluster_uuid" : "xxxxxxx",
      "version" : {
        "number" : "5.0.2",
        "build_hash" : "f6b4951",
        "build_date" : "2016-11-24T10:07:18.101Z",
        "build_snapshot" : false,
        "lucene_version" : "6.2.1"
      },
      "tagline" : "You Know, for Search"
    }
     
  14. eva2000

    eva2000 Administrator Staff Member

    44,750
    10,204
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +15,815
    Local Time:
    9:08 PM
    Nginx 1.17.x
    MariaDB 5.5/10.x
    living on the edge :)
     
  15. Sunka

    Sunka Well-Known Member

    1,140
    312
    83
    Oct 31, 2015
    Pula, Croatia
    Ratings:
    +508
    Local Time:
    1:08 PM
    Nginx 1.17.9
    MariaDB 10.3.22
    Yep :)
    If something get bad, it is 10 minute job to reinstall elasticsearch.
     
  16. Xon

    Xon Active Member

    170
    61
    28
    Nov 16, 2015
    Ratings:
    +223
    Local Time:
    7:08 PM
    1.15.x
    MariaDB 10.3.x
    Then a 12 hour job to re-index :p
     
  17. rdan

    rdan Well-Known Member

    4,962
    1,183
    113
    May 25, 2014
    Ratings:
    +1,797
    Local Time:
    7:08 PM
    Mainline
    10.2
    Is this reasonable size for index?
     
  18. Sunka

    Sunka Well-Known Member

    1,140
    312
    83
    Oct 31, 2015
    Pula, Croatia
    Ratings:
    +508
    Local Time:
    1:08 PM
    Nginx 1.17.9
    MariaDB 10.3.22
    Mine...

    Code:
    Documents:   322,709 (189.3 MB)
     
  19. rdan

    rdan Well-Known Member

    4,962
    1,183
    113
    May 25, 2014
    Ratings:
    +1,797
    Local Time:
    7:08 PM
    Mainline
    10.2
    Mine got an average of 0.27kb per document.
    Yours got an average of 0.58kb per document.
     
  20. Sunka

    Sunka Well-Known Member

    1,140
    312
    83
    Oct 31, 2015
    Pula, Croatia
    Ratings:
    +508
    Local Time:
    1:08 PM
    Nginx 1.17.9
    MariaDB 10.3.22
    Yep, so mine is worse than yours :penguin:
    What would be OK?