Want to subscribe to topics you're interested in?
Become a Member

Xenforo Elasticsearch Install for Xenforo

Discussion in 'Forum software usage' started by Sunka, Nov 27, 2015.

  1. rdan

    rdan Well-Known Member

    4,962
    1,183
    113
    May 25, 2014
    Ratings:
    +1,797
    Local Time:
    5:57 PM
    Mainline
    10.2
    Haha no. Maybe most of your post has a lot of data/characters.
    Not sure :D
     
  2. eva2000

    eva2000 Administrator Staff Member

    44,750
    10,204
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +15,814
    Local Time:
    7:57 PM
    Nginx 1.17.x
    MariaDB 5.5/10.x
    well mine
     
  3. rdan

    rdan Well-Known Member

    4,962
    1,183
    113
    May 25, 2014
    Ratings:
    +1,797
    Local Time:
    5:57 PM
    Mainline
    10.2
    Average of 0.59kb per document :D.
     
  4. Sunka

    Sunka Well-Known Member

    1,140
    312
    83
    Oct 31, 2015
    Pula, Croatia
    Ratings:
    +508
    Local Time:
    11:57 AM
    Nginx 1.17.9
    MariaDB 10.3.22
    Lot of them are "big", so yep :)
     
  5. Sunka

    Sunka Well-Known Member

    1,140
    312
    83
    Oct 31, 2015
    Pula, Croatia
    Ratings:
    +508
    Local Time:
    11:57 AM
    Nginx 1.17.9
    MariaDB 10.3.22
    Java consume more ram that I set for elasticsearch.
    Elasticsearch stop every day 1-2 times.

    I think I will roll back to ES 2.x

    Remove elasticsearch with yum and then install ES 2.x would be OK, but what with java?
    I have java v8.0 installed.
    Should I not touch anythng related to java or to downgrade to v7?
     
  6. rdan

    rdan Well-Known Member

    4,962
    1,183
    113
    May 25, 2014
    Ratings:
    +1,797
    Local Time:
    5:57 PM
    Mainline
    10.2
    I'm using ES 5.x for a month now without any issue.
     
  7. Sunka

    Sunka Well-Known Member

    1,140
    312
    83
    Oct 31, 2015
    Pula, Croatia
    Ratings:
    +508
    Local Time:
    11:57 AM
    Nginx 1.17.9
    MariaDB 10.3.22
    I set 768MB ram to elasticsearch, but still Java consuming above 1GB

    Capture.PNG
     
  8. eva2000

    eva2000 Administrator Staff Member

    44,750
    10,204
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +15,814
    Local Time:
    7:57 PM
    Nginx 1.17.x
    MariaDB 5.5/10.x
    maybe

    For 2.4 Circuit Breaker | Elasticsearch Reference [2.4] | Elastic
     
    Last edited: Jan 11, 2017
  9. Sunka

    Sunka Well-Known Member

    1,140
    312
    83
    Oct 31, 2015
    Pula, Croatia
    Ratings:
    +508
    Local Time:
    11:57 AM
    Nginx 1.17.9
    MariaDB 10.3.22
    Yep, but from all my ram (4GB) I give elasticsearch no more than 512MB and it is crashed once or twice in 24 hours. I tried also with 768MB and 1024MB but result is same.

    ES 2.x works fine, and also ES 5.x till 2-3 weeks (I do not know from which 5.x version start that crashing).

    What is confusing me, is that Java (elasticsearch service) consume more memory than I order in settings.

    Maybe to delete clam (maldet) because then I will get 500 MB ram free to give to ES.

    I have 4GB RAM
    2gb ram is located for mariadb
    512 to ES
    64 to redis
    rest for all other when needed
     
  10. eva2000

    eva2000 Administrator Staff Member

    44,750
    10,204
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +15,814
    Local Time:
    7:57 PM
    Nginx 1.17.x
    MariaDB 5.5/10.x
    from these 2
    maybe set in /etc/elasticsearch/elasticsearch.yml

    Code (Text):
    # for ES 2.4
    # defaults 70%
    indices.breaker.total.limit: 60%
    # defaults 60%
    # this value has to be higher than indices.fielddata.cache.size
    indices.breaker.fielddata.limit: 50%
    # defaults 40%
    indices.breaker.request.limit: 35%
    # defaults unbound
    indices.fielddata.cache.size: 35%


    but
    I believe @Xon for his forum's Elasticsearch is leveraging Linode's fast disk i/o filesystem as an extended memory area when it comes to elasticsearch so allowing it to swap from memory to disk to keep memory requirements lower ? So basically letting his servers swap to disk.

    I am using ES heap size of 257MB in ES 2.4 and using around 295MB RSS memory
    Code (Text):
    ps aufxw | grep java | grep -v grep
    496       5004  1.9  3.6 3926944 297052 ?      Sl   17:14   0:30 /usr/bin/java -Xms257m -Xmx257m
    


    Getting fielddata per index per node memory usage and eviction stats
    Code (Text):
    curl -s 'localhost:9200/_nodes/stats/indices/fielddata?level=indices&fields=*'
    

    which for me fileddata memory usage is very small = 194,024 bytes
    Code (Text):
    "memory_size_in_bytes":194024,"evictions":0
    
     
    Last edited: Jan 11, 2017
  11. Sunka

    Sunka Well-Known Member

    1,140
    312
    83
    Oct 31, 2015
    Pula, Croatia
    Ratings:
    +508
    Local Time:
    11:57 AM
    Nginx 1.17.9
    MariaDB 10.3.22
    What is this?

    Also same in code tags

    Code:
    ps aufxw | grep java | grep -v grep
    elastic+ 29911  0.6 22.4 4473252 870132 ?      Ssl  15:45   1:35 /bin/java -Xms512m -Xmx512m -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+DisableExplicitGC -XX:+AlwaysPreTouch -server -Xss1m -Djava.awt.headless=true -Dfile.encoding=UTF-8 -Djna.nosys=true -Djdk.io.permissionsUseCanonicalPath=true -Dio.netty.noUnsafe=true -Dio.netty.noKeySetOptimization=true -Dlog4j.shutdownHookEnabled=false -Dlog4j2.disable.jmx=true -Dlog4j.skipJansi=true -XX:+HeapDumpOnOutOfMemoryError -Des.path.home=/usr/share/elasticsearch -cp /usr/share/elasticsearch/lib/elasticsearch-5.1.1.jar:/usr/share/elasticsearch/lib/* org.elasticsearch.bootstrap.Elasticsearch -p /var/run/elasticsearch/elasticsearch.pid --quiet -Edefault.path.logs=/var/log/elasticsearch -Edefault.path.data=/var/lib/elasticsearch -Edefault.path.conf=/etc/elasticsearch
     
  12. Sunka

    Sunka Well-Known Member

    1,140
    312
    83
    Oct 31, 2015
    Pula, Croatia
    Ratings:
    +508
    Local Time:
    11:57 AM
    Nginx 1.17.9
    MariaDB 10.3.22
    and

    Code:
    curl -s '127.0.0.1:9200/_nodes/stats/indices/fielddata?level=indices&fields=*'
    {"_nodes":{"total":1,"successful":1,"failed":0},"cluster_name":"elastictvor","nodes":{"-QxWHO2jTfWe2CKpqDO7-w":{"timestamp":1484075751931,"name":"-QxWHO2","transport_address":"127.0.0.1:9300","host":"127.0.0.1","ip":"127.0.0.1:9300","roles":["master","data","ingest"],"indices":{"fielddata":{"memory_size_in_bytes":0,"evictions":0,"fields":{"_parent":{"memory_size_in_bytes":0}}},"indices":{"elastictvor":{"fielddata":{"memory_size_in_bytes":0,"evictions":0,"fields":{"_parent":{"memory_size_in_bytes":0}}}}}}}}}
     
  13. eva2000

    eva2000 Administrator Staff Member

    44,750
    10,204
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +15,814
    Local Time:
    7:57 PM
    Nginx 1.17.x
    MariaDB 5.5/10.x
    ps aufx headers
    Code (Text):
    ps aufx
    USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
    

    so that is your VSZ and RSS memory sizes so RSS used memory is 870132 kb
    no fielddata memory being used so probably not related

    so only ~850MB of memory used when ES heap size set at 512MB (/bin/java -Xms512m -Xmx512m)
     
  14. Sunka

    Sunka Well-Known Member

    1,140
    312
    83
    Oct 31, 2015
    Pula, Croatia
    Ratings:
    +508
    Local Time:
    11:57 AM
    Nginx 1.17.9
    MariaDB 10.3.22
    Elasticsearch 5.2.0 is out there
    Release notes

    Code (Text):
    [root@upcloud ~]# yum history info 107
    Loaded plugins: fastestmirror, priorities
    Transaction ID : 107
    Begin time     : Tue Jan 31 17:53:51 2017
    Begin rpmdb    : 780:c4ecfaf4409ab378a30cffcbe386d12fb8d90763
    End time       :            17:53:52 2017 (1 seconds)
    End rpmdb      : 780:67c68ed802b58da59e5083429635ba9141488cf1
    User           : root <root>
    Return-Code    : Success
    Command Line   : update
    Transaction performed with:
        Installed     rpm-4.11.3-21.el7.x86_64                      @base
        Installed     yum-3.4.3-150.el7.centos.noarch               @base
        Installed     yum-metadata-parser-1.1.4-10.el7.x86_64       @anaconda
        Installed     yum-plugin-fastestmirror-1.1.31-40.el7.noarch @base
    Packages Altered:
        Updated elasticsearch-5.1.2-1.noarch @elasticsearch-5.x
        Update                5.2.0-1.noarch @elasticsearch-5.x
    history info
    
    [root@upcloud ~]# curl -XGET '127.0.0.1:9200'
    {
      "name" : "-QxWHO2",
      "cluster_name" : "xxxxx",
      "cluster_uuid" : "xxxxxxx",
      "version" : {
        "number" : "5.2.0",
        "build_hash" : "24e05b9",
        "build_date" : "2017-01-24T19:52:35.800Z",
        "build_snapshot" : false,
        "lucene_version" : "6.4.0"
      },
      "tagline" : "You Know, for Search"
    }
    
     
  15. BamaStangGuy

    BamaStangGuy Premium Member Premium Member

    650
    188
    43
    May 25, 2014
    Ratings:
    +263
    Local Time:
    4:57 AM
    Haven't seen the update come across on my end.
     
  16. Sunka

    Sunka Well-Known Member

    1,140
    312
    83
    Oct 31, 2015
    Pula, Croatia
    Ratings:
    +508
    Local Time:
    11:57 AM
    Nginx 1.17.9
    MariaDB 10.3.22
    try clean yum
     
  17. BamaStangGuy

    BamaStangGuy Premium Member Premium Member

    650
    188
    43
    May 25, 2014
    Ratings:
    +263
    Local Time:
    4:57 AM
    That worked.
     
  18. viralpoet

    viralpoet Member

    43
    3
    8
    Mar 5, 2017
    Ratings:
    +9
    Local Time:
    9:57 PM
    1.10.3
    10
    Just need one of your nice little step by step guides now ;)
     
  19. Sunka

    Sunka Well-Known Member

    1,140
    312
    83
    Oct 31, 2015
    Pula, Croatia
    Ratings:
    +508
    Local Time:
    11:57 AM
    Nginx 1.17.9
    MariaDB 10.3.22
    Installing or upgrading?
     
  20. viralpoet

    viralpoet Member

    43
    3
    8
    Mar 5, 2017
    Ratings:
    +9
    Local Time:
    9:57 PM
    1.10.3
    10
    On my centmin server? I'd like to install version 5. Not upgrade.