/ Register

Want to subscribe to topics you're interested in?
Become a Member

Featured Install Compiling Newer Versions of Clang 4.0

Discussion in 'Centmin Mod User Tutorials & Guides' started by Revenge, Jan 31, 2017.

Tags:
Previous Thread Next Thread
Loading...
Page 1 of 3
  1. Jan 31, 2017 #1
    Revenge

    Revenge Active Member

    469
    93
    28
    Feb 21, 2016
    Portugal
    Ratings:
    +354
    Local Time:
    2:45 AM
    1.9.x
    10.1.x
    How many of you loves to be using the last version of everything? Living in the edge of tecnology? Well, im one of those persons.

    So, today i was compiling nginx using clang 3.4.2 and i thought... Why not use the last version of it instead of using a version that already have 2 and a half years?

    I will explain here how to do it in a Centos 7 build and its really very easy. I will be installing clang 4.0.0 that is the last release in their svn.

    First, to compile Clang 4.0, you need Cmake 3. The version that comes with Centos is 2.8.12, so it won't work.
    You just need to install it using the epel repo:
    Code:
    yum install cmake3
    This will install cmake3 but not overwrite the cmake that comes by default with the OS, which is a good thing.

    We will use svn to download clang, so if you haven't it already installed:
    Code:
    yum install svn
    Now we will download clang source code and put everything in the correct folder. I will use a /src folder. If you want to install clang 3.9.1 instead of 4.0.0, just change to release_39, and so on.
    Code:
    mkdir src
cd /src
svn co http://llvm.org/svn/llvm-project/llvm/branches/release_40/ llvm
cd llvm/tools
svn co http://llvm.org/svn/llvm-project/cfe/branches/release_40/ clang
cd clang/tools
svn co http://llvm.org/svn/llvm-project/clang-tools-extra/branches/release_40/ extra
cd ../../../projects
svn co http://llvm.org/svn/llvm-project/compiler-rt/branches/release_40/ compiler-rt
cd ../..
    Now lets create the folder where we will compile it. Attention that this folder will have 2.2Gb at the end of the compile. So make sure you have the space.
    Code:
    mkdir llvm.build
cd llvm.build
    In the next step we will build clang. I use make -j4 to use the 4 cores of my vps to be faster, but you can adjust it to your needs. I will also install it in folder /opt/sbin/llvm, this way i will have clang 3.4.2 and this new clang 4.0.0. You can change the folder to only /usr and it will overwrite clang 3.4.2 and will work flawless with centminmod.

    Code:
    cmake3 -G "Unix Makefiles" -DCMAKE_BUILD_TYPE=Release -DCMAKE_INSTALL_PREFIX=/opt/sbin/llvm ../llvm
make -j4
make install
    Now we just compile Nginx with the new clang. In my case i start the configure with CC=/opt/sbin/llvm/bin/clang, if you overwrite the 3.4.2, its just CC=clang.

    And here it is, nginx compiled with the new clang working flawless:
    Code:
    nginx version: nginx/1.11.9
built by clang 4.0.0 (branches/release_40 293485)
built with OpenSSL 1.1.0d  26 Jan 2017
TLS SNI support enabled
     
  2. Jan 31, 2017 #2
    eva2000

    eva2000 Administrator Staff Member

    53,614
    12,139
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +18,683
    Local Time:
    11:45 AM
    Nginx 1.27.x
    MariaDB 10.x/11.4+
    very nice. I have thought about updating clang but the whole point of using clang is faster overall centmin mod initial install time as well. But if you had to compile clang, that extra time overweighs the benefits gained from using clang in the first place. Hence, why no clang update as yet :)
     
  3. Jan 31, 2017 #3
    Revenge

    Revenge Active Member

    469
    93
    28
    Feb 21, 2016
    Portugal
    Ratings:
    +354
    Local Time:
    2:45 AM
    1.9.x
    10.1.x
    Yes, thats true. The only option for centminmod would be to update clang as a option and not at first install. People that like to have the very last version of everything, could update it. In my vps, it took about 2 hours to fully compile it.
     
  4. Jan 31, 2017 #4
    eva2000

    eva2000 Administrator Staff Member

    53,614
    12,139
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +18,683
    Local Time:
    11:45 AM
    Nginx 1.27.x
    MariaDB 10.x/11.4+
    Yup as an optional upgrade addon would be possible. IIRC, clang builds would need CentOS 7 as Clang still needs a GCC minimum version requirement and CentOS 6 doesn't have that out of the box. Though devtoolset-4's GCC 5.3.1 should work for CentOS 6 ? Haven't tried Clang compile in ages now so don't recall :)
     
  5. Jan 31, 2017 #5
    Revenge

    Revenge Active Member

    469
    93
    28
    Feb 21, 2016
    Portugal
    Ratings:
    +354
    Local Time:
    2:45 AM
    1.9.x
    10.1.x
    I never used clang in Centos 6, but its not possible to install clang 3.4.2 on it? With that version of clang you can compile newer versions.
     
  6. Jan 31, 2017 #6
    eva2000

    eva2000 Administrator Staff Member

    53,614
    12,139
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +18,683
    Local Time:
    11:45 AM
    Nginx 1.27.x
    MariaDB 10.x/11.4+
    Yeah CentOS 6 uses clang 3.4.2, but IIRC clang compile still needs GCC in some form which is what stopped clang 3.4.2+ rpm build development for CentOS as the GCC requirements kept creeping up beyond what CentOS 6 GCC 4.4 could provide. It's also why Fedora is able to offer newer clang versions as Fedora's default GCC version is much higher.
     
  7. Jan 31, 2017 #7
    Revenge

    Revenge Active Member

    469
    93
    28
    Feb 21, 2016
    Portugal
    Ratings:
    +354
    Local Time:
    2:45 AM
    1.9.x
    10.1.x
    Never tried it on centos 6, so i didn't know.

    Just another suggestion, linking is a bit ram memory intensive for low vps's. We can change the default link from ld.bfd to ld.gold that is faster and uses less ram.

    We just need to run this:
    Code:
    alternatives --config ld
    It will show:
    Code:
    There is 2 program that provides 'ld'.

  Selection    Command
-----------------------------------------------
*+ 1           /usr/bin/ld.bfd
   2           /usr/bin/ld.gold

Enter to keep the current selection[+], or type selection number: 2
    We choose 2 and its done.
     
  8. Jan 31, 2017 #8
    eva2000

    eva2000 Administrator Staff Member

    53,614
    12,139
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +18,683
    Local Time:
    11:45 AM
    Nginx 1.27.x
    MariaDB 10.x/11.4+
    thanks will try compile on Core i7 4790K @4.0Ghz :D

    hmm CentOS 7 ld.gold ain't multi-threaded Andrew's Tech Blog: Multithreaded linking with ld.gold, currently minimal benefit
    so devtoolset-4 ld.gold might be the one needed

    edit: guess not
    Code (Text):
    /opt/rh/devtoolset-4/root/usr/bin/ld.gold --threads
/opt/rh/devtoolset-4/root/usr/bin/ld.gold: warning: ignoring --threads: /opt/rh/devtoolset-4/root/usr/bin/ld.gold was compiled without thread support
/opt/rh/devtoolset-4/root/usr/bin/ld.gold: fatal error: no input files
     
    Last edited: Jan 31, 2017
  9. Jan 31, 2017 #9
    eva2000

    eva2000 Administrator Staff Member

    53,614
    12,139
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +18,683
    Local Time:
    11:45 AM
    Nginx 1.27.x
    MariaDB 10.x/11.4+
    first attempt on Intel Core i7 4790K 4.0Ghz with 32GB Ram 2x 240GB SSD server took ~1,338 seconds = 22.3 minutes using raw clock speed :)
    Code (Text):
    /opt/sbin/llvm/bin/clang --version
clang version 4.0.0 (branches/release_40 293572)
Target: x86_64-unknown-linux-gnu
Thread model: posix
InstalledDir: /opt/sbin/llvm/bin

    Code (Text):
    ./llvm.sh

    Code (Text):
    ls -lahrt /root/centminlogs/ |grep llvm | tail -1
-rw-r--r--  1 root root 2.8M Jan 31 00:23 centminmod_llvm_310117-000105.log

    Code (Text):
    tail -1 /root/centminlogs/centminmod_llvm_310117-000105.log
Total LLVM 4 Build Time: 1338.368736902 seconds

    Code (Text):
    ls -lah /opt/sbin/llvm/bin/
total 738M
drwxr-xr-x 2 root root 4.0K Jan 31 00:23 .
drwxr-xr-x 7 root root 4.0K Jan 31 00:23 ..
-rwxr-xr-x 1 root root  16M Jan 31 00:18 bugpoint
-rwxr-xr-x 1 root root  18M Jan 31 00:23 c-index-test
lrwxrwxrwx 1 root root    9 Jan 31 00:23 clang -> clang-4.0
lrwxrwxrwx 1 root root    5 Jan 31 00:23 clang++ -> clang
-rwxr-xr-x 1 root root  74M Jan 31 00:22 clang-4.0
-rwxr-xr-x 1 root root 1.9M Jan 31 00:22 clang-apply-replacements
-rwxr-xr-x 1 root root  19M Jan 31 00:22 clang-change-namespace
-rwxr-xr-x 1 root root  54M Jan 31 00:22 clang-check
lrwxrwxrwx 1 root root    5 Jan 31 00:23 clang-cl -> clang
lrwxrwxrwx 1 root root    5 Jan 31 00:23 clang-cpp -> clang
-rwxr-xr-x 1 root root 2.1M Jan 31 00:12 clang-format
-rwxr-xr-x 1 root root  22M Jan 31 00:22 clang-import-test
-rwxr-xr-x 1 root root  18M Jan 31 00:18 clang-include-fixer
-rwxr-xr-x 1 root root 2.7M Jan 31 00:11 clang-offload-bundler
-rwxr-xr-x 1 root root  20M Jan 31 00:22 clang-query
-rwxr-xr-x 1 root root  18M Jan 31 00:22 clang-rename
-rwxr-xr-x 1 root root  18M Jan 31 00:22 clang-reorder-fields
-rwxr-xr-x 1 root root  26M Jan 31 00:22 clang-tidy
-rwxr-xr-x 1 root root  18M Jan 31 00:18 find-all-symbols
-rwxr-xr-x 1 root root  21K Jan 31 00:01 git-clang-format
-rwxr-xr-x 1 root root  38M Jan 31 00:22 llc
-rwxr-xr-x 1 root root  20M Jan 31 00:23 lli
-rwxr-xr-x 1 root root  13M Jan 31 00:22 llvm-ar
-rwxr-xr-x 1 root root 2.2M Jan 31 00:18 llvm-as
-rwxr-xr-x 1 root root 357K Jan 31 00:18 llvm-bcanalyzer
-rwxr-xr-x 1 root root 2.1M Jan 31 00:18 llvm-cat
-rwxr-xr-x 1 root root 139K Jan 31 00:03 llvm-config
-rwxr-xr-x 1 root root 2.9M Jan 31 00:18 llvm-cov
-rwxr-xr-x 1 root root  31M Jan 31 00:23 llvm-c-test
-rwxr-xr-x 1 root root 2.5M Jan 31 00:22 llvm-cxxdump
-rwxr-xr-x 1 root root 122K Jan 31 00:03 llvm-cxxfilt
-rwxr-xr-x 1 root root 2.0M Jan 31 00:18 llvm-diff
-rwxr-xr-x 1 root root 1.7M Jan 31 00:18 llvm-dis
-rwxr-xr-x 1 root root  31M Jan 31 00:22 llvm-dsymutil
-rwxr-xr-x 1 root root 2.7M Jan 31 00:18 llvm-dwarfdump
-rwxr-xr-x 1 root root  30M Jan 31 00:23 llvm-dwp
-rwxr-xr-x 1 root root 3.0M Jan 31 00:18 llvm-extract
lrwxrwxrwx 1 root root    7 Jan 31 00:23 llvm-lib -> llvm-ar
-rwxr-xr-x 1 root root 2.9M Jan 31 00:18 llvm-link
-rwxr-xr-x 1 root root  38M Jan 31 00:22 llvm-lto
-rwxr-xr-x 1 root root  39M Jan 31 00:23 llvm-lto2
-rwxr-xr-x 1 root root  15M Jan 31 00:22 llvm-mc
-rwxr-xr-x 1 root root 294K Jan 31 00:03 llvm-mcmarkup
-rwxr-xr-x 1 root root 1.8M Jan 31 00:18 llvm-modextract
-rwxr-xr-x 1 root root  14M Jan 31 00:22 llvm-nm
-rwxr-xr-x 1 root root  14M Jan 31 00:22 llvm-objdump
-rwxr-xr-x 1 root root 378K Jan 31 00:18 llvm-opt-report
-rwxr-xr-x 1 root root 1.4M Jan 31 00:18 llvm-pdbdump
-rwxr-xr-x 1 root root 1.1M Jan 31 00:07 llvm-profdata
lrwxrwxrwx 1 root root    7 Jan 31 00:23 llvm-ranlib -> llvm-ar
-rwxr-xr-x 1 root root 3.6M Jan 31 00:18 llvm-readobj
-rwxr-xr-x 1 root root  12M Jan 31 00:23 llvm-rtdyld
-rwxr-xr-x 1 root root 2.5M Jan 31 00:18 llvm-size
-rwxr-xr-x 1 root root 2.4M Jan 31 00:18 llvm-split
-rwxr-xr-x 1 root root 3.0M Jan 31 00:18 llvm-stress
-rwxr-xr-x 1 root root 282K Jan 31 00:18 llvm-strings
-rwxr-xr-x 1 root root 2.8M Jan 31 00:18 llvm-symbolizer
-rwxr-xr-x 1 root root 2.1M Jan 31 00:03 llvm-tblgen
-rwxr-xr-x 1 root root 2.9M Jan 31 00:23 llvm-xray
-rwxr-xr-x 1 root root  18M Jan 31 00:17 modularize
-rwxr-xr-x 1 root root 3.0M Jan 31 00:18 obj2yaml
-rwxr-xr-x 1 root root  44M Jan 31 00:23 opt
-rwxr-xr-x 1 root root  13M Jan 31 00:22 sancov
-rwxr-xr-x 1 root root 2.6M Jan 31 00:18 sanstats
-rwxr-xr-x 1 root root  53K Jan 31 00:01 scan-build
-rwxr-xr-x 1 root root 4.4K Jan 31 00:01 scan-view
-rwxr-xr-x 1 root root 2.4M Jan 31 00:18 verify-uselistorder
-rwxr-xr-x 1 root root 769K Jan 31 00:18 yaml2obj


    Clang 4.0 without ccache as first time compile seems to compile Nginx ~10-15 seconds faster than GCC at ~75 seconds. So is it worth the 1,338+ seconds compile time ? :)

    Clang 3.4.2 with ccache'd time from prior builds took ~44 seconds

    rebuild with clang 4.0 with ccache this time too 43.5 seconds so 0.5 seconds faster than clang 3.4.2 ccache'd time !

    native CentOS 7.3 GCC 4.8.5 compile with ccache took ~66 seconds
     
    Last edited: Jan 31, 2017
  10. Jan 31, 2017 #10
    Revenge

    Revenge Active Member

    469
    93
    28
    Feb 21, 2016
    Portugal
    Ratings:
    +354
    Local Time:
    2:45 AM
    1.9.x
    10.1.x
    But if with ld.gold with only one thread is 1-2% faster than ld.bfd with 4 threads when linking, thats very good because with 4 threads, linking will simple use 4x more memory. For low vps's that can be a problem.

    22 minutes on your server, thats very good. I compiled mine again with ld.gold instead and i decreased the time from 2 hours to 1 hour and half.
     
  11. Jan 31, 2017 #11
    eva2000

    eva2000 Administrator Staff Member

    53,614
    12,139
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +18,683
    Local Time:
    11:45 AM
    Nginx 1.27.x
    MariaDB 10.x/11.4+
    sweet will try with ld.gold next :)

    Interesting ld.gold for clang 4.0 ccache'd based Nginx compile took ~32.7 seconds !
     
  12. Jan 31, 2017 #12
    Revenge

    Revenge Active Member

    469
    93
    28
    Feb 21, 2016
    Portugal
    Ratings:
    +354
    Local Time:
    2:45 AM
    1.9.x
    10.1.x
    So that means ld.gold also decreases nginx compile times. Very good. It seems i will stick with ld.gold.
     
  13. Jan 31, 2017 #13
    eva2000

    eva2000 Administrator Staff Member

    53,614
    12,139
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +18,683
    Local Time:
    11:45 AM
    Nginx 1.27.x
    MariaDB 10.x/11.4+
  14. Jan 31, 2017 #14
    eva2000

    eva2000 Administrator Staff Member

    53,614
    12,139
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +18,683
    Local Time:
    11:45 AM
    Nginx 1.27.x
    MariaDB 10.x/11.4+
    @Revenge with ld.gold for clang 4.0 compile only slightly faster ~1,313 seconds so ~25 seconds faster :)
    Code (Text):
    tail -1 /root/centminlogs/centminmod_llvm_310117-005629.log
Total LLVM 4 Build Time: 1313.191681593 seconds

    So whatever way you swing it, Clang 4 with/without ld.gold means 10-15 seconds faster for Nginx compiles on i7 4790K server at the expense of an extra 1,300+ second compile time !
     
  15. Jan 31, 2017 #15
    Revenge

    Revenge Active Member

    469
    93
    28
    Feb 21, 2016
    Portugal
    Ratings:
    +354
    Local Time:
    2:45 AM
    1.9.x
    10.1.x
    Can you test it on a slower server to see if the diference is bigger?
    Well, its a one time compile time and it will make every nginx compile faster after that. You just need to compile nginx 85 times to start to compensate :LOL:
     
  16. Jan 31, 2017 #16
    eva2000

    eva2000 Administrator Staff Member

    53,614
    12,139
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +18,683
    Local Time:
    11:45 AM
    Nginx 1.27.x
    MariaDB 10.x/11.4+
    that's as much free time i have to dig into this for now. So will have to play with this later :)
     
  17. Jan 31, 2017 #17
    eva2000

    eva2000 Administrator Staff Member

    53,614
    12,139
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +18,683
    Local Time:
    11:45 AM
    Nginx 1.27.x
    MariaDB 10.x/11.4+
  18. Jan 31, 2017 #18
    eva2000

    eva2000 Administrator Staff Member

    53,614
    12,139
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +18,683
    Local Time:
    11:45 AM
    Nginx 1.27.x
    MariaDB 10.x/11.4+
    More interesting reading https://blogs.s-osg.org/an-introduction-to-accelerating-your-build-with-clang/ :)

    Which tracks and supports Centmin Mod Nginx ccache compile times of ~66 seconds for GCC 4.8.5 vs ~44 seconds for Clang 3.4.2 and 4.0. Though the resulting Nginx binary performance, GCC is better than Clang especially with Intel optimised GCC flags in play.

    Also confirms my results above, ld.gold linker doesn't improve release builds much at all compared to debug builds.

    gsplit-dwarf flag looks interesting too, GCC 4.7+ support it - Bug 10005 – Doesn't support -gsplit-dwarf

    as Centmin Mod uses ccache compiler caching too so not wonder if ccache now supports such ? :)
    as ccache 3.2.3+ support it ccache news
    fyi, centmin mod 123.08stable uses ccache 3.2.2 while 123.09beta01 uses ccache 3.3.2, so only 123.09beta01+ and higher would have ccache supporting gsplit-dwarf
     
    Last edited: Jan 31, 2017
  19. Jan 31, 2017 #19
    eva2000

    eva2000 Administrator Staff Member

    53,614
    12,139
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +18,683
    Local Time:
    11:45 AM
    Nginx 1.27.x
    MariaDB 10.x/11.4+
  20. Feb 1, 2017 #20
    eva2000

    eva2000 Administrator Staff Member

    53,614
    12,139
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +18,683
    Local Time:
    11:45 AM
    Nginx 1.27.x
    MariaDB 10.x/11.4+
    @Revenge added Nginx support for using ld.gold linker for clang 3.4.2 and GCC via NGX_LDGOLD='y' variable which is disabled by default.

    Example for persistent /etc/centminmod/custom_config.inc config file options to enable GCC 5.3.1 + gsplit-dwarf + ld.gold linker
    Code (Text):
    NGX_LDGOLD='y'
NGX_GSPLITDWARF='y'
PHP_GSPLITDWARF='y'
CLANG='n'
NGINX_DEVTOOLSETGCC='y'

    resulting GCC 5.3.1 ccache'd upgrade time for i7 4790K
    Code (Text):
    Total Nginx Upgrade Time: 31.384081342 seconds

    For clang 3.4.2 + gsplit-dwarf + ld.gold
    Code (Text):
    NGX_LDGOLD='y'
NGX_GSPLITDWARF='y'
PHP_GSPLITDWARF='y'

    resulting clang 3.4.2 ccache'd upgrade time for i7 4790K = only 0.19s slower than GCC 5.3.1 for ccache'd results with ld.gold linker and gsplit-dwarf
    Code (Text):
    Total Nginx Upgrade Time: 31.575658021 seconds


    With Clang 4.0 + ld.gold auto disabled + gsplit-dwarf ccache'd
    ~0.5s faster than GCC 5.3.1 ccache'd above and ~~0.31s faster than Clang 3.4.2 ccache'd above.
    Code (Text):
    Total Nginx Upgrade Time: 31.072155199 seconds
     
    Last edited: Feb 1, 2017
Previous Thread Next Thread
Loading...
Page 1 of 3

.