Sysadmin Random Sanity Project

Jimmy · Jun 4, 2017

The Random Sanity Project is a free, open source service that helps secure the Internet by sanity-checking sources of randomness. If you are a CTO or system administrator responsible for a security-critical web server or application running on the Internet you should consider using this service to alert you of catastrophic hardware or software failures that could completely compromise the security of your website or application.

http://www.randomsanity.org/

eva2000 · Jun 4, 2017

Moved to more appropriate forum. Looks interesting and more at about how it came about and relationship with bitcoin http://dailycoin.info/andresen-back-one-man-security-project-inspired-bitcoin-mishaps/

Gavin Andresen, the former lead developer of bitcoin, is breaking his silence.

While in recent months he’s been more active on Twitter discussing the block size debate (even stamping his name on a new bitcoin scaling ‘agreement‘), Andresen has largely been absent from the bitcoin developer community for about a year.

But, that doesn’t mean the prodigious worker, who did much to help build out bitcoin’s early developer team and market, hasn’t been busy.
Click to expand...

Keeping an eye on bitcoin

In May 2015, a vulnerability in Blockchain’s Android bitcoin wallet left several users out money. According to Softpedia, the vulnerability allowed duplicate bitcoin addresses to be created and given to different users. At its core, the problem was with Blockchain’s random number generator, random.org, which provided insufficient entropy on certain versions of the Android operating system.

And two years before, in August 2013, all bitcoin wallet applications on Android operating systems were potentially at risk when several vulnerabilities were found within another random number generator, Java SecureRandom.
Click to expand...

Andresen has been working on the Random Sanity Project for about six months. According to him, the project is not intended to be a profit-making business. Instead, ideally, the project would be sponsored by an entity like the Linux Foundation to offer the service to anyone for free.

So how does Random Sanity work? Every system and every programming language has a way of getting random bytes – for instance, Linux has a special folder called ‘/dev/urandom’ and OpenSSL provides several random number generators (which Bitcoin Core uses).

Users of the Random Sanity Project can take those random numbers – from 16 to 64 bytes – and input them into the service, which will return a ‘true’ if the bytes look random, or a ‘false’ if the numbers don’t.

“The problem of detecting whether your random numbers are good enough is a tricky problem,” Andresen told CoinDesk. “There are a bunch of ways you can screw up.”
Click to expand...

FYI, Centmin Mod installs and uses haveged to boost the level of Entropy servers have out of the box for non-OpenVZ systems. Details at http://www.issihosts.com/haveged/

Introduction
The haveged project is an attempt to provide an easy-to-use, unpredictable random number generator based upon an adaptation of the HAVEGE algorithm. Haveged was created to remedy low-entropy conditions in the Linux random device that can occur under some workloads, especially on headless servers. Current development of haveged is directed towards improving overall reliability and adaptability while minimizing the barriers to using haveged for other tasks.

Downloads and release notes for haveged are found here.

Background on the details of the linux random device, a survey of other solutions to entropy shortages, and notes on the design and evolution of haveged are extracted from the home page that graced this site for several years.
Click to expand...

on my OVH MC-32 server for random sanity project check http://www.randomsanity.org/details
Code (Text):
BYTES=$(openssl rand -hex 64)
curl https://rest.randomsanity.org/v1/q/$BYTES
true
curl https://rest.randomsanity.org/v1/q/$BYTES
false
interesting for RHEL 7 & CentOS 7 https://github.com/lmacken/randomsanity_redhat actual bash shell script https://github.com/lmacken/randomsanity_redhat/blob/master/randomsanity

testing randomness of the generated data with dieharder test suite

The diehard tests are a battery of statistical tests for measuring the quality of a random number generator. They were developed by George Marsaglia over several years and first published in 1995 on a CD-ROM of random numbers
Click to expand...

On OVH MC-32 server with Centmin Mod 123.09beta01
Code (Text):
yum -y install dieharder rng-utils
cat /proc/sys/kernel/random/entropy_avail
cat /dev/random | rngtest -c 1000
haveged -n 0 | dieharder -g 200 -a
Entropy available never drops below 4067 during dieharder test. Closer to 4096 bits = better randomness and SSL related performance vs closer to 0 kernel block at generating random data = poorer SSL performance
Code (Text):
cat /proc/sys/kernel/random/entropy_avail
4067
output from rngtest to check the randomness of data https://access.redhat.com/documenta...yption-Using_the_Random_Number_Generator.html

A high number of failures shown in the output of the rngtest tool indicates that the randomness of the tested data is sub-optimal and should not be relied upon
Click to expand...
Code (Text):
cat /dev/random | rngtest -c 1000
rngtest 5
Copyright (c) 2004 by Henrique de Moraes Holschuh
This is free software; see the source for copying conditions.  There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

rngtest: starting FIPS tests...
rngtest: bits received from input: 20000032
rngtest: FIPS 140-2 successes: 999
rngtest: FIPS 140-2 failures: 1
rngtest: FIPS 140-2(2001-10-10) Monobit: 1
rngtest: FIPS 140-2(2001-10-10) Poker: 0
rngtest: FIPS 140-2(2001-10-10) Runs: 0
rngtest: FIPS 140-2(2001-10-10) Long run: 0
rngtest: FIPS 140-2(2001-10-10) Continuous run: 0
rngtest: input channel speed: (min=3.325; avg=21.557; max=31.423)Mibits/s
rngtest: FIPS tests speed: (min=131.541; avg=192.172; max=211.928)Mibits/s
rngtest: Program run time: 984153 microseconds
output from dieharder all PASSED as opposed to listing as WEAK which means entropy being supplied isn't completely random. Click spoiler link to expand results
Code (Text):
haveged -n 0 | dieharder -g 200 -a
Writing unlimited bytes to stdout
#=============================================================================#
#            dieharder version 3.31.1 Copyright 2003 Robert G. Brown          #
#=============================================================================#
   rng_name    |rands/second|   Seed   |
stdin_input_raw|  4.11e+07  |2555998644|
#=============================================================================#
        test_name   |ntup| tsamples |psamples|  p-value |Assessment
#=============================================================================#
   diehard_birthdays|   0|       100|     100|0.58466400|  PASSED 
      diehard_operm5|   0|   1000000|     100|0.34741491|  PASSED 
  diehard_rank_32x32|   0|     40000|     100|0.58924299|  PASSED 
    diehard_rank_6x8|   0|    100000|     100|0.73616445|  PASSED 
   diehard_bitstream|   0|   2097152|     100|0.74486962|  PASSED 
        diehard_opso|   0|   2097152|     100|0.69833033|  PASSED 
        diehard_oqso|   0|   2097152|     100|0.95739780|  PASSED 
         diehard_dna|   0|   2097152|     100|0.15888544|  PASSED 
diehard_count_1s_str|   0|    256000|     100|0.49224258|  PASSED 
diehard_count_1s_byt|   0|    256000|     100|0.94721478|  PASSED 
 diehard_parking_lot|   0|     12000|     100|0.37605494|  PASSED 
    diehard_2dsphere|   2|      8000|     100|0.42252561|  PASSED 
    diehard_3dsphere|   3|      4000|     100|0.54756912|  PASSED 
     diehard_squeeze|   0|    100000|     100|0.70081215|  PASSED 
        diehard_sums|   0|       100|     100|0.03207225|  PASSED 
        diehard_runs|   0|    100000|     100|0.98345098|  PASSED 
        diehard_runs|   0|    100000|     100|0.48667985|  PASSED 
       diehard_craps|   0|    200000|     100|0.97647211|  PASSED 
       diehard_craps|   0|    200000|     100|0.84142025|  PASSED 
 marsaglia_tsang_gcd|   0|  10000000|     100|0.93260709|  PASSED 
 marsaglia_tsang_gcd|   0|  10000000|     100|0.65510932|  PASSED 
         sts_monobit|   1|    100000|     100|0.37642452|  PASSED 
            sts_runs|   2|    100000|     100|0.72907473|  PASSED 
          sts_serial|   1|    100000|     100|0.56484065|  PASSED 
          sts_serial|   2|    100000|     100|0.78245002|  PASSED 
          sts_serial|   3|    100000|     100|0.75464262|  PASSED 
          sts_serial|   3|    100000|     100|0.93411292|  PASSED 
          sts_serial|   4|    100000|     100|0.43683705|  PASSED 
          sts_serial|   4|    100000|     100|0.31600703|  PASSED 
          sts_serial|   5|    100000|     100|0.90585806|  PASSED 
          sts_serial|   5|    100000|     100|0.95413919|  PASSED 
          sts_serial|   6|    100000|     100|0.61627529|  PASSED 
          sts_serial|   6|    100000|     100|0.48429656|  PASSED 
          sts_serial|   7|    100000|     100|0.16817216|  PASSED 
          sts_serial|   7|    100000|     100|0.15561448|  PASSED 
          sts_serial|   8|    100000|     100|0.01516734|  PASSED 
          sts_serial|   8|    100000|     100|0.10012547|  PASSED 
          sts_serial|   9|    100000|     100|0.03920821|  PASSED 
          sts_serial|   9|    100000|     100|0.33063611|  PASSED 
          sts_serial|  10|    100000|     100|0.01733869|  PASSED 
          sts_serial|  10|    100000|     100|0.05875474|  PASSED 
          sts_serial|  11|    100000|     100|0.05293556|  PASSED 
          sts_serial|  11|    100000|     100|0.98119654|  PASSED 
          sts_serial|  12|    100000|     100|0.09514860|  PASSED 
          sts_serial|  12|    100000|     100|0.69278435|  PASSED 
          sts_serial|  13|    100000|     100|0.28010951|  PASSED 
          sts_serial|  13|    100000|     100|0.60023662|  PASSED 
          sts_serial|  14|    100000|     100|0.29961227|  PASSED 
          sts_serial|  14|    100000|     100|0.42328548|  PASSED 
          sts_serial|  15|    100000|     100|0.14600592|  PASSED 
          sts_serial|  15|    100000|     100|0.60146485|  PASSED 
          sts_serial|  16|    100000|     100|0.12137347|  PASSED 
          sts_serial|  16|    100000|     100|0.97444505|  PASSED 
         rgb_bitdist|   1|    100000|     100|0.37052849|  PASSED 
         rgb_bitdist|   2|    100000|     100|0.83575017|  PASSED 
         rgb_bitdist|   3|    100000|     100|0.30034690|  PASSED 
         rgb_bitdist|   4|    100000|     100|0.70184743|  PASSED 
         rgb_bitdist|   5|    100000|     100|0.91389874|  PASSED 
         rgb_bitdist|   6|    100000|     100|0.71450474|  PASSED 
         rgb_bitdist|   7|    100000|     100|0.51110178|  PASSED 
         rgb_bitdist|   8|    100000|     100|0.31148799|  PASSED 
         rgb_bitdist|   9|    100000|     100|0.69787661|  PASSED 
         rgb_bitdist|  10|    100000|     100|0.60763156|  PASSED 
         rgb_bitdist|  11|    100000|     100|0.37104923|  PASSED 
         rgb_bitdist|  12|    100000|     100|0.03615341|  PASSED 
rgb_minimum_distance|   2|     10000|    1000|0.74809078|  PASSED 
rgb_minimum_distance|   3|     10000|    1000|0.21609783|  PASSED 
rgb_minimum_distance|   4|     10000|    1000|0.71609594|  PASSED 
rgb_minimum_distance|   5|     10000|    1000|0.85874785|  PASSED 
    rgb_permutations|   2|    100000|     100|0.78737303|  PASSED 
    rgb_permutations|   3|    100000|     100|0.61896838|  PASSED 
    rgb_permutations|   4|    100000|     100|0.02561445|  PASSED 
    rgb_permutations|   5|    100000|     100|0.28153360|  PASSED 
      rgb_lagged_sum|   0|   1000000|     100|0.68613613|  PASSED 
      rgb_lagged_sum|   1|   1000000|     100|0.55712141|  PASSED 
      rgb_lagged_sum|   2|   1000000|     100|0.81716761|  PASSED 
      rgb_lagged_sum|   3|   1000000|     100|0.93693052|  PASSED 
      rgb_lagged_sum|   4|   1000000|     100|0.63680500|  PASSED 
      rgb_lagged_sum|   5|   1000000|     100|0.97482845|  PASSED 
      rgb_lagged_sum|   6|   1000000|     100|0.46970603|  PASSED 
      rgb_lagged_sum|   7|   1000000|     100|0.78321335|  PASSED 
      rgb_lagged_sum|   8|   1000000|     100|0.21592473|  PASSED 
      rgb_lagged_sum|   9|   1000000|     100|0.63013875|  PASSED 
      rgb_lagged_sum|  10|   1000000|     100|0.96707651|  PASSED 
      rgb_lagged_sum|  11|   1000000|     100|0.05556949|  PASSED 
      rgb_lagged_sum|  12|   1000000|     100|0.93387628|  PASSED 
      rgb_lagged_sum|  13|   1000000|     100|0.26417530|  PASSED 
      rgb_lagged_sum|  14|   1000000|     100|0.40688115|  PASSED 
      rgb_lagged_sum|  15|   1000000|     100|0.67391425|  PASSED 
      rgb_lagged_sum|  16|   1000000|     100|0.75166577|  PASSED 
      rgb_lagged_sum|  17|   1000000|     100|0.07421644|  PASSED 
      rgb_lagged_sum|  18|   1000000|     100|0.89115827|  PASSED 
      rgb_lagged_sum|  19|   1000000|     100|0.63754741|  PASSED 
      rgb_lagged_sum|  20|   1000000|     100|0.19772907|  PASSED 
      rgb_lagged_sum|  21|   1000000|     100|0.07741357|  PASSED 
      rgb_lagged_sum|  22|   1000000|     100|0.07929148|  PASSED 
      rgb_lagged_sum|  23|   1000000|     100|0.35269016|  PASSED 
      rgb_lagged_sum|  24|   1000000|     100|0.58477945|  PASSED 
      rgb_lagged_sum|  25|   1000000|     100|0.80145591|  PASSED 
      rgb_lagged_sum|  26|   1000000|     100|0.58836258|  PASSED 
      rgb_lagged_sum|  27|   1000000|     100|0.21149789|  PASSED 
      rgb_lagged_sum|  28|   1000000|     100|0.97520376|  PASSED 
      rgb_lagged_sum|  29|   1000000|     100|0.31053275|  PASSED 
      rgb_lagged_sum|  30|   1000000|     100|0.21627329|  PASSED 
      rgb_lagged_sum|  31|   1000000|     100|0.21344191|  PASSED 
      rgb_lagged_sum|  32|   1000000|     100|0.85708927|  PASSED 
     rgb_kstest_test|   0|     10000|    1000|0.66413543|  PASSED 
     dab_bytedistrib|   0|  51200000|       1|0.25151904|  PASSED 
             dab_dct| 256|     50000|       1|0.35925012|  PASSED 
Preparing to run test 207.  ntuple = 0
        dab_filltree|  32|  15000000|       1|0.38070121|  PASSED 
        dab_filltree|  32|  15000000|       1|0.95650990|  PASSED 
Preparing to run test 208.  ntuple = 0
       dab_filltree2|   0|   5000000|       1|0.01486836|  PASSED 
       dab_filltree2|   1|   5000000|       1|0.05904405|  PASSED 
Preparing to run test 209.  ntuple = 0
        dab_monobit2|  12|  65000000|       1|0.03673123|  PASSED
and more info https://blog.cryptographyengineering.com/2014/03/19/how-do-you-know-if-rng-is-working/

What this means is that an attacker who can predict the output of your RNG — perhaps by taking advantage of a bug, or even compromising it at a design level — can often completely decrypt your communications. The Debian project learned this firsthand, as have many others. This certainly hasn’t escaped NSA’s notice, if the allegations regarding its Dual EC random number generator are true.

All of this bring us back to Snowden’s quote above, and the question he throws open for us. How do you know that an RNG is working? What kind of tests can we run on our code to avoid flaws ranging from the idiotic to the highly malicious?
Click to expand...

Statistical Tests
If you look at the literature on random number generators, you’ll find a lot of references to statistical randomness testing suites like Diehard or NIST’s SP 800-22. The gist of these systems is that they look a the output of an RNG and run tests to determine whether the output is, from a statistical perspective, “good enough” for government work (very literally, in the case of the NIST suite.)
Click to expand...

Jimmy · Jun 4, 2017

Yup. Thought it might useful for other applications.

eva2000 · Jun 4, 2017

updated my 2nd post above with more info and example tests of randomness to better frame this discussion

eva2000 · Jun 4, 2017

More on randomness https://blog.cloudflare.com/ensuring-randomness-with-linuxs-random-number-generator/

Entropy and Randomness
Not all randomness is created equally. There are two sorts of randomness to think about: uniformity and unpredictability. A random number generator provides ‘uniform’ output if all numbers will come up equally often if run long enough. That’s useful for modeling random processes, but not good enough for security.

For computer security, random numbers need to be hard to guess: they need to be unpredictable. The predictability of numbers is quantified in a measure called entropy.

If a fair coin is tossed it provides one bit of entropy: the coin lands with equal probability on heads or tails (which can be thought of as 0 and 1). Because the probability is equal there’s no predictability in the coin’s ‘output’. We say it provides one bit of entropy.

An unfair coin toss provides less than one bit, since it’s much easier to guess when you know the bias. Flipping a coin with heads on both sides provides no entropy, since the result of a coin toss can be guessed with absolute certainty.
Click to expand...
Take a dip in the pool
On Linux, the root of all randomness is something called the kernel entropy pool. This is a large (4,096 bit) number kept privately in the kernel’s memory. There are 24096 possibilities for this number so it can contain up to 4,096 bits of entropy. There is one caveat - the kernel needs to be able to fill that memory from a source with 4,096 bits of entropy. And that’s the hard part: finding that much randomness.

The entropy pool is used in two ways: random numbers are generated from it and it is replenished with entropy by the kernel. When random numbers are generated from the pool the entropy of the pool is diminished (because the person receiving the random number has some information about the pool itself). So as the pool’s entropy diminishes as random numbers are handed out, the pool must be replenished.

Replenishing the pool is called stirring: new sources of entropy are stirred into the mix of bits in the pool.

This is the key to how random number generation works on Linux. If randomness is needed, it’s derived from the entropy pool. When available, other sources of randomness are used to stir the entropy pool and make it less predictable. The details are a little mathematical, but it’s interesting to understand how the Linux random number generator works as the principles and techniques apply to random number generation in other software and systems.

The kernel keeps a rough estimate of the number of bits of entropy in the pool. You can check the value of this estimate through the following command:
Code (Text):
cat /proc/sys/kernel/random/entropy_avail
A healthy Linux system with a lot of entropy available will have return close to the full 4,096 bits of entropy. If the value returned is less than 200, the system is running low on entropy.
Click to expand...
Running out of entropy
One of the dangers of a system is running out of entropy. When the system’s entropy estimate drops to around the 160 bit level, the length of a SHA-1 hash, things get tricky, and how they effect programs and performance depends on which of two Linux random number generators are used.

Linux exposes two interfaces for random data that behave differently when the entropy level is low. They are /dev/random and /dev/urandom. When the entropy pool becomes predictable, both interfaces for requesting random numbers become problematic.

When the entropy level is too low, /dev/random blocks and does not return until the level of entropy in the system is high enough. This guarantees high entropy random numbers. If /dev/random is used in a time-critical service and the system runs low on entropy, the delays could be detrimental to the quality of service.

On the other hand, /dev/urandom does not block. It continues to return the hashed value of its entropy pool even though there is little to no entropy in it. This low-entropy data is not suited for cryptographic use.

The solution to the problem is to simply add more entropy into the system.
Click to expand...

Log in / Register

Forums

Discover Centmin Mod today

Sysadmin Random Sanity Project

Jimmy Well-Known Member

eva2000 Administrator Staff Member

Jimmy Well-Known Member

eva2000 Administrator Staff Member

eva2000 Administrator Staff Member

.

Log in / Register

Useful Searches

Discover Centmin Mod today

Sysadmin Random Sanity Project

Jimmy Well-Known Member

eva2000 Administrator Staff Member

Jimmy Well-Known Member

eva2000 Administrator Staff Member

eva2000 Administrator Staff Member

.