Weblogic server takes long time to start. Performance tuning of Weblogic server startup.

Weblogic server takes long time to start. Performance tuning of Weblogic server startup.

Cause:

It is observed on some Linux boxes that WebLogic server startup takes several minutes and hangs for a while. Similar behavior happens during the domain creation, when the security information gets populated.

It has also been observed WLS Admin Console slowness due to low entropy.

Linux has two devices to provide random data at any time: /dev/random and /dev/urandom. Both ways should be secure enough to use them in generating PGP keys, ssh challenges, and other applications where secure random numbers are required. Starting on kernel 2.6, default entropy is 4096 bits and problem arises when the entropy available on the system is minimum (around 100 bits or less).

The main difference between those two devices is that /dev/random runs out of random bits and makes you wait for more to be accumulated. Note that on some systems, it can block for a long time waiting for new user-generated entropy to be entered into the system.

In terms of the outcome, /dev/random is categorized as a high quality entropy device if we compare it with /dev/urandom. The latter uses the entropy pool as long as it is available, but falls back on pseudo random numeric algorithms when depleted.

Why a system could be running out of entropy?

You have to consider that an Operating System performs cryptographic operations frequently (on ssh challenges, https connections, etc.) so the /dev/random pool gets consumed quite quickly. OS also expects to feed that pool with I/O operations coming from disk, network, mouse or keyboard but that situation does not happen as quickly. This is a common pattern on virtualized environments or headless boxes.

Is important to mention that Java uses /dev/random by default as entropy generator device.

How to verify if you are encountering this issue?

1. Check the default system entropy.

$ cat /proc/sys/kernel/random/poolsize
4096

2. Check the available entropy.

$ cat /proc/sys/kernel/random/entropy_avail
125

On previous example, entropy is too low.

3. Monitor the current entropy of the system by using the following command:

$ for i in $(seq 500); do cat /proc/sys/kernel/random/entropy_avail ; sleep 5; done

4. Start a WebLogic server instance. You should see that entropy decreases or stalls.

Solution:

a) WebLogic Server Scope

i.   Edit the Weblogic startup script ($DOMAIN_HOME/bin/startWebLogic.sh)
ii.  Add the following to the JAVA_OPTIONS variable: -Djava.security.egd=file:/dev/./urandom
iii. Save the file.
iv. Set the domain environment. ($DOMAIN_HOME/bin/setDomainEnv.sh)
v.  Start WebLogic instances.

b) JDK Scope

i.   Edit the Java Security Properties file ($JAVA_HOME/jre/lib/security/java.security)
ii.  The securerandom.source property specifies the source of seed data for secure random. If that property points to /dev/random, set it as one of the options listed below.
securerandom.source=file:/dev/./urandom

securerandom.source=file:/dev/urandom

iii.  Save changes and start the WebLogic Server instances.

Temporary solution :

i. Override the JAVA_OPTIONS environment variable before starting WebLogic Server via shell scripts.
$ export JAVA_OPTIONS="${JAVA_OPTIONS} -Djava.security.egd=file:/dev/./urandom"

ii. Start WebLogic instances.

Happy Learning!