What is Oracle WebLogic?
Oracle WebLogic Server (WLS) is one of the leading Java Enterprise Edition (Java EE) application servers in the market today. Offering a robust, mature, and scalable implementation of the J2EE specification, the WebLogic Platform is a unified, extensible platform for developing and deploying applications based on Service-Oriented Architectures (SOA). According to the reviews from TrustRadius, WebLogic is the most trusted choice for an enterprise-ready Java EE application server with a close to 8.5 out of 10 market score.
What are the Key WebLogic Server Performance Metrics?
To ensure the availability and performance of Java applications running on Oracle WebLogic, performance monitoring of all aspects of Oracle WebLogic’s functioning is essential. In this blog, we discuss the top 8 areas that performance monitoring solutions need to cover to provide IT operations teams with the insights they need for proactively monitoring the performance of a WebLogic application server and the applications hosted on it.
A Peek into the Architecture of a WebLogic Application Server
To determine the key areas that we need to focus on to get the best performance out of a WebLogic application server, let’s look at the typical application architecture that Oracle WebLogic is deployed in.
A WebLogic Server centralizes application services such as Web server functionality, business components, and access to backend enterprise systems. It uses technologies like caching and connection pooling to improve resource use and application performance. A WebLogic Server also provides enterprise-level security and powerful administration facilities.
A WebLogic Server operates in the middle tier of a multitier (or n-tier) architecture. This is the tier that is placed between the front-facing external applications “Client Layer” – and the database servers “Backend Layer.” From its middleware position, a WebLogic Server delivers application services like the multiple functions of a web server, the business logic and connectivity to backend data storage.
The WebLogic Server cluster option allows administrators to distribute client requests and back-end services among multiple cooperating WebLogic Servers. Clients access the cluster as if it were a single WebLogic Server. As the workload increases, you can add WebLogic Servers to the cluster to share the work. The cluster uses a selectable load-balancing algorithm to choose a WebLogic Server in the cluster that is capable of handling the request.
#1 Monitoring of WebLogic JVM Performance
WebLogic application servers operate on top of Java Virtual Machines (JVM). Therefore, bottlenecks in the JVM can severely affect the performance of the WebLogic application server. Hence, it’s essential to monitor the performance of the WebLogic server’s JVM to troubleshoot server-side bottlenecks and answer the following questions:
- Are the JVM heap and non-heap memory pools sized correctly?
- Is any thread taking an excessive amount of CPU, and which code snippet is being executed when this is observed?
- Are there any out-of-memory-exceptions occurring during application execution?
- Is garbage collection taking too much time, affecting the performance of the application?
- If there is a constant increase in memory usage in the JVM, is there a memory leak in the application deployed on Oracle WebLogic? If yes, where?
- Are there any instances of thread blocking or deadlocks? If yes, which threads and which lines of code are involved?
- When does garbage collection happen, and how much memory is freed up each time?
Monitoring of the JVM must be an essential part of any Java application performance monitoring strategy. IT Ops and DevOps teams use JVM performance metrics to troubleshoot server-side bottlenecks. Developers and architects can also benefit from JVM monitoring by uncovering code-level issues.
Helpful Tips:
- The max limit for the individual memory spaces also must be tuned correctly.
- Monitor JVM memory spaces and growth patterns continuously to proactively detect situations when there is a memory shortfall.
- Fix code-level issues that cause unused objects to use up heap memory.
- High CPU usage of the JVM can happen when there is excessive garbage collection happening. When you don’t see your application threads taking CPU, check the performance of garbage collection. A memory issue can manifest as high CPU usage, making performance diagnosis difficult.
#2 Monitoring WebLogic’s Execute Queues
When a request is received by a WebLogic Server, it is placed in an execute queue. This request is then assigned to a thread that does the work on it. Threads consume resources on the server, so a large number of threads handling requests can cause more memory to be used and also increases context switching, which can degrade performance. WebLogic administrators often configure individual execute queues for key applications. By using multiple execute queues, you can guarantee that selected applications have access to a fixed number of execute threads, regardless of the load on WebLogic Server.
Monitoring the execute queues is important because this is where you can determine whether the WebLogic server is able to keep up with its request workload. Track the number of threads in use in each execute queue and the thread utilization level. This can help you proactively determine times when the WebLogic server is overloaded.
If the number of open sockets is higher than the number of socket reader threads allocated, incoming requests may be required to wait until a socket reader thread is free. However, it is essential to balance the number of execute threads that are dedicated to reading messages from a socket and that perform the execution of tasks in the server. Continuously monitor the number of open sockets – excessive number of sockets opened can result in performance degradation.
Helpful Tips:
- Tuning of these thread pools and discovering the proper estimate of threads is time-consuming and requires many trials. If the number of incoming client requests has resulted in all the execute threads being allocated, then at 100% utilization, the WebLogic server will not have any threads available to service incoming requests. This degrades your performance.
- Monitor the WebLogic server’s throughput as the execute queue length is increased. At some point, increasing the number of threads will lead to enough context switching that the throughput for the queue begins to decrease.
- Continuously track thread pool metrics, such as allocated, freed, created, closed, and managed requests.
#3 Monitoring WebLogic’s WorkManagers and Stuck Threads
WebLogic uses a concept called WorkManager in order to prioritize work and maintain threads and thread pools. WorkManager can be created and configured by the administrator on the WebLogic level or by application developers on the application level. The WorkManager enables you to guarantee that each application will get their chunk of the available resources (threads/connections) or you can limit the amount of resources.
During the operation of the WebLogic server, it is important to track the requests being processed by each WorkManager and the requests pending for each WorkManager. Another key metric is stuck threads in the WorkManager. WebLogic Server diagnoses a thread as stuck if it is continually working (not idle) for a set period of time. You can tune a server’s thread detection behavior by changing the length of time before a thread is diagnosed as stuck (Stuck Thread Max Time), and by changing the frequency with which the server checks for stuck threads.
A stuck thread cannot complete its current work or accept new work. If many stuck threads are detected, the WorkManager itself can be shutdown. Hence, monitoring the number of stuck threads can provide proactive indicators of problem situations.
Helpful Tips:
- The recommended way to track stuck threads is by tracking the thread activity in the JVM.
#4 Monitoring WebLogic JDBC Connection Pool Performance
Applications running on WebLogic often need to access the backend database. Establishing and tearing down connections to the database server often take time and include a lot of overhead. WebLogic includes built-in support for JDBC connection pools. Each connection pool is a group of connections that the WebLogic establishes and maintains to the backend database server. When an application requests for a JDBC connection, the WebLogic server provides it a connection from the existing pool. When the application is done with its work, it returns the connection to the pool for use by other connections. The use of a connection pool improves application response time and minimizes overhead as connections are re-used across multiple client requests.
Monitoring the JDBC connection pools is vitally important as database access is often a bottleneck for Java applications. For a start, it is essential to track the availability of the connection pool. During operation of the WebLogic server, track the utilization of the connection pool – i.e., how many of the connections in the pool are in use, and how close is the connection pool to using the max connections allocated to it. If the connection pool runs out of capacity, application access to the database server will fail. Therefore, if the connection pool usage is reaching the max number of connections possible in the pool, you should consider increasing the max setting for the connection pool. Another important measure is the wait time to get a database connection from the pool. A larger wait time is usually an indicator that the connection pool capacity is too small for the workload the server is handling. Consider increasing the JDBC connection pool size if the wait time for a connection is high.
Another common problem encountered when dealing with JDBC connection pools is connection leaks. An application may request a connection from the pool but not return it back to the pool. WebLogic detects such situations and records leaked connections in the connection pool. Tracking the number of leaked connections can provide indications of potential issues in the application logic.
Helpful Tips:
- Analyze the utilization of available JDBC connections in a connection pool to avoid performance degradation caused by having to wait for a JDBC connection.
- The best way to diagnose JDBC connection leaks is by analyzing the WebLogic server’s log. A warning is logged whenever a connection leak is detected, and the stack trace provided often gives a clue as to where the application code issue lies.
#5 Monitoring WebLogic JMS Performance
WebLogic Java Message Service (JMS) enables applications to communicate with one another through the exchange of messages. Communications can be point-to-point or publish/subscribe.
With point-to-point messaging, the message arrives at a queue, and it is held in the queue until either the consumer processes the message, or the message expires. Deletion of the queue or shutting down of the server (in the case of a non-persistent message) can also cause the message to be removed.
With the publish-and-subscribe model, messages are sent to a topic and delivered only to the active consumers.
Communication issues between applications can be detected by monitoring key JMS metrics for the WebLogic server. Continuously track the current messages queued, messages pending, and the message send and receive rates. If any of these metrics shows an unusual trend, this may be indicative of potential application problems. When point-to-point messaging is used, monitor the above metrics for each of the queues. If the messages in the queue are not being processed quickly enough or not being processed at all, then there is a communication issue on the WebLogic server that will affect the functioning and performance of one or more applications executing on it.
In the case of publish-and-subscribe messaging, monitor the messages in a topic, the subscribers to a topic and pending messages over time.
Helpful Tips:
- A large message backlog is often problematic. This can lead to out-of-memory exceptions on the server, which in turn prevents the server from doing productive work. A large backlog can also lead to high garbage collection overhead.
- There are several ways to tune applications for improved JMS performance. Using multi-threading consumers is one way to avoid excessive message queueing. See https://docs.oracle.com/middleware/11119/wls/PERFM/jmstuning.htm#CHDBHBGF
- Leverage distributed JMS queues where possible for improved JMS performance.
#6 Monitoring WebLogic Java Transaction Processing
Java Transaction API (JTA) is built into WebLogic to manage transactions. JTA allows applications to perform distributed transactions, i.e., transactions that access and update data on two or more networked computer resources. The JTA specifies standard Java interfaces between a transaction manager and the parties involved in a distributed transaction system: the application, the application server, and the resource manager that controls access to the shared resources affected by the transactions.
The transaction manager is responsible for making the final decision either to commit or rollback any distributed transaction. A commit decision should lead to a successful transaction; rollback leaves the data in the database unaltered.
Administrators must track the number of active transactions, transaction aborts, rollbacks, commits over time to determine times when application performance will suffer because there are excessive transaction aborts and rollbacks.
Helpful Tips:
- WebLogic includes support for the Logging Last Resource (LLR) transaction optimization. LLR offers significant performance improvements for transaction processing.
#7 Monitoring of JSP & Servlet Performance in the WebLogic Application Server
The core application logic of a web module is implemented in the JSP pages and servlet files on the server-side. When a user accesses a JSP page for the first time, WebLogic automatically translates the JSP file into a servlet. The primary function of a servlet is to accept an HTTP request from a web browser and return an HTTP response.
When a web module is experiencing a lot of requests, or the processing time is high, the immediate question would be, is it the increased traffic or slower processing is attributable to one or more servlets being processed. Monitoring tools for WebLogic provide insights into the requests handled by each servlet and the processing time of each servlet. By comparing the processing time across servlets, administrators can determine the areas of the web application that need to be optimized.
Helpful Tips:
- Monitor the average execution time for a servlet and the workload to each servlet.
- Changes to the workload over time can cause performance issues and must be investigated.
- Comparing execution times across servlets may indicate areas of code that may need to be optimized.
#8 Monitoring the Performance of Business Transactions on WebLogic
Knowing that a particular servlet is taking time is useful, but an immediate next question is, “Why is the servlet taking time to execute?” Administrators would like to know the exact line of code that is causing slowness. They would also like to know if the slowness is in Java processing (e.g., expensive method calls) or whether it is due to an external call to a remote service (e.g., a REST API call to a payment processing service), or whether the slowness is caused by database processing and, if so, which query/queries are causing slowness.
Getting to this level of detail requires a deep dive into the application code. This is where business transaction tracing comes in. This is a common approach used by Java application monitoring tools, wherein the instrumentation API of modern JVMs is used to add byte-code instrumentation that captures all method calls in Java and external calls from the JVM. This capability provides administrators with the insights they need to discover and troubleshoot Java application slowness issues.
Helpful Tips:
- Transaction tracing to determine code-level issues is not free. Consider the additional overhead of transaction processing when sizing the WebLogic server (heap memory, CPU available, etc.).
- Monitoring tools often have a development mode and a production mode. Make sure when deploying transaction tracing in production that you have the production mode turned on for transaction tracing.
#9 Monitoring WebLogic Server’s Uptime
Many times server crashes can happen and go unnoticed because automatic actions have been configured to restart the server in the event of a crash. Track the uptime of the WebLogic server’s JVM and be alerted to instances when the WebLogic server restarted.
Helpful Tips:
- Investigate CPU, memory, and thread stack trace of the JVM prior to restart to detect possible reasons why the server had to be restarted.
Monitoring All Aspects of WebLogic Application Server Performance
It is essential to have a proactive monitoring tool in place to monitor all of the above nine key areas of WebLogic performance. Administrators can use these metrics to learn about WebLogic configuration issues and bottlenecks. Development teams can also benefit from the insights provided by these monitoring tools. They can identify which line of application code or which database query is causing application slowness.
eG Enterprise from eG Innovations offers 360° visibility into all aspects of WebLogic application server performance. From a single pane of glass, administrators can see the performance of the WebLogic server, the underlying JVM, the server operating system, as well as the virtual or containerized environment in which WebLogic is running.
eG Enterprise is an Observability solution for Modern IT. Monitor digital workspaces,
web applications, SaaS services, cloud and containers from a single pane of glass.
