Several weeks ago, me and my partner, Shimi Bandiel, were asked by one of the largest global software companies to help them with a strange performance problem.
The problem was indeed strange, they had a fairly simple web application which was running perfectly well. The programmers working on this project have read that migrating to from Java 5 to Java 6 will result in better performance without any additional effort. The strange thing was that as soon as they have moved to version 6, JSP pages were awfully slow the first time they were accessed, and when I say slow I mean it, it was about 60 seconds per page to load.
Shimi and me rolled up our sleeves and started to work. Soon enough, we have discovered that the loading of the Servlet class generated from the JSP was the time consuming operation. A short thinking made us assume the problem is with the new class version of Java 6, we have made sure that the JSP is compiling the classes to Java 6 and that everything is set correctly. Everything seems to be fine. Then Shimi came up with a new idea, maybe it is a problem with the new verifier introduced in Java 6, the Split Verifier. In Java 6 the verification process is split to two phases: type inferencing (off-line) and type checking (on-line). This means the off-line part is done by the compiler which inserts data to the class file. This new verification process should speed up the class-loading process significantly. Unfortunately for some mysterious reason, this verification process failed when loading the JSP classes, when fails, this process falls back to the old verification and the result was a VERY long verification.
Since that project had to go online quickly, we had to find a fast solution, again, Shimi pooled something out of his sleeve, two flags that control the new verifier:
-XX:+UseSplitVerifier
-XX:+FailOverToOldVerifier
By using these flags we have manged to disable the split verifier and solve the problem.
Following are my conclusions from this experience:
- When trying to figure why systems act diferently on Java 6, keep in mind the Split Verfier.
- When you have performance problems, call Shimi
