““Help! Our PaperCut Application server is acting unpredictably, and we see lots of intermittent issues especially during peak usage times; how should we go about troubleshooting this problem?”
Symptoms of a performance problem
If you’ve already tried “turning it off and on again” and you still see some of the symptoms in the list below, then this might be the right troubleshooting article for you.
- The PaperCut server’s web interface is slow to load.
- Logging into MFDs may be slow or time out. These devices may display an error similar to “unable to connect to server” or “connection timed out” when users try to log in, especially in the middle of the day.
- The server might show consistently high CPU or Memory usage, plateauing at 100%.
- The common thread is that the problem often cannot be reproduced consistently, and is usually reported during the busiest times of the day for printing.
- You may see the error Warning: Print Provider “printsrv01|192.0.1.5”: Print job “Document-1.pdf” printed by “user1” is offline and has been temporarily hidden from the Print Release screen. as described in the article Why was this job ‘Temporarily Hidden’? but you have ruled out the other causes.
Slow Printing or Disappearing Print Jobs can also be indicators of a performance issue but also have many other causes.
Basic checks
You may have already gone through the suggestions in our article on Keeping a healthy system, and you’re still having trouble. So what’s the next thing to check?
Every server OS has a built-in utility for checking which apps are using the greatest amount of CPU or Memory. Here’s how to find how much CPU and memory is getting used by PaperCut on each type of server:
- Windows - in the native OS Resource Monitor or Task Manager applications, look for pc-app.exe.
- macOS - in the native OS’ Activity Monitor application, look for the process named Java running as the user papercut.
- Linux - open the Terminal command window and run the top command, look for the process named pc-app.
Is there anything else running on the PaperCut NG/MF server that is consuming excessive resources? As an IT best practice, we strongly recommend having a dedicated server in your environment for PaperCut NG/MF.
However, if you see that one of those PaperCut processes is the sole culprit then read on…
Make sure the PaperCut NG/MF server has adequate CPU/Memory allocation
Check our Server Sizing Guide to make sure that the server is scaled with the appropriate resources. Remember that Site Servers and Secondary PaperCut servers need adequate resources as well to function as part of the printing ecosystem.
One extreme sign of a performance issue is to look in the service.log file found on any PaperCut Application server for messages like JVM Process has not received any CPU time for X seconds. This indicates that the PaperCut application, which runs as a Java Virtual Machine (JVM), is not getting enough resources from the operating system or virtual host. The service.log file can be found in the \[app-path]\PaperCut NG/MF\server\log.
INFO | jvm 1 | 2020/04/14 13:11:31 | JVM Process has not received any CPU time for 23 seconds. Extending timeouts.
INFO | jvm 1 | 2020/04/15 05:08:17 | JVM Process has not received any CPU time for 21 seconds. Extending timeouts.
INFO | jvm 1 | 2020/04/15 20:56:36 | JVM Process has not received any CPU time for 19 seconds. Extending timeouts.
…but don’t give the server too much memory
It is possible to have too much of a good thing. Follow our Server Sizing Guide to give your PaperCut server the right amount of memory, and don’t go overboard.
Why? PaperCut NG and PaperCut MF are Java applications, and hence subject to Java “garbage collection” activities, which maintain proper memory allocation. When too much memory is available to the Application Server, full garbage collections will happen less frequently but take longer to complete, creating a drain on resources.
By default the PaperCut server will use one quarter (¼) of system memory. If you want to fine-tune this value, but retain the same amount of physical memory allocated to this server, then follow our guide on how to adjust the amount of memory available to PaperCut NG/MF.
…and reserve the memory for servers that are VMs
This point is extremely important and often overlooked. If the PaperCut server is running on a virtual machine, make sure the memory is reserved and not dynamically allocated. We have seen that the Java Virtual Machine that PaperCut runs on top of does not like it when memory is part of a pool that can be dynamically allocated to other servers.
For a VM running in a VMware ESXi host, this setting is called “Reserve all guest memory” and you want it enabled. For a VM running in a Microsoft Hyper-V host, this setting is called “Dynamic Memory” and you want it disabled.
We describe this problem in more detail in this article which answers the question, Does PaperCut work on a Virtual Machine (VM)?.
When troubleshooting a performance issue for Virtual Machines, don’t forget to make sure that your virtual host is happy and healthy as well.
Check how many print jobs are being held
PaperCut NG/MF has to track each print job held on the server, and more print jobs held means more server resources used. You might start to run into a problem after configuring PaperCut NG/MF to hold print jobs for longer than the default of 2–4 hours as this can cause too many print jobs to begin to accumulate that PaperCut NG/MF must continually account for.
To see how many print jobs are being held, log into the PaperCut server as administrator and look for the number of “Hold/release jobs” on the dashboard. More than 1500 held jobs at any moment should be considered a red flag and you should strongly consider changing the Job Timeout period to a lower value to reduce this number. This change will not take effect immediately, because it will only apply to any new print jobs that reach the server so it may be a few hours before you start to see the benefits.
Too many clients running the Direct Print Monitor
The PaperCut Direct Print Monitor may be installed on user’s workstations to track printing to local or USB printers. However when there are several hundred or thousands of clients running the Direct Print Monitor this may put a strain on the PaperCut server.
This scenario is discussed in much greater detail in our article Environments with large numbers of Direct Print Monitors.
Find out if the database is a bottleneck
Check in the logs for the term DatabaseHealthChecker. In the example below the database connection takes 6000 ms (6 whole seconds!) which is not healthy.
DEBUG DatabaseHealthChecker:69 - Database connection test - success. Connect duration: 6017 ms. Query duration: 0 ms. [db-connection-test]
This particular log snippet was due to an issue we discovered with the default driver for SQL databases that comes bundled with Java, since resolved, that we have documented here.
If NG/MF is configured to use the Internal Derby Database (default configuration)…
- If this server has been around for many years, consider Optimizing your database. This operation is a bit like defragmenting a hard drive in that it might provide a slight performance boost, but is not going to help much if there is a larger underlying issue.
- It may be time to upgrade to an external database hosted on another server. When implemented properly this will take a load off the server running PaperCut.
If NG/MF is already configured to use an external database…
- Is the database server properly resourced and is it near enough to the PaperCut Application server to ensure a good connection?
- Run a 3rd party I/O test on your database server to help identify if there is a latency issue when writing to the disk array.
- To narrow down whether this is a problem with the specific database you are using, consider switching database types to see if the problem goes away.
Turn off debug when not needed
We sometimes see that Debug Mode is left enabled on busy servers. While debug mode is important to catch and diagnose an issue, it shouldn’t be left on indefinitely because the additional read/write operations can cause a measurable increase in load at large sites (on the scale of tens of thousands of clients). If you are troubleshooting a performance issue and leave debug mode enabled, this can compound the problem while clients have to wait for the server to update the log file. Heed this advice:
- Standard logging in most cases will include the information we need to search for key messages.
- Leave debug logging disabled on the server, unless it’s required to diagnose an active problem.
- If planning a restart during high daytime load, disable debug logging before the restart.
- When troubleshooting a performance issue, check to make sure that the different types of debug modes have not been left on by accident, including Application Server Debug Mode, SSL Debug, and SMTP Debug.