“Help! The web interface of the PaperCut server won’t load, and nothing seems to be working at all. How do we figure out what’s stopping our PaperCut server from starting?”
What do we mean by “the server won’t start”?
This simple phrase can mean a lot of different things, but for the purpose of this article “the server won’t start” means that many PaperCut features are not working and admins cannot log into the web interface locally, http://localhost:9191 or http://127.0.0.1:9191. This may be the case because the PaperCut Primary Application encountered an error when starting up and is not running.
If you are able to log on using one of those two URLs from the server, but can’t reach the server from another workstation, that sounds like a networking issue (wrong IP, wrong hostname, the ports are blocked on the firewall, something to do with DNS, etc…) which is outside the scope of this article. Instead you should check our article PaperCut Web Interface Won’t Load.
If you are the system administrator for your organization, great! If that doesn’t sound like you we recommend looping in your IT department/Help Desk/Service Desk at this point before going any further.
We see this symptom happen for a few reasons that we’ll explore in this article, like…
- PaperCut was configured to run as a service account, but the password has expired.
- The database has become corrupted because of tampering.
- Another application is trying to use the same ports as the PaperCut server.
- The Keystore which holds the SSL certificate has been misconfigured.
- A problem connecting an external database.
Did you try turning it off and on again?
We know this is a cliche, but in the world of IT this works more often than it should so we always ask.
Establish a timeline
Ask what changes have been made to the server or environment recently. Knowing if there are any changes made days or weeks before the problem surfaced helps identify the root cause.
- Was it working before or is this a new setup?
- Was PaperCut recently updated?
- Have any patches just been installed on the server?
- Have any changes been made to the network or firewall?
- Was a custom certificate recently installed on the PaperCut server?
- Has the server (or external database server) recently run out of disk space?
Is the PaperCut service started?
The first thing to check is to ensure the PaperCut Primary Application server service is started. See Stopping and Starting PaperCut Services for information on how to check whether the service is started on other platforms like macOS and Linux.
Steps:
- On a Windows PaperCut server open the Services menu by pressing Windows key + R, then type services.msc and hit the enter key.
- Look for the PaperCut Primary Application Server service and make a note about whether it is started or not.
- Right-click on this service and choose Restart.
- Look for any errors when the service starts up.
If you see a message like “Error 1069: The service did not start due to a logon failure” this occurs when PaperCut has been configured to run as a Service Account but the password used for this account has expired or changed. To resolve, right click on the service, choose Properties, click the logon tab, and have the Admin update the password for this service account. Alternatively, configure the service to run as the default Local System until the issues can be resolved.
If you see a message like “Error 1053: The service did not respond to the start or control request in a timely fashion” there’s no need to panic just yet. This happens because PaperCut occasionally takes a long time to start up. Wait a minute or two and then try logging into the web interface to see if the issue is resolved. If not, please continue reading.
The HTTP Error 404 Page
A 404 error means the page wasn’t found. This might be seen if you are browsing to a non-existent URL. Make certain you are browsing to the correct login URL, for example: http://YourPaperCutServerName:9191/. For more information, see our article How do I log in?
The “System currently unavailable” Page
Instead of seeing the PaperCut login page, you might see a simple page with the message “System currently unavailable” with information about the database connection details, and what error was received when PaperCut tried to connect. In the example above, the message is “Connection refused” and this issue was reproduced by simply pointing to a nonexistent database.
More information about the root cause can be found in the server.log file (further down in this article we explain where this is stored) if you open this file and look for the string “Unable to establish database connection”. The lines that follow that message may offer clues to what the problem is.
ERROR DatabaseUtils:316 - Unable to establish database connection. (next exception 1) - Page Page(3,Container(0, 66064)) is at version 430, the log file contains change version 1,641, either there are log records of this page missing, or this page did not get written out to disk properly. [main]
java.sql.SQLException: Page Page(3,Container(0, 66064)) is at version 430, the log file contains change version 1,641, either there are log records of this page missing, or this page did not get written out to disk properly.
The above error is reporting that the server log file (this is a file that acts like an audit log) is out of sync with the database it is trying to connect to. This will prevent the database from loading correctly.
Reasons for database connection issues…
- The server hosting the database cannot write because it is out of disk space or the transaction logs are full.
- The external database is unavailable because it is not running or the connection details have changed. Confirm that your External Database (Oracle, PostgreSQL, MySQL, MSSQL) is online, that the services are running, and that PaperCut has the current login details for this database in the server.properties file.
- The time settings are off between the database and the PaperCut server. Confirm both servers are pointed at the same time source (NTP server).
- If the above steps don’t help then it’s possible the database may be corrupt, particularly if it has been tampered with recently. To identify and fix a corrupt database, please refer to our guide: Detecting and Resolving Database Corruption or just jump ahead to the section in the manual that discusses restoring your database from a backup.
The “An error occurred starting the PaperCut MF application server service” Page
Some customers have reported that the service starts successfully but when trying to access the Web Interface of the PaperCut Server they encounter a message “An error occurred starting the PaperCut MF application server service.” Followed by one of these messages in the “Error description” section…
- “Error description: There is not enough space on the disk”
- Error description: Checksum does not match, data broken
- Error description: Lock wait timeout exceeded; try restarting transaction
- Error description: GC overhead limit exceeded
- Error description: Invalid object name ‘tbl_config’
- Error description: null
- Error description: 0
- And others…
Sometimes, these are straightforward to resolve. “There is not enough space on the disk” means exactly what it says. Free up some disk space on your PaperCut server and try restarting it. The message “Lock wait timeout exceeded; try restarting transaction” is less clear but points towards an issue with the database.
A few of the errors are a bit more enigmatic, but thankfully these issues are few and far between. If you do encounter this page please make a copy of the full text to share with us and start a support ticket.
Some of these errors were a bit more common before 18.3.6, so if you are on an older version of PaperCut please try following the steps in our article: MapDB Errors in PaperCut.
If no page loads at all… check the logs
If your PaperCut server is unable to start then you won’t be able to log into the Web Interface to download the diagnostic bundle, but you still can access these log files manually.
Steps:
- Log onto the server running PaperCut.
- Browse to where the PaperCut logs are stored on your server, [app-path]/server/logs/. (On a 64-bit Windows server running PaperCut MF, this path would be C:\Program Files\PaperCut MF\server\logs.)
- Make a copy of the entire logs folder, zip it up so that you can share it with us.
At this point, you can start a Support ticket with us or if you know what you’re doing, you can skip to the bottom of the server.log and service.log file to see a reason why the service has failed to start up. Look for the string “Stopping server” like in the example below because whatever error messages are right above that often give us a clue as to what’s going awry with the server.
INFO AppServer:371 - Stopping server. [WrapperSimpleAppMain]
INFO AppServer:373 - Stopped server. [WrapperSimpleAppMain]
Below we’ve described some errors we’ve seen in the server.log file or the service.log file.
“Unable to start server. Address already in use: bind” in the logs
ERROR AppServer:174 - Unable to start server. Address already in use: bind [WrapperSimpleAppMain] java.net.BindException: Address already in use: bind
In our experience, this error which is found in the server.log file suggests that some other application is trying to use the same ports as the PaperCut server. It could be that both PaperCut and a 3rd party application are both attempting to listen on port 9191 or 9192, which could cause this conflict. However, we find this error is much more likely to happen if PaperCut has been configured to listen on standard ports (80 and 443) or after enabling a Payment Gateway that listens on 80 or 443 because it’s much more likely that another application will be using these ports. For Windows servers, the problem is often that the IIS role is installed or some other web server application is running.
The best way to find out what other applications are using these ports is to open PowerShell (Windows) or Terminal (macOS and Linux) and run one of the following commands to look for any application listening on port 9191.
- Windows - open PowerShell as administrator and run Get-Process -Id (Get-NetTCPConnection -LocalPort 9191).OwningProcess then look for any app other than pc-app.exe using this port. (The old school method was netstat -ano but it’s easier to parse the output from the equivalent PowerShell command).
- MacOS - open Terminal and run sudo lsof -nP -i$PORT | grep LISTEN | grep “9191” then look for any app other than java using this port.
- Linux - open Terminal and run sudo netstat -ap | grep “9191” then look for any app other than pc-app using this port.
If you identify that another application is trying to listen on the same ports as PaperCut, then please stop or remove the conflicting application and restart the PaperCut server to see if that resolves the issue.
If your PaperCut server is configured to listen on standard ports as mentioned above, try running those same commands but substitute 9191 with 80 or 443.
“Unable to start server… Cannot recover key” in the logs
ERROR AppServer:178 - Unable to start server. Multiple exceptions [WrapperSimpleAppMain]
MultiException[java.security.UnrecoverableKeyException: Cannot recover key, java.security.UnrecoverableKeyException: Cannot recover key]"
These messages found in the server.log file means that the server.properties file was not configured with the correct password for the key in the KeyStore. We recommend following our guide on how to Install an SSL Certificate the Easy Way and make sure the server.properties file is configured with the correct values on the lines server.ssl.keystore-password= and server.ssl.key-password=.
If you’re encountering this error after an upgrade, please reach out to us for more information and mention PC-14894.
“Could not reserve enough space for object heap” in the logs
STATUS | wrapper | 2011/05/23 18:02:12 | Launching a JVM...
INFO | jvm 5 | 2011/05/23 18:02:13 | Error occurred during initialization of VM
INFO | jvm 5 | 2011/05/23 18:02:13 | Could not reserve enough space for object heap
INFO | jvm 5 | 2011/05/23 18:02:13 | Could not create the Java virtual machine.
ERROR | wrapper | 2011/05/23 18:02:13 | JVM exited while loading the application.
This error found in the service.log file suggests a problem related to the memory allocation for the application. If you see these messages, verify that your server meets the minimum memory requirements listed on our PaperCut System Requirements page.
We also suspect systems with excessive amounts of memory could see this problem as well if the Java runtime tries to allocate a large block of memory for PaperCut, but this is not available in a single contiguous block. To resolve this problem, it’s possible to configure the exact amount of memory that PaperCut can use by following the instructions in our article, Managing the amount of memory used by PaperCut NG/MF.
PaperCut fails to start on SUSE Linux
We have some reports that even though Yast (Yast2 etc) shows that PaperCut’s Application Server is set up correctly at the correct run levels, the service fails to start. If you invoke it manually via the /etc/init.d/papercut start command, it starts correctly.
In these cases it was reported that changes needed to be made to the SUSE program AppArmor. The exact changes required are unknown at this time.
Starting in SUSE Linux Enterprise 12, you can now use systemd to start/stop or check the status of the PaperCut Application Server service. To check the status, simply run the following command: systemctl status pc-app-server.service.
If the service isn’t running correctly, you should see some telling error messages. If no problems are showing with the service, you should a message like this:
pc-app-server.service - papercut Application Server
Loaded: loaded (/etc/systemd/system/pc-app-server.service; enabled; vendor preset: enabled)
Active: active (running) since Thu 2020-04-02 11:58:48 PDT; 22min ago
Main PID: 785 (app-server)
Tasks: 84 (limit: 2317)
...