Community Discussions and Support
Mercury/32 v4.72 crashing several times a day

It's basically the same procedure again. Start by checking the folder with mbxmaint. If it reports an error try repairing it. If this should fail, use the other tool I linked to extract the messages, and try creating the folder again from scratch.

/Rolf 

<p>It's basically the same procedure again. Start by checking the folder with mbxmaint. If it reports an error try repairing it. If this should fail, use the other tool I linked to extract the messages, and try creating the folder again from scratch.</p><p>/Rolf </p>

Hi all. We have a Mercury v4.72 instance that was running pretty well on Windows 7 Professional up until recently when it started crashing several times a day, some times right after starting up, other times lasting for several hours before crashing. We've tried all the usual stuff -- clearing the temp folders and caches, setting compatibility modes, reinstalling the software, and a few other strategies. We thought to post this message in case others have already seen and/or resolved this issue. Checking the Windows 7 Reliability Monitor, every one of the crashes have the same basic information:

mercury.exe

Problem

Stopped Working

Description
Faulting application name: mercury.exe, version: 4.7.2.0, time stamp: 0x00000000
Faulting module name: mercury.exe, version: 4.7.2.0, time stamp: 0x00000000
Exception code: 0xc0000005
Fault offset: 0x000592cf
Faulting process id: 0x1528
Faulting application start time: 0x01cb4f450e8f751a
Faulting application path: C:\mercury\mercury.exe
Faulting module path: C:\mercury\mercury.exe
Report Id: 64517119-bbc0-11df-8d53-90fba68727e5

For the 30 or so failures during the past seven days, the application name, module name, exception code and fault offset are always the same. The other data -- the process id, start time and report id -- are of course different.

Judging from the same exception code (0xc0000005 - protection fault) and the same app and offset, there is one specific issue within Mercury that is responsible for all of these crashes (fault offset: 0x000592cf). If anyone knows what this is and how to fix it, including and updates or workarounds, we would be greatly appreciative.

On a related note, we have another Windows process that would restart Mercury on a crash except that the Windows 7 "Your program has stopped responding" box always appears and stops Mercury from terminating and therefore prevents that process from noticing that Mercury isn't running anymore and launching a new instance. We know not having the crash to begin with is better (see above), but if others have figured out a way to have Mercury terminate completely under Windows 7 on a crash, that would be a good automatic recovery. The recovery process used to work on Windows XP without a problem, but now we have to spend time to constantly monitor Mercury for crashes and then, when they occur, manually close the "stopped responding" boxes and then restart Mercury.

Thank you all in advance for your enlightening comments. 

<p>Hi all. We have a Mercury v4.72 instance that was running pretty well on Windows 7 Professional up until recently when it started crashing several times a day, some times right after starting up, other times lasting for several hours before crashing. We've tried all the usual stuff -- clearing the temp folders and caches, setting compatibility modes, reinstalling the software, and a few other strategies. We thought to post this message in case others have already seen and/or resolved this issue. Checking the Windows 7 Reliability Monitor, every one of the crashes have the same basic information:</p><p>mercury.exe</p><p>Problem</p><p>Stopped Working</p><p>Description Faulting application name: mercury.exe, version: 4.7.2.0, time stamp: 0x00000000 Faulting module name: mercury.exe, version: 4.7.2.0, time stamp: 0x00000000 Exception code: 0xc0000005 Fault offset: 0x000592cf Faulting process id: 0x1528 Faulting application start time: 0x01cb4f450e8f751a Faulting application path: C:\mercury\mercury.exe Faulting module path: C:\mercury\mercury.exe Report Id: 64517119-bbc0-11df-8d53-90fba68727e5</p><p>For the 30 or so failures during the past seven days, the application name, module name, exception code and fault offset are always the same. The other data -- the process id, start time and report id -- are of course different.</p><p>Judging from the same exception code (0xc0000005 - protection fault) and the same app and offset, there is one specific issue within Mercury that is responsible for all of these crashes (fault offset: 0x000592cf). If anyone knows what this is and how to fix it, including and updates or workarounds, we would be greatly appreciative.</p><p>On a related note, we have another Windows process that would restart Mercury on a crash except that the Windows 7 "Your program has stopped responding" box always appears and stops Mercury from terminating and therefore prevents that process from noticing that Mercury isn't running anymore and launching a new instance. We know not having the crash to begin with is better (see above), but if others have figured out a way to have Mercury terminate completely under Windows 7 on a crash, that would be a good automatic recovery. The recovery process used to work on Windows XP without a problem, but now we have to spend time to constantly monitor Mercury for crashes and then, when they occur, manually close the "stopped responding" boxes and then restart Mercury.</p><p>Thank you all in advance for your enlightening comments.  </p>

If Mercury was working before on this computer there has presumably been some change recently that is creating the instability. Start by checking any software that might interfere with memory access or disk access, such as various anti-virus and anti-malware programs. Run chkdsk an verify that the disk is error free. Deactivate daemons and filtering rules in Mercury. If you have POP3 or IMAP access to Mercury check if there are any new client programs (or program versions) that have been introduced recently. Check Mercury logs and see if there is some reoccurring event that happens before each crash.

/Rolf

<p>If Mercury was working before on this computer there has presumably been some change recently that is creating the instability. Start by checking any software that might interfere with memory access or disk access, such as various anti-virus and anti-malware programs. Run chkdsk an verify that the disk is error free. Deactivate daemons and filtering rules in Mercury. If you have POP3 or IMAP access to Mercury check if there are any new client programs (or program versions) that have been introduced recently. Check Mercury logs and see if there is some reoccurring event that happens before each crash. </p><p>/Rolf </p>

Hi Rolf,

Thank you for those ideas and also for your willingness to help.  We really and greatly appreciate your quick response.

A couple of your ideas are good ones and ones we hadn't thought of before, so we are "on them," especially the logging. A couple of the ideas may not be terribly practical in a production system (e.g. turning off all filtering rules), so we'll have to work around those to some extent. (Perhaps we can run in a limited mode late at night.) I'll have another report in a day or so. In the meantime, let me list some additional information prompted by your comments:

1. No new applications loaded except for Window 7 critical updates. The Windows 7 box is a fairly lean and dedicated WAMP box running only Apache 2.2, MySQL 5.1, PHP v5 (Wampserver) and some simply FTP server.  Except for Windows 7 Firewall, there is no anti-virus or anti-malware software.

2. chkdsk /f was perfectly clean.

3. Deactivating the one daemon had no effect (Mercury still crashes).

4. New POP3 or IMAP client access -- are you talking about what email client software users are using? How would we know?

5. Mercury logs -- we'll check. Unfortunately, the crashes aren't part of the log so we'll have to correlate the Mercury logs with the report from Window's Reliability Monitor.

6. Mercury modules -- we are using S, P, E and I.

We have tried varying some of the multitude of options within Mercury and its modules, some of them are difficult to change as they interfere unfortunately with proper operation, but we'll try. From a software engineering POV, is there a way to figure out at least the general area within Mercury/32 v4.72 that is crashing based on the compiled fault offset address of 0x000592cf (decimal 365,263)? Knowing that would go a long way to help narrow the extremely broad range of possibilities. We would appreciate any more comments that advance our ability to diagnose this issue.

/Harry

<p>Hi Rolf,</p><p>Thank you for those ideas and also for your willingness to help.  We really and greatly appreciate your quick response. </p><p>A couple of your ideas are good ones and ones we hadn't thought of before, so we are "on them," especially the logging. A couple of the ideas may not be terribly practical in a production system (e.g. turning off all filtering rules), so we'll have to work around those to some extent. (Perhaps we can run in a limited mode late at night.) I'll have another report in a day or so. In the meantime, let me list some additional information prompted by your comments:</p><p>1. No new applications loaded except for Window 7 critical updates. The Windows 7 box is a fairly lean and dedicated WAMP box running only Apache 2.2, MySQL 5.1, PHP v5 (Wampserver) and some simply FTP server.  Except for Windows 7 Firewall, there is no anti-virus or anti-malware software. </p><p>2. chkdsk /f was perfectly clean.</p><p>3. Deactivating the one daemon had no effect (Mercury still crashes).</p><p>4. New POP3 or IMAP client access -- are you talking about what email client software users are using? How would we know? </p><p>5. Mercury logs -- we'll check. Unfortunately, the crashes aren't part of the log so we'll have to correlate the Mercury logs with the report from Window's Reliability Monitor.</p><p>6. Mercury modules -- we are using S, P, E and I. </p><p>We have tried varying some of the multitude of options within Mercury and its modules, some of them are difficult to change as they interfere unfortunately with proper operation, but we'll try. From a software engineering POV, is there a way to figure out at least the general area within Mercury/32 v4.72 that is crashing based on the compiled fault offset address of 0x000592cf (decimal 365,263)? Knowing that would go a long way to help narrow the extremely broad range of possibilities. We would appreciate any more comments that advance our ability to diagnose this issue. </p><p>/Harry </p>

It can be a challenge to track down what's causing crashes like this, but with a bit of luck you might find a clue in the log files. If there hasn't been any changes in filtering rules around the time the crashes started to happen then that's not a likely culprit. If there has been changes, you could try deactivating one rule at the time and see if it makes a difference.

Same goes for different configuration settings in Mercury: unless they were changed recently they are probably not to blame.

I doubt very much that we can get any hints from the offset address, unfortunately. If it had been an error message generated by the program itself it would have been much more useful.

If MercuryP and MercuryI logs indicate that crashes occur right after certain users have logged in it would be good if you could find out if perhaps a specific mail client (regular program or mobile device) was used.

/Rolf

<p>It can be a challenge to track down what's causing crashes like this, but with a bit of luck you might find a clue in the log files. If there hasn't been any changes in filtering rules around the time the crashes started to happen then that's not a likely culprit. If there has been changes, you could try deactivating one rule at the time and see if it makes a difference. </p><p>Same goes for different configuration settings in Mercury: unless they were changed recently they are probably not to blame.</p><p>I doubt very much that we can get any hints from the offset address, unfortunately. If it had been an error message generated by the program itself it would have been much more useful.</p><p>If MercuryP and MercuryI logs indicate that crashes occur right after certain users have logged in it would be good if you could find out if perhaps a specific mail client (regular program or mobile device) was used.</p><p>/Rolf</p>

Update Oct. 5, 2010

Mercury/32 v4.72 still crashing several times per day.  We've done everything Rolf suggested to try and resolve the situation and more.  Here's a list:

  1. clearing queues and scratch
  2. turning off the daemons
  3. turning off spam control (in SMTP Server)
  4. changing parameters
  5. turning off POP3
  6. turning off IMAP
  7. changing filtering rules
  8. changing active domains (we have a few)
  9. backing out Windows 7 updates (that took a while)
  10. changing compatibility options
  11. changing UAC parameters
  12. turning on all logging
  13. looking at crash entries, attempting to correlate to the log entries
  14. some other things we tried but forgotten about


We are hardware and software engineers, so we know a thing or two about tracking down problems, but this one is tough.  It seems pretty random, crashing Mercury from within seconds after launch to over a couple of hours later.  Short of moving to a non-Windows 7 box, which would be a shame since that it the only Microsoft OS worth its salt these days (and can be purchased).  One possibility is to try an earlier version of Mercury, say 4.5x, although of course there would be guarantee that the problem would go away and there would be other issues with which to content.  Does anyone have any experience with that on Windows 7? 

We have been running a number of Windbg sessions with Windbg attached to the mercury.exe process and setting a breakpoint on the exception.  When Mercury crashes, WinDbg shows the offending instruction which seems to be an indirect jump into a non-existent portion of memory, hence the exception.  Interestingly, when we change the Mercury instance's memory layout by, say, turning off on of the modules or turning off the daemons, the exception changes to a different memory location and instruction type.  For example, when we turned off daemons so that their DLLs didn't load, the exception was then caused by a data read from a non-existent memory location.

We'll continue to diagnose our problem and report back any additional findings.  Should any of our notes trigger any thoughts, please feel free to reply in this thread.  Thank you in advance!

<p>Update Oct. 5, 2010 </p><p>Mercury/32 v4.72 still crashing several times per day.  We've done everything Rolf suggested to try and resolve the situation and more.  Here's a list: </p><ol><li>clearing queues and scratch</li><li>turning off the daemons</li><li>turning off spam control (in SMTP Server) </li><li>changing parameters</li><li>turning off POP3</li><li>turning off IMAP</li><li>changing filtering rules</li><li>changing active domains (we have a few)</li><li>backing out Windows 7 updates (that took a while)</li><li>changing compatibility options</li><li>changing UAC parameters</li><li>turning on all logging</li><li>looking at crash entries, attempting to correlate to the log entries</li><li>some other things we tried but forgotten about</li></ol><p> We are hardware and software engineers, so we know a thing or two about tracking down problems, but this one is tough.  It seems pretty random, crashing Mercury from within seconds after launch to over a couple of hours later.  Short of moving to a non-Windows 7 box, which would be a shame since that it the only Microsoft OS worth its salt these days (and can be purchased).  One possibility is to try an earlier version of Mercury, say 4.5x, although of course there would be guarantee that the problem would go away and there would be other issues with which to content.  Does anyone have any experience with that on Windows 7?  </p><p>We have been running a number of Windbg sessions with Windbg attached to the mercury.exe process and setting a breakpoint on the exception.  When Mercury crashes, WinDbg shows the offending instruction which seems to be an indirect jump into a non-existent portion of memory, hence the exception.  Interestingly, when we change the Mercury instance's memory layout by, say, turning off on of the modules or turning off the daemons, the exception changes to a different memory location and instruction type.  For example, when we turned off daemons so that their DLLs didn't load, the exception was then caused by a data read from a non-existent memory location.</p><p>We'll continue to diagnose our problem and report back any additional findings.  Should any of our notes trigger any thoughts, please feel free to reply in this thread.  Thank you in advance! </p>

Until you can find some common trait it's very difficult to guess. I assume you already checked that system RAM is faultless?

One suggestion that has been brought up on the beta list in this context by David Harris is that if an IMAP folder for some reason is damaged (for instance due to crashing clients) that might cause problems for Mercury. There is a small command line tool for rebuilding mail folders if you suspect this has happened for some user.

/Rolf 

<p>Until you can find some common trait it's very difficult to guess. I assume you already checked that system RAM is faultless?</p><p>One suggestion that has been brought up on the beta list in this context by David Harris is that if an IMAP folder for some reason is damaged (for instance due to crashing clients) that might cause problems for Mercury. There is a small command line tool for rebuilding mail folders if you suspect this has happened for some user.</p><p>/Rolf </p>

Hi Rolf

 I am interested in this command line tool because I also suspect that the imap folders cause various mercury server crashes daily

Where I can find this tool?

 

regards

 

V.G.
<P>Hi Rolf</P> <P> <SPAN title="" closure_uid_9tik3a="66" te="I am interested in this little tool because I also suspect that my imap folders cause various server crashes daily mercury

" se="estoy interesado en esa pequeña herramienta porque yo tambien sospecho que las carpetas imap me provocan varias caidas del servidor mercury cada dia">I am interested in this command line tool because I also suspect that the imap folders cause various mercury server crashes daily </SPAN><SPAN style="BACKGROUND-COLOR: #e6ecf9; COLOR: #000" title="" closure_uid_9tik3a="67" te="Where I can find this useful?" se="¿donde puedo encontrar esa utilidad?">Where I can find this tool?</SPAN></P> <DIV id=input_tts_button class=" tts_vertical_bt"> </DIV> <DIV id=gt-res-c class=g-unit> <DIV id=gt-res-p> <DIV id=gt-res-data> <DIV id=gt-res-content class=almost_half_cell>regards</DIV> <DIV class=almost_half_cell> </DIV> <DIV class=almost_half_cell>V.G.</DIV></DIV></DIV></DIV>

[quote user="Rolf Lindby"]

Until you can find some common trait it's very difficult to guess. I assume you already checked that system RAM is faultless?

One suggestion that has been brought up on the beta list in this context by David Harris is that if an IMAP folder for some reason is damaged (for instance due to crashing clients) that might cause problems for Mercury. There is a small command line tool for rebuilding mail folders if you suspect this has happened for some user.

/Rolf 

[/quote]

 I'm also interested in trying this tool.  Mercury had been running very well for me until I switched to using IMAP a few months ago.   Now I'm having fairly frequent crashes (once a week or so but I'm the only user -- with four email accounts).   No changes had been made to my server around the time the crashes began and I haven't switched email clients (Outlook, SquirrelMail and the iPhone email client).  I've done all the normal checks / tests to my server so I'm fairly sure it's clean.

[quote user="Rolf Lindby"]<p>Until you can find some common trait it's very difficult to guess. I assume you already checked that system RAM is faultless?</p><p>One suggestion that has been brought up on the beta list in this context by David Harris is that if an IMAP folder for some reason is damaged (for instance due to crashing clients) that might cause problems for Mercury. There is a small command line tool for rebuilding mail folders if you suspect this has happened for some user.</p><p>/Rolf </p><p>[/quote]</p><p> I'm also interested in trying this tool.  Mercury had been running very well for me until I switched to using IMAP a few months ago.   Now I'm having fairly frequent crashes (once a week or so but I'm the only user -- with four email accounts).   No changes had been made to my server around the time the crashes began and I haven't switched email clients (Outlook, SquirrelMail and the iPhone email client).  I've done all the normal checks / tests to my server so I'm fairly sure it's clean. </p>

  I am interested in this command line tool because I also suspect that the imap folders cause various mercury server crashes daily 

 Where I can find this tool?

Download a copy of Pegasus Mail v.4.52, the utility MBXMAINT.exe comes with it.

 

<p><span class="Apple-style-span" style="font-family: Tahoma, Arial, Helvetica; "> <span class="Apple-tab-span" style="white-space:pre"> </span><span>I am interested in this command line tool because I also suspect that the imap folders cause various mercury server crashes daily  </span><span style="background-color: rgb(230, 236, 249); color: rgb(0, 0, 0); "><span class="Apple-tab-span" style="white-space:pre"> </span>Where I can find this tool?</span></span></p><p>Download a copy of Pegasus Mail v.4.52, the utility MBXMAINT.exe comes with it.</p><p> </p>

Hi Guys,

It has been a week, and we think we have resolved mostly our "Mercury crashing several times a day" issue.  It was either a corrupt mail folder or a mail folder that had over 3,000 emails. We disassembled the code around the fault offset a bit and determined that there was some kind of index at play which would occasionally exceed the allocated physical memory. We conjectured that this might have to do with some array-based or link-based memory object, something to do with the number of folders, the number of emails, etc. Following this line of thinking, and among other things we tried to ameliorate the situation, we copied the content of a mail folder to an archive folder, and the frequency of the crash dropped from around ten time per day to once per week (so far). That is at least acceptable if short of ideal. 

We didn't get around to trying Rolf's folder fixup tool. Our copying may have had the same effect though as the larger folder was the "inbox" and the copy process seemed to reset it to a very small size. Any corruption that might have been in it could have been inherently addressed by this action. The archive folder has just as many emails but having our clients access it does not seem to crash the mail server. We wonder if it is possible that the increased activity on the Inbox might have played a role in the crashes?  Also, we had the thought that there may be different code within Mercury for a mail account's inbox and its other folders in which case corruptions and crashes on the Inbox might be independent from any of those in the other folders.

While the number of crashed have dropped 98%, we are keeping a watchful eye and clearing out large volume inboxes on a regular basis.

Thank you all for your inputs, and especially thank you, Rolf,

Harry

 

<p>Hi Guys,</p><p>It has been a week, and we think we have resolved mostly our "Mercury crashing several times a day" issue.  It was either a corrupt mail folder or a mail folder that had over 3,000 emails. We disassembled the code around the fault offset a bit and determined that there was some kind of index at play which would occasionally exceed the allocated physical memory. We conjectured that this might have to do with some array-based or link-based memory object, something to do with the number of folders, the number of emails, etc. Following this line of thinking, and among other things we tried to ameliorate the situation, we copied the content of a mail folder to an archive folder, and the frequency of the crash dropped from around ten time per day to once per week (so far). That is at least acceptable if short of ideal. </p><p>We didn't get around to trying Rolf's folder fixup tool. Our copying may have had the same effect though as the larger folder was the "inbox" and the copy process seemed to reset it to a very small size. Any corruption that might have been in it could have been inherently addressed by this action. The archive folder has just as many emails but having our clients access it does not seem to crash the mail server. We wonder if it is possible that the increased activity on the Inbox might have played a role in the crashes?  Also, we had the thought that there may be different code within Mercury for a mail account's inbox and its other folders in which case corruptions and crashes on the Inbox might be independent from any of those in the other folders.</p><p>While the number of crashed have dropped 98%, we are keeping a watchful eye and clearing out large volume inboxes on a regular basis.</p><p>Thank you all for your inputs, and especially thank you, Rolf, </p><p>Harry</p><p> </p>

Hi Rolf

 I've tried the tool and, if anything, things are worse now.  Through some messing around I think I've determined that the issue is with the INBOX.Trash folder for one of my users.  Every time a client connects to that folder Mercury crashes.  I'd like to remove the messages from that folder but I'm not sure how to go about it.  I see in the HIERARCH.PM file that the folder in question is FOL15AD1.  Can I just go ahead and delete the FOL15AD1.PMM file?  Are all the messages in the .PMM file copies of the individual *.CNM files?  I don't quite understand the structure here and I'd prefer not to make things worse!

 Is there a document outlining the mail store structure anywhere?

 Thanks

John

<p>Hi Rolf</p><p> I've tried the tool and, if anything, things are worse now.  Through some messing around I think I've determined that the issue is with the INBOX.Trash folder for one of my users.  Every time a client connects to that folder Mercury crashes.  I'd like to remove the messages from that folder but I'm not sure how to go about it.  I see in the HIERARCH.PM file that the folder in question is FOL15AD1.  Can I just go ahead and delete the FOL15AD1.PMM file?  Are all the messages in the .PMM file copies of the individual *.CNM files?  I don't quite understand the structure here and I'd prefer not to make things worse!</p><p> Is there a document outlining the mail store structure anywhere? </p><p> Thanks</p><p>John </p>

If the folder can't be repaired with MBXMAINT you can use this tool to split it up in individual message files:

http://community.pmail.com/files/folders/utils/entry3962.aspx

If you want to delete the messages altogether just delete or rename the FOL15AD1.* files, and remove the reference to the folder from HIERARCH.PM.

/Rolf

<p>If the folder can't be repaired with MBXMAINT you can use this tool to split it up in individual message files:</p><p>http://community.pmail.com/files/folders/utils/entry3962.aspx</p><p>If you want to delete the messages altogether just delete or rename the FOL15AD1.* files, and remove the reference to the folder from HIERARCH.PM.</p><p>/Rolf </p>

Hi Rolf

I deleted the INBOX.Trash directory / file and removed the references from the HIERARCH.PM file for both my main users and I was able to connect to the server again using IMAP.  But, now I'm having crashes when trying to open a different folder.  This folder doesn't contain trash so I don't want to be deleting it...  Something is definitely amiss with the IMAP functionality here.

 

John

<p>Hi Rolf</p><p>I deleted the INBOX.Trash directory / file and removed the references from the HIERARCH.PM file for both my main users and I was able to connect to the server again using IMAP.  But, now I'm having crashes when trying to open a different folder.  This folder doesn't contain trash so I don't want to be deleting it...  Something is definitely amiss with the IMAP functionality here.</p><p>  </p><p>John </p>
live preview
enter atleast 10 characters
WARNING: You mentioned %MENTIONS%, but they cannot see this message and will not be notified
Saving...
Saved
With selected deselect posts show selected posts
All posts under this topic will be deleted ?
Cancel
Delete
Pending draft ... Click to resume editing
Discard draft