Rolf,

My problem appears to be solved.  Thanks for your suggestion to use Process Monitor.  That tool provided the clue needed to fix this.  

 For the benefit of others that may encounter a similar problem I am stating the cause and solution below.

Problem:

Temporary files were being orphaned in the SCRATCH directory at infrequent intervals.  On an even less frequent interval the result file returned from my policy program would not be included in the Policy Exception Notification email generated by Mercury in response to a non-zero return code.

Policy Program Design:

The policy program was using COUT to write to STDOUT.  The program issued a standard C call similar to this "freopen_s( &stream, argv[2], "w", stdout );" to reassign stdout to the file being returned to Mercury.  This mechanism worked about 99% of time.  When it failed it was because the return file was still busy even though the policy program had terminated.  It seems that there is a delay in Windows wrapping up the stdout file and if the delay was long enough Mercury would attempt to access the return file and be denied.  The end result was that the return file would be orphaned in the SCRATCH directory.

Policy Program Modification:

To fix the problem I modified the policy program to not use "freopen_s" but instead to use an ofstream definition of the return file and to redirect COUT to that file with this statement "cout.rdbuf(return_ot.rdbuf());"  Thus, the normal process of closing the return file prior to program exit ensured that the buffers were flushed and the file was no longer busy when Mercury resumed processing. No changes to any COUT statements had to be made.

I have written a policy program and it works about 99% of the time.  I have made numerous changes trying to isolate the problem so now I seek help from other users.

Background:  Mercury/32 v4.62, Vista with all fixes and SP applied, VS2005 C++.

Program receives an ~A file and uses a specified Result file "policyresults.tmp" but no sentinel file.  The specification of a Result file name is a partial fix to the problem which is more prevalent when allowing Mercury to create a temporary Result file. Following is the invocation command line.

F:\FILES\MERCURY\MAIL\emailspamdetector.exe ~A ~R F:\Files\Mercury\MAIL\whitelist.csv F:\Files\Mercury\MAIL\policy.log

First, a work around but not my current problem.  The program only examines headers but I had to specify ~A instead of ~X in order to have the entire message attached to the Policy Exception message returned by Mercury when the policy program exits with other than a 0 return code.  I didn't pick up on this when reading the documentation but my workaround is to stop processing headers when I encounter the first blank line and that works fine.

The problem is:

1.  When I originally had Mercury returning a random named temporary file for the Result that file would sometimes be left in the SCRATCH folder instead of being deleted.  That would happen at random times but about 2 or 3 times a day, usually when the computer was busy doing other background work like file transfers, backups, and such.  However, it also would happen when the computer was idle.

2.  I suspected a timing problem where Mercury was perhaps given an error when trying to delete the file and just ignored it leaving the file.  I tried numerous techniques to try to see if it was timing problem.  My PC is a low cost Gateway dual core with 1gb.  I modified my program to have a 700 millisecond delay after closing all files prior to exit 0 or exit 1.  The problem occurs with either kind of exit.  The 700 millisecond delay before program exit did not seem to make any difference. I should point out that except for the .tmp file being orphaned the rest of the process works flawlessly.

3.  I changed to a specified Result file and I have not seen it orphaned although if it is I assume it will be reused on a subsequent message.  Thus, using the specified Result file seems to be a work around for the orphaned .tmp files and I'm fine with that.

4.  The remaining problem is that the Result file will sometimes have none of the lines that I wrote to the file.  In other words, Mercury gives the standard Policy Exception Notification email but no Result file contents.  This happens about once every few days.  I get about 10 - 20 policy exceptions per day out of a total email volume of about 50 - 100 messages per day.

What my program does is it reads the ~A file examining TO:, FROM:, and ENVELOPE-TO: headers in order to catch spam that has certain characteristics.  For example, mail where the FROM: and the TO: are the same is, in my case, likely to be spam.  I also check the FROM: as being in a white list whose file name is passed in the command line show above.  I also append everything written to the Result file to a policy log file, also specified in the command line show above, so that I have a continuous history of all policy processing.  I orginally did not have the policy log file but I added that when I was having the problem with orphaned .tmp files in the SCRATCH directory.  

 In summary, I suspect that the problem is Mercury being unable to access the Result file, either for purposes of adding the content to the Policy Exception message or for deleting the file.  I have read that Mercury has been slowly adding multiple thread processing so perhaps this is related to that?  Although I have been using Mercury for years the policy process is new and only used with 4.62.  

Any thoughts are appreciated.  I see that Mercury 5 is soon to be released and will include more thread processes.  I hope to head off any problems with that.

 Thanks.

It would seem that there is something that occasionally blocks access for Mercury to the result file.It's presumably not thread related, Mercury is already processing that message and just waiting for your policy program to finish.

Make sure you have no anti-virus program or other file protection application accessing the Mercury directories.

/Rolf
 

[quote user="Rolf Lindby"]

It would seem that there is something that occasionally blocks access for Mercury to the result file.It's presumably not thread related, Mercury is already processing that message and just waiting for your policy program to finish.

Make sure you have no anti-virus program or other file protection application accessing the Mercury directories.

/Rolf
 

[/quote]

 When reading through old posts I saw that suggested and have already done that.  It does not seem to have made any difference.  Is there some way that you know of to monitor file access denials, either in Windows or Mercury?

You will need some rather low-level tool to do this, such as Process Monitor from Sysinternals/Microsoft (http://technet.microsoft.com/en-us/sysinternals/bb896645.aspx). It's a free tool, but there might be quite a lot of data to analyze.

/Rolf 

Thanks, Rolf.  I have the various Sysinternals tools and I'll see if I can figure out a way to capture the data and analyze it.

Bruce

Rolf,

Last night I started Process Monitor and it seems to indicate that there is a problem with Mercury and my Policy program executing concurrently.  Here is one line from the monitor.  Note the SHARING VIOLATION and that is Mercury trying to access the Result file while it is still in use in my Policy program.

719    22:11:35.8965823    mercury.exe    5256    IRP_MJ_CREATE    C:\MERCURY\policyresults.tmp    SHARING VIOLATION    Desired Access: Read Attributes, Delete, Disposition: Open, Options: Non-Directory File, Open Reparse Point, Attributes: n/a, ShareMode: Read, Write, Delete, AllocationSize: n/a    0.0000288    File System        1824    1884 

I will set up a better filtering combination to more clearly show the flow through the processes.  One thing that I'm doing that perhaps is causing this is that I am doing a redirect on STDOUT in my program.  The Result file is the target of the redirect.  Perhaps there is some sort of delay in Windows releasing the file used for STDOUT and that is causing Mercury to go after the file while it is still busy?  Just guessing obviously.

Another interesting detail is that the Result file is being created subordinate to c:\mercury (the install directory) rather than in the SCRATCH directory.  Not a problem but a surprise.

And the final interesting detail is that Mercury creates two temporary files in the SCRATCH directory, one being the ~X file and the other being unused except for being created.  Mercury is creating a .tmp file for use as a Result file and then realizes it does not need it?  Just a guess.

 Rolf, thanks for the great idea of using Process Monitor to try to pin this problem down.  I used it a couple of years ago as a learning tool but forgot all about it.  The version I had was from 2007 and I have downloaded the latest version.

OK good, then you know the reason for the problem. The trick will be to make sure the file is really closed before ending the policy program.

Try entering the name of the result file with a full path to make sure it's created in the SCRATCH directory. I have no theory about that other .tmp file, though...

/Rolf