Here is the follow-up:

With ClamD working properly and no

content control to worry about, 1313 messages of about 10kb to active addresses on

our main mailing list passed through the core process in just under 8

minutes. The list is still running as a VERP list and this is with ClamWall also scanning all outgoing mail. This is running on a 2.8Ghz P4 with 1Gb RAM, running on Server

2003 SP 2.

Very impressive indeed. Thank you, David 

I have upgraded to M32/4.51 - apart from moving from F-Secure to

using Clamwall and implementing Graywall (both with supplied defaults),

nothing has changed.

We have a weekly newsletter mailed to about 1600 recipients on a VERP enabled list.

On the previous version of Mercury, the list would pass through

the "Core Process" windows within about 5 mintes and all be handed to

MercuryE which would then take over.Other messages would then start

passing through the Core Process again and our outgoing traffic (as per

MRTG) would peak for the next hour or so as newsletters were delivered.

Since

upgrading, the mailing list messages pass through the Core window MUCH

slower. Almost 2 hours later they are still the only traffic I see there. Even

though other messages have been delivered and accepted by MercuryS,

they are yet to be processed by the Core and delivered.

FWIW, my

MercuryE is still set to do a maximum of 10 delivery threads at a time,

but I see no more than 2 at any moment. MRTG tells me that my outgoing

traffic is barely going above 5%. Am I missing something?

I have

checked the SYSTEM log and find that on average, the newsletter of

about 4630 bytes is being delivered to 5 addresses per minute. 

Any ideas?

Something is *very* wrong. The queuing mechanism has been totally overhauled in v4.51 (it's probably the biggest and most important change in the whole release). Queue traversal is now at least 100 times faster than in previous versions, locking issues no longer exist, and almost all queue access operations should be at least an order of magnitude faster.

I have no idea at all why you are seeing the utter opposite of what I have seen repeatably in testing here, and I have to say it's a little distressing. I put *months* into that overhaul, and it has been so hugely tested that I do not believe it can inherently go wrong.

I'm afraid I don't have much in the way of helpful suggestions (other than the standard ones of making sure that no external process is interfering with Mercury's access to its queue files). If you encounter a solution, please share it here.

Cheers!

-- David --

Hi!

 Try to disable Clamwall.

 
Regards
Jyrki Aarni

Problem and Resolution # 1 

According to the Clamwall

log, I did not have ClamD set up properly. As you can see, it took more

than a second for Clamwall to timeout when ClamD failder to

respond. 

>  20070620 110846.113 MG00057B From: verp.bWFya0Bkb29kbGVncmFwaGljcy5jby56YQ==.106@server.treverton.co.za

>  20070620 110846.113 MG00057B To: mark@somewhere.co.za

>C 20070620 110856.550 MG00057B ClamD version:

>C 20070620 110856.550 MG00057B ClamD started

>C 20070620 110856.550 MG00057B FreshClam started

>E 20070620 110857.629 MG00057B ClamD not responding!

>_ 20070620 110857.644 MG00057B Done! (11544)

 I fixed

this by downloading and installing the full clamav-090-3.exe

installation package. I had previously just downloaded the ClamD.exe

andFreshClam.exe files that appeared to promise full functionality

without a much larger installation routine and related files. 

This was my major mistake - with ClamD now running properly and

scanning is significantly faster. With Lucas's Clamwall, Graywall and

SpamHalter now running properly, my bandwidth consumption is much

improved, processing time way down and users much happier.

Problem and Resolution # 2 

I also had a rule that

scanned all outgoing messages for inappropriate words and phrases. For

quite innocent reasons, a phrase was picked up in the newsletter, which

triggered each copy of the message to be forwarded to the special

mailbox set up for futher investigation. When I sent the message to the

forum I was not aware of this issue, as the forwarded messages were

still futher behind in the queue.

I have now put the sender of the mailing list onto a whitelist

for that rule, so that it is never triggered again under those

circumstances.

In conclusion 

I did not intend to lay

blame for the slow processing at David's feet, and if that was implied,

I humbly apologize. My post to the forum was an appeal for advice and

that I got. I can now see a significant improvement in speed over the

previous version. Well done. Unfortunately the school is now closing

for the winter vacation (no teachers or pupils to interrupt the IT

maintenance work!) and it will be a month before I can report on the

next mailing.