Community Discussions and Support
Suppressing deliberately duplicated messages

Good to hear! We have very well behaved users that won't attempt to paste a zillion copies of the same address into the To: field, but I've been running the modified daemon in production here anyway for the past days. Looks like it's quite stable.

It was mentioned in the Beta group discussion that Outlook only delivers a single copy of a message in a similar situation, so perhaps this should be considered for Pegasus as well. 

 /Rolf 

<p>Good to hear! We have very well behaved users that won't attempt to paste a zillion copies of the same address into the To: field, but I've been running the modified daemon in production here anyway for the past days. Looks like it's quite stable.</p><p>It was mentioned in the Beta group discussion that Outlook only delivers a single copy of a message in a similar situation, so perhaps this should be considered for Pegasus as well. </p><p> /Rolf </p>

Don't know if anyone else suffers this problem with educational users (little tykes in the Year 7 to Year 11 age group), but the cherubs who use my system love to copy-and-paste a single target address in the "To:" box, so they send thousands of copies of the same insult to some unsuspecting victim. The victim then usually does the same thing back again. Result - mailboxes bloated with duplicate (multiplicate?) junk. Anyone have any useful ideas about intercepting this in Mercury? Like scanning the address list for duplicates and deleting the excess entries? Could also be useful to suppress accidental duplication (I can't see any legitimate reason why you would want to send duplicates to the same target address). I can see how I might script a duplicate detector and just delete the whole message, but that seems a bit harsh for accidents.

<P>Don't know if anyone else suffers this problem with educational users (little tykes in the Year 7 to Year 11 age group), but the cherubs who use my system love to copy-and-paste a single target address in the "To:" box, so they send thousands of copies of the same insult to some unsuspecting victim. The victim then usually does the same thing back again. Result - mailboxes bloated with duplicate (multiplicate?) junk. Anyone have any useful ideas about intercepting this in Mercury? Like scanning the address list for duplicates and deleting the excess entries? Could also be useful to suppress accidental duplication (I can't see any legitimate reason why you would want to send duplicates to the same target address). I can see how I might script a duplicate detector and just delete the whole message, but that seems a bit harsh for accidents.</P>

You've got a good'un there, I'd say!

I've just had a look through Mercury's options and I can't see anything simple. The best options I can come up with both involve you recording a list of the insults being bandied about.

You'd then either write a content control test which just marked them as spam (or deleted them), or make a filter which, whenever it found one of the insults in a message, adds the sender to a mailing list and e-mails you. If you test every incoming message against this mailing list, you could then do something with that sender AND drop the message.

I'd suggest that you send them a mail saying that none of their e-mails to anybody are being delivered anymore (maybe send this three or four times for each message they send!) and let them be automatically removed from the list if they send an admission of their behaviour and an apology to another mail address - the headmaster!

<P>You've got a good'un there, I'd say!</P><P>I've just had a look through Mercury's options and I can't see anything simple. The best options I can come up with both involve you recording a list of the insults being bandied about.</P><P>You'd then either write a content control test which just marked them as spam (or deleted them), or make a filter which, whenever it found one of the insults in a message, adds the sender to a mailing list and e-mails you. If you test every incoming message against this mailing list, you could then do something with that sender AND drop the message.</P><P>I'd suggest that you send them a mail saying that none of their e-mails to anybody are being delivered anymore (maybe send this three or four times for each message they send!) and let them be automatically removed from the list if they send an admission of their behaviour and an apology to another mail address - the headmaster!</P>

Just had another think and I suppose that you could write a regular expression test to limit the number of internal addresses that a message could have.

If you're really talking about endless copy-pasting into the To: field, I guess this would at least limit the damage. I mean, if you stated no more than 20 internal recipients before deleting the message, that would at least allow staff to continue work.

I'm really intrigued now to hear about any progress you make with this issue. Please post back.

<P>Just had another think and I suppose that you could write a regular expression test to limit the number of internal addresses that a message could have.</P><P>If you're really talking about endless copy-pasting into the To: field, I guess this would at least limit the damage. I mean, if you stated no more than 20 internal recipients before deleting the message, that would at least allow staff to continue work.</P><P>I'm really intrigued now to hear about any progress you make with this issue. Please post back.</P>

but the cherubs who use my system love to copy-and-paste a single

target address in the "To:" box, so they send thousands of copies of

the same insult to some unsuspecting victim

Could you be more specific as to what they are doing?   Are they sending one message with 1000 addresses or multiple email messages with the same body.  The one with 1000 identical addresses in the same message it a lot easier to spot that the 1000 separate messages to the same user.

 

<blockquote>but the cherubs who use my system love to copy-and-paste a single target address in the "To:" box, so they send thousands of copies of the same insult to some unsuspecting victim</blockquote><p>Could you be more specific as to what they are doing?   Are they sending one message with 1000 addresses or multiple email messages with the same body.  The one with 1000 identical addresses in the same message it a lot easier to spot that the 1000 separate messages to the same user.</p><p> </p>

We brought the matter up in the Beta group. Thomas Stephenson, who has had similar experiences, suggests to go to the root of the problem:

I supported systems for middle school (grades 6-8) and high school with Pegasus Mail and Mercury and had problems like this. We reduced the problem to a manageable level by creating terms and conditions of use for the e-mail system and these were signed by both the parents and the students before they were given an account. They violate the T&C and their account was suspended. The kids really did not want to be the only ones without e-mail access and so this worked quite well.
We also discussed various ways to counter this with rule sets, but I ended up modifying an existing anti-spam daemon to reject duplicate recipients during the SMTP session. I'll send you a message with download details to let you try it out if you like to. In the long run Thomas' idea might be better, though, as the technical solution only will block this particular mischief, not any other interesting stuff all those future hackers of yours may think of.
/Rolf
<p>We brought the matter up in the Beta group. Thomas Stephenson, who has had similar experiences, suggests to go to the root of the problem:</p><blockquote><div><i>I supported systems for middle school (grades 6-8) and high school with Pegasus Mail and Mercury and had problems like this. We reduced the problem to a manageable level by creating terms and conditions of use for the e-mail system and these were signed by both the parents and the students before they were given an account. They violate the T&C and their account was suspended. The kids really did not want to be the only ones without e-mail access and so this worked quite well.</i></div></blockquote><div> </div><div>We also discussed various ways to counter this with rule sets, but I ended up modifying an existing anti-spam daemon to reject duplicate recipients during the SMTP session. I'll send you a message with download details to let you try it out if you like to. In the long run Thomas' idea might be better, though, as the technical solution only will block this particular mischief, not any other interesting stuff all those future hackers of yours may think of.</div><div> </div><div>/Rolf </div>

Thomas: to clarify, I'm talking about a single message with duplicated addresses in the To: box. Evidently what they do is to enter the address once with a comma, ctrl-C it and then add more by leaning on the ctrl-V key. Yes, of course we suspend email service to culprits - it's easy, I just withdraw an AD group membership, but that's after-the-event and doesn't tidy up the trash. In fact, we have quite sophisticated IT Policy enforcement and notification, but I prefer to be pro-active if possible, especially where it reduces our workload.

 Rolf: thanks for the message and when I've got a little time, I'll set it up and see if it nails the problem. I'm very grateful for the contribution!

<P>Thomas: to clarify, I'm talking about a single message with duplicated addresses in the To: box. Evidently what they do is to enter the address once with a comma, ctrl-C it and then add more by leaning on the ctrl-V key. Yes, of course we suspend email service to culprits - it's easy, I just withdraw an AD group membership, but that's after-the-event and doesn't tidy up the trash. In fact, we have quite sophisticated IT Policy enforcement and notification, but I prefer to be pro-active if possible, especially where it reduces our workload.</P> <P> Rolf: thanks for the message and when I've got a little time, I'll set it up and see if it nails the problem. I'm very grateful for the contribution!</P>

[quote user="Rammie"]

Thomas: to clarify, I'm talking about a single message with duplicated addresses in the To: box. Evidently what they do is to enter the address once with a comma, ctrl-C it and then add more by leaning on the ctrl-V key.

Ok, that's not what I thought they were doing.  One message to multiple addresses as opposed to multiple messages to a single address.

Yes, of course we suspend email service to culprits - it's easy, I just withdraw an AD group membership, but that's after-the-event and doesn't tidy up the trash. In fact, we have quite sophisticated IT Policy enforcement and notification, but I prefer to be pro-active if possible, especially where it reduces our workload.

Granted, but since these are so easy to catch the "little devils" will simply move to the next level where they are sending multiple messages with the same address or dumping the CNM files directly into the queue or other users mail directory.  You can spend all of your time fighting a never ending battle that you can control but not win.  ;-)  Since these are a lot like spam I find something like POPFileD or SpamHalter with their Bayesian filtering are a lot better about catching and moving these off to a different location are a lot more effective.  If something simply does not work, even the spammers give up after awhile.  ;-)

 Rolf: thanks for the message and when I've got a little time, I'll set it up and see if it nails the problem. I'm very grateful for the contribution!

Rolf's daemon with catch this sort of threat.[/quote]

<blockquote>[quote user="Rammie"]<p>Thomas: to clarify, I'm talking about a single message with duplicated addresses in the To: box. Evidently what they do is to enter the address once with a comma, ctrl-C it and then add more by leaning on the ctrl-V key. </p></blockquote><p>Ok, that's not what I thought they were doing.  One message to multiple addresses as opposed to multiple messages to a single address. </p><blockquote><p>Yes, of course we suspend email service to culprits - it's easy, I just withdraw an AD group membership, but that's after-the-event and doesn't tidy up the trash. In fact, we have quite sophisticated IT Policy enforcement and notification, but I prefer to be pro-active if possible, especially where it reduces our workload.</p></blockquote><p>Granted, but since these are so easy to catch the "little devils" will simply move to the next level where they are sending multiple messages with the same address or dumping the CNM files directly into the queue or other users mail directory.  You can spend all of your time fighting a never ending battle that you can control but not win.  ;-)  Since these are a lot like spam I find something like POPFileD or SpamHalter with their Bayesian filtering are a lot better about catching and moving these off to a different location are a lot more effective.  If something simply does not work, even the spammers give up after awhile.  ;-) </p><blockquote><p> Rolf: thanks for the message and when I've got a little time, I'll set it up and see if it nails the problem. I'm very grateful for the contribution!</p></blockquote><p>Rolf's daemon with catch this sort of threat.[/quote]</p>

Ha ha! I said it was easy to withdraw their privileges, Thomas, not so easy to catch them red-handed! We capture every message sent by students in an audit-trail mailbox, where it's filtered, flagged and filed (filtered for bullying or obscenities). There can be a delay of hours or days before we detect an offence (we don't necessarily look that often because we have a big network to run as well). This is rather unsatisfying - punishments work best when meted out immediately. Hence the urge to automate. We sometimes have a blitz on offences, just so they know we're watching.

I like your thinking, Thomas, but I’m pleased to say that they haven’t discovered any back-doors to the messaging system yet. The top-level mailbox folder is a hidden-share and they only have rights to their own folder. All messages are forced to use Mercury’s routing (no direct file moves) and they cannot use the Pegasus control panel (nor most other options) to find out how we’ve set it up (or fiddle with it). No other clients are permitted. Indeed, students cannot install software of any kind. I’m not complacent, but hopefully this hog's fairly well tied.

<P class=MsoNormal style="MARGIN: 0cm 0cm 10pt"><SPAN style="FONT-SIZE: 10pt; LINE-HEIGHT: 115%; FONT-FAMILY: 'Arial','sans-serif'">Ha ha! I said it was easy to withdraw their privileges, Thomas, not so easy to catch them red-handed! We capture <U>every</U> message sent by students in an audit-trail mailbox, where it's filtered, flagged and filed (filtered for bullying or obscenities). There can be a delay of hours or days before we detect an offence (we don't necessarily look that often because we have a big network to run as well). This is rather unsatisfying - punishments work best when meted out immediately. Hence the urge to automate. We sometimes have a blitz on offences, just so they know we're watching.</SPAN></P> <P class=MsoNormal style="MARGIN: 0cm 0cm 10pt"><SPAN style="FONT-SIZE: 10pt; LINE-HEIGHT: 115%; FONT-FAMILY: 'Arial','sans-serif'">I like your thinking, Thomas, but I’m pleased to say that they haven’t discovered any back-doors to the messaging system yet. The top-level mailbox folder is a hidden-share and they only have rights to their own folder. All messages are forced to use Mercury’s routing (no direct file moves) and they cannot use the Pegasus control panel (nor most other options) to find out how we’ve set it up (or fiddle with it). No other clients are permitted. Indeed, students cannot <EM>install</EM> software of any kind. I’m not complacent, but hopefully this hog's fairly well tied.</SPAN></P>

[quote user="Rammie"]

...and they cannot use the Pegasus control panel (nor most other options) to find out how we’ve set it up (or fiddle with it).

[/quote]

I'm intrigued. Although I have no need to know, I really want to know: how have you locked out the options screen and other menu items? Is it possible to modify the menus? If so, may I be pointed at a how-to?

[quote user="Rammie"]<P class="MsoNormal" style="MARGIN: 0cm 0cm 10pt"><SPAN style="FONT-SIZE: 10pt; LINE-HEIGHT: 115%; FONT-FAMILY: 'Arial','sans-serif'">...and they cannot use the Pegasus control panel (nor most other options) to find out how we’ve set it up (or fiddle with it).</SPAN></P><P>[/quote]</P><P>I'm intrigued. Although I have no <EM>need</EM> to know, I really <EM>want</EM> to know: how have you locked out the options screen and other menu items? Is it possible to modify the menus? If so, may I be pointed at a how-to?</P>

Ha ha! I said it was easy to withdraw their privileges, Thomas, not so easy to catch them red-handed! We capture every

message sent by students in an audit-trail mailbox, where it's

filtered, flagged and filed (filtered for bullying or obscenities).

There can be a delay of hours or days before we detect an offense (we

don't necessarily look that often because we have a big network to run

as well). This is rather unsatisfying - punishments work best when

meted out immediately. Hence the urge to automate. We sometimes have a

blitz on offenses, just so they know we're watching.

I said it was easy to catch a message with a 1000 email addresses in one messages via filtering, Now I've only worked middle schools (much more difficult than high schools) with between 300-800 students and I was doing this on a volunteer basis but that kind of message was not at all that hard to handle.  It was just deleted or re-routed to the postmaster and the account suspended until the kid and their parents had a discussion with the teaching staff.  The sender got an automated message back from the postmaster saying "gotcha" though in many cases. 

Those single messages sent from external throwaway email addresses are a much tougher problem.  You can get essentially a DOS attack from almost any of these services.  The kids here in the Silicon Valley also have access to all sorts of hardware/software and many of the 11-13 year old students know how to use these tools.  These are not your normal IPOD and Mac users, they cut their eye teeth on Linux and DOS.  You have to become the BOFH to be able to handle these.  ;-)

I

like your thinking, Thomas, but I’m pleased to say that they haven’t

discovered any back-doors to the messaging system yet. The top-level

mailbox folder is a hidden-share and they only have rights to their own

folder. All messages are forced to use Mercury’s routing (no direct

file moves) and they cannot use the Pegasus control panel (nor most

other options) to find out how we’ve set it up (or fiddle with it). No

other clients are permitted. Indeed, students cannot install software of any kind. I’m not complacent, but hopefully this hog's fairly well tied.

Do the allow cell phones in the school?  If they have a cell phone there are all sorts of tools available to provide web access and contact to your SMTP server via telnet.  They can also go to their home systems and send the mail.  ;-)  The kids are a lot more comfortable and capable with the current technology than most adults and the local teachers assigned the task of teaching them technology are easily overwhelmed.  The guy at the district office providing the schools tech support is generally undermanned and overwhelmed.  It can be managed but trying to do this automatically with technology is a never ending battle and a pretty thankless task. 

Again this is why I recommend treating all of this pretty much the same way as I do spam.   The Bayesian filtering catches all the bad stuff and either deletes it or routes if off to a separate user for later analysis.  If this is done in an educational environment all of the mail can be printed out for the teacher/parent conference.  ;-)

 

 

 

<blockquote><p><span style="font-size: 10pt; line-height: 115%; font-family: 'Arial','sans-serif';">Ha ha! I said it was easy to withdraw their privileges, Thomas, not so easy to catch them red-handed! We capture <u>every</u> message sent by students in an audit-trail mailbox, where it's filtered, flagged and filed (filtered for bullying or obscenities). There can be a delay of hours or days before we detect an offense (we don't necessarily look that often because we have a big network to run as well). This is rather unsatisfying - punishments work best when meted out immediately. Hence the urge to automate. We sometimes have a blitz on offenses, just so they know we're watching.</span></p></blockquote><p>I said it was easy to catch a message with a 1000 email addresses in one messages via filtering, Now I've only worked middle schools (much more difficult than high schools) with between 300-800 students and I was doing this on a volunteer basis but that kind of message was not at all that hard to handle.  It was just deleted or re-routed to the postmaster and the account suspended until the kid and their parents had a discussion with the teaching staff.  The sender got an automated message back from the postmaster saying "gotcha" though in many cases.  </p><p>Those single messages sent from external throwaway email addresses are a much tougher problem.  You can get essentially a DOS attack from almost any of these services.  The kids here in the Silicon Valley also have access to all sorts of hardware/software and many of the 11-13 year old students know how to use these tools.  These are not your normal IPOD and Mac users, they cut their eye teeth on Linux and DOS.  You have to become the BOFH to be able to handle these.  ;-) </p><blockquote><p><span style="font-size: 10pt; line-height: 115%; font-family: 'Arial','sans-serif';">I like your thinking, Thomas, but I’m pleased to say that they haven’t discovered any back-doors to the messaging system yet. The top-level mailbox folder is a hidden-share and they only have rights to their own folder. All messages are forced to use Mercury’s routing (no direct file moves) and they cannot use the Pegasus control panel (nor most other options) to find out how we’ve set it up (or fiddle with it). No other clients are permitted. Indeed, students cannot <em>install</em> software of any kind. I’m not complacent, but hopefully this hog's fairly well tied.</span></p></blockquote><p>Do the allow cell phones in the school?  If they have a cell phone there are all sorts of tools available to provide web access and contact to your SMTP server via telnet.  They can also go to their home systems and send the mail.  ;-)  The kids are a lot more comfortable and capable with the current technology than most adults and the local teachers assigned the task of teaching them technology are easily overwhelmed.  The guy at the district office providing the schools tech support is generally undermanned and overwhelmed.  It can be managed but trying to do this automatically with technology is a never ending battle and a pretty thankless task.  </p><p>Again this is why I recommend treating all of this pretty much the same way as I do spam.   The Bayesian filtering catches all the bad stuff and either deletes it or routes if off to a separate user for later analysis.  If this is done in an educational environment all of the mail can be printed out for the teacher/parent conference.  ;-)</p><p> </p><p> </p><p> </p>

> I'm intrigued. Although I have no need to know, I really want to know:
> how have you locked out the options screen and other menu items? Is it
> possible to modify the menus? If so, may I be pointed at a how-to?

The DISABLE extension allows you to override almost any main
menu function within WinPMail.  It simply pops up a message box
with the text "This option has been disabled by the Network
Supervisor" when the user either selects the option from the menu,
or presses the shortcut key, or clicks the associatied button bar
button.

New versions of the DISABLE extension are now available in both
16-bit and 32-bit flavours at:

                    ftp://tui.lincoln.ac.nz

Filenames are DISABL16.ZIP and DISABL32.ZIP.

If you can't find it there I've got a copy i can send oyu.

<p>> I'm intrigued. Although I have no need to know, I really want to know: > how have you locked out the options screen and other menu items? Is it > possible to modify the menus? If so, may I be pointed at a how-to? The DISABLE extension allows you to override almost any main menu function within WinPMail.  It simply pops up a message box with the text "This option has been disabled by the Network Supervisor" when the user either selects the option from the menu, or presses the shortcut key, or clicks the associatied button bar button. New versions of the DISABLE extension are now available in both 16-bit and 32-bit flavours at:                     ftp://tui.lincoln.ac.nz Filenames are DISABL16.ZIP and DISABL32.ZIP.</p><p>If you can't find it there I've got a copy i can send oyu. </p>

Thanks for filling in Sharkfin on controlling Pegasus menu options, Thomas. That's exactly the tool we use. In fact, Pegasus would never have been entertained as a solution, had we not been able to do this. It's even handy in a business context, because with a shared copy of Pegasus, you may not want people messing about with some of the settings.

Thomas, I manage the ICT for a large (1800+) high school, but as of the moment, we operate a different support model to the one you're used to. I say "as of the moment", because this is going to change (Sharkfin may recognise the acronym "BSF"). Almost all support is done with local staff, but we use a consortium for our Internet access. This means external access is strictly limited and filtered, and our mail goes though a smart-host. Our wireless access is closed too (we don't divulge the WPA codes to anyone and we don't permit casual users). We accept that pupils can misuse email at home, but we cannot legislate for what they do off our premises. We just blacklist troublesome addresses along with the myiad of ICT companies trying to sell me stuff.

<P>Thanks for filling in Sharkfin on controlling Pegasus menu options, Thomas. That's exactly the tool we use. In fact, Pegasus would never have been entertained as a solution, had we not been able to do this. It's even handy in a business context, because with a shared copy of Pegasus, you may not want people messing about with some of the settings.</P> <P>Thomas, I manage the ICT for a large (1800+) high school, but as of the moment, we operate a different support model to the one you're used to. I say "as of the moment", because this is going to change (Sharkfin may recognise the acronym "BSF"). Almost all support is done with local staff, but we use a consortium for our Internet access. This means external access is strictly limited and filtered, and our mail goes though a smart-host. Our wireless access is closed too (we don't divulge the WPA codes to anyone and we don't permit casual users). We accept that pupils can misuse email at home, but we cannot legislate for what they do off our premises. We just blacklist troublesome addresses along with the myiad of ICT companies trying to sell me stuff.</P>

Rolf,

I've installed and tested your daemon and so far, it seems to be working fine. I've not noticed any other messages breaking and I've tested the duplication detection: works brilliantly! Now it just remains to let it run for a week and check the logs.

 Thanks very much indeed, Rolf, I appreciate you sharing the results of your labours. Perhaps David might consider including some similar code in a later version of Mercury, with a tick-box option selection, although it's hardly a big priority I'm sure. High on my personal wish-list is broader support for IMAP functions (calendars etc), but that's another story and another thread.

 

<P>Rolf,</P> <P>I've installed and tested your daemon and so far, it seems to be working fine. I've not noticed any other messages breaking and I've tested the duplication detection: works brilliantly! Now it just remains to let it run for a week and check the logs.</P> <P> Thanks very much indeed, Rolf, I appreciate you sharing the results of your labours. Perhaps David might consider including some similar code in a later version of Mercury, with a tick-box option selection, although it's hardly a big priority I'm sure. High on my personal wish-list is broader support for IMAP functions (calendars etc), but that's another story and another thread.</P> <P mce_keep="true"> </P>
live preview
enter atleast 10 characters
WARNING: You mentioned %MENTIONS%, but they cannot see this message and will not be notified
Saving...
Saved
With selected deselect posts show selected posts
All posts under this topic will be deleted ?
Pending draft ... Click to resume editing
Discard draft