Good and Bad News!

First, the bad news is that the trailing asterisk in  .ru/B*  did not help.

In the above two rules, each of the 13 caught by Russia was

also caught by Russia B.  Each of the 12 missed was missed by

both. 

Good news is, after closely analyzing the hex in what was caught and missed, I think I finally figured it out.

The catch is when there are 2 CRLF adjoining the .ru:     www.somewhere.ruCRLFCRLF next line may be blank or may have text CRLF       (or sometimes no next line at all)

The miss is when it is followed by only 1 CRLF:              www.somewhere.ruCRLF  (next line is  blank in the ones I looked at) CRLF

 So, it appears the bug is not at EOM, but at the end of a line.

Mike

I suspect that Content Control has a bug in scanning the last byte of the inbound message.

I will furnish the entire rules list upon request. The applicable rules are:

IF BODY  MATCHES "*http*.ru"       WEIGHT 35 TAG "nRussia"
IF BODY  MATCHES "*http*.ru[/ ]*" WEIGHT 55 TAG "Russia"

The inbound Email is posted below in raw form  (please note the weights assigned):

Email 1:

================

Received: from spooler by MountainThyme.com (Mercury/32 v4.74); 23 Apr 2012 23:32:25 -0500
X-Envelope-To: truedelete
Resent-Date: Mon, 23 Apr 2012 23:32:14 -0500
X-Autoforward: 1
Received: from POP3D by MountainThyme.com with MercuryD (v4.74); 23 Apr 2012 23:32:08 -0500
Return-Path: <masonite@intarget.net>
Delivered-To: rhonda@mountainthyme.com
Received: (qmail 21555 invoked by uid 20868); 24 Apr 2012 03:52:47 -0000
X-RBL: (sbl-xbl.spamhaus.org) tells us See http://www.spamhaus.org/SBL
Received: from unknown (HELO 189.221.190.78.cable.dyn.cableonline.com.mx) ([189.221.190.78])
          (envelope-sender <masonite@intarget.net>)
          by 198.66.209.132 (qmail-ldap-1.03) with SMTP
          for <rhonda@mountainthyme.com>; 24 Apr 2012 03:52:47 -0000
Message-ID: <DLLKKIGAIJRVXOTPLZKJCNQK.22221856477080@vujeq.eurowebwatch.com>
From: "Beatrice Obrien" <Beatrice.Obrien@eurowebwatch.com>
Organization: honeydew
To: rhonda@mountainthyme.com
Subject: Best-quality pharmwarehouse
Date: Tue, 24 Apr 2012 00:49:45 -0300
MIME-Version: 1.0
Content-Type: text/plain; charset="iso-8859-1"
Content-Transfer-Encoding: 7Bit
X-Priority: 3
X-UC-Weight: [#   ] 80
X-CC-Diagnostic: Header X-RBL contains "See http://www.spamhaus.org/SBL" (25),
 Russia (55)

Internet Shop of RX
http://eich.soondrugs.ru

=================

Note above has correct ".ru" value assigned.

=================

Received: from spooler by MountainThyme.com (Mercury/32 v4.74); 25 Apr 2012 08:18:53 -0500
X-Envelope-To: truedelete
Resent-Date: Wed, 25 Apr 2012 08:18:22 -0500
X-Autoforward: 1
Received: from POP3D by MountainThyme.com with MercuryD (v4.74); 25 Apr 2012 08:17:55 -0500
Return-Path: <shizukoardath@unidial.com>
Delivered-To: inn@mountainthyme.com
Received: (qmail 32476 invoked by uid 20868); 25 Apr 2012 09:50:26 -0000
X-RBL: (sbl-xbl.spamhaus.org) tells us See http://www.spamhaus.org/SBL
Received: from unknown (HELO fdcqrcykjdi3.gf) ([84.108.43.241])
          (envelope-sender <shizukoardath@unidial.com>)
          by 198.66.209.132 (qmail-ldap-1.03) with SMTP
          for <inn@mountainthyme.com>; 25 Apr 2012 09:50:26 -0000
Date: Wed, 25 Apr 2012 03:46:26 -0700
MIME-Version: 1.0
From: "Ashely Mirian" <shizukoardath@unidial.com>
To: <inn@mountainthyme.com>
Subject: high luxury quality replica Swiss watches. We have a lot of famous brands. Such as Jacob & Co, TAG Heuer, ... 4bapec
Message-ID: <4f97d602.91d5a1ae@unidial.com>
Content-Type: text/plain; charset=iso-8859-1
Content-Transfer-Encoding: 7bit
X-UC-Weight: [#   ] 50
X-CC-Diagnostic: Header X-RBL contains "See http://www.spamhaus.org/SBL" (25),
 Subject contains "replica" (0)

              
On-line Shop High Quality Replica Swiss Watches & Jewelry ...
Replica Watches & Jewelry at ProWatches - high luxury quality replica Swiss watches. We have a lot of famous brands. Such as Jacob & Co, TAG Heuer, ...
http://bogute.ru
                 
===================

Note above does NOT have correct ".ru" value assigned.

====================

Based on review of multiple instances of this, the failures appear to be when the ".ru" is the last character string in the message. I wonder if there is a small error in calculating the length of the compare.

Tests where I forward the email results in the correct value being assigned.  I assume the difference may be in the length of the text file as dowloaded

as compared with the text file as stored.  Just a thought.

=================

mah

[quote user="mah0110"]

I suspect that Content Control has a bug in scanning the last byte of the inbound message.

I will furnish the entire rules list upon request. The applicable rules are:

IF BODY  MATCHES "*http*.ru"       WEIGHT 35 TAG "nRussia"
IF BODY  MATCHES "*http*.ru[/ ]*" WEIGHT 55 TAG "Russia"[/quote]

What are you trying to catch with that last one?  .ru followed by a slash or a space?

[quote]The inbound Email is posted below in raw form  (please note the weights assigned):

email 1.....[snipped]

http://eich.soondrugs.ru

email 2....[snipped]

http://bogute.ru

[/quote]

We need to know what follows the .ru - it may be a space or a tab, or perhaps a line ending (CRLF).  A hex dump of just that part would show it.

The second match was the original  looking for domain followed by a slash or space. The first is a debug entry trying to understand why main match wasn't working.

Hex dump of applicable section follows.  (1st time I've gone looking for hex display in 10 years or more.  Didn't know was so unusual nowadays..... )

00000000  20 53 75 63 68 20 61 73-20 4A 61 63 6F 62 20 26   Such as Jacob &
00000010  20 43 6F 2C 20 54 41 47-20 48 65 75 65 72 2C 20   Co, TAG Heuer,
00000020  2E 2E 2E 0D 0A 68 74 74-70 3A 2F 2F 62 75 6E 6E  .....http://bunn
00000030  65 74 2E 72 75 0D 0A 20-20 20 20 20 20 20 20 20  et.ru..        
00000040  20 20 20 20 20 20 20 20-20 20 20 20 20 20 20 20                 
00000050  20 0D 0A 0D 0A         -                          ....

 The crlf explains the miss on the [/ ] match.  But why the miss on the other?

Thanks.

Mike

[quote user="mah0110"]00000020  2E 2E 2E 0D 0A 68 74 74-70 3A 2F 2F 62 75 6E 6E  .....http://bunn
00000030  65 74 2E 72 75 0D 0A 20-20 20 20 20 20 20 20 20  et.ru..        
00000040  20 20 20 20 20 20 20 20-20 20 20 20 20 20 20 20                 
00000050  20 0D 0A 0D 0A         -                          ....

 The crlf explains the miss on the [/ ] match.  But why the miss on the other?[/quote]

Do you still have the debug entry and it didn't match?  Does it work with .ru* or .ru/B ?

(I use Ztree for hex checking/editing, others will have their own favourites.)

Yeah, both entries are still in there.

Will add .ru* and .ru/B .... shouldn't take long to find out.  Getting a ton of these nowadays.  Will post results asap.

Mike

Well, it is kind of interesting....

I created a set of TEST rules as follows:
IF BODY  MATCHES "*http*.ru"      WEIGHT 50 TAG "nRussia"
IF BODY  MATCHES "*http*.ru[/ ]*" WEIGHT 51 TAG "Russia"
IF BODY  MATCHES "*http*.ru/B"    WEIGHT 52 TAG "Russia B"
IF BODY  MATCHES "*http*.ru*"     WEIGHT 53  TAG "Russia *"

The above rule set was first in line. The next rule set to process is my "Delete" rule set which contains, among a lot of other things,  the 2 rules noted in the original post.

=====

This one passed thru my TEST rules, but then got caught in my regular spam filter as shown

000005D0  6C 69 76 65 72 79 2C 20-32 34 2F 37 20 63 75 73  livery, 24/7 cus
000005E0  74 6F 6D 65 72 20 73 75-70 70 6F 72 74 2E 20 0D  tomer support. .
000005F0  0A 68 74 74 70 3A 2F 2F-70 68 61 72 6D 2D 6D 61  .http://pharm-ma
00000600  6C 6C 37 37 37 2E 72 75-0D 0A 20 20 20 20 20 20  ll777.ru..      
00000610  20 20 20 20 0D 0A 0D 0A-                             ....

X-UC-Weight: [#   ] 25
X-CC-Diagnostic: Header X-RBL contains "See http://www.spamhaus.org/SBL" (25)

                           
Buy Hydrocodone | Discount Prices, OVERNIGHT Delivery, >> Low Cost ...
Buy Hydrocodone online >> Low Cost, Express Delivery <<, discount prices, discrete packaging, express delivery, 24/7 customer support.
http://pharm-mall777.ru

=====

The TEST rules set did catch this one:

       
00000690  0A 33 35 30 30 30 30 30-2B 20 73 61 74 69 73 66  .3500000+ satisf
000006A0  69 65 64 20 63 75 73 74-6F 6D 65 72 73 0D 0A 0D  ied customers...
000006B0  0A 0D 0A 68 74 74 70 3A-2F 2F 61 72 63 6C 63 2E  ...http://arclc.
000006C0  6D 65 64 69 63 62 61 73-65 6D 2E 72 75 0D 0A 0D  medicbasem.ru...
000006D0  0A                     -                         .

X-UC-Weight: [### ] 156
X-CC-Diagnostic: Russia (51), Russia B (52), Russia * (53)

USPS - Fast Delivery Shipping 1-4 day USA

PRODUCT QUALITY - 100% Guaranteed
 
    * U.S. Licensed Pharmacies
    * U.S. Licensed Physicians
    * Discreet Packaging
    * Confidential Ordering
    * Next Day Delivery Available

3500000+ satisfied customers


http://arclc.medicbasem.ru

=====

As near as I can tell, 

1. If Russia B caught it, then so did both Russia and Russia* (as in the weight 156 example above).

2. Russia does work in obvious cases.

3. Russia * does get false positives, as would be expected ..... (EG www.somebody.ruleset.com) Well, not false exactly, just not what I was fishing for.

4. No case of nRussia has appeared.

 Mike

[quote user="mah0110"]Well, it is kind of interesting....

I created a set of TEST rules as follows:
IF BODY  MATCHES "*http*.ru"      WEIGHT 50 TAG "nRussia"
IF BODY  MATCHES "*http*.ru[/ ]*" WEIGHT 51 TAG "Russia"
IF BODY  MATCHES "*http*.ru/B"    WEIGHT 52 TAG "Russia B"
IF BODY  MATCHES "*http*.ru*"     WEIGHT 53  TAG "Russia *"

[message snipped]

=====

As near as I can tell, 

1. If Russia B caught it, then so did both Russia and Russia* (as in the weight 156 example above).

2. Russia does work in obvious cases.

3. Russia * does get false positives, as would be expected ..... (EG www.somebody.ruleset.com) Well, not false exactly, just not what I was fishing for.

4. No case of nRussia has appeared.[/quote]

It seems as if trying to match something with the end of the message doesn't give the expected results.  However, you should be able to pick up all .ru domains - try using 'Russia B' as "*http*.ru/B*"  to match anywhere.

(I find I don't have to use CC much anymore.  Graywall and Clamwall+Sanesecurity do a good job for me.)

I'll post one last test with
IF BODY  MATCHES "*http*.ru[/ ]*"   WEIGHT 51 TAG "Russia"
IF BODY  MATCHES "*http*.ru/B*"    WEIGHT 52 TAG "Russia B"

Thanks for the suggestion.

Mike