Pegasus Mail & Mercury

Welcome to the Community for Pegasus Mail and
The Mercury Mail Transport System, the Internet's longest-serving PC e-mail system!
Welcome to Pegasus Mail & Mercury Sign in | Join | Help
in
Home Blogs Forums Downloads Pegasus Mail Overview Mercury Overview Wiki

Serious bug report and request for solution

Last post 03-26-2009, 21:57 by Lythande. 18 replies.
Page 1 of 2 (19 items)   1 2 Next >
Sort Posts: Previous Next
  •  03-13-2009, 11:23

    • PEC is not online. Last active: 04-14-2009, 15:02 PEC
    • Not Ranked
    • Joined on 03-13-2009
    • Member
    • Points 115

    Serious bug report and request for solution

    This is not a common problem but when it happens it is can be serious and far reaching as well as hard to detect.  I would appreciate knowing if I can solve it myself and having it examined by the developer/s.  Here is a description of the bug:

     In the normal view text gets cut off, it just never appears.  To verify this I look at the 'raw view' and each time the fuller text is there.

    There doesn't seem to be any reason that I can identify that might explain this. There are no odd characters in the raw view and nothing else to suspect might be the trigger for truncating the text.

    If this cannot be detected I am willing to send a copy of the latest example.

     In fact this most recent example came to my attention as a result of a contact telling me he had already told me something that I was unaware of.  I nearly argued with him but then remembered this flaw (bug) and double checked his original email and sure enough, in the raw view I read the extra, missing, text. Imagine if I had chosen to argue instead!

    PEC

  •  03-13-2009, 11:32

    • PEC is not online. Last active: 04-14-2009, 15:02 PEC
    • Not Ranked
    • Joined on 03-13-2009
    • Member
    • Points 115

    Re: Serious bug report and request for solution

    Addendum...

    Whoops!  Should mention I'm using Pegasus 4.41 for Windows 32bit.

    It seems that it is longer paragraphs that get cut off and mid word too.  Has someone set a very low string length somewhere and not tested it for other than short paragraphs?

  •  03-13-2009, 17:11

    Re: Serious bug report and request for solution

    PEC:
    This is not a common problem but when it happens it is can be serious and far reaching as well as hard to detect.  I would appreciate knowing if I can solve it myself and having it examined by the developer/s.  Here is a description of the bug:

     In the normal view text gets cut off, it just never appears.  To verify this I look at the 'raw view' and each time the fuller text is there.

    I'll be willing to bet the problem lines are longer that 1000 characters.  The RFC 2821says that each line of a SMTP mail message MUST be longer than 998 characters plus C/R and L/F.   The originator's email client is writing entire paragraphs as a single line. Once they get to 1000 characters they will be truncated.  Actually a SMTP mail system should reject these messages with a "Line too long" error but most do not.

    There doesn't seem to be any reason that I can identify that might explain this. There are no odd characters in the raw view and nothing else to suspect might be the trigger for truncating the text.

    If this cannot be detected I am willing to send a copy of the latest example.

    Send the copy of the CNM file zipped as an attachment to techsupp@tstephenson.com

     In fact this most recent example came to my attention as a result of a contact telling me he had already told me something that I was unaware of.  I nearly argued with him but then remembered this flaw (bug) and double checked his original email and sure enough, in the raw view I read the extra, missing, text. Imagine if I had chosen to argue instead!

    PEC


    Thomas R. Stephenson
    San Jose, California
    Member of Pegasus Mail Support Team

    I do not answer private messages from the forum. If you want to contact me use email to techsupp@tstephenson.com.
  •  03-13-2009, 17:52

    • dkocmoud is not online. Last active: 03-07-2013, 10:45 dkocmoud
    • Top 100 Contributor
    • Joined on 02-06-2007
    • Texas, USA
    • Member
    • Points 680
    • BetaTeam Moderator

    Re: Serious bug report and request for solution

    Just to make sure we're not missing the obvious, does pressing F5 (Reformat Long Lines) or Ctrl-F5 (Wrap Long Lines) have any effect while reading that message?

     


    David Kocmoud
    Pegasus Mail and Mercury beta tester
    Pegasus Mail and Mercury tech support volunteer
  •  03-13-2009, 17:52

    • PEC is not online. Last active: 04-14-2009, 15:02 PEC
    • Not Ranked
    • Joined on 03-13-2009
    • Member
    • Points 115

    Re: Serious bug report and request for solution

     Thanks for the very prompt response.

    Where it says "MUST be" I will assume you mean "MUST NOT be".

     I checked the number of characters with this latest example and it's 1009. If I do the same check with the raw view text then it is 1023.

    0 to 1023 is 1024 bytes.

    256, 512, 1024 comes to mind when I see that.

    I will zip up the example today or tomorrow and send it as requested.

    BUT, I have to say, even if the RFC is as you say, the raw view shows the full text so shouldn't we expect to see it also in the normal message view (surely?).

    Paul E. Coughlin

     

  •  03-13-2009, 17:55

    • PEC is not online. Last active: 04-14-2009, 15:02 PEC
    • Not Ranked
    • Joined on 03-13-2009
    • Member
    • Points 115

    Re: Serious bug report and request for solution

    Just tried and no it doesn't. 

    F5 has this effect:

    The short lines become a few long ones but the missing text is still missing.

     

    PEC

  •  03-13-2009, 18:11

    • dkocmoud is not online. Last active: 03-07-2013, 10:45 dkocmoud
    • Top 100 Contributor
    • Joined on 02-06-2007
    • Texas, USA
    • Member
    • Points 680
    • BetaTeam Moderator

    Re: Serious bug report and request for solution

    The raw view is the exact message how it came across the wire and is stored on the computer, which is why you can see it there.  However, when Pegasus Mail views the message it has to do some decoding work before it can display it in a presentable form.  In doing so, it makes some assumptions on the largest line size based on RFC 2821 saying that a line MUST NOT be longer than 1000 characters, including the CR/LF characters.  This is required because WinPmail reads a whole line of raw text into a memory buffer, which has to have a finite size (otherwise you suffer a buffer overflow and potential security vulnerability issue).

    The problem here is that there are programmers of email systems out there that simply CANNOT READ.  Angry  Because of them, we cannot be sure of anything, even though RFC 2821 exists, saying that email systems will never spit out a line longer than 1000 characters (including CR/LF).  So some idiot programmer breaks the "rules" and other email systems get the blame for it.

    I would strongly encourage you to respond back to your correspondent to tell them that their email client or email system is broken and that they should demand a fix immediately to abide by RFC2821 concerning the max length of a line of text in an SMTP email message.


    David Kocmoud
    Pegasus Mail and Mercury beta tester
    Pegasus Mail and Mercury tech support volunteer
  •  03-14-2009, 3:03

    Re: Serious bug report and request for solution

    Did I read incorrectly, or did you suggest we tell our correspondents to adapt to our Pmail?
  •  03-14-2009, 6:09

    Re: Serious bug report and request for solution

    Did I read incorrectly, or did you suggest we tell our correspondents to adapt to our Pmail?

    Nope, what you read was you should tell the correspondents that they should use email clients that comply with the MUST requirements of the RFC.  Para 4.5.3.1 Size limits and minimums of RFC 2821 says

       There are several objects that have required minimum/maximum sizes.
    Every implementation MUST be able to receive objects of at least
    these sizes.

     

    text line
    The maximum total length of a text line including the <CRLF> is
    1000 characters (not counting the leading dot duplicated for
    transparency). This number may be increased by the use of SMTP
    Service Extensions.
     

    Thomas R. Stephenson
    San Jose, California
    Member of Pegasus Mail Support Team

    I do not answer private messages from the forum. If you want to contact me use email to techsupp@tstephenson.com.
  •  03-14-2009, 6:29

    Re: Serious bug report and request for solution

    PEC, can you read the Raw View and tell us what the mail software of your correspondent is? You should find it after "X-Mailer".

     

    Or if you don't find it, copy/paste the header here if you can. Be careful to remove anything that could be illegal to publish before (confidential).

     

    PEC:
    BUT, I have to say, even if the RFC is as you say, the raw view shows the full text so shouldn't we expect to see it also in the normal message view (surely?).

     

    I would find it extremely difficult to argue with you on that point. In other words, you seem more than perfectly right.

     

    Thomas R. Stephenson:
    Nope, what you read was you should tell the correspondents that they should use email clients that comply with the MUST requirements of the RFC.

     

    I would humbly suggest that Pmail MUST read the text that is already in Raw View, and that the receiver SHOULD be very cautious before disturbing the sender about a mail compatibility issue ^^

     

    PEC, can you also try the same account with Thunderbird and see what happens?

  •  03-14-2009, 11:17

    • Phil is not online. Last active: Tue, Jan 12 2016, 11:44 Phil
    • Top 50 Contributor
    • Joined on 03-25-2007
    • Paris (France)
    • Participant
    • Points 2,285
    • BetaTeam Moderator Translators

    Re: Serious bug report and request for solution

    arnaudherve:

     I would humbly suggest that Pmail MUST read the text that is already in Raw View, and that the receiver SHOULD be very cautious before disturbing the sender about a mail compatibility issue ^^

    Martin Ireland David Kocmoud (sorry David) wrote : "This is required because WinPmail reads a whole line of raw text into a memory buffer, which has to have a finite size (otherwise you suffer a buffer overflow and potential security vulnerability issue)."

    Do you understand what it means?

    To display a raw view Pegasus Mail has almost nothing to do than to display what it reads in the original file, to display a presentable form, Pegasus Mail has to do some decoding, to do it it has to load the line into a memory buffer prior to launch the decoding process, that memory buffer has a limited size, as the RFC says that a line can't be longer than 1000 car, Pegasus Mail has a buffer whom size is 1000 car, it could be 1100 car but in that case you would ask why Pegasus Mail can't display a 1110 car line.

    Example taken from a mail I received:

    In raw view:

    Ca c'est aussi ing=E9rable, les postes mobiles voire m=EAme les changements  de prise r=E9seau entraineront des blocages permanents, je vais passer ma vie =E0 d=E9bloquer des ports.

    In display mode:

    Ca c'est aussi ingérable, les postes mobiles voire même les changements de prise réseau entraineront des blocages permanents, je vais passer ma vie à débloquer des ports.

    You can see the difference and then imagine that Pegasus Mail has some decoding work to process to display a message in a presentable form.

     David Harris, Thomas Stephenson, Martin Ireland and many others are very skillful in programming, you should trust them. As I told you in another post some skills in computing are not useless...


    Philippe Chartier
    French translation team leader
  •  03-14-2009, 12:02

    Re: Serious bug report and request for solution

    I think I understand. You give an example about the display of accented characters, which is ok but a bit off-topic. We are discussing in this thread the loss of display of entire paragraphs due to line length.

     

    With my poor skills, I also understand that messages violating  protocols like RFC 2821 are often detected at the level of servers and produce an error message. I would be surprised if the sender had been totally unaware of that bug.

  •  03-14-2009, 13:01

    • Phil is not online. Last active: Tue, Jan 12 2016, 11:44 Phil
    • Top 50 Contributor
    • Joined on 03-25-2007
    • Paris (France)
    • Participant
    • Points 2,285
    • BetaTeam Moderator Translators

    Re: Serious bug report and request for solution

    arnaudherve:

    I think I understand. You give an example about the display of accented characters, which is ok but a bit off-topic. We are discussing in this thread the loss of display of entire paragraphs due to line length.

    No you don't understand. We are discussing about the non displaying of a line because of its length, the display of accented characters is just an example to show you that there is a difference between displaying the raw view and displaying the definitive text, you said that if Pegasus can display the line in raw view it can do it in the display view and the answer is no because there is a process difference between the 2.

    arnaudherve:

    With my poor skills, I also understand that messages violating  protocols like RFC 2821 are often detected at the level of servers and produce an error message. I would be surprised if the sender had been totally unaware of that bug.

    The sender is probably totally unaware of the problem and probably unaware also that rfc exist and what they are for, he bought or simply downloaded a software and the only thing he wants is that the software works just like it should.


    Philippe Chartier
    French translation team leader
  •  03-14-2009, 13:16

    Re: Serious bug report and request for solution

    Phil:
    arnaudherve:

    I think I understand. (...) We are discussing in this thread the loss of display of entire paragraphs due to line length.

    No you don't understand. We are discussing about the non displaying of a line because of its length (...)

     

    You are right on everything. Let us concentrate on helping the thread starter now.

  •  03-14-2009, 15:42

    • PEC is not online. Last active: 04-14-2009, 15:02 PEC
    • Not Ranked
    • Joined on 03-13-2009
    • Member
    • Points 115

    Re: Serious bug report and request for solution

    To: David, Arnaud, Thomas and Phil,

    I have read each contribution with care and want to thank you all for your responses.

    Below I am pasting in the raw and the normal view with a few edits. I first had this problem occur a couple of years ago but there was no support system in place at that time so I just upgraded and hoped for the best.  Now, this time it has happened, I was able to find out what the problem is - just as explained by yourselves, I am sure. 

    I have also done a few tests but I would need to use a faulty sending email, (such as, it seems, the AOL web mail system). But, as a user, I am not interested in the cause if I cannot overcome it. 

    What I AM interested in is reading emails that I can rely on to be faithful to the original OR I get a warning message telling me something is wrong.  For that reason I have just switched (after about twelve years of PMail use) to Thunderbird.  But I am sad to leave. I very much believe, as a philosophy, in the "small" programmer - I am one myself as it happens.

    From a programming point of view - rather than as just the user - I can see what the problem is (thanks to you all). 

    Now, some of you won't like this at all so let me add this is my own personal view - there is no law or moral dictate that I can reach out to as support (maybe there should be though) - the fact that Pmail is strictly going about it's business isn't good enough, it should be catering for all the bad email senders as part of it's own agenda, policy or philosophy.  I emphasize, that is my personal view.

     What that means, here, is that it isn't good enough for Pmail to simply ignore long lines and offer up excuses about buffer sizes and the like (sorry, I said you wouldn't like it).

    So what SHOULD Pmail be doing (and have done already anyway)?  

    ASSUME that there will be non compliant messages received and, I believe the term is "gracefully" (no religion involved) deal with it regardless of compliance.

    And what does that mean?

     Well, if I were coding for this in VB (not net) then first off string sizes don't matter much so I would have to test for the end of the line by looking for what always comes next then I know that however long the line is, I have reached the end, not by relying of dubious quality input (people are well known for not sticking to the rules) but by positively testing for it.

     At first I called this a serious bug.  However, maybe I should rephrase it and call it a serious flaw in the philosophy behind the programming which gives rise to what is tantamount to a serious flaw in the program whether that is strictly a bug or not doesn't matter much, it's still the job of the programmers and designers (to my mind) to watch out for and at least be aware of the implications and act accordingly.  

    The implications, for me,  are that I have had emails where important information has been completely missed due to this one flaw on it's own and I have only found out by accident.

     I realise I have now seriously ruffled many feathers. So I will, gracefully, stop there and reitterate my thanks to each of you (regardless whether I appear to have criticised or confirmed your statements - each has been appreciated very much).  

    I look forward to reading the forthcoming furore...

    Here now is the promised paste job...

    Received: (qmail 20121 invoked from network); 10 Mar 2009 14:51:17 -0000
    Received: from unknown (HELO p3presmtp01-10.prod.phx3.secureserver.net) ([208.189.88.159])
              (envelope-sender <{Privacy Edit - pec}@aol.com>)
              by smtp17-02.prod.mesa1.secureserver.net (qmail-1.03) with SMTP
              for <pec@sanethinking.com>; 10 Mar 2009 14:51:17 -0000
    Received: (qmail 22556 invoked from network); 10 Mar 2009 14:51:17 -0000
    Received: from imo-d04.mx.aol.com ([205.188.157.36])
              (envelope-sender <{Privacy Edit - pec}@aol.com>)
              by p3presmtp01-10.prod.phx3.secureserver.net (qmail-ldap-1.03) with SMTP
              for <pec@sanethinking.com>; 10 Mar 2009 14:51:12 -0000
    Received: from {Privacy Edit - pec}@aol.com
        by imo-d04.mx.aol.com  (mail_out_v39.1.) id z.cc8.432703ac (34899)
         for <pec@sanethinking.com>; Tue, 10 Mar 2009 10:50:59 -0400 (EDT)
    Received: from smtprly-mb03.mx.aol.com (smtprly-mb03.mx.aol.com [64.12.207.150]) by cia-da01.mx.aol.com (v123.3) with ESMTP id MAILCIADA018-5c7849b67e4c25c; Tue, 10 Mar 2009 10:50:59 -0400
    Received: from webmail-de07 (webmail-de07.webmail.aol.com [205.188.104.28]) by smtprly-mb03.mx.aol.com (v121_r5.5) with ESMTP id MAILSMTPRLYMB037-5c7849b67e4c25c; Tue, 10 Mar 2009 10:50:52 -0400
    References: <49B10731.18706.516F34@pec.SaneThinking.com>, <8CB6DDF3EFD771B-1140-7AD@webmail-da15.sysops.aol.com> <49B672A8.14154.BBBEC8@pec.SaneThinking.com>
    To: pec@SaneThinking.com
    Subject: Re: Your letter in MEN
    Date: Tue, 10 Mar 2009 10:50:51 -0400
    X-AOL-IP: 92.237.43.114
    In-Reply-To: <49B672A8.14154.BBBEC8@pec.SaneThinking.com>
    X-MB-Message-Source: WebUI
    MIME-Version: 1.0
    From: {Privacy Edit - pec}@aol.com
    X-MB-Message-Type: User
    Content-Type: multipart/alternative;
     boundary="--------MB_8CB6FA6BE0A04CE_91C_132_webmail-de07.sysops.aol.com"
    X-Mailer: AOL Webmail 41757-STANDARD
    Received: from 92.237.43.114 by webmail-de07.sysops.aol.com (205.188.104.28) with HTTP (WebMailUI); Tue, 10 Mar 2009 10:50:51 -0400
    Message-Id: <8CB6FA6BD9793BE-91C-A1@webmail-de07.sysops.aol.com>
    X-Spam-Flag:NO
    X-Nonspam: Statistical 65%
    X-PMFLAGS: 570950016 0 1 P4HF0DQQ.CNM                      


    ----------MB_8CB6FA6BE0A04CE_91C_132_webmail-de07.sysops.aol.com
    Content-Transfer-Encoding: 7bit
    Content-Type: text/plain; charset="us-ascii"

    Hi Paul
    Thanks for the reply and the advice. You're right I didn't have the site when I wrote to the MEN which was a mistake I should have waited until I'd sorted the site before I wrote the letters. I actually wrote to every local and national newspaper in the country but on the MEN, BEN and a couple of the freebies actually printed it. I've written to the BBC gardeners World magazine, The national Trust, The RHA and ITV and to every local authority allotments officer in the country. Although I'm getting the 'it's a great idea' returns, I'm not getting much by way of practical support. Your suggestions of links to other sites is good but my knowledge and understanding of the net is quite limited. I've started a face book group but again I don't really know how to promote it there either. I can run the scheme and the legal stuff with my eyes closed but getting the support and the name out there is proving more difficult than I imagined. I'd certainly appreciate any practical help you can gi
    ve in that direction. I take on board your comments about forums and guest books and I don't want to get tied down with inane chatter or to be constantly erasing viagra adverts so it might be best if I turn them off altogether.


    As I said thanks for the advice and any practical help would be greatly appreciated. I have applied for some lottery funding and a few other grants but they take months for the applications to be sorted but if I get some funding I'll certainly be spending it on the web site. You mentioned about changing the host, can you recommend someone?

    Cheers

    {Privacy Edit - pec}

    -----------------------------------------------------------------------------------------------------------
    Below is the "message" view; (above is the "Raw view")
    -----------------------------------------------------------------------------------------------------------

    Hi Paul

    Thanks for the reply and the advice. You're right I didn't have the site
    when I wrote to the MEN which was a mistake I should have waited until I'd
    sorted the site before I wrote the letters. I actually wrote to every
    local and national newspaper in the country but on the MEN, BEN and a
    couple of the freebies actually printed it. I've written to the BBC
    gardeners World magazine, The national Trust, The RHA and ITV and to every
    local authority allotments officer in the country. Although I'm getting
    the 'it's a great idea' returns, I'm not getting much by way of practical
    support. Your suggestions of links to other sites is good but my knowledge
    and understanding of the net is quite limited. I've started a face book
    group but again I don't really know how to promote it there either. I can
    run the scheme and the legal stuff with my eyes closed but getting the
    support and the name out there is proving more difficult than I imagined.
    I'd certainly appreciate any practical help you can give in that
    direction. I ta

    As I said thanks for the advice and any practical help would be greatly
    appreciated. I have applied for some lottery funding and a few other
    grants but they take months for the applications to be sorted but if I get
    some funding I'll certainly be spending it on the web site. You mentioned
    about changing the host, can you recommend someone?

    Cheers

    {Privacy Edit - pec}

Page 1 of 2 (19 items)   1 2 Next >
View as RSS news feed in XML

Contact | Advertise | Host provider: PraktIT | Terms of Use | Privacy Statement
Copyright © 2007-2011 David Harris / Peter Strömblad. | Pegasus Mail Home Page