getMsgTextFromStream: inserting breaklines problem

Talk about add-ons and extension development.
user2018
Posts: 196
Joined: September 23rd, 2018, 11:07 am

getMsgTextFromStream: inserting breaklines problem

Post by user2018 »

I have the following function defined in userChrome.js:

Code: Select all

function getMessageBody(aMessageHeader)
{
	let messenger = Components.classes["@mozilla.org/messenger;1"].createInstance(Components.interfaces.nsIMessenger);
	let listener = Components.classes["@mozilla.org/network/sync-stream-listener;1"].createInstance(Components.interfaces.nsISyncStreamListener);
	let uri = aMessageHeader.folder.getUriForMsg(aMessageHeader);
	messenger.messageServiceFromURI(uri).streamMessage(uri, listener, null, null, false, "");
	let folder = aMessageHeader.folder;
	return folder.getMsgTextFromStream(listener.inputStream, aMessageHeader.Charset, 65536, 32768, false, true, { });
}
This was working well, but I have noticed that now, when I use it to retrieve the body of a message as text, newlines are artificially inserted in the text retrieved (I do not see those newlines in the message view, they are not there). It seems like an artificial breakline is added to limit the line's lenght. This is inconvenient because then I have to manually remove the breaklines.

This change might have been introduced with some Thunderbird's update, as I think this problem did not exist before.

Any idea about how this could be solved?

Thank you.
morat
Posts: 6394
Joined: February 3rd, 2009, 6:29 pm

Re: getMsgTextFromStream: inserting breaklines problem

Post by morat »

I don't know. That exact code is still being used in a calendar function.

Reference
http://searchfox.org/comm-esr78/search? ... romMessage

P.S.

It's best to post the Thunderbird userChrome.js topics in the Extension Development forum.
User avatar
LIMPET235
Moderator
Posts: 39920
Joined: October 19th, 2007, 1:53 am
Location: The South Coast of N.S.W. Oz.

Re: getMsgTextFromStream: inserting breaklines problem

Post by LIMPET235 »

Moving this to the Extension Dev forum...
[Ancient Amateur Astronomer.]
Win-10-H/64 bit/500G SSD/16 Gig Ram/450Watt PSU/350WattUPS/Firefox-115.0.2/T-bird-115.3.2./SnagIt-v10.0.1/MWP-7.12.125.

(Always choose the "Custom" Install.)
morat
Posts: 6394
Joined: February 3rd, 2009, 6:29 pm

Re: getMsgTextFromStream: inserting breaklines problem

Post by morat »

The following test code works for me. I don't see extra newlines.

Code: Select all

(function () {
  var msgHdr = gFolderDisplay.selectedMessage;
  var messenger = Components.classes["@mozilla.org/messenger;1"].
    createInstance(Components.interfaces.nsIMessenger);
  var listener = Components.classes["@mozilla.org/network/sync-stream-listener;1"].
    createInstance(Components.interfaces.nsISyncStreamListener);
  var uri = msgHdr.folder.getUriForMsg(msgHdr);
  messenger.messageServiceFromURI(uri).streamMessage(uri, listener, null, null, false, "");
  var folder = msgHdr.folder;
  var text = folder.getMsgTextFromStream(listener.inputStream, msgHdr.Charset, 65536, 32768, false, true, {});
  var clipboard = Components.classes["@mozilla.org/widget/clipboardhelper;1"].
    getService(Components.interfaces.nsIClipboardHelper);
  clipboard.copyString(text);
  alert("Plain text message copied to clipboard.");
})();
user2018
Posts: 196
Joined: September 23rd, 2018, 11:07 am

Re: getMsgTextFromStream: inserting breaklines problem

Post by user2018 »

I tried the code above and I see the "extra" breaklines in the messages I receive. However, by playing with this I have realized that the breaklines actually seem to exist in the message received (if I save the email as .eml and I open it with a text editor, I also see the "extra" breaklines). However, they are not shown in the message view panel of Thunderbird, so if I copy and paste the contents of the message by selecting the text from the message view panel of Thunderbird the "extra" breaklines are not copied. I guess I have some property defined in Thunderbird to automatically remove the breaklines (not sure about this... I know is that I have auto-wrapping, but this seems to be something else).

It would be great if I could modify the JavaScript code to be able to get the text in the body as shown in the message view panel of Thunderbird (that is, without the "extra" breaklines), if possible.

Any suggestion is welcme.
morat
Posts: 6394
Joined: February 3rd, 2009, 6:29 pm

Re: getMsgTextFromStream: inserting breaklines problem

Post by morat »

IMO, the getMsgTextFromStream method works correctly.

You are stripping html tags from the output with the getMsgTextFromStream method.

i.e. aStripHTMLTags param is true

Reference
http://searchfox.org/comm-esr78/search? ... m&path=idl

So of course the .eml file will be different.
user2018
Posts: 196
Joined: September 23rd, 2018, 11:07 am

Re: getMsgTextFromStream: inserting breaklines problem

Post by user2018 »

I continued trying to find a solution for my problem. The problem has nothing to do with HTML contents, as this happens to me with plain text.

I suspect that it has something to do with a Config Editor property (or maybe several of them). In particular, I thought that "mailnews.wraplength" was the culprit, as its default value is 72, which is about the maximum number of characters in the text lines I get (it is like an artificial break line is added if the line is going to exceed that length). However, I still cannot make it work. Could this be a good hint? Any ideas are welcome.
morat
Posts: 6394
Joined: February 3rd, 2009, 6:29 pm

Re: getMsgTextFromStream: inserting breaklines problem

Post by morat »

I can't reproduce the problem so I really have no idea.

More info about plain text wrapping
http://kb.mozillazine.org/Plain_text_e- ... d#Advanced
user2018
Posts: 196
Joined: September 23rd, 2018, 11:07 am

Re: getMsgTextFromStream: inserting breaklines problem

Post by user2018 »

I'm still trying to solve this problem, as it takes a long time for me to manually remove the unwanted breaklines. I discovered something that could help. I noticed that the problem happens only with messages sent to me as a HTML. It's for the case of HTML messages that getMsgTextFromStream returns these extra breaklines. Anyway, the message is visualized correctly, with no extra breaklines, in the message viewer of Thunderbird; indeed, if I copy from the message viewer and I paste in a text file the extra breaklines do not appear. Any suggestion is very welcome!

(Displaying the message as HTML or plain text does not seem to have any impact on this problem)
morat
Posts: 6394
Joined: February 3rd, 2009, 6:29 pm

Re: getMsgTextFromStream: inserting breaklines problem

Post by morat »

I need a reproducible test case.

Try finding a .eml file with the problem. Then post a link. Then I will import and test.

Sample .eml file
http://www.phpclasses.org/browse/file/14672.html

ImportExportTools NG
http://addons.thunderbird.net/thunderbird/addon/986686
user2018
Posts: 196
Joined: September 23rd, 2018, 11:07 am

Re: getMsgTextFromStream: inserting breaklines problem

Post by user2018 »

I left an example at https://we.tl/t-RudrizINd5
Thank you.
morat
Posts: 6394
Joined: February 3rd, 2009, 6:29 pm

Re: getMsgTextFromStream: inserting breaklines problem

Post by morat »

Okay. I understand the problem now. I can't fix the artificial break line issue with the getMsgTextFromStream method. I believe the wrap width is hardcoded in the nsMsgDBFolder.cpp file.

Reference
http://searchfox.org/comm-esr78/search? ... Folder.cpp
http://searchfox.org/comm-esr78/search? ... Folder.cpp

What are you doing with the |getMessageBody| function in the userChrome.js file? Are you copying the message text to the clipboard? You can use the ImportExportTools helper functions to do that.

Code: Select all

copyMSGtoClip();

Code: Select all

(function () {
  var msgHdr = gFolderDisplay.selectedMessage;
  var uri = msgHdr.folder.getUriForMsg(msgHdr);
  var text = IETstoreBody(uri);
  var clipboard = Components.classes["@mozilla.org/widget/clipboardhelper;1"].
    getService(Components.interfaces.nsIClipboardHelper);
  clipboard.copyString(text);
})();
ImportExportTools exportTools.js
http://github.com/thundernest/import-ex ... rtTools.js
user2018
Posts: 196
Joined: September 23rd, 2018, 11:07 am

Re: getMsgTextFromStream: inserting breaklines problem

Post by user2018 »

Thank you very much for the suggestion! I have a simple question: how can I import the exportTools.js file in userChrome.js? I guess this is what I should do (rather than copy and paste the whole contents of exportTools.js into my file?

Yes, I end up copying the text in the clipboard, but I guess the important part to adopt is the line "var text = IETstoreBody(uri);".
morat
Posts: 6394
Joined: February 3rd, 2009, 6:29 pm

Re: getMsgTextFromStream: inserting breaklines problem

Post by morat »

You need to install the ImportExportTools addon.

ImportExportTools NG
http://addons.thunderbird.net/thunderbird/addon/986686

The ImportExportTools copy message command calls the |copyMSGtoClip| function.

i.e. right click Message Pane > Copy to clipboard > Message

The |copyMSGtoClip| and |IETstoreBody| functions are global functions like the |alert| function.

The above code snippets work in the error console after installing the addon.

Here is a code snippet to set the clipboard content with the |copyMSGtoClip| function, get the clipboard content, replace text, then set the clipboard content again.

Code: Select all

(function () {
  copyMSGtoClip(); // asynchronous
  alert("Wait.");
  var type = Ci.nsIClipboard.kGlobalClipboard;
  var text = EnigmailClipboard.getClipboardContent(window, type);
  text = text.replace(/.*/, "").substr(1); // remove "Subject:" line
  text = text.replace(/.*/, "").substr(1); // remove "From:" line
  text = text.replace(/.*/, "").substr(1); // remove "Date:" line
  text = text.replace(/.*/, "").substr(1); // remove "To:" line
  text = text.replace(/.*/, "").substr(1);
  EnigmailClipboard.setClipboardContent(text, type);
  alert("Message copied to clipboard.");
})();
Here is a code snippet based on the |IETstoreBody| function that works without the addon.

Code: Select all

(function() {
  var msghdr = gFolderDisplay.selectedMessage;
  var msguri = msghdr.folder.getUriForMsg(msghdr);
  var content = "";
  var MsgService = messenger.messageServiceFromURI(msguri);
  var MsgStream = Cc["@mozilla.org/network/sync-stream-listener;1"].createInstance();
  var consumer = MsgStream.QueryInterface(Ci.nsIInputStream);
  var ScriptInput = Cc["@mozilla.org/scriptableinputstream;1"].createInstance();
  var ScriptInputStream = ScriptInput.QueryInterface(Ci.nsIScriptableInputStream);
  ScriptInputStream.init(consumer);
  try {
    MsgService.streamMessage(msguri, MsgStream, null, null, true, "header=filter");
  } catch (e) {
    return content;
  }
  ScriptInputStream.available();
  while (ScriptInputStream.available()) {
    content = content + ScriptInputStream.read(512);
  }
  var toStr = {value: null};
  var formatConverter = Cc["@mozilla.org/widget/htmlformatconverter;1"].
    createInstance(Ci.nsIFormatConverter);
  var fromStr = Cc["@mozilla.org/supports-string;1"].
    createInstance(Ci.nsISupportsString);
  var converter = Cc["@mozilla.org/intl/scriptableunicodeconverter"].
    createInstance(Ci.nsIScriptableUnicodeConverter);
  converter.charset = "UTF-8";
  var dataUTF8 = converter.ConvertToUnicode(content);
  var text;
  fromStr.data = dataUTF8;
  try {
    const versionChecker = Services.vc;
    const currentVersion = Services.appinfo.platformVersion;
    if (versionChecker.compare(currentVersion, "61") >= 0) {
      formatConverter.convert("text/html", fromStr, "text/unicode", toStr);
    } else {
      formatConverter.convert("text/html", fromStr, fromStr.toString().length, "text/unicode", toStr, {});
    }
  } catch (e) {
    text = dataUTF8;
  }
  if (toStr.value) {
    toStr = toStr.value.QueryInterface(Ci.nsISupportsString);
    var os = navigator.platform.toLowerCase();
    var strValue = toStr.toString();
    if (os.indexOf("win") > -1)
      text = strValue.replace(/(.+\r\n?)*/, "");
    else
      text = strValue.replace(/(.+\n?)*/, "");
    text = text.replace(/\r?\n+/g, "\r\n");
    text = text.replace(/^(\r\n)/g, "");
    text = text.replace(/(\r\n)$/g, "");
  }
  text = text.replace("$%$%$", ":");
  var clipboard = Components.classes["@mozilla.org/widget/clipboardhelper;1"].
    getService(Components.interfaces.nsIClipboardHelper);
  clipboard.copyString(text);
  alert("Message copied to clipboard.");
})();
Last edited by morat on July 10th, 2021, 7:15 am, edited 3 times in total.
user2018
Posts: 196
Joined: September 23rd, 2018, 11:07 am

Re: getMsgTextFromStream: inserting breaklines problem

Post by user2018 »

This is great!! It solves the problem, except for one minor issue: the text copied includes the headers of the messages (from, to, subject). I would like to retrieve the body text without the headers. Is there any slight modification that can be performed to achieve this? I tried with some minor modifications with no luck.
Post Reply