IMAP save attachement filter bug workarounds

User Help for Mozilla Thunderbird
Post Reply
Patch2
Posts: 6
Joined: November 21st, 2016, 2:59 am

IMAP save attachement filter bug workarounds

Post by Patch2 »

I receive faxes via a third party who emails the faxes to me as a pdf attachment.
I'm trying to set up Thunderbird to copy the pdf attachment to a local file server directory then move the email out of my inbox, to a archival folder (from where it can be deleted in about month or the original accessed if needed). I'm using IMAP so the inbox cleaning functionality benefits all devices I access my email from.

The FiltaQuilla addon enables saving attachments to a directory, it works when run manually but when run automatically as a message is received no attachment is saved. I believe the problem is thunderbird runs the filter before the attachment is downloaded. I think it worked better when set up as a Pop mail account however I have not fully tested that as pop is not a useful configuration for my application.

So I suppose my questions are
  1. Is there a way to force thunderbird to download full IMAP messages?
  2. Am I doing something wrong? Does this work for other users?
  3. Is this beyond ThunderBird or FiltaQuilla? If so does anyone have suggestions for a better platform?
In more detail
Filters in Thunderbird should execute in the order they are listed however FiltaQuilla executes asynchronously. As as result in the following filter

Code: Select all

      {Wait for items referenced in search to be available}
From is MyHostedFaxServer@theirURL
Subject starts with "Fax from"
Attachment Status is has attachment
      {Perform actions (in stated order)}
Mark as Read
      {Wait for above to complete}
      {Wait for attachment to download}
Save Attachment to \\MyServer\ReceivedFaxes
      {Wait for file write to complete}
Move Message to zFaxProcessed on FaxEmailAddress@myEmailHost {an IMAP subfolder}
This filter should clear all incoming faxes from my inbox by copying the actual fax (a pdf document) to a local share on my network and moving the email to a subfolder from where it can be deleted at a later date. Emails not processed for any reason should be easy to spot as they will sit in my inbox. Any interruption to processing may result in a duplicate pdf in my received fax directory but not missing documents.

In practice it does not work using FiltaQuilla because the components run asynchronously, without waiting for input data to be available or the proceeding operation to finish. This creates a race condition or in the above pseudo code "Wait for" is implemented as "Don't wait for".
The proposed work around for combining moves with complex actions is to:
  1. apply filter getting new mail "Filter after Junk classification" rather than "Filter before Junk Classification".
  2. And replace "Move Message to" with "Move later"
Unfortunately adding small fixed delays may work some of the time but will fail under some load conditions and almost never works on my computer if run "When getting new mail" as
  1. no document is copied to network drive and
  2. email in Thunderbird zFaxProcessed subfolder on FaxEmailAddress@myEmailHost has unknown or zero length [until "Repair Folder" is run in ThunderBird, as reported by other mozillaZine users).
Note I'm running the current release versions of Thunderbird v45.4.0 and FiltaQuilla v1.3.2 on a Windows 7 professional 64 bit. Email is accessed via IMAP mail server (so inbox is maintained for all of my devices).

Also FiltaQuilla bug saving to network drives has persisted since at least 2010, causing the leading double backslash is converted to a single backslash on reloading. Fortunately this can be readily circumvented by mapping the network drive to a local drive.
Although there appears to be many outstanding filter issues


I have looked at AttachmentExtractor which had good attachment renaming capabilities but is designed to run interactively and is no longer maintained.
User avatar
tanstaafl
Moderator
Posts: 49647
Joined: July 30th, 2003, 5:06 pm

Re: IMAP save attachement filter bug workarounds

Post by tanstaafl »

See http://kb.mozillazine.org/Entire_messag ... AP_message . Try setting mail.imap.mime_parts_on_demand false. That should force it to fetch the entire message when it is opened. The drawback is that setting effects all IMAP accounts.
Patch2
Posts: 6
Joined: November 21st, 2016, 2:59 am

Re: IMAP save attachement filter bug workarounds

Post by Patch2 »

tanstaafl wrote:See http://kb.mozillazine.org/Entire_messag ... AP_message . Try setting mail.imap.mime_parts_on_demand false.
Thanks for the suggestion and reference. I Set
  1. setting mail.imap.mime_parts_on_demand false
  2. mail.server.default.mime_parts_on_demand false
This appeared to fixed the unknown or zero length attachment after moving and email via a filter (saves having to "Repair Folder") but actually results in zero length documents being saved.
Also the "Save Attachment to" filter action still fails to actually save the attachment when run automatically as a message is received (but works when run manually).

I'm out of ideas. I suspect getting IMAP "Save Attachment to" filter action working is beyond Thunderbird/FiltaQuilla/My current capabilities.

Is there something else worth trying with Thunderbird?
If not, what tool would moxillaZine's recommend using? Claws Mail is multiplatform, distributed under GPL and appears supports Save attachment via munpack, however I'm unsure how well this works on a windows system. Has anyone had more luck doing similar things with this email client or any other.
Patch2
Posts: 6
Joined: November 21st, 2016, 2:59 am

Re: IMAP save attachement filter bug workarounds

Post by Patch2 »

Just in case anyone else has the same problem, the bug continues unchanged in Thunderbird v52.0b4

I believe the underlying issue is filters are run synchronously but possibly slow operations such as full message download and msf file generation is run asynchronously.
This caused a problem when an synchronous task needs an asynchronous task to complete.

The solution will involve
  • A mechanism for a synchronous task to wait for an asynchronous task when the selected filter operation requires it
  • The synchronous task reporting to the user when a wait over a set amount occurs, possibly with the option of stopping the filtering / script.
Patch2
Posts: 6
Joined: November 21st, 2016, 2:59 am

Re: IMAP save attachement filter bug workarounds

Post by Patch2 »

Update Just in case anyone else has the same problem

Part of the problem is likely to be IMAP maildir store bugs
Bug 1182686 Unable to view attachment moved to account with maildir store

Changing Thunderbird Message Store_Type (Options -> Advanced -> General) from maildir -> mbox
Results in:
  • attachments visible after move (Improved functionality)
  • still not saved to a directory if filter run on "Getting new message" (ie Bug persists)
  • Filter save attachment a local directory then move message to a IMAP directory continues to function correctly if run manually (ie unchanged functionality).
The residual behaviour suggests a bug in the IMAP cache code resulting in message date being accessed prior to it being available. A work around which worked for me is:
  1. mail.imap.fetch_by_chunks → false (default true)
  2. setting mail.imap.mime_parts_on_demand → false (Default true)
  3. mail.server.default.mime_parts_on_demand → false (Default true)
I'm yet to test if changing the Chunk setting, or mbox with mime_parts_on_demand was the solution, but doing both does work.
Emale
Posts: 4
Joined: December 4th, 2017, 10:09 am

Re: IMAP save attachement filter bug workarounds

Post by Emale »

FYI:

I had the same problem when using the 'Run file' action along with the 'Save Attachments to' action in filtaQuilla.

FiltaQuilla seems to run the file before the attachments are completely copied to the destination. This causes error 32 - file is in use. (or something like that)

The solution that worked for me was here:

https://www.databasejournal.com/feature ... essing.htm

download the sql function shown on the website(page 1), then incorporate it into your SQL code by calling it (see page 2 of the website)

This worked great for me.
Post Reply