Help considering move TB to maildir (SSD, ram disc,Dropbox)

[cwinte]

Hi, first post. Have dug about a bit but not found answers/confidence in how to move ahead (or stay as I am)...

My TB profile uses the mbox structure and lives in Dropbox, on SSD in my Macbook Pro and in cloud, this results in many Dropbox updates & actions (though it MIGHT be doing block level updates, not sure on that though).
I have 5 or so active IMAP accounts plus a local structure where I file a small % of mails (but about 15k mails) from the accounts for purposes such as: folder-struct/off-line/search/collation/ref/long term/archival/don't-rely-on-externals. I probably use this local store only a few times a week, most work is live IMAP and multiple device.

I wonder if SSDs will be suffering from read/write/erase patterns in mbox that were certainly not designed with SSDs in mind.

I maintain a ram drive for daily ephemera (such as logs, browser caches, test data and notes) though changes are backed up to permanent external rotating disc media every hour or so. At restarts my last saved ram drive state is rebuilt...

So Option 1: move my TB profile onto the ram drive - no changes, keep mbox. Should be fairly easy, however TB profile 1.2GB is larger than my free ram drive space and reconfiguring not very attractive.
Option 2: as 1 but also build new setup as maildir. From reading it seems I would best do that by making a new profile and setting up my IMAP accounts...
However there is a great deal of config in each of my IMAP setups, exact account options, spam settings.
Plus there is the local store and global settings options that would need to be migrated.

Questions:
For Opt2 what files could usefully be copy-migrated into a new profile to duplicate the original config and white/black/spam list etc data?

How will maildir size of a given 10k mails compare with mbox size of the same mail?

Am I correct that recovering individual files from old backups will be much easier (no need to restore whole mbox and juggle reading from that while running live mail etc)? This assumes that suitable search/metadata would exist (my backups are incremental by file with backup copies of deleted files getting moved into an archive stream, so I can't rely on the meta data living in snapshot indexes etc. I assume I might be having to text search/grep/date locate individual files but feel I'm up to that, it is going to be rare event anyway...

Where can I look to compact/reduce size of the IMAp indexes? (I noted with interest http://kb.mozillazine.org/Minimize_the_ ... _a_profile, and the search text copy of IMAP mails being run on the servers may well mean I have a great deal of not-really-required index data. Data that Spotlight might be able to find directly, though at present Spotlight does not get to operate on my small 1.5GB ram drive...

ANY OTHER POINTS-NOTES-SUGGESTIONS that I should be considering or aware of?
Thanks for your time.

(Sorry if I missed any auto corrects, must have had to type mbox 10 times!)

[JYLD]

If you are using IMAP there is no need to have your profile inside of dropbox folder. Just keep your profile on your SSD drive. That is plenty fast enough. No need to have it in your ram disk. With IMAP your emails are already backed up in the cloud on your email provider's servers. Dropbox is redundant at this point. If you want a back up of your profile itself, get a backup addon such as Moz Backup or a newer one that is still under development. It backs up your profile to a file and you can place that backup file in your dropbox folder.

[cwinte]

Thanks JYLD.
I have had at least one mail provider "migrate" their service with a few weeks notice and then I promptly had NO back history. My policy is "don't rely on them/anyone to do what you need/expect/hope" even when they try they may make mistakes or have fails/go out of biz etc...
I see the ISP server as just one possible source of data, unreliable and not under my control, I also auto forward most emails to a gmail account to increase my odds and for unified long term archival / storage / search. Sometimes I have to go back 10 years... and consider what providers may be around and offering in 10 years time too.
I know the SSD is fast enough but wanted to reduce the ageing read/write cycles that actually seem very frequent (I can see them from trace tools in OS). That would be the benefit of putting it on the Ram drive - fully volatile and I would just copy useful info from the profile there... Something already happening with my other data on the ram drive which all can take some loss of data (e.g. between a backup and crash) without problem (as can my email).
So, I'll look into Moz Backup, thanks for that. Even if I only use it as a guide to what files to copy etc and then take that into one of the tools I already schedule, run, check etc.


I'll also need to trim the profile data to get on the ram drive as I mentioned, I had a few ideas there. This is on a laptop that travels where there is no internet sometimes...

Still very unclear about the maildir feature though... anyone?

[tanstaafl]

There used to be a mbox to maildir built-in converter that I used in the past. It worked well. It was eventually removed. I tried searching for what version it was in (thinking you could do a custom setup of that version in a different directory to convert your account, and then uninstall it and resume using the current version) but I can't find it.

https://superuser.com/questions/1169371 ... to-maildir claims there is an experimental converter in version 60, but I can't find any sign of it. https://support.mozilla.org/ca/kb/novet ... al-feature says to see https://support.mozilla.org/en-US/kb/ma ... hunderbird (which has some errors such as claiming the maildir files use a .eml file extension) . It points to https://www.wilderssecurity.com/threads ... ir.389599/ , instead of to a built-in convertor. Its basically advocating you create a new profile that defaults to maildir and then import mail from the old profile.

A better solution might be to set maildir as the default in tools -> options -> advanced -> general -> message store type for new accounts, delete your existing IMAP accounts and then add them again. It will sync your remote folders in the new accounts, pulling the messages down and saving them using maildir. Note: The contents of your local directory for the deleted IMAP accounts aren't deleted unless you explicitly check the checkbox to delete the accounts data when deleting the account.

If you are short of SSD space think about keeping your profile there but reconfiguring the IMAP account to use the hard disk to store mail instead. If you are working online the slower performance doesn't effect you . It will also make it easier to backup your profile to dropbox since far fewer files get modified.

See http://kb.mozillazine.org/Moving_your_m ... hunderbird

http://kb.mozillazine.org/IMAP_backup might also be useful. Don't overlook getting a free Gmail IMAP account that you use just to backup the inbox (check "getting new mail" checkbox) and sent folders (check "after sending" checkbox) of your 5 existing IMAP accounts using a message filter.

[cwinte]

Thanks for a fulsome reply, I will be working through that.
I already do the Gmail mechanism, though in a different way so that it works as required whether if I am using web, phone or other clients. This allows me to cater for the fact that phones do send and delete folder names differently for instance on some types of account than TB or web clients.
I tried making a maildir default profile built in the ramdrive and pulling my small but most trafficked IMAP to it. The filestore ended up being pretty much exactly the same size (although for 5 or 6 times more files) as maildir than mbox - that's as far as time allowed so far. I'll see how/where I can trim data and how may accounts I can fit... (My scheduled backup scheme will keep the incremental data on externals as & when they are available).
What would be useful would be a way to copy over useful config files such as logins.json and others that would help retain the spam filtering, options and account settings evolved over the years. Maybe they are too interwoven for that to be a safe option... even logins.json looked to maybe contain state info such as last access time that might confuse a fresh install. ANY REFS ON THIS?
My only spinning discs on laptop are external, but not always with me when travelling, so reluctant to use them for main storage (and I guess any disappearing drive and IMAP redownload elsewhere when no external could end up in a real spaghetti to sort out later?)
SSD space is OK, it was the ramdisk space that I only had 1.2GB spare if I was going to fit TB on it. I might continue building in the maildir profile for just the IMAP accounts with no local files, then decide if the local store is needed and maybe move files waiting to go local in an IMAP folder "to local" then at my convenience and when there is enough swap to the profile with local stores and move them. Feels like its not going to be sexy, new download and update of all the IMAP stuff again.
I'll make a first step also to put the profile outside dropbox, I do like the maildir approach as it would allow me to do individual email level recovery from longer term backup (admitted, this is only needed if user error/gmail/ISPs have lost my data).
I seem to recall using a to mbox conversion tool some years ago when I migrated all my history from MS Outlook. But also found pushing all mail from a client up to gmail and back gave something but gmail tags behave very different to folders and that can cause issues, it was also a x-per-day-many-days approach.

[tanstaafl]

See http://kb.mozillazine.org/Files_and_fol ... hunderbird for a description of what each file in the profile is used for.

[cwinte]

wonderful, many thanks!

[wsmwk]

also, dropbox is strictly not supported

[cwinte]

@wsmwk: certainly I had once had conflicted file arise when I concurrently opened TB on another laptop using the same (Dropbox transferred) profile info. I no longer see what it could be doing for me other than being a different cloud backup from my email ISP.

[cwinte]

OP follow up: have put my 3 main IMAP accounts on the ram drive in about 550MB and working very well. No problems food with maildir yet at all.
Oddly on one account I had INBOX set for offline use yet with just a single 250kB email the msg file was 4k and there was no other inbox folder etc. So where was the message? Also text in the message was not seen by search so not include in the global text sql although again this option was set on.
Not a big bug/issue for me, just a puzzlement when I thought I knew what I should observe...

[tanstaafl]

http://kb.mozillazine.org/Entire_messag ... AP_message

Was the message body about 4k, with a approx. 250KB attachment? If so its possible "MIME Parts On Demand" might work differently for a maildir folder than a mbox based folder. i.e. it downloads only the headers and message body unless you open the attachment.

[cwinte]

Thanks for the reply, you may well be correct... I no longer have any of those exact recent mails to hand. Looking at one from 2016 (though only 70k this time) the source shows:
Content-Type: multipart/alternative; boundary=94eb2c045ee050ebfa053e37c6c9
and there do seem to be two parts in that case.

It does not show as attachments in the classic "send a file" manner.
I'm sure more will arrive soon and I will have a look through the source view...

This multipart behaviour would normally be actually pretty smart for me but as I want to scan these visually pretty fast there is a significant delay while the message (html part presumably) starts to arrive, despite being on better than 30Mb fibre the ISP servers always seem slow to pump me data.

Having read the ref you gave I would have expected I would have the full file since I have set "download for offline" on that folder. Maybe that was only done recently - since my move to maildir I have been setting many folder attributes across my accounts. (There are probably 200 folders in all). I quit/restart TB in case it is a pending change of config...