Many charset can not be displayed(lack of CharsetAlias)

Discussion of bugs in Mozilla Thunderbird
Post Reply
User avatar
SoaRex
Posts: 369
Joined: April 20th, 2003, 10:01 pm
Location: Japan, the earth, the solar system

Many charset can not be displayed(lack of CharsetAlias)

Post by SoaRex »

Mail/news can not be displayed correctly if some charsets is specified, for example iso-8859-15, iso-2022-jp , big-5.
Copying charsetalias.properties from 2003-04-09 build has resolved the problem.

Following is res\*.properties file list, which were deleted from size-reduced Thunderbird.
charsetalias.properties
charsetData.properties
langGroups.properties
language.properties
viewer.properties
wincharset.properties
Although charsetalias.properties is sufficient for displaying plain text mail, I think many of above *.properties will be required for HTML mails.
mscott
Posts: 2516
Joined: April 2nd, 2003, 4:10 pm
Location: Thunderbird Research Center, CA
Contact:

Re: Many charset can not be displayed(lack of CharsetAlias)

Post by mscott »

SoaRex wrote:Mail/news can not be displayed correctly if some charsets is specified, for example iso-8859-15, iso-2022-jp , big-5.
Copying charsetalias.properties from 2003-04-09 build has resolved the problem.

Following is res\*.properties file list, which were deleted from size-reduced Thunderbird.
charsetalias.properties
charsetData.properties
langGroups.properties
language.properties
viewer.properties
wincharset.properties
Although charsetalias.properties is sufficient for displaying plain text mail, I think many of above *.properties will be required for HTML mails.


Do you think the fact that I stripped these particular files out of the thunderbird windows build is why Thunderbird has some issues displaying some charsets in HTML?

I can try to make a special build that has these files included if some of you that were complaining about charset issue regressions in Thunderbird over Mozilla Mail could try it out.

It sure would be great if this small thing was the issue! If so, good catch =).
Thunderbirds are Go!
User avatar
SoaRex
Posts: 369
Joined: April 20th, 2003, 10:01 pm
Location: Japan, the earth, the solar system

Re: Many charset can not be displayed(lack of CharsetAlias)

Post by SoaRex »

mscott wrote:Do you think the fact that I stripped these particular files out of the thunderbird windows build is why Thunderbird has some issues displaying some charsets in HTML?

Yes. I think you had removed at least one required file because;
(1) Some Japanese people reported that charset=iso-2022-jp is NOT displayed correctly even though charset=ISO-2022-JP is displayed properly.
(2) Copying of res\charsetalias.properties(this file only) resolved problem on iso-2022-jp and iso-8859-15 case in my environment(5/30 build Thunderbird,Win-Me).
(3) (1)&(2) indicate that lack of charsetalias.properties is the cause of the problem.

But I believe that the real fault is in programing.
Since charsetalias.properties is very important(charset value is case insensitive),
error message should have been issued in order to detect user error such as deletion of important files.

mscott wrote:I can try to make a special build that has these files included if some of you that were complaining about charset issue regressions in Thunderbird over Mozilla Mail could try it out.
It sure would be great if this small thing was the issue! If so, good catch =).

Thanks but I think special build is not so important because we can copy res/*.properties from 4/09 build.
I prefer usual build which includes all required and only required res/*.properties files.
You can observe and verify the problem at newsgroups of news.opera.com as I described in thread of iso-8859-15 case.
iso-8859-15 case : opera.italiano/opera.deutsch at news.opera.com
iso-2022-jp case : opera.japanese at news.opera.com

Best regards, M.Wada
mscott
Posts: 2516
Joined: April 2nd, 2003, 4:10 pm
Location: Thunderbird Research Center, CA
Contact:

Post by mscott »

This fix is checked in and should show up in the next build.
Thunderbirds are Go!
mscott
Posts: 2516
Joined: April 2nd, 2003, 4:10 pm
Location: Thunderbird Research Center, CA
Contact:

Post by mscott »

Can you verify that this is indeed working again in the latest 6-3 windows build?

Thanks!
Thunderbirds are Go!
User avatar
RIV@NVX
Posts: 467
Joined: December 24th, 2002, 7:32 am

Post by RIV@NVX »

With imported mail from Mozilla Mail & Newsgroups, Japanese messages still aren't shown correctly. :(

Tried fresh profile too. Any ideas?
Why would you even consider to use the OS that is older and more obsolete than your computer?
See, that's just one of the reasons why I pick Linux.
User avatar
SoaRex
Posts: 369
Joined: April 20th, 2003, 10:01 pm
Location: Japan, the earth, the solar system

Post by SoaRex »

mscott wrote:Can you verify that this is indeed working again in the latest 6-3 windows build?


2003-06-03 build successfully displayed body part of mail/news with "Content-Type: text/plain; charset=iso-2022-jp".
\res\entityTables\transliterate.properties seems to be the trick :-)
Thanks a lot, Scott.

However, subject text in "Subject:" header still has problem when subject header is encoded in ISO-2022-JP but charset name is specified in small letters, "iso-2022-jp".
Copying of \res\charsetalias.properties was required for proper display of iso-2022-jp encoded subject.
Please note that encoding in Subject: header is independent from charset value in Content-Type: header, which is for body text only.

Regards.
M.Wada
mscott
Posts: 2516
Joined: April 2nd, 2003, 4:10 pm
Location: Thunderbird Research Center, CA
Contact:

Post by mscott »

SoaRex wrote:
mscott wrote:Can you verify that this is indeed working again in the latest 6-3 windows build?


2003-06-03 build successfully displayed body part of mail/news with "Content-Type: text/plain; charset=iso-2022-jp".
\res\entityTables\transliterate.properties seems to be the trick :-)
Thanks a lot, Scott.

However, subject text in "Subject:" header still has problem when subject header is encoded in ISO-2022-JP but charset name is specified in small letters, "iso-2022-jp".
Copying of \res\charsetalias.properties was required for proper display of iso-2022-jp encoded subject.
Please note that encoding in Subject: header is independent from charset value in Content-Type: header, which is for body text only.

Regards.
M.Wada


Thanks for testing this. Does the remaining behavior of the subject header differ from mozilla mail in anyway or are we at least consistant with moz mail?
Thunderbirds are Go!
User avatar
SoaRex
Posts: 369
Joined: April 20th, 2003, 10:01 pm
Location: Japan, the earth, the solar system

Post by SoaRex »

mscott wrote:Does the remaining behavior of the subject header differ from mozilla mail in anyway or are we at least consistant with moz mail?


Thunderbird behavior is completely different from not only Mozilla Mail but also all mailers which can handle and display encoded subject text properly.
Current Thundebird still ignores the rule of "charset name is case insensitive" for Subject: header.

"Copying of \res\charsetalias.properties from 4/09 build to 6/3 build" resolved the remaining iso-2022-jp encoded subject problem.
This indicates that charset name validation logic for encoded Subject: header is different from one for charset name in Content-Type: header.
I guess that some other module(s) still uses some function(s) which requires \res\charsetalias.properties in charset name validation and the decoding process of encoded mail header is included in one of such modules, which you had not successfully modified yet.

Regards.
M.Wada
User avatar
SoaRex
Posts: 369
Joined: April 20th, 2003, 10:01 pm
Location: Japan, the earth, the solar system

6/11 build resolved the problem

Post by SoaRex »

I confirmed that 6/11 build resolved iso-2022-jp encoded subject problem.
I hope many charset related problems are completely resolved by your solution.
Thanks, mscott.

M.Wada from Japan
User avatar
RIV@NVX
Posts: 467
Joined: December 24th, 2002, 7:32 am

Re: 6/11 build resolved the problem

Post by RIV@NVX »

Central European is now resolved too.
Why would you even consider to use the OS that is older and more obsolete than your computer?
See, that's just one of the reasons why I pick Linux.
Post Reply