MozillaZine

[EXT] XMLTV Guide and Search

Announce and Discuss the Latest Theme and Extension Releases.
Benjamin Markson

User avatar
 
Posts: 324
Joined: November 19th, 2011, 3:57 am
Location: en-GB

Post Posted July 19th, 2016, 12:43 am

colin1951uk wrote:Legacy may have died but I think the new feed is only as good as the life cycle of Windows 7 as when that comes to pass and Microsoft no longer support the o/s then anything that goes with it will be dropped as well I would have thought.

I believe the legacy feed is for XP (okay dead) and Vista which is supposed to be supported until 2017? There again, I think the whole Microsoft feed is a bit of a hack.

colin1951uk wrote:So running with Schedules Direct is the better policy imo.

My only real bug-bear is the Premiere flag for films. Some providers (C5, for example) don't seem to honour it at all, while others (C4, for example) have lots of false positives - Voyage to the Bottom of the Sea (1961) ain't no Premiere!

Ben.
XUL is dead. Long live the Google Chrome Clones.

keithy397

User avatar
 
Posts: 2339
Joined: August 29th, 2004, 6:49 pm
Location: North Wales, UK.

Post Posted July 19th, 2016, 2:41 am

I'm having real problems. I did what you stated, it takes a while to execute but the import crashes Firefox every time.

Any suggestions guys?
Cheers,
Keith

Benjamin Markson

User avatar
 
Posts: 324
Joined: November 19th, 2011, 3:57 am
Location: en-GB

Post Posted July 19th, 2016, 3:01 am

keithy397 wrote:I'm having real problems. I did what you stated, it takes a while to execute but the import crashes Firefox every time.

Any suggestions guys?

I suspect you're trying to import more data than your computer can handle - zip back to this post: viewtopic.php?p=14619215#p14619215

Benjamin Markson wrote:I would definitely suggest using an mc2xml.chl file to either include the channels you want, or exclude channels you don't. If you look at the mc2xml documentation this file is pretty versatile... it allows you to include/exclude ranges of channels, for example.

The feed really is full of a lot of dross, so this would speed up the fetching of the xmltv.xml file as well.

Yoss follows it up with a handy example.

Ben.
XUL is dead. Long live the Google Chrome Clones.

keithy397

User avatar
 
Posts: 2339
Joined: August 29th, 2004, 6:49 pm
Location: North Wales, UK.

Post Posted July 19th, 2016, 7:07 am

Ok, so I ran the mc2xml.exe again. I select SD JSON and inserted my area etc. How do I now select the correct channels?
Cheers,
Keith

Benjamin Markson

User avatar
 
Posts: 324
Joined: November 19th, 2011, 3:57 am
Location: en-GB

Post Posted July 19th, 2016, 8:29 am

keithy397 wrote:How do I now select the correct channels?

You can select the channels to download by manually creating a file called mc2xml.chl in the same folder as mc2xml.exe - see the instructions in post 1 of this thread. :)

Benjamim Marksom wrote:Optionally, in the same folder, create a file called mc2xml.chl - for speed, this file can limit the list of channels you grab, although, you may prefer to use this option later. For example:

Code: Select all
    # includes
    1
    2
    3
    4
    5
    15
    70
    72
    704

This will have the effect of speeding up the execute and massively reducing the size of the resultant file to import.

Yoss gives a more comprehensive example of this file.

Ben.
XUL is dead. Long live the Google Chrome Clones.

keithy397

User avatar
 
Posts: 2339
Joined: August 29th, 2004, 6:49 pm
Location: North Wales, UK.

Post Posted July 19th, 2016, 1:18 pm

I saw the info about putting the channel numbers in the mc2xml.chl file Ben but what are the identifiers per channel? 1 = BBC1 etc. Is there a list in existence or something I can refer to please?

Update:
Even doing a list 1 to 40 still crashes Firefox...... ](*,) ](*,) ](*,)
Cheers,
Keith

Benjamin Markson

User avatar
 
Posts: 324
Joined: November 19th, 2011, 3:57 am
Location: en-GB

Post Posted July 20th, 2016, 12:42 am

keithy397 wrote:I saw the info about putting the channel numbers in the mc2xml.chl file Ben but what are the identifiers per channel? 1 = BBC1 etc. Is there a list in existence or something I can refer to please?

The channel numbers are the 'regular' numbers used by whatever service you are using, for example, Freeview: https://www.freeview.co.uk/why-freeview/channels [see the List View] - I'm guessing they all produce similar lists?

keithy397 wrote:Even doing a list 1 to 40 still crashes Firefox......

I'm surprised by that. How big is the xmltv.xml file you're trying to import? An unfiltered (all channels) file is a massive 91.5meg in size, whereas a 1-40 channels file is a more manageable 11.5meg in size.

The symptom some people have seen when importing really big files is a huge (albeit temporary) memory spike for Firefox. I can happily import the full 91.5meg file with Firefox spiking at around 2gig during the import. We came to the conclusion before that success may depend on the spec of your computer.

Ben.
XUL is dead. Long live the Google Chrome Clones.

keithy397

User avatar
 
Posts: 2339
Joined: August 29th, 2004, 6:49 pm
Location: North Wales, UK.

Post Posted July 20th, 2016, 2:39 am

Then I have problem somewhere Ben. When I run the set up with only the .exe .chl and .ren files in the folder, the .xml file populated at a whopping 235mb?!
Is it perhaps the way I've set up SD?
Cheers,
Keith

Benjamin Markson

User avatar
 
Posts: 324
Joined: November 19th, 2011, 3:57 am
Location: en-GB

Post Posted July 20th, 2016, 3:24 am

keithy397 wrote:...the .xml file populated at a whopping 235mb?!

Ouch.

I don't think the .ren file is going to make any difference to the file size. Can you post a copy of the contents of your .chl [that is: CHL] file?

The 'full' file I described was for Freeview, I guess Messrs Virgin and Sky would be a lot bigger. In any event, it certainly sounds like you are downloading every single channel.

Incidently, when you execute mc2xml.exe, in the DOS box it should report something like: Preparing.... n stations, where n is the number of channels it's about to fetch. This should correspond to the channels you have defined in the .chl file.

Ben.
XUL is dead. Long live the Google Chrome Clones.

keithy397

User avatar
 
Posts: 2339
Joined: August 29th, 2004, 6:49 pm
Location: North Wales, UK.

Post Posted July 20th, 2016, 7:23 am

At the moment my broadband is kaput and I'm not getting an Engineer until Monday. MONDAY!!
I'm sick of Sky as a provider as it is and this is now the straw that broke my patience. I'll get back to you when I next have a connection.
Cheers...
Cheers,
Keith

Benjamin Markson

User avatar
 
Posts: 324
Joined: November 19th, 2011, 3:57 am
Location: en-GB

Post Posted July 22nd, 2016, 1:48 am

For any that are interested, at the time of typing, the microsoft legacy feed is back.

Ben.
XUL is dead. Long live the Google Chrome Clones.

Benjamin Markson

User avatar
 
Posts: 324
Joined: November 19th, 2011, 3:57 am
Location: en-GB

Post Posted July 27th, 2016, 7:41 am

The new v1.0.8 release includes a big rewrite of the way I'm importing xmltv.xml files into the database. It's apparent that loading large xml files can have a big memory hit. I'm now processing the file in smaller, more memory friendly, chunks. The bigger the file, the smaller the chunks. Files with more than 100,000 schedule entries get handled 2,500 at a time, 10,001 to 100,000 entries 5,000 at a time, and 10,000 or less entries get written in one go (which is the what was happening previously for all file sizes).

Some of these files can be big. Importing the full Freeview lineup at 90 meg (226 channels with 100,000+ schedules - across over 2 million rows of data!) now stays below the 1 gig mark. I've tested this on a very old Pentium 4 with 2 gig of memory and it took, a somewhat browser lagging, 5 minutes. On a more capable Core 2 with 4 gig of memory it took nearer 2 minutes without the lag, and a Core i3 with 8 gig of memory took it down to about 1 minute.

Hopefully this means that we can be less cautious when importing the xmltv.xml files but I'd still strongly recommend that people only fetch what they need as this is still going to be a lot quicker and more considerate to the upstream servers.

Ben.
XUL is dead. Long live the Google Chrome Clones.

Yoss1960

User avatar
 
Posts: 112
Joined: January 6th, 2013, 11:23 am

Post Posted August 2nd, 2016, 5:37 am

Hello Ben,

Thanks for v1.0.8. I've not been around for a while (you may have noticed) and am pleased to see there's been a fair bit of activity.

Unfortunately, in Pale Moon (I don't use Ff) v1.0.8 never imports listings, but will display those already imported. Just noticed v1.0.9 on AMO so tried that; It won't display anything. It won't even allow me to re-enter settings as all buttons (add, refresh, etc) are ghosted.

I have Ff available (for the odd badly programmed web site [usually banks] that won't work in PM) and of course everything works there, v1.0.9 included.

v1.0.7 works in Pale Moon so I'm sticking with that for the time being. Thankfully, the memory issues that were a problem for some don't affect me.
Regards,
¥ø$$1960
"The blindingly obvious is never immediately apparent"

Benjamin Markson

User avatar
 
Posts: 324
Joined: November 19th, 2011, 3:57 am
Location: en-GB

Post Posted August 2nd, 2016, 11:30 am

I'm a bit surprised that v1.0.8 doesn't work as it's not really that different to earlier versions (it uses something called XmlHttpRequest) - the main difference being that it processes the response in smaller chunks.

The new v1.0.9, on the other hand, is using a very different approach (nsIFileInputStream inside a Generator function) - this actually reads the xml file itself in smaller chunks. In this case I don't think PM currently supports Generator functions and I think you will have error(s) in the error console. I don't think FF itself supports this type of function until around FF26.

A quick look at the Pale Moon forums suggests that support for Generator functions is planned - see: The Future of Pale Moon.

Unless you were importing really big files I think the earlier versions worked fine. The only downside will be if I add any new features, although I have nothing planned. My main motivation for all the change is that, by default, mc2xml.exe invites users to download really big files. I'm therefore concerned that a first-time user of the add-on will simply not get it to work and give up.

Of course, if I'm forced onto PM myself I'll be doing a big row-back - I wait to see what carnage ensues when Electrolysis hits my tens of installed add-ons. 8-[

Ben.
XUL is dead. Long live the Google Chrome Clones.

Yoss1960

User avatar
 
Posts: 112
Joined: January 6th, 2013, 11:23 am

Post Posted August 2nd, 2016, 12:00 pm

I'm a bit surprised that v1.0.8 doesn't work


TBH, it surprised me, too. I thought it had worked, until I saw some "To Be Announced" notices yesterday. So I tried re-importing data. I should have kept the log, but didn't. I can easily get one, if you wish. If memory serves, the log reported Import was started, but I never saw any evidence of it actually happening (no spinner), despite waiting several minutes. I'm guessing it's some quirk of PM's API at fault.

Nice to know the PM people have Generator functions (all Greek to me, TBH) etc. on the horizon.

File import size I deliberately keep as low as possible, for the reasons you state.

I had to look up Electrolysis (1). Hopefully, no negative effects on your add ons. Or anybody else's, come to that.

(1) I don't keep abreast of what Mozilla are up to any more - it usually annoys and/or depresses me, although this change does seem to be for the better.
Regards,
¥ø$$1960
"The blindingly obvious is never immediately apparent"

Return to Extension/Theme Releases


Who is online

Users browsing this forum: No registered users and 2 guests