[EXT] XMLTV Guide and Search

Announce and Discuss the Latest Theme and Extension Releases.
Post Reply
User avatar
Benjamin Markson
Posts: 397
Joined: November 19th, 2011, 3:57 am
Location: en-GB

Re: [EXT] XMLTV Guide and Search

Post by Benjamin Markson »

colin1951uk wrote:Legacy may have died but I think the new feed is only as good as the life cycle of Windows 7 as when that comes to pass and Microsoft no longer support the o/s then anything that goes with it will be dropped as well I would have thought.
I believe the legacy feed is for XP (okay dead) and Vista which is supposed to be supported until 2017? There again, I think the whole Microsoft feed is a bit of a hack.
colin1951uk wrote:So running with Schedules Direct is the better policy imo.
My only real bug-bear is the Premiere flag for films. Some providers (C5, for example) don't seem to honour it at all, while others (C4, for example) have lots of false positives - Voyage to the Bottom of the Sea (1961) ain't no Premiere!

Ben.
XUL is dead. Long live the Google Chrome Clones.
User avatar
keithy397
Posts: 2352
Joined: August 29th, 2004, 6:49 pm
Location: North Wales, UK.

Re: [EXT] XMLTV Guide and Search

Post by keithy397 »

I'm having real problems. I did what you stated, it takes a while to execute but the import crashes Firefox every time.

Any suggestions guys?
Cheers,
Keith
User avatar
Benjamin Markson
Posts: 397
Joined: November 19th, 2011, 3:57 am
Location: en-GB

Re: [EXT] XMLTV Guide and Search

Post by Benjamin Markson »

keithy397 wrote:I'm having real problems. I did what you stated, it takes a while to execute but the import crashes Firefox every time.

Any suggestions guys?
I suspect you're trying to import more data than your computer can handle - zip back to this post: http://forums.mozillazine.org/viewtopic ... #p14619215
Benjamin Markson wrote:I would definitely suggest using an mc2xml.chl file to either include the channels you want, or exclude channels you don't. If you look at the mc2xml documentation this file is pretty versatile... it allows you to include/exclude ranges of channels, for example.

The feed really is full of a lot of dross, so this would speed up the fetching of the xmltv.xml file as well.
Yoss follows it up with a handy example.

Ben.
XUL is dead. Long live the Google Chrome Clones.
User avatar
keithy397
Posts: 2352
Joined: August 29th, 2004, 6:49 pm
Location: North Wales, UK.

Re: [EXT] XMLTV Guide and Search

Post by keithy397 »

Ok, so I ran the mc2xml.exe again. I select SD JSON and inserted my area etc. How do I now select the correct channels?
Cheers,
Keith
User avatar
Benjamin Markson
Posts: 397
Joined: November 19th, 2011, 3:57 am
Location: en-GB

Re: [EXT] XMLTV Guide and Search

Post by Benjamin Markson »

keithy397 wrote:How do I now select the correct channels?
You can select the channels to download by manually creating a file called mc2xml.chl in the same folder as mc2xml.exe - see the instructions in post 1 of this thread. :)
Benjamim Marksom wrote:Optionally, in the same folder, create a file called mc2xml.chl - for speed, this file can limit the list of channels you grab, although, you may prefer to use this option later. For example:

Code: Select all

    # includes
    1
    2
    3
    4
    5
    15
    70
    72
    704
This will have the effect of speeding up the execute and massively reducing the size of the resultant file to import.

Yoss gives a more comprehensive example of this file.

Ben.
XUL is dead. Long live the Google Chrome Clones.
User avatar
keithy397
Posts: 2352
Joined: August 29th, 2004, 6:49 pm
Location: North Wales, UK.

Re: [EXT] XMLTV Guide and Search

Post by keithy397 »

I saw the info about putting the channel numbers in the mc2xml.chl file Ben but what are the identifiers per channel? 1 = BBC1 etc. Is there a list in existence or something I can refer to please?

Update:
Even doing a list 1 to 40 still crashes Firefox...... ](*,) ](*,) ](*,)
Cheers,
Keith
User avatar
Benjamin Markson
Posts: 397
Joined: November 19th, 2011, 3:57 am
Location: en-GB

Re: [EXT] XMLTV Guide and Search

Post by Benjamin Markson »

keithy397 wrote:I saw the info about putting the channel numbers in the mc2xml.chl file Ben but what are the identifiers per channel? 1 = BBC1 etc. Is there a list in existence or something I can refer to please?
The channel numbers are the 'regular' numbers used by whatever service you are using, for example, Freeview: https://www.freeview.co.uk/why-freeview/channels [see the List View] - I'm guessing they all produce similar lists?
keithy397 wrote:Even doing a list 1 to 40 still crashes Firefox......
I'm surprised by that. How big is the xmltv.xml file you're trying to import? An unfiltered (all channels) file is a massive 91.5meg in size, whereas a 1-40 channels file is a more manageable 11.5meg in size.

The symptom some people have seen when importing really big files is a huge (albeit temporary) memory spike for Firefox. I can happily import the full 91.5meg file with Firefox spiking at around 2gig during the import. We came to the conclusion before that success may depend on the spec of your computer.

Ben.
XUL is dead. Long live the Google Chrome Clones.
User avatar
keithy397
Posts: 2352
Joined: August 29th, 2004, 6:49 pm
Location: North Wales, UK.

Re: [EXT] XMLTV Guide and Search

Post by keithy397 »

Then I have problem somewhere Ben. When I run the set up with only the .exe .chl and .ren files in the folder, the .xml file populated at a whopping 235mb?!
Is it perhaps the way I've set up SD?
Cheers,
Keith
User avatar
Benjamin Markson
Posts: 397
Joined: November 19th, 2011, 3:57 am
Location: en-GB

Re: [EXT] XMLTV Guide and Search

Post by Benjamin Markson »

keithy397 wrote:...the .xml file populated at a whopping 235mb?!
Ouch.

I don't think the .ren file is going to make any difference to the file size. Can you post a copy of the contents of your .chl [that is: CHL] file?

The 'full' file I described was for Freeview, I guess Messrs Virgin and Sky would be a lot bigger. In any event, it certainly sounds like you are downloading every single channel.

Incidently, when you execute mc2xml.exe, in the DOS box it should report something like: Preparing.... n stations, where n is the number of channels it's about to fetch. This should correspond to the channels you have defined in the .chl file.

Ben.
XUL is dead. Long live the Google Chrome Clones.
User avatar
keithy397
Posts: 2352
Joined: August 29th, 2004, 6:49 pm
Location: North Wales, UK.

Re: [EXT] XMLTV Guide and Search

Post by keithy397 »

At the moment my broadband is kaput and I'm not getting an Engineer until Monday. MONDAY!!
I'm sick of Sky as a provider as it is and this is now the straw that broke my patience. I'll get back to you when I next have a connection.
Cheers...
Cheers,
Keith
User avatar
Benjamin Markson
Posts: 397
Joined: November 19th, 2011, 3:57 am
Location: en-GB

Re: [EXT] XMLTV Guide and Search

Post by Benjamin Markson »

For any that are interested, at the time of typing, the microsoft legacy feed is back.

Ben.
XUL is dead. Long live the Google Chrome Clones.
User avatar
Benjamin Markson
Posts: 397
Joined: November 19th, 2011, 3:57 am
Location: en-GB

Re: [EXT] XMLTV Guide and Search

Post by Benjamin Markson »

The new v1.0.8 release includes a big rewrite of the way I'm importing xmltv.xml files into the database. It's apparent that loading large xml files can have a big memory hit. I'm now processing the file in smaller, more memory friendly, chunks. The bigger the file, the smaller the chunks. Files with more than 100,000 schedule entries get handled 2,500 at a time, 10,001 to 100,000 entries 5,000 at a time, and 10,000 or less entries get written in one go (which is the what was happening previously for all file sizes).

Some of these files can be big. Importing the full Freeview lineup at 90 meg (226 channels with 100,000+ schedules - across over 2 million rows of data!) now stays below the 1 gig mark. I've tested this on a very old Pentium 4 with 2 gig of memory and it took, a somewhat browser lagging, 5 minutes. On a more capable Core 2 with 4 gig of memory it took nearer 2 minutes without the lag, and a Core i3 with 8 gig of memory took it down to about 1 minute.

Hopefully this means that we can be less cautious when importing the xmltv.xml files but I'd still strongly recommend that people only fetch what they need as this is still going to be a lot quicker and more considerate to the upstream servers.

Ben.
XUL is dead. Long live the Google Chrome Clones.
User avatar
Yoss1960
Posts: 119
Joined: January 6th, 2013, 11:23 am

Re: [EXT] XMLTV Guide and Search

Post by Yoss1960 »

Hello Ben,

Thanks for v1.0.8. I've not been around for a while (you may have noticed) and am pleased to see there's been a fair bit of activity.

Unfortunately, in Pale Moon (I don't use Ff) v1.0.8 never imports listings, but will display those already imported. Just noticed v1.0.9 on AMO so tried that; It won't display anything. It won't even allow me to re-enter settings as all buttons (add, refresh, etc) are ghosted.

I have Ff available (for the odd badly programmed web site [usually banks] that won't work in PM) and of course everything works there, v1.0.9 included.

v1.0.7 works in Pale Moon so I'm sticking with that for the time being. Thankfully, the memory issues that were a problem for some don't affect me.
Regards,
¥ø$$1960
"The blindingly obvious is never immediately apparent"
User avatar
Benjamin Markson
Posts: 397
Joined: November 19th, 2011, 3:57 am
Location: en-GB

Re: [EXT] XMLTV Guide and Search

Post by Benjamin Markson »

I'm a bit surprised that v1.0.8 doesn't work as it's not really that different to earlier versions (it uses something called XmlHttpRequest) - the main difference being that it processes the response in smaller chunks.

The new v1.0.9, on the other hand, is using a very different approach (nsIFileInputStream inside a Generator function) - this actually reads the xml file itself in smaller chunks. In this case I don't think PM currently supports Generator functions and I think you will have error(s) in the error console. I don't think FF itself supports this type of function until around FF26.

A quick look at the Pale Moon forums suggests that support for Generator functions is planned - see: The Future of Pale Moon.

Unless you were importing really big files I think the earlier versions worked fine. The only downside will be if I add any new features, although I have nothing planned. My main motivation for all the change is that, by default, mc2xml.exe invites users to download really big files. I'm therefore concerned that a first-time user of the add-on will simply not get it to work and give up.

Of course, if I'm forced onto PM myself I'll be doing a big row-back - I wait to see what carnage ensues when Electrolysis hits my tens of installed add-ons. 8-[

Ben.
XUL is dead. Long live the Google Chrome Clones.
User avatar
Yoss1960
Posts: 119
Joined: January 6th, 2013, 11:23 am

Re: [EXT] XMLTV Guide and Search

Post by Yoss1960 »

I'm a bit surprised that v1.0.8 doesn't work
TBH, it surprised me, too. I thought it had worked, until I saw some "To Be Announced" notices yesterday. So I tried re-importing data. I should have kept the log, but didn't. I can easily get one, if you wish. If memory serves, the log reported Import was started, but I never saw any evidence of it actually happening (no spinner), despite waiting several minutes. I'm guessing it's some quirk of PM's API at fault.

Nice to know the PM people have Generator functions (all Greek to me, TBH) etc. on the horizon.

File import size I deliberately keep as low as possible, for the reasons you state.

I had to look up Electrolysis (1). Hopefully, no negative effects on your add ons. Or anybody else's, come to that.

(1) I don't keep abreast of what Mozilla are up to any more - it usually annoys and/or depresses me, although this change does seem to be for the better.
Regards,
¥ø$$1960
"The blindingly obvious is never immediately apparent"
Post Reply