Have you checked your Quicktime preferences, MIME settings, to make sure m3u files are selected? The default action for downloaded files are to open in the plugin - See: http://kb.mozillazine.org/Opening_files_using_plugins
If that doesn't help, you could try removing the M3U entry in Firefox Download Actions, or try the suggestion here to reset the Firefox Download Actions:
http://kb.mozillazine.org/Changing_medi ... ad_actions
If you still have problems you might be better off starting a new topic with "Mac" in the subject, or trying Apple ITunes support or the Apple ITunes for Mac forum. For example, here is a quote from a recent Apple Itunes forum topic that refers to WAV files, but which you can apply th M3U or other filetypes:
Re: Quicktime VS ITunes
Posted: Jun 3, 2006 2:02 PM in response to: moneykube
monkeykube, first click on a wav file icon to highlight it, then Get Info from the File menu and change the application to open to QuickTime from iTunes and then check change all. Any similar wav file should open with QuickTime thereafter. Also check the wav MIME Setting in the QuickTime System Preferences.