MozillaZine

automated bug info gathering

Discuss application theming and theme development.
mcdavis

User avatar
 
Posts: 3195
Joined: December 9th, 2005, 5:51 am

Post Posted May 3rd, 2013, 1:44 am

http://xkcd.com/1205/

I took a stab at automating some of my manual processes to see if I could save some time, and this seems to be working out. It's quick and dirty code that goes out and gets bug and changeset information for a list of bugs that you give it (e.g., the ones that Alfred posts, which is where I always start) then prints it out in summary form (with dates, changesets, bug descriptions and commit messages) in a way that matches my workflow.

I made it just good enough to get the job done, after which I dropped it like a hot rock. It's Windows-only, and it depends on already having perl and several perl modules installed, and on having a local copy of screen-scraped pushlog info from each of the hg repos of interest. It's not the best code, but it is going to save me a lot of transcribing and copypasting, so that alone is going to be great. Just in case it gives anyone else some ideas, here it is.

If you have any better ideas, please say so. (For example, there may be Bugzilla/hg APIs that give the same information, but I don't know anything about that.)

Code: Select all
#!/usr/bin/perl

# SYNOPSIS ###########################################################
#
# bugdata
#
#   Processes a list of bugs of interest to identify and report relevant hg changesets.
#
# Usage:
#
#   cat buglist.txt | perl -w bugdata.pl > out.txt
#
# Command-line arguments:
#
#    - none -
#
# Relevant files
#
#   bugdata.pl                              - this script
#
#   temporary files, relative to the bugdata root dir:
#
#       temp/showbughtml/wgetbugs.bat       - command file generated by us
#       temp/showbughtml/*.html             - downloaded bug pages from bugzilla (for title and changeset ids)
#       temp/showbughtml/junk.txt           - egrep results, listing changesets for each bug
#       temp/showbughtml/junk1.txt          - egrep results, listing bug title for each bug
#       temp/csethtml/wgetcsets.bat         - command file generated by us
#       temp/csethtml/*.html                - downloaded changesets from hg (for commit message)
#       temp/pushloghtml/wgetpushlogs.bat   - command file generated by us
#       temp/pushloghtml/*.html             - downloaded pushlogs from hg (for date)
#       temp/fullscan/fullscan.bat          - command file generated by us
#       temp/fullscan/junk2.txt             - grep results, listing all references to any bug and any related changeset for all repos across all years
#
#   existing external data, relative to the nlnext root dir:
#
#       m-c-pushlog/                        - directory containing downloaded historical pushlog for the entire m-c repo
#       m-a-pushlog/                        - directory containing downloaded historical pushlog for the entire m-a repo
#       m-b-pushlog/                        - directory containing downloaded historical pushlog for the entire m-b repo
#       m-r-pushlog/                        - directory containing downloaded historical pushlog for the entire m-r repo
#
# Output
#
#   Output written to stdout.
#
# DONE what data structure for the compiled data?
# DONE what final output?
# DONE convert pushlog date to our NNL-ish date format
# DONE change our expected input to begin with bug number list
# DONE extract bug title (replacing any &ndash; with simple hyphen) <title>648675 &ndash; Allow comments in content/plugin crash UI</title>
# DONE MSDOS wget doesn't result in unique saved changeset filenames
# DONE stop using @cset_commands
# DONE stop using @pushlog_commands
# DONE do we want to parse our all-year changeset data?
# DONE produce list of bugs in our META format
# DONE honor the various options
# DONE CLEANUP get rid of extra prints
# DONE what date to use in quicklist A: date should be earliest patch, fxversion should be lowest fx version number containing the patch set, even though date and version may not correspond
# DONE detect historical pushlog files
# DONE write the grep file
# DONE run the grep file
# DONE output the results
# DONE in the historical grep, sort out verbose output from normal
#
# TODO some way to figure out which Fx version corresponds to a pushlog date?
# TODO check for failed wgets
# TODO figure out how to report error and continue on missing filename
# TODO start keeping nlnext/m-r-pushlog/
# TODO get nlnext/m-r-pushlog/2010.release.pushlog.txt
# TODO get nlnext/m-r-pushlog/2011.release.pushlog.txt
# TODO get nlnext/m-r-pushlog/2012.release.pushlog.txt
# TODO get nlnext/m-r-pushlog/2013.release.pushlog.txt
# TODO figure out first m-c push
# DONE figure out first m-a push A: 2011-04-11
# TODO figure out first m-b push
# TODO figure out first m-r push
# TODO check and report last update time for each historical pushlog file.
# TODO report if no files found in historical dir
# TODO update all our historical pushlogs
# TODO in scan_historical_pushlogs() .. change explicit enumeration to repo-indexed arrays?
# TODO in final output, merge historical report into write-ups (grouping per-bug historical data with the write-up for that bug)
# TODO use perl's built-in grep?
# TODO maybe columnate historical pushlog grep output
# TODO use find_matching_files() instead of inline code, throughout
# TODO doc caveats about what you can conclude from the output
# TODO could iterate through historical pushlogs, repeating searches, following each new changeset as it's discovered until the trail runs out (maybe when switching to grouping historical results with bug write-ups)
# TODO turn options into command-line flags
#
# ####################################################################

# GPL'd.  Original Code: mcdavis941@netscape.net



use strict;
use warnings;
use File::Find;
use vars qw/@PDT @MM/;
use Getopt::EvaP;
use Data::Dumper;
use Time::Local;

use constant false => 0;
use constant true  => 1;

# We operate out of one base directory and on fetched data in subdirectories thereunder.
# This is a Windows-oriented representation; would need more work to run on other platforms.
my $basedir_ming_NOTUSED = '/d/mcd/dvl/moztheme/nightlaunchnext/bugdata/';
my $basedrive            = 'd:';
my $basedir              = 'd:/mcd/dvl/moztheme/nightlaunchnext/bugdata/';
my $showbug_subdir       = 'temp/showbughtml/';
my $cset_subdir          = 'temp/csethtml/';
my $pushlog_subdir       = 'temp/pushloghtml/';
my $full_scan_subdir     = 'temp/fullscan/';

# Historical pushlog data, saved locally.  We only read from them, and never write to them; it's maintained separately from this script.
my %historical_dirs = (
    'm-c' => 'd:/mcd/dvl/moztheme/nightlaunchnext/m-c-pushlog/',
    'm-a' => 'd:/mcd/dvl/moztheme/nightlaunchnext/m-a-pushlog/',
    'm-b' => 'd:/mcd/dvl/moztheme/nightlaunchnext/m-b-pushlog/',
    'm-r' => 'd:/mcd/dvl/moztheme/nightlaunchnext/m-r-pushlog/',
);

# We create these temporary files to execute wget commands and grep commands en masse.
my $showbug_batch_filename          = "wgetbugs.bat";
my $cset_batch_filename             = "wgetcsets.bat";
my $pushlog_batch_filename          = "wgetpushlogs.bat";
my $full_scan_batch_filename        = "fullscan.bat";
my $wget_command_prefix             = "wget -E --no-check-certificate ";
my $cset_grep_results_filename      = "junk.txt";
my $title_grep_results_filename     = "junk1.txt";
my $full_scan_grep_results_filename = "junk2.txt";

# Option to fetch data from online source or, alternatively, to work with data we already have.
my $fetch_showbug_html = true;
my $fetch_cset_html    = true;
my $fetch_pushlog_html = true;

# Option to include interim data in final output.
my $verbose = false;

# ---------------------------------------------------------------------------------
# process command line args
# ---------------------------------------------------------------------------------

# ---------------------------------------------------------------------------------
# main routine
# ---------------------------------------------------------------------------------

{
    ###############################################################################################
    #
    # Representation of information for each bug, compiled from various online sources.
    #
    # We use three separate data structures:
    #
    #   1 - an array of bug numbers, ordered as received from input, with duplicates removed.
    #   2 - a hash, keyed by bug number, of data associated with and collected for each bug number.
    #   3 - a hash, keyed by changeset repo + changeset id, of data associated with and collected for each changeset.
    #
    # The data structures are
    #
    #     @buglist = ( '802546', '702532' )
    #
    #     %bug_details =
    #         (
    #         '802546' =>
    #             {
    #             'bugnum'            => '802546',
    #             'bugtitle'          => 'Prettify the Stackframes UI',
    #             'changeset_list'    => [ 'm-c/1234567890ab', 'm-c/444444aaabbb' ],
    #             },
    #         '702532' =>
    #             {
    #             'bugnum'            => '702532',
    #             'bugtitle'          => 'Make something else better.',
    #             'changeset_list'    => [ 'm-c/baba12345678', 'm-a/dd34cc78bbab' ],
    #             },
    #         )
    #
    #     %changeset_details =
    #         (
    #         'm-c/1234567890ab' =>
    #             {
    #             repo        => 'm-c',
    #             changeset   => '1234567890ab',
    #             commitmsg   => 'Bug whatever - fix whatever.',
    #             pushlogdate => 'Mon Mar 18 13:06:48 2013 -0700',
    #             },
    #         'm-c/444444aaabbb' =>
    #             {
    #             repo        => 'm-c',
    #             changeset   => '444444aaabbb',
    #             commitmsg   => 'Bug whatever - fix a major problem.',
    #             pushlogdate => 'Sun Apr 07 02:12:16 2013 -0700',
    #             },
    #         'm-c/baba12345678' =>
    #             {
    #             repo        => 'm-c',
    #             changeset   => 'baba12345678',
    #             commitmsg   => 'Bug whatever - fix something important.',
    #             pushlogdate => 'Wed Apr 10 02:29:04 2013 -0700',
    #             },
    #         'm-a/dd34cc78bbab' =>
    #             {
    #             repo        => 'm-a',
    #             changeset   => 'dd34cc78bbab',
    #             commitmsg   => 'Bug whatever - fix something else.',
    #             pushlogdate => 'Thu Apr 11 06:08:46 2013 -0700',
    #             },
    #          )
    #
    ###############################################################################################

    my @buglist = ();
    my %bug_details = ();
    my %changeset_details = ();

    # Working variables.
    my @showbug_commands = ();
    my $output_filename;
    my $repo;
    my $bugnum;
    my $bugtitle;
    my $cset;

    ###############################################################################################
    #
    # Read list of bug numbers to process from standard input.
    #
    ###############################################################################################

    my @bug_input = ();
    while(<>) {
        chomp;
        s/\s//g;
        next if /^#/;   # skip comment lines
        next if /^$/;   # skip empty lines
        if(/^([\d]+)$/) {
            print "Got bug $1\n" if $verbose;
            push @bug_input, $1;
        } else {
            die "Bug list contains non-bug data: $_\n";
        }
    }
    @buglist = uniq(@bug_input);

    if ($verbose) {
        print "=====================================================================\n";
        print "Sanitized Incoming Bug List:\n";
        print "---------------------------------------------------------------------\n";
        print "$_\n" for @buglist;
    }

    # Initialize bug details.

    $bug_details{$_} = { "bugnum" => $_, "bugtitle" => "", "changeset_list" => [] } for @buglist;
    print Data::Dumper::Dumper( \%bug_details ) if $verbose;

    ###############################################################################################
    #
    # Pull bug info for each bug from online bugzilla with wget and save results locally,
    # then grep them all to extract bug title and changeset information.
    #
    # Because we're on Windows with its StupidShell(tm), write out a batch file and execute that
    # to fetch them all at once.  For the same reason, include the grep commands in the batch
    # file too.
    #
    # The form of the wget command is:
    #
    #   wget -E --no-check-certificate https://bugzilla.mozilla.org/show_bug.cgi?id=648675
    #
    # The grep commands are:
    #
    #   egrep "mozilla-central/rev|releases/mozilla-aurora/rev|releases/mozilla-beta/rev|releases/mozilla-release" *.html > junk.txt
    #   egrep "<title>.*?</title>" *.html > junk1.txt
    #
    ###############################################################################################

    $output_filename = $basedir . $showbug_subdir . $showbug_batch_filename;
    open(OUTPUTFILE, "> $output_filename") || die "Unable to open $output_filename.\n";
    print OUTPUTFILE "\@echo off\n";
    print OUTPUTFILE "$basedrive\n";
    print OUTPUTFILE "cd $basedir$showbug_subdir\n";
    print OUTPUTFILE "del .\\*.html /Q\n" if $fetch_showbug_html;
    print OUTPUTFILE "del .\\*.txt /Q\n" if $fetch_showbug_html;
    if ($fetch_showbug_html) {
        print OUTPUTFILE "$wget_command_prefix https://bugzilla.mozilla.org/show_bug.cgi?id=$_\n" for @buglist;
    }
    print OUTPUTFILE "egrep \"mozilla-central/rev|releases/mozilla-aurora/rev|releases/mozilla-beta/rev|releases/mozilla-release\" *.html > $basedir$showbug_subdir$cset_grep_results_filename\n";
    print OUTPUTFILE "egrep \"<title>.*?</title>\" *.html > $basedir$showbug_subdir$title_grep_results_filename\n";
    close(OUTPUTFILE);

    if ($verbose) {
        open SRCFILE, "<$output_filename" or die "Error: can't open $output_filename";
        # read in the entire file in one big gulp
        undef $/;
        my $entire_file = <SRCFILE>;
        $/ = "\n";
        close(SRCFILE);
        print "=====================================================================\n";
        print "$output_filename\n";
        print "---------------------------------------------------------------------\n";
        print $entire_file;
    }

    system($output_filename);

    if ($verbose) {
        open SRCFILE, "<$basedir$showbug_subdir$cset_grep_results_filename" or die "Error: can't open $basedir$showbug_subdir$cset_grep_results_filename";
        # read in the entire file in one big gulp
        undef $/;
        my $entire_file = <SRCFILE>;
        $/ = "\n";
        close(SRCFILE);
        print "=====================================================================\n";
        print "$basedir$showbug_subdir$cset_grep_results_filename\n";
        print "---------------------------------------------------------------------\n";
        print $entire_file;
    }

    if ($verbose) {
        open SRCFILE, "<$basedir$showbug_subdir$title_grep_results_filename" or die "Error: can't open $basedir$showbug_subdir$title_grep_results_filename";
        # read in the entire file in one big gulp
        undef $/;
        my $entire_file = <SRCFILE>;
        $/ = "\n";
        close(SRCFILE);
        print "=====================================================================\n";
        print "$basedir$showbug_subdir$title_grep_results_filename\n";
        print "---------------------------------------------------------------------\n";
        print $entire_file;
    }

    ###############################################################################################
    #
    # Process the result of grepping showbug html and extract bug title and changeset data.
    #
    ###############################################################################################

    if ($verbose) {
        print "=====================================================================\n";
        print "Extracting changeset info from bugs\n";
        print "---------------------------------------------------------------------\n";
    }

    open SRCFILE, "<$basedir$showbug_subdir$cset_grep_results_filename" or die "Error: can't open $cset_grep_results_filename";

    while(<SRCFILE>) {
        $bugnum = "";
        $repo = "";
        $cset = "";
        chomp;

        if (/^show_bug\.cgi\@id=([\d]+)\.html:/ ) {
            s/^show_bug\.cgi\@id=([\d]+)\.html:/Bug $1: /;
            $bugnum = $1;
        }
        s/<pre class="bz_comment_text" >//g;
        s/<\/pre>//g;

        # detect repo and changeset
        if (/hg\.mozilla\.org\/mozilla-central\/rev\/([0-9a-fA-F]{12})/) {
            $repo = "m-c";
            $cset = $1;
        }
        if (/hg\.mozilla\.org\/releases\/mozilla-aurora\/rev\/([0-9a-fA-F]{12})/) {
            $repo = "m-a";
            $cset = $1;
        }
        if (/hg\.mozilla\.org\/releases\/mozilla-beta\/rev\/([0-9a-fA-F]{12})/) {
            $repo = "m-b";
            $cset = $1;
        }
        if (/hg\.mozilla\.org\/releases\/mozilla-release\/rev\/([0-9a-fA-F]{12})/) {
            $repo = "m-r";
            $cset = $1;
        }

        if ($repo and $cset) {
            my $scoped_changeset_id = $repo . "/" . $cset;
            # We do NOT do uniqueness tests here, because the input is messy. (We don't control
            # what people type into Bugzilla and we have no way to programmatically disambiguate it.
            # We'll live with the ambiguity and repetition.)
            push @{$bug_details{$bugnum}->{'changeset_list'}}, $scoped_changeset_id;
            $changeset_details{$scoped_changeset_id} = { 'changeset' => $cset, 'repo' => $repo, 'commitmsg' => 'Unable to find commit message.', 'pushlogdate' => 'Unable to find pushlog date.' };
        }

        print if $verbose;
        print " repo:" . $repo . ", cset:" . $cset if ($repo ne "" and $verbose);
        print "\n" if $verbose;
    }
    close(SRCFILE);

    if ($verbose) {
        print "=====================================================================\n";
        print "Extracting bug title info from saved bugs\n";
        print "---------------------------------------------------------------------\n";
    }

    open SRCFILE, "<$basedir$showbug_subdir$title_grep_results_filename" or die "Error: can't open $title_grep_results_filename";

    while(<SRCFILE>) {
        $bugnum = "";
        $bugtitle = "";
        chomp;

        if (/^show_bug\.cgi\@id=([\d]+)\.html:\s*<title>\d+ \&ndash; (.*?)<\/title>/ ) {
            # s/^show_bug\.cgi\@id=([\d]+)\.html:/Bug $1: /;
            $bugnum = $1;
            $bugtitle = $2;
            $bug_details{$bugnum}->{'bugtitle'} = $bugtitle;
        }

        print if $verbose;
        print " bugnum:" . $bugnum . ", bugtitle:" . $bugtitle if ($verbose);
        print "\n" if $verbose;
    }
    close(SRCFILE);

    ###############################################################################################
    #
    # Pull changeset info for each bug from online hg with wget and save results locally.
    # The resulting files are then read one by one by us.
    #
    # Because we're on Windows with its StupidShell(tm), write out a batch file and execute that
    # to fetch them all at once.
    #
    # The form of the wget command, depending on the repo in question, is:
    #
    #   wget -E --no-check-certificate https://hg.mozilla.org/mozilla-central/rev/2779fab6ba58 -O m-c-2779fab6ba58.html
    #   wget -E --no-check-certificate https://hg.mozilla.org/releases/mozilla-aurora/rev/df268e1e0987 -O m-a-df268e1e0987.html
    #   wget -E --no-check-certificate https://hg.mozilla.org/releases/mozilla-beta/rev/df268e1e0987 -O m-b-df268e1e0987.html
    #   wget -E --no-check-certificate https://hg.mozilla.org/releases/mozilla-release/rev/df268e1e0987 -O m-r-df268e1e0987.html
    #
    # Note that changeset ids are only unique within their associated repos, and not universally unique.
    # To avoid name collisions of downloaded files, we tell wget to save to a filename that includes both repo
    # and changeset information.
    #
    ###############################################################################################

    $output_filename = $basedir . $cset_subdir . $cset_batch_filename;
    open(OUTPUTFILE, "> $output_filename") || die "Unable to open $output_filename.\n";
    print OUTPUTFILE "\@echo off\n";
    print OUTPUTFILE "$basedrive\n";
    print OUTPUTFILE "cd $basedir$cset_subdir\n";
    print OUTPUTFILE "del .\\*.html /Q\n" if $fetch_cset_html;
    if ($fetch_cset_html) {
        foreach my $key (sort(keys %changeset_details)) {
            print OUTPUTFILE gen_changeset_wget_command( $changeset_details{$key}->{'repo'}, $changeset_details{$key}->{'changeset'} ) . "\n";
        }
    }
    close(OUTPUTFILE);

    if ($verbose) {
        open SRCFILE, "<$output_filename" or die "Error: can't open $output_filename";
        # read in the entire file in one big gulp
        undef $/;
        my $entire_file = <SRCFILE>;
        $/ = "\n";
        close(SRCFILE);
        print "=====================================================================\n";
        print "$output_filename\n";
        print "---------------------------------------------------------------------\n";
        print $entire_file;
    }

    # Execute our batch file.
    system($output_filename);

    if ($verbose) {
        my @src_file_list = ();
        find(
            \&{sub {
                # my @results = ();
                if( not -d $File::Find::name and
                    $File::Find::name =~ /\.html$/
                    ) {
                    push @src_file_list, $File::Find::name;
                    }
                }},
            $basedir . $cset_subdir
            );

        print "=====================================================================\n";
        print "Downloaded Changeset Files:\n";
        print "---------------------------------------------------------------------\n";
        print map { "$_\n" } sort @src_file_list;
    }

    ###############################################################################################
    #
    # Pull pushlog info for each changeset from online hg with wget and save results locally,
    # The resulting files are then read one by one by us.
    #
    # Because we're on Windows with its LamerShell(tm), write out a batch file and execute that
    # to fetch them all at once.
    #
    # To determine pushlog URL for corresponding changeset:
    #
    #   if mozilla-central/rev/f22b5313db35 then    https://hg.mozilla.org/mozilla-central/pushloghtml?changeset=f22b5313db35
    #   if mozilla-aurora/rev/f22b5313db35  then    https://hg.mozilla.org/releases/mozilla-aurora/pushloghtml?changeset=f22b5313db35
    #   if mozilla-beta/rev/f22b5313db35    then    https://hg.mozilla.org/releases/mozilla-beta/pushloghtml?changeset=f22b5313db35
    #   if mozilla-release/rev/f22b5313db35 then    https://hg.mozilla.org/releases/mozilla-release/pushloghtml?changeset=f22b5313db35
    #
    # The form of the wget command, depending on the repo in question, is:
    #
    #   wget -E --no-check-certificate https://hg.mozilla.org/mozilla-central/pushloghtml?changeset=528411b6f628 -O m-c-528411b6f628.html
    #   wget -E --no-check-certificate https://hg.mozilla.org/releases/mozilla-aurora/pushloghtml?changeset=f22b5313db35 -O m-a-f22b5313db35.html
    #   wget -E --no-check-certificate https://hg.mozilla.org/releases/mozilla-beta/pushloghtml?changeset=8ac5f560ac98 -O m-b-8ac5f560ac98.html
    #   wget -E --no-check-certificate https://hg.mozilla.org/releases/mozilla-release/pushloghtml?changeset=8ac5f560ac98 -O m-r-8ac5f560ac98.html
    #
    # Note that changeset ids are only unique within their associated repos, and not universally unique.
    # To avoid name collisions of downloaded files, we tell wget to save to a filename that includes both repo
    # and changeset information.
    #
    ###############################################################################################

    $output_filename = $basedir . $pushlog_subdir . $pushlog_batch_filename;
    open(OUTPUTFILE, "> $output_filename") || die "Unable to open $output_filename.\n";
    print OUTPUTFILE "\@echo off\n";
    print OUTPUTFILE "$basedrive\n";
    print OUTPUTFILE "cd $basedir$pushlog_subdir\n";
    if ($fetch_pushlog_html) {
        print OUTPUTFILE "del .\\*.html /Q\n";
        foreach my $key (sort(keys %changeset_details)) {
            print OUTPUTFILE gen_pushlog_wget_command( $changeset_details{$key}->{'repo'}, $changeset_details{$key}->{'changeset'} ) . "\n";
        }
    }
    close(OUTPUTFILE);

    if ($verbose) {
        open SRCFILE, "<$output_filename" or die "Error: can't open $output_filename";
        # read in the entire file in one big gulp
        undef $/;
        my $entire_file = <SRCFILE>;
        $/ = "\n";
        close(SRCFILE);
        print "=====================================================================\n";
        print "$output_filename\n";
        print "---------------------------------------------------------------------\n";
        print $entire_file;
    }

    # Execute our batch file.
    system($output_filename);

    if ($verbose) {
        my @src_file_list = ();
        find(
            \&{sub {
                # my @results = ();
                if( not -d $File::Find::name and
                    $File::Find::name =~ /\.html$/
                    ) {
                    push @src_file_list, $File::Find::name;
                    }
                }},
            $basedir . $pushlog_subdir
            );

        print "=====================================================================\n";
        print "Downloaded Pushlog Files:\n";
        print "---------------------------------------------------------------------\n";
        print map { "$_\n" } sort @src_file_list;
    }

    ###############################################################################################
    #
    # Extract commit message from individual changesets.
    #
    ###############################################################################################

    my $re_commit_msg = qr/<div class="title">\n(.*)<span class="logtags"><\/span>\n<\/div>/mi;
    my $commit_msg;

    if ($verbose) {
        print "=====================================================================\n";
        print "Extracting commit messages\n";
        print "---------------------------------------------------------------------\n";
    }

    foreach my $key (sort(keys %changeset_details)) {
        my $srcfilename = $basedir . $cset_subdir . compose_local_changeset_filename( $changeset_details{$key}->{'repo'}, $changeset_details{$key}->{'changeset'} );
        print "Reading $srcfilename\n" if $verbose;
        open SRCFILE, "<$srcfilename" or die "Error: can't open $srcfilename";

        # read in the entire file in one big gulp
        undef $/;
        my $entire_file = <SRCFILE>;
        $/ = "\n";

        close(SRCFILE);

        # print "----------------------\n";
        # print $entire_file;

        if ($entire_file =~ $re_commit_msg) {
            $commit_msg = $1;
            $commit_msg =~ s/<a.*?>//gmi;
            $commit_msg =~ s/<\/a>//gmi;
            $commit_msg =~ s/\n/ /gm;
            $commit_msg =~ s/ +$//gm;

            print $commit_msg . "\n" if $verbose;

            $changeset_details{$key}->{'commitmsg'} = $commit_msg;
         }
    }

    ###############################################################################################
    #
    # Extract commit date from changeset's pushlog.
    #
    ###############################################################################################

    my $re_commit_date = qr/<span class="date">(.*?)<\/span>/mi;
    my $commit_date;

    if ($verbose) {
        print "=====================================================================\n";
        print "Extracting commit dates\n";
        print "---------------------------------------------------------------------\n";
    }

    foreach my $key (sort(keys %changeset_details)) {
        my $srcfilename = $basedir . $pushlog_subdir . compose_local_pushlog_filename( $changeset_details{$key}->{'repo'}, $changeset_details{$key}->{'changeset'} );
        print "Reading $srcfilename\n" if $verbose;
        open SRCFILE, "<$srcfilename" or die "Error: can't open $srcfilename";

        # read in the entire file in one big gulp
        undef $/;
        my $entire_file = <SRCFILE>;
        $/ = "\n";

        close(SRCFILE);

        # print "----------------------\n";
        # print $entire_file;

        if ($entire_file =~ $re_commit_date) {
            $commit_date = $1;
            print $commit_date . " (" . format_patch_date($commit_date) .")" . "\n" if $verbose;

            $changeset_details{$key}->{'pushlogdate'} = $commit_date;
        }
    }

    ###############################################################################################
    #
    # Scan for bug and changeset references through all saved historical pushlog data.
    #
    ###############################################################################################

    scan_historical_pushlogs(\@buglist, \%bug_details, \%changeset_details);

    ###############################################################################################
    #
    # Finally, the moment we've all been waiting for.
    #
    ###############################################################################################

    output_bug_quicklist(\@buglist, \%bug_details, \%changeset_details);
    output_meta_buglist(\@buglist, \%bug_details, \%changeset_details);
    output_bug_summaries(\@buglist, \%bug_details, \%changeset_details);
    output_historical_pushlog_scan_results(\@buglist, \%bug_details, \%changeset_details);
}


# ---------------------------------------------------------------------------------
# subroutines
# ---------------------------------------------------------------------------------

# Outputs bug information in NNL-specific write-up format.
sub output_bug_summaries {
    my( $r_buglist, $r_bug_details, $r_changeset_details ) = @_;

    print "=====================================================================\n";
    print "Final Report\n";
    print "---------------------------------------------------------------------\n";

    print Data::Dumper::Dumper( $r_bug_details ) if $verbose;
    print Data::Dumper::Dumper( $r_changeset_details ) if $verbose;

    foreach my $bugnum (@$r_buglist) {
        print "TOPIC: Handle Bug " . $bugnum . " - ";
        if ($$r_bug_details{$bugnum}->{"bugtitle"}) {
          print $$r_bug_details{$bugnum}->{"bugtitle"};
        }
        else {
          print "no bug title";
        }
        print "\n";
        print "TAGS: \n";
        print "=========================================================================================================================\n";
        print "patch:\n";
        print "** REQUIRES EYES-ON CONFIRMATION **\n";

        foreach my $csetid (@{$$r_bug_details{$bugnum}->{'changeset_list'}}) {
          print "- (GUESSING:" . format_patch_date($$r_changeset_details{$csetid}->{'pushlogdate'}) . ") (FxNN) " . gen_changeset_url($$r_changeset_details{$csetid}->{'repo'}, $$r_changeset_details{$csetid}->{'changeset'}) . " () " . $$r_changeset_details{$csetid}->{'commitmsg'} . "\n";
        }

        print "outcome:\n";
        print "NNL impact:\n";
        print "- TODO\n";
        print "\n";
    }
}

sub uniq {
    my %seen;
    grep !$seen{$_}++, @_
}

# Converts this:    Mon Mar 18 13:06:48 2013 -0700
# into this:        2013-03-18
#
sub format_patch_date {
    my( $changeset_date ) = @_;
    my $retval = "-bad date-";

    my %mon = (
        'JAN' => '01',
        'FEB' => '02',
        'MAR' => '03',
        'APR' => '04',
        'MAY' => '05',
        'JUN' => '06',
        'JUL' => '07',
        'AUG' => '08',
        'SEP' => '09',
        'OCT' => '10',
        'NOV' => '11',
        'DEC' => '12',
        );

    if ($changeset_date =~ /[a-zA-Z]{3} (Jan|Feb|Mar|Apr|May|Jun|Jul|Aug|Sep|Oct|Nov|Dec) (\d\d) \d\d:\d\d:\d\d (\d\d\d\d)/) {
      $retval = $3 . "-" . $mon{uc($1)} . "-" . $2;
    }
    return $retval;
}

# Returns a complete wget command for retrieving the bug page for a specific given bug from bugzilla.
sub gen_showbug_wget_command {
    my( $bugnum ) = @_;
    return 'wget -E --no-check-certificate ' . gen_showbug_url($bugnum);
}

# Returns a complete wget command for retrieving a given changeset from hg.
sub gen_changeset_wget_command {
    my( $repo, $cset ) = @_;
    return 'wget -E --no-check-certificate ' . gen_changeset_url($repo, $cset) . ' -O ' . compose_local_changeset_filename($repo, $cset);
}

# Returns a complete wget command for retrieving the pushlog for a given changeset from hg.
sub gen_pushlog_wget_command {
    my( $repo, $cset ) = @_;
    return 'wget -E --no-check-certificate ' . gen_pushlog_url($repo, $cset) . ' -O ' . compose_local_pushlog_filename($repo, $cset);
}

# Returns the url for a bug page for a specific given bug from bugzilla.
sub gen_showbug_url {
    my( $bugnum ) = @_;
    return 'https://bugzilla.mozilla.org/show_bug.cgi?id=' . $bugnum;
}

# Returns the url for a specific given changeset on hg.
sub gen_changeset_url {
    my( $repo, $cset ) = @_;
    return hg_root_for_repo($repo) . 'rev/' . $cset;
}

# Returns the url for the pushlog for a specific given changeset on hg.
sub gen_pushlog_url {
    my( $repo, $cset ) = @_;
    return hg_root_for_repo($repo) . 'pushloghtml?changeset=' . $cset;
}

# Given repo and changetset, returns the filename to use for a locally-saved copy of that changeset from hg.
# It's a bare filename (just the name, not the full path) and it's just the repo signifier plus the changeset id,
# with an html extensions, e.g. m-c-1234567890ab.html.
sub compose_local_changeset_filename {
    my( $repo, $cset ) = @_;
    return $repo . '-' . $cset . '.html';
}

# Given repo and changetset, returns the filename to use for a locally-saved copy of that changeset's pushlog from hg.
# It's a bare filename (just the name, not the full path) and it's just the repo signifier plus the changeset id,
# with an html extensions, e.g. m-c-1234567890ab.html.
sub compose_local_pushlog_filename {
    my( $repo, $cset ) = @_;
    return $repo . '-' . $cset . '.html';
}

# Given repo signifier, returns URL for that repo's root on hg.
sub hg_root_for_repo {
    my( $repo ) = @_;
    my %repo_roots = (
        'm-c' => 'https://hg.mozilla.org/mozilla-central/',
        'm-a' => 'https://hg.mozilla.org/releases/mozilla-aurora/',
        'm-b' => 'https://hg.mozilla.org/releases/mozilla-beta/',
        'm-r' => 'https://hg.mozilla.org/releases/mozilla-release/',
    );
    return $repo_roots{$repo};
}

# Outputs bug information in NNL-specific "meta" digest overview format.
sub output_meta_buglist {
    my( $r_buglist, $r_bug_details, $r_changeset_details ) = @_;
    my $encoded_cset;
    my $repo;
    my $cset;

    print "=====================================================================\n";
    print "Meta Buglist\n";
    print "---------------------------------------------------------------------\n";

    print "\n";
    print "    Imp Rv  IThm    NLN Sta BugNum      Date [1]        Res         Appver          Platform[2] Cset            Description                                                                                                                \n";
    print "    --- --- ------- ------- ----------- --------------- ----------- --------------- ----------- --------------- ---------------------------------------------------------------------------------------------------------------------------\n";
    print "\n";
    print "    -   -   ?       ????    Bug ??????  (??????????)    ?           Fx??            All         ????????????    - ??????                                                                                                                   \n";
    print "\n";

    foreach my $bugnum (@$r_buglist) {
        foreach my $csetid (@{$$r_bug_details{$bugnum}->{'changeset_list'}}) {
          # Prefix with repo signifier if other than m-c, and right-pad if m-c so that following columns align.
          # Yes, this means csets on repos other than m-c are slightly inset, intended to catch the reader's eye.
          $repo = $$r_changeset_details{$csetid}->{'repo'};
          $cset = $$r_changeset_details{$csetid}->{'changeset'};
          if ($repo eq 'm-a') {
            $encoded_cset = 'a/' . $cset;
          }
          elsif ($repo eq 'm-b') {
            $encoded_cset = 'b/' . $cset;
          }
          elsif ($repo eq 'm-r') {
            $encoded_cset = 'r/' . $cset;
          }
          else {
            # m-c and default
            $encoded_cset = $cset . '  ';
          }
          # print "Bugnum:$bugnum\n";
          printf( "    -   -   ?       TODO    Bug %6d  (%s)    ?           Fx??            ?           %s  - %s\n", $bugnum, format_patch_date($$r_changeset_details{$csetid}->{'pushlogdate'}), $encoded_cset, $$r_bug_details{$bugnum}->{'bugtitle'} );
        }
    }
    print "\n";
}

# Outputs bug information in NNL-specific "Quick List" format.
sub output_bug_quicklist {
    my( $r_buglist, $r_bug_details, $r_changeset_details ) = @_;
    my $guessed_first_patch_date = "- unable to guess date -";

    print "=====================================================================\n";
    print "Bug Quicklist\n";
    print "---------------------------------------------------------------------\n";

    print "\n";
    print "Note: dates may be incorrect and need to be checked.  They may well be based on backed-out patches.\n";

    # It's unclear which date we want here.
    # This date is used to determine the order in which we should address bugs.
    # Want:
    # - earliest date
    # - that corresp to a cset that sticks
    # - that's on m-c (or do we want furthest-downstream repo on which it lands?)
    # The most common case is that
    # - there's one patch
    # - if there's more than one they all have the same date (because they land as part of the same push), and
    # - all stick on first landing
    # in which case the date of the first patch detected is as good as any (presuming we preserve order of detection in our representation).
    # However, this will fail if patches are backed out, which is not uncommon.
    # That makes the date a good first guess which must be verified.

    foreach my $bugnum (@$r_buglist) {
        foreach my $csetid (@{$$r_bug_details{$bugnum}->{'changeset_list'}}) {
            $guessed_first_patch_date = format_patch_date($$r_changeset_details{$csetid}->{'pushlogdate'});
            print "guessing quicklist date: $guessed_first_patch_date\n" if $verbose;
            last;
        }
        printf( "TODO (GUESSING:%s) (FxNN) Bug %6d - %s\n", $guessed_first_patch_date, $bugnum, $$r_bug_details{$bugnum}->{'bugtitle'} );
    }
    print "\n";
}

# TODO this whole comment needs to be rewritten.
#
# Outputs grep command search terms for each bug.
# This is defined as, for each repo, the bug number or any changeset ids we've identified as associated with the bug.
# The result is a set of grep searches to be run on each repo, specific to each repo.
# We always search each repo for bug number, and we also search each repo for any changeset ids we've identified
# as associated with that bug in that repo.
# Note that grepping for bug number will occasionally turn up matches on changeset ids containing bug number
# as a substring.
# This grep is done to gather all information about patches for each bug to help determine the actual changeset list.
# Historical pushlog data is screenscraped for a data-ranged query of the whole-repo pushlog.
# It's organized into files, each file containing the data for a full year, as in the following:
# There may not be any files at all, or there may be more.  To work with whatever we have, we assume simply that all files in the directory contain historical pushlog data and should be searched, that they're text files, and that they begin with year number.
#
#   m-c-pushlog/
#     2010.central.pushlog.txt
#     2011.central.pushlog.txt
#     2012.central.pushlog.txt
#     2013.central.pushlog.txt
#
# Sample output for a bug for which we've identified candidate changesets in m-c and m-a only:
#
#   grep m-c: 802546|1234567890ab|444444aaabbb
#   grep m-a: 802546|dd34cc78bbab
#   grep m-b: 802546
#   grep m-r: 802546
#
# Generated grep commands have the following form.
#
#   d:\MCD\dvl\moztheme\nightlaunchnext\bugdata>grep -EiH "858759|d1264794ca7e" D:/MCD/dvl/moztheme/nightlaunchnext/m-c-pushlog/2013.all.pushlog.txt
#
# and produce output like the following (with indentation from pushlog txt file).
#
#       d1264794ca7e    Anton Kovalyov â?" Bug 858759 - Move profiler.css to browser/themes. r=vporof
#
# although some have a pushlog date prepended, because they are the first or only changeset for a give push, like the following:
#
#   Tue Jan 15 08:26:10 2013 -0800  8233d14cdc57    Christian Sonne — Bug 811469 - Indicator progress bar gradient leaks into border. r=dolske a=gavin
#
# Approach, in general:
#
#   for each bug
#     for each historical pushlog repo
#       for each historical data file in that directory
#           scan for references to this bug number and for any known changesets on this repo for this bug

sub scan_historical_pushlogs {
    my( $r_buglist, $r_bug_details, $r_changeset_details ) = @_;
    my @repos = ('m-c', 'm-a', 'm-b', 'm-r');
    my @m_c_changesets = ();
    my @m_a_changesets = ();
    my @m_b_changesets = ();
    my @m_r_changesets = ();
    my $m_c_pattern = '';
    my $m_a_pattern = '';
    my $m_b_pattern = '';
    my $m_r_pattern = '';
    my $output_filename;
    my @temp_list;

    my %historical_pushlog_files = (
        'm-c' => [],
        'm-a' => [],
        'm-b' => [],
        'm-r' => [],
    );

    if ($verbose) {
        print "=====================================================================\n";
        print "Writing Historical Pushlog Grep Commands\n";
        print "---------------------------------------------------------------------\n";
        print "\n";
    }

    # Get the list of historical pushlog files to grep against.
    foreach my $repo (@repos) {
      print $historical_dirs{$repo} . "\n" if $verbose;
      # Use helper variable to establish list context for returned value.
      @temp_list = find_matching_files($historical_dirs{$repo}, '.txt$');
      $historical_pushlog_files{$repo} = [ sort @temp_list ];
      print join(", ", @{$historical_pushlog_files{$repo}}) . "\n" if $verbose;
    }

    $output_filename = $basedir . $full_scan_subdir . $full_scan_batch_filename;
    open(OUTPUTFILE, "> $output_filename") || die "Unable to open $output_filename.\n";
    print OUTPUTFILE "\@echo off\n";
    print OUTPUTFILE "$basedrive\n";
    print OUTPUTFILE "cd $basedir$full_scan_subdir\n";

    foreach my $bugnum (@$r_buglist) {

        # Determine changesets of interest for this bug, grouped by repo.

        @m_c_changesets = ();
        @m_a_changesets = ();
        @m_b_changesets = ();
        @m_r_changesets = ();
        foreach my $csetid (@{$$r_bug_details{$bugnum}->{'changeset_list'}}) {
            if ($$r_changeset_details{$csetid}->{'repo'} eq 'm-c') {
                push(@m_c_changesets, $$r_changeset_details{$csetid}->{'changeset'});
            }
            elsif ($$r_changeset_details{$csetid}->{'repo'} eq 'm-a') {
                push(@m_a_changesets, $$r_changeset_details{$csetid}->{'changeset'});
            }
            elsif ($$r_changeset_details{$csetid}->{'repo'} eq 'm-b') {
                push(@m_b_changesets, $$r_changeset_details{$csetid}->{'changeset'});
            }
            elsif ($$r_changeset_details{$csetid}->{'repo'} eq 'm-r') {
                push(@m_r_changesets, $$r_changeset_details{$csetid}->{'changeset'});
            }
        }

        # Knowing changesets of interest, construct grep regexs to find references to this bug and those changesets.

        $m_c_pattern = $bugnum . join('', map({ "|$_" } @m_c_changesets));
        $m_a_pattern = $bugnum . join('', map({ "|$_" } @m_a_changesets));
        $m_b_pattern = $bugnum . join('', map({ "|$_" } @m_b_changesets));
        $m_r_pattern = $bugnum . join('', map({ "|$_" } @m_r_changesets));

        if ($verbose) {
            print "grep for $bugnum:\n";
            print "  on m-c: $m_c_pattern\n";
            print "  on m-a: $m_a_pattern\n";
            print "  on m-b: $m_b_pattern\n";
            print "  on m-r: $m_r_pattern\n";
        }

        # To keep things manageable, historical pushlog data is segregated into separate files by year.
        # When scanning historical data for a repo, write grep commands against each file for that repo.
        # Strange but true: use echo. to output a blank line from a batch file.
        print OUTPUTFILE "echo.\n";
        print OUTPUTFILE "echo *** Grepping Bug $bugnum ***\n";
        foreach my $filename (@{$historical_pushlog_files{'m-c'}}) {
            print OUTPUTFILE "grep -EiH \"$m_c_pattern\" $filename\n";
        }
        foreach my $filename (@{$historical_pushlog_files{'m-a'}}) {
            print OUTPUTFILE "grep -EiH \"$m_a_pattern\" $filename\n";
        }
        foreach my $filename (@{$historical_pushlog_files{'m-b'}}) {
            print OUTPUTFILE "grep -EiH \"$m_b_pattern\" $filename\n";
        }
        foreach my $filename (@{$historical_pushlog_files{'m-r'}}) {
            print OUTPUTFILE "grep -EiH \"$m_r_pattern\" $filename\n";
        }
        print OUTPUTFILE "\n";
    }
    print OUTPUTFILE "\n";
    close(OUTPUTFILE);

    # Run the historical pushlog grep commands en masse.  This may take some time.
    system("$output_filename > $basedir$full_scan_subdir$full_scan_grep_results_filename");
}

# Read the file previously written by us and add it to our output.
sub output_historical_pushlog_scan_results {
    my( $r_buglist, $r_bug_details, $r_changeset_details ) = @_;

    print "=====================================================================\n";
    print "Historical Pushlog Grep Results\n";
    print "---------------------------------------------------------------------\n";
    print "\n";

    my $srcfilename = $basedir . $full_scan_subdir . $full_scan_grep_results_filename;
    open SRCFILE, "<$srcfilename" or die "Error: can't open $srcfilename";

    # read in the entire file in one big gulp
    undef $/;
    my $entire_file = <SRCFILE>;
    $/ = "\n";

    close(SRCFILE);
    print $entire_file;
}

# Searches the given directory for files matching the given pattern.
# Subdirectories will not match; only non-directory files are included in results.
# The given pattern will be used in a regex; it should be a string, and not a precompiled regex,
# and should not contain the match delimiters (slashes), i.e., THIS '.html$' but NOT this '/.html$/'.
# Returns a possibly-empty list of fully-qualified matching filenames.
sub find_matching_files {
    my( $dir, $pattern ) = @_;
    my @files = ();
    find(
        \&{sub {
            if( not -d $File::Find::name and
                $File::Find::name =~ /$pattern/
                ) {
                push @files, $File::Find::name;
                }
            }},
        $dir
        );
    return @files;
}
Theme Development is Radical Participation.
NNL Beta Builds for Current and Up-coming Firefox
Dear User: Your Help is Needed

mcdavis

User avatar
 
Posts: 3195
Joined: December 9th, 2005, 5:51 am

Post Posted May 3rd, 2013, 10:33 pm

So for example, looking at the bugs in Alfred's list that I haven't yet scoped out, if I feed in this list:

Code: Select all
851546
855502
837351
860175
862117
748894
832672
856917
425561
842780
853151
854555
865746
588305


then I get this output:

Code: Select all
=====================================================================
Bug Quicklist
---------------------------------------------------------------------

Note: dates may be incorrect and need to be checked.  They may well be based on backed-out patches.
TODO (GUESSING:2013-04-19) (FxNN) Bug 851546 - Create an Options panel for the toolbox
TODO (GUESSING:2013-04-19) (FxNN) Bug 855502 - we need a proper design for the new checkboxes
TODO (GUESSING:2013-04-23) (FxNN) Bug 837351 - When mixed content is blocked, show the blocked URL in the Error Console and WebConsole
TODO (GUESSING:2013-04-23) (FxNN) Bug 860175 - On netmonitor examining the specific details of a network action leads to bad overlapping display
TODO (GUESSING:2013-04-23) (FxNN) Bug 862117 - Zoom cursors don't appear with third party full theme or in framed images
TODO (GUESSING:2013-04-23) (FxNN) Bug 748894 - Move the bookmark star button outside of the location bar
TODO (GUESSING:2013-04-24) (FxNN) Bug 832672 - Downloads Panel gives no indication or feedback on missing files
TODO (GUESSING:2013-04-24) (FxNN) Bug 856917 - Improve about:memory's functional UI
TODO (GUESSING:2013-04-24) (FxNN) Bug 425561 - Sidebar background color off on OSX
TODO (GUESSING:2013-04-25) (FxNN) Bug 842780 - about:newaddon doesn't restrict size of the add-on's icon
TODO (GUESSING:2013-04-25) (FxNN) Bug 853151 - refactoring of recommend into SocialMark
TODO (GUESSING:2013-04-25) (FxNN) Bug 854555 - icons in the Downloads view are slightly &quot;squashed&quot; on OS X
TODO (GUESSING:2013-04-26) (FxNN) Bug 865746 - [HiDPI] Downloads panel icons are disproportionately large
TODO (GUESSING:2013-04-29) (FxNN) Bug 588305 - Convert offline storage notifications to a doorhanger panel

=====================================================================
Meta Buglist
---------------------------------------------------------------------

    Imp Rv  IThm    NLN Sta BugNum      Date [1]        Res         Appver          Platform[2] Cset            Description                                                                                                               
    --- --- ------- ------- ----------- --------------- ----------- --------------- ----------- --------------- ---------------------------------------------------------------------------------------------------------------------------

    -   -   ?       ????    Bug ??????  (??????????)    ?           Fx??            All         ????????????    - ??????                                                                                                                   

    -   -   ?       TODO    Bug 851546  (2013-04-19)    ?           Fx??            ?           fe573a582083    - Create an Options panel for the toolbox
    -   -   ?       TODO    Bug 855502  (2013-04-19)    ?           Fx??            ?           630fa3185e7e    - we need a proper design for the new checkboxes
    -   -   ?       TODO    Bug 837351  (2013-04-23)    ?           Fx??            ?           cd89b7d8dbf0    - When mixed content is blocked, show the blocked URL in the Error Console and WebConsole
    -   -   ?       TODO    Bug 837351  (2013-04-23)    ?           Fx??            ?           af21a5a0956e    - When mixed content is blocked, show the blocked URL in the Error Console and WebConsole
    -   -   ?       TODO    Bug 860175  (2013-04-23)    ?           Fx??            ?           ca1f0cbdff36    - On netmonitor examining the specific details of a network action leads to bad overlapping display
    -   -   ?       TODO    Bug 862117  (2013-04-23)    ?           Fx??            ?           2e822e8396c2    - Zoom cursors don't appear with third party full theme or in framed images
    -   -   ?       TODO    Bug 748894  (2013-04-23)    ?           Fx??            ?           1706d6e0514b    - Move the bookmark star button outside of the location bar
    -   -   ?       TODO    Bug 748894  (2013-04-23)    ?           Fx??            ?           4002c6963c4e    - Move the bookmark star button outside of the location bar
    -   -   ?       TODO    Bug 832672  (2013-04-24)    ?           Fx??            ?           12b482031469    - Downloads Panel gives no indication or feedback on missing files
    -   -   ?       TODO    Bug 856917  (2013-04-24)    ?           Fx??            ?           0b2c39985587    - Improve about:memory's functional UI
    -   -   ?       TODO    Bug 856917  (2013-04-24)    ?           Fx??            ?           1c5977e8d52f    - Improve about:memory's functional UI
    -   -   ?       TODO    Bug 425561  (2013-04-24)    ?           Fx??            ?           50e931146b0e    - Sidebar background color off on OSX
    -   -   ?       TODO    Bug 842780  (2013-04-25)    ?           Fx??            ?           6b6e79e65b37    - about:newaddon doesn't restrict size of the add-on's icon
    -   -   ?       TODO    Bug 854555  (2013-04-25)    ?           Fx??            ?           a1ead1605ade    - icons in the Downloads view are slightly &quot;squashed&quot; on OS X
    -   -   ?       TODO    Bug 865746  (2013-04-26)    ?           Fx??            ?           da9f02088406    - [HiDPI] Downloads panel icons are disproportionately large
    -   -   ?       TODO    Bug 588305  (2013-04-29)    ?           Fx??            ?           ea5490a3bca7    - Convert offline storage notifications to a doorhanger panel

=====================================================================
Final Report
---------------------------------------------------------------------
TOPIC: Handle Bug 851546 - Create an Options panel for the toolbox
TAGS:
=========================================================================================================================
patch:
** REQUIRES EYES-ON CONFIRMATION **
- (GUESSING:2013-04-19) (FxNN) https://hg.mozilla.org/mozilla-central/rev/fe573a582083 () Bug 851546 - Options panel for DevTools Toolbox, r=jwalker,vporof,past
outcome:
NNL impact:
- TODO

TOPIC: Handle Bug 855502 - we need a proper design for the new checkboxes
TAGS:
=========================================================================================================================
patch:
** REQUIRES EYES-ON CONFIRMATION **
- (GUESSING:2013-04-19) (FxNN) https://hg.mozilla.org/mozilla-central/rev/630fa3185e7e () Bug 855502 - we need a proper design for the new checkboxes r=benvie
outcome:
NNL impact:
- TODO

TOPIC: Handle Bug 837351 - When mixed content is blocked, show the blocked URL in the Error Console and WebConsole
TAGS:
=========================================================================================================================
patch:
** REQUIRES EYES-ON CONFIRMATION **
- (GUESSING:2013-04-23) (FxNN) https://hg.mozilla.org/mozilla-central/rev/cd89b7d8dbf0 () Bug 837351 - Security Errors in Web Console. r=msucan
- (GUESSING:2013-04-23) (FxNN) https://hg.mozilla.org/mozilla-central/rev/af21a5a0956e () Bug 837351 - Log blocked mixed content to the Error and Web Consoles. r=smaug
outcome:
NNL impact:
- TODO

TOPIC: Handle Bug 860175 - On netmonitor examining the specific details of a network action leads to bad overlapping display
TAGS:
=========================================================================================================================
patch:
** REQUIRES EYES-ON CONFIRMATION **
- (GUESSING:2013-04-23) (FxNN) https://hg.mozilla.org/mozilla-central/rev/ca1f0cbdff36 () Bug 860175 - On netmonitor examining the specific details of a network action leads to bad overlapping display, r=rcampbell
outcome:
NNL impact:
- TODO

TOPIC: Handle Bug 862117 - Zoom cursors don't appear with third party full theme or in framed images
TAGS:
=========================================================================================================================
patch:
** REQUIRES EYES-ON CONFIRMATION **
- (GUESSING:2013-04-23) (FxNN) https://hg.mozilla.org/mozilla-central/rev/2e822e8396c2 () Bug 862117 - Move CSS rule classes relevant to image documents for cursor appearance (zoom in / zoom out) into layout/style/ImageDocument.css. r=roc, r=jaws
outcome:
NNL impact:
- TODO

TOPIC: Handle Bug 748894 - Move the bookmark star button outside of the location bar
TAGS:
=========================================================================================================================
patch:
** REQUIRES EYES-ON CONFIRMATION **
- (GUESSING:2013-04-23) (FxNN) https://hg.mozilla.org/mozilla-central/rev/1706d6e0514b () Bug 748894 - Move the bookmark star button outside of the location bar.
- (GUESSING:2013-04-23) (FxNN) https://hg.mozilla.org/mozilla-central/rev/4002c6963c4e () Bug 748894 follow-up - increase menu-button dropmarker clickable area.
outcome:
NNL impact:
- TODO

TOPIC: Handle Bug 832672 - Downloads Panel gives no indication or feedback on missing files
TAGS:
=========================================================================================================================
patch:
** REQUIRES EYES-ON CONFIRMATION **
- (GUESSING:2013-04-24) (FxNN) https://hg.mozilla.org/mozilla-central/rev/12b482031469 () Bug 832672 - Downloads Panel gives no indication or feedback on missing files. r=mak ui-r=shorlander
outcome:
NNL impact:
- TODO

TOPIC: Handle Bug 856917 - Improve about:memory's functional UI
TAGS:
=========================================================================================================================
patch:
** REQUIRES EYES-ON CONFIRMATION **
- (GUESSING:2013-04-24) (FxNN) https://hg.mozilla.org/mozilla-central/rev/0b2c39985587 () Bug 856917 (part 1) - Improve about:memory's functional UI.  r=kats.
- (GUESSING:2013-04-24) (FxNN) https://hg.mozilla.org/mozilla-central/rev/1c5977e8d52f () Bug 856917 (part 2) - Add a test for &quot;?file=&quot; loading in about:memory.  r=jlebar.
outcome:
NNL impact:
- TODO

TOPIC: Handle Bug 425561 - Sidebar background color off on OSX
TAGS:
=========================================================================================================================
patch:
** REQUIRES EYES-ON CONFIRMATION **
- (GUESSING:2013-04-24) (FxNN) https://hg.mozilla.org/mozilla-central/rev/50e931146b0e () Bug 425561 - Make all sidebars (Including inside Organizer) lighter and consistent. r=dao, ui-r=shorlander
outcome:
NNL impact:
- TODO

TOPIC: Handle Bug 842780 - about:newaddon doesn't restrict size of the add-on's icon
TAGS:
=========================================================================================================================
patch:
** REQUIRES EYES-ON CONFIRMATION **
- (GUESSING:2013-04-25) (FxNN) https://hg.mozilla.org/mozilla-central/rev/6b6e79e65b37 () Bug 842780 - about:newaddon doesn't restrict size of the add-on's icon. r=Unfocused
outcome:
NNL impact:
- TODO

TOPIC: Handle Bug 853151 - refactoring of recommend into SocialMark
TAGS:
=========================================================================================================================
patch:
** REQUIRES EYES-ON CONFIRMATION **
outcome:
NNL impact:
- TODO

TOPIC: Handle Bug 854555 - icons in the Downloads view are slightly &quot;squashed&quot; on OS X
TAGS:
=========================================================================================================================
patch:
** REQUIRES EYES-ON CONFIRMATION **
- (GUESSING:2013-04-25) (FxNN) https://hg.mozilla.org/mozilla-central/rev/a1ead1605ade () bug 854555 - ensure download icons are 32px even on Retina Macs. r=shorlander
outcome:
NNL impact:
- TODO

TOPIC: Handle Bug 865746 - [HiDPI] Downloads panel icons are disproportionately large
TAGS:
=========================================================================================================================
patch:
** REQUIRES EYES-ON CONFIRMATION **
- (GUESSING:2013-04-26) (FxNN) https://hg.mozilla.org/mozilla-central/rev/da9f02088406 () bug 865746 - [win-hidpi] explicitly set size of download item icons. r=mconley
outcome:
NNL impact:
- TODO

TOPIC: Handle Bug 588305 - Convert offline storage notifications to a doorhanger panel
TAGS:
=========================================================================================================================
patch:
** REQUIRES EYES-ON CONFIRMATION **
- (GUESSING:2013-04-29) (FxNN) https://hg.mozilla.org/mozilla-central/rev/ea5490a3bca7 () Bug 588305 - Convert offline storage notifications to a doorhanger panel.  r=MattN
outcome:
NNL impact:
- TODO

=====================================================================
Historical Pushlog Grep Results
---------------------------------------------------------------------


*** Grepping Bug 851546 ***
d:/mcd/dvl/moztheme/nightlaunchnext/m-c-pushlog/2013.central.pushlog.txt:    fe573a582083    Girish Sharma — Bug 851546 - Options panel for DevTools Toolbox, r=jwalker,vporof,past
d:/mcd/dvl/moztheme/nightlaunchnext/m-c-pushlog/2013.central.pushlog.txt:    543594f2647a    Panos Astithas — Backed out changeset 371ecfa8df92 (bug 851546) for breakage
d:/mcd/dvl/moztheme/nightlaunchnext/m-c-pushlog/2013.central.pushlog.txt:    371ecfa8df92    Girish Sharma — Bug 851546 - Options panel for DevTools Toolbox, r=jwalker

*** Grepping Bug 855502 ***
d:/mcd/dvl/moztheme/nightlaunchnext/m-c-pushlog/2013.central.pushlog.txt:    630fa3185e7e    Michael Ratcliffe — Bug 855502 - we need a proper design for the new checkboxes r=benvie Bug 855502 - we need a proper design for the new checkboxes r=benvie

*** Grepping Bug 837351 ***
d:/mcd/dvl/moztheme/nightlaunchnext/m-c-pushlog/2013.central.pushlog.txt:    af21a5a0956e    Garrett Robinson — Bug 837351 - Log blocked mixed content to the Error and Web Consoles. r=smaug
d:/mcd/dvl/moztheme/nightlaunchnext/m-c-pushlog/2013.central.pushlog.txt:    cd89b7d8dbf0    Garrett Robinson — Bug 837351 - Security Errors in Web Console. r=msucan

*** Grepping Bug 860175 ***
d:/mcd/dvl/moztheme/nightlaunchnext/m-c-pushlog/2013.central.pushlog.txt:    ca1f0cbdff36    Victor Porof — Bug 860175 - On netmonitor examining the specific details of a network action leads to bad overlapping display, r=rcampbell

*** Grepping Bug 862117 ***
d:/mcd/dvl/moztheme/nightlaunchnext/m-c-pushlog/2013.central.pushlog.txt:    2e822e8396c2    Brandon Waterloo — Bug 862117 - Move CSS rule classes relevant to image documents for cursor appearance (zoom in / zoom out) into layout/style/ImageDocument.css. r=roc, r=jaws This stylesheet applies to all ImageDocuments, including those in frames. In old location for CSS rules, some full themes would not show cursors correctly, nor would ImageDocuments in frames. Image rotation CSS rule classes moved to layout/style/TopLevelImageDocument.css.

*** Grepping Bug 748894 ***
d:/mcd/dvl/moztheme/nightlaunchnext/m-c-pushlog/2013.central.pushlog.txt:    4002c6963c4e    Marco Bonardo — Bug 748894 follow-up - increase menu-button dropmarker clickable area. r=dao
d:/mcd/dvl/moztheme/nightlaunchnext/m-c-pushlog/2013.central.pushlog.txt:    1706d6e0514b    Marco Bonardo — Bug 748894 - Move the bookmark star button outside of the location bar. r=Mano

*** Grepping Bug 832672 ***
d:/mcd/dvl/moztheme/nightlaunchnext/m-c-pushlog/2013.central.pushlog.txt:    12b482031469    Paolo Amadini — Bug 832672 - Downloads Panel gives no indication or feedback on missing files. r=mak ui-r=shorlander

*** Grepping Bug 856917 ***
d:/mcd/dvl/moztheme/nightlaunchnext/m-c-pushlog/2013.central.pushlog.txt:Wed Apr 24 05:15:57 2013 -0700  1c5977e8d52f    Nicholas Nethercote — Bug 856917 (part 2) - Add a test for "?file=" loading in about:memory. r=jlebar.
d:/mcd/dvl/moztheme/nightlaunchnext/m-c-pushlog/2013.central.pushlog.txt:    0b2c39985587    Nicholas Nethercote — Bug 856917 (part 1) - Improve about:memory's functional UI. r=kats.

*** Grepping Bug 425561 ***
d:/mcd/dvl/moztheme/nightlaunchnext/m-c-pushlog/2013.central.pushlog.txt:    50e931146b0e    JosiahOne — Bug 425561 - Make all sidebars (Including inside Organizer) lighter and consistent. r=dao, ui-r=shorlander

*** Grepping Bug 842780 ***
d:/mcd/dvl/moztheme/nightlaunchnext/m-c-pushlog/2013.central.pushlog.txt:    6b6e79e65b37    Sachin Hosmani — Bug 842780 - about:newaddon doesn't restrict size of the add-on's icon. r=Unfocused

*** Grepping Bug 853151 ***
d:/mcd/dvl/moztheme/nightlaunchnext/m-c-pushlog/2013.central.pushlog.txt:    29c172213ff7    Shane Caraveo — bug 853151 rework recommend into social marks, r=felipe
d:/mcd/dvl/moztheme/nightlaunchnext/m-c-pushlog/2013.central.pushlog.txt:    ed39a9db0fb1    Ed Morley — Backed out changeset 8a9a40bfa8e3 (bug 853151) for browser_social_markButton.js failures
d:/mcd/dvl/moztheme/nightlaunchnext/m-c-pushlog/2013.central.pushlog.txt:    8a9a40bfa8e3    Shane Caraveo — bug 853151 refactoring recommend into SocialMark, r=felipe

*** Grepping Bug 854555 ***
d:/mcd/dvl/moztheme/nightlaunchnext/m-c-pushlog/2013.central.pushlog.txt:    a1ead1605ade    Jonathan Kew — bug 854555 - ensure download icons are 32px even on Retina Macs. r=shorlander

*** Grepping Bug 865746 ***
d:/mcd/dvl/moztheme/nightlaunchnext/m-c-pushlog/2013.central.pushlog.txt:    da9f02088406    Jonathan Kew — bug 865746 - [win-hidpi] explicitly set size of download item icons. r=mconley

*** Grepping Bug 588305 ***
d:/mcd/dvl/moztheme/nightlaunchnext/m-c-pushlog/2013.central.pushlog.txt:    ea5490a3bca7    Mark Hammond — Bug 588305 - Convert offline storage notifications to a doorhanger panel. r=MattN
Theme Development is Radical Participation.
NNL Beta Builds for Current and Up-coming Firefox
Dear User: Your Help is Needed

LoudNoise
Moderator

User avatar
 
Posts: 40048
Joined: October 18th, 2007, 1:45 pm
Location: Next door to the west

Post Posted May 3rd, 2013, 10:50 pm

I am certainly not your audience here but my first question would be how useful is this without a link to the bug?
Post wrangler
"Choose between the Food Select Feature or other Functions. If no food or function is chosen, Toast is the default."

mcdavis

User avatar
 
Posts: 3195
Joined: December 9th, 2005, 5:51 am

Post Posted May 3rd, 2013, 11:19 pm

LoudNoise wrote:I am certainly not your audience here but my first question would be how useful is this without a link to the bug?


Yeah, that's true, it would be more generally useful that way. (Although that's already in Alfred's list, since that's what I take as input.)

I don't know if this is useful to anybody else but me, it really is meant to fit into my workflow. But I do spend too many hours each release cycle just figuring out what the changesets are, which ones stuck and which ones got backed out, what order they landed in, and what Firefox version they landed for, so I think this'll cut that time down a lot. Copypasting is exhausting, and cutting that down is a big win too. (Also, part of my way of working is to keep any documentation in text specifically because it can't be formatted or linked, to keep me from engaging my perfectionist tendencies. If it were in HTML or say a Word or OpenOffice document, I'd be helpless to resist spending time making it pretty.)
Theme Development is Radical Participation.
NNL Beta Builds for Current and Up-coming Firefox
Dear User: Your Help is Needed

mcdavis

User avatar
 
Posts: 3195
Joined: December 9th, 2005, 5:51 am

Post Posted October 13th, 2013, 12:30 pm

A little more show and tell ...

I've continued working on this and it's now a key part of my process. I can produce a report like this by passing this input into this script which goes a long way toward making me feel like I have an idea of what happened in Firefox source without having to do a lot of work or wait for a bunch of page loads to figure it out.

The script depends on two other input files: 1) a list of tags identifying the associated areas of functionality for each bug and 2) the dates of version bumps for each Firefox version in m-c. The dates let me positively determine the Firefox version for which a patch on m-c landed, by comparing the pushlog date of the patch with the date ranges in that list.

One other thing the script does is scan pushlogs for any mention of each bug and for any changesets thought to be associated with the bug, and then report any matches.

The thing to know about pushlog vs. hg, for anyone who hasn't bumped up against that yet: patch date is not the same as push date. Patch date is whatever the date was on the computer where the patch was made, at the time it was made; if you're looking at an individual changeset, the date shown is the patch date. In contrast, push date (which is the date shown in pushlog) is the date on which the patch was applied to the repo. Mercurial itself knows only patch dates, and knows nothing about pushes. This seems strange, but I've been told this several times by several people. In fact (I was told) this is exactly why pushlog was created, to record and report the order in which patches landed on the repo, with dates recorded relative to the date on the machine holding the repo master (as opposed to whatever the date was on the computer of the person who created the patch). Or so I've heard.

Back to my stuff: this pushlog scan is really good for feeling like you're seeing the big picture. It helps make sure you found all the bugs and changesets, and helps confirm or deny that the ones you think you found are actually the right ones.

To support this pushlog scanning, I save out a local copy of the entire pushlog for each repo (just a screen-scrape of pushlog web output), then just search through that. Working on a local copy is a lot faster than hitting up the web server.

However, there's a complication: while you can easily get the whole pushlog for mozilla-central and mozilla-aurora through the web interface, you can't do it for mozilla-beta and mozilla-release on any day that includes one of the big uplift merges. The web pushlog query works by taking its own list of pushes and asking hg for data on each of the changesets that are part of the push, and for those really big pushes hg just can't keep up and pushlog just times out without reporting anything. Pushlog on m-b and m-r works as desired as long as your query doesn't include one of those big pushes, but if it does, forget it. For example, this pushlog query, running from midnight August 10 to midnight on September 9, does work, while this query, spanning August 10 to September 10, just one day more, fails because of the big push that happened during that one additional day.

So, there's a work-around for this (and many thanks to Ted M for pointing it out). Instead of getting nice pushlog output, you can get raw unprocessed push data which contains information about the pushes that pushlog itself knows about, without involving hg, thus avoiding the bottleneck, and this works fine. It's not human-readable, so you have to run some post-processing on it to make it look like it came from the pushlog web interface, and I run a little JS in Scratchpad which does that.

One final complication: pushlog stores push dates in numeric form, as an integer number of seconds since the epoch. When JS, running in the browser, converts those into string form, it uses whatever the timezone is on the computer on which the browser is running. To get the reported date for each push to be the same as it would have been had it come from pushlog web output, I first set the timezone on my computer to be the same as the one on the pushlog master server (i.e., California time); that way, when my JS, running in Scratchpad, converts the numeric time into a date string, it comes out the same as if it came from pushlog itself. (JavaScript itself doesn't come with built-in routines to do the TZ conversion, and all the JS date-time packages I surveyed had some flaw which made them less than 100% reliable across all cases, so I just went with what works.) And if I'm lucky, I remember to reset my timezone when I'm done.
Theme Development is Radical Participation.
NNL Beta Builds for Current and Up-coming Firefox
Dear User: Your Help is Needed

Return to Theme Development


Who is online

Users browsing this forum: No registered users and 1 guest