expanding support options/needs as the fx user base grows

[chofmann]

Asa and I have started looking at ways that we can meet the needs of the firefox user base as it continues to grow. I'd like to start a discussion on how things are going with the great effort that you all are involved in to provide support on the forums and build the knowledge base.

What are the current problems that you encounter and how can things get better in areas like:
--server loading,
--community building for support experts
--need for better tools to diagnose problems, assist users, and translate to kb articles as needed
--other stuff...

What are the problems that we need to be watching for in the future?

One perspective that I think we are starting to see as the user base grows is a difference among some users on their willingness to do research on support problems. There seems to be a general hierarchy of:

1) users that might just want to yell at their spouse across the room, pick up the phone, or open a real-time chat session on a service like http://alpha.qunu.com/index.html?search=firefox -- users like this might have a profile of a) they hate technology b) they just want their problem solved right now, and are not willing to invest much time or effort to fix the problem. you know the type

2) if a user doesn't find this kind of real-time help they might struggle to move to find answer by going to something like the mozillazine forums, yahoo! answers, or similar service by asking questions and getting responses there.

3) another class of users might be more willing to put in time to do research on a problem or question and are best served via the knowledge base and on-line documentation that is clear and up-to-date. This is also the resource needed to support the experts that volunteer to help out on areas 1 and 2.

How can we best tie all these kinds of support needs together? What are the tools and processes we need to make that happen. Things like
-sharing of expert resources among all the channels
-statistical analysis to categorize problems, build a knowledge base, and surface the feedback into to have the best impact future product releases.

How can we make sure we are using the best of breed tools in each of these areas?

http://qunu.com is interesting in that they are taking an open approach to building support tools for real-time support help and creating communities a lot like what has happened with firefox. I'd be interested in any comments or experiences that folks have had with qunu or similar kinds of tools to providing end-user support.

thoughts and comments welcome

chris h.

[old np]

The first problem we encounter is the integration of what we have available to us for support on mozillaZine. We have a MediaWiki, we have a phpBB forum, and we have an IRC channel. I don't know how much customization has been done to these things, but they need to be further customized and integrated with each other, or replaced with something that does. Instead of requiring users to go to separate areas to look for documentation on an issue, chat to someone about an issue, and post a thread about an issue, why not have it all in the same place? My ideal software would allow the user to post a problem, get a few automated replies based on the question, and if that doesn't help, give the user a forum/chat session with one or more experts.

Further to this, we can't get simple functionality improvements. Replacing the MediaWiki search with a much better Google site search. Not requiring KB users to e-mail me to get an account. I realize that Alex and kerz may be busy, but we have hundreds of technically-capable people who want to improve this site, but have no way to.

Another problem is the sheer number of users posting. The forum isn't working for this. Looking right now, someone who posted an issue an hour ago is already off the first page of the forum. People providing support are spending much of their time telling users about the same few problems rather than helping users who are truly stuck on "interesting problems" (and then documenting the solution to help those who look at the KB).

We have no tools to understand the frequency of problems or the usefulness of solutions. A Knowledge Base article may have three possible solutions to a problem, with a 95%, 4%, and 1% chance of working. All choices are presented to the user equally.

We also require additional attention from the browser developers as well. Case in point: the number one support issue for people upgrading is the fact that their firewall may block access. There's a bug that would help people with this issue, but it wasn't deemed important enough to include in Firefox 2. There needs to be some way to indicate to Mozilla how widespread certain issues are, and Mozilla needs to listen.

[Lucy]

I think this is a hard question to answer without getting everyone together from all the places. I'm not sure how much integrating the support areas is really a good thing. Some people just need help documents, others really need to be hand held. I've spent 5 hours on irc with someone and ended up with a resolution to their problem and that just wouldn't have happened on the forums. I think we need a better maintained help base where those of us answering issues can go to find answers.

np- do you spend much time on irc at all? I ask because some of the things you're asking for, I feel we have working well in #firefox so I dont know if it's a matter that you would still change how things are done, or if it's a matter that you haven't been there to see it.

As for the sheer number of threads, so many are duplicates. IMO moderators need to do a better job of editing thread titles to be properly descriptive (so they turn up in searches) and when something is a duplicate, refer to the original thread and lock it. The ability to mark threads as duplicates ala bugzilla I think would be incredible. Also, from what I've seen moderators are being chosen for their time being regulars on the forums and not for their technical ability. When someone sees a moderator say "I have no idea why that's happening" to a problem that's explained in the knowledge base, the users are going to assume there is no answer, or the moderator would know. I've seen that in itself create extra threads, because the old thread with the solution gets burried under 10 threads where people get no answer.

As for the knowledge base... I think whatever help repository mozilla officially uses MUST be moderated. I got into some fun trying to make the lost bookmarks article easier for users to navigate to their solution, but another kb contributor disagreed with part of changes. Instead of communicating to me so that we could sort out the right thing for the document it ended up being changed back and forth several times. With no one to appeal to who could lock down the article, or at least pass judgement on how the article would best help users, I gave up. It also makes me disinclined to try updating other articles where I know they're out of date, or I've discovered the solution in one of my 5 hour support marathons with a happily cooperating user (which is a wonderful benefit of IRC, you have them in real time and they don't need their browser to talk to you. Combine that with neither of you having anything else to do, and it's amazing what you can sort out).

I think the thing that's putting me off the most about this thread is that a) it's only on the forums here, b) you mention live support, but not IRC and c) it doesn't seem like the rest of the mozilla developer community has been approached first about how *they* think it's best to meet user support needs. This has definitely come up among other parts of the community, and there are some really exciting ideas.

All in all, I think the most important thing for improving user support is to have a *Mozilla* hosted central support repository, contributed to by all the different branches of user support, as well as bugzilla triagers, that anyone giving support whether it be here or on IRC or a friend supporting another friend over the phone, can draw answers from and know that they're current, and correct. By having the right answers in one official place gives everyone the freedom to provide support in whatever manner they wish, wherever they wish and ensures the end users get the same quality support.

[Mossop]

We have an interesting challenge with providing consistent support to the user base. We currently have 4 main support options that I can think of, newsgroups, forums, IRC and the knowledge base and other associated help style pages. This is pretty good in that it allows users to use the medium they are most familiar with to get support, and likewise it allows us volunteers to choose which method they are more comfortable giving support over. The problem I feel is the lack of crossover. How many of the active guys in the support forums also offer support on IRC or the newsgroups? Not many in my estimation.

Personally I think one of the biggest steps we could take would be to facilitate all the supporters from all the areas in getting together (figuratively) so that they can easily discuss the hot issues, point out new approaches to fixing things and just generally share the information they hold. All of that could then be disseminated out to the rest of the supporters and even the developers.

I don't know what form would work, maybe it hasn't happened because there isn't one but I could imagine each support area choosing a couple of "champions". When a supporter sees something noteworthy they pass it on to their champion who then discusses it with the champions elsewhere, maybe by an email list or something. If agreed the info gets pushed onto a supporters blog which anyone interested can subscribe to, and we would hope most supporters and a couple of key developers would. Maybe we don't even need the mailing list, maybe just allowing a few from each area to push information to a blog that most of the supporters read is enough?

Incidentally this is not a new discussion. I can recall having talks like this on IRC and the newsgroups in the past, the only example I could find being this discussion that came out with some great ideas in my opinion but didn't really get anywhere.

On the subject of tools, I made a suggestion some time ago that could be useful. Ryan and myself even started some groundwork on it though I have to confess we haven't really got anywhere with it.

N.B. I apologise wholeheartedly for the use of the word "champion" in the post, it's somewhat depressing that I couldn't think of anything better before I had to post this.

[trolly]

As for the sheer number of threads, so many are duplicates. IMO moderators need to do a better job of editing thread titles to be properly descriptive (so they turn up in searches) and when something is a duplicate, refer to the original thread and lock it. The ability to mark threads as duplicates ala bugzilla I think would be incredible. Also, from what I've seen moderators are being chosen for their time being regulars on the forums and not for their technical ability. When someone sees a moderator say "I have no idea why that's happening" to a problem that's explained in the knowledge base, the users are going to assume there is no answer, or the moderator would know. I've seen that in itself create extra threads, because the old thread with the solution gets buried under 10 threads where people get no answer.

I'm a programmer myself and have the required basic technical knowledge but not special inside in FF. So even if i can see a problem i still do not know why this problem appears. For example i got a thread where someone complained about not connecting to the net. I and the OP went through every know issue and still FF can not connect. So i did really not know what is going on. I have special knowledge of networking but not of the details in FF to prepare special tests.
And for the time i'm here in my free time and even if i'm on work and i'm not able to review all posts. There are too much of them. Most of them are answered by the regulars but they can not change subject titles. For myself i can point well over 90% of all issues to existing threads and knowledge base entries but there are still a few issues for which i can not find a thread or entry because a) there is none or b) the description does not match with the posters description to get usable words for a search.
I think we need a better classifying of the known issues. Something like a search tree. Some sort of automated problem analyzer to sort out the more basic problems.

[Lucy]

I don't see how an automated problem analyzer makes sense in the context of a message board. What is needed is more active members who have some sort of "listen to them, they know what they're talking about" designation, and more moderators to handle the grunt work. Also a clearer example of how to moderate and lock/merge threads etc. so that community members who work their way up can jump into moderation. It's the same principles that grow the triagers in bugzilla and the helpers in irc.

[Pike2]

I agree with Lucy on that a thread in the mz forum is likely only going to get a fragment of folks into the discussion. I had to ponder back and forth if I want to check if I could resurrect my, I think second account on this forum. And 3rd or 4th on mz.

I think there are two points here, which are somewhat orthogonal. One is writing end-user documentation. We've been shooting ourselves in the foot there for quite a while. Writing and maintaining that documentation is a huge task, and it includes stuff like branching (you want to have places docs without throwing the bookmarks docs right away), and product management. Luckily, I recall us putting end-user docs on the feature check-list on the summit. Oh, do I have to mention l10n? That's a biggie, in particular in the "maintaining" part, I think. Jeff had some ideas, I have some, but that's a different thread (hopefully on news.m.o).

The other point is making that documentation available to users. I recall that we have a suite of PDF files for firewall problems, one for each major firewall vendor. Not that I recall us shipping them somehow.

I think that a tiered approach to support is working quite nicely, and I'm afraid that we're missing at least one tier. I guess we're doing good at the level-2 and level-3 supporters, but should think about how to do level-1 right. The qunu approach that chofmann pointed at is interesting as one, though it's probably such that it'd be more in our genes to hook such a service up to an IRC channel rather than jabber. But I think there should be a more automated approach before that, something that would catch plain text questions and try to map them to documentation. This is probably some machine-learning problem, where users could do the "I found this helpful" thing that you see in other KBs quite often, and assist that with folks from chat that could jump on questions and help out to point users to docs (and make the system learn from that).
I'm pretty sure that a user forum like phpBB will not survive the incoming requests if we ever get to 30 on a worldwide level, not that I'm fond of the user experience thread-duping either. There should be something in front of that, which scales better without being a "dial 1 to hang up" service-refusal-automaton.

[old np]

IRC is essentially a chat implementation. Chatting is definitely something that's needed to help users; one reason being that some users need their hands help to fix whatever problem they're having. The thing about chatting, though (and this is one reason I stay away from the IRC channel), is the user and the helper have to both be there at the same time for one to help the other. I don't know how many people who regularly post on forums or newsgroups would want to provide that sort of support. I personally would rather jump through a page of forum threads in ten minutes and provide whatever help I can. Others wouldn't. I don't think it'd be useful to force one type of helper to use the other helping method. The other reason I stay away from IRC is that I barely know how to work it. I suspect a large number of users are the same. Providing a web front-end to whatever chat option we use is a must. What I mean by integrating the options is that people who want to do IRC can keep doing IRC, etc., but there is a common entry point to users where they can search documentation and choose what person-to-person support method they prefer. This common entry point may also try to help the user automatically (without being obnoxious as Pike2 mentions).

The Knowledge Base I think is a key component. Whether you provide person-to-person support via chatting, a forum, or a newsgroup, you always need documentation that both the user and the helper can look up. It's key that we encourage people to contribute there. Having some sort of dispute resolution system is definitely needed; I'll give it some thought. It also needs a usability and functionality upgrade or five.

I would be against running a forum like Bugzilla, marking things as duplicates of others. I don't think users would take too kindly to their question being marked as a duplicate. What sounds like the same problem could have different causes, and even if it is the same problem, the user might require hand-holding. The sheer number of posts would also make this very difficult. It may be useful, however, to classify an issue internally so that we can keep track of what the common problems and solutions are.

I think it's unrealistic to expect to have a squad of super-helpers who know everything. A helper may know about how to get a web page to work, but know nothing of plugins. I wouldn't want to have that helper hesitate to attempt to help a user having a plugin problem because they fear they might be wrong and lead the user to a bad solution, not to mention possibly lose their "super-helper" status. Having a hierarchy where people are moved up and down based on a group of people's opinions would also create bitterness. It may work if we hook it up to an algorithm of some kind that would pull from the data on if the user thought something was helpful.

[Alice]

As for the knowledge base... I think whatever help repository mozilla officially uses MUST be moderated. I got into some fun trying to make the lost bookmarks article easier for users to navigate to their solution, but another kb contributor disagreed with part of changes. Instead of communicating to me so that we could sort out the right thing for the document it ended up being changed back and forth several times. With no one to appeal to who could lock down the article, or at least pass judgement on how the article would best help users, I gave up. It also makes me disinclined to try updating other articles where I know they're out of date, or I've discovered the solution

I'm sorry if our disagreement as to the layout and content of the Lost Bookmarks article discouraged you from making further contributions to the KB.... it shouldn't have. The KB needs contributors so please don't give up on it.

Contrary to the impression your comments might have made, I DID communicate my disagreements to you, both via forum PM and on the Lost Bookmarks discussion page. The discussion was going along nicely with give and take on both sides, except that we were the only two people involved! I think that's one of the biggest problems with the KB, lack of involvement. You can count on one hand the number of active KB contributors on any given day.

As for moderation of KB changes and the accuracy of KB articles in general:
I know that np pays attention to the recent changes page, which brings up the question of which KB sysops are currently active, whether moderating the KB editing process and article discussion is one of their functions and how the quality of KB articles should be checked. Is there an quality control procedure, other than checking the recent changes and hoping that existing errors are found and corrected?

As for conflict resolution, I'm all for it, if the parties involved can't come to a consensus.

[Guest]

> I think the thing that's putting me off the most about this thread is that a) it's only on the forums here,

There was no intention of wanting to put anyone off, and I apolgise if I have by starting this discussion in only one place.

Lets get the discussion going in a few other places if that would help, but we need to consolidate at least some of the issues in one place, and we also need to figure out how to get the support communities working better together. Your example of cross talk on the lost bookmarks kb entry highlights the kind of problem that we have.

suggestions on how to approach? which news group should we open some of the newsgoup support related problems?

I've set up a wiki page that might point to discussions that might be specific to various channels, and a place that we might consolidate central issues.

http://wiki.mozilla.org/Support:ImprovingSupportOptions

Will that help, or confuse? Not sure what the right answer is.

b) you mention live support, but not IRC

sorry also for leaving that out. lets definitly include IRC in the support systems that we need in this complicated ecosystem. The live support qunu ideas were intended to open up discussion on some tools that are zero software install. I see lots of frustrantion on webmaster with comments like "I've lost my bookmarks and you want me to install this strange thing called IRC so I can talk to someone about solviing the problem?... I'm out of here!" We need to figure out a solution for this growing class of users.

Figuring out a way to provide voice support services also needs to be looked at as part of the real-time options.

> and c) it doesn't seem like the rest of the mozilla developer community has been approached first about how *they* think it's best to meet user support needs. This has definitely come up among other parts of the community, and there are some really exciting ideas.

Let start sharing the ideas, raising the visibility, and working on some of them, and try to coordinate some of the work where it makes sense. That's all I'm trying to get rolling...

chris h.

[VanillaMozilla]

Users often do not give complete, accurate information. Getting it from them is a struggle. Too many questions, too few answers. The KB is much more comprehensive, so you just point to an article. But the article can be overwhelming (and a lot of effort has been expended to keep it simple and clear -- but we're dealing with computers).

The KB has two, somewhat contradictory functions: (1) to be a knowledge base, so users can find out how things work and do their own troubleshooting; and (2) to give simple instructions on what to do. Most of the time there's no way to say "Do this."

Success Stories
There are some success stories. One of them is that the number of firewall problems is down dramatically from previous levels. Trolly's enumprocess tool helped in a number of cases, and it helped turn the tide. I think improvements in the KB firewall article helped a lot, but I can't prove it. Finally, manufacturers got their acts together -- or at least Symantec did. Now if we can get that firewall detection....

Trolly's two tools have been very helpful. The new version of wdnslookup includes an nslookup variant, ping, and tracert. Now I need to write a KB section on how to use it for diagnosis. I'm pretty sure most "can't-connect" problems are networking problems, not Firefox problems, and a simple, systematic approach with that should show people why they can't connect. This is a promising approach. Getting people to use it is another problem, but the KB should help.

<b>Finally, the Bottom Line -- (I think)</b>
Given support is largely a process of working through a complex tree, with typical users giving wrong answers at about 30% of the branches. That's why tools are needed. The user can work through an automated tree much faster than through a quarrelsome discussion group. Mozilla ought to develop tools. There are too many ways to screw up to cover all problems, but some of the common ones could be covered, e.g.:

(1) A nice, Web-based Q&A tree like everyone else has. ("Screen still black? Try plugging in your computer.")
(1a) An automated text device like np suggested.

(2) "Did you really, really search for hidden files? OK, Here's where your bookmarks are. Try to remember next time. Mozilla promises to fix it Real Soon Now."

(3) How about an automated Trolly tool for network diagnosis? ("Can't ping Google. You don't have a network connection. At all.") ("I'm still waiting for an IP address. Contact your ISP or consider voodoo.")

(4) The Firefox Diagnostics tool

[VanillaMozilla]

The KB <i>does</i> have good ownership and arbitration. There are two people who look after it constantly, and one or two others who also take part. These people are knowledgeable and write well. The discussion pages work well. For a committee effort I think the KB is a miracle. One possible improvement would be to find someone who has infinite time and knows everything, but I don't see that as a useful suggestion.

A Google search engine for it would help a lot, however.

[Lucy]

For the people objecting to duplicating threads - the exact same issue is faced on Bugzilla. If someone mistakenly dupes an issue in bugzilla it's not just a matter of someone not getting an answer, it's an issue of something not getting fixed (which isn't to say that one isn't as important than the other, just that one has larger impact). If there isn't enough information to be sure of the issue, then a thread shouldnt be duped, just as bugzilla bugs shouldn't be until the issue is clear.

The great thing about dupes is it leaves myriads of ways to get to the same solution. Eventually you end up with enough dupes that no matter how a user searches for an issue, they will find the parent bug because at least one of the dupes turns up in their results. Also, as triagers, the more bugs you see get duped to a certain bug, the easier it is for you to remember where the parent bugs are that actually contain the complete information.

In the end though these are only my ideas on how I would run the forums. I do think the best idea is to have a Mozilla controlled knowledge base so that everyone has reliable information that they can then give to their users however they wish, and Mossop's champions can make sure to share new information as each different support area stumbles upon it.

[Pike2]

I'd follow Lucy in that duping user support requests is a bad idea in principle. From an asynchronous mail-based system, we're using otrs on mozilla-europe.org, which isn't perfect. But that's where I'm conceptually coming from. (They have demos etc on http://otrs.org, but it didn't turn me to it right now, and it's been a while since I worked on the system in our installation, I'm afraid.)

Each user opens a request, basically in a free form web-based thingie, which would fork the following actions:

- paste the entry to an IRC channel, including a reference to a ticket-specific channel
- query the KB based on text analysis and history, which should hopefully suggest a few KB articles

The user can evaluate the KB articles and mark the ticket as "I'm good", or do a "wtf???", which would harrass the IRC channel again.

The web interface would include a chat interface proxying the discussion in the ticket specific channel.

The ticket would be associated with the query, the suggested KB articles including their rating, plus the chat log.

All that somewhat more or less sandboxed by language.


As our guest chofmann mentioned it, if we're taking voice support into the game, this system could serve as level-2 - -3 backup and training material for the call center. In which case some kind of micropayment system behind might be appropriate, to share the revenue from the phone calls between those with the phone skills and those with the in-depth skills.

Is that vague and complex enough to be totally unrealistic? I guess . Anyway, that's how I think about it.

[old np]

In regards to "duping" threads, what often happens is the reply to an often-asking question is a link to a Knowledge Base article. The effect of that is that a search leading to any of the duplicate threads leads to the KB article, which should be the most up-to-date information even if the original question was posted a year earlier.