01:00:17 <cprofitt> #startmeeting ubuntu-friendly
01:00:17 <meetingology> Meeting started Wed Feb  6 01:00:17 2013 UTC.  The chair is cprofitt. Information about MeetBot at http://wiki.ubuntu.com/meetingology.
01:00:17 <meetingology> 
01:00:17 <meetingology> Available commands: #accept #accepted #action #agree #agreed #chair #commands #endmeeting #endvote #halp #help #idea #info #link #lurk #meetingname #meetingtopic #nick #progress #rejected #replay #restrictlogs #save #startmeeting #subtopic #topic #unchair #undo #unlurk #vote #voters #votesrequired
01:00:35 <cprofitt> #meetingtopic welcome back
01:00:46 <cprofitt> hello all -- and welcome to the Ubuntu Friendly meeting
01:01:00 <cprofitt> can everyone that is here for the meeting just let me know you are in the room
01:02:09 <SergioMeneses> o/
01:02:12 <phillw> o/
01:02:40 <cprofitt> thanks for coming everyone
01:02:51 <cprofitt> our agenda is:
01:02:59 <pleia2> o/
01:03:15 <cprofitt> hexr, wiki cleanup, how to improve parts of the Ubuntu Friendly system
01:03:24 <cprofitt> your questions and any advice/ideas you all have
01:03:49 <cprofitt> baloons are you here?
01:03:56 <SergioMeneses> #link https://wiki.ubuntu.com/UbuntuFriendly/Meetings
01:04:29 <cprofitt> alright... I might have baloons take over if he comes in, but let me tell you a bit about hexr
01:04:51 <cprofitt> #meetingtopic hexr
01:05:12 <cprofitt> #link https://launchpad.net/hexr
01:06:30 <cprofitt> hexr, from my understanding, will potentially make use of the same data Ubuntu Friendly does, but it will do so to see what harware components have been tested
01:06:50 <cprofitt> so it will seek to find out of a specific Nvidia card has been tested or a specific wireless card
01:07:06 <cprofitt> and not look at systems, but components
01:07:21 <cprofitt> #link https://launchpad.net/~hexr-dev/+members#active
01:07:36 <cprofitt> the developers are listed on that page
01:08:24 <cprofitt> I think knowing that could help Ubuntu Friendly add some more functionality to the site -- where people might be able to sort systems based on a desired video card or wireless card...
01:08:43 <cprofitt> for now it is just something for us to know as we move forward
01:09:22 <cprofitt> #topic wiki clean up
01:09:41 <cprofitt> how many of you have had time to look at the wiki page for UF?
01:09:58 <cprofitt> #link https://wiki.ubuntu.com/UbuntuFriendly/
01:10:53 <cprofitt> would anyone  be willing to take on reviewing the pages to ensure that they are up-to-date?
01:11:00 <SergioMeneses> cprofitt, we can put some icons
01:11:04 <SergioMeneses> cprofitt, I can do it
01:11:30 <cprofitt> #action SergioMeneses will review the Ubuntu Friendly wiki
01:11:30 * meetingology SergioMeneses will review the Ubuntu Friendly wiki
01:11:34 <cprofitt> thanks SergioMeneses
01:11:45 <cprofitt> please feel free to ask for help from others as you do that
01:12:05 <cprofitt> if anyone else knows of pages that need updates please let SergioMeneses or me know
01:12:14 <SergioMeneses> cprofitt, ok
01:12:26 <chilicuil> ok =)
01:12:36 <cprofitt> #topic improvements
01:13:12 <cprofitt> I know from personal experience we have some things to improve moving forward
01:13:21 <cprofitt> phillw: did you want to discuss some of the thoughts you had?
01:13:56 <cprofitt> anyone else?
01:14:28 <cprofitt> #subtopic submissions
01:14:34 <phillw> cprofitt: just to mention that you and laptop team are quite similar and I hope there is a good flow of informations between you both :)
01:14:47 <cprofitt> phillw: I would hope so as well.
01:15:01 <cprofitt> One of the things that got me involved was submissions
01:15:29 <cprofitt> right now it is my understanding that submissions are not getting in to UF
01:15:35 <cprofitt> but I am not sure why that is
01:15:40 <pleia2> phillw: hm, I thought the laptop testing team was for testing Ubuntu on laptops for upcoming releases, not reporting on how well supported laptops are in current releases
01:16:31 <SergioMeneses> Im totally agree with pleia2,
01:17:10 <phillw> pleia2: ergo, one a release is out, that information is available to what is then the 'current' release?
01:17:21 <phillw> *once*
01:18:00 <cprofitt> I would like to see two things done to improve in regards to submissions
01:18:05 <pleia2> phillw: but it would be results for when it was in development, so maybe the graphics card for that one laptop finally did end up working once the release happened, the results would be from when it wasn't working :\
01:18:15 <pleia2> for example
01:18:39 <phillw> i see, I'll have a chat with the laptop team :)
01:18:49 <cprofitt> pleia2: I agree.. the results from pre-release could would not be invluded in Ubuntu Friendly
01:19:19 <pleia2> they're both valueable and needed projects, for sure, I just thing they provide different data to different audiences
01:19:25 <pleia2> s/thing/think
01:19:28 <cprofitt> but the laptop testing team could, upon release, be a group of people who ran the tests for UF
01:19:36 <pleia2> cprofitt: indeed!
01:20:05 <cprofitt> and from my understanding the database we are pulling from might actually be the same -- but with different criteria for what information is included
01:20:13 <cprofitt> and also differences in how it gets displayed
01:20:21 <pleia2> it makes sense for both projects to test the same things
01:20:27 <cprofitt> but I would need one of the developers to give me more information on that
01:20:52 <cprofitt> I would like to see two things done to improve in regards to submissions
01:21:19 <cprofitt> 1.  some system be put in place to make it easy for those submitting results to know that the results were processed
01:21:31 <cprofitt> if the result was invalid - let them know why
01:22:05 <cprofitt> as a person who has submitted results and never saw my results in UF I know how that can cause someone to lose interest in contributing
01:22:52 <pleia2> +1
01:23:13 <cprofitt> 2.  I would also want to have a group of people that could look at invalid results to ensure that how the data is being processed is accurate -- that the automated system is not makring valid results as invalid
01:23:30 <cprofitt> this is particularly important as hardware changes...
01:23:52 <cprofitt> failure to test a card reader when there is no such device present should not invalidate results
01:24:04 <cprofitt> any other comments in regards to submissions?
01:24:58 <cprofitt> #subtopic checkbox
01:25:10 <cprofitt> that brings me to one more idea
01:25:24 <cprofitt> this one is not mine, but one that has been discussed in the qa channel
01:25:52 <cprofitt> there may be a need to refine checkbox a bit to allow for a 'simple' test
01:26:06 <cprofitt> or in the case of hexr a test of specific hardware
01:26:36 <cprofitt> the simple test would be an abbreviated test to make sure that basic things like wireless work etc.
01:27:07 <cprofitt> I also had cases when I tested where the automated test 'failed' itself, but I knew the device was working
01:27:39 <cprofitt> I think there should be some sort of mechanism for making a note of that issue built in to checkbox so that changes can be addressed with those issues
01:28:03 <cprofitt> having devices 'fail' a test when they are actually working undercuts people's faith in the process and the results.
01:28:20 <cprofitt> any thoughts in that area?
01:28:32 <roadmr> o/
01:28:43 <cprofitt> yes, roadmr
01:29:02 <roadmr> hello! first, if an automated test fails when the device is working, that means a bug that we'd like to get fixed
01:29:21 <roadmr> I understand this adds a lot of friction to things, so maybe thinking of ways to make this more effortless may be worthwhile
01:29:31 * cprofitt nods
01:29:42 <roadmr> second, the automated tests are in general more reliable but they are also quite valuable in high-volume environments
01:30:07 <roadmr> if we assume that people can be trusted, we can potentially use manual test cases instead of automated ones that are most unreliable
01:30:30 <roadmr> even so, it would be good to collect some information that can be automatically analyzed, maybe server-side
01:30:46 <roadmr> this way we can potentially re-score a submission given the raw data, even after-the-fact
01:30:51 <pleia2> admittedly there is some trust to an extent already in the testing (you can hear a sound, etc)
01:30:59 <cprofitt> I too, would prefer to get the automated tests working... I think human interaction always introduces some potential for error
01:31:08 <roadmr> this would however introduce a "we're collecting stuff and sending it up to canonical" factor that some people don't like
01:31:19 <cprofitt> I do like rescoring a submission after the fact
01:31:33 <roadmr> so this is an area for improvement but we always have the option of doing this manually
01:31:47 <cprofitt> roadmr: in this case I think the 'collecting data' part is voluntary
01:31:54 <cprofitt> at least with checkbox
01:32:15 <roadmr> cprofitt: yes, well in order to determine what's going on, some extensive logs would need to be collected
01:32:22 * cprofitt nods
01:32:30 <roadmr> cprofitt: but keeping the user in control would be the best
01:32:48 <roadmr> pleia2: hehe, we sought to remove the human factor from that by adding an automated audio_test :) so the computer listens to itself
01:32:56 <cprofitt> could that be an optional part of checkbox -- that a user would opt-in to collecting additional data?
01:32:57 <roadmr> anyway, those are my ideas on this
01:33:32 <pleia2> roadmr: does it work? I guess I haven't run it in a bit :)
01:33:35 <cprofitt> I appreciate the feedback and information roadmr
01:33:35 <roadmr> cprofitt: sure, with the current checkbox architecture it feels a bit complex to do, but we can always change that, we should bend the tool to the needs, not the other way around
01:33:51 <roadmr> pleia2: as long as you have a microphone and speakers, it works and it's very reliable
01:33:59 <pleia2> nice
01:34:15 <cprofitt> roadmr: +1 we want to make it easy for people to collect data for this type of use -- improving user experience
01:34:58 <cprofitt> roadmr: do you have any insight as to why results are not getting included in to Ubuntu Friendly or is that outside of your scope?
01:35:42 <roadmr> cprofitt: I have a pretty good idea why, yes
01:35:52 <cprofitt> could you go a bit in to that for us?
01:36:13 <roadmr> cprofitt: two components are involved in this, the first is checkbox, which runs the tests, assembles a report and pushes that to launchpad
01:36:45 <roadmr> the second is the UF website/application, which pulls the reports (submissions) from launchpad and adds the results to the friendly database (what you see in the website) - does the scoring, aggregation and such
01:37:14 <roadmr> for a submission to appear at all, it needs to contain (and pass) all tests identified as "core"
01:37:30 <roadmr> the basics for a computer to be usable, if any core tests are failed the submission will not appear in UF
01:37:52 <roadmr> if it passes the "core" tests, it gets 3 stars, and any additional tests it passes increase its rating, up to 5 stars
01:38:06 <roadmr> a 5-star rating means that all the components we test work perfectly
01:38:14 <roadmr> so why would a submission not appear?
01:38:39 <roadmr> some of the core tests are automated, and checkbox is not very good at 1) telling you one of them failed, and 2) letting you rerun/amend the results
01:39:19 <roadmr> checkbox itself has no idea of core tests, so it can't tell you "hey, your card reader didn't work and that means you won't make it into UF - please rerun this before submitting"
01:39:42 <cprofitt> is there currenlty anyway for a person who has submitted the test to see their submissions on launchpad?
01:40:16 <roadmr> that's one reason, the second is that since the UF website needs some love, there may have been a bit of "drift" into the tests checkbox submits and performs, the UF site has not been updated as checkbox whitelists (sets of tests to run) change
01:40:23 <roadmr> those would be the two main reasons
01:40:36 <roadmr> cprofitt: there's a cryptic url that you an access to see your submissions, let me find it
01:40:53 <cprofitt> so, it sounds like the key is giving the user some feedback as to if their device failed a core test... and also allowing for them to resubmit
01:40:53 <roadmr> cprofitt: https://launchpad.net/~/+hwdb-submissions
01:41:37 <roadmr> cprofitt: yep, again, something that's a bit difficult to do with the current checkbox architecture which is very linear :( think of the difference between linear and non-linear video editing
01:41:47 <roadmr> in a sense, with checkbox we can only ffwd, rewind, and look at the results at the very end
01:41:58 * cprofitt nods
01:42:00 <roadmr> with no/little chance of going back to a specific test.
01:42:08 <cprofitt> exactly...
01:42:28 <cprofitt> it is also my understanding there is no ability to 'pause' a test and come back and pick up where you left off
01:42:49 <roadmr> there is, but it's very awkward heheh
01:42:51 <cprofitt> roadmr: is there currently any team that reviews the failed submissions?
01:43:07 <roadmr> what you do basically is murder checkbox, when you restart it does its best to pick up where it left off
01:43:30 <roadmr> it gives you three choices: rerun the last test, skip to the next one (in case the last test outright crashed the system), or start anew
01:43:42 * cprofitt nods
01:43:50 <roadmr> the message is quite terse and the choices are somewhat ambiguous, so most people are/will be confused by this
01:44:36 <roadmr> cprofitt: reviewing failed submissions, not really, no. It can be done but pretty much the only person who can and knows how to do it is jedimike, by looking at how the UF results importer processed a submission
01:44:48 <roadmr> he'd have more specifics on how to do it, but it's done on an ad-hoc basis
01:45:00 <cprofitt> roadmr: thanks
01:45:07 <roadmr> np :)
01:45:14 <cprofitt> I asked because I saw that as part of the launchpad team description
01:45:27 <cprofitt> does anyone else have any questions for roadmr?
01:46:19 <cprofitt> I would like to set an initial priority for the team moving forward
01:46:33 <cprofitt> I think that we have two high value items
01:47:12 <cprofitt> 1.  Improving the UF website process so that there is some feedback to the person who submitted the system as to the fact the results were processed but failed
01:47:35 <cprofitt> 2.  Documenting the current process - so others can understand it as we start to look at improving it
01:47:42 <cprofitt> are there any other items people see?
01:48:47 <cprofitt> roadmr: would you be willing to procude a flow chart of what you described that we could them use on the wiki?
01:48:57 <cprofitt> just a rough outline of how the process currently flows?
01:49:08 <roadmr> cprofitt: sure
01:49:22 <roadmr> cprofitt: there's some documentation (an outline plus more detailed links) here: https://friendly.ubuntu.com/what-is-ubuntu-friendly/
01:49:23 <cprofitt> #action roadmr will produce a flow chart of the current process
01:49:23 * meetingology roadmr will produce a flow chart of the current process
01:49:53 <cprofitt> #vote agree that improving feedback on submissions is important in moving foward
01:49:53 <meetingology> Please vote on: agree that improving feedback on submissions is important in moving foward
01:49:53 <meetingology> Public votes can be registered by saying +1, +0 or -1 in channel, (private votes don't work yet, but when they do it will be by messaging the channel followed by +1/-1/+0 to me)
01:49:57 <cprofitt> +1
01:49:57 <meetingology> +1 received from cprofitt
01:50:12 <roadmr> +1 definitely heh
01:50:12 <meetingology> +1 definitely heh received from roadmr
01:50:33 <chilicuil> +1
01:50:33 <meetingology> +1 received from chilicuil
01:50:35 <SergioMeneses> +1
01:50:35 <meetingology> +1 received from SergioMeneses
01:50:44 <cprofitt> pleia2: ?
01:50:48 <cprofitt> phillw: ?
01:50:56 <phillw> +1
01:50:56 <meetingology> +1 received from phillw
01:51:05 <phillw> sorry wasnm't sure if i was to vote :)
01:51:23 <cprofitt> #endvote
01:51:23 <meetingology> Voting ended on: agree that improving feedback on submissions is important in moving foward
01:51:23 <meetingology> Votes for:5 Votes against:0 Abstentions:0
01:51:23 <meetingology> Motion carried
01:52:01 <cprofitt> I think for now we should move towards that; after we get that feedback improved we need to look to make some improvements to checkbox as roadmr mentioned
01:52:16 <cprofitt> any other topics, questions or thoughts?
01:52:37 <roadmr> o/
01:52:42 <cprofitt> yes, roadmr
01:53:00 <SergioMeneses> when is our next meeting?
01:53:10 <roadmr> hehe, one thing that we *can* potentially do with current checkbox is make it submit directly to a (possibly updated) UF application, so feedback is quicker
01:53:33 <roadmr> this needs writing a "plugin" and an accompanying web application, so it would need some coordination
01:53:33 <cprofitt> +1 roadmr that should be explored
01:53:51 <roadmr> we're still on time to make that happen before Ubuntu Feature Freeze, but we'd have to start working soon
01:54:13 * cprofitt nods
01:54:24 <roadmr> ideally we sould ask jedimike if there's a way to submit directly to the UF application, because we may not have a lot of manpower to come up with a new one
01:54:31 <roadmr> sould/should
01:54:46 <cprofitt> SergioMeneses: I would like us to meet again in two weeks -- I think to be fair to some of the UTC
01:54:49 <roadmr> that may be a quick way to reduce the decoupling between submission and actual result processing
01:55:22 <cprofitt> folks we should have the meeting earlier; though that would require a different person to chair the meeting
01:55:36 <SergioMeneses> perfect
01:55:41 <cprofitt> roadmr: I know jedimike has spoken to me about that so I will try to touch bases with him about that
01:55:56 <cprofitt> are any of you in the UTC 0/+1 TZ?
01:57:01 <phillw> 'im on utc
01:57:01 <cprofitt> alright... I will send a message to the list and see if I can get a person in that TZ to chair a meeting.
01:57:02 <roadmr> cprofitt: I'm EST (UTC-5)
01:57:19 <cprofitt> phillw: would you be willing to chair a meeting on the 19th?
01:57:27 <cprofitt> roadmr: I am on your TZ as well
01:57:33 <phillw> I can if no one steps forward :)
01:57:49 <SergioMeneses> Im UTC-5
01:57:55 <cprofitt> phillw: ok, I will send the request to the list and you will be the fall back
01:58:36 <cprofitt> #action next meeting February 19th UTC 17:00 - cprofitt will send a message to the list to get a chair
01:58:36 * meetingology next meeting February 19th UTC 17:00 - cprofitt will send a message to the list to get a chair
01:58:36 <phillw> what time on 19th are you thinking of?
01:59:05 <phillw> yeah, 1700 utc would be okay if no one steps forward.
01:59:09 <SergioMeneses> 17 utc
01:59:12 * cprofitt nods
01:59:20 <cprofitt> yes, SergioMeneses so noon our time
01:59:52 <SergioMeneses> cprofitt, it's not problem for me
01:59:58 <cprofitt> thanks SergioMeneses phillw pleia2 chilicuil roadmr for attending. I think this has been very productive for an initial start
02:00:04 <cprofitt> #endmeeting