#title #ubuntu-meeting: QA Meeting Meeting started by gema at 17:02:01 UTC. The full logs are available at http://ubottu.com/meetingology/logs/ubuntu-meeting/2011/ubuntu-meeting.2011-12-21-17.02.log.html . == Meeting summary == *Previous Actions ''ACTION:'' patrickmw to publish a list of launchpad projects that conform our automated testing in jenkins (gema, 17:04:01) *Blueprints Update Precise *https://blueprints.launchpad.net/ubuntu/+spec/other-p-builds-smoke-testing (gema, 17:06:49) *#subtopic https://blueprints.launchpad.net/ubuntu/+spec/other-p-qa-boot-speed-testing (gema, 17:12:44) *https://blueprints.launchpad.net/ubuntu/+spec/other-p-qa-boot-speed-testing (gema, 17:13:08) *https://blueprints.launchpad.net/ubuntu/+spec/other-p-qa-kernel-sru (gema, 17:14:36) *https://blueprints.launchpad.net/ubuntu/+spec/other-p-qa-test-case-management-tool (gema, 17:14:59) *https://blueprints.launchpad.net/ubuntu/+spec/other-p-qa-backlog (gema, 17:39:19) *https://blueprints.launchpad.net/ubuntu/+spec/other-p-qa-qa-regression-testing (gema, 17:42:25) ''LINK:'' https://launchpad.net/~nuclearbob/+archive/ppa (nuclearbob, 17:43:20) *https://blueprints.launchpad.net/ubuntu/+spec/other-p-qa-automated-test-submissions (gema, 17:44:10) *https://blueprints.launchpad.net/ubuntu/+spec/other-p-qa-automated-test-submissions (gema, 17:44:45) *https://blueprints.launchpad.net/ubuntu/+spec/other-p-qa-metrics (gema, 17:45:12) *https://blueprints.launchpad.net/ubuntu/+spec/other-p-qa-standard-sru-testing (gema, 17:45:26) *Community Tasks - https://wiki.ubuntu.com/QATeam/TasksPrecise (gema, 17:45:45) *Update Lubuntu *Update Xubuntu *Update Ubuntu *Other Topics Meeting ended at 17:57:08 UTC. == Votes == == Action items == * patrickmw to publish a list of launchpad projects that conform our automated testing in jenkins == People present (lines said) == * gema (173) * alourie (93) * roignac (15) * nuclearbob (12) * ScottK (12) * phillw (9) * meetingology (4) * balloons (4) == Full Log == 17:02:01 #startmeeting QA Meeting 17:02:01 Meeting started Wed Dec 21 17:02:01 2011 UTC. The chair is gema. Information about MeetBot at http://wiki.ubuntu.com/AlanBell/mootbot. 17:02:01 17:02:01 Available commands: #accept #accepted #action #agree #agreed #chair #commands #endmeeting #endvote #halp #help #idea #info #link #lurk #meetingname #meetingtopic #nick #progress #rejected #replay #restrictlogs #save #startmeeting #subtopic #topic #unchair #undo #unlurk #vote #voters #votesrequired 17:02:10 who is with us today? 17:02:14 o/ 17:02:15 I am 17:02:23 o/ 17:02:39 I'll try to ping roignac 17:02:44 ok, it sounds like we have many people already on holidays 17:03:04 let's get started then 17:03:08 #topic Previous Actions 17:03:38 we had an action on patrick but he is not available today, so I will postpone it again! 17:04:01 #action patrickmw to publish a list of launchpad projects that conform our automated testing in jenkins 17:04:01 * meetingology patrickmw to publish a list of launchpad projects that conform our automated testing in jenkins 17:04:11 * gema gema to talk to hggdh about some jenkins training for the community 17:04:35 I spoke to hggdh and he has organised a training session in one developer event that is happening past mid January 17:04:55 I have asked him to send an email to the list but he was going on holidays, so I guess that'll happen next year 17:05:10 * gema gema to move the standard sru testing blueprint to the end 17:05:17 yea!! 17:05:22 I did that on the agenda 17:05:53 anything I am missing, anyone? 17:06:00 if we could contribute to anything jenkins oriented, would be great 17:06:16 yep, let me get to that later 17:06:21 k 17:06:30 #topic Blueprints Update Precise 17:06:49 #subtopic https://blueprints.launchpad.net/ubuntu/+spec/other-p-builds-smoke-testing 17:06:59 I have started reviewing the code of our ISO smoke tests and will have some feedback early in January. 17:07:16 I am not sure who to send the feedback to, but will figure it out 17:07:31 o/ 17:07:40 alourie: go ahead 17:07:56 I think that generic stuff like that should go to the list anyway 17:08:03 s/list/maillist 17:08:18 yes, I can copy the list about it, but I don't know who is responsible to fix it 17:08:21 and then other channels if necessary 17:08:21 .. 17:08:23 ah 17:08:25 ok 17:08:44 I will definitely copy ubuntu-qa whenever I send it 17:08:50 great 17:08:52 thank you 17:08:59 no problem 17:09:02 o/ 17:09:07 roignac: go ahead 17:09:24 you should also include ubuntu-server-iso-testing-dev@lists.launchpad.net 17:09:43 this is a maillist of Ubuntu Server Iso Testing Developers 17:09:44 ahh, so those are the people responsible for it, you reckon? 17:09:48 ok 17:09:53 thanks roignac 17:10:17 i guess so, please include other drivers/maintainers of automated test project (based on info from patrickw) 17:10:30 s/patrickw/patrickmw/ 17:10:35 yes, will do 17:10:56 I am also thinking that they may tell me to fix it myself, I am not sure how these things work 17:11:01 but I guess we'll find out 17:11:28 we are also looking into modifying the jenkins plugin we installed for the ISO testing dashboard 17:11:47 so that it shows the build we are testing, aggregates results, etc 17:12:09 we'll let you know if we go ahead with the enhancements, the plan would be to contribute those upstream 17:12:44 #subtopic #subtopic https://blueprints.launchpad.net/ubuntu/+spec/other-p-qa-boot-speed-testing 17:12:57 too many subtopics, let me try again 17:13:08 #subtopic https://blueprints.launchpad.net/ubuntu/+spec/other-p-qa-boot-speed-testing 17:13:27 patrickmw is not here, nuclearbob do you have any update on this? 17:13:40 yeah, we're working on adding architecture to the results now 17:13:52 as part of that we're redoing the directory structure 17:14:05 and we'll be adding some information about deltas on a given day and between days 17:14:17 that's all I've got for now 17:14:24 cool, thanks 17:14:36 #subtopic https://blueprints.launchpad.net/ubuntu/+spec/other-p-qa-kernel-sru 17:14:48 sconklin is not here either 17:14:49 moving on 17:14:59 #subtopic https://blueprints.launchpad.net/ubuntu/+spec/other-p-qa-test-case-management-tool 17:15:16 the Mozilla team plan to have the new version of Case Conductor ready for mid January 17:15:38 o/ 17:15:41 they answered yesterday night to my email and are happy to discuss our requests at some point in January 17:15:45 yes, roignac ? 17:16:09 could you please ask them whether we could use cc.oddsites.net and cc-dev.oddsotes.net freely? 17:16:22 As I've taken liberty to create a Ubiquity product 17:16:26 I did, they are happy for us to perform our tasks there 17:16:34 as long as we don't destroy their testing envs 17:16:38 since they are also using it 17:16:41 and later realised that this could break some of their automated tests 17:16:52 oh, ok. No destruction anyway 17:16:53 no, I think they run their automated tests in other place 17:16:58 cool 17:17:03 By the way 17:17:17 o/ 17:17:17 I'd like someone to try review feature in CC 17:17:30 roignac: which feature? 17:17:44 alourie, you can speak freely 17:17:48 ok 17:17:59 Case Conductor allows us to review testcases - though I haven't figured how for now 17:18:17 So we could review new testcases there, how about that? 17:18:18 roignac: I don't think it does, we asked for them to add the under review state 17:18:27 From what I've seen , CC also offers flow of testing. 17:18:41 roignac: that is a test environment, we shouldn't be using it to store data 17:18:48 it can be teared apart any moment 17:18:49 we use ISO tracker for that (well, in context of testing ISO images, at least) 17:19:03 so what is our purpose with CC? Is it only for storing test cases? 17:19:17 alourie: storing test cases and results, I would say 17:19:28 if it shows them in a meaningful way 17:19:28 alourie, it has some reports too 17:19:31 so, I would use it for reporting as well? 17:19:38 I think so 17:19:38 instead of ISO tracker? 17:19:42 eventually 17:19:46 Ah... 17:19:48 but that's not going to happen any time soon 17:19:55 it's going to take us a while to get there 17:19:59 hm, ok 17:20:06 and if it is not better than the iso tracker, then we won't use it for that 17:20:14 got ya 17:20:27 so it could only be used for storing test cases 17:20:33 to start with, yes 17:20:48 and if it does good job with it, we can only use it for storing :-) 17:20:51 and for community testing to report results 17:20:55 and defects 17:21:04 we just need to find a way to connect it to the tracker database 17:21:05 gema: but that's what ISO tracker is for! 17:21:10 hm 17:21:10 which doesn't seem very difficult 17:21:18 hello everyone -- late, but thought I would lurk 17:21:24 balloons: welcome! 17:21:47 gema: ok, I'm not clear on this, but let's just waut 17:21:54 alourie: there are two functionalities on the tracker, one is to report results, the other one is to display them 17:22:06 alourie: on a first step we are trying to replace the wiki 17:22:14 sure 17:22:15 which is horrible to store test cases 17:22:24 +1 17:22:28 after that, we'll see if we use the tracker fully, or just partially 17:22:35 depending on how CC integrates with our workflow 17:22:43 but this is not going to happen overnight in any case 17:22:50 I think that test content should show on the flow, either tracker or CC 17:23:01 we'll see how CC progresses, and make our minds later. 17:23:06 and we don't want to screw up any release so we will do a gradual move to CC 17:23:15 alourie: agreed 17:23:33 hey, if they can handle 'test case management', this is already an improvement 17:23:38 yes 17:23:41 even without flows 17:23:48 it'll flow, just give it time 17:23:55 sure it will :-) 17:24:03 and this will probably free some of our good devs to do some more interesting stuff 17:24:22 ok, the other thing I wanted to say on this topic 17:24:24 roignac: if you find that review option, ping me and we'll schedule something 17:24:36 I have been thinking about how to classify the test cases and how to structure the testing, I am not sure whether we should be testing images or whether our 17:24:45 test cases should start targetting packages themselves. 17:24:59 o/ 17:25:04 alourie: go ahead 17:25:15 I think these are different kinds of testing 17:25:25 you test images for being a complete product 17:25:42 that's the thing, alourie they are not 17:25:43 but you test packages for verifying that they work 17:25:47 gema: why not? 17:26:11 alourie: because two different people choose two different options at install time and end up with two different set of packages installed 17:26:25 so there is no one product in an image 17:26:33 there is virtually infite products in it 17:26:45 hm 17:27:25 so what would be purpose of ISO testing? 17:27:25 so I guess the question is whether we are testing the end product or an artificial construct we call default image 17:27:45 testing as many flavors of the ISO as we can 17:27:51 taking them into account 17:27:55 gema: would it be too inaccurate to assume that most people install the default image? 17:27:56 unlike now 17:28:05 alourie: I believe so 17:28:14 so testing the default flow makes sense 17:28:30 yes, and testing others makes equal sense 17:28:51 The question I'd ask is "what are you testing". 17:29:04 Generally for ISO testing you're testing the image and the installer. 17:29:15 You aren't really testing all the applications. 17:29:23 so, we are actually questioning ourselves with "what purpose does ISO testing have?" 17:29:25 That can allow you to narrow your test focus. 17:29:37 I'd agree with ScottK on this 17:30:08 People who are running the development release need to test does the stuff work after it's installed and running. 17:30:38 indeed, getting it installed and running is an important part 17:31:14 agreed, so what would you propose we do, ScottK ? 17:32:03 I think the test cases we've had are appropriately focused on what ISO testing needs to accomplish. 17:32:21 That doesn't mean they can't be improved, but they are at least in the ballpark. 17:32:43 What I've seen though is that as the installer evolves, the test cases don't keep up. 17:33:03 that's the thing, I don't think they are appropriate 17:33:22 because they are mainly installer tests, not ISO tests 17:33:30 ScottK: sounds like the classic issue of adding a feature (code-wise), but not docs or tests, etc 17:33:36 they don't tend to verify much of what's actually installed works 17:34:12 gema: so do we touch here definition of "ISO" as the complete system? 17:34:34 so, by "ISO testing" we imply testing that EVERYTHING works... 17:34:42 I would have thought so 17:34:44 or, rather, "tested" 17:34:48 hm 17:34:55 Then you better set aside a month or two for testing in the release cycle. 17:35:08 that "EVERYTHING" bit worries me a little 17:35:13 ScottK: or get more people collaborating and get as much as you can automated 17:35:23 yikes, test every default app that the standard iso holds? 17:35:25 We're an integrator and have to, to some extent, rely on upstreams to deliver working code. 17:35:30 phillw: yes 17:35:39 wow 17:35:42 this would take some time! 17:35:47 ScottK: we are an integrator and have to verify things integrate properly 17:35:48 I could see a problem here :-) 17:36:18 gema: True, but not all of that verification needs to be done in the context of ISO testing. 17:36:18 well, this is a bigger problem than this meeting, just wanted to let you know I have started to think about it 17:36:20 is it even viable? 17:36:26 ScottK: agreed 17:36:42 ScottK: but it needs to be done by ISO release time 17:36:50 alourie: it is 17:36:53 gema: so here's an idea 17:36:55 It needs to be done. 17:37:00 ok 17:37:02 what if we split this into 2: 17:37:13 1. ISO testing part 1, which includes installer and image 17:37:26 2. ISO testing part 2, which includes testing that apps work 17:37:27 .. 17:37:46 in general, they could be done in parallel/separately 17:37:55 alourie: the apps part we need to automate, I don't think running it manually unless it is necessary due to the nature of the test case works 17:38:21 alourie: I don't see how you can test that the apps work without the installer working properly 17:38:31 cos you don't have the system under test installed in that case 17:38:39 anyway, we can keep discussing this on the list 17:38:48 ok, let's take it offline 17:38:56 it was just a thought that occurred to me whilst trying to figure out how to populate CC 17:39:03 gema: ok 17:39:12 gema: I can also tell you about my secret idea then 17:39:16 :-) 17:39:19 #subtopic https://blueprints.launchpad.net/ubuntu/+spec/other-p-qa-backlog 17:39:22 alourie: cool :D 17:39:30 any backlog info to share today? 17:39:54 are community tasks separate item nowdays? 17:40:08 they are in case something got lost 17:40:12 but we can discuss here too 17:40:12 ok 17:40:21 so I can go with wiki updates 17:40:23 I haven't added the tasks to the blueprints yet because I didn't find the time 17:40:27 but you can go ahead, yes 17:40:49 ok, so I didn't have much time to work on wiki this week, but I did make some progress 17:41:10 I will also make a hierarchy of the thing I want to update, and work on them too 17:41:22 so, I hope that I'll have something to update with soon 17:41:30 and then I'll send it to the list for review 17:41:31 .. 17:41:57 alourie: excellent, thanks a lot 17:42:11 anything else anyone 17:42:12 ? 17:42:25 #subtopic https://blueprints.launchpad.net/ubuntu/+spec/other-p-qa-qa-regression-testing 17:42:28 nuclearbob: ? 17:42:44 I've got autotest packaged now, autotest-server and autotest-client are in my ppa 17:43:00 could you post the launchpad link to it? 17:43:01 I've been using those to run tests on the most recent images available in our openstack cluster 17:43:03 yeah 17:43:20 https://launchpad.net/~nuclearbob/+archive/ppa 17:43:22 thanks 17:43:55 ok, thanks a lot nuclearbob 17:44:10 #subtopic https://blueprints.launchpad.net/ubuntu/+spec/other-p-qa-automated-test-submissions 17:44:12 some of the tests are failing on the precise dailies, so I'm doing more investigation to determine whether the failures are actual problems or whether the test code needs to be updated 17:44:15 looks like some of both 17:44:16 ups, sorry 17:44:18 that's all from me 17:44:35 ok, sounds good, thanks 17:44:45 #subtopic https://blueprints.launchpad.net/ubuntu/+spec/other-p-qa-automated-test-submissions 17:45:04 no progress on this task this week, so nothing to report 17:45:12 #subtopic https://blueprints.launchpad.net/ubuntu/+spec/other-p-qa-metrics 17:45:17 no progress on this one either 17:45:26 #subtopic https://blueprints.launchpad.net/ubuntu/+spec/other-p-qa-standard-sru-testing 17:45:37 we don't have jibel around, so no progress on this one either 17:45:45 #subtopic Community Tasks - https://wiki.ubuntu.com/QATeam/TasksPrecise 17:45:53 have we missed any community task ? 17:46:06 that we have people here to discuss, I mean 17:46:10 I don't think so 17:46:14 ok, so moving on 17:46:28 #topic Update Lubuntu 17:46:33 phillw: ? 17:46:47 all is peaceful, no horrible show-stopping bugs reported. 17:47:01 good 17:47:18 anything else, phillw ? 17:47:41 at the meeting tonight they are looking to form some teams... I'm hoping for a QA / testing team :) 17:47:57 that'd be good, yes :D 17:48:08 phillw, gema: would you elaborate a bit? 17:48:21 a, the Lubuntu ;-) 17:48:44 I guess phillw is getting himself a QA Team for Lubuntu, yes 17:49:00 gema, phillw: this reminds me 17:49:05 alourie: currently everything lubuntu is on one mailing list, we are looking to start to split groups so, for example, normal users do not get the chatter from the devs about what they're working on 17:49:23 phillw: sure, 17:49:46 currently, when we list different QA subgroups on wiki, we list Kubuntu QA 17:50:10 so, I think we need to list all the QA that we have in -buntu universe there 17:50:17 alourie: that would be correct, as I'm the only person on the Lubuntu QA team (and I get lonely :P ) 17:50:23 :-) 17:50:49 good, alourie will you take care of that? 17:50:52 we are running out of time 17:50:54 gema: of course 17:51:10 cool, thanks 17:51:14 #topic Update Xubuntu 17:51:24 charlie doesn't seem to be around 17:51:28 so we are moving on 17:51:47 #topic Update Ubuntu 17:52:01 I don't think we have any updates for Ubuntu either, besides what we've already discussed 17:52:10 #topic Other Topics 17:52:23 I'd like to mention that we are not having our weekly meeting next week 17:52:28 due to holidays and festivities 17:52:34 are we all happy with that? 17:52:41 or do you guys want to have it? 17:52:41 sure 17:52:47 okay with me. 17:52:53 ok here 17:53:05 ok, so our next meeting will be the 4th of January 17:53:32 if anyone wants to chair it, let me know, otherwise I will do it :) 17:53:47 any other topic ? 17:53:50 gema: what was that bit about "participating in jenkins" stuff, that you wanted to mention later? 17:54:04 ahh, we'll start with the training that hggdh is going to give 17:54:14 and then we could have ubuntu-qa triaging issues 17:54:28 or at least, trying to learn how to do it 17:54:31 triaging issues? 17:54:44 as in, when a test fails, determine why and raise a defect if required 17:54:52 aha 17:54:53 or try to reproduce on your environment to make sure it happens 17:54:56 ok 17:54:56 that sort of thing 17:55:05 sure 17:55:05 but we need some training for that 17:55:16 nod 17:55:23 * alourie loves this stuff 17:55:27 does the bugsquad have anything for it now? 17:55:42 no, bugsquad are doing different things 17:55:47 afaik 17:56:19 we are doing it within canonical QA 17:56:31 but there is no reason why the community contributors cannot do it if they like to 17:56:38 right right.. 17:56:56 so that'd be it, any further qustions, please send to the list! :D 17:56:59 balloons: we split qa and bugsquad to 2 teams 17:57:08 #endmeeting Generated by MeetBot 0.1.5 (http://wiki.ubuntu.com/AlanBell/mootbot)