Meeting 20111109

Development

• Joseph has been testing and such with scalability issues
  • Successes with a large number of files
  • He will build today the path around the watched directories for SIP creation
  • He thinks he still need to look at the thread creation (SIP creation)... looks like some threads spawn and aren't removed. Deciding how to manage that, might get bumped to 0.9.
• Peter suggests to have a scalability refactor discussion very early on in the 0.9 development schedule.
• There are lot of remaining issues, a Monday code freeze is not realistic
• We can bump code freeze by a week, that then still leaves 2 weeks between code freeze and release.
• Evelyn to update the issue list and mark more criticals now that we're getting close
• Austin: ace is up and monitoring ~1000 documents
• Meeting scheduled on Mon, Nov 14 11am

Documentation

Evelyn updated Significant characteristics of raster images wiki page

Deployment

• Tomorrow, CVA and UBC
• Evelyn and Austin will do the RAC install the week of Dec 12
  • That could get us an extra week in the 0.8 dev schedule

Chat log

(10:34:09 AM) peterVG: okay, let's roll
(10:34:29 AM) berwin22: Been testing and such with scalability issues
(10:34:44 AM) berwin22: Successes with a large number of files
(10:34:51 AM) epmclellan: nice!
(10:35:02 AM) epmclellan: that's been a big issue this past week
(10:35:16 AM) epmclellan: Sevein has been doing all kinds of dashboard dev
(10:35:20 AM) berwin22: today I'll build the path around the watched directories for SIP creation
(10:35:41 AM) peterVG: okay
(10:36:03 AM) peterVG: so the background story to ^ this is that we're running into what appears to be a Python threading limitation
(10:36:16 AM) Sevein: yep, I did a big refactor in the dashboard pages using twitter bootstrap and lot of new code to manage transfer and sips individually
(10:36:26 AM) berwin22: I think I still need to look at the thread creation there (sip creation)... looks like some threads spawn and aren't removed. Deciding how to manage that. Might get bumped to 0.9
(10:36:30 AM) peterVG: which means we have a small, hard cap on total number of files that can be processed at once
(10:37:14 AM) peterVG: ...and we'll have to skip/have ability to turn off Transfer SIP creation for 0.8 as it (appears to) spawn a thread for each file in the Transfer area
(10:37:14 AM) djjuhasz: :(
(10:37:45 AM) peterVG: so we'll have to have a scalibility refactor discussion very early on in the 0.9 development schedule.
(10:37:47 AM) berwin22: I did some more testing on that. I think it's for each SIP/Transfer
(10:37:57 AM) djjuhasz: sounds like some reasonable compromise
(10:38:03 AM) berwin22: I may be able to get around it, spent some time looking into it yesterday.
(10:38:18 AM) peterVG: berwin22: fingers crossed
(10:38:31 AM) berwin22: I'm finding it challenging to debug python threads.
(10:39:20 AM) berwin22: Looking at the calendar... Monday is code freeze, and tomorrow we're at CVA
(10:39:40 AM) epmclellan: I updated the scalability testing page: http://www.archivematica.org/wiki/index.php?title=Scalability_testing
(10:39:42 AM) berwin22: I think there are other issues we wanted me to work on before release
(10:39:59 AM) epmclellan: there are a lot of remaining issues, a Monday code freeze is not realistic
(10:40:04 AM) peterVG: agreed
(10:40:07 AM) epmclellan: quite a few METS issues
(10:40:08 AM) ARTi2: cva&ubc?
(10:40:21 AM) epmclellan: ARTi2: I'll be at CVA
(10:40:27 AM) epmclellan: youdon't really need me at UBC
(10:40:36 AM) ARTi2: ok
(10:40:47 AM) epmclellan: Lisa will be there, she has been experiencing the problems
(10:40:53 AM) epmclellan: and can explain/test them
(10:41:07 AM) epmclellan: anyway, yes, we have quite a few outstanding issues
(10:41:09 AM) ARTi2: great
(10:41:15 AM) epmclellan: and some more will crop as dashboard changes are made
(10:41:37 AM) epmclellan: because sometimes issues don't become apparent until you use the new dashboard features
(10:41:44 AM) epmclellan: but I'll try to be realistic :)
(10:41:51 AM) peterVG: epmclellan & ARTi2 would you guys be able to do the RAC install the week of Dec 12?
(10:41:56 AM) epmclellan: yes
(10:42:05 AM) peterVG: that could get us an extra week in the 0.8 dev schedule
(10:42:06 AM) ARTi2: yes
(10:42:12 AM) peterVG: okay, good. 
(10:42:14 AM) epmclellan: assuming we can get flights
(10:42:19 AM) epmclellan: it's the busy season
(10:42:25 AM) epmclellan: we'd better book asap
(10:42:33 AM) peterVG: hmm, yes. let's get on that today then. first i'll confirm dates with RAC
(10:42:37 AM) epmclellan: don't wanna go to New York via the Yukon
(10:42:41 AM) peterVG: for sure
(10:42:52 AM) ARTi2: ;-) 
(10:42:54 AM) peterVG: but let's assume that's the plan. we can bump code freeze by a week
(10:43:00 AM) epmclellan: peterVG: and with the UN?
(10:43:20 AM) epmclellan: dates, I mean
(10:43:21 AM) peterVG: that then still leaves 2 weeks between code freeze and release
(10:43:36 AM) peterVG: epmclellan: yes, I'll confirm those as well. tag a day onto RAC
(10:43:41 AM) epmclellan: ok, thx
(10:44:49 AM) epmclellan: ARTi2: re scalability testing
(10:44:58 AM) epmclellan: can we have zip of Corpora files on wiki?
(10:45:04 AM) epmclellan: For testing purposes
(10:45:24 AM) peterVG: so do we have the list of criticals for code freeze then? http://code.google.com/p/archivematica/issues/list?can=2&q=milestone=Release-0.8&sort=priority&colspec=ID%20Milestone%20Component%20Type%20Status%20Priority%20Owner%20Summary
(10:45:40 AM) epmclellan: I think I need to mark more criticals
(10:45:44 AM) epmclellan: now that we're getting close
(10:45:54 AM) epmclellan: I've marked a lot of issues as high that are probably critical
(10:45:56 AM) peterVG: is there an issue(s) for the transfer scalability & SIP scalablity problems?
(10:46:03 AM) epmclellan: hmm
(10:46:12 AM) epmclellan: I think so
(10:46:25 AM) ARTi2: yes @ digicopora files
(10:46:30 AM) epmclellan: thanks
(10:46:59 AM) epmclellan: berwin22: I believe you filed the scalability issue? Just looking for it
(10:47:00 AM) peterVG: ok, well we now have a hard deadline for the 0.8 code freeze so that will be the determinant on how many 'highs' we can accomodate as 'criticals'
(10:47:21 AM) peterVG: should we book a mtg next week to review this list?
(10:47:25 AM) ARTi2: ace is up and monitoring ~1000 documents
(10:47:31 AM) berwin22: I filed one on our internal issues, not sure about google
(10:47:33 AM) peterVG: ARTi2: v. cool!
(10:47:41 AM) epmclellan: yes, it's very cool
(10:47:54 AM) epmclellan: berwin22: can you put it on google too?
(10:48:18 AM) epmclellan: peterVG: I'll assign what I think are the criticals, we can review next week
(10:48:30 AM) peterVG: okay, let's schedule a mtg now
(10:48:52 AM) peterVG: joseph isn't here on mon
(10:48:57 AM) peterVG: i'm at ubc on tue
(10:49:07 AM) peterVG: wed the 16th?
(10:49:08 AM) epmclellan: berwin22: you're planning to come in on Monday, yes?
(10:49:19 AM) peterVG: oh, right, instead of the Fri?
(10:49:23 AM) peterVG: that would work well
(10:49:28 AM) epmclellan: Monday would work for me
(10:49:58 AM) epmclellan: berwin22: ?
(10:50:14 AM) berwin22: yes, monday in
(10:50:17 AM) berwin22: is the plan
(10:50:18 AM) epmclellan: great
(10:51:03 AM) peterVG: cool
(10:51:04 AM) Sevein: Evelyn updated "Significant characteristics of raster images" wiki page
(10:51:06 AM) peterVG: 11am?
(10:51:10 AM) Sevein: on docs
(10:51:11 AM) berwin22: Issue 388:Stress tests with multiple clients < what's the status of this issue
(10:51:27 AM) epmclellan: thanks Sevein
(10:51:40 AM) epmclellan: 11:00 is fine
(10:52:11 AM) peterVG: ARTi2: may be our last chance to be all together prior to 0.8 release so would be good if you could make it so we cover all logistics for upcoming weeks
(10:52:12 AM) berwin22: http://code.google.com/p/archivematica/issues/detail?id=388
(10:52:23 AM) epmclellan: thanks berwin22
(10:52:48 AM) ARTi2: sounds good.
(10:53:02 AM) ARTi2: be there Monday.
(10:53:20 AM) peterVG: okay, great. i've just put in calendar for 11am. 
(10:53:41 AM) berwin22: btw... I have plans the third, probably won't make it
(10:54:08 AM) peterVG: oh no!
(10:55:28 AM) ARTi2: were talking the next Monday right?
(10:55:46 AM) epmclellan: yeah, the 14th
(10:56:07 AM) ARTi2: thanks
(10:56:47 AM) JessicaB1 [~Jessica@70.71.0.109] entered the room.
(10:56:47 AM) JessicaB left the room (quit: Read error: Connection reset by peer).
(10:57:30 AM) Sevein: http://archivematica.org/wiki/index.php?title=Meeting_20111109
(10:57:35 AM) Sevein: could someone paste the chat log, please?
(10:57:41 AM) epmclellan: I can do that
(10:57:46 AM) Sevein: thank you!