Meeting 20111012

Meeting 20111012

Development

• Austin fixed the archivematica-mcp-server package to install its databases, it looks good. The transcoder package seems to be finished, but looking at the logs he thinks it can use some touching up.. maybe Joseph and Austin can hack on it next week.
• Joseph made some backend changes to the database, that should help Jesus distinguish between SIP and Transfer JOBS. There's a lot of debuggin info printed.
• Austin fixed upstart which is now successfully starting/stopping/restarting, with gearman.
• Joseph added new normalization workflows: no normalization and access normalization only.
• Evelyn almost finished figuring out the METS profile, will put it through a validator before finalizing.
• Joseph added a means to group "jobs" into "microservices" on the back end. He thinks this will need a bunch of tweaking when we see it operational.
• Austin built some ica-atom packages to ease with archivematica builds. archivematica sans xubuntu modifications can install completely from packages now w/ out any post installation scripts. Good for 0.8.
• Joseph thinks he fixed the issue where the MCP would disconnect from the database, if it was left alone.
• Feature freeze is set for Oct. 24, code freeze for Nov. 7, release for Nov. 21. We'll have an in office feature freeze/release schedule meeting on Tue Oct 18 at 11:00 am. Evelyn will tackle the googlecode issue list for review at oct18 meeting.

Chat log

(10:32:14 AM) peterVG: archivematica mtg
(10:32:22 AM) epmclellan: I can take notes
(10:32:31 AM) ARTi: epmclellan++
(10:32:34 AM) berwin22: thanks epmclellan
(10:32:43 AM) epmclellan: np
(10:33:22 AM) peterVG: dev?
(10:34:01 AM) berwin22: is this one or two weeks worth of dev to report?
(10:34:08 AM) epmclellan: two weeks, I think
(10:34:25 AM) ARTi: two weeks ehh..  already cleared that cache..
(10:34:56 AM) ARTi: I fixed the archivematica-mcp-server package to install its databasses, it looks good.   the transcoder package seems to be finish, but looking at the logs I think it can use some touch up.. maybe berwin22 and I can hack on it next week
(10:34:57 AM) berwin22: I made some backend changes to the database, that should help sevien distinguish SIP and Transfer JOBS
(10:35:19 AM) peterVG: great
(10:35:37 AM) berwin22: yeah, there is a lot of debugging info printed
(10:36:54 AM) berwin22: I've done a LOT of bug fixing, but haven't been keeping great track of issue numbers unfortunatly. 
(10:36:58 AM) ARTi: upstart is now successfully starting/stopping/restarting, with gearman.
(10:37:04 AM) berwin22: added a no normalization path
(10:37:07 AM) peterVG: excellent
(10:37:12 AM) berwin22: added access normalization only
(10:37:12 AM) peterVG: x2
(10:38:04 AM) epmclellan: I'm almost finished figuring out the METS profile
(10:38:12 AM) epmclellan: which will be a big chunk of work for Joseph
(10:38:15 AM) berwin22: Austin getting the daemons working was a great help for testing
(10:38:37 AM) peterVG: epmclellan: great progress on that
(10:38:48 AM) epmclellan: thx, it's really interesting
(10:38:57 AM) peterVG: pls try to fire through validator before finalizing for joseph and jesus
(10:39:04 AM) epmclellan: yup
(10:39:11 AM) peterVG: if not premis-in-mets validator we should do basic xml validation
(10:39:29 AM) epmclellan: I always do basic xml validation
(10:39:36 AM) peterVG: of course you do :-)
(10:39:39 AM) epmclellan: otherwise the xml doesn't render in a web browser
(10:39:52 AM) epmclellan: :)
(10:40:00 AM) peterVG: sorry i meant mets schema validation
(10:40:05 AM) berwin22: I've added a means to group "jobs" into "microservices" on the back end. I think this will need a bunch of tweaking when we see it operational
(10:40:12 AM) peterVG: i.e. it might be valid xml but not compliant with METS schema
(10:40:17 AM) epmclellan: ok
(10:40:37 AM) ARTi: also built some ica-atom packages to ease with archivematica builds
(10:40:39 AM) peterVG: we can use a xml editor tool for that if the online validator isnt an option
(10:40:46 AM) berwin22: I think I fixed the issue where the MCP would disconnect from the database, if it was left alone
(10:40:49 AM) ARTi:  archivematica sans xubuntu modifications can install completely from packages now w/ out any post installation scripts.  good for .8
(10:40:51 AM) peterVG: ARTi: yes, that's great
(10:41:01 AM) peterVG: it also means we can advertise an additional ica-atom install option
(10:41:05 AM) peterVG: when we release 1.2
(10:41:21 AM) peterVG: ARTi: that's great, nice work
(10:41:35 AM) Sevein: ARTi: thank you for the ica-atom packages by the way! that could be into qubit notes actually!
(10:41:37 AM) ARTi: yah, maybe I can work with djjuhasz Sevein on the database installation
(10:41:50 AM) Sevein: sure
(10:41:54 AM) ARTi: http://people.debian.org/~seanius/policy/dbconfig-common-using.html/  
(10:42:07 AM) peterVG: cool, we should start testing the packages after code freeze on fri
(10:43:38 AM) djjuhasz: ARTi: "good for .8"?
(10:44:08 AM) djjuhasz: ARTi: yes, let's get that package working!
(10:44:13 AM) ARTi: archivematica release
(10:44:21 AM) ARTi: djjuhasz: sounds good
(10:44:28 AM) Sevein: ARTi: does these package work in 10.04 or just 11.04?
(10:44:32 AM) ARTi: its basicly just coming up with some good dbconfig magic
(10:44:49 AM) ARTi: Sevein: its very simple, i think it should working with any ubuntu/debian
(10:45:03 AM) ARTi: it should work*
(10:45:14 AM) djjuhasz: ARTi: dbconfig magic sounds like your bailiwick 
(10:45:18 AM) djjuhasz: :)
(10:45:21 AM) Sevein: ARTi: great, that worth a wiki page or an Installation page update!
(10:45:35 AM) djjuhasz: Sevein: agreed :)
(10:46:08 AM) berwin22: I don't see an archivematica feature freeze in the calendar
(10:46:46 AM) peterVG: hmm, i may have erased that by mistake yesterday
(10:46:58 AM) epmclellan: excellent, no more deadline!
(10:47:03 AM) ARTi: weeee
(10:47:14 AM) Sevein: :D
(10:47:15 AM) djjuhasz: we're free, we're freee!
(10:47:34 AM) berwin22: there is still a code freeze and a release
(10:47:37 AM) peterVG: there was a feature freeze meeting on oct 15 but i thought that was based on a timeline established in spring
(10:48:05 AM) epmclellan: I think we need to have a meeting about implementing METS
(10:48:28 AM) epmclellan: to make sure it all makes sense
(10:48:38 AM) peterVG: yes, but when :-(
(10:48:43 AM) epmclellan: hmm
(10:48:49 AM) epmclellan: this week?
(10:48:57 AM) peterVG: archivesspace rft due fri 2pm, expect to have short night before
(10:49:08 AM) epmclellan: ok
(10:49:11 AM) peterVG: won't be up for meeting post 2pm on fri
(10:49:18 AM) epmclellan: well, maybe Joseph and I can have a meeting
(10:49:26 AM) djjuhasz: Looks like Sevein is implementing METS from the SWORD end too?
(10:49:30 AM) peterVG: monday will be qubit 1.3 dev discussion while MJ is in town
(10:49:31 AM) epmclellan: yes
(10:49:54 AM) peterVG: maybe next tue?
(10:50:03 AM) epmclellan: that would be fine with me
(10:50:11 AM) peterVG: ....but will be working on Access presentation and gettting ready for user meeting at 3pm
(10:50:21 AM) peterVG: ...:-(
(10:50:33 AM) epmclellan: oh, well maybe berwin22 and I can meet and see if we can get everyhting finalized
(10:50:46 AM) peterVG: yes, definitely. please don't let me stop you
(10:51:10 AM) peterVG: i can always do a QA on our METS structure after next week
(10:51:14 AM) epmclellan: sure
(10:51:31 AM) peterVG: okay, cool thanks
(10:51:57 AM) peterVG: main issue will be writing and parsing the METS XML and processing its contents
(10:52:30 AM) epmclellan: yes, and there are different requirements for different types of transfers
(10:52:32 AM) peterVG: some tweaks to which elements are used where as result of QA shouldn't be too disruptive, hopefully just some XPATH updates if any at all
(10:52:43 AM) epmclellan: same basic structure, but if it's from DSpace there are additional requirements, etc
(10:53:36 AM) peterVG: epmclellan: METS used by Archivematica AIP/DIP should be consistent, regardless of source, right?
(10:53:47 AM) epmclellan: yes, it is consistent
(10:53:57 AM) peterVG: "if it's from DSpace there are additional requirement"?
(10:53:58 AM) epmclellan: but there are more elements for more complex AIPs
(10:54:19 AM) epmclellan: it's on the wiki
(10:54:25 AM) peterVG: :-)
(10:54:31 AM) epmclellan: same generic structure underlying the AIP
(10:54:42 AM) epmclellan: but additional filegroups etc
(10:54:53 AM) peterVG: k, makes sense
(10:54:54 AM) epmclellan: don't worry :)
(10:55:19 AM) JessicaB left the room (quit: Read error: Connection reset by peer).
(10:55:22 AM) JessicaB [~Jessica@70.71.0.109] entered the room.
(10:55:33 AM) peterVG: not worried :-)
(10:56:16 AM) epmclellan: I've studied METS for over a week now, very confident ;)
(10:56:30 AM) peterVG: awesome
(10:56:50 AM) peterVG: so i guess we should double check our freeze deadlines?
(10:57:14 AM) epmclellan: code freeze is Nov 7
(10:57:26 AM) peterVG: release is Nov 21
(10:57:28 AM) epmclellan: but that may be flexible?
(10:57:47 AM) peterVG: no, we need to make nov 21 release
(10:57:50 AM) epmclellan: ok
(10:57:54 AM) peterVG: to complete 2011 installs
(10:57:58 AM) epmclellan: right
(10:58:17 AM) peterVG: i'd say oct 21 is good feature freeze deadline
(10:58:30 AM) peterVG: we want to scalability test trunk at CVA starting week of oct 24
(10:58:42 AM) epmclellan: ok
(10:58:52 AM) peterVG: that leaves two weeks for that testing with bug fixes, adjustments etc
(10:59:02 AM) peterVG: until nov 7 code freeze
(10:59:12 AM) epmclellan: seems reasonable
(10:59:18 AM) peterVG: so another week and a half
(10:59:26 AM) epmclellan: berwin22: any thoughts?
(10:59:28 AM) peterVG: yikes!
(10:59:36 AM) epmclellan: yeah
(10:59:37 AM) berwin22: that's tight
(10:59:42 AM) berwin22: not sure trim will make it
(10:59:50 AM) peterVG: fair enough
(10:59:52 AM) epmclellan: agree
(10:59:53 AM) berwin22: what's the word on that stuff anyways?
(10:59:59 AM) epmclellan: there's a lot of dashboard stuff
(11:00:08 AM) epmclellan: berwin22: still waiting to hear
(11:00:44 AM) berwin22: last release I helped sevien with a number of queries, not sure what my schedule will look like on this release.
(11:01:22 AM) berwin22: it's going to be tight, but the deadline needs to be met
(11:01:32 AM) berwin22: we'll need to be strict about the freezes
(11:01:42 AM) peterVG: k, well we HAVE to get Sevein starting on dashboard dev on Monday
(11:02:40 AM) peterVG: i'll find out more about TRIM today
(11:02:45 AM) berwin22: there is some stuff that may be important I haven't had a chance to cleanup
(11:02:50 AM) berwin22: special characters
(11:03:03 AM) epmclellan: that's post feature-freeze?
(11:03:09 AM) berwin22: special characters = sip fails, it's a different back end
(11:03:39 AM) berwin22: so the old escape sequences have some "flaws"
(11:04:08 AM) berwin22: I don't have a good measure of how important those are, and how brittle they are
(11:04:21 AM) peterVG: let's figure out a testing plan for that
(11:04:21 AM) epmclellan: we're getting close to the end of the meeting, any deployment, testing, docs news?
(11:04:37 AM) peterVG: let's schedule an in office feature freeze/release schedule meeting on Tue Oct 25
(11:04:59 AM) epmclellan: ok
(11:05:00 AM) peterVG: i'll finalizing client committments by then so we can see how much wiggle room into Dec that gives us
(11:05:11 AM) peterVG: highest priority now is Dashboard dev
(11:05:42 AM) epmclellan: METS plus dashboard, without the METS we're not dealing with DSpace imports properly
(11:05:44 AM) peterVG: is googlecode issues list a good reflection of roadmap priorities?
(11:05:49 AM) peterVG: epmclellan: got it
(11:06:00 AM) peterVG: i.e. sort by priority
(11:06:18 AM) peterVG: ARTi: can you make Oct 25 in office?
(11:06:21 AM) epmclellan: dashboard, rights, METS
(11:06:58 AM) ARTi: peterVG: yeah, that should work
(11:07:07 AM) peterVG: actually, there's no way around it. we need to move this meeting up to next tue, before the Archivematica user group meeting
(11:07:36 AM) peterVG: we need clear path for 0.8 by time of that meeting
(11:07:47 AM) epmclellan: agreed
(11:07:50 AM) peterVG: user meeting is at 3pm
(11:08:12 AM) peterVG: so ARTi sorry, forget about oct 25
(11:08:26 AM) peterVG: what's the earliest you can be in office for tue oct 18?
(11:08:34 AM) peterVG: 10am too early?
(11:08:40 AM) peterVG: 11am?
(11:08:49 AM) ARTi: 11am is best
(11:09:10 AM) peterVG: actually, that will give us a chance to discuss any ES integration for 0.8 with MJ too
(11:09:14 AM) peterVG: okay, 11am
(11:09:30 AM) peterVG: ARTi: you're going to make hackfest the next day too right?
(11:09:45 AM) ARTi: yeah
(11:09:48 AM) peterVG: great
(11:09:58 AM) ARTi: oh, yeah I was planning on both days next week..
(11:10:00 AM) peterVG: okay so feature freeze meeting on Tue oct 18, 11am
(11:10:04 AM) peterVG: ARTi: great
(11:10:13 AM) peterVG: feature freeze : Mon Oct 24
(11:10:16 AM) ARTi: so scratch oct 25th?
(11:10:21 AM) peterVG: yes
(11:10:23 AM) peterVG: but stay tuned
(11:10:23 AM) ARTi: cool
(11:10:38 AM) peterVG: we'll want to go out to CVA that week to upgrade to trunk and do scalability testing
(11:10:53 AM) peterVG: we can discuss steps, schedule for that at oct 18 meeting
(11:10:56 AM) peterVG: is cool?
(11:11:30 AM) peterVG: i.e. which day(s) we'll go out to CVA during week of Oct 24
(11:12:12 AM) peterVG: so let us know asap if you have any days that won't work for that week
(11:12:47 AM) ARTi: Im pretty much open all month.. planning on some halloween parties though :]
(11:12:52 AM) peterVG: okay great
(11:13:10 AM) peterVG: epmclellan: you'll tackle the googlecode issue list for review at oct18 meeting?
(11:14:15 AM) epmclellan: sure
(11:14:19 AM) peterVG: thx
(11:14:36 AM) peterVG: okay sorry about going over but some critical scheduling to deal with there
(11:14:44 AM) epmclellan: np
(11:14:47 AM) berwin22: np