Meeting 20111130

Development

• Joseph and Evelyn have been working on producing valid mets files
  • The files are validating in the PREMIS in METS toolkit
• Jesús has been working on the upload DIP scalability issue, the work affects both qubit and Archivematica
  • He's trying to make the information object target a choice, he'll be working on it today
  • And he's trying to do the job queue in Qubit more reliable, and some optimizations, but he didn't make much progress
  • He discovered yesterday that the qubit worker daemon does not free used memory, has to fix that
• Joseph has been working on a number of the criticals:
  • Issue 782 - unicode problems
  • Archivematica permission error on submission documentation
  • Sanitizing the names in submission documentation/SIP was loosing the file UUID.
  • Issue 762 - having a workflow for service copies
  • Issue 756 - one transfer to one sip workflows
  • An attempt to kill -9 the previous instance of an MCP if it's already running.
  • Issue 797 - Manual SIP creation fails
  • Issue 765 - added text group for dspace exports
  • Issue 710 - transfer submission documentation copied to sip submission documentation
• Critical work areas for next week or two: PREMIS/METS implementation, rights module, DIP upload scalability

Documentation

Evelyn has also been updating roadmap to schedule 2012 priorities: Development roadmap

Chat log

(10:34:38 AM) peterVG: moving on to Archivematica
(10:34:39 AM) epmclellan: hi
(10:34:43 AM) epmclellan: I can take minutes
(10:35:02 AM) epmclellan: we haven't had a real meeting for several weeks
(10:35:07 AM) epmclellan: lots of dev going on
(10:35:13 AM) berwin221: yeah
(10:35:20 AM) berwin221: thanks for taking notes
(10:35:35 AM) epmclellan: np
(10:35:50 AM) Sevein: dev news!
(10:35:53 AM) peterVG: bring it
(10:36:08 AM) epmclellan: mets mets mets mets
(10:36:34 AM) epmclellan: Joseph and I have been working closely on the mets implementation
(10:36:40 AM) berwin221: sorry, had a package delivered
(10:36:45 AM) epmclellan: no worries
(10:36:55 AM) epmclellan: our mets files are validating in the PREMIS in METS toolkit
(10:37:06 AM) peterVG: yay!
(10:37:08 AM) epmclellan: we've confirmed that it checks the mets as well as the premis
(10:37:12 AM) epmclellan: still have work to do
(10:37:13 AM) peterVG: we've been validated
(10:37:21 AM) peterVG: epmclellan: oh great, good news
(10:37:30 AM) peterVG: so its validating against the full METS schema?
(10:37:30 AM) epmclellan: yeah
(10:37:43 AM) peterVG: ...the tool is using the full METS schema
(10:37:45 AM) epmclellan: but some of our content still needs a little work
(10:37:53 AM) berwin221: shotgun for the week before last...
(10:37:53 AM) berwin221: I've been working on a number of the criticals: 
(10:37:53 AM) berwin221: 782 unicode problems
(10:37:53 AM) berwin221: archivematica permission error on submission documentation
(10:37:53 AM) berwin221: Sanitizing the names in submission documentation/SIP was loosing the file UUID.
(10:37:54 AM) berwin221: 762 Having a workflow for service copies
(10:37:54 AM) berwin221: 756 one transfer to one sip workflows
(10:37:56 AM) berwin221: An attempt to kill -9 the previous instance of an MCP if it's already running.
(10:37:56 AM) berwin221: 785 bagit size limit
(10:37:56 AM) epmclellan: yes, it's using the full mets schema, it points out mets errors if there are any
(10:37:58 AM) berwin221: 797 Manual SIP creation fails
(10:37:58 AM) berwin221: 765 added text group for dspace exports
(10:38:00 AM) berwin221: 710 transfer submission documentation copied to sip submission documentation
(10:38:16 AM) peterVG: epmclellan: that's great
(10:38:43 AM) epmclellan: standard transfer mets file is really close
(10:38:55 AM) epmclellan: DSpace transfer mets file is taking a little longer, it's quite complex
(10:39:45 AM) epmclellan: there are some PREMIS issues left too
(10:40:02 AM) epmclellan: but I think it's looking very good for a full and valid PREMIS/METS implementation by Dec 16
(10:40:08 AM) epmclellan: which is the day that CVA goes into production
(10:40:33 AM) peterVG: epmclellan: that's great, nice work
(10:40:53 AM) epmclellan: thanks, Joseph's been going full out
(10:41:12 AM) berwin221: I wanted to discuss this issues in the meeting http://code.google.com/p/archivematica/issues/detail?id=823 All submission documentation files need to be under the submission documentation directory.
(10:41:12 AM) berwin221: I'm thinking that that may not always be the desired case.
(10:41:14 AM) peterVG: yes, really coming through
berwin22 berwin221 
(10:41:28 AM) epmclellan: berwin221 yes?
(10:41:35 AM) epmclellan: what would be the preferred option?
(10:41:59 AM) berwin221: if we are ingesting from various systems, it may make some sense to maintain the original order in the AIP, unless the archivist wishes to change that.
(10:42:20 AM) epmclellan: we maintain the original order by capturing the transfer structure in the transfer mets file
(10:42:26 AM) epmclellan: that's the whole purpose of the transfer mets file
(10:42:49 AM) peterVG: seems like a better topic for discussion offline, purpose of meeting is to check-in on status of tasks
(10:42:49 AM) epmclellan: to allow the system or the archivist to then re-arrange the objects
(10:42:53 AM) epmclellan: ok
(10:43:11 AM) peterVG: but should be discussed, I can referee ;-)
(10:43:18 AM) epmclellan: sounds good :)
(10:43:32 AM) peterVG: berwin221 you coming in today?
(10:43:36 AM) Sevein: Jesús has been working in the upload DIP scalability issue, the work affects both ubit and Archiveamtica
(10:43:49 AM) berwin221: sorry probably not at this rate, just got my butt out of bed for the meeting
(10:43:59 AM) berwin221: rough sleep seems to be my issue these days
(10:44:04 AM) peterVG: likewise
(10:44:28 AM) peterVG: berwin221 we're expecting pcs and workstations to be delivered today right?
(10:44:35 AM) berwin221: yes
(10:44:38 AM) peterVG: sorry PC & monitors
(10:44:40 AM) peterVG: okay
(10:44:44 AM) berwin221: not monitors
(10:44:56 AM) peterVG: oh, will cheque your email 
(10:45:21 AM) epmclellan: Sevein: what's the status of DIP upload?
(10:45:33 AM) peterVG: *will check your email (for delivery info) and write you a reimburse *cheque*
(10:46:02 AM) berwin221: cheers. the monitors were scheduled for pickup at burnaby ncix
(10:46:23 AM) peterVG: oh okay
(10:46:28 AM) berwin221: I checked their stock and it said they had two, but when I ordered them, they said they were out, and needed to get more from the warehouse
(10:46:49 AM) Sevein: epmclellan: I'm trying to make the information object target a choice, I'll be working on it today
(10:47:01 AM) epmclellan: nice
(10:47:17 AM) Sevein: epmclellan: and trying to do the job queue in Qubit more reliable, and some optimizations, but I didn't make much progress :(
(10:47:30 AM) epmclellan: oh well
(10:48:22 AM) Sevein: I discovered yesterday that the qubit worker daemon does not free used memory, have to fix that
(10:48:52 AM) Sevein: It is not a problem when we use qubit as a web site (request/response model) but it can be when we have a process in the background
(10:49:02 AM) epmclellan: so after a while qubit runs out of memory and DIP upload quits?
(10:49:13 AM) Sevein: I have been playing with xdebug traces to find leaks
(10:49:30 AM) Sevein: yes, basically
(10:49:36 AM) peterVG: balls
(10:49:51 AM) epmclellan: yeah
(10:49:56 AM) Sevein: and the memory used grows for each upload done
(10:50:13 AM) epmclellan: oh dear
(10:50:14 AM) peterVG: Sevein: maybe an isolated task that mcantelon can help with debugging?
(10:50:32 AM) peterVG: after he assembles IKEA furniture :-)
(10:50:36 AM) berwin221: can you hack it - spawn a sub process and kill it, when finished?
(10:50:46 AM) mcantelon: mcantelon: Haha... yeah, I'm willing to poke around for sure.
(10:51:36 AM) Sevein: berwin221: I have never done that in PHP, but it would be a nice workaround
(10:52:06 AM) berwin221: I think it would depend where the memory leak was. 
(10:52:28 AM) peterVG: okay, lets revisit after meeting
(10:52:35 AM) Sevein: yes
(10:52:48 AM) epmclellan: deployment?
(10:52:59 AM) peterVG: one major archivematica decision this week is to postpone 0.8 to allow for ongoing CVA testing/deployment
(10:53:14 AM) epmclellan: yes
(10:53:26 AM) epmclellan: getting CVA into production is our deadline
(10:53:30 AM) peterVG: evelyn has also been updating roadmap to schedule 2012 priorities
(10:53:31 AM) peterVG: http://archivematica.org/wiki/index.php?title=Development_roadmap
(10:53:39 AM) epmclellan: yes, that's very draft though
(10:53:56 AM) peterVG: sure, and definitely open for more discussion with full dev team
(10:54:00 AM) epmclellan: yup
(10:54:17 AM) peterVG: but needed right now for some 2012 planning
(10:54:55 AM) berwin221: code freeze is the 5th of Dec?
(10:55:06 AM) peterVG: berwin221 we'll move that now
(10:55:09 AM) peterVG: somewhat fluid
(10:55:25 AM) peterVG: our deadline now is to have 0.8 packaged and ready for upgrade/install at RAC by mid-January
(10:55:46 AM) peterVG: gives us the flexibility we need in meanwhile to continue on with CVA scalability testing
(10:55:56 AM) peterVG: that said, we should update these dates accordingly
(10:56:07 AM) peterVG: evelyn and i can do so today
(10:56:29 AM) peterVG: but likely means code freeze should be Xmas leaving the 2-3 weeks in new year for final QA and packaging
(10:56:31 AM) epmclellan: I took the release date off the main page of the wiki ages ago
(10:57:01 AM) berwin221: k
(10:57:02 AM) epmclellan: since it's been so fluid
(10:57:36 AM) peterVG: so epmclellan what are dev priorities then for next week?
(10:57:47 AM) epmclellan: PREMIS and METS
(10:57:52 AM) epmclellan: Qubit upload
(10:58:26 AM) epmclellan: ...checking roadmap
(10:58:31 AM) peterVG: okay, and then rights metadata right?
(10:58:34 AM) epmclellan: Rights management
(10:58:40 AM) epmclellan: yes, that's critical
(10:58:40 AM) peterVG: or is that what you meant by PREMIS and METS?
(10:58:43 AM) Sevein: rights? this is where I think mcantelon could be very helpful now
(10:58:53 AM) Sevein: as an totally isolated django task
(10:58:58 AM) epmclellan: peterVG: there are other PREMIS and METS issues, they've all been filed
(10:59:04 AM) peterVG: k
(10:59:16 AM) epmclellan: but it's true that METS is not final until rights are done
(10:59:57 AM) epmclellan: I think those are the key priorities
(11:00:10 AM) epmclellan: that will get us into production at CVA
(11:00:13 AM) peterVG: Sevein: okay, plan was to have mcantelon shadow djjuhasz this week to get started with migration work but looks like that may slip to next week so maybe focus on pitching in with Django dev and debug tasks until then?
(11:00:28 AM) peterVG: can discuss during Skype chat
(11:00:36 AM) mcantelon: Sounds good to me.