Meeting 20130807

Archivematica

Development

• Holly has been working on a bunch of thing to integrate the dashboard and storage service. #5427 Added 'browse' endpoint for Locations to storage service, for use when picking transfers. Added ability to configure storage service URL, both on install (#5259), and from the administration page.
• Holly added the ability to enable and disable pipelines (which affects all of their locations) and locations from the GUI #5403 #5260 There's also a section for configuration in the storage service now, with one setting of 'pipelines enabled on creation'.
• Holly started fixing a bug that meant AIP storage locations added through the storage service didn't show up in the microservice dropdown. #5440 In progress.
• Mike C got AIP deletion/rejection working (issue #5249).
• Mike C worked on integrating the dashboard with the storage server (issue #5427).
• Mike C added logic to the archival storage page to factor out AIP pending deletion from size and file count calculation (issue #5313).
• Misty and Justin S discussed some potential optimizations to FITSing, that might provide significant time improvements

Chat log

[10:34am] epmclellan: Archivematica meeting time?
[10:34am] JustinS: archivematica dev news: main story is storage service
[10:34am] djjuhasz: yes Archivematica!
[10:35am] peterVG: go go storage service!
[10:35am] epmclellan: JustinS: and how does the story go?
[10:35am] epmclellan: (can someone pls take notes?)
[10:35am] JustinS: Storage service features are ready to be shown to sfu, but sfu has not looked at it yet.  Justin has deployed storage service to a server for them to use, but
[10:35am] mcantelon: It's a story of hope and wonder.
[10:35am] mcantelon: I'll take notes.
[10:35am] JustinS: they want to install locally, so Justin made an installer
[10:35am] epmclellan: mcantelon: it always is
[10:35am] epmclellan: thanks mcantelon
[10:35am] JustinS: there is a conflict with the dashboard that I am resolving this morning
[10:36am] Holly|gone_home: Holly has been working on a bunch of thing to integrate the dashboard and storage service.  #5427 Added 'browse' endpoint for Locations to storage service, for use when picking transfers.  Added ability to configure storage service URL, both on install (#5259), and from the administration page.
[10:36am] qubot: Links: https://projects.artefactual.com/issues/5427 | https://projects.artefactual.com/issues/5259
[10:36am] epmclellan: JustinS: any chance we'll be ready to bill by the end of the week?
[10:36am] JustinS: so today hopefully is the day that Mark can start looking
[10:36am] Holly|gone_home is now known as Holly.
[10:36am] Holly: Holly added the ability to enable and disable pipelines (which affects all of their locations) and locations from the GUI #5403 #5260  There's also a section for configuration in the storage service now, with one setting of 'pipelines enabled on creation'.
[10:36am] qubot: Links: https://projects.artefactual.com/issues/5403 | https://projects.artefactual.com/issues/5260
[10:36am] JustinS: epmclellan I hope so/think so, depends on marks feedback  I suppose
[10:36am] epmclellan: wow, Holly
[10:37am] Holly: Holly started fixing a bug that meant AIP storage locations added through the storage service didn't show up in the microservice dropdown. #5440  In progress.
[10:37am] qubot: Links: https://projects.artefactual.com/issues/5440
[10:37am] epmclellan: if there are minor issues we can bill, if major I guess we would wait
[10:37am] Holly: I can agree with the 'lots of storage service'
[10:37am] mcantelon: Mike C got AIP deletion/rejection working (issue #5249).
[10:37am] qubot: Links: https://projects.artefactual.com/issues/5249
[10:37am] Holly: mcantelon: Exciting!
[10:38am] epmclellan: super
[10:38am] peterVG: epmclellan: JustinS we need to sfu (mark) to confirm that the features are done. i.e. there are no feature gaps. fixes to completed features can then still follow as small fixes
[10:38am] epmclellan: I should go away more often
[10:38am] epmclellan: peterVG: so if there are small fixes we can bill?
[10:38am] courtney: mcantelon: so have you and Holly made the storage service and AIP deletion in the dashboard work together?
[10:38am] JustinS: peterVG agreed, installer was the main feature gap
[10:39am] peterVG: epmclellan: we just need mark to say "yup, you guys made a storage service"
[10:39am] epmclellan: ok
[10:39am] epmclellan: great
[10:39am] JustinS: CIFS support is still missing, but I do not think SFU Library is going to care
[10:39am] peterVG: JustinS: thats fine. dont even mention it
[10:39am] peterVG: they're all Linux
[10:40am] JustinS: peterVG yes, I was not planning on pointing out any missing features
[10:40am] epmclellan: heh
[10:40am] peterVG:
[10:40am] mcantelon: courtney: Yes.
[10:40am] djjuhasz: It's a feature!  CIFS sucks, don't go
[10:41am] JustinS: I think that is it for dev news.  deployment news next?
[10:41am] peterVG: mcantelon: Holly courtney JustinS that's super! go go storage service
[10:41am] epmclellan: JustinS: status of FCD DIP upload at RAC?
[10:41am] mcantelon: Mike C added logic to the archival storage page to factor out AIP pending deletion from size and file count calculation (issue #5313).
[10:41am] qubot: Links: https://projects.artefactual.com/issues/5313
[10:41am] JustinS: rockefeller has given the green light to run the fcd dip upload, it is scheduled for 2pm pacific time tomorrow
[10:41am] epmclellan: ooh!
[10:41am] mcantelon: Mike C worked on integrating the dashboard with the storage server (issue #5427).
[10:41am] qubot: Links: https://projects.artefactual.com/issues/5427
[10:42am] peterVG: JustinS: any upgrades?
[10:42am] peterVG: any hardware upgrades?
[10:42am] peterVG: at RAC?
[10:42am] JustinS: not since they put in more ram 6 or 8 weeks ago
[10:42am] JustinS: for the fcd dip upload, new hardware is not required
[10:42am] peterVG: okay, how big is that DIP?
[10:42am] JustinS: it is 100 dips
[10:43am] peterVG: right, cause DIPs already created
[10:43am] JustinS: yes
[10:43am] JustinS: we have run the script with all but 2 dips in dev and it worked
[10:43am] ARTi: have been/am helping henery @ UofA with some dev deployment issues
[10:43am] peterVG: JustinS: so this will then finally complete their production ingest to access workflow!
[10:43am] peterVG: for their big collection
[10:43am] epmclellan: very exciting!!
[10:43am] epmclellan: ARTi: great, do they still have problems?
[10:44am] JustinS: once fcd dip upload is complete, then we will deploy the gui mcantelon built
[10:44am] peterVG: JustinS: cool
[10:44am] JustinS: then we can try to run maildir on thier production machine
[10:44am] JustinS: then we are done at rac
[10:44am] peterVG: JustinS: sweet!
[10:44am] epmclellan: oh my GOD
[10:44am] ARTi: epmclellan: yes, currently with the ica-atom install something happened with their check out w/ dev-installer
[10:44am] epmclellan: done!!
[10:44am] ARTi: I have to follow up with henry
[10:44am] epmclellan: thanks ARTi
[10:44am] JustinS: ARTi you started scalability tests at yale this week?
[10:44am] peterVG: JustinS: ARTi is UofA replicating our git repo installs?
[10:44am] ARTi: been getting up to speed w/ the scalability tests at yale, and have run the first one
[10:45am] epmclellan: excellent
[10:45am] courtney: ARTi and JustinS -- and he structured it with the md in the md folder and it worked
[10:45am] ARTi: peterVG: the first single went through fine
[10:45am] epmclellan: actually, I'll have to finish RAC documentation, another two days on that should get it done
[10:45am] JustinS: peterVG on the one vm they gave us access to , it is using the git repo installs, I am not sure about the one ARTi is helping Henry with
[10:45am] ARTi: running the first one with 5 different sets of objects/media infos now
[10:45am] JustinS: separate vm
[10:46am] JustinS: UBC deployment to educloud is still waiting on UBC IT, once that it done they will need some help deploying to the new vms
[10:46am] peterVG: ARTi: JustinS just wondering how we can reign in support for custom UofA vms that they keep toying with
[10:46am] JustinS: there are 2 that I am aware of, one that Umar is using (that is the one ARTi is helping with) one is for scalibliity
[10:46am] ARTi: peterVG: we had ssh access before, but I think some things got changed.    We might need to follow up with their lead tech again
[10:46am] JustinS: once those two are functional, there should be no more support
[10:47am] epmclellan: we weren't expecting them to try AtOM
[10:47am] JustinS: Umar's vm is to test workflow stuff I believe, and they decided to install atom
[10:47am] epmclellan: so that has added some work
[10:47am] peterVG: okay, epmclellan maybe we should be clear on that. if they want to switch to a maintenance agreement in Nov then we would also want more control of the VM to make sure we're on top of any changes
[10:47am] epmclellan: ok
[10:47am] epmclellan: understood
[10:47am] peterVG: epmclellan: JustinS okay, we'll thats fine
[10:48am] peterVG: happy that he's trying it
[10:48am] peterVG: i was just trying to make point about how we manage client-installed configs
[10:48am] epmclellan: for sure
[10:48am] peterVG: reduce workload
[10:48am] epmclellan: yeah, I have some ideas for Pasadena and Harvard
[10:48am] peterVG: if they replicate our git repo configs then at least we can track and troubleshoot changes in their config
[10:48am] epmclellan: to reduce after-install support
[10:49am] JustinS: re Yale scalability tests - Misty and I discussed some potential optimizations to FITSing, that might provide significant time improvements, I'd like to figure out if/when Misty can spend more time on that this month, with ARTi and me
[10:49am] epmclellan: will talk to JustinS and ARTi about it
[10:49am] djjuhasz: courtney is advocating that we dump the Archivematica VM for 1.0, due to all the installation support we are doing, and have a demo site instead
[10:49am] epmclellan: that would be a huge improvement
[10:49am] ARTi: JustinS:  awesome
[10:49am] JustinS: deployment news at Pasadena: we had a conference call with them last week
[10:49am] epmclellan: djjuhasz: saw that and think it's a good idea
[10:49am] epmclellan: JustinS: thanks for doing that
[10:49am] courtney: to go forward, we need a clear release plan... djjuhasz, peterVG and JustinS should sit down and talk about it this week
[10:49am] djjuhasz: I think it sounds like a good plan, but there are some technical details to figure out
[10:50am] JustinS: they want to use RedHat, their IT is unsure if they will let them use ubuntu, waiting for them to get back
[10:50am] courtney: advocating/begging/pleading.. same diff
[10:50am] djjuhasz: Agreed @ having a clear release plan
[10:50am] epmclellan: I'd like to be involved too if possible
[10:50am] JustinS: djjuhasz implementation details, there are always a few implementation details
[10:50am] courtney: djjuhasz: and soon so we can plan for dev needs
[10:50am] peterVG: epmclellan: lets book the meeting for next wed. today is full
[10:51am] courtney: epmclellan: for reference, i sent an email to the archivematica list... it's probably somewhere buried in your backlog
[10:51am] epmclellan: sure
[10:51am] ARTi: @ online demo,  I think we can use the digital ocean API to script destroying and recreating a demo
[10:51am] epmclellan: courtney: I try not to read the archivematica list
[10:51am] ARTi: could probably get a couple demos online there for ~$10 a month
[10:51am] JustinS: ARTi there is an ansible module for DO, which we can use to make that scripting pretty easy
[10:51am] ARTi: awesome
[10:52am] djjuhasz: Okay, I'll figure out a time for the Archivematica demo site + VM discussion and add to calendar
[10:52am] peterVG: ARTi: that sounds good
[10:52am] peterVG: re: digital ocean demos
[10:52am] peterVG: and cost
[10:53am] courtney: deployment: JustinS brought up some good points about avoiding MoMA scope creep... another email you should try to locate epmclellan
[10:53am] epmclellan: ok
[10:53am] epmclellan: I have 10 to the power of 15 emails tho
[10:53am] courtney: peterVG: and there is more to it, too... JustinS mentioned the need for an uninstall script...it's all laid out in my email to the archivematica internal list
[10:53am] epmclellan: so may take a while
[10:53am] JustinS: Ben wanted to add a metadata entry template for setting up controlled vocabularies, to archivematica, as part of DRMC-MA
[10:53am] courtney: peterVG: do you need me to forward it
[10:54am] epmclellan: JustinS: a simple metadata entry template may not be much scope creep
[10:54am] epmclellan: hmm
[10:54am] epmclellan: to Archivematica?
[10:54am] epmclellan: sorry, thought it was Atom
[10:55am] courtney: epmclellan: it's not a new template, it's setting preconfigured choices and defaults
[10:55am] epmclellan: ok, yeah, that's scope creep
[10:55am] courtney: epmclellan: from what i understand
[10:55am] JustinS: ya, that is the thing, it is an AtoM type feature, for cataloguing metadata, it should probably be discussed in DRMC-MA meeting this afternoon
[10:55am] courtney: epmclellan: but still creeeepy
[10:55am] epmclellan: ok, we'll review
[10:55am] epmclellan: yeah, we gotta keep the metadata enhancement in atom
[10:55am] epmclellan: as much as possible
[10:56am] courtney: since it's approaching end of the meeting.... can i offer a fun thing?
[10:56am] epmclellan: please do!
[10:56am] mcantelon: Fun thing!
[10:56am] courtney: Spruce, OPF and CurateCamp are holding a hackathon at iPres and asking for suggestions... http://wiki.opf-labs.org/display/SPR/iPREShack+-+SPRUCE,+OPF+and+CURATEcamp+hackathon+at+iPRES2013
[10:57am] Misty: Cool!
[10:57am] courtney: I think our team might have some fun ideas for people to hack that could also 1-benefit us and 2- get people downloading and playing with archivematica
[10:57am] JustinS: courtney fix FITS
[10:57am] courtney: You can make suggestions to them by scrolling down on the link I sent -- link at the bottom
[10:57am] courtney: JustinS: don't tell me, tell them! : ) and be specific about what "fix" means
[10:58am] courtney: Anywho... please please contribute ideas... I'll be there to answer questions from the dev teams if they have any
[10:59am] JustinS: thanks for the link courtney
[10:59am] epmclellan: courtney: nice job linking PREMIS in METS to the Simpsons
[10:59am] epmclellan: reviewing now
[11:00am] courtney: epmclellan: yes, peterVG will be happy to know that we are neither a pipeline or a sausage maker, we are donut makers
[11:00am] epmclellan: mmmm....
[11:00am] epmclellan: they taste better than pipelines
[11:00am] epmclellan: ok, reviewing now
[11:00am] kerry is now known as kerryBANK.
[11:00am] courtney: meeting adjourned