Meeting 20120425

Development

• Joseph has been working on a polling replacement for watched directories, so we can use network or distributed file systems
• Joseph did a quick tweet to our dev-helper script, made stderr on a number of scripts appear in red, to find problems faster
• Joseph has been working on Issue 886, make overiding the default assigned threads by core count configurable.
  • Default is detect number of cores
• Mike C worked on the underpinnings needed for a file browser: added the ability to return directory structure data as JSON, adding the ability to configure source directories in the administration interface and using the file browser JavaScript widget I made earlier for the source directory selection.
• Joseph did a lot of work on the thumbnail generation. Put in some placeholders.
• Courtney has been working on issue 882: Management of persistent MCP metadata that does not end up in AIP.
• Joseph needs Austin's help on a few normalization problems in Ubuntu 12.04 LTS

Deployment

• CVA SIPs are freezing at the Verify checksums microservice, so Courtney's bringing Joseph some examples for testing next week

Testing

• Austin has been working on the scalability testing: http://archivematica.org/wiki/index.php?title=Scalability_testing#Test_results. He will be running more test data sets through with different parameters once he gets Joseph's changes into packages and into the images (tonight/tomorrow).
• Evelyn will ensure that all those test cases get completed and documented.
• Peter feels that compability between VM packaging and hosting environments can give us unknown/new issues at time of deploy so he'd like for a way to test for that. But that's seperate from this round of scalability testing, it's something to work on for later.

Documentation

• Courtney is uploading some new mockups/doc to wiki at http://archivematica.org/wiki/index.php?title=Transfer_Backup_Requirements#Administration_Tab_in_Dashboard and on the file browser interface

Chat log

(10:37:54 AM) epmclellan: I can take notes
(10:38:17 AM) epmclellan: dev news? berwin22, ARTi and mcantelon?
(10:38:49 AM) berwin22: been working on a polling replacement for watched directories, so we can use network or distributed file systems
(10:39:06 AM) ARTi: berwin22: awesome
(10:39:10 AM) peterVG: sweet
(10:40:02 AM) berwin22: did a quick tweet to our dev-helper script, made stderr on a number of scripts appear in red, to find problems faster
(10:40:12 AM) berwin22: tweak*
(10:40:21 AM) epmclellan: oh, I liked "tweet"
(10:40:51 AM) epmclellan: mcantelon: you've been doing some dashboard work, yes?
(10:40:56 AM) berwin22: Issue 886:	make overiding the default assigned threads by core count configurable.
(10:40:56 AM) berwin22: worked on that: default is detect number of cores
(10:41:18 AM) courtney: epmclellan peer pressured me to advise on Issue 882, so I done did
(10:41:19 AM) mcantelon: epmclellan: For sure...
(10:41:20 AM) mcantelon: Mike C worked on the underpinnings needed for a file browser: added the ability to return directory structure data as JSON, adding the ability to configure source directories in the administration interface and using the file browser JavaScript widget I made earlier for the source directory selection.
(10:41:46 AM) berwin22: I did a lot of work on the thumbnail generation. Put in some placeholders.
(10:42:47 AM) epmclellan: I haven't heard from Mark Jordan for a few days re CONTENTdm integration
(10:43:18 AM) epmclellan: I'm waiting on answers to a couple of questions re user-supplied METS structMaps
(10:43:57 AM) epmclellan: that's it for dev?
(10:44:09 AM) epmclellan: deployment?
(10:44:49 AM) courtney: CVA SIPs are freezing at the Verify checksums microservice, so I'm bringing berwin22 some examples for testing here next week
(10:45:14 AM) epmclellan: sounds bad, are you stalled out?
(10:45:19 AM) courtney: yes
(10:45:29 AM) ARTi: Ive been working on the scalability testing. berwin22 I told you the wrong package revisions the other day its actually on something much more recent.. does the output here look correct : http://archivematica.org/wiki/index.php?title=Scalability_testing#Test_results ?
(10:45:30 AM) epmclellan: sh*t, that's too bad
(10:45:37 AM) courtney: only processed one fully since last week
(10:45:46 AM) epmclellan: courtney's problem, not ARTi's great testing
(10:46:04 AM) berwin22: checking
(10:46:26 AM) ARTi: My tests were shut down a few times over the weekend because of billing issues, but once the test data looks good Ill do a test with 6 more machines
(10:46:47 AM) epmclellan: ARTi: great!
(10:46:49 AM) berwin22: courtney: let's try a good old fashioned reboot 
(10:46:59 AM) courtney: we have done that
(10:47:36 AM) epmclellan: why has it conked out now after so many successful transfers?
(10:47:51 AM) berwin22: ARTi: looks incomplete
(10:48:27 AM) berwin22: where's the output of these:  http://code.google.com/p/archivematica/source/browse/trunk#trunk%2Fsrc%2FtestingTools%2FdistributedTesting
(10:49:06 AM) ARTi: yeah, thats a more recent version.
(10:49:17 AM) ARTi: I can move that to my server and run
(10:49:38 AM) berwin22: you will also need the updated views in the mysql file
(10:49:47 AM) ARTi: ?
(10:49:56 AM) ARTi: it wasn't complete?
(10:50:03 AM) ARTi: oohh
(10:50:06 AM) ARTi: looks incomplete
(10:50:08 AM) ARTi: gotcha
(10:50:09 AM) ARTi: ok then
(10:51:12 AM) ARTi: Ive gotta update these images before testing continues.
(10:51:26 AM) peterVG: ETA for running the various test data sets through with the different parameters/combinations?
(10:52:48 AM) ARTi: as soon as I get josephs changes into packages and into the images
(10:53:28 AM) ARTi: tonight/tomorrow
(10:54:26 AM) peterVG: cool, evelyn you are on deck for ensuring all those test cases get completed & documented?
(10:54:32 AM) epmclellan: yes
(10:54:37 AM) peterVG: okay,cool
(10:54:44 AM) peterVG: ARTi: openhosting looks promising eh?
(10:55:17 AM) ARTi: yeah, lots of good features.  prices are not bad either
(10:55:59 AM) peterVG: ARTi: what virtualization platform are you using there. Should you be testing a Vsphere server there as well for closer compability to our client deploys (almost all VMware VMs)
(10:56:35 AM) ARTi: I dont think so
(10:56:47 AM) ARTi: I have not seen differences between this and working on vmware 
(10:56:47 AM) peterVG: ARTi: i think if we can get billed for those by the hour then we can make good use of them for a number of things
(10:57:26 AM) epmclellan: guys, for note-taking purposes, what are you talking about?
(10:57:36 AM) ARTi: virtualization platforms
(10:57:49 AM) epmclellan: to deploy to clients?
(10:58:07 AM) berwin22: what are we being billed by the hour for?
(10:58:08 AM) ARTi: peterVG: yeah its all billed by the hour, and cheap $0.0187 / GHz-hour  , $0.0229 / GB-hour
(10:58:11 AM) epmclellan: to deploy in client institutions, I mean?
(10:58:14 AM) peterVG: okay, we'll that's berwin22 put in the time to install it in ciaos, let's use it now if only to test for any differences in vm format (acknowledging that ciaus testing is more for software compatibility than performance/scalability)
(10:58:31 AM) ARTi: disk space is $0.0850 / GB-month
(10:58:53 AM) peterVG: sorry test the VMDK's within VMware VMs running inside Vsphere server console (on ciaus)
(10:59:31 AM) ARTi: peterVG, instead of testing on openhosting?
(10:59:38 AM) peterVG: as parallel tests
(10:59:39 AM) berwin22: peter: I dont' think the schedule allows time for me to do that.
(10:59:48 AM) peterVG: berwin22: no fair enough, just realized that as well
(11:00:12 AM) berwin22: and I need ARTi's help on a few normalization problems
(11:00:29 AM) peterVG: we took time to configure ciaus for vsphere so that we could have an environment that closely replicated client installs
(11:00:47 AM) ARTi: peterVG: yeah, but a single machine is not going to give us great scalability testing results
(11:00:51 AM) peterVG: still useful for QA or troubleshooting deploys
(11:01:02 AM) ARTi: our stuff at ohi might be on a single machine but I kinda doubt it
(11:01:29 AM) ARTi: peterVG: yeah great for troubleshooting,
(11:01:43 AM) ARTi: and making sure vmdk images work before sending them out ;)
(11:02:05 AM) peterVG: ARTi: i still feel that compability between VM packaging and hosting environments can give us unknown/new issues at time of deploy so I'd like for a way to test for that
(11:02:32 AM) peterVG: but that's seperate from this round of scalability testing, it's something to work on for later
(11:02:36 AM) ARTi: ok
(11:03:25 AM) epmclellan: Docs? courtney?
(11:05:08 AM) courtney: just uploading some new mockups/doc to wiki at  http://archivematica.org/wiki/index.php?title=Transfer_Backup_Requirements#Administration_Tab_in_Dashboard and on the file browser interface