Meeting 20111116

Development

• Jesús has been working on DIP upload issue discovered at CVA (can't upload large numbers of large files)
  • 2gb is the apache/current limit so we're maybe going to have to bypass Apache http upload
  • Peter, Jesús and David will discuss the issue in a separate meeting today

Documentation

(10:39:06 AM) epmclellan: I'll take minutes
(10:39:26 AM) Sevein: I have spent two days working on a solution for the uploads, unfortunately nothing clear yet
(10:39:39 AM) Sevein: every part of the puzzle I try to imagine is lot of work
(10:39:46 AM) ARTi: Ive just been working on qubit packages... feeling clogged/under the weather, gonna sit this one out
(10:39:48 AM) Sevein: sorry guys, I don't think we can to this for Friday
(10:40:10 AM) ARTi: ohh, Sevein, if I can help let me know
(10:40:22 AM) Sevein: I started a mail trying to get some ideas from the team but guess people is really busy now
(10:40:27 AM) Sevein: ARTi: ok, thank you
(10:40:53 AM) epmclellan: so no workarounds, Sevein?
(10:41:51 AM) Sevein: I don't think so, both sides need to talk each other, qubit needs an offline queue, etc
(10:41:58 AM) epmclellan: what's our current upper limit on a DIP upload?
(10:42:04 AM) epmclellan: eg number of files, size of files
(10:42:12 AM) epmclellan: total size
(10:42:17 AM) epmclellan: not sure how you're measuring
(10:42:44 AM) ARTi: 2gb is the apache/current limit
(10:43:11 AM) Sevein: oh nice, í didn't know that
(10:43:26 AM) Sevein: we are dealing with the same problem we do when trying to send an attachment to a friend by email of 8G
(10:43:31 AM) peterVG: so i think we're going to have to bypass Apache http upload?
(10:43:39 AM) Sevein: it is something crazy, you normally upload the file somehwere and then send it the reference
(10:43:46 AM) djjuhasz: Sevein: Have you looked into Gearman for the Qubit queue?
(10:43:48 AM) peterVG: we discussed SFTP as the preferred approach
(10:43:50 AM) Sevein: but normally you don't have to code your friend to get that file and put it into the database, etc hehehe
(10:44:01 AM) peterVG: ^ right
(10:44:14 AM) peterVG: Sevein: should we discuss further over lunch with david today?
(10:44:19 AM) Sevein: yes
(10:44:25 AM) Sevein: djjuhasz: not really
(10:44:30 AM) djjuhasz: I think the main advantages are that we're already using it for Archivematica and it seems more robust/mature than beanstalkd
(10:44:47 AM) peterVG: it looks like we'll need a short term 1.2.1 solution and then start to make some decisions on how to implement this fully for 1.3?
(10:44:56 AM) djjuhasz: Sevein: I'll have a look at your piratepad today, and see if I can offer any feedback
(10:45:34 AM) Sevein: the problem is that I guess we need an extra table in our qubit model to keep some kind of new object status, right?
(10:45:58 AM) Sevein: an object is created when someone deposits
(10:46:06 AM) Sevein: I say to Archivematica the temporary URI
(10:46:27 AM) Sevein: then qubit should be able to send the status of that object
(10:46:59 AM) djjuhasz: Sevein: depends on if we use a queue manager - I think Gearman and beanstalkd don't use a database for jobs
(10:47:19 AM) djjuhasz: hmm
(10:49:06 AM) peterVG:  ARTi is sick and Joseph is AWOL so we'll cancel the Archivematica dev mtg for this week. Had good discussion on status back on Monday
(10:50:56 AM) epmclellan: ok, that made the minutes easy