Recent changes to the Mirage User Interface

There have been a number of changes to the Mirage user interface in the past few days.

Source metadata

Mirage now displays "administrative" metadata about the data capture process; e.g. who was logged in, what instrument was used, and the "save" file and folder names for each captured Datafile.

Downloaded ZIP / TAR files

When you download an Experiment or selected Datasets or Datafiles the TAR / ZIP archive now has file and folder names that match the names used to save the files on the instrument.   If you are downloading an entire Experiment, you can still get the "classic" MyTardis organization.

Update - 2013-05-06:  Mac users are no longer offered the option of downloading in ZIP format because of Mac's problems with reading the "streamed" ZIP file format that MyTardis uses.  You can use TAR format instead.

Sharing data with other Mirage users

It is now possible to share Experiments with other Mirage users, or groups using the "sharing" tab in the Experiment view.  You can grant read-only access, read-write access, or full ownership rights.

If you wish to create a Mirage group for sharing data within your research group, please contact me.

Credits

The "data sharing" changes and other ones that you may notice are the work of Steve Androulakis and Grischa Meyer at Monash Uni.  For the MyTardis audience, this the result of the "master-3.0 merge".  Kudos to Steve A!

Data Migration in production on Mirage

The long awaited MyTardis "datafile migration" extensions are now deployed to CMM's Mirage production system.  The data is being mirrored as I write this.

What does this mean?  The short answer is that I can now add the rest of CMM's instruments to Mirage without filling our local file servers with a few months of data.  So expect to see the remaining AIBN and Hawken instruments added to Mirage in the next few days and weeks.

---

For people interested in developing / deploying MyTardis, there is a "pull request" waiting for merging into mytardis:master on Github.  Read the pull request comments for details on what is involved.

Update: The pull request has been merged, so the "migration" code is not in the "master" MyTardis codebase.

The Mirage project web pages

If you are interested in how Mirage is implemented, and/or are considering implementing your own data management system, take a look at the Mirage Project web pages.  They include an architectural overview, a description of the components we developed, and a section on how we interface with ACLS.

Progress on Mirage Data Migration

The Disk Space Issue

The main issue that is preventing us from rolling out Mirage for the majority of CMM's instruments is disc space.  Basically, if we did a "full" roll-out, we would fill up the available disc space in a small number of months, and then we would have to start deleting older files to make space That would mean that users would need to manage their own long-term file storage, and we would essentially be back where we started.

Then there is the new 3View system that is capable of generating 10Gb of data in a single night, or 20 Gb after rendering the images as tif files.

Managing Lots Of Data

We are addressing the issue of "too much data, not enough disc space" in two ways:

• For data that needs to be kept online, we are currently implementing a scheme where individual data files are "migrated" to secondary online locations within UQ.  As far as users are concerned, the data files will be accessed as before, except that access will be a just little bit slower.
• For data that no longer needs to be online, we will be implementing an archiving scheme in which snapshots of entire experiments complete with all relevant metadata are saved to offline storage.

Progress So Far

I am currently developing the code for the datafile migration subsystem for Mirage.  The basic file migration code is working in the Mirage test system, and the code that will decide what files to migrate and when to migrate them is in progress.  The initial migration system will take into account the size of the individual files, their file types, and when they were created and last accessed.  Later on, I intend to allow users to indicate the relative importance of files, datasets and experiments to influence the migration decision making.

(For MyTardis folks, the Mirage migration code is actually a MyTardis "app".  To use it, you will need to set up one or more secondary "destinations", which can simply be private WebDAV servers on some other machine with lots of disk space.  Look in my MyTardis repo on GitHub for the code.)

The other aspect that needs to be sorted out is actual disk space provisioning.  I have negotiated some space on the UQ HPC cluster for interim storage, but "the real thing" will be implemented on the QERN system that QCIF is currently developing.  We are currently "on the list" for transition onto QERN.

About this blog

Hi Folks,

This blog is about the Data Management Systems that we are building and running at the Centre for Microscopy and Microanalysis (CMM) at the University of Queensland (UQ).

CMM is the UQ's main microscopy centre, running a number of Electron Microscopes, X-ray Diffraction instruments and Mass Spectrometers in 5 separate laboratories on the St Lucia campus. There are in the region of 20 major instruments, and we provide a service to over 400 regular clients (UQ staff and students, staff and students of other universities, and commercial clients).  UQ CMM is a foundation node of the Australian Microscopy & Microanalysis Research Facility (AMMRF).

All of CMM's instruments produce results in digital form; 2D and 3D images, and various other kinds of characterization data, often in proprietary formats.   The problems we (the data wranglers) are trying to solve are:

• Getting the data off the instruments and into a place where the clients can access it.
• Providing long term storage for the data.
• Allowing clients to view, organize, search and process their data collections.
• Allowing clients to share data with other clients, and "publish" it in various ways for other people to fund and use.
• Do all of the above in a secure and sustainable way.

In this blog, I'm going to talk about where we are, where we are going, and how we implement things.  I'm going to try to cover things in ways that are relevant to both the users of the CMM facilities, and to other people who are building their own data management systems.

Stay tuned ...