CMM's Chemistry X-ray Facility is on Mirage

As of a couple of days ago, the "B2" and "B5" instruments in the CMM's Lab in Chemistry are now connected to Mirage.

So now all 4 CMM Labs are "on Mirage" to some extent.

Technical stuff

The Chem Lab instruments don't save their data directly to a Samba server (like the other Labs).  Instead, they save their data locally, and the data is "pulled" by a periodic rsync from CMMData.  (This used to happen, once a day, but now it happens every 20 minutes.)

This presented a couple of "interesting challenges":

• The data grabber software needed significant changes to cope with files delivered by rsync.  The files get touched repeatedly by the rsync, so we need to take special steps to prevent them being ingested multiple times.
• The B5 machine is one of a few machines that are operated by CMM staff exclusively doing runs for CMM users. So we needed to implement a special mechanism for automatically assigning files to the right CMM user.

Data Migration in production on Mirage

The long awaited MyTardis "datafile migration" extensions are now deployed to CMM's Mirage production system.  The data is being mirrored as I write this.

What does this mean?  The short answer is that I can now add the rest of CMM's instruments to Mirage without filling our local file servers with a few months of data.  So expect to see the remaining AIBN and Hawken instruments added to Mirage in the next few days and weeks.

---

For people interested in developing / deploying MyTardis, there is a "pull request" waiting for merging into mytardis:master on Github.  Read the pull request comments for details on what is involved.

Update: The pull request has been merged, so the "migration" code is not in the "master" MyTardis codebase.

The Mirage project web pages

If you are interested in how Mirage is implemented, and/or are considering implementing your own data management system, take a look at the Mirage Project web pages.  They include an architectural overview, a description of the components we developed, and a section on how we interface with ACLS.

About this blog

Hi Folks,

This blog is about the Data Management Systems that we are building and running at the Centre for Microscopy and Microanalysis (CMM) at the University of Queensland (UQ).

CMM is the UQ's main microscopy centre, running a number of Electron Microscopes, X-ray Diffraction instruments and Mass Spectrometers in 5 separate laboratories on the St Lucia campus. There are in the region of 20 major instruments, and we provide a service to over 400 regular clients (UQ staff and students, staff and students of other universities, and commercial clients).  UQ CMM is a foundation node of the Australian Microscopy & Microanalysis Research Facility (AMMRF).

All of CMM's instruments produce results in digital form; 2D and 3D images, and various other kinds of characterization data, often in proprietary formats.   The problems we (the data wranglers) are trying to solve are:

• Getting the data off the instruments and into a place where the clients can access it.
• Providing long term storage for the data.
• Allowing clients to view, organize, search and process their data collections.
• Allowing clients to share data with other clients, and "publish" it in various ways for other people to fund and use.
• Do all of the above in a secure and sustainable way.

In this blog, I'm going to talk about where we are, where we are going, and how we implement things.  I'm going to try to cover things in ways that are relevant to both the users of the CMM facilities, and to other people who are building their own data management systems.

Stay tuned ...