{% extends "base.html" %} {% block title %}Help pages{% endblock %} {% load static %} {% block stylesheets %} {% endblock %} {% block body %}

Help Pages

Get in touch with the Support Team

Category:

Accessing my reduced data

The reduced data can be access in two ways. Either through an IDAaaS virtual machine or using the IDAaaS FileBrowser.

Access reduced files within IDAaaS Virtual Machines

  1. Visit the IDAaaS page https://isis.analysis.stfc.ac.uk
  2. Open an existing or create and open a new IDAaaS virtual machine / instance
  3. The reduced files are then available within the virtual machine within folders named "autoreduced" within RB folders; and at data locations as detailed on completed Reduction Job pages on the WebApp

Use the IDAaaS FileBrowser to copy reduced data elsewhere

  1. Visit the page https://data.analysis.stfc.ac.uk and login
  2. Click on "Data" on the left menu
  3. Reduced data are then available within RB folders that you have access to from HOME or by stepping through to the relevant RB folder from INSTRUMENT. Within these reduced file are within "autoreduced" sub-folders; and at data locations as detailed on completed Reduction Job pages on the WebApp

Reduction Variables Script

reduce_vars.py

It is possible to define some additional variables for scripts that can be used during reduction. They can be altered via the web interface. These should be defined in the //isis/NDX.../user/script/autoreduction/reduce_vars.py file. This file has the following format:

standard_vars = {
    'perform corrections': False,
    'reduction option': "Default value"
}

advanced_vars = {
    'crop range': [100, 10000]
}

variable_help = {
    'standard_vars': {'do_absorb_corrections' : "Do you want corrections on?",
                      'reduction option' : "message to explain option - this is displayed in the webapp"},
    'advanced_vars': {'crop range' : "advanced settings are handled in a separate section"}
}

Reduction Script

reduce.py

In order to reduce your data you must have a reduction script written in python. This script should be called reduce.py and should be placed in the user/scripts/autoreduction/ directory in the ISIS data archive. A full example script can be seen below:

import os
import sys
from mantid.simpleapi import (Load, Rebin, SaveNexus)
sys.path.append("/isis/NDXTEST/user/scripts/autoreduction")
import reduce_vars as params

def validate(file, dir):
    """
    Function that validates if a file and/or directory exist. If not a
    RunTimeError is raised which is picked up by Autoreduction.

    :param file: full path of data file. Provide empty string to ignore
    :type file: str
    :param dir: full path of a directory. Provide empty string to ignore
    :type dir: str
    """
    print("Running validation")
    if file:
        if not os.path.isfile(file):
            raise RuntimeError("Unable to find file: {}".format(file))
    if dir:
        if not os.path.isdir(dir):
            raise RuntimeError("Unable to find directory: {}".format(dir))
    print("Validation successful")

def main(input_file, output_dir):
   validate(input_file, output_dir)
   workspace = Load(input_file)
   # use rebin params defined in reduce_vars
   workspace = Rebin(workspace, params['rebin'])
   SaveNexus(workspace, output_dir)

Imports

All your imports should be placed at the top of the file. These can include, python packages, mantid packages and the reduce_vars we defined in the above section.

# import standard python packages
import os
import sys

# import mantid packages
from mantid.simpleapi import (Rebin, ReductionAlgorithm, SaveNexusProcessed)

# import reduce_vars
import reduce_vars as params

Validate

The first function you define should validate that the files and directories you require exist. This should include all files and directories. If any can not be found, an exception should be raised. Below is an example for reference:

def validate(file, dir):
    """
    Function that validates if a file and/or directory exist. If not a
    RunTimeError is raised which is picked up by Autoreduction.

    :param file: full path of data file. Provide empty string to ignore
    :type file: str
    :param dir: full path of a directory. Provide empty string to ignore
    :type dir: str
    """
    print("Running validation")
    if file:
        if not os.path.isfile(file):
            raise RuntimeError("Unable to find file: {}".format(file))
    if dir:
        if not os.path.isdir(dir):
            raise RuntimeError("Unable to find directory: {}".format(dir))
    print("Validation successful")

Main

This function should call the validate function to check inputs and then perform reduction steps. The function must take an input_file and output_dir as parameters. Both arguments should be validated in the validate function.

def main(input_file, output_dir):
    # validate
    validate(input_file, output_dir)
    # Perform reduction
    reduced_workspace = ReductionAlgorithm(input_file, ...)
    # Save output
    SaveNexusProcessed(reduced_workspace, output_dir)

If you have any difficulty please get in touch and we'll be happy to help

If you feel a value is incorrectly being marked as invalid please Contact Us.

Autoreduction GitHub Repository

Here is a link to the Autoreduction Github repository

How do I add new instrument variables?

To add new instrument variables first you must be part of the instrument scientist team within ICAT. Once you are part of this team you should see an "Edit Reduction Variables" link on the right of the instrument name on the main run list page or "Add new run variables" Clicking either of these links will take you to a form in which you can set the values of variables exposed by that instrument's reduce script.

When changing variables why is value X invalid for variable Y?

Any value entered for a variable must match the type of the default value already in the reduction script. If a value doesn't match, you should see a warning message explaining the reason why you will be prevented from submitting new variables until resolved.

If you feel a value is incorrectly being marked as invalid please Contact Us.

I've modified my reduction scripts, why aren't my new runs using it?

To make sure new reductions coming straight from the instrument use new reduce.py scripts automatically, you can navigate to the instrument summary page for a given instrument. From here, select the "See instrument variables" link located on the right of the status table to view ongoing and upcoming instrument configurations. From here, you can select "edit variables" on the left side of the upcoming variables table. When presented with the variables, ensure that the "Track script changes" checkbox is checked. This is located on the right side of the page. You should then select the "Submit Changes" button at the bottom of the page to save this change.

When re-running a job, if you want the re-run to use the current reduce.py script click the "Reset to values in the current reduce_vars script" link under "Additional Actions" located on the right side of the page.

To update instrument variables to the new ones entered into the reduce_vars.py script, navigate to the "Instrument Summary" page and select the "Configure New Runs" button. This will take you to the instrument variables page. From here, click "Reset to values in the current reduce_vars script", located on the right side of the page. This must be done any time reduce_vars.py is modified.

How can I make a reduction script compatible with Autoreduction?

For a reduction script to be compatible with the Autoreduction web application, it must be named reduce.py Please make sure to expose what reduction variables can be modified. These variables must be made available in a file named reduce_vars.py. For an example of this please See Here for documentation. See Here for the full script. Autoreduction expects the reduction script to have a main() method that takes in two arguments, data and output. These are passed in a kwargs but it may be desirable for these to also be accepted from the command line. For an example of how to do this please See Here . The reduction script should perform all operations on the provided data file and save the output to the output directory provided to the script. The output will be copied to the cache upon completion. If additional save locations are required these can be returned as a list of system paths from the main method.

If you feel your script will need to save to additional locations please Contact Us to ensure the locations are accessible from the Autoreduction machine.

Where should I put reduction scripts?

When you have made sure your reduction script is in the correct format and the variables are exposed, it must be placed within the Autoreduction directory within the user script directory of the appropriate instrument.

For example, the reduction script for MERLIN would be located at: \\isis\inst$\NDXMERLIN\user\scripts\autoreduction\reduce.py
and the variables at: \\isis\inst$\NDXMERLIN\user\scripts\autoreduction\reduce_vars.py

Where does my reduced data go?

When a reduction job has completed, the reduced data is moved to the CEPH storage cluster. The full file path to the data is detailed in the run summary pages for completed runs.

An example file path in CEPH would be: /instrument/GEM/RBNumber/[RB NUMBER]/autoreduced/[RUN NUMBER].

What do the reduction statuses mean?

There are 5 different states a reduction can be in:
  • Queued - The run is waiting to be processed
  • Processing - The reduction is in progress
  • Complete - The reduction has been complete and the reduced data is available
  • Error - Something has gone wrong processing the reduction or in the reduction script. If this happens you can contact us for support.
  • Skipped - The run was marked to be skipped and has not been reduced

How do I save out plots from my reduce.py?

To get a plot to show into the web app you need to save it into the reduction output directory with the following file name:

<name-part-of-the-data-file>*<.png or other extension>

where e.g. if the data file is "GEM90963.nxs" then "GEM90963" is the name-part-of-the-data-file, and the plot file extensions supported are "png", "jpg", "bmp", "gif", "tiff".

The following function is provided to help you get that information from the input file parameter Autoreduction called the main() method with in reduce.py:

def get_name_of_data_file(datafile):
    """
    Finds only the name of the data file and returns it. Removes the rest
    of the input full path and the file extension.

    :param datafile: The input file provided in the reduce.py file
    :return: Only the name of the data file
    """
    import os
    filename_and_extension = os.path.basename(datafile)
    just_filename, extension = os.path.splitext(filename_and_extension)
    return just_filename

To save out your matplotlib figure you can do:

 ... some plotting code with matplotlib ...

filename = get_name_of_data_file(input_file)
my_matplotlib_figure.savefig(os.path.join(output_dir, f"{filename}.png"), dpi=None)

How do I save out interactive plots with Plotly?

To get an interactive plot in the webapp you need to do the plotting with Plotly and save it to a .json file.

<name-part-of-the-data-file>.json

where e.g. if the data file is "GEM90963.nxs" then "GEM90963" is the name-part-of-the-data-file, and the plot file extension must be "json".

The save code that you need in the reduction script is:

# Serialise the figure to a JSON that can be shown in the webapp
figure_json = my_plotly_figure.to_json()

# Save the file in the output_dir
filename = get_name_of_data_file(input_file)
with open(os.path.join(output_dir, f"{filename}.json")) as figfile:
    figfile.write(figure_json)

Note: the above uses Python 3's f-string syntax to make the output JSON filename.

Plotting with Plotly

Plotly is slightly different from Matplotlib, but the difference is mostly in the syntax, while the fundamental features remain the same. As long as Plotly successfully serialises the plot to_json, it should also work in the webapp.

Because Mantid integrates Matplotlib, it is not possible to plot workspaces directly into Plotly. You will have to extract the data manually via the readX, readY, and similar methods.

To avoid adding plotting code here that will get out of date, it is recommended to have a look at the documentation.

Plotly express or graphics_objects (go)?

Both of Plotly's plotting APIs should work. Just use whatever works best for your case.

What are the output structures available in autoreduction?

There are 2 different output structures for reduction runs:

  • Flat Output: In flat output mode all files are outputted into a flat structure. This means that all runs for the same RB will be in the same folder. This mode offers the most simple structure but means every subsequent rerun of a run would overwrite the previous run.

  • Versioned output: In Versioned output mode, each run all reruns of runs will be outputted into their own separate folder. For example RB123/1234/run-version-1. This structure results in more files and folders but has the benefit that every version of a reduction run will be kept.

Sorry, no help topics matched your query.
If you still require help, please Contact Us.
{% endblock %} {% block scripts %} {% endblock %}