{% extends "base.html" %} {% block title %}Help pages{% endblock %} {% load static %} {% block stylesheets %} {% endblock %} {% block body %}
Get in touch with the Support Team
The reduced data can be access in two ways. Either through an IDAaaS virtual machine or using the IDAaaS FileBrowser.
It is possible to define some additional variables for scripts that can be used
during reduction. They can be altered via the web interface. These should be defined in the
//isis/NDX.../user/script/autoreduction/reduce_vars.py
file. This file has the
following format:
standard_vars = {
'perform corrections': False,
'reduction option': "Default value"
}
advanced_vars = {
'crop range': [100, 10000]
}
variable_help = {
'standard_vars': {'do_absorb_corrections' : "Do you want corrections on?",
'reduction option' : "message to explain option - this is displayed in the webapp"},
'advanced_vars': {'crop range' : "advanced settings are handled in a separate section"}
}
In order to reduce your data you must have a reduction script written in python.
This script should be called reduce.py
and should be placed in the
user/scripts/autoreduction/
directory in the ISIS data archive.
A full example script can be seen below:
import os
import sys
from mantid.simpleapi import (Load, Rebin, SaveNexus)
sys.path.append("/isis/NDXTEST/user/scripts/autoreduction")
import reduce_vars as params
def validate(file, dir):
"""
Function that validates if a file and/or directory exist. If not a
RunTimeError is raised which is picked up by Autoreduction.
:param file: full path of data file. Provide empty string to ignore
:type file: str
:param dir: full path of a directory. Provide empty string to ignore
:type dir: str
"""
print("Running validation")
if file:
if not os.path.isfile(file):
raise RuntimeError("Unable to find file: {}".format(file))
if dir:
if not os.path.isdir(dir):
raise RuntimeError("Unable to find directory: {}".format(dir))
print("Validation successful")
def main(input_file, output_dir):
validate(input_file, output_dir)
workspace = Load(input_file)
# use rebin params defined in reduce_vars
workspace = Rebin(workspace, params['rebin'])
SaveNexus(workspace, output_dir)
All your imports should be placed at the top of the file. These can include, python
packages, mantid packages and the reduce_vars
we defined in the above section.
# import standard python packages
import os
import sys
# import mantid packages
from mantid.simpleapi import (Rebin, ReductionAlgorithm, SaveNexusProcessed)
# import reduce_vars
import reduce_vars as params
The first function you define should validate that the files and directories you require exist. This should include all files and directories. If any can not be found, an exception should be raised. Below is an example for reference:
def validate(file, dir):
"""
Function that validates if a file and/or directory exist. If not a
RunTimeError is raised which is picked up by Autoreduction.
:param file: full path of data file. Provide empty string to ignore
:type file: str
:param dir: full path of a directory. Provide empty string to ignore
:type dir: str
"""
print("Running validation")
if file:
if not os.path.isfile(file):
raise RuntimeError("Unable to find file: {}".format(file))
if dir:
if not os.path.isdir(dir):
raise RuntimeError("Unable to find directory: {}".format(dir))
print("Validation successful")
This function should call the validate function to check inputs and then perform
reduction steps. The function must take an input_file
and output_dir
as parameters. Both arguments should be validated in the validate function.
def main(input_file, output_dir):
# validate
validate(input_file, output_dir)
# Perform reduction
reduced_workspace = ReductionAlgorithm(input_file, ...)
# Save output
SaveNexusProcessed(reduced_workspace, output_dir)
If you have any difficulty please get in touch and we'll be happy to help
If you feel a value is incorrectly being marked as invalid please Contact Us.
To add new instrument variables first you must be part of the instrument scientist team within ICAT. Once you are part of this team you should see an "Edit Reduction Variables" link on the right of the instrument name on the main run list page or "Add new run variables" Clicking either of these links will take you to a form in which you can set the values of variables exposed by that instrument's reduce script.
Any value entered for a variable must match the type of the default value already in the reduction script. If a value doesn't match, you should see a warning message explaining the reason why you will be prevented from submitting new variables until resolved.
If you feel a value is incorrectly being marked as invalid please Contact Us.
To make sure new reductions coming straight from the instrument use new reduce.py
scripts automatically, you can navigate to the instrument summary page for a
given instrument. From here, select the "See instrument variables" link located on
the right of the status table to view ongoing and upcoming instrument
configurations.
From here, you can select "edit variables" on the left side of the upcoming
variables table. When presented with the variables, ensure that the
"Track script changes" checkbox is checked. This is located on the right side of
the page. You should then select the "Submit Changes" button at the bottom of the
page to save this change.
When re-running a job, if you want the re-run to use the current reduce.py
script
click the "Reset to values in the current reduce_vars script" link under "Additional Actions" located
on the right side of the page.
To update instrument variables to the new ones entered into the reduce_vars.py
script, navigate to the "Instrument Summary" page and select the
"Configure New Runs" button. This will take you to the instrument variables page.
From here, click "Reset to values in the current reduce_vars script", located on the right side of the page.
This must be done any time reduce_vars.py
is modified.
For a reduction script to be compatible with the Autoreduction web application,
it must be named reduce.py
Please make sure to expose what reduction variables
can be modified.
These variables must be made available in a file named
reduce_vars.py
. For an example of this please
See Here
for documentation.
See Here
for the full script.
Autoreduction expects the reduction script to have a main()
method
that takes in two arguments, data and output.
These are passed in a kwargs but it may be desirable for these to also be accepted from the
command line.
For an example of how to do this please
See Here
.
The reduction script should perform all operations on the provided data file and
save the output to the output directory provided to the script. The output will
be copied to the cache upon completion. If additional save locations are
required these can be returned as a list of system paths from the main method.
If you feel your script will need to save to additional locations please Contact Us to ensure the locations are accessible from the Autoreduction machine.
When you have made sure your reduction script is in the correct format and the variables are exposed, it must be placed within the Autoreduction directory within the user script directory of the appropriate instrument.
For example, the reduction script for MERLIN would be located at:
\\isis\inst$\NDXMERLIN\user\scripts\autoreduction\reduce.py
and the variables at:
\\isis\inst$\NDXMERLIN\user\scripts\autoreduction\reduce_vars.py
When a reduction job has completed, the reduced data is moved to the CEPH storage cluster. The full file path to the data is detailed in the run summary pages for completed runs.
An example file path in CEPH would be:
/instrument/GEM/RBNumber/[RB NUMBER]/autoreduced/[RUN NUMBER]
.
To get a plot to show into the web app you need to save it into the reduction output directory with the following file name:
<name-part-of-the-data-file>*<.png or other extension>
where e.g. if the data file is "GEM90963.nxs" then "GEM90963" is the name-part-of-the-data-file, and the plot file extensions supported are "png", "jpg", "bmp", "gif", "tiff".
The following function is provided to help you get that information from the input file parameter Autoreduction called the main() method with in reduce.py:
def get_name_of_data_file(datafile):
"""
Finds only the name of the data file and returns it. Removes the rest
of the input full path and the file extension.
:param datafile: The input file provided in the reduce.py file
:return: Only the name of the data file
"""
import os
filename_and_extension = os.path.basename(datafile)
just_filename, extension = os.path.splitext(filename_and_extension)
return just_filename
To save out your matplotlib figure you can do:
... some plotting code with matplotlib ...
filename = get_name_of_data_file(input_file)
my_matplotlib_figure.savefig(os.path.join(output_dir, f"{filename}.png"), dpi=None)
To get an interactive plot in the webapp you need to do the plotting with Plotly and save it to a .json file.
<name-part-of-the-data-file>.json
where e.g. if the data file is "GEM90963.nxs" then "GEM90963" is the name-part-of-the-data-file
,
and the plot file extension must be "json".
The save code that you need in the reduction script is:
# Serialise the figure to a JSON that can be shown in the webapp
figure_json = my_plotly_figure.to_json()
# Save the file in the output_dir
filename = get_name_of_data_file(input_file)
with open(os.path.join(output_dir, f"{filename}.json")) as figfile:
figfile.write(figure_json)
Note: the above uses Python 3's f-string syntax to make the output JSON filename.
Plotly is slightly different from Matplotlib, but the difference is mostly in the syntax, while the fundamental features remain the same.
As long as Plotly successfully serialises the plot to_json
, it should also work in the webapp.
Because Mantid integrates Matplotlib, it is not possible to plot workspaces directly into Plotly. You will have to extract the data manually via the readX
, readY
, and similar methods.
To avoid adding plotting code here that will get out of date, it is recommended to have a look at the documentation.
Both of Plotly's plotting APIs should work. Just use whatever works best for your case.
There are 2 different output structures for reduction runs:
Flat Output: In flat output mode all files are outputted into a flat structure. This means that all runs for the same RB will be in the same folder. This mode offers the most simple structure but means every subsequent rerun of a run would overwrite the previous run.
Versioned output: In Versioned output mode, each run all reruns of
runs will be outputted into their own separate folder. For example
RB123/1234/run-version-1
. This structure results in more files and folders
but has the benefit that every version of a reduction run will be kept.