Rucio demo

Starting a Rucio demo instance

The instructions can be found in Setting up a Rucio demo environment

Boostrap the Rucio demo

Once everything is ready you can log into the container and start playing around with rucio:

$ docker exec -i -t demo_rucio_1 /bin/bash

The bash tab completion is by default enabled for the Rucio CLIs. The rucio configuration file is located in /opt/rucio/etc/rucio.cfg. The clients are configured to talk to the local server instance:

$ cat /opt/rucio/etc/rucio.cfg
...
[client]
rucio_host = https://localhost:443
auth_host = https://localhost:443
...

In this demo, an apache server runs and the log files are located in ´/var/log/rucio/´ To query the Rucio server, you can ping it with:

$ curl -k https://localhost/ping
 {"version": "1.14.9.post1"}

In the corresponding apache log file, you can see the access log entry:

$ tail -f  /var/log/rucio/httpd_access_log
  localhost - - [27/Feb/2018:11:57:59 +0000] "GET /ping HTTP/1.1" 200 27

The equivalent command exists with the rucio CLI:

$ rucio ping
cannot get auth_token
2018-02-27 13:22:54,297       ERROR   Cannot authenticate.
Details: userpass authentication failed

But it fails since the command also authenticates with Rucio and this Rucio instance is not configured.

Configuring Rucio

A bootstrap script is provided at /setup_data.py. This script creates the database tables and creates 2 accounts: root and jdoe and 2 local Rucio storage elements (RSEs): SITE1_DISK, SITE2_DISK mounted on the /tmp partition.

This example script uses the Rucio python client module.To execute it:

$ /setup_data.py

By default, it will create a MySQL database as specified in the configuration:

$ cat /opt/rucio/etc/rucio.cfg
 ...
 [database]
 default = mysql://rucio:rucio@mysql/rucio
 ...

To create a different one like sqlite, you need to change this section, e.g.,:

$ cat /opt/rucio/etc/rucio.cfg
 ...
 [database]
 default = sqlite:////tmp/rucio.db
 ...

Execute the script and restart apache:

$ /setup_data.py
$  httpd -k restart

Rucio supports mysql, mariadb, oracle and postgresql.

rucio ping now works:

$ rucio ping
1.14.9.post1

The token is stored in /tmp/root/.rucio_root/auth_token_root.

The equivalent can be done with the Rucio python clients:

$ python
 ...
 >>> from rucio.client import Client
 >>> rucio_client = Client()
 >>> rucio_client.ping()
 {u'version': u'1.14.9.post1'}

You can also check your account:

[root@3a6d4527e1f6 rucio]# rucio whoami
status     : ACTIVE
account    : root
account_type : SERVICE
created_at : 2018-02-08T15:37:26
suspended_at : None
updated_at : 2018-02-08T15:37:26
deleted_at : None
email      : None

To list the RSEs:

$ rucio list-rses
SITE1_DISK
SITE2_DISK

We can add some RSE attributes:

$ rucio-admin rse set-attribute  --rse SITE1_DISK --key zone --value eu-west-3
Added new RSE attribute for SITE1_DISK: zone-eu-west-3

$ rucio-admin rse set-attribute  --rse SITE2_DISK --key zone --value us-west-1
Added new RSE attribute for SITE2_DISK:  rucio list-rses --expression 'zone=us-west-1'

$ rucio list-rses --expression 'zone=eu-west-3'
   SITE1_DISK

To list the accounts:

$ # rucio-admin account list
jdoe
root

Testing dataset upload, creation of dataset and rules

There are no datasets created yet. To generate datasets and copy them to one of the RSEs, you can use a daemon called automatix:

[root@3a6d4527e1f6 rucio]# /usr/bin/rucio-automatix --run-once --input-file /opt/rucio/etc/automatix.json
...
2018-02-19 13:47:07,532 277     DEBUG   https://localhost:443 "POST /dids/tests/test.24659.automatix_stream.recon.AOD.917/dids HTTP/1.1" 201 None
2018-02-19 13:47:07,533 277     INFO    Thread [1/1] : Upload operation for tests:test.24659.automatix_stream.recon.AOD.917 done
2018-02-19 13:47:07,534 277     INFO    Thread [1/1] : Run with once mode. Exiting
2018-02-19 13:47:07,541 277     INFO    Thread [1/1] : Graceful stop requested
2018-02-19 13:47:07,541 277     INFO    Thread [1/1] : Graceful stop done

The daemon has created and uploaded a new dataset in the tests scope. One can list all the DIDs in this scope:

[root@3a6d4527e1f6 rucio]# rucio list-dids tests:*
+-------------------------------------------------+--------------+
| SCOPE:NAME                                      | [DID TYPE]   |
|-------------------------------------------------+--------------|
| tests:AOD.a9753781316c4b2f8bd88c60e9dd3570      | FILE         |
| tests:AOD.fc50eb5e2b1949919880f8218bf62108      | FILE         |
| tests:test.24659.automatix_stream.recon.AOD.917 | DATASET      |
+-------------------------------------------------+--------------+

And one can list the content of the dataset:

[root@3a6d4527e1f6 rucio]# rucio list-files tests:test.24659.automatix_stream.recon.AOD.917
+--------------------------------------------+--------------------------------------+-------------+------------+----------+
| SCOPE:NAME                                 | GUID                                 | ADLER32     | FILESIZE   | EVENTS   |
|--------------------------------------------+--------------------------------------+-------------+------------+----------|
| tests:AOD.a9753781316c4b2f8bd88c60e9dd3570 | 32C89A5A-F0BD-43F6-A958-099C46954C7F | ad:480900d5 | 1.000 MB   |          |
| tests:AOD.fc50eb5e2b1949919880f8218bf62108 | 1937B5B8-BFE3-4AE0-B5CE-28AFE964F5F8 | ad:32bf834e | 1.000 MB   |          |
+--------------------------------------------+--------------------------------------+-------------+------------+----------+
Total files : 2
Total size : 2.000 MB

Now if you list the rule for this dataset, you will see that there are no files listed on SITE1_DISK:

[root@3a6d4527e1f6 rucio]# rucio list-rules tests:test.24659.automatix_stream.recon.AOD.917
ID                                ACCOUNT    SCOPE:NAME                                       STATE[OK/REPL/STUCK]    RSE_EXPRESSION      COPIES  EXPIRES (UTC)    CREATED (UTC)
--------------------------------  ---------  -----------------------------------------------  ----------------------  ----------------  --------  ---------------  -------------------
7744c0e0dcce4243b906a2afbc8bc87f  root       tests:test.24659.automatix_stream.recon.AOD.917  OK[0/0/0]               SITE1_DISK               1                   2018-02-19 13:47:06

The information needs to be updated by another daemon called the judge. To run it once:

[root@3a6d4527e1f6 rucio]# /usr/bin/rucio-judge-evaluator --run-once
2018-02-19 13:47:37,943 328     DEBUG   re_evaluator[0/0] index query time 0.003242 fetch size is 1
2018-02-19 13:47:37,951 328     INFO    Re-Evaluating did tests:test.24659.automatix_stream.recon.AOD.917 for ATTACH
2018-02-19 13:47:38,010 328     DEBUG   Creating locks and replicas for rule 7744c0e0dcce4243b906a2afbc8bc87f [0/0/0]
2018-02-19 13:47:38,011 328     DEBUG   Creating OK Lock tests:AOD.a9753781316c4b2f8bd88c60e9dd3570 for rule 7744c0e0dcce4243b906a2afbc8bc87f
2018-02-19 13:47:38,011 328     DEBUG   Creating OK Lock tests:AOD.fc50eb5e2b1949919880f8218bf62108 for rule 7744c0e0dcce4243b906a2afbc8bc87f
2018-02-19 13:47:38,030 328     DEBUG   Rule 7744c0e0dcce4243b906a2afbc8bc87f  [2/0/0] queued 0 transfers
2018-02-19 13:47:38,031 328     DEBUG   queue requests
2018-02-19 13:47:38,031 328     DEBUG   Finished creating locks and replicas for rule 7744c0e0dcce4243b906a2afbc8bc87f [2/0/0]
2018-02-19 13:47:38,045 328     DEBUG   re_evaluator[0/0]: evaluation of tests:test.24659.automatix_stream.recon.AOD.917 took 0.101811

After this one can see that the 2 files from the dataset are located at SITE1_DISK:

[root@3a6d4527e1f6 rucio]# rucio list-rules tests:test.24659.automatix_stream.recon.AOD.917
ID                                ACCOUNT    SCOPE:NAME                                       STATE[OK/REPL/STUCK]    RSE_EXPRESSION      COPIES  EXPIRES (UTC)    CREATED (UTC)
--------------------------------  ---------  -----------------------------------------------  ----------------------  ----------------  --------  ---------------  -------------------
7744c0e0dcce4243b906a2afbc8bc87f  root       tests:test.24659.automatix_stream.recon.AOD.917  OK[2/0/0]               SITE1_DISK               1                   2018-02-19 13:47:06

One can then create another rule:

[root@3a6d4527e1f6 rucio]# rucio add-rule tests:test.24659.automatix_stream.recon.AOD.917 1 SITE2_DISK
[root@3a6d4527e1f6 rucio]# rucio list-rules tests:test.24659.automatix_stream.recon.AOD.917
ID                                ACCOUNT    SCOPE:NAME                                       STATE[OK/REPL/STUCK]    RSE_EXPRESSION      COPIES  EXPIRES (UTC)    CREATED (UTC)
--------------------------------  ---------  -----------------------------------------------  ----------------------  ----------------  --------  ---------------  -------------------
7744c0e0dcce4243b906a2afbc8bc87f  root       tests:test.24659.automatix_stream.recon.AOD.917  OK[2/0/0]               SITE1_DISK               1                   2018-02-19 13:47:06
f528e0681ebd404c90d534b7f7a254be  root       tests:test.24659.automatix_stream.recon.AOD.917  REPLICATING[0/2/0]      SITE2_DISK               1                   2018-02-19 13:51:42

Then you can download with:

[root@3a6d4527e1f6 rucio]# rucio download tests:test.24659.automatix_stream.recon.AOD.917
2018-02-19 19:32:22,868 INFO    Thread 1/2 : Starting the download of tests:AOD.a9753781316c4b2f8bd88c60e9dd3570
2018-02-19 19:32:22,869 INFO    Thread 2/2 : Starting the download of tests:AOD.fc50eb5e2b1949919880f8218bf62108
2018-02-19 19:32:22,922 INFO    Thread 1/2 : File tests:AOD.a9753781316c4b2f8bd88c60e9dd3570 trying from SITE1_DISK
2018-02-19 19:32:22,922 INFO    Thread 2/2 : File tests:AOD.fc50eb5e2b1949919880f8218bf62108 trying from SITE1_DISK
...
2018-02-19 19:32:23,410 INFO    Thread 1/2 : File tests:AOD.a9753781316c4b2f8bd88c60e9dd3570 successfully downloaded. 1.000 MB in 0.11 seconds = 9.09 MBps
2018-02-19 19:32:23,410 1032    INFO    Thread 1/2 : File tests:AOD.a9753781316c4b2f8bd88c60e9dd3570 successfully downloaded. 1.000 MB in 0.11 seconds = 9.09 MBps
----------------------------------
Download summary
----------------------------------------
DID tests:test.24659.automatix_stream.recon.AOD.917
Total files :                                 2
Downloaded files :                            2
Files already found locally :                 0
Files that cannot be downloaded :             0

To delete the rule:

[root@3a6d4527e1f6 rucio]# rucio update-rule --lifetime -7200 f528e0681ebd404c90d534b7f7a254be
Updated Rule

[root@3a6d4527e1f6 rucio]# /usr/bin/rucio-judge-cleaner --run-once
2018-02-19 19:59:16,258 1388    DEBUG   rule_cleaner[0/0] index query time 0.008735 fetch size is 1
2018-02-19 19:59:16,258 1388    INFO    rule_cleaner[0/0]: Deleting rule f528e0681ebd404c90d534b7f7a254be with expression SITE2_DISK
2018-02-19 19:59:16,273 1388    DEBUG   Deleting lock tests:AOD.a9753781316c4b2f8bd88c60e9dd3570 for rule f528e0681ebd404c90d534b7f7a254be
2018-02-19 19:59:16,281 1388    DEBUG   Deleting lock tests:AOD.fc50eb5e2b1949919880f8218bf62108 for rule f528e0681ebd404c90d534b7f7a254be
2018-02-19 19:59:16,359 1388    DEBUG   rule_cleaner[0/0]: deletion of f528e0681ebd404c90d534b7f7a254be took 0.100267

[root@3a6d4527e1f6 rucio]# rucio list-rules tests:test.24659.automatix_stream.recon.AOD.917
ID                                ACCOUNT    SCOPE:NAME                                       STATE[OK/REPL/STUCK]    RSE_EXPRESSION      COPIES  EXPIRES (UTC)    CREATED (UTC)
--------------------------------  ---------  -----------------------------------------------  ----------------------  ----------------  --------  ---------------  -------------------
7744c0e0dcce4243b906a2afbc8bc87f  root       tests:test.24659.automatix_stream.recon.AOD.917  OK[2/0/0]               SITE1_DISK               1                   2018-02-19 13:47:06