Internal API reference

Request

Request class used to handle HTTP.

process_cookies(raw_cookies, pconfig)[source]

Process the raw cookies and replace with the real data.

Return type

Dict[str, str]

process_headers(raw_headers, pconfig)[source]

Process the raw headers and replace with the real data.

Return type

Dict[str, str]

process_data(raw_data, pconfig)[source]

Process the raw HTTP data and replace with the real data.

Return type

Dict[str, str]

class Request(function, url, method, **kwargs)[source]

Class holding the elements of the HTTP request.

When a Flow object is created, it defines a Request object with the information necessary to create a HTTP request. The “method” and “url” attributes are required. Everything else is optional.

The Request object can contain Plugins which will be evaluated and its value replaced in the HTTP request.

method

A string with the HTTP request method. Only GET and POST is supported for now.

url

A string with the URL of the HTTP request.

cookies

A list of Cookie objects to be sent with the HTTP request.

headers

A list of Header objects to be sent with the HTTP request.

data

A dictionary of Any objects. Can contain strings and Plugins. When a key or a value of the dictionary is a Plugin, it will be evaluated and its value will be used in the HTTP request. If the “method” is GET those values will be put inside the URL parameters, and if the “method” is POST they will be inside the POST request body.

__init__(function, url, method, **kwargs)[source]

Initializes the Request object.

list_inputs()[source]

Returns a list of request’s inputs.

Return type

Optional[Dict[str, Plugin]]

send(pconfig)[source]

Sends the HTTP request.

With the given user information, replaces the input plugins with their values, and sends the HTTP request. Returns the response.

Parameters
  • user – A User object with the user specific data to be used when processing inputs.

  • pconfig – A Config object with the global Raider configuration.

Return type

Optional[Response]

Returns

A requests.models.Response object with the HTTP response received after sending the generated request.

class Template(method, url=None, cookies=None, headers=None, data=None)[source]

Template class to hold requests.

It will initiate itself with a Request parent, and when called will return a copy of itself with the modified parameters.

__init__(method, url=None, cookies=None, headers=None, data=None)[source]

Initializes the template object.

__call__(method=None, url=None, cookies=None, headers=None, data=None)[source]

Allow the object to be called.

Accepts the same arguments as the Request class. When called, will return a copy of itself with the modified parameters.

Return type

Template

Structures

Data structures used in Raider.

class DataStore(data)[source]

Class defining a dictionary-like data structure.

This class was created to hold information relevant to Raider in a structure similar to Python dictionaries.

__init__(data)[source]

Initializes the DataStore object.

Given a dictionary with the data, store them in this object.

Parameters

data (Optional[Dict[Any, Any]]) – A dictionary with Any elements to be stored.

__getitem__(key)[source]

Getter to return an element with the key.

Return type

Any

__setitem__(key, value)[source]

Setter to add a new element to DataStore.

Return type

None

__iter__()[source]

Iterator to yield the keys.

Return type

Iterator[Any]

__next__()[source]

Iterator to get the next element.

Return type

Any

update(data)[source]

Updates the DataStore with a new element.

Return type

None

pop(name)[source]

Pops an element from the DataStore.

Return type

Any

keys()[source]

Returns a list of the keys in the DataStore.

Return type

List[Any]

values()[source]

Returns a list of the values in the DataStore.

Return type

List[Any]

to_dict()[source]

Returns the DataStore elements as a dictionary.

Return type

Dict[Any, Any]

items()[source]

Returns a list of tuples containing the keys and values.

Return type

List[Tuple[Any, Any]]

class HeaderStore(data)[source]

Class storing the HTTP headers.

This class inherits from DataStore, and converts the values into Header objects.

__init__(data)[source]

Initializes the HeaderStore object.

Creates a HeaderStore object out of the given Header list.

Parameters

data (Optional[List[Header]]) – A list of Header objects to store.

set(header)[source]

Sets the value of a Header.

Given a Header object, add or update its value in the HeaderStore.

Parameters

header (Header) – A Header object to be added to the HeaderStore.

Return type

None

merge(headerstore)[source]

Merge HeaderStore object with another one.

Return type

None

classmethod from_dict(data)[source]

Creates a HeaderStore object from a dictionary.

Given a dictionary with header values, creates a HeaderStore object and returns it.

Parameters

data (Optional[Dict[str, str]]) – A dictionary with header values. Those will be mapped in Header objects.

Return type

HeaderStore

Returns

A HeaderStore object containing the headers created from the supplied dictionary.

class CookieStore(data)[source]

Class storing the HTTP cookies.

This class inherits from DataStore, and converts the values into Cookie objects.

__init__(data)[source]

Initializes a CookieStore object.

Given a list of Cookie objects, create the CookieStore containing them.

Parameters

data (Optional[List[Cookie]]) – A list of Cookies to be added to the CookieStore.

set(cookie)[source]

Sets the value of a Cookie.

Given a Cookie object, add or update its value in the CookieStore.

Parameters

cookie (Cookie) – A Cookie object to be added to the CookieStore

Return type

None

merge(cookiestore)[source]

Merge CookieStore object with another one.

Return type

None

classmethod from_dict(data)[source]

Creates a CookieStore object from a dictionary.

Given a dictionary with cookie values, creates a CookieStore object and returns it.

Parameters

data (Optional[Dict[str, str]]) – A dictionary with cookie values. Those will be mapped in Cookie objects.

Return type

CookieStore

Returns

A CookieStore object containing the cookies created from the supplied dictionary.

User

Classes used for handling users.

class User(username=None, password=None, **kwargs)[source]

Class holding user related information.

User objects are created inside the Users. Each User object contains at least the username and the password. Every time a Plugin generates an output, it is saved in the User object. If the Plugin is a Cookie or a Header, the output will be stored in the the cookies and headers attributes respectively. Otherwise they’ll be saved inside data.

username

A string containing the user’s email or username used to log in.

password

A string containing the user’s password.

cookies

A CookieStore object containing all of the collected cookies for this user. The Cookie plugin only writes here.

headers

A HeaderStore object containing all of the collected headers for this user. The Header plugin only writes here.

data

A DataStore object containing the rest of the data collected from plugins for this user.

__init__(username=None, password=None, **kwargs)[source]

Initializes a User object.

Creates an object for easy access to user specific information. It’s used to store the username, password, cookies, headers, and other data extracted from the Plugin objects.

Parameters
  • username (Optional[str]) – A string with the username used for the login process.

  • password (Optional[str]) – A string with the password used for the login process.

  • **kwargs (Dict[str, str]) – A dictionary with additional data about the user.

Sets the cookies for the user.

Given a Cookie object, update the user’s cookies attribute to include this Cookie's value.

Parameters

cookie (Cookie) – A Cookie Plugin object with the data to be added.

Return type

None

set_cookies_from_dict(data)[source]

Set user’s cookies from a dictionary.

Given a dictionary of cookie values as strings, convert them to Cookie objects, and load them in the User object respectively.

Parameters

data (Dict[str, str]) – A dictionary of strings corresponding to cookie keys and values.

Return type

None

set_header(header)[source]

Sets the headers for the user.

Given a Header object, update the user’s headers attribute to include this header value.

Parameters

header (Header) – A Header Plugin <raider.plugins.common.Plugin object with the data to be added.

Return type

None

set_headers_from_dict(data)[source]

Set user’s headers from a dictionary.

Given a dictionary of header values as strings, convert them to Header objects, and load them in the User object respectively.

Parameters

data (Dict[str, str]) – A dictionary of strings corresponding to header keys and values.

Return type

None

set_data(data)[source]

Sets the data for the user.

Given a Plugin, update the user’s data attribute to include this data.

Parameters

data (Plugin) – A Plugin object with the data to be added.

Return type

None

set_data_from_dict(data)[source]

Set user’s data from a dictionary.

Given a dictionary of data items made out of strings, update the data attribute accordingly.

Parameters

data (Dict[str, str]) – A dictionary of strings corresponding to data keys and values.

Return type

None

to_dict()[source]

Returns this object’s data in a dictionary format.

Return type

Dict[str, str]

class Users(users=None, active_user='DEFAULT')[source]

Class holding all the users of the application.

Users inherits from DataStructure, and contains the users set up in hyfiles. Each user is an User object. The data from a Users object can be accessed same way like from the DataStore.

active_user

A string with the username attribute of the currently active User.

__init__(users=None, active_user='DEFAULT')[source]

Initializes the Users object.

Given a list of dictionaries, map them to a User object and store them in this Users object.

Parameters
  • users (Optional[List[Dict[Keyword, str]]]) – A list of dictionaries. Dictionary’s data is mapped to a User object.

  • active_user (str) – An optional string specifying the default User.

to_dict()[source]

Returns the Users object data in dictionary format.

Return type

Dict[str, str]

property active: User

Returns the active User as an Users object.

Return type

User

utils

Functions that are used within Raider.

default_user_agent()[source]

Gets the default user agent.

Gets the current version of Raider and creates the user agent string.

Return type

str

Returns

A string with the user agent.

get_config_dir()[source]

Gets the configuration directory.

Returns the path of the directory with the Raider configuration files.

Return type

str

Returns

A string with the path of the configuration directory.

get_config_file(filename)[source]

Gets the configuration file.

Given the file name, it returns the path of this file in the Raider configuration directory.

Parameters

filename (str) – A string with the name of the file to look up for in the main configuration directory.

Return type

str

Returns

A string with the path of the file.

get_project_dir(project)[source]

Gets the directory of the project.

Given the name of the project, returns the path to the directory containing the configuration files for this project.

Parameters

project (str) – A string with the name of the project.

Return type

str

Returns

A string with the path of the directory where the config files for the project are located.

get_project_file(project, filename)[source]

Gets a file from a project.

Given the project name and the file name, it returns the path to that file.

Parameters
  • project (str) – A string with the name of the project.

  • filename (str) – A string with the file name.

Return type

str

Returns

The path of the file in the project directory.

import_raider_objects()[source]

Imports Raider objects to use inside hy configuration files.

To make Raider objects visible inside hy files without using separate imports, this function does the imports and returns the locals() which is later used when evaluating hy files.

Return type

Dict[str, Any]

Returns

A dictionary with the locals() containing all the Raider objects that can be used in hy files.

hy_dict_to_python(hy_dict)[source]

Converts a hy dictionary to a python dictionary.

When creating dictionaries in hylang using :parameters they become hy.models.Keyword objects. This function converts them to normal python dictionaries.

Parameters

hy_dict (Dict[Keyword, Any]) – A dictionary created in hy, which uses hy.models.Keyword instead of simple strings as keys.

Return type

Dict[str, Any]

Returns

A dictionary with the same elements only with hy.models.Keyword keys converted into normal strings.

py_dict_to_hy_list(data)[source]

Converts a python dictionary to a hylang list.

In hy, dictionaries are created out of lists, and this function converts a normal python dictionary to a list made out of hy symbols that will be later used to create the hy dictionary.

Parameters

data (Dict[str, Any]) – A python dictionary with the data to convert.

Return type

List[Union[String, Dict, Symbol]]

Returns

A list with hy objects that can be used to create a hy dictionary.

create_hy_expression(variable, value)[source]

Creates a hy expression.

Raider configuration is saved in hy format, and this function creates the assignments in this format.

Parameters
  • variable (str) – A string with the name of the variable to be created.

  • value (Union[str, Dict[Any, Any], List[Any]]) – The value of the variable.

Return type

str

Returns

A string with the valid hy expression.

serialize_hy(form)[source]

Serializes hy expression.

This function serializes the supplied hy expression and returns it in a string format, so that it can be later saved in a file.

Parameters

form (Union[Expression, Dict, List, Symbol, Integer, Keyword, String]) – A hy expression to convert to a string.

Return type

str

Returns

A string with the serialized form.

eval_file(filename, shared_locals=None)[source]

Evaluate hy file.

This function evaluates all the content inside the supplied hy file, and returns the created locals() so that it can be later used for other files.

Parameters
  • filename (str) – A string with the file name to be evaluated.

  • shared_locals (Optional[Dict[str, Any]]) – A dictionary with the locals() that will be considered when evaluating the file.

Return type

Dict[str, Any]

Returns

A dictionary with the updated locals() after evaluating the hy file.

eval_project_file(project, filename, shared_locals)[source]

Evaluate a hy file from a project.

This function evaluates the specified file inside the project and returns the locals() which are updated after evaluating the file.

Parameters
  • project (str) – A string with the name of the project.

  • filename (str) – A string with the file name to be evaluated.

  • shared_locals (Dict[str, Any]) – A dictionary of locals() to be included when evaluating the file.

Return type

Dict[str, Any]

Returns

A dictionary of locals() updated after evaluating the file.

list_projects()[source]

List existing projects.

This function returns the list of projects that have been configured in Raider.

Return type

List[str]

Returns

A list with the strings of the project found in the configuration directory.

list_hyfiles(project)[source]

List hyfiles for a project.

This function returns the list of hyfiles that have been configured in Raider for the provided project.

Parameters

project (str) – A string with the project name.

Return type

List[str]

Returns

A list with the strings of the project found in the configuration directory.

match_tag(html_tag, attributes)[source]

Tells if a tag matches the search.

This function checks whether the supplied tag matches the attributes. The attributes is a dictionary, and the values are treated as a regular expression, to allow checking for tags that don’t have a static value.

Parameters
  • html_tag (Tag) – A bs4.element.Tag object with the tag to be checked.

  • attributes (Dict[str, str]) – A dictionary of attributes to check whether they match with the tag.

Return type

bool

Returns

A boolean saying whether the tag matched with the attributes or not.

parse_json_filter(raw)[source]

Parses a raw JSON filter and returns a list with the items.

Parameters

raw (str) – A string with the expected JSON filter.

Return type

List[str]

Returns

A list with all items found in the filter.