--- title: Utils keywords: fastai sidebar: home_sidebar summary: "General utilities. Should probably split up into `utils.time` and `utils.download`" description: "General utilities. Should probably split up into `utils.time` and `utils.download`" nb_path: "notebooks/01_utils.ipynb" ---
{% raw %}
{% endraw %} {% raw %}
{% endraw %}

Time format strings

These are the different format strings these utils convert from and to.

An identifier with _dt_ in its name signifies a full datetime format as compared to dates only.

{% raw %}
{% endraw %} {% raw %}

nasa_date_to_iso[source]

nasa_date_to_iso(datestr, with_hours=False)

Convert the day-number based NASA date format to ISO.

Parameters

datestr : str Date string in the form Y-j

Returns

Datestring in ISO standard yyyy-mm-ddTHH:MM:SS.MMMMMM

Parameters:

  • datestr : <class 'inspect._empty'>

  • with_hours : <class 'bool'>, optional

{% endraw %} {% raw %}
{% endraw %} {% raw %}
nasa_date = "2010-110"
iso_date = "2010-4-20"
{% endraw %} {% raw %}
assert nasa_date_to_iso(nasa_date, with_hours=True) == "2010-04-20T00:00:00"
assert nasa_date_to_iso(nasa_date) == "2010-04-20"
{% endraw %} {% raw %}

iso_to_nasa_date[source]

iso_to_nasa_date(datestr)

Convert iso date to day-number based NASA date.

Parameters

datestr : str Date string in the form Y-m-d

Returns

Datestring in NASA standard yyyy-jjj

Parameters:

  • datestr : <class 'inspect._empty'>
{% endraw %} {% raw %}
{% endraw %} {% raw %}
assert iso_to_nasa_date(iso_date) == nasa_date
{% endraw %} {% raw %}

nasa_datetime_to_iso[source]

nasa_datetime_to_iso(dtimestr)

Convert the day-number based NASA datetime format to ISO.

Note: This is dateTIME vs nasa_date_to_iso which is just DATE.

Parameters

dtimestr : str Datetime string in the form Y-jTH:M:S

Returns

Datestring in ISO standard yyyy-mm-ddTHH:MM:SS.MMMMMM

Parameters:

  • dtimestr : <class 'inspect._empty'>
{% endraw %} {% raw %}
{% endraw %} {% raw %}
nasa_datetime = "2010-110T10:12:14"
nasa_datetime_with_ms = nasa_datetime + ".123000"
iso_datetime = "2010-04-20T10:12:14"
iso_datetime_with_ms = iso_datetime + ".123000"
{% endraw %} {% raw %}
assert nasa_datetime_to_iso(nasa_datetime) == iso_datetime
assert nasa_datetime_to_iso(nasa_datetime_with_ms) == iso_datetime_with_ms
{% endraw %} {% raw %}

iso_to_nasa_datetime[source]

iso_to_nasa_datetime(dtimestr)

Convert iso datetime to day-number based NASA datetime.

Parameters

dtimestr : str Datetime string in the form yyyy-mm-ddTHH-MM-SS

Returns

Datestring in NASA standard yyyy-jjjTHH-MM-SS

Parameters:

  • dtimestr : <class 'inspect._empty'>
{% endraw %} {% raw %}
{% endraw %} {% raw %}
assert iso_to_nasa_datetime(iso_datetime) == nasa_datetime
assert iso_to_nasa_datetime(iso_datetime_with_ms) == nasa_datetime_with_ms
{% endraw %} {% raw %}

replace_all_nasa_times[source]

replace_all_nasa_times(df)

Find all NASA times in dataframe and replace with ISO.

Changes will be implemented on incoming dataframe!

This will be done for all columns with the word TIME in the column name.

Parameters

df: pandas.DataFrame DataFrame with NASA time columns.

Returns

Nothing (Changes implemented in-place)

Parameters:

  • df : <class 'inspect._empty'>
{% endraw %} {% raw %}
{% endraw %}

Network utils

{% raw %}

class ProgressBar[source]

ProgressBar(*_, **__) :: tqdm

Provides update_to(n) which uses tqdm.update(delta_n).

Parameters:

  • args : <class 'inspect._empty'>

  • kwargs : <class 'inspect._empty'>

{% endraw %} {% raw %}

parse_http_date[source]

parse_http_date(text)

Parse date string retrieved via urllib.request.

Parameters:

  • text : <class 'inspect._empty'>
{% endraw %} {% raw %}

get_remote_timestamp[source]

get_remote_timestamp(url)

Parameters:

  • url : <class 'inspect._empty'>
{% endraw %} {% raw %}

download[source]

download(url, local_dir='.', use_tqdm=True, **kwargs)

Simple wrapper of urlretrieve

Adding a default path to urlretrieve

Parameters:

url : str HTTP(S) URL to download local_dir : str,pathlib.Path Local directory where to store the download. **kwargs : {dict} Keyword args to be handed to urlretrieve.

Returns

Tuple Tuple returned by urlretrieve

Parameters:

  • url : <class 'inspect._empty'>

  • local_dir : <class 'str'>, optional

  • use_tqdm : <class 'bool'>, optional

  • kwargs : <class 'inspect._empty'>

{% endraw %} {% raw %}

url_retrieve[source]

url_retrieve(url:str, outfile:str, chunk_size:int=128)

Improved urlretrieve with progressbar, timeout and chunker.

This downloader has built-in progress bar using tqdm and using the requests package it improves standard urllib behavior by adding time-out capability.

I tested different chunk_sizes and most of the time 128 was actually fastest, YMMV.

Parameters

url : str, urlpath.URL The URL to download outfile: str, pathlib.Path The path where to store the downloaded file. chunk_size : int, optional The size of the chunk for the request.iter_content call. Default: 128

See also

Inspired by https://stackoverflow.com/a/61575758/680232

Parameters:

  • url : <class 'str'>

  • outfile : <class 'str'>

  • chunk_size : <class 'int'>, optional

{% endraw %} {% raw %}

have_internet[source]

have_internet()

Fastest way to check for active internet connection.

From https://stackoverflow.com/a/29854274/680232

{% endraw %} {% raw %}
{% endraw %}

Image processing helpers

{% raw %}

height_from_shadow[source]

height_from_shadow(shadow_in_pixels, sun_elev)

Calculate height of an object from its shadow length.

Note, that your image might have been binned. You need to correct shadow_in_pixels for that.

Parameters

shadow_in_pixels : float Measured length of shadow in pixels sun_elev : angle(float) Angle of sun over horizon

Returns

height [meter]

Parameters:

  • shadow_in_pixels : <class 'inspect._empty'>

  • sun_elev : <class 'inspect._empty'>

{% endraw %} {% raw %}

get_gdal_center_coords[source]

get_gdal_center_coords(imgpath)

Get center rows/cols pixel coordinate for GDAL-readable dataset.

Check CLI gdalinfo --formats to see all formats that GDAL can open.

Parameters

imgpath: str, pathlib.Path Path to raster image that is readable by GDLA

Parameters:

  • imgpath : <class 'inspect._empty'>
{% endraw %} {% raw %}

file_variations[source]

file_variations(filename, extensions)

Create a variation of file names.

Generate a list of variations on a filename by replacing the extension with the provided list.

Parameters

filename: str, pathlib.Path The original file name to use as a base. extensions: list-like A list of file extensions to generate new filenames.

Returns

list of pathlib.Paths

Notes

Adapted from T. Olsens `file_variations of the pysis module for using pathlib.

Parameters:

  • filename : <class 'inspect._empty'>

  • extensions : <class 'inspect._empty'>

{% endraw %} {% raw %}
{% endraw %} {% raw %}
fname = 'abc.txt'
{% endraw %} {% raw %}
extensions = ".cub .cal.cub .map.cal.cub".split()
{% endraw %} {% raw %}
file_variations(fname, extensions)
[PosixPath('abc.cub'), PosixPath('abc.cal.cub'), PosixPath('abc.map.cal.cub')]
{% endraw %}