19.1.40. camcops_server.cc_modules.cc_sqla_coltypes¶
Copyright (C) 2012-2018 Rudolf Cardinal (rudolf@pobox.com).
This file is part of CamCOPS.
CamCOPS is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any later version.
CamCOPS is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details.
You should have received a copy of the GNU General Public License along with CamCOPS. If not, see <http://www.gnu.org/licenses/>.
Note these built-in SQLAlchemy types: http://docs.sqlalchemy.org/en/latest/core/type_basics.html#generic-types
SQLAlchemy type Comment BigInteger MySQL: -9,223,372,036,854,775,808 to 9,223,372,036,854,775,807 (64-bit) (compare NHS number: up to 9,999,999,999) Boolean Date DateTime Enum Float Integer MySQL: -2,147,483,648 to 2,147,483,647 (32-bit) Interval for datetime.timedelta LargeBinary under MySQL, maps to BLOB MatchType for the return type of the MATCH operator Numeric for fixed-precision numbers like NUMERIC or DECIMAL PickleType SchemaType SmallInteger String VARCHAR Text variably sized string type … under MySQL, renders as TEXT Time Unicode implies that the underlying column explicitly supports Unicode UnicodeText variably sized version of Unicode … under MySQL, renders as TEXT too
Not supported across all platforms:
SQL type Comment BIGINT UNSIGNED MySQL: 0 to 18,446,744,073,709,551,615 (64-bit) … use sqlalchemy.dialects.mysql.BIGINT(unsigned=True) INT UNSIGNED MySQL: 0 to 4,294,967,295 (32-bit) … use sqlalchemy.dialects.mysql.INTEGER(unsigned=True)
Other MySQL sizes:
MySQL type Comment TINYBLOB 2^8 bytes = 256 bytes BLOB 2^16 bytes = 64 KiB MEDIUMBLOB 2^24 bytes = 16 MiB LONGBLOB 2^32 bytes = 4 GiB TINYTEXT 255 (2^8 - 1) bytes TEXT 65,535 bytes (2^16 - 1) = 64 KiB MEDIUMTEXT 16,777,215 (2^24 - 1) bytes = 16 MiB LONGTEXT 4,294,967,295 (2^32 - 1) bytes = 4 GiB
Also:
- Columns may need their character set specified explicitly under MySQL: https://stackoverflow.com/questions/2108824/mysql-incorrect-string-value-error-when-save-unicode-string-in-django
-
class
camcops_server.cc_modules.cc_sqla_coltypes.
CamcopsColumn
(*args, cris_include: bool = False, exempt_from_anonymisation: bool = False, identifies_patient: bool = False, is_blob_id_field: bool = False, blob_relationship_attr_name: str = '', permitted_value_checker: camcops_server.cc_modules.cc_sqla_coltypes.PermittedValueChecker = None, **kwargs)[source]¶ A Column class that supports some CamCOPS-specific flags, such as whether a field is a BLOB reference; how it should be treated for anonymisation; and which values are permitted in the field (in a soft sense: duff values cause errors to be reported, but they’re still stored).
-
class
camcops_server.cc_modules.cc_sqla_coltypes.
IdNumReferenceListColType
(*args, **kwargs)[source]¶ Stores a list of IdNumReference objects. On the database side, uses a comma-separated list of integers.
-
process_bind_param
(value: Union[List[camcops_server.cc_modules.cc_simpleobjects.IdNumReference], NoneType], dialect: sqlalchemy.engine.interfaces.Dialect) → str[source]¶ Convert things on the way from Python to the database.
-
process_literal_param
(value: Union[List[camcops_server.cc_modules.cc_simpleobjects.IdNumReference], NoneType], dialect: sqlalchemy.engine.interfaces.Dialect) → str[source]¶ Convert things on the way from Python to the database.
-
process_result_value
(value: Union[str, NoneType], dialect: sqlalchemy.engine.interfaces.Dialect) → List[camcops_server.cc_modules.cc_simpleobjects.IdNumReference][source]¶ Convert things on the way from the database to Python.
-
python_type
¶ Return the Python type object expected to be returned by instances of this type, if known.
Basically, for those types which enforce a return type, or are known across the board to do such for all common DBAPIs (like
int
for example), will return that type.If a return type is not defined, raises
NotImplementedError
.Note that any type also accommodates NULL in SQL which means you can also get back
None
from any type in practice.
-
-
class
camcops_server.cc_modules.cc_sqla_coltypes.
PendulumDateTimeAsIsoTextColType
(*args, **kwargs)[source]¶ Stores date/time values as ISO-8601, in a specific format. Uses Pendulum on the Python side.
-
class
comparator_factory
(expr)[source]¶ Process SQL for when we are comparing our column, in the database, to something else.
We make this dialect-independent by calling functions like
unknown_field_to_utcdatetime isotzdatetime_to_utcdatetime
… which we then specialize for specific dialects.
-
operate
(op, *other, **kwargs)[source]¶ Operate on an argument.
This is the lowest level of operation, raises
NotImplementedError
by default.Overriding this on a subclass can allow common behavior to be applied to all operations. For example, overriding
ColumnOperators
to applyfunc.lower()
to the left and right side:class MyComparator(ColumnOperators): def operate(self, op, other): return op(func.lower(self), func.lower(other))
Parameters: - op – Operator callable.
- *other – the ‘other’ side of the operation. Will be a single scalar for most operations.
- **kwargs – modifiers. These may be passed by special
operators such as
ColumnOperators.contains()
.
-
-
static
isostring_to_pendulum
(x: Union[str, NoneType]) → Union[pendulum.datetime.DateTime, NoneType][source]¶ From an ISO-formatted string to a Python Pendulum, with timezone.
-
static
pendulum_to_isostring
(x: Union[NoneType, datetime.date, str, arrow.arrow.Arrow]) → Union[str, NoneType][source]¶ From a Python datetime to an ISO-formatted string in our particular format.
-
process_bind_param
(value: Union[pendulum.datetime.DateTime, NoneType], dialect: sqlalchemy.engine.interfaces.Dialect) → Union[str, NoneType][source]¶ Convert things on the way from Python to the database.
-
process_literal_param
(value: Union[pendulum.datetime.DateTime, NoneType], dialect: sqlalchemy.engine.interfaces.Dialect) → Union[str, NoneType][source]¶ Convert things on the way from Python to the database.
-
process_result_value
(value: Union[str, NoneType], dialect: sqlalchemy.engine.interfaces.Dialect) → Union[pendulum.datetime.DateTime, NoneType][source]¶ Convert things on the way from the database to Python.
-
python_type
¶ Return the Python type object expected to be returned by instances of this type, if known.
Basically, for those types which enforce a return type, or are known across the board to do such for all common DBAPIs (like
int
for example), will return that type.If a return type is not defined, raises
NotImplementedError
.Note that any type also accommodates NULL in SQL which means you can also get back
None
from any type in practice.
-
class
-
class
camcops_server.cc_modules.cc_sqla_coltypes.
PermittedValueChecker
(not_null: bool = False, minimum: Union[int, float] = None, maximum: Union[int, float] = None, permitted_values: List[Any] = None)[source]¶ Represents permitted values, and checks a value against them.
-
class
camcops_server.cc_modules.cc_sqla_coltypes.
RelationshipInfo
[source]¶ Used for the “info” parameter to SQLAlchemy “relationship” calls.
-
class
camcops_server.cc_modules.cc_sqla_coltypes.
SemanticVersionColType
(*args, **kwargs)[source]¶ Stores semantic versions in the database. Uses semantic_version.Version on the Python side.
-
class
comparator_factory
(expr)[source]¶ Process SQL for when we are comparing our column, in the database, to something else.
-
operate
(op, *other, **kwargs)[source]¶ Operate on an argument.
This is the lowest level of operation, raises
NotImplementedError
by default.Overriding this on a subclass can allow common behavior to be applied to all operations. For example, overriding
ColumnOperators
to applyfunc.lower()
to the left and right side:class MyComparator(ColumnOperators): def operate(self, op, other): return op(func.lower(self), func.lower(other))
Parameters: - op – Operator callable.
- *other – the ‘other’ side of the operation. Will be a single scalar for most operations.
- **kwargs – modifiers. These may be passed by special
operators such as
ColumnOperators.contains()
.
-
-
process_bind_param
(value: Union[semantic_version.base.Version, NoneType], dialect: sqlalchemy.engine.interfaces.Dialect) → Union[str, NoneType][source]¶ Convert things on the way from Python to the database.
-
process_literal_param
(value: Union[semantic_version.base.Version, NoneType], dialect: sqlalchemy.engine.interfaces.Dialect) → Union[str, NoneType][source]¶ Convert things on the way from Python to the database.
-
process_result_value
(value: Union[str, NoneType], dialect: sqlalchemy.engine.interfaces.Dialect) → Union[semantic_version.base.Version, NoneType][source]¶ Convert things on the way from the database to Python.
-
python_type
¶ Return the Python type object expected to be returned by instances of this type, if known.
Basically, for those types which enforce a return type, or are known across the board to do such for all common DBAPIs (like
int
for example), will return that type.If a return type is not defined, raises
NotImplementedError
.Note that any type also accommodates NULL in SQL which means you can also get back
None
from any type in practice.
-
class
-
camcops_server.cc_modules.cc_sqla_coltypes.
gen_ancillary_relationships
(obj) → Generator[[Tuple[str, sqlalchemy.orm.relationships.RelationshipProperty, Type[GenericTabletRecordMixin]], NoneType], NoneType][source]¶ Yields tuples of
(attrname, RelationshipProperty, related_class)
for all relationships that are marked as a CamCOPS ancillary relationship.
-
camcops_server.cc_modules.cc_sqla_coltypes.
gen_blob_relationships
(obj) → Generator[[Tuple[str, sqlalchemy.orm.relationships.RelationshipProperty, Type[GenericTabletRecordMixin]], NoneType], NoneType][source]¶ Yields tuples of
(attrname, RelationshipProperty, related_class)
for all relationships that are marked as a CamCOPS BLOB relationship.
-
camcops_server.cc_modules.cc_sqla_coltypes.
gen_camcops_blob_columns
(obj) → Generator[[Tuple[str, camcops_server.cc_modules.cc_sqla_coltypes.CamcopsColumn], NoneType], NoneType][source]¶ Yields tuples of (attr_name, CamcopsColumn) from a class where those CamcopsColumns are for fields referencing the BLOB table.
-
camcops_server.cc_modules.cc_sqla_coltypes.
gen_camcops_columns
(obj) → Generator[[Tuple[str, camcops_server.cc_modules.cc_sqla_coltypes.CamcopsColumn], NoneType], NoneType][source]¶ Yields tuples of (attr_name, CamcopsColumn) from an object.
-
class
camcops_server.cc_modules.cc_sqla_coltypes.
isotzdatetime_to_utcdatetime
(*clauses, **kwargs)[source]¶ Used as an SQL operation by PendulumDateTimeAsIsoTextColType.
Creates an SQL expression wrapping a field containing our ISO-8601 text, making a DATETIME out of it, in the UTC timezone.
Implemented for different SQL dialects.
-
camcops_server.cc_modules.cc_sqla_coltypes.
isotzdatetime_to_utcdatetime_default
(element: ClauseElement, compiler: SQLCompiler, **kw) → None[source]¶ Default implementation for isotzdatetime_to_utcdatetime: fail.
-
camcops_server.cc_modules.cc_sqla_coltypes.
isotzdatetime_to_utcdatetime_mysql
(element: ClauseElement, compiler: SQLCompiler, **kw) → str[source]¶ Implementation of isotzdatetime_to_utcdatetime for MySQL.
For format, see https://dev.mysql.com/doc/refman/5.5/en/date-and-time-functions.html#function_date-format
Note the use of “%i” for minutes.
Things after “func.” get passed to the database engine as literal SQL functions; http://docs.sqlalchemy.org/en/latest/core/tutorial.html
-
camcops_server.cc_modules.cc_sqla_coltypes.
isotzdatetime_to_utcdatetime_sqlite
(element: ClauseElement, compiler: SQLCompiler, **kw) → str[source]¶ Implementation of isotzdatetime_to_utcdatetime for SQLite.
-
camcops_server.cc_modules.cc_sqla_coltypes.
isotzdatetime_to_utcdatetime_sqlserver
(element: ClauseElement, compiler: SQLCompiler, **kw) → str[source]¶ Implementation of isotzdatetime_to_utcdatetime for SQL Server.
Converting strings to DATETIME values
- CAST(): Part of ANSI SQL.
- CONVERT(): Not part of ANSI SQL; has some extra formatting options.
Both methods work:
SELECT CAST('2001-01-31T21:30:49.123' AS DATETIME) AS via_cast, CONVERT(DATETIME, '2001-01-31T21:30:49.123') AS via_convert;
… fine on SQL Server 2005, with milliseconds in both cases. However, going beyond milliseconds doesn’t fail gracefully, it causes an error (e.g. “…21:30.49.123456”) both for CAST and CONVERT.
The DATETIME2 format accepts greater precision, but requires SQL Server 2008 or higher. Then this works:
SELECT CAST('2001-01-31T21:30:49.123456' AS DATETIME2) AS via_cast, CONVERT(DATETIME2, '2001-01-31T21:30:49.123456') AS via_convert;
So as not to be too optimistic: CAST(x AS DATETIME2) ignores (silently) any timezone information in the string. So does CONVERT(DATETIME2, x, {0 or 1}).
Converting between time zones
NO TIME ZONE SUPPORT in SQL Server 2005. e.g. https://stackoverflow.com/questions/3200827/how-to-convert-timezones-in-sql-server-2005 # noqa
TODATETIMEOFFSET(expression, time_zone): expression: something that evaluates to a DATETIME2 value time_zone: integer minutes, or string hours/minutes e.g. "+13.00" -> produces a DATETIMEOFFSET value
Available from SQL Server 2008. https://docs.microsoft.com/en-us/sql/t-sql/functions/todatetimeoffset-transact-sql # noqa
SWITCHOFFSET -> converts one DATETIMEOFFSET value to another, preserving its UTC time, but changing the displayed (local) time zone.
… however, is that unnecessary? We want a plain
DATETIME2
in UTC, and .conversion to UTC is automatically achieved byCONVERT(DATETIME2, .some_datetimeoffset, 1)
… but not by
CAST(some_datetimeoffset AS DATETIME2)
, and not byCONVERT(DATETIME2, some_datetimeoffset, 0)
… and styles 0 and 1 are the only ones permissible from SQL Server 2012 and up, empirically and (documented for the reverse direction at: https://docs.microsoft.com/en-us/sql/t-sql/functions/cast-and-convert-transact-sql?view=sql-server-2017 # noqa
… this is not properly documented re UTC conversion, as far as I can see. Let’s use SWITCHOFFSET -> CAST to be explicit and clear.
AT TIME ZONE: From SQL Server 2016 only. https://docs.microsoft.com/en-us/sql/t-sql/queries/at-time-zone-transact-sql?view=sql-server-2017
Therefore
- We need to require SQL Server 2008 or higher.
- Therefore we can use the DATETIME2 type.
- Note that LEN(), not LENGTH(), is ANSI SQL; SQL Server only supports LEN.
Example (tested on SQL Server 2014)
DECLARE @source AS VARCHAR(100) = '2001-01-31T21:30:49.123456+07:00'; SELECT CAST( SWITCHOFFSET( TODATETIMEOFFSET( CAST(LEFT(@source, LEN(@source) - 6) AS DATETIME2), RIGHT(@source, 6) ), '+00:00' ) AS DATETIME2 ) -- 2001-01-31 14:30:49.1234560
-
camcops_server.cc_modules.cc_sqla_coltypes.
permitted_value_failure_msgs
(obj) → List[str][source]¶ Checks a SQLAlchemy ORM object instance against its PermittedValueColumn checks, if it has any. Returns a list of failure messages (empty list means all OK).
-
camcops_server.cc_modules.cc_sqla_coltypes.
permitted_values_ok
(obj) → bool[source]¶ Faster way of checking WHETHER an instance passes its PermittedValueColumn checks, if it has any.
-
class
camcops_server.cc_modules.cc_sqla_coltypes.
unknown_field_to_utcdatetime
(*clauses, **kwargs)[source]¶ Used as an SQL operation by PendulumDateTimeAsIsoTextColType.
Creates an SQL expression wrapping a field containing something unknown, which might be a DATETIME or an ISO-formatted field, and making a DATETIME out of it, in the UTC timezone.
Implemented for different SQL dialects.
-
camcops_server.cc_modules.cc_sqla_coltypes.
unknown_field_to_utcdatetime_default
(element: ClauseElement, compiler: SQLCompiler, **kw) → None[source]¶ Default implementation for unknown_field_to_utcdatetime: fail.
-
camcops_server.cc_modules.cc_sqla_coltypes.
unknown_field_to_utcdatetime_mysql
(element: ClauseElement, compiler: SQLCompiler, **kw) → str[source]¶ Implementation of unknown_field_to_utcdatetime for MySQL.
If it’s the length of a plain DATETIME e.g. “2013-05-30 00:00:00” (19), leave it as a DATETIME; otherwise convert ISO -> DATETIME log.critical(result)
-
camcops_server.cc_modules.cc_sqla_coltypes.
unknown_field_to_utcdatetime_sqlite
(element: ClauseElement, compiler: SQLCompiler, **kw) → str[source]¶ Implementation of unknown_field_to_utcdatetime for SQLite.
-
camcops_server.cc_modules.cc_sqla_coltypes.
unknown_field_to_utcdatetime_sqlserver
(element: ClauseElement, compiler: SQLCompiler, **kw) → str[source]¶ Implementation of unknown_field_to_utcdatetime for SQL Server.
We should cope also with the possibility of a DATETIME2 field, not just DATETIME. It seems consistent that LEN(DATETIME2) = 27, with precision tenth of a microsecond, e.g.
2001-01-31 21:30:49.1234567
(27).So, if it looks like a DATETIME or a DATETIME2, then we leave it alone; otherwise we put it through our ISO-to-datetime function.
Importantly, note that neither _SQLSERVER_DATETIME_LEN nor _SQLSERVER_DATETIME2_LEN are the length of any of our ISO strings.