Coverage for cc_modules/client_api.py: 18%
927 statements
« prev ^ index » next coverage.py v6.5.0, created at 2022-11-08 23:14 +0000
« prev ^ index » next coverage.py v6.5.0, created at 2022-11-08 23:14 +0000
1#!/usr/bin/env python
3"""
4camcops_server/cc_modules/client_api.py
6===============================================================================
8 Copyright (C) 2012, University of Cambridge, Department of Psychiatry.
9 Created by Rudolf Cardinal (rnc1001@cam.ac.uk).
11 This file is part of CamCOPS.
13 CamCOPS is free software: you can redistribute it and/or modify
14 it under the terms of the GNU General Public License as published by
15 the Free Software Foundation, either version 3 of the License, or
16 (at your option) any later version.
18 CamCOPS is distributed in the hope that it will be useful,
19 but WITHOUT ANY WARRANTY; without even the implied warranty of
20 MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
21 GNU General Public License for more details.
23 You should have received a copy of the GNU General Public License
24 along with CamCOPS. If not, see <https://www.gnu.org/licenses/>.
26===============================================================================
28**Implements the API through which client devices (tablets etc.) upload and
29download data.**
31We use primarily SQLAlchemy Core here (in contrast to the ORM used elsewhere).
33This code is optimized to a degree for speed over clarity, aiming primarily to
34reduce the number of database hits.
36**The overall upload method is as follows**
38Everything that follows refers to records relating to a specific client device
39in the "current" era, only.
41In the preamble, the client:
43- verifies authorization via :func:`op_check_device_registered` and
44 :func:`op_check_upload_user_and_device`;
45- fetches and checks server ID information via :func:`op_get_id_info`;
46- checks its patients are acceptable via :func:`op_validate_patients`;
47- checks which tables are permitted via :func:`op_get_allowed_tables`;
48- performs some internal validity checks.
50Then, in the usual stepwise upload:
52- :func:`op_start_upload`
54 - Rolls back any previous incomplete changes via :func:`rollback_all`.
55 - Creates an upload batch, via :func:`get_batch_details_start_if_needed`.
57- If were are in a preserving/finalizing upload: :func:`op_start_preservation`.
59 - Marks all tables as dirty.
60 - Marks the upload batch as a "preserving" batch.
62- Then call some or all of:
64 - For tables that are empty on the client, :func:`op_upload_empty_tables`.
66 - Current client records are marked as ``_removal_pending``.
67 - Any table that had previous client records is marked as dirty.
68 - If preserving, any table without current records is marked as clean.
70 - For tables that the client wishes to send in one go,
71 :func:`op_upload_table`.
73 - Find current server records.
74 - Use :func:`upload_record_core` to add new records and modify existing
75 ones, and :func:`flag_deleted` to delete ones that weren't on the client.
76 - If any records are new, modified, or deleted, mark the table as dirty.
77 - If preserving and there were no server records in this table, mark the
78 table as clean.
80 - For tables (e.g. BLOBs) that might be too big to send in one go:
82 - client sends PKs to :func:`op_delete_where_key_not`, which "deletes" all
83 other records, via :func:`flag_deleted_where_clientpk_not`.
84 - client sends PK and timestamp values to :func:`op_which_keys_to_send`
85 - server "deletes" records that are not in the list (via
86 :func:`flag_deleted_where_clientpk_not`, which marks the table as dirty
87 if any records were thus modified). Note REDUNDANCY here re
88 :func:`op_delete_where_key_not`.
89 - server tells the client which records are new or need to be updated
90 - client sends each of those via :func:`op_upload_record`
92 - Calls :func`upload_record_core`.
93 - Marks the table as dirty, unless the client erroneously sent an
94 unchanged record.
96- In addition, specific records can be marked as ``_move_off_tablet``.
98 - :func:`upload_record_core` checks this for otherwise "identical" records
99 and applies that flag to the server.
101- When the client's finished, it calls :func:`op_end_upload`.
103 - Calls :func:`commit_all`;
104 - ... which, for all dirty tables, calls :func:`commit_table`;
105 - ... which executes the "add", "remove", and "preserve" functions for the
106 table;
107 - ... and triggers the updating of special server indexes on patient ID
108 numbers and tasks, via :func:`update_indexes`.
109 - At the end, :func:`commit_all` clears the dirty-table list.
111There's a little bit of special code to handle old tablet software, too.
113As of v2.3.0, the function :func:`op_upload_entire_database` does this in one
114step (faster; for use if the network packets are not excessively large).
116- Code relating to this uses ``batchdetails.onestep``.
118**Setup for the upload code**
120- Fire up a CamCOPS client with an empty database, e.g. from the build
121 directory via
123 .. code-block:: bash
125 ./camcops --dbdir ~/tmp/camcops_client_test
127- Fire up a web browser showing both (a) the task list via the index, and (b)
128 the task list without using the index. We'll use this to verify correct
129 indexing. **The two versions of the view should never be different.**
131- Ensure the test client device has no current records (force-finalize if
132 required).
134- Ensure the server's index is proper. Run ``camcops_server reindex`` if
135 required.
137- If required, fire up MySQL with the server database. You may wish to use
138 ``pager less -SFX``, for better display of large tables.
140**Testing the upload code**
142Perform the following steps both (1) with the client forced to the stepwise
143upload method, and (2) with it forced to one-step upload.
145Note that the number of patient ID numbers uploaded (etc.) is ignored below.
147*Single record*
149[Checked for one-step and multi-step upload, 2018-11-21.]
151#. Create a blank ReferrerSatisfactionSurvey (table ``ref_satis_gen``).
152 This has the advantage of being an anonymous single-record task.
154#. Upload/copy.
156 - The server log should show 1 × ref_satis_gen added.
158 - The task lists should show the task as current and incomplete.
160#. Modify it, so it's complete.
162#. Upload/copy.
164 - The server log should show 1 × ref_satis_gen added, 1 × ref_satis_gen
165 modified out.
167 - The task lists should show the task as current and complete.
169#. Upload/move.
171 - The server log should show 2 × ref_satis_gen preserved.
173 - The task lists should show the task as no longer current.
175#. Create another blank one.
177#. Upload/copy.
179#. Modify it so it's complete.
181#. Specifically flag it for preservation (the chequered flags).
183#. Upload/copy.
185 - The server log should show 1 × ref_satis_gen added, 1 × ref_satis_gen
186 modified out, 2 × ref_satis_gen preserved.
188 - The task lists should show the task as complete and no longer current.
190*With a patient*
192[Checked for one-step and multi-step upload, 2018-11-21.]
194#. Create a dummy patient that the server will accept.
196#. Create a Progress Note with location "loc1" and abort its creation, giving
197 an incomplete task.
199#. Create a second Progress Note with location "loc2" and contents "note2".
201#. Create a third Progress Note with location "loc3" and contents "note3".
203#. Upload/copy. Verify. This checks *addition*.
205 - The server log should show 1 × patient added; 3 × progressnote added.
206 (Also however many patientidnum records you chose.)
207 - All three tasks should be "current".
208 - The first should be "incomplete".
210#. Modify the first note by adding contents "note1".
212#. Delete the second note.
214#. Upload/copy. Verify. This checks *modification*, *deletion*,
215 *no-change detection*, and *reindexing*.
217 - The server log should show 1 × progressnote added, 1 × progressnote
218 modified out, 1 × progressnote deleted.
219 - The first note should now appear as complete.
220 - The second should have vanished.
221 - The third should be unchanged.
222 - The two remaining tasks should still be "current".
224#. Delete the contents from the first note again.
226#. Upload/move (or move-keeping-patients; that's only different on the
227 client side). Verify. This checks *preservation (finalizing)* and
228 *reindexing*.
230 - The server log should show 1 × patient preserved; 1 × progressnote added,
231 1 × progressnote modified out, 5 × progressnote preserved.
232 - The two remaining tasks should no longer be "current".
233 - The first should no longer be "complete".
235#. Create a complete "note 4" and an incomplete "note 5".
237#. Upload/copy.
239#. Force-finalize from the server. This tests force-finalizing including
240 reindexing.
242 - The "tasks to finalize" list should have just two tasks in it.
243 - After force-finalizing, the tasks should remain in the index but no
244 longer be marked as current.
246#. Upload/move to get rid of the residual tasks on the client.
248 - The server log should show 1 × patient added, 1 × patient preserved; 2 ×
249 progressnote added, 2 × progressnote preserved.
251*With ancillary tables and BLOBs*
253[Checked for one-step and multi-step upload, 2018-11-21.]
255#. Create a PhotoSequence with text "t1", one photo named "p1" of you holding
256 up one finger vertically, and another photo named "p2" of you holding up
257 two fingers vertically.
259#. Upload/copy.
261 - The server log should show:
263 - blobs: 2 × added;
264 - patient: 1 × added;
265 - photosequence: 1 × added;
266 - photosequence_photos: 2 × added.
268 - The task lists should look sensible.
270#. Clear the second photo and replace it with a photo of you holding up
271 two fingers horizontally.
273#. Upload/copy.
275 - The server log should show:
277 - blobs: 1 × added, 1 × modified out;
278 - photosequence: 1 × added, 1 × modified out;
279 - photosequence_photos: 1 × added, 1 × modified out.
281 - The task lists should look sensible.
283#. Back to two fingers vertically. (This is the fourth photo overall.)
285#. Mark that patient for specific finalization.
287#. Upload/copy.
289 - The server log should show:
291 - blobs: 1 × added, 1 × modified out, 4 × preserved;
292 - patient: 1 × preserved;
293 - photosequence: 1 × added, 1 × modified out, 3 × preserved;
294 - photosequence_photos: 1 × added, 1 × modified out, 4 × preserved.
296 - The tasks should no longer be current.
297 - A fresh "vertical fingers" photo should be visible.
299#. Create another patient and another PhotoSequence with one photo of three
300 fingers.
302#. Upload-copy.
304#. Force-finalize.
306 - Should finalize: 1 × blobs, 1 × patient, 1 × photosequence, 1 ×
307 photosequence_photos.
309#. Upload/move.
311During any MySQL debugging, remember:
313.. code-block:: none
315 -- For better display:
316 pager less -SFX;
318 -- To view relevant parts of the BLOB table without the actual BLOB:
320 SELECT
321 _pk, _group_id, _device_id, _era,
322 _current, _predecessor_pk, _successor_pk,
323 _addition_pending, _when_added_batch_utc, _adding_user_id,
324 _removal_pending, _when_removed_batch_utc, _removing_user_id,
325 _move_off_tablet,
326 _preserving_user_id, _forcibly_preserved,
327 id, tablename, tablepk, fieldname, mimetype, when_last_modified
328 FROM blobs;
330"""
332# =============================================================================
333# Imports
334# =============================================================================
336import logging
337import json
339# from pprint import pformat
340import secrets
341import string
342import time
343from typing import (
344 Any,
345 Dict,
346 Iterable,
347 List,
348 Optional,
349 Sequence,
350 Set,
351 Tuple,
352 TYPE_CHECKING,
353)
354from cardinal_pythonlib.datetimefunc import (
355 coerce_to_pendulum,
356 coerce_to_pendulum_date,
357 format_datetime,
358)
359from cardinal_pythonlib.httpconst import HttpMethod
360from cardinal_pythonlib.logs import BraceStyleAdapter
361from cardinal_pythonlib.pyramid.responses import TextResponse
362from cardinal_pythonlib.sqlalchemy.core_query import (
363 exists_in_table,
364 fetch_all_first_values,
365)
366from cardinal_pythonlib.text import escape_newlines
367from pyramid.httpexceptions import HTTPBadRequest
368from pyramid.view import view_config
369from pyramid.response import Response
370from pyramid.security import NO_PERMISSION_REQUIRED
371from semantic_version import Version
372from sqlalchemy.engine.result import ResultProxy
373from sqlalchemy.exc import IntegrityError
374from sqlalchemy.orm import joinedload
375from sqlalchemy.sql.expression import exists, select, update
376from sqlalchemy.sql.schema import Table
378from camcops_server.cc_modules import cc_audit # avoids "audit" name clash
379from camcops_server.cc_modules.cc_all_models import CLIENT_TABLE_MAP
380from camcops_server.cc_modules.cc_blob import Blob
381from camcops_server.cc_modules.cc_cache import cache_region_static, fkg
382from camcops_server.cc_modules.cc_client_api_core import (
383 AllowedTablesFieldNames,
384 BatchDetails,
385 exception_description,
386 ExtraStringFieldNames,
387 fail_server_error,
388 fail_unsupported_operation,
389 fail_user_error,
390 get_server_live_records,
391 IgnoringAntiqueTableException,
392 require_keys,
393 ServerErrorException,
394 ServerRecord,
395 TabletParam,
396 UploadRecordResult,
397 UploadTableChanges,
398 UserErrorException,
399 values_delete_later,
400 values_delete_now,
401 values_preserve_now,
402 WhichKeyToSendInfo,
403)
404from camcops_server.cc_modules.cc_client_api_helpers import (
405 upload_commit_order_sorter,
406)
407from camcops_server.cc_modules.cc_constants import (
408 CLIENT_DATE_FIELD,
409 DateFormat,
410 ERA_NOW,
411 FP_ID_NUM,
412 FP_ID_DESC,
413 FP_ID_SHORT_DESC,
414 MOVE_OFF_TABLET_FIELD,
415 NUMBER_OF_IDNUMS_DEFUNCT, # allowed; for old tablet versions
416 POSSIBLE_SEX_VALUES,
417 TABLET_ID_FIELD,
418)
419from camcops_server.cc_modules.cc_convert import (
420 decode_single_value,
421 decode_values,
422 encode_single_value,
423)
424from camcops_server.cc_modules.cc_db import (
425 FN_ADDING_USER_ID,
426 FN_ADDITION_PENDING,
427 FN_CAMCOPS_VERSION,
428 FN_CURRENT,
429 FN_DEVICE_ID,
430 FN_ERA,
431 FN_GROUP_ID,
432 FN_PK,
433 FN_PREDECESSOR_PK,
434 FN_REMOVAL_PENDING,
435 FN_REMOVING_USER_ID,
436 FN_SUCCESSOR_PK,
437 FN_WHEN_ADDED_BATCH_UTC,
438 FN_WHEN_ADDED_EXACT,
439 FN_WHEN_REMOVED_BATCH_UTC,
440 FN_WHEN_REMOVED_EXACT,
441 RESERVED_FIELDS,
442)
443from camcops_server.cc_modules.cc_device import Device
444from camcops_server.cc_modules.cc_dirtytables import DirtyTable
445from camcops_server.cc_modules.cc_group import Group
446from camcops_server.cc_modules.cc_ipuse import IpUse
447from camcops_server.cc_modules.cc_membership import UserGroupMembership
448from camcops_server.cc_modules.cc_patient import (
449 Patient,
450 is_candidate_patient_valid_for_group,
451 is_candidate_patient_valid_for_restricted_user,
452)
453from camcops_server.cc_modules.cc_patientidnum import (
454 fake_tablet_id_for_patientidnum,
455 PatientIdNum,
456)
457from camcops_server.cc_modules.cc_proquint import (
458 InvalidProquintException,
459 uuid_from_proquint,
460)
461from camcops_server.cc_modules.cc_pyramid import Routes
462from camcops_server.cc_modules.cc_simpleobjects import (
463 BarePatientInfo,
464 IdNumReference,
465)
466from camcops_server.cc_modules.cc_specialnote import SpecialNote
467from camcops_server.cc_modules.cc_task import (
468 all_task_tables_with_min_client_version,
469)
470from camcops_server.cc_modules.cc_taskindex import (
471 update_indexes_and_push_exports,
472)
473from camcops_server.cc_modules.cc_user import User
474from camcops_server.cc_modules.cc_validators import (
475 STRING_VALIDATOR_TYPE,
476 validate_anything,
477 validate_email,
478)
479from camcops_server.cc_modules.cc_version import (
480 CAMCOPS_SERVER_VERSION_STRING,
481 MINIMUM_TABLET_VERSION,
482)
484if TYPE_CHECKING:
485 from camcops_server.cc_modules.cc_request import CamcopsRequest
487log = BraceStyleAdapter(logging.getLogger(__name__))
490# =============================================================================
491# Constants
492# =============================================================================
494COPE_WITH_DELETED_PATIENT_DESCRIPTIONS = True
495# ... as of client 2.0.0, ID descriptions are no longer duplicated.
496# As of server 2.0.0, the fields still exist in the database, but the reporting
497# and consistency check has been removed. In the next version of the server,
498# the fields will be removed, and then the server should cope with old clients,
499# at least for a while.
501DUPLICATE_FAILED = "Failed to duplicate record"
502INSERT_FAILED = "Failed to insert record"
504# REGEX_INVALID_TABLE_FIELD_CHARS = re.compile("[^a-zA-Z0-9_]")
505# ... the ^ within the [] means the expression will match any character NOT in
506# the specified range
508DEVICE_STORED_VAR_TABLENAME_DEFUNCT = "storedvars"
509# ... old table, no longer in use, that Titanium clients used to upload.
510# We recognize and ignore it now so that old clients can still work.
512SILENTLY_IGNORE_TABLENAMES = [DEVICE_STORED_VAR_TABLENAME_DEFUNCT]
514IGNORING_ANTIQUE_TABLE_MESSAGE = (
515 "Ignoring user request to upload antique/defunct table, but reporting "
516 "success to the client"
517)
519SUCCESS_MSG = "Success"
520SUCCESS_CODE = "1"
521FAILURE_CODE = "0"
523DEBUG_UPLOAD = False
526# =============================================================================
527# Quasi-constants
528# =============================================================================
530DB_JSON_DECODER = json.JSONDecoder() # just a plain one
531PATIENT_INFO_JSON_DECODER = json.JSONDecoder() # just a plain one
534# =============================================================================
535# Cached information
536# =============================================================================
539@cache_region_static.cache_on_arguments(function_key_generator=fkg)
540def all_tables_with_min_client_version() -> Dict[str, Version]:
541 """
542 For all tables that the client might upload to, return a mapping from the
543 table name to the corresponding minimum client version.
544 """
545 d = all_task_tables_with_min_client_version()
546 d[Blob.__tablename__] = MINIMUM_TABLET_VERSION
547 d[Patient.__tablename__] = MINIMUM_TABLET_VERSION
548 d[PatientIdNum.__tablename__] = MINIMUM_TABLET_VERSION
549 return d
552# =============================================================================
553# Validators
554# =============================================================================
557def ensure_valid_table_name(req: "CamcopsRequest", tablename: str) -> None:
558 """
559 Ensures a table name:
561 - doesn't contain bad characters,
562 - isn't a reserved table that the user is prohibited from accessing, and
563 - is a valid table name that's in the database.
565 Raises :exc:`UserErrorException` upon failure.
567 - 2017-10-08: shortcut to all that: it's OK if it's listed as a valid
568 client table.
569 - 2018-01-16 (v2.2.0): check also that client version is OK
570 """
571 if tablename not in CLIENT_TABLE_MAP:
572 fail_user_error(f"Invalid client table name: {tablename}")
573 tables_versions = all_tables_with_min_client_version()
574 assert tablename in tables_versions
575 client_version = req.tabletsession.tablet_version_ver
576 minimum_client_version = tables_versions[tablename]
577 if client_version < minimum_client_version:
578 fail_user_error(
579 f"Client CamCOPS version {client_version} is less than the "
580 f"version ({minimum_client_version}) "
581 f"required to handle table {tablename}"
582 )
585def ensure_valid_field_name(table: Table, fieldname: str) -> None:
586 """
587 Ensures a field name contains only valid characters, and isn't a
588 reserved fieldname that the user isn't allowed to access.
590 Raises :exc:`UserErrorException` upon failure.
592 - 2017-10-08: shortcut: it's OK if it's a column name for a particular
593 table.
594 """
595 if fieldname.startswith("_"): # all reserved fields start with _
596 # ... but not all fields starting with "_" are reserved; e.g.
597 # "_move_off_tablet" is allowed.
598 if fieldname in RESERVED_FIELDS:
599 fail_user_error(
600 f"Reserved field name for table {table.name!r}: {fieldname!r}"
601 )
602 if fieldname not in table.columns.keys():
603 fail_user_error(
604 f"Invalid field name for table {table.name!r}: {fieldname!r}"
605 )
606 # Note that the reserved-field check is case-sensitive, but so is the
607 # "present in table" check. So for a malicious uploader trying to use, for
608 # example, "_PK", this would not be picked up as a reserved field (so would
609 # pass that check) but then wouldn't be recognized as a valid field (so
610 # would fail).
613def ensure_string(value: Any, allow_none: bool = True) -> None:
614 """
615 Used when processing JSON information about patients: ensures that a value
616 is a string, or raises.
618 Args:
619 value: value to test
620 allow_none: is ``None`` allowed (not just an empty string)?
621 """
622 if value is None:
623 if allow_none:
624 return # OK
625 else:
626 fail_user_error("Patient JSON contains absent string")
627 if not isinstance(value, str):
628 fail_user_error(f"Patient JSON contains invalid non-string: {value!r}")
631def ensure_valid_patient_json(
632 req: "CamcopsRequest", group: Group, pt_dict: Dict[str, Any]
633) -> None:
634 """
635 Ensures that the JSON dictionary contains valid patient details (valid for
636 the group into which it's being uploaded), and that (if applicable) this
637 user is allowed to upload this patient.
639 Args:
640 req:
641 the :class:`camcops_server.cc_modules.cc_request.CamcopsRequest`
642 group:
643 the :class:`camcops_server.cc_modules.cc_group.Group` into which
644 the upload is going
645 pt_dict:
646 a JSON dictionary from the client
648 Raises:
649 :exc:`UserErrorException` if invalid
651 """
652 if not isinstance(pt_dict, dict):
653 fail_user_error("Patient JSON is not a dict")
654 if not pt_dict:
655 fail_user_error("Patient JSON is empty")
656 valid_which_idnums = req.valid_which_idnums
657 errors = [] # type: List[str]
658 finalizing = None
659 ptinfo = BarePatientInfo()
660 idnum_types_seen = set() # type: Set[int]
661 for k, v in pt_dict.items():
662 ensure_string(k, allow_none=False)
664 if k == TabletParam.FORENAME:
665 ensure_string(v)
666 ptinfo.forename = v
668 elif k == TabletParam.SURNAME:
669 ensure_string(v)
670 ptinfo.surname = v
672 elif k == TabletParam.SEX:
673 if v not in POSSIBLE_SEX_VALUES:
674 fail_user_error(f"Bad sex value: {v!r}")
675 ptinfo.sex = v
677 elif k == TabletParam.DOB:
678 ensure_string(v)
679 if v:
680 dob = coerce_to_pendulum_date(v)
681 if dob is None:
682 fail_user_error(f"Invalid DOB: {v!r}")
683 else:
684 dob = None
685 ptinfo.dob = dob
687 elif k == TabletParam.EMAIL:
688 ensure_string(v)
689 if v:
690 try:
691 validate_email(v)
692 except ValueError:
693 fail_user_error(f"Bad e-mail address: {v!r}")
694 ptinfo.email = v
696 elif k == TabletParam.ADDRESS:
697 ensure_string(v)
698 ptinfo.address = v
700 elif k == TabletParam.GP:
701 ensure_string(v)
702 ptinfo.gp = v
704 elif k == TabletParam.OTHER:
705 ensure_string(v)
706 ptinfo.otherdetails = v
708 elif k.startswith(TabletParam.IDNUM_PREFIX):
709 nstr = k[len(TabletParam.IDNUM_PREFIX) :] # noqa: E203
710 try:
711 which_idnum = int(nstr)
712 except (TypeError, ValueError):
713 fail_user_error(f"Bad idnum key: {k!r}")
714 # noinspection PyUnboundLocalVariable
715 if which_idnum not in valid_which_idnums:
716 fail_user_error(f"Bad ID number type: {which_idnum}")
717 if which_idnum in idnum_types_seen:
718 fail_user_error(
719 f"More than one ID number supplied for ID "
720 f"number type {which_idnum}"
721 )
722 idnum_types_seen.add(which_idnum)
723 if v is not None and not isinstance(v, int):
724 fail_user_error(f"Bad ID number value: {v!r}")
725 idref = IdNumReference(which_idnum, v)
726 if not idref.is_valid():
727 fail_user_error(f"Bad ID number: {idref!r}")
728 ptinfo.add_idnum(idref)
730 elif k == TabletParam.FINALIZING:
731 if not isinstance(v, bool):
732 fail_user_error(f"Bad {k!r} value: {v!r}")
733 finalizing = v
735 else:
736 fail_user_error(f"Unknown JSON key: {k!r}")
738 if finalizing is None:
739 fail_user_error(f"Missing {TabletParam.FINALIZING!r} JSON key")
741 pt_ok, reason = is_candidate_patient_valid_for_group(
742 ptinfo, group, finalizing
743 )
744 if not pt_ok:
745 errors.append(f"{ptinfo} -> {reason}")
746 pt_ok, reason = is_candidate_patient_valid_for_restricted_user(req, ptinfo)
747 if not pt_ok:
748 errors.append(f"{ptinfo} -> {reason}")
749 if errors:
750 fail_user_error(f"Invalid patient: {' // '.join(errors)}")
753# =============================================================================
754# Extracting information from the POST request
755# =============================================================================
758def get_str_var(
759 req: "CamcopsRequest",
760 var: str,
761 mandatory: bool = True,
762 validator: STRING_VALIDATOR_TYPE = validate_anything,
763) -> Optional[str]:
764 """
765 Retrieves a string variable from the CamcopsRequest.
767 By default this performs no validation (because, for example, these strings
768 can contain SQL-encoded data or JSON), but there are a number of subsequent
769 operation-specific validation steps.
771 Args:
772 req: the :class:`camcops_server.cc_modules.cc_request.CamcopsRequest`
773 var: name of variable to retrieve
774 mandatory: if ``True``, raise an exception if the variable is missing
775 validator: validator function to use
777 Returns:
778 value
780 Raises:
781 :exc:`UserErrorException` if the variable was mandatory and
782 no value was provided
783 """
784 try:
785 val = req.get_str_param(var, default=None, validator=validator)
786 if mandatory and val is None:
787 fail_user_error(f"Must provide the variable: {var}")
788 return val
789 except HTTPBadRequest as e: # failed the validator
790 fail_user_error(str(e))
793def get_int_var(req: "CamcopsRequest", var: str) -> int:
794 """
795 Retrieves an integer variable from the CamcopsRequest.
797 Args:
798 req: the :class:`camcops_server.cc_modules.cc_request.CamcopsRequest`
799 var: name of variable to retrieve
801 Returns:
802 value
804 Raises:
805 :exc:`UserErrorException` if no value was provided, or if it wasn't an
806 integer
807 """
808 s = get_str_var(req, var, mandatory=True)
809 try:
810 return int(s)
811 except (TypeError, ValueError):
812 fail_user_error(f"Variable {var} is not a valid integer; was {s!r}")
815def get_bool_int_var(req: "CamcopsRequest", var: str) -> bool:
816 """
817 Retrieves a Boolean variable (encoded as an integer) from the
818 CamcopsRequest. Zero represents false; nonzero represents true.
820 Args:
821 req: the :class:`camcops_server.cc_modules.cc_request.CamcopsRequest`
822 var: name of variable to retrieve
824 Returns:
825 value
827 Raises:
828 :exc:`UserErrorException` if no value was provided, or if it wasn't an
829 integer
830 """
831 num = get_int_var(req, var)
832 return bool(num)
835def get_table_from_req(req: "CamcopsRequest", var: str) -> Table:
836 """
837 Retrieves a table name from a HTTP request, checks it's a valid client
838 table, and returns that table.
840 Args:
841 req: the :class:`camcops_server.cc_modules.cc_request.CamcopsRequest`
842 var: variable name (the variable's should be the table name)
844 Returns:
845 a SQLAlchemy :class:`Table`
847 Raises:
848 :exc:`UserErrorException` if the variable wasn't provided
850 :exc:`IgnoringAntiqueTableException` if the table is one to
851 ignore quietly (requested by an antique client)
852 """
853 tablename = get_str_var(req, var, mandatory=True)
854 if tablename in SILENTLY_IGNORE_TABLENAMES:
855 raise IgnoringAntiqueTableException(f"Ignoring table {tablename}")
856 ensure_valid_table_name(req, tablename)
857 return CLIENT_TABLE_MAP[tablename]
860def get_tables_from_post_var(
861 req: "CamcopsRequest", var: str, mandatory: bool = True
862) -> List[Table]:
863 """
864 Gets a list of tables from an HTTP request variable, and ensures all are
865 valid.
867 Args:
868 req: the :class:`camcops_server.cc_modules.cc_request.CamcopsRequest`
869 var: name of variable to retrieve
870 mandatory: if ``True``, raise an exception if the variable is missing
872 Returns:
873 a list of SQLAlchemy :class:`Table` objects
875 Raises:
876 :exc:`UserErrorException` if the variable was mandatory and
877 no value was provided, or if one or more tables was not valid
878 """
879 cstables = get_str_var(req, var, mandatory=mandatory)
880 if not cstables:
881 return []
882 # can't have any commas in table names, so it's OK to use a simple
883 # split() command
884 tablenames = [x.strip() for x in cstables.split(",")]
885 tables = [] # type: List[Table]
886 for tn in tablenames:
887 if tn in SILENTLY_IGNORE_TABLENAMES:
888 log.warning(IGNORING_ANTIQUE_TABLE_MESSAGE)
889 continue
890 ensure_valid_table_name(req, tn)
891 tables.append(CLIENT_TABLE_MAP[tn])
892 return tables
895def get_single_field_from_post_var(
896 req: "CamcopsRequest", table: Table, var: str, mandatory: bool = True
897) -> str:
898 """
899 Retrieves a field (column) name from a the request and checks it's not a
900 bad fieldname for the specified table.
902 Args:
903 req: the :class:`camcops_server.cc_modules.cc_request.CamcopsRequest`
904 table: SQLAlchemy :class:`Table` in which the column should exist
905 var: name of variable to retrieve
906 mandatory: if ``True``, raise an exception if the variable is missing
908 Returns:
909 the field (column) name
911 Raises:
912 :exc:`UserErrorException` if the variable was mandatory and
913 no value was provided, or if the field was not valid for the specified
914 table
915 """
916 field = get_str_var(req, var, mandatory=mandatory)
917 ensure_valid_field_name(table, field)
918 return field
921def get_fields_from_post_var(
922 req: "CamcopsRequest",
923 table: Table,
924 var: str,
925 mandatory: bool = True,
926 allowed_nonexistent_fields: List[str] = None,
927) -> List[str]:
928 """
929 Get a comma-separated list of field names from a request and checks that
930 all are acceptable. Returns a list of fieldnames.
932 Args:
933 req: the :class:`camcops_server.cc_modules.cc_request.CamcopsRequest`
934 table: SQLAlchemy :class:`Table` in which the columns should exist
935 var: name of variable to retrieve
936 mandatory: if ``True``, raise an exception if the variable is missing
937 allowed_nonexistent_fields: fields that are allowed to be in the
938 upload but not in the database (special exemptions!)
940 Returns:
941 a list of the field (column) names
943 Raises:
944 :exc:`UserErrorException` if the variable was mandatory and
945 no value was provided, or if any field was not valid for the specified
946 table
947 """
948 csfields = get_str_var(req, var, mandatory=mandatory)
949 if not csfields:
950 return []
951 allowed_nonexistent_fields = (
952 allowed_nonexistent_fields or []
953 ) # type: List[str] # noqa
954 # can't have any commas in fields, so it's OK to use a simple
955 # split() command
956 fields = [x.strip() for x in csfields.split(",")]
957 for f in fields:
958 if f in allowed_nonexistent_fields:
959 continue
960 ensure_valid_field_name(table, f)
961 return fields
964def get_values_from_post_var(
965 req: "CamcopsRequest", var: str, mandatory: bool = True
966) -> List[Any]:
967 """
968 Retrieves a list of values from a CSV-separated list of SQL values
969 stored in a CGI form (including e.g. NULL, numbers, quoted strings, and
970 special handling for base-64/hex-encoded BLOBs.)
972 Args:
973 req: the :class:`camcops_server.cc_modules.cc_request.CamcopsRequest`
974 var: name of variable to retrieve
975 mandatory: if ``True``, raise an exception if the variable is missing
976 """
977 csvalues = get_str_var(req, var, mandatory=mandatory)
978 if not csvalues:
979 return []
980 return decode_values(csvalues)
983def get_fields_and_values(
984 req: "CamcopsRequest",
985 table: Table,
986 fields_var: str,
987 values_var: str,
988 mandatory: bool = True,
989) -> Dict[str, Any]:
990 """
991 Gets fieldnames and matching values from two variables in a request.
993 See :func:`get_fields_from_post_var`, :func:`get_values_from_post_var`.
995 Args:
996 req: the :class:`camcops_server.cc_modules.cc_request.CamcopsRequest`
997 table: SQLAlchemy :class:`Table` in which the columns should exist
998 fields_var: name of CSV "column names" variable to retrieve
999 values_var: name of CSV "corresponding values" variable to retrieve
1000 mandatory: if ``True``, raise an exception if the variable is missing
1002 Returns:
1003 a dictionary mapping column names to decoded values
1005 Raises:
1006 :exc:`UserErrorException` if the variable was mandatory and
1007 no value was provided, or if any field was not valid for the specified
1008 table
1009 """
1010 fields = get_fields_from_post_var(
1011 req, table, fields_var, mandatory=mandatory
1012 )
1013 values = get_values_from_post_var(req, values_var, mandatory=mandatory)
1014 if len(fields) != len(values):
1015 fail_user_error(
1016 f"Number of fields ({len(fields)}) doesn't match number of values "
1017 f"({len(values)})"
1018 )
1019 return dict(list(zip(fields, values)))
1022def get_json_from_post_var(
1023 req: "CamcopsRequest",
1024 key: str,
1025 decoder: json.JSONDecoder = None,
1026 mandatory: bool = True,
1027) -> Any:
1028 """
1029 Returns a Python object from a JSON-encoded value.
1031 Args:
1032 req: the :class:`camcops_server.cc_modules.cc_request.CamcopsRequest`
1033 key: the name of the variable to retrieve
1034 decoder: the JSON decoder object to use; if ``None``, a default is
1035 created
1036 mandatory: if ``True``, raise an exception if the variable is missing
1038 Returns:
1039 Python object, e.g. a list of values, or ``None`` if the object is
1040 invalid and not mandatory
1042 Raises:
1043 :exc:`UserErrorException` if the variable was mandatory and
1044 no value was provided or the value was invalid JSON
1045 """
1046 decoder = decoder or json.JSONDecoder()
1047 j = get_str_var(req, key, mandatory=mandatory) # may raise
1048 if not j: # missing but not mandatory
1049 return None
1050 try:
1051 return decoder.decode(j)
1052 except json.JSONDecodeError:
1053 msg = f"Bad JSON for key {key!r}"
1054 if mandatory:
1055 fail_user_error(msg)
1056 else:
1057 log.warning(msg)
1058 return None
1061# =============================================================================
1062# Sending stuff to the client
1063# =============================================================================
1066def get_server_id_info(req: "CamcopsRequest") -> Dict[str, str]:
1067 """
1068 Returns a reply for the tablet, as a variable-to-value dictionary, giving
1069 details of the server.
1070 """
1071 group = Group.get_group_by_id(req.dbsession, req.user.upload_group_id)
1072 reply = {
1073 TabletParam.DATABASE_TITLE: req.database_title,
1074 TabletParam.ID_POLICY_UPLOAD: group.upload_policy or "",
1075 TabletParam.ID_POLICY_FINALIZE: group.finalize_policy or "",
1076 TabletParam.SERVER_CAMCOPS_VERSION: CAMCOPS_SERVER_VERSION_STRING,
1077 }
1078 for iddef in req.idnum_definitions:
1079 n = iddef.which_idnum
1080 nstr = str(n)
1081 reply[TabletParam.ID_DESCRIPTION_PREFIX + nstr] = (
1082 iddef.description or ""
1083 )
1084 reply[TabletParam.ID_SHORT_DESCRIPTION_PREFIX + nstr] = (
1085 iddef.short_description or ""
1086 )
1087 reply[TabletParam.ID_VALIDATION_METHOD_PREFIX + nstr] = (
1088 iddef.validation_method or ""
1089 )
1090 return reply
1093def get_select_reply(
1094 fields: Sequence[str], rows: Sequence[Sequence[Any]]
1095) -> Dict[str, str]:
1096 """
1097 Formats the result of a ``SELECT`` query for the client as a dictionary
1098 reply.
1100 Args:
1101 fields: list of field names
1102 rows: list of rows, where each row is a list of values in the same
1103 order as ``fields``
1105 Returns:
1107 a dictionary of the format:
1109 .. code-block:: none
1111 {
1112 "nfields": NUMBER_OF_FIELDS,
1113 "fields": FIELDNAMES_AS_CSV,
1114 "nrecords": NUMBER_OF_RECORDS,
1115 "record0": VALUES_AS_CSV_LIST_OF_ENCODED_SQL_VALUES,
1116 ...
1117 "record{nrecords - 1}": VALUES_AS_CSV_LIST_OF_ENCODED_SQL_VALUES
1118 }
1120 The final reply to the server is then formatted as text as per
1121 :func:`client_api`.
1123 """ # noqa
1124 nrecords = len(rows)
1125 reply = {
1126 TabletParam.NFIELDS: len(fields),
1127 TabletParam.FIELDS: ",".join(fields),
1128 TabletParam.NRECORDS: nrecords,
1129 }
1130 for r in range(nrecords):
1131 row = rows[r]
1132 encodedvalues = [] # type: List[str]
1133 for val in row:
1134 encodedvalues.append(encode_single_value(val))
1135 reply[TabletParam.RECORD_PREFIX + str(r)] = ",".join(encodedvalues)
1136 return reply
1139# =============================================================================
1140# CamCOPS table reading functions
1141# =============================================================================
1144def record_exists(
1145 req: "CamcopsRequest",
1146 table: Table,
1147 clientpk_name: str,
1148 clientpk_value: Any,
1149) -> ServerRecord:
1150 """
1151 Checks if a record exists, using the device's perspective of a
1152 table/client PK combination.
1154 Args:
1155 req: the :class:`camcops_server.cc_modules.cc_request.CamcopsRequest`
1156 table: an SQLAlchemy :class:`Table`
1157 clientpk_name: the column name of the client's PK
1158 clientpk_value: the client's PK value
1160 Returns:
1161 a :class:`ServerRecord` with the required information
1163 """
1164 query = (
1165 select(
1166 [
1167 table.c[FN_PK], # server PK
1168 table.c[
1169 CLIENT_DATE_FIELD
1170 ], # when last modified (on the server)
1171 table.c[MOVE_OFF_TABLET_FIELD], # move_off_tablet
1172 ]
1173 )
1174 .where(table.c[FN_DEVICE_ID] == req.tabletsession.device_id)
1175 .where(table.c[FN_CURRENT])
1176 .where(table.c[FN_ERA] == ERA_NOW)
1177 .where(table.c[clientpk_name] == clientpk_value)
1178 )
1179 row = req.dbsession.execute(query).fetchone()
1180 if not row:
1181 return ServerRecord(clientpk_value, False)
1182 server_pk, server_when, move_off_tablet = row
1183 return ServerRecord(
1184 clientpk_value, True, server_pk, server_when, move_off_tablet
1185 )
1186 # Consider a warning/failure if we have >1 row meeting these criteria.
1187 # Not currently checked for.
1190def client_pks_that_exist(
1191 req: "CamcopsRequest",
1192 table: Table,
1193 clientpk_name: str,
1194 clientpk_values: List[int],
1195) -> Dict[int, ServerRecord]:
1196 """
1197 Searches for client PK values (for this device, current, and 'now')
1198 matching the input list.
1200 Args:
1201 req: the :class:`camcops_server.cc_modules.cc_request.CamcopsRequest`
1202 table: an SQLAlchemy :class:`Table`
1203 clientpk_name: the column name of the client's PK
1204 clientpk_values: a list of the client's PK values
1206 Returns:
1207 a dictionary mapping client_pk to a :class:`ServerRecord` objects, for
1208 those records that match
1209 """
1210 query = (
1211 select(
1212 [
1213 table.c[FN_PK], # server PK
1214 table.c[clientpk_name], # client PK
1215 table.c[
1216 CLIENT_DATE_FIELD
1217 ], # when last modified (on the server)
1218 table.c[MOVE_OFF_TABLET_FIELD], # move_off_tablet
1219 ]
1220 )
1221 .where(table.c[FN_DEVICE_ID] == req.tabletsession.device_id)
1222 .where(table.c[FN_CURRENT])
1223 .where(table.c[FN_ERA] == ERA_NOW)
1224 .where(table.c[clientpk_name].in_(clientpk_values))
1225 )
1226 rows = req.dbsession.execute(query)
1227 d = {} # type: Dict[int, ServerRecord]
1228 for server_pk, client_pk, server_when, move_off_tablet in rows:
1229 d[client_pk] = ServerRecord(
1230 client_pk, True, server_pk, server_when, move_off_tablet
1231 )
1232 return d
1235def get_all_predecessor_pks(
1236 req: "CamcopsRequest",
1237 table: Table,
1238 last_pk: int,
1239 include_last: bool = True,
1240) -> List[int]:
1241 """
1242 Retrieves the PKs of all records that are predecessors of the specified one
1244 Args:
1245 req: the :class:`camcops_server.cc_modules.cc_request.CamcopsRequest`
1246 table: an SQLAlchemy :class:`Table`
1247 last_pk: the PK to start with, and work backwards
1248 include_last: include ``last_pk`` in the list
1250 Returns:
1251 the PKs
1253 """
1254 dbsession = req.dbsession
1255 pks = [] # type: List[int]
1256 if include_last:
1257 pks.append(last_pk)
1258 current_pk = last_pk
1259 finished = False
1260 while not finished:
1261 next_pk = dbsession.execute(
1262 select([table.c[FN_PREDECESSOR_PK]]).where(
1263 table.c[FN_PK] == current_pk
1264 )
1265 ).scalar() # type: Optional[int]
1266 if next_pk is None:
1267 finished = True
1268 else:
1269 pks.append(next_pk)
1270 current_pk = next_pk
1271 return sorted(pks)
1274# =============================================================================
1275# Record modification functions
1276# =============================================================================
1279def flag_deleted(
1280 req: "CamcopsRequest",
1281 batchdetails: BatchDetails,
1282 table: Table,
1283 pklist: Iterable[int],
1284) -> None:
1285 """
1286 Marks record(s) as deleted, specified by a list of server PKs within a
1287 table. (Note: "deleted" means "deleted with no successor", not "modified
1288 and replaced by a successor record".)
1289 """
1290 if batchdetails.onestep:
1291 values = values_delete_now(req, batchdetails)
1292 else:
1293 values = values_delete_later()
1294 req.dbsession.execute(
1295 update(table).where(table.c[FN_PK].in_(pklist)).values(values)
1296 )
1299def flag_all_records_deleted(req: "CamcopsRequest", table: Table) -> int:
1300 """
1301 Marks all records in a table as deleted (that are current and in the
1302 current era).
1304 Returns the number of rows affected.
1305 """
1306 rp = req.dbsession.execute(
1307 update(table)
1308 .where(table.c[FN_DEVICE_ID] == req.tabletsession.device_id)
1309 .where(table.c[FN_CURRENT])
1310 .where(table.c[FN_ERA] == ERA_NOW)
1311 .values(values_delete_later())
1312 ) # type: ResultProxy
1313 return rp.rowcount
1314 # https://docs.sqlalchemy.org/en/latest/core/connections.html?highlight=rowcount#sqlalchemy.engine.ResultProxy.rowcount # noqa
1317def flag_deleted_where_clientpk_not(
1318 req: "CamcopsRequest",
1319 table: Table,
1320 clientpk_name: str,
1321 clientpk_values: Sequence[Any],
1322) -> None:
1323 """
1324 Marks for deletion all current/current-era records for a device, within a
1325 specific table, defined by a list of client-side PK values (and the name of
1326 the client-side PK column).
1327 """
1328 rp = req.dbsession.execute(
1329 update(table)
1330 .where(table.c[FN_DEVICE_ID] == req.tabletsession.device_id)
1331 .where(table.c[FN_CURRENT])
1332 .where(table.c[FN_ERA] == ERA_NOW)
1333 .where(table.c[clientpk_name].notin_(clientpk_values))
1334 .values(values_delete_later())
1335 ) # type: ResultProxy
1336 if rp.rowcount > 0:
1337 mark_table_dirty(req, table)
1338 # ... but if we are preserving, do NOT mark this table as clean; there may
1339 # be other records that still require preserving.
1342def flag_modified(
1343 req: "CamcopsRequest",
1344 batchdetails: BatchDetails,
1345 table: Table,
1346 pk: int,
1347 successor_pk: int,
1348) -> None:
1349 """
1350 Marks a record as old, storing its successor's details.
1352 Args:
1353 req: the :class:`camcops_server.cc_modules.cc_request.CamcopsRequest`
1354 batchdetails: the :class:`BatchDetails`
1355 table: SQLAlchemy :class:`Table`
1356 pk: server PK of the record to mark as old
1357 successor_pk: server PK of its successor
1358 """
1359 if batchdetails.onestep:
1360 req.dbsession.execute(
1361 update(table)
1362 .where(table.c[FN_PK] == pk)
1363 .values(
1364 {
1365 FN_CURRENT: 0,
1366 FN_REMOVAL_PENDING: 0,
1367 FN_SUCCESSOR_PK: successor_pk,
1368 FN_REMOVING_USER_ID: req.user_id,
1369 FN_WHEN_REMOVED_EXACT: req.now,
1370 FN_WHEN_REMOVED_BATCH_UTC: batchdetails.batchtime,
1371 }
1372 )
1373 )
1374 else:
1375 req.dbsession.execute(
1376 update(table)
1377 .where(table.c[FN_PK] == pk)
1378 .values({FN_REMOVAL_PENDING: 1, FN_SUCCESSOR_PK: successor_pk})
1379 )
1382def flag_multiple_records_for_preservation(
1383 req: "CamcopsRequest",
1384 batchdetails: BatchDetails,
1385 table: Table,
1386 pks_to_preserve: List[int],
1387) -> None:
1388 """
1389 Low-level function to mark records for preservation by server PK.
1390 Does not concern itself with the predecessor chain (for which, see
1391 :func:`flag_record_for_preservation`).
1393 Args:
1394 req: the :class:`camcops_server.cc_modules.cc_request.CamcopsRequest`
1395 batchdetails: the :class:`BatchDetails`
1396 table: SQLAlchemy :class:`Table`
1397 pks_to_preserve: server PK of the records to mark as preserved
1398 """
1399 if batchdetails.onestep:
1400 req.dbsession.execute(
1401 update(table)
1402 .where(table.c[FN_PK].in_(pks_to_preserve))
1403 .values(values_preserve_now(req, batchdetails))
1404 )
1405 # Also any associated special notes:
1406 new_era = batchdetails.new_era
1407 # noinspection PyUnresolvedReferences
1408 req.dbsession.execute(
1409 update(SpecialNote.__table__)
1410 .where(SpecialNote.basetable == table.name)
1411 .where(SpecialNote.device_id == req.tabletsession.device_id)
1412 .where(SpecialNote.era == ERA_NOW)
1413 .where(
1414 exists()
1415 .select_from(table)
1416 .where(table.c[TABLET_ID_FIELD] == SpecialNote.task_id)
1417 .where(table.c[FN_DEVICE_ID] == SpecialNote.device_id)
1418 .where(table.c[FN_ERA] == new_era)
1419 )
1420 # ^^^^^^^^^^^^^^^^^^^^^^^^^^
1421 # This bit restricts to records being preserved.
1422 .values(era=new_era)
1423 )
1424 else:
1425 req.dbsession.execute(
1426 update(table)
1427 .where(table.c[FN_PK].in_(pks_to_preserve))
1428 .values({MOVE_OFF_TABLET_FIELD: 1})
1429 )
1432def flag_record_for_preservation(
1433 req: "CamcopsRequest", batchdetails: BatchDetails, table: Table, pk: int
1434) -> List[int]:
1435 """
1436 Marks a record for preservation (moving off the tablet, changing its
1437 era details).
1439 2018-11-18: works back through the predecessor chain too, fixing an old
1440 bug.
1442 Args:
1443 req: the :class:`camcops_server.cc_modules.cc_request.CamcopsRequest`
1444 batchdetails: the :class:`BatchDetails`
1445 table: SQLAlchemy :class:`Table`
1446 pk: server PK of the record to mark
1448 Returns:
1449 list: all PKs being preserved
1450 """
1451 pks_to_preserve = get_all_predecessor_pks(req, table, pk)
1452 flag_multiple_records_for_preservation(
1453 req, batchdetails, table, pks_to_preserve
1454 )
1455 return pks_to_preserve
1458def preserve_all(
1459 req: "CamcopsRequest", batchdetails: BatchDetails, table: Table
1460) -> None:
1461 """
1462 Preserves all records in a table for a device, including non-current ones.
1464 Args:
1465 req: the :class:`camcops_server.cc_modules.cc_request.CamcopsRequest`
1466 batchdetails: the :class:`BatchDetails`
1467 table: SQLAlchemy :class:`Table`
1468 """
1469 device_id = req.tabletsession.device_id
1470 req.dbsession.execute(
1471 update(table)
1472 .where(table.c[FN_DEVICE_ID] == device_id)
1473 .where(table.c[FN_ERA] == ERA_NOW)
1474 .values(values_preserve_now(req, batchdetails))
1475 )
1478# =============================================================================
1479# Upload helper functions
1480# =============================================================================
1483def process_upload_record_special(
1484 req: "CamcopsRequest",
1485 batchdetails: BatchDetails,
1486 table: Table,
1487 valuedict: Dict[str, Any],
1488) -> None:
1489 """
1490 Special processing function for upload, in which we inspect the data.
1491 Called by :func:`upload_record_core`.
1493 1. Handles old clients with ID information in the patient table, etc.
1494 (Note: this can be IGNORED for any client using
1495 :func:`op_upload_entire_database`, as these are newer.)
1497 2. Validates ID numbers.
1499 Args:
1500 req: the :class:`camcops_server.cc_modules.cc_request.CamcopsRequest`
1501 batchdetails: the :class:`BatchDetails`
1502 table: an SQLAlchemy :class:`Table`
1503 valuedict: a dictionary of {colname: value} pairs from the client.
1504 May be modified.
1505 """
1506 ts = req.tabletsession
1507 tablename = table.name
1509 if tablename == Patient.__tablename__:
1510 # ---------------------------------------------------------------------
1511 # Deal with old tablets that had ID numbers in a less flexible format.
1512 # ---------------------------------------------------------------------
1513 if ts.cope_with_deleted_patient_descriptors:
1514 # Old tablets (pre-2.0.0) will upload copies of the ID
1515 # descriptions with the patient. To cope with that, we
1516 # remove those here:
1517 for n in range(1, NUMBER_OF_IDNUMS_DEFUNCT + 1):
1518 nstr = str(n)
1519 fn_desc = FP_ID_DESC + nstr
1520 fn_shortdesc = FP_ID_SHORT_DESC + nstr
1521 valuedict.pop(fn_desc, None) # remove item, if exists
1522 valuedict.pop(fn_shortdesc, None)
1524 if ts.cope_with_old_idnums:
1525 # Insert records into the new ID number table from the old
1526 # patient table:
1527 for which_idnum in range(1, NUMBER_OF_IDNUMS_DEFUNCT + 1):
1528 nstr = str(which_idnum)
1529 fn_idnum = FP_ID_NUM + nstr
1530 idnum_value = valuedict.pop(fn_idnum, None)
1531 # ... and remove it from our new Patient record
1532 patient_id = valuedict.get("id", None)
1533 if idnum_value is None or patient_id is None:
1534 continue
1535 # noinspection PyUnresolvedReferences
1536 mark_table_dirty(req, PatientIdNum.__table__)
1537 client_date_value = coerce_to_pendulum(
1538 valuedict[CLIENT_DATE_FIELD]
1539 )
1540 # noinspection PyUnresolvedReferences
1541 upload_record_core(
1542 req=req,
1543 batchdetails=batchdetails,
1544 table=PatientIdNum.__table__,
1545 clientpk_name="id",
1546 valuedict={
1547 "id": fake_tablet_id_for_patientidnum(
1548 patient_id=patient_id, which_idnum=which_idnum
1549 ), # ... guarantees a pseudo client PK
1550 "patient_id": patient_id,
1551 "which_idnum": which_idnum,
1552 "idnum_value": idnum_value,
1553 CLIENT_DATE_FIELD: client_date_value,
1554 MOVE_OFF_TABLET_FIELD: valuedict[
1555 MOVE_OFF_TABLET_FIELD
1556 ], # noqa
1557 },
1558 )
1559 # Now, how to deal with deletion, i.e. records missing from the
1560 # tablet? See our caller, op_upload_table(), which has a special
1561 # handler for this.
1562 #
1563 # Note that op_upload_record() is/was only used for BLOBs, so we
1564 # don't have to worry about special processing for that aspect
1565 # here; also, that method handles deletion in a different way.
1567 elif tablename == PatientIdNum.__tablename__:
1568 # ---------------------------------------------------------------------
1569 # Validate ID numbers.
1570 # ---------------------------------------------------------------------
1571 which_idnum = valuedict.get("which_idnum", None)
1572 if which_idnum not in req.valid_which_idnums:
1573 fail_user_error(f"No such ID number type: {which_idnum}")
1574 idnum_value = valuedict.get("idnum_value", None)
1575 if not req.is_idnum_valid(which_idnum, idnum_value):
1576 why_invalid = req.why_idnum_invalid(which_idnum, idnum_value)
1577 fail_user_error(
1578 f"For ID type {which_idnum}, ID number {idnum_value} is "
1579 f"invalid: {why_invalid}"
1580 )
1583def upload_record_core(
1584 req: "CamcopsRequest",
1585 batchdetails: BatchDetails,
1586 table: Table,
1587 clientpk_name: str,
1588 valuedict: Dict[str, Any],
1589 server_live_current_records: List[ServerRecord] = None,
1590) -> UploadRecordResult:
1591 """
1592 Uploads a record. Deals with IDENTICAL, NEW, and MODIFIED records.
1594 Used by :func:`upload_table` and :func:`upload_record`.
1596 Args:
1597 req: the :class:`camcops_server.cc_modules.cc_request.CamcopsRequest`
1598 batchdetails: the :class:`BatchDetails`
1599 table: an SQLAlchemy :class:`Table`
1600 clientpk_name: the column name of the client's PK
1601 valuedict: a dictionary of {colname: value} pairs from the client
1602 server_live_current_records: list of :class:`ServerRecord` objects for
1603 the active records on the server for this client, in this table
1605 Returns:
1606 a :class:`UploadRecordResult` object
1607 """
1608 require_keys(
1609 valuedict, [clientpk_name, CLIENT_DATE_FIELD, MOVE_OFF_TABLET_FIELD]
1610 )
1611 clientpk_value = valuedict[clientpk_name]
1613 if server_live_current_records:
1614 # All server records for this table/device/era have been prefetched.
1615 serverrec = next(
1616 (
1617 r
1618 for r in server_live_current_records
1619 if r.client_pk == clientpk_value
1620 ),
1621 None,
1622 )
1623 if serverrec is None:
1624 serverrec = ServerRecord(clientpk_value, False)
1625 else:
1626 # Look up this record specifically.
1627 serverrec = record_exists(req, table, clientpk_name, clientpk_value)
1629 if DEBUG_UPLOAD:
1630 log.debug("upload_record_core: {}, {}", table.name, serverrec)
1632 oldserverpk = serverrec.server_pk
1633 urr = UploadRecordResult(
1634 oldserverpk=oldserverpk,
1635 specifically_marked_for_preservation=bool(
1636 valuedict[MOVE_OFF_TABLET_FIELD]
1637 ),
1638 dirty=True,
1639 )
1640 if serverrec.exists:
1641 # There's an existing record, which is either identical or not.
1642 client_date_value = coerce_to_pendulum(valuedict[CLIENT_DATE_FIELD])
1643 if serverrec.server_when == client_date_value:
1644 # The existing record is identical.
1645 # No action needed unless MOVE_OFF_TABLET_FIELDNAME is set.
1646 if not urr.specifically_marked_for_preservation:
1647 urr.dirty = False
1648 else:
1649 # The existing record is different. We need a logical UPDATE, but
1650 # maintaining an audit trail.
1651 process_upload_record_special(req, batchdetails, table, valuedict)
1652 urr.newserverpk = insert_record(
1653 req, batchdetails, table, valuedict, oldserverpk
1654 )
1655 flag_modified(
1656 req, batchdetails, table, oldserverpk, urr.newserverpk
1657 )
1658 else:
1659 # The record is NEW. We need to INSERT it.
1660 process_upload_record_special(req, batchdetails, table, valuedict)
1661 urr.newserverpk = insert_record(
1662 req, batchdetails, table, valuedict, None
1663 )
1664 if urr.specifically_marked_for_preservation:
1665 preservation_pks = flag_record_for_preservation(
1666 req, batchdetails, table, urr.latest_pk
1667 )
1668 urr.note_specifically_marked_preservation_pks(preservation_pks)
1670 if DEBUG_UPLOAD:
1671 log.debug("upload_record_core: {}, {!r}", table.name, urr)
1672 return urr
1675def insert_record(
1676 req: "CamcopsRequest",
1677 batchdetails: BatchDetails,
1678 table: Table,
1679 valuedict: Dict[str, Any],
1680 predecessor_pk: Optional[int],
1681) -> int:
1682 """
1683 Inserts a record, or raises an exception if that fails.
1685 Args:
1686 req: the :class:`camcops_server.cc_modules.cc_request.CamcopsRequest`
1687 batchdetails: the :class:`BatchDetails`
1688 table: an SQLAlchemy :class:`Table`
1689 valuedict: a dictionary of {colname: value} pairs from the client
1690 predecessor_pk: an optional server PK of the record's predecessor
1692 Returns:
1693 the server PK of the new record
1694 """
1695 ts = req.tabletsession
1696 valuedict.update(
1697 {
1698 FN_DEVICE_ID: ts.device_id,
1699 FN_ERA: ERA_NOW,
1700 FN_REMOVAL_PENDING: 0,
1701 FN_PREDECESSOR_PK: predecessor_pk,
1702 FN_CAMCOPS_VERSION: ts.tablet_version_str,
1703 FN_GROUP_ID: req.user.upload_group_id,
1704 }
1705 )
1706 if batchdetails.onestep:
1707 valuedict.update(
1708 {
1709 FN_CURRENT: 1,
1710 FN_ADDITION_PENDING: 0,
1711 FN_ADDING_USER_ID: req.user_id,
1712 FN_WHEN_ADDED_EXACT: req.now,
1713 FN_WHEN_ADDED_BATCH_UTC: batchdetails.batchtime,
1714 }
1715 )
1716 else:
1717 valuedict.update({FN_CURRENT: 0, FN_ADDITION_PENDING: 1})
1718 rp = req.dbsession.execute(
1719 table.insert().values(valuedict)
1720 ) # type: ResultProxy
1721 inserted_pks = rp.inserted_primary_key
1722 assert isinstance(inserted_pks, list) and len(inserted_pks) == 1
1723 return inserted_pks[0]
1726def audit_upload(
1727 req: "CamcopsRequest", changes: List[UploadTableChanges]
1728) -> None:
1729 """
1730 Writes audit information for an upload.
1732 Args:
1733 req: the :class:`camcops_server.cc_modules.cc_request.CamcopsRequest`
1734 changes: a list of :class:`UploadTableChanges` objects, one per table
1735 """
1736 msg = (
1737 f"Upload from device {req.tabletsession.device_id}, "
1738 f"username {req.tabletsession.username!r}: "
1739 )
1740 changes = [x for x in changes if x.any_changes]
1741 if changes:
1742 changes.sort(key=lambda x: x.tablename)
1743 msg += ", ".join(x.description() for x in changes)
1744 else:
1745 msg += "No changes"
1746 log.info("audit_upload: {}", msg)
1747 audit(req, msg)
1750# =============================================================================
1751# Batch (atomic) upload and preserving
1752# =============================================================================
1755def get_batch_details(req: "CamcopsRequest") -> BatchDetails:
1756 """
1757 Returns the :class:`BatchDetails` for the current upload. If none exists,
1758 a new batch is created and returned.
1760 SIDE EFFECT: if the username is different from the username that started
1761 a previous upload batch for this device, we restart the upload batch (thus
1762 rolling back previous pending changes).
1764 Raises:
1765 :exc:`camcops_server.cc_modules.cc_client_api_core.ServerErrorException`
1766 if the device doesn't exist
1767 """
1768 device_id = req.tabletsession.device_id
1769 # noinspection PyUnresolvedReferences
1770 query = (
1771 select(
1772 [
1773 Device.ongoing_upload_batch_utc,
1774 Device.uploading_user_id,
1775 Device.currently_preserving,
1776 ]
1777 )
1778 .select_from(Device.__table__)
1779 .where(Device.id == device_id)
1780 )
1781 row = req.dbsession.execute(query).fetchone()
1782 if not row:
1783 fail_server_error(
1784 f"Device {device_id} missing from Device table"
1785 ) # will raise # noqa
1786 upload_batch_utc, uploading_user_id, currently_preserving = row
1787 if not upload_batch_utc or uploading_user_id != req.user_id:
1788 # SIDE EFFECT: if the username changes, we restart (and thus roll back
1789 # previous pending changes)
1790 start_device_upload_batch(req)
1791 return BatchDetails(req.now_utc, False)
1792 return BatchDetails(upload_batch_utc, currently_preserving)
1795def start_device_upload_batch(req: "CamcopsRequest") -> None:
1796 """
1797 Starts an upload batch for a device.
1798 """
1799 rollback_all(req)
1800 # noinspection PyUnresolvedReferences
1801 req.dbsession.execute(
1802 update(Device.__table__)
1803 .where(Device.id == req.tabletsession.device_id)
1804 .values(
1805 last_upload_batch_utc=req.now_utc,
1806 ongoing_upload_batch_utc=req.now_utc,
1807 uploading_user_id=req.tabletsession.user_id,
1808 )
1809 )
1812def _clear_ongoing_upload_batch_details(req: "CamcopsRequest") -> None:
1813 """
1814 Clears upload batch details from the Device table.
1816 Args:
1817 req: the :class:`camcops_server.cc_modules.cc_request.CamcopsRequest`
1818 """
1819 # noinspection PyUnresolvedReferences
1820 req.dbsession.execute(
1821 update(Device.__table__)
1822 .where(Device.id == req.tabletsession.device_id)
1823 .values(
1824 ongoing_upload_batch_utc=None,
1825 uploading_user_id=None,
1826 currently_preserving=0,
1827 )
1828 )
1831def end_device_upload_batch(
1832 req: "CamcopsRequest", batchdetails: BatchDetails
1833) -> None:
1834 """
1835 Ends an upload batch, committing all changes made thus far.
1837 Args:
1838 req: the :class:`camcops_server.cc_modules.cc_request.CamcopsRequest`
1839 batchdetails: the :class:`BatchDetails`
1840 """
1841 commit_all(req, batchdetails)
1842 _clear_ongoing_upload_batch_details(req)
1845def clear_device_upload_batch(req: "CamcopsRequest") -> None:
1846 """
1847 Ensures there is nothing pending. Rools back previous changes. Wipes any
1848 ongoing batch details.
1850 Args:
1851 req: the :class:`camcops_server.cc_modules.cc_request.CamcopsRequest`
1852 """
1853 rollback_all(req)
1854 _clear_ongoing_upload_batch_details(req)
1857def start_preserving(req: "CamcopsRequest") -> None:
1858 """
1859 Starts preservation (the process of moving records from the NOW era to
1860 an older era, so they can be removed safely from the tablet).
1862 Called by :func:`op_start_preservation`.
1864 In this situation, we start by assuming that ALL tables are "dirty",
1865 because they may have live records from a previous upload.
1866 """
1867 # noinspection PyUnresolvedReferences
1868 req.dbsession.execute(
1869 update(Device.__table__)
1870 .where(Device.id == req.tabletsession.device_id)
1871 .values(currently_preserving=1)
1872 )
1873 mark_all_tables_dirty(req)
1876def mark_table_dirty(req: "CamcopsRequest", table: Table) -> None:
1877 """
1878 Marks a table as having been modified during the current upload.
1879 """
1880 tablename = table.name
1881 device_id = req.tabletsession.device_id
1882 dbsession = req.dbsession
1883 # noinspection PyUnresolvedReferences
1884 table_already_dirty = exists_in_table(
1885 dbsession,
1886 DirtyTable.__table__,
1887 DirtyTable.device_id == device_id,
1888 DirtyTable.tablename == tablename,
1889 )
1890 if not table_already_dirty:
1891 # noinspection PyUnresolvedReferences
1892 dbsession.execute(
1893 DirtyTable.__table__.insert().values(
1894 device_id=device_id, tablename=tablename
1895 )
1896 )
1899def mark_tables_dirty(req: "CamcopsRequest", tables: List[Table]) -> None:
1900 """
1901 Marks multiple tables as dirty.
1902 """
1903 if not tables:
1904 return
1905 device_id = req.tabletsession.device_id
1906 tablenames = [t.name for t in tables]
1907 # Delete first
1908 # noinspection PyUnresolvedReferences
1909 req.dbsession.execute(
1910 DirtyTable.__table__.delete()
1911 .where(DirtyTable.device_id == device_id)
1912 .where(DirtyTable.tablename.in_(tablenames))
1913 )
1914 # Then insert
1915 insert_values = [
1916 {"device_id": device_id, "tablename": tn} for tn in tablenames
1917 ]
1918 # noinspection PyUnresolvedReferences
1919 req.dbsession.execute(DirtyTable.__table__.insert(), insert_values)
1922def mark_all_tables_dirty(req: "CamcopsRequest") -> None:
1923 """
1924 If we are preserving, we assume that all tables are "dirty" (require work
1925 when we complete the upload) unless we specifically mark them clean.
1926 """
1927 device_id = req.tabletsession.device_id
1928 # Delete first
1929 # noinspection PyUnresolvedReferences
1930 req.dbsession.execute(
1931 DirtyTable.__table__.delete().where(DirtyTable.device_id == device_id)
1932 )
1933 # Now insert
1934 # https://docs.sqlalchemy.org/en/latest/core/tutorial.html#execute-multiple
1935 all_client_tablenames = list(CLIENT_TABLE_MAP.keys())
1936 insert_values = [
1937 {"device_id": device_id, "tablename": tn}
1938 for tn in all_client_tablenames
1939 ]
1940 # noinspection PyUnresolvedReferences
1941 req.dbsession.execute(DirtyTable.__table__.insert(), insert_values)
1944def mark_table_clean(req: "CamcopsRequest", table: Table) -> None:
1945 """
1946 Marks a table as being clean: that is,
1948 - the table has been scanned during the current upload
1949 - there is nothing to do (either from the current upload, OR A PREVIOUS
1950 UPLOAD).
1951 """
1952 tablename = table.name
1953 device_id = req.tabletsession.device_id
1954 # noinspection PyUnresolvedReferences
1955 req.dbsession.execute(
1956 DirtyTable.__table__.delete()
1957 .where(DirtyTable.device_id == device_id)
1958 .where(DirtyTable.tablename == tablename)
1959 )
1962def mark_tables_clean(req: "CamcopsRequest", tables: List[Table]) -> None:
1963 """
1964 Marks multiple tables as clean.
1965 """
1966 if not tables:
1967 return
1968 device_id = req.tabletsession.device_id
1969 tablenames = [t.name for t in tables]
1970 # Delete first
1971 # noinspection PyUnresolvedReferences
1972 req.dbsession.execute(
1973 DirtyTable.__table__.delete()
1974 .where(DirtyTable.device_id == device_id)
1975 .where(DirtyTable.tablename.in_(tablenames))
1976 )
1979def get_dirty_tables(req: "CamcopsRequest") -> List[Table]:
1980 """
1981 Returns tables marked as dirty for this device. (See
1982 :func:`mark_table_dirty`.)
1983 """
1984 query = select([DirtyTable.tablename]).where(
1985 DirtyTable.device_id == req.tabletsession.device_id
1986 )
1987 tablenames = fetch_all_first_values(req.dbsession, query)
1988 return [CLIENT_TABLE_MAP[tn] for tn in tablenames]
1991def commit_all(req: "CamcopsRequest", batchdetails: BatchDetails) -> None:
1992 """
1993 Commits additions, removals, and preservations for all tables.
1995 Args:
1996 req: the :class:`camcops_server.cc_modules.cc_request.CamcopsRequest`
1997 batchdetails: the :class:`BatchDetails`
1998 """
1999 tables = get_dirty_tables(req)
2000 # log.debug("Dirty tables: {}", list(t.name for t in tables))
2001 tables.sort(key=upload_commit_order_sorter)
2003 changelist = [] # type: List[UploadTableChanges]
2004 for table in tables:
2005 auditinfo = commit_table(req, batchdetails, table, clear_dirty=False)
2006 changelist.append(auditinfo)
2008 if batchdetails.preserving:
2009 # Also preserve/finalize any corresponding special notes (2015-02-01),
2010 # but all in one go (2018-11-13).
2011 # noinspection PyUnresolvedReferences
2012 req.dbsession.execute(
2013 update(SpecialNote.__table__)
2014 .where(SpecialNote.device_id == req.tabletsession.device_id)
2015 .where(SpecialNote.era == ERA_NOW)
2016 .values(era=batchdetails.new_era)
2017 )
2019 clear_dirty_tables(req)
2020 audit_upload(req, changelist)
2022 # Performance 2018-11-13:
2023 # - start at 2.407 s
2024 # - remove final temptable clearance and COUNT(*): 1.626 to 2.118 s
2025 # - IN clause using Python literal not temptable: 1.18 to 1.905 s
2026 # - minor tidy: 1.075 to 1.65
2027 # - remove ORDER BY from task indexing: 1.093 to 1.607
2028 # - optimize special note code won't affect this: 1.076 to 1.617
2029 # At this point, entire upload process ~5s.
2030 # - big difference from commit_table() query optimization
2031 # - huge difference from being more careful with mark_table_dirty()
2032 # - further table scanning optimizations: fewer queries
2033 # Overall upload down to ~2.4s
2036def commit_table(
2037 req: "CamcopsRequest",
2038 batchdetails: BatchDetails,
2039 table: Table,
2040 clear_dirty: bool = True,
2041) -> UploadTableChanges:
2042 """
2043 Commits additions, removals, and preservations for one table.
2045 Should ONLY be called by :func:`commit_all`.
2047 Also updates task indexes.
2049 Args:
2050 req: the :class:`camcops_server.cc_modules.cc_request.CamcopsRequest`
2051 batchdetails: the :class:`BatchDetails`
2052 table: SQLAlchemy :class:`Table`
2053 clear_dirty: remove the table from the record of dirty tables for
2054 this device? (If called from :func:`commit_all`, this should be
2055 ``False``, since it's faster to clear all dirty tables for the
2056 device simultaneously than one-by-one.)
2058 Returns:
2059 an :class:`UploadTableChanges` object
2060 """
2062 # Tried storing PKs in temporary tables, rather than using an IN clause
2063 # with Python values, as per
2064 # https://www.xaprb.com/blog/2006/06/28/why-large-in-clauses-are-problematic/ # noqa
2065 # However, it was slow.
2066 # We can gain a lot of efficiency (empirically) by:
2067 # - Storing PKs in Python
2068 # - Only performing updates when we need to
2069 # - Using a single query per table to get "add/remove/preserve" PKs
2071 # -------------------------------------------------------------------------
2072 # Helpful temporary variables
2073 # -------------------------------------------------------------------------
2074 user_id = req.user_id
2075 device_id = req.tabletsession.device_id
2076 exacttime = req.now
2077 dbsession = req.dbsession
2078 tablename = table.name
2079 batchtime = batchdetails.batchtime
2080 preserving = batchdetails.preserving
2082 # -------------------------------------------------------------------------
2083 # Fetch addition, removal, preservation, current PKs in a single query
2084 # -------------------------------------------------------------------------
2085 tablechanges = UploadTableChanges(table)
2086 serverrecs = get_server_live_records(
2087 req, device_id, table, current_only=False
2088 )
2089 for sr in serverrecs:
2090 tablechanges.note_serverrec(sr, preserving=preserving)
2092 # -------------------------------------------------------------------------
2093 # Additions
2094 # -------------------------------------------------------------------------
2095 # Update the records we're adding
2096 addition_pks = tablechanges.addition_pks
2097 if addition_pks:
2098 # log.debug("commit_table: {}, adding server PKs {}",
2099 # tablename, addition_pks)
2100 dbsession.execute(
2101 update(table)
2102 .where(table.c[FN_PK].in_(addition_pks))
2103 .values(
2104 {
2105 FN_CURRENT: 1,
2106 FN_ADDITION_PENDING: 0,
2107 FN_ADDING_USER_ID: user_id,
2108 FN_WHEN_ADDED_EXACT: exacttime,
2109 FN_WHEN_ADDED_BATCH_UTC: batchtime,
2110 }
2111 )
2112 )
2114 # -------------------------------------------------------------------------
2115 # Removals
2116 # -------------------------------------------------------------------------
2117 # Update the records we're removing
2118 removal_pks = tablechanges.removal_pks
2119 if removal_pks:
2120 # log.debug("commit_table: {}, removing server PKs {}",
2121 # tablename, removal_pks)
2122 dbsession.execute(
2123 update(table)
2124 .where(table.c[FN_PK].in_(removal_pks))
2125 .values(values_delete_now(req, batchdetails))
2126 )
2128 # -------------------------------------------------------------------------
2129 # Preservation
2130 # -------------------------------------------------------------------------
2131 # Preserve necessary records
2132 preservation_pks = tablechanges.preservation_pks
2133 if preservation_pks:
2134 # log.debug("commit_table: {}, preserving server PKs {}",
2135 # tablename, preservation_pks)
2136 new_era = batchdetails.new_era
2137 dbsession.execute(
2138 update(table)
2139 .where(table.c[FN_PK].in_(preservation_pks))
2140 .values(values_preserve_now(req, batchdetails))
2141 )
2142 if not preserving:
2143 # Also preserve/finalize any corresponding special notes
2144 # (2015-02-01), just for records being specifically preserved. If
2145 # we are preserving, this step happens in one go in commit_all()
2146 # (2018-11-13).
2147 # noinspection PyUnresolvedReferences
2148 dbsession.execute(
2149 update(SpecialNote.__table__)
2150 .where(SpecialNote.basetable == tablename)
2151 .where(SpecialNote.device_id == device_id)
2152 .where(SpecialNote.era == ERA_NOW)
2153 .where(
2154 exists()
2155 .select_from(table)
2156 .where(table.c[TABLET_ID_FIELD] == SpecialNote.task_id)
2157 .where(table.c[FN_DEVICE_ID] == SpecialNote.device_id)
2158 .where(table.c[FN_ERA] == new_era)
2159 )
2160 # ^^^^^^^^^^^^^^^^^^^^^^^^^^
2161 # This bit restricts to records being preserved.
2162 .values(era=new_era)
2163 )
2165 # -------------------------------------------------------------------------
2166 # Update special indexes
2167 # -------------------------------------------------------------------------
2168 update_indexes_and_push_exports(req, batchdetails, tablechanges)
2170 # -------------------------------------------------------------------------
2171 # Remove individually from list of dirty tables?
2172 # -------------------------------------------------------------------------
2173 if clear_dirty:
2174 # noinspection PyUnresolvedReferences
2175 dbsession.execute(
2176 DirtyTable.__table__.delete()
2177 .where(DirtyTable.device_id == device_id)
2178 .where(DirtyTable.tablename == tablename)
2179 )
2180 # ... otherwise a call to clear_dirty_tables() must be made.
2182 if DEBUG_UPLOAD:
2183 log.debug("commit_table: {}", tablechanges)
2185 return tablechanges
2188def rollback_all(req: "CamcopsRequest") -> None:
2189 """
2190 Rolls back all pending changes for a device.
2191 """
2192 tables = get_dirty_tables(req)
2193 for table in tables:
2194 rollback_table(req, table)
2195 clear_dirty_tables(req)
2198def rollback_table(req: "CamcopsRequest", table: Table) -> None:
2199 """
2200 Rolls back changes for an individual table for a device.
2201 """
2202 device_id = req.tabletsession.device_id
2203 # Pending additions
2204 req.dbsession.execute(
2205 table.delete()
2206 .where(table.c[FN_DEVICE_ID] == device_id)
2207 .where(table.c[FN_ADDITION_PENDING])
2208 )
2209 # Pending deletions
2210 req.dbsession.execute(
2211 update(table)
2212 .where(table.c[FN_DEVICE_ID] == device_id)
2213 .where(table.c[FN_REMOVAL_PENDING])
2214 .values(
2215 {
2216 FN_REMOVAL_PENDING: 0,
2217 FN_WHEN_ADDED_EXACT: None,
2218 FN_WHEN_REMOVED_BATCH_UTC: None,
2219 FN_REMOVING_USER_ID: None,
2220 FN_SUCCESSOR_PK: None,
2221 }
2222 )
2223 )
2224 # Record-specific preservation (set by flag_record_for_preservation())
2225 req.dbsession.execute(
2226 update(table)
2227 .where(table.c[FN_DEVICE_ID] == device_id)
2228 .values({MOVE_OFF_TABLET_FIELD: 0})
2229 )
2232def clear_dirty_tables(req: "CamcopsRequest") -> None:
2233 """
2234 Clears the dirty-table list for a device.
2235 """
2236 device_id = req.tabletsession.device_id
2237 # noinspection PyUnresolvedReferences
2238 req.dbsession.execute(
2239 DirtyTable.__table__.delete().where(DirtyTable.device_id == device_id)
2240 )
2243# =============================================================================
2244# Additional helper functions for one-step upload
2245# =============================================================================
2248def process_table_for_onestep_upload(
2249 req: "CamcopsRequest",
2250 batchdetails: BatchDetails,
2251 table: Table,
2252 clientpk_name: str,
2253 rows: List[Dict[str, Any]],
2254) -> UploadTableChanges:
2255 """
2256 Performs all upload steps for a table.
2258 Note that we arrive here in a specific and safe table order; search for
2259 :func:`camcops_server.cc_modules.cc_client_api_helpers.upload_commit_order_sorter`.
2261 Args:
2262 req: the :class:`camcops_server.cc_modules.cc_request.CamcopsRequest`
2263 batchdetails: the :class:`BatchDetails`
2264 table: an SQLAlchemy :class:`Table`
2265 clientpk_name: the name of the PK field on the client
2266 rows: a list of rows, where each row is a dictionary mapping field
2267 (column) names to values (those values being encoded as SQL-style
2268 literals in our extended syntax)
2270 Returns:
2271 an :class:`UploadTableChanges` object
2272 """ # noqa
2273 serverrecs = get_server_live_records(
2274 req,
2275 req.tabletsession.device_id,
2276 table,
2277 clientpk_name,
2278 current_only=False,
2279 )
2280 servercurrentrecs = [r for r in serverrecs if r.current]
2281 if rows and not clientpk_name:
2282 fail_user_error(
2283 f"Client-side PK name not specified by client for "
2284 f"non-empty table {table.name!r}"
2285 )
2286 tablechanges = UploadTableChanges(table)
2287 server_pks_uploaded = [] # type: List[int]
2288 for row in rows:
2289 valuedict = {k: decode_single_value(v) for k, v in row.items()}
2290 urr = upload_record_core(
2291 req,
2292 batchdetails,
2293 table,
2294 clientpk_name,
2295 valuedict,
2296 server_live_current_records=servercurrentrecs,
2297 )
2298 # ... handles addition, modification, preservation, special processing
2299 # But we also make a note of these for indexing:
2300 if urr.oldserverpk is not None:
2301 server_pks_uploaded.append(urr.oldserverpk)
2302 tablechanges.note_urr(
2303 urr, preserving_new_records=batchdetails.preserving
2304 )
2305 # Which leaves:
2306 # (*) Deletion (where no record was uploaded at all)
2307 server_pks_for_deletion = [
2308 r.server_pk
2309 for r in servercurrentrecs
2310 if r.server_pk not in server_pks_uploaded
2311 ]
2312 if server_pks_for_deletion:
2313 flag_deleted(req, batchdetails, table, server_pks_for_deletion)
2314 tablechanges.note_removal_deleted_pks(server_pks_for_deletion)
2316 # Preserving all records not specifically processed above, too
2317 if batchdetails.preserving:
2318 # Preserve all, including noncurrent:
2319 preserve_all(req, batchdetails, table)
2320 # Note other preserved records, for indexing:
2321 tablechanges.note_preservation_pks(r.server_pk for r in serverrecs)
2323 # (*) Indexing (and push exports)
2324 update_indexes_and_push_exports(req, batchdetails, tablechanges)
2326 if DEBUG_UPLOAD:
2327 log.debug("process_table_for_onestep_upload: {}", tablechanges)
2329 return tablechanges
2332# =============================================================================
2333# Audit functions
2334# =============================================================================
2337def audit(
2338 req: "CamcopsRequest",
2339 details: str,
2340 patient_server_pk: int = None,
2341 tablename: str = None,
2342 server_pk: int = None,
2343) -> None:
2344 """
2345 Audit something.
2346 """
2347 # Add parameters and pass on:
2348 cc_audit.audit(
2349 req=req,
2350 details=details,
2351 patient_server_pk=patient_server_pk,
2352 table=tablename,
2353 server_pk=server_pk,
2354 device_id=req.tabletsession.device_id, # added
2355 remote_addr=req.remote_addr, # added
2356 user_id=req.user_id, # added
2357 from_console=False, # added
2358 from_dbclient=True, # added
2359 )
2362# =============================================================================
2363# Helper functions for single-user mode
2364# =============================================================================
2367def make_single_user_mode_username(
2368 client_device_name: str, patient_pk: int
2369) -> str:
2370 """
2371 Returns the username for single-user mode.
2372 """
2373 return f"user-{client_device_name}-{patient_pk}"
2376def json_patient_info(patient: Patient) -> str:
2377 """
2378 Converts patient details to a string representation of a JSON list (one
2379 patient) containing a single JSON dictionary (detailing that patient), with
2380 keys/formats known to the client.
2382 (One item list to be consistent with patients uploaded from the tablet.)
2384 Args:
2385 patient: :class:`camcops_server.cc_modules.cc_patient.Patient`
2386 """
2387 patient_dict = {
2388 TabletParam.SURNAME: patient.surname,
2389 TabletParam.FORENAME: patient.forename,
2390 TabletParam.SEX: patient.sex,
2391 TabletParam.DOB: format_datetime(
2392 patient.dob, DateFormat.ISO8601_DATE_ONLY
2393 ),
2394 TabletParam.EMAIL: patient.email,
2395 TabletParam.ADDRESS: patient.address,
2396 TabletParam.GP: patient.gp,
2397 TabletParam.OTHER: patient.other,
2398 }
2399 for idnum in patient.idnums:
2400 key = f"{TabletParam.IDNUM_PREFIX}{idnum.which_idnum}"
2401 patient_dict[key] = idnum.idnum_value
2402 # One item list to be consistent with patients uploaded from the tablet
2403 return json.dumps([patient_dict])
2406def get_single_server_patient(req: "CamcopsRequest") -> Patient:
2407 """
2408 Returns the patient identified by the proquint access key present in this
2409 request, or raises.
2410 """
2411 _ = req.gettext
2413 patient_proquint = get_str_var(req, TabletParam.PATIENT_PROQUINT)
2414 assert patient_proquint is not None # For type checker
2416 try:
2417 uuid_obj = uuid_from_proquint(patient_proquint)
2418 except InvalidProquintException:
2419 # Checksum failed or characters in wrong place
2420 # We'll do the same validation on the client so in theory
2421 # should never get here
2422 fail_user_error(
2423 _(
2424 "There is no patient with access key '{access_key}'. "
2425 "Have you entered the key correctly?"
2426 ).format(access_key=patient_proquint)
2427 )
2429 server_device = Device.get_server_device(req.dbsession)
2431 # noinspection PyUnboundLocalVariable,PyProtectedMember
2432 patient = (
2433 req.dbsession.query(Patient)
2434 .filter(
2435 Patient.uuid == uuid_obj,
2436 Patient._device_id == server_device.id,
2437 Patient._era == ERA_NOW,
2438 Patient._current == True, # noqa: E712
2439 )
2440 .options(joinedload(Patient.task_schedules))
2441 .one_or_none()
2442 )
2444 if patient is None:
2445 fail_user_error(
2446 _(
2447 "There is no patient with access key '{access_key}'. "
2448 "Have you entered the key correctly?"
2449 ).format(access_key=patient_proquint)
2450 )
2452 if not patient.idnums:
2453 # In theory should never happen. The patient must be created with at
2454 # least one ID number. We did see this once in testing (possibly when
2455 # a patient created on a device was registered)
2456 _ = req.gettext
2457 fail_server_error(_("Patient has no ID numbers"))
2459 return patient
2462def get_or_create_single_user(
2463 req: "CamcopsRequest", name: str, patient: Patient
2464) -> Tuple[User, str]:
2465 """
2466 Creates a user for a patient (who's using single-user mode).
2468 The user is associated (via its name) with the combination of a client
2469 device and a patient. (If a device is re-registered to another patient, the
2470 username will change.)
2472 If the username already exists, then since we can't look up the password
2473 (it's irreversibly encrypted), we will set it afresh.
2475 - Why is a user associated with a patient? So we can enforce that the user
2476 can upload only data relating to that patient.
2478 - Why is a user associated with a device?
2480 - If it is: then two users (e.g. "Device1-Bob" and "Device2-Bob") can
2481 independently work with the same patient. This will be highly
2482 confusing (mainly because it will allow "double" copies of tasks to be
2483 created, though only by manually entering things twice).
2485 - If it isn't (e.g. user "Bob"): then, because registering the patient on
2486 Device2 will reset the password for the user, registering a new device
2487 for a patient will "take over" from a previous device. That has some
2488 potential for data loss if work was in progress (incomplete tasks won't
2489 be uploadable any more, and re-registering [to fix the password on the
2490 first device] would delete data).
2492 - Since some confusion is better than some data loss, we associate users
2493 with a device/patient combination.
2495 Args:
2496 req:
2497 a :class:`camcops_server.cc_modules.cc_request.CamcopsRequest`
2498 name:
2499 username
2500 patient:
2501 associated :class:`camcops_server.cc_modules.cc_patient.Patient`,
2502 which also tells us the group in which to place this user
2504 Returns:
2505 tuple: :class:`camcops_server.cc_modules.cc_user.User`, password
2507 """
2508 dbsession = req.dbsession
2509 password = random_password()
2510 group = patient.group
2511 assert group is not None # for type checker
2513 user = User.get_user_by_name(dbsession, name)
2514 creating_new_user = user is None
2515 if creating_new_user:
2516 # Create a fresh user.
2517 user = User(username=name)
2518 user.upload_group = group
2519 user.auto_generated = True
2520 user.superuser = False # should be redundant!
2521 # noinspection PyProtectedMember
2522 user.single_patient_pk = patient._pk
2523 user.set_password(req, password)
2524 if creating_new_user:
2525 dbsession.add(user)
2526 # As the username is based on a UUID, we're pretty sure another
2527 # request won't have created the same user, otherwise we'd need
2528 # to catch IntegrityError
2529 dbsession.flush()
2531 membership = UserGroupMembership(user_id=user.id, group_id=group.id)
2532 membership.may_register_devices = True
2533 membership.may_upload = True
2534 user.user_group_memberships = [membership] # ... only these permissions
2536 return user, password
2539def random_password(length: int = 32) -> str:
2540 """
2541 Create a random password.
2542 """
2543 # Not trying anything clever with distributions of letters, digits etc
2544 characters = string.ascii_letters + string.digits + string.punctuation
2545 # We use secrets.choice() rather than random.choices() as it's better
2546 # for security/cryptography purposes.
2547 return "".join(secrets.choice(characters) for _ in range(length))
2550def get_task_schedules(req: "CamcopsRequest", patient: Patient) -> str:
2551 """
2552 Gets a JSON string representation of the task schedules for a specified
2553 patient.
2554 """
2555 dbsession = req.dbsession
2557 schedules = []
2559 for pts in patient.task_schedules:
2560 if pts.start_datetime is None:
2561 # Minutes granularity so we are consistent with the form
2562 pts.start_datetime = req.now_utc.replace(second=0, microsecond=0)
2563 dbsession.add(pts)
2565 items = []
2567 for task_info in pts.get_list_of_scheduled_tasks(req):
2568 due_from = task_info.start_datetime.to_iso8601_string()
2569 due_by = task_info.end_datetime.to_iso8601_string()
2571 complete = False
2572 when_completed = None
2573 task = task_info.task
2574 if task:
2575 complete = task.is_complete()
2576 if complete and task.when_last_modified:
2577 when_completed = (
2578 task.when_last_modified.to_iso8601_string()
2579 )
2581 if pts.settings is not None:
2582 settings = pts.settings.get(task_info.tablename, {})
2583 else:
2584 settings = {}
2586 items.append(
2587 {
2588 TabletParam.TABLE: task_info.tablename,
2589 TabletParam.ANONYMOUS: task_info.is_anonymous,
2590 TabletParam.SETTINGS: settings,
2591 TabletParam.DUE_FROM: due_from,
2592 TabletParam.DUE_BY: due_by,
2593 TabletParam.COMPLETE: complete,
2594 TabletParam.WHEN_COMPLETED: when_completed,
2595 }
2596 )
2598 schedules.append(
2599 {
2600 TabletParam.TASK_SCHEDULE_NAME: pts.task_schedule.name,
2601 TabletParam.TASK_SCHEDULE_ITEMS: items,
2602 }
2603 )
2605 return json.dumps(schedules)
2608# =============================================================================
2609# Action processors: allowed to any user
2610# =============================================================================
2611# If they return None, the framework uses the operation name as the reply in
2612# the success message. Not returning anything is the same as returning None.
2613# Authentication is performed in advance of these.
2616def op_check_device_registered(req: "CamcopsRequest") -> None:
2617 """
2618 Check that a device is registered, or raise
2619 :exc:`UserErrorException`.
2620 """
2621 req.tabletsession.ensure_device_registered()
2624def op_register_patient(req: "CamcopsRequest") -> Dict[str, Any]:
2625 """
2626 Registers a patient. That is, the client provides an access key. If all
2627 is well, the server returns details of that patient, as well as key
2628 server parameters, plus (if required) the username/password to use.
2629 """
2630 # -------------------------------------------------------------------------
2631 # Patient details
2632 # -------------------------------------------------------------------------
2633 patient = get_single_server_patient(req) # may fail/raise
2634 patient_info = json_patient_info(patient)
2635 reply_dict = {TabletParam.PATIENT_INFO: patient_info}
2637 # -------------------------------------------------------------------------
2638 # Username/password
2639 # -------------------------------------------------------------------------
2640 client_device_name = get_str_var(req, TabletParam.DEVICE)
2641 # noinspection PyProtectedMember
2642 user_name = make_single_user_mode_username(client_device_name, patient._pk)
2643 user, password = get_or_create_single_user(req, user_name, patient)
2644 reply_dict[TabletParam.USER] = user.username
2645 reply_dict[TabletParam.PASSWORD] = password
2647 # -------------------------------------------------------------------------
2648 # Intellectual property settings
2649 # -------------------------------------------------------------------------
2650 ip_use = patient.group.ip_use or IpUse()
2651 # ... if the group doesn't have an associated ip_use object, use defaults
2652 ip_dict = {
2653 TabletParam.IP_USE_COMMERCIAL: int(ip_use.commercial),
2654 TabletParam.IP_USE_CLINICAL: int(ip_use.clinical),
2655 TabletParam.IP_USE_EDUCATIONAL: int(ip_use.educational),
2656 TabletParam.IP_USE_RESEARCH: int(ip_use.research),
2657 }
2658 reply_dict[TabletParam.IP_USE_INFO] = json.dumps(ip_dict)
2660 return reply_dict
2663# =============================================================================
2664# Action processors that require REGISTRATION privilege
2665# =============================================================================
2668def op_register_device(req: "CamcopsRequest") -> Dict[str, Any]:
2669 """
2670 Register a device with the server.
2672 Returns:
2673 server information dictionary (from :func:`get_server_id_info`)
2674 """
2675 dbsession = req.dbsession
2676 ts = req.tabletsession
2677 device_friendly_name = get_str_var(
2678 req, TabletParam.DEVICE_FRIENDLY_NAME, mandatory=False
2679 )
2680 # noinspection PyUnresolvedReferences
2681 device_exists = exists_in_table(
2682 dbsession, Device.__table__, Device.name == ts.device_name
2683 )
2684 if device_exists:
2685 # device already registered, but accept re-registration
2686 # noinspection PyUnresolvedReferences
2687 dbsession.execute(
2688 update(Device.__table__)
2689 .where(Device.name == ts.device_name)
2690 .values(
2691 friendly_name=device_friendly_name,
2692 camcops_version=ts.tablet_version_str,
2693 registered_by_user_id=req.user_id,
2694 when_registered_utc=req.now_utc,
2695 )
2696 )
2697 else:
2698 # new registration
2699 try:
2700 # noinspection PyUnresolvedReferences
2701 dbsession.execute(
2702 Device.__table__.insert().values(
2703 name=ts.device_name,
2704 friendly_name=device_friendly_name,
2705 camcops_version=ts.tablet_version_str,
2706 registered_by_user_id=req.user_id,
2707 when_registered_utc=req.now_utc,
2708 )
2709 )
2710 except IntegrityError:
2711 fail_user_error(INSERT_FAILED)
2713 ts.reload_device()
2714 audit(
2715 req,
2716 f"register, device_id={ts.device_id}, "
2717 f"friendly_name={device_friendly_name}",
2718 tablename=Device.__tablename__,
2719 )
2720 return get_server_id_info(req)
2723def op_get_extra_strings(req: "CamcopsRequest") -> Dict[str, str]:
2724 """
2725 Fetch all local extra strings from the server.
2727 Returns:
2728 a SELECT-style reply (see :func:`get_select_reply`) for the
2729 extra-string table
2730 """
2731 fields = [
2732 ExtraStringFieldNames.TASK,
2733 ExtraStringFieldNames.NAME,
2734 ExtraStringFieldNames.LANGUAGE,
2735 ExtraStringFieldNames.VALUE,
2736 ]
2737 rows = req.get_all_extra_strings()
2738 reply = get_select_reply(fields, rows)
2739 audit(req, "get_extra_strings")
2740 return reply
2743# noinspection PyUnusedLocal
2744def op_get_allowed_tables(req: "CamcopsRequest") -> Dict[str, str]:
2745 """
2746 Returns the names of all possible tables on the server, each paired with
2747 the minimum client (tablet) version that will be accepted for each table.
2748 (Typically these are all the same as the minimum global tablet version.)
2750 Uses the SELECT-like syntax (see :func:`get_select_reply`).
2751 """
2752 tables_versions = all_tables_with_min_client_version()
2753 fields = [
2754 AllowedTablesFieldNames.TABLENAME,
2755 AllowedTablesFieldNames.MIN_CLIENT_VERSION,
2756 ]
2757 rows = [[k, str(v)] for k, v in tables_versions.items()]
2758 reply = get_select_reply(fields, rows)
2759 audit(req, "get_allowed_tables")
2760 return reply
2763def op_get_task_schedules(req: "CamcopsRequest") -> Dict[str, str]:
2764 """
2765 Return details of the task schedules for the patient associated with
2766 this request, for single-user mode. Also returns details of the single
2767 patient, in case that's changed.
2768 """
2769 patient = get_single_server_patient(req)
2770 patient_info = json_patient_info(patient)
2771 task_schedules = get_task_schedules(req, patient)
2772 return {
2773 TabletParam.PATIENT_INFO: patient_info,
2774 TabletParam.TASK_SCHEDULES: task_schedules,
2775 }
2778# =============================================================================
2779# Action processors that require UPLOAD privilege
2780# =============================================================================
2782# noinspection PyUnusedLocal
2783def op_check_upload_user_and_device(req: "CamcopsRequest") -> None:
2784 """
2785 Stub function for the operation to check that a user is valid.
2787 To get this far, the user has to be valid, so this function doesn't
2788 actually have to do anything.
2789 """
2790 pass # don't need to do anything!
2793# noinspection PyUnusedLocal
2794def op_get_id_info(req: "CamcopsRequest") -> Dict[str, Any]:
2795 """
2796 Fetch server ID information; see :func:`get_server_id_info`.
2797 """
2798 return get_server_id_info(req)
2801def op_start_upload(req: "CamcopsRequest") -> None:
2802 """
2803 Begin an upload.
2804 """
2805 start_device_upload_batch(req)
2808def op_end_upload(req: "CamcopsRequest") -> None:
2809 """
2810 Ends an upload and commits changes.
2811 """
2812 batchdetails = get_batch_details(req)
2813 # ensure it's the same user finishing as starting!
2814 end_device_upload_batch(req, batchdetails)
2817def op_upload_table(req: "CamcopsRequest") -> str:
2818 """
2819 Upload a table.
2821 Incoming information in the POST request includes a CSV list of fields, a
2822 count of the number of records being provided, and a set of variables named
2823 ``record0`` ... ``record{nrecords - 1}``, each containing a CSV list of
2824 SQL-encoded values.
2826 Typically used for smaller tables, i.e. most except for BLOBs.
2827 """
2828 table = get_table_from_req(req, TabletParam.TABLE)
2830 allowed_nonexistent_fields = [] # type: List[str]
2831 # noinspection PyUnresolvedReferences
2832 if req.tabletsession.cope_with_old_idnums and table == Patient.__table__:
2833 for x in range(1, NUMBER_OF_IDNUMS_DEFUNCT + 1):
2834 allowed_nonexistent_fields.extend(
2835 [
2836 FP_ID_NUM + str(x),
2837 FP_ID_DESC + str(x),
2838 FP_ID_SHORT_DESC + str(x),
2839 ]
2840 )
2842 fields = get_fields_from_post_var(
2843 req,
2844 table,
2845 TabletParam.FIELDS,
2846 allowed_nonexistent_fields=allowed_nonexistent_fields,
2847 )
2848 nrecords = get_int_var(req, TabletParam.NRECORDS)
2850 nfields = len(fields)
2851 if nfields < 1:
2852 fail_user_error(
2853 f"{TabletParam.FIELDS}={nfields}: can't be less than 1"
2854 )
2855 if nrecords < 0:
2856 fail_user_error(
2857 f"{TabletParam.NRECORDS}={nrecords}: can't be less than 0"
2858 )
2860 batchdetails = get_batch_details(req)
2862 ts = req.tabletsession
2863 if ts.explicit_pkname_for_upload_table: # q.v.
2864 # New client: tells us the PK name explicitly.
2865 clientpk_name = get_single_field_from_post_var(
2866 req, table, TabletParam.PKNAME
2867 )
2868 else:
2869 # Old client. Either (a) old Titanium client, in which the client PK
2870 # is in fields[0], or (b) an early C++ client, in which there was no
2871 # guaranteed order (and no explicit PK name was sent). However, in
2872 # either case, the client PK name was (is) always "id".
2873 clientpk_name = TABLET_ID_FIELD
2874 ensure_valid_field_name(table, clientpk_name)
2875 server_pks_uploaded = [] # type: List[int]
2876 n_new = 0
2877 n_modified = 0
2878 n_identical = 0
2879 dirty = False
2880 serverrecs = get_server_live_records(
2881 req,
2882 ts.device_id,
2883 table,
2884 clientpk_name=clientpk_name,
2885 current_only=True,
2886 )
2887 for r in range(nrecords):
2888 recname = TabletParam.RECORD_PREFIX + str(r)
2889 values = get_values_from_post_var(req, recname)
2890 nvalues = len(values)
2891 if nvalues != nfields:
2892 errmsg = (
2893 f"Number of fields in field list ({nfields}) doesn't match "
2894 f"number of values in record {r} ({nvalues})"
2895 )
2896 log.warning(errmsg + f"\nfields: {fields!r}\nvalues: {values!r}")
2897 fail_user_error(errmsg)
2898 valuedict = dict(zip(fields, values))
2899 # log.debug("table {!r}, record {}: {!r}", table.name, r, valuedict)
2900 # CORE: CALLS upload_record_core
2901 urr = upload_record_core(
2902 req,
2903 batchdetails,
2904 table,
2905 clientpk_name,
2906 valuedict,
2907 server_live_current_records=serverrecs,
2908 )
2909 if urr.oldserverpk is not None: # was an existing record
2910 server_pks_uploaded.append(urr.oldserverpk)
2911 if urr.newserverpk is None:
2912 n_identical += 1
2913 else:
2914 n_modified += 1
2915 else: # entirely new
2916 n_new += 1
2917 if urr.dirty:
2918 dirty = True
2920 # Now deal with any ABSENT (not in uploaded data set) conditions.
2921 server_pks_for_deletion = [
2922 r.server_pk
2923 for r in serverrecs
2924 if r.server_pk not in server_pks_uploaded
2925 ]
2926 # Note that "deletion" means "end of the line"; records that are modified
2927 # and replaced were handled by upload_record_core().
2928 n_deleted = len(server_pks_for_deletion)
2929 if n_deleted > 0:
2930 flag_deleted(req, batchdetails, table, server_pks_for_deletion)
2932 # Set dirty/clean status
2933 if (
2934 dirty
2935 or n_new > 0
2936 or n_modified > 0
2937 or n_deleted > 0
2938 or any(sr.move_off_tablet for sr in serverrecs)
2939 ):
2940 # ... checks on n_new and n_modified are redundant; dirty will be True
2941 mark_table_dirty(req, table)
2942 elif batchdetails.preserving and not serverrecs:
2943 # We've scanned this table, and there would be no work to do to
2944 # preserve records from previous uploads.
2945 mark_table_clean(req, table)
2947 # Special for old tablets:
2948 # noinspection PyUnresolvedReferences
2949 if req.tabletsession.cope_with_old_idnums and table == Patient.__table__:
2950 # noinspection PyUnresolvedReferences
2951 mark_table_dirty(req, PatientIdNum.__table__)
2952 # Mark patient ID numbers for deletion if their parent Patient is
2953 # similarly being marked for deletion
2954 # noinspection PyUnresolvedReferences,PyProtectedMember
2955 req.dbsession.execute(
2956 update(PatientIdNum.__table__)
2957 .where(PatientIdNum._device_id == Patient._device_id)
2958 .where(PatientIdNum._era == ERA_NOW)
2959 .where(PatientIdNum.patient_id == Patient.id)
2960 .where(Patient._pk.in_(server_pks_for_deletion))
2961 .where(Patient._era == ERA_NOW) # shouldn't be in doubt!
2962 .values(_removal_pending=1, _successor_pk=None)
2963 )
2965 # Auditing occurs at commit_all.
2966 log.info(
2967 "Upload successful; {n} records uploaded to table {t} "
2968 "({new} new, {mod} modified, {i} identical, {nd} deleted)",
2969 n=nrecords,
2970 t=table.name,
2971 new=n_new,
2972 mod=n_modified,
2973 i=n_identical,
2974 nd=n_deleted,
2975 )
2976 return f"Table {table.name} upload successful"
2979def op_upload_record(req: "CamcopsRequest") -> str:
2980 """
2981 Upload an individual record. (Typically used for BLOBs.)
2982 Incoming POST information includes a CSV list of fields and a CSV list of
2983 values.
2984 """
2985 batchdetails = get_batch_details(req)
2986 table = get_table_from_req(req, TabletParam.TABLE)
2987 clientpk_name = get_single_field_from_post_var(
2988 req, table, TabletParam.PKNAME
2989 )
2990 valuedict = get_fields_and_values(
2991 req, table, TabletParam.FIELDS, TabletParam.VALUES
2992 )
2993 urr = upload_record_core(
2994 req, batchdetails, table, clientpk_name, valuedict
2995 )
2996 if urr.dirty:
2997 mark_table_dirty(req, table)
2998 if urr.oldserverpk is None:
2999 log.info("upload-insert")
3000 return "UPLOAD-INSERT"
3001 else:
3002 if urr.newserverpk is None:
3003 log.info("upload-update: skipping existing record")
3004 else:
3005 log.info("upload-update")
3006 return "UPLOAD-UPDATE"
3007 # Auditing occurs at commit_all.
3010def op_upload_empty_tables(req: "CamcopsRequest") -> str:
3011 """
3012 The tablet supplies a list of tables that are empty at its end, and we
3013 will 'wipe' all appropriate tables; this reduces the number of HTTP
3014 requests.
3015 """
3016 tables = get_tables_from_post_var(req, TabletParam.TABLES)
3017 batchdetails = get_batch_details(req)
3018 to_dirty = [] # type: List[Table]
3019 to_clean = [] # type: List[Table]
3020 for table in tables:
3021 nrows_affected = flag_all_records_deleted(req, table)
3022 if nrows_affected > 0:
3023 to_dirty.append(table)
3024 elif batchdetails.preserving:
3025 # There are no records in the current era for this device.
3026 to_clean.append(table)
3027 # In the fewest number of queries:
3028 mark_tables_dirty(req, to_dirty)
3029 mark_tables_clean(req, to_clean)
3030 log.info("upload_empty_tables")
3031 # Auditing occurs at commit_all.
3032 return "UPLOAD-EMPTY-TABLES"
3035def op_start_preservation(req: "CamcopsRequest") -> str:
3036 """
3037 Marks this upload batch as one in which all records will be preserved
3038 (i.e. moved from NOW-era to an older era, so they can be deleted safely
3039 from the tablet).
3041 Without this, individual records can still be marked for preservation if
3042 their MOVE_OFF_TABLET_FIELD field (``_move_off_tablet``) is set; see
3043 :func:`upload_record` and the functions it calls.
3044 """
3045 get_batch_details(req)
3046 start_preserving(req)
3047 log.info("start_preservation successful")
3048 # Auditing occurs at commit_all.
3049 return "STARTPRESERVATION"
3052def op_delete_where_key_not(req: "CamcopsRequest") -> str:
3053 """
3054 Marks records for deletion, for a device/table, where the client PK
3055 is not in a specified list.
3056 """
3057 table = get_table_from_req(req, TabletParam.TABLE)
3058 clientpk_name = get_single_field_from_post_var(
3059 req, table, TabletParam.PKNAME
3060 )
3061 clientpk_values = get_values_from_post_var(req, TabletParam.PKVALUES)
3063 get_batch_details(req)
3064 flag_deleted_where_clientpk_not(req, table, clientpk_name, clientpk_values)
3065 # Auditing occurs at commit_all.
3066 # log.info("delete_where_key_not successful; table {} trimmed", table)
3067 return "Trimmed"
3070def op_which_keys_to_send(req: "CamcopsRequest") -> str:
3071 """
3072 Intended use: "For my device, and a specified table, here are my client-
3073 side PKs (as a CSV list), and the modification dates for each corresponding
3074 record (as a CSV list). Please tell me which records have mismatching dates
3075 on the server, i.e. those that I need to re-upload."
3077 Used particularly for BLOBs, to reduce traffic, i.e. so we don't have to
3078 send a lot of BLOBs.
3080 Note new ``TabletParam.MOVE_OFF_TABLET_VALUES`` parameter in server v2.3.0,
3081 with bugfix for pre-2.3.0 clients that won't send this; see changelog.
3082 """
3083 # -------------------------------------------------------------------------
3084 # Get details
3085 # -------------------------------------------------------------------------
3086 try:
3087 table = get_table_from_req(req, TabletParam.TABLE)
3088 except IgnoringAntiqueTableException:
3089 raise IgnoringAntiqueTableException("")
3090 clientpk_name = get_single_field_from_post_var(
3091 req, table, TabletParam.PKNAME
3092 )
3093 clientpk_values = get_values_from_post_var(
3094 req, TabletParam.PKVALUES, mandatory=False
3095 )
3096 # ... should be autoconverted to int, but we check below
3097 client_dates = get_values_from_post_var(
3098 req, TabletParam.DATEVALUES, mandatory=False
3099 )
3100 # ... will be in string format
3102 npkvalues = len(clientpk_values)
3103 ndatevalues = len(client_dates)
3104 if npkvalues != ndatevalues:
3105 fail_user_error(
3106 f"Number of PK values ({npkvalues}) doesn't match number of dates "
3107 f"({ndatevalues})"
3108 )
3110 # v2.3.0:
3111 move_off_tablet_values = [] # type: List[int] # for type checker
3112 if req.has_param(TabletParam.MOVE_OFF_TABLET_VALUES):
3113 client_reports_move_off_tablet = True
3114 move_off_tablet_values = get_values_from_post_var(
3115 req, TabletParam.MOVE_OFF_TABLET_VALUES, mandatory=True
3116 )
3117 # ... should be autoconverted to int
3118 n_motv = len(move_off_tablet_values)
3119 if n_motv != npkvalues:
3120 fail_user_error(
3121 f"Number of move-off-tablet values ({n_motv}) doesn't match "
3122 f"number of PKs ({npkvalues})"
3123 )
3124 try:
3125 move_off_tablet_values = [bool(x) for x in move_off_tablet_values]
3126 except (TypeError, ValueError):
3127 fail_user_error(
3128 f"Bad move-off-tablet values: {move_off_tablet_values!r}"
3129 )
3130 else:
3131 client_reports_move_off_tablet = False
3132 log.warning(
3133 "op_which_keys_to_send: old client not reporting "
3134 "{}; requesting all records",
3135 TabletParam.MOVE_OFF_TABLET_VALUES,
3136 )
3138 clientinfo = [] # type: List[WhichKeyToSendInfo]
3140 for i in range(npkvalues):
3141 cpkv = clientpk_values[i]
3142 if not isinstance(cpkv, int):
3143 fail_user_error(f"Bad (non-integer) client PK: {cpkv!r}")
3144 dt = None # for type checker
3145 try:
3146 dt = coerce_to_pendulum(client_dates[i])
3147 if dt is None:
3148 fail_user_error(f"Missing date/time for client PK {cpkv}")
3149 except ValueError:
3150 fail_user_error(f"Bad date/time: {client_dates[i]!r}")
3151 clientinfo.append(
3152 WhichKeyToSendInfo(
3153 client_pk=cpkv,
3154 client_when=dt,
3155 client_move_off_tablet=(
3156 move_off_tablet_values[i]
3157 if client_reports_move_off_tablet
3158 else False
3159 ),
3160 )
3161 )
3163 # -------------------------------------------------------------------------
3164 # Work out the answer
3165 # -------------------------------------------------------------------------
3166 batchdetails = get_batch_details(req)
3168 # 1. The client sends us all its PKs. So "delete" anything not in that
3169 # list.
3170 flag_deleted_where_clientpk_not(req, table, clientpk_name, clientpk_values)
3172 # 2. See which ones are new or updates.
3173 client_pks_needed = [] # type: List[int]
3174 client_pk_to_serverrec = client_pks_that_exist(
3175 req, table, clientpk_name, clientpk_values
3176 )
3177 for wk in clientinfo:
3178 if client_reports_move_off_tablet:
3179 if wk.client_pk not in client_pk_to_serverrec:
3180 # New on the client; we want it
3181 client_pks_needed.append(wk.client_pk)
3182 else:
3183 # We know about some version of this client record.
3184 serverrec = client_pk_to_serverrec[wk.client_pk]
3185 if serverrec.server_when != wk.client_when:
3186 # Modified on the client; we want it
3187 client_pks_needed.append(wk.client_pk)
3188 elif serverrec.move_off_tablet != wk.client_move_off_tablet:
3189 # Not modified on the client. But it is being preserved.
3190 # We don't need to ask the client for it again, but we do
3191 # need to mark the preservation.
3192 flag_record_for_preservation(
3193 req, batchdetails, table, serverrec.server_pk
3194 )
3196 else:
3197 # Client hasn't told us about the _move_off_tablet flag. Always
3198 # request the record (workaround potential bug in old clients).
3199 client_pks_needed.append(wk.client_pk)
3201 # Success
3202 pk_csv_list = ",".join(
3203 [str(x) for x in client_pks_needed if x is not None]
3204 )
3205 # log.info("which_keys_to_send successful: table {}", table.name)
3206 return pk_csv_list
3209def op_validate_patients(req: "CamcopsRequest") -> str:
3210 """
3211 As of v2.3.0, the client can use this command to validate patients against
3212 arbitrary server criteria -- definitely the upload/finalize ID policies,
3213 but potentially also other criteria of the server's (like matching against
3214 a bank of predefined patients).
3216 Compare ``NetworkManager::getPatientInfoJson()`` on the client.
3218 There is a slight weakness with respect to "single-patient" users, in that
3219 the client *asks* if the patients are OK (rather than the server
3220 *enforcing* that they are OK, via hooks into :func:`op_upload_table`,
3221 :func:`op_upload_record`, :func:`op_upload_entire_database` -- made more
3222 complex because ID numbers are not uploaded to the same table...). In
3223 principle, the weakness is that a user could (a) crack their assigned
3224 password and (b) rework the CamCOPS client, in order to upload "bad"
3225 patient data into their assigned group.
3227 todo:
3228 address this by having the server *require* patient validation for
3229 all uploads?
3231 """
3232 pt_json_list = get_json_from_post_var(
3233 req,
3234 TabletParam.PATIENT_INFO,
3235 decoder=PATIENT_INFO_JSON_DECODER,
3236 mandatory=True,
3237 )
3238 if not isinstance(pt_json_list, list):
3239 fail_user_error("Top-level JSON is not a list")
3240 group = Group.get_group_by_id(req.dbsession, req.user.upload_group_id)
3241 for pt_dict in pt_json_list:
3242 ensure_valid_patient_json(req, group, pt_dict)
3243 return SUCCESS_MSG
3246def op_upload_entire_database(req: "CamcopsRequest") -> str:
3247 """
3248 Perform a one-step upload of the entire database.
3250 - From v2.3.0.
3251 - Therefore, we do not have to cope with old-style ID numbers.
3252 """
3253 # Roll back and clear any outstanding changes
3254 clear_device_upload_batch(req)
3256 # Fetch the JSON, with sanity checks
3257 preserving = get_bool_int_var(req, TabletParam.FINALIZING)
3258 pknameinfo = get_json_from_post_var(
3259 req, TabletParam.PKNAMEINFO, decoder=DB_JSON_DECODER, mandatory=True
3260 )
3261 if not isinstance(pknameinfo, dict):
3262 fail_user_error("PK name info JSON is not a dict")
3263 dbdata = get_json_from_post_var(
3264 req, TabletParam.DBDATA, decoder=DB_JSON_DECODER, mandatory=True
3265 )
3266 if not isinstance(dbdata, dict):
3267 fail_user_error("Database data JSON is not a dict")
3269 # Sanity checks
3270 dbdata_tablenames = sorted(dbdata.keys())
3271 pkinfo_tablenames = sorted(pknameinfo.keys())
3272 if pkinfo_tablenames != dbdata_tablenames:
3273 fail_user_error("Table names don't match from (1) DB data (2) PK info")
3274 duff_tablenames = sorted(
3275 list(set(dbdata_tablenames) - set(CLIENT_TABLE_MAP.keys()))
3276 )
3277 if duff_tablenames:
3278 fail_user_error(
3279 f"Attempt to upload nonexistent tables: {duff_tablenames!r}"
3280 )
3282 # Perform the upload
3283 batchdetails = BatchDetails(
3284 req.now_utc, preserving=preserving, onestep=True
3285 ) # NB special "onestep" option
3286 # Process the tables in a certain order:
3287 tables = sorted(CLIENT_TABLE_MAP.values(), key=upload_commit_order_sorter)
3288 changelist = [] # type: List[UploadTableChanges]
3289 for table in tables:
3290 clientpk_name = pknameinfo.get(table.name, "")
3291 rows = dbdata.get(table.name, [])
3292 tablechanges = process_table_for_onestep_upload(
3293 req, batchdetails, table, clientpk_name, rows
3294 )
3295 changelist.append(tablechanges)
3297 # Audit
3298 audit_upload(req, changelist)
3300 # Done
3301 return SUCCESS_MSG
3304# =============================================================================
3305# Action maps
3306# =============================================================================
3309class Operations:
3310 """
3311 Constants giving the name of operations (commands) accepted by this API.
3312 """
3314 CHECK_DEVICE_REGISTERED = "check_device_registered"
3315 CHECK_UPLOAD_USER_DEVICE = "check_upload_user_and_device"
3316 DELETE_WHERE_KEY_NOT = "delete_where_key_not"
3317 END_UPLOAD = "end_upload"
3318 GET_ALLOWED_TABLES = "get_allowed_tables" # v2.2.0
3319 GET_EXTRA_STRINGS = "get_extra_strings"
3320 GET_ID_INFO = "get_id_info"
3321 GET_TASK_SCHEDULES = "get_task_schedules" # v2.4.0
3322 REGISTER = "register"
3323 REGISTER_PATIENT = "register_patient" # v2.4.0
3324 START_PRESERVATION = "start_preservation"
3325 START_UPLOAD = "start_upload"
3326 UPLOAD_EMPTY_TABLES = "upload_empty_tables"
3327 UPLOAD_ENTIRE_DATABASE = "upload_entire_database" # v2.3.0
3328 UPLOAD_RECORD = "upload_record"
3329 UPLOAD_TABLE = "upload_table"
3330 VALIDATE_PATIENTS = "validate_patients" # v2.3.0
3331 WHICH_KEYS_TO_SEND = "which_keys_to_send"
3334OPERATIONS_ANYONE = {
3335 Operations.CHECK_DEVICE_REGISTERED: op_check_device_registered,
3336 # Anyone can register a patient provided they have the right unique code
3337 Operations.REGISTER_PATIENT: op_register_patient,
3338}
3339OPERATIONS_REGISTRATION = {
3340 Operations.GET_ALLOWED_TABLES: op_get_allowed_tables, # v2.2.0
3341 Operations.GET_EXTRA_STRINGS: op_get_extra_strings,
3342 Operations.GET_TASK_SCHEDULES: op_get_task_schedules,
3343 Operations.REGISTER: op_register_device,
3344}
3345OPERATIONS_UPLOAD = {
3346 Operations.CHECK_UPLOAD_USER_DEVICE: op_check_upload_user_and_device,
3347 Operations.DELETE_WHERE_KEY_NOT: op_delete_where_key_not,
3348 Operations.END_UPLOAD: op_end_upload,
3349 Operations.GET_ID_INFO: op_get_id_info,
3350 Operations.START_PRESERVATION: op_start_preservation,
3351 Operations.START_UPLOAD: op_start_upload,
3352 Operations.UPLOAD_EMPTY_TABLES: op_upload_empty_tables,
3353 Operations.UPLOAD_ENTIRE_DATABASE: op_upload_entire_database,
3354 Operations.UPLOAD_RECORD: op_upload_record,
3355 Operations.UPLOAD_TABLE: op_upload_table,
3356 Operations.VALIDATE_PATIENTS: op_validate_patients, # v2.3.0
3357 Operations.WHICH_KEYS_TO_SEND: op_which_keys_to_send,
3358}
3361# =============================================================================
3362# Client API main functions
3363# =============================================================================
3366def main_client_api(req: "CamcopsRequest") -> Dict[str, str]:
3367 """
3368 Main HTTP processor.
3370 For success, returns a dictionary to send (will use status '200 OK')
3371 For failure, raises an exception.
3372 """
3373 # log.info("CamCOPS database script starting at {}",
3374 # format_datetime(req.now, DateFormat.ISO8601))
3375 ts = req.tabletsession
3376 fn = None
3378 if ts.operation in OPERATIONS_ANYONE:
3379 fn = OPERATIONS_ANYONE.get(ts.operation)
3381 elif ts.operation in OPERATIONS_REGISTRATION:
3382 ts.ensure_valid_user_for_device_registration()
3383 fn = OPERATIONS_REGISTRATION.get(ts.operation)
3385 elif ts.operation in OPERATIONS_UPLOAD:
3386 ts.ensure_valid_device_and_user_for_uploading()
3387 fn = OPERATIONS_UPLOAD.get(ts.operation)
3389 if not fn:
3390 fail_unsupported_operation(ts.operation)
3391 result = fn(req)
3392 if result is None:
3393 # generic success
3394 result = {TabletParam.RESULT: ts.operation}
3395 elif not isinstance(result, dict):
3396 # convert strings (etc.) to a dictionary
3397 result = {TabletParam.RESULT: result}
3398 return result
3401@view_config(
3402 route_name=Routes.CLIENT_API,
3403 request_method=HttpMethod.POST,
3404 permission=NO_PERMISSION_REQUIRED,
3405)
3406@view_config(
3407 route_name=Routes.CLIENT_API_ALIAS,
3408 request_method=HttpMethod.POST,
3409 permission=NO_PERMISSION_REQUIRED,
3410)
3411def client_api(req: "CamcopsRequest") -> Response:
3412 """
3413 View for client API. All tablet interaction comes through here.
3414 Wraps :func:`main_client_api`.
3416 Internally, replies are managed as dictionaries.
3417 For the final reply, the dictionary is converted to text in this format:
3419 .. code-block:: none
3421 k1:v1
3422 k2:v2
3423 k3:v3
3424 ...
3425 """
3426 # log.debug("{!r}", req.environ)
3427 # log.debug("{!r}", req.params)
3428 t0 = time.time() # in seconds
3430 try:
3431 resultdict = main_client_api(req)
3432 resultdict[TabletParam.SUCCESS] = SUCCESS_CODE
3433 status = "200 OK"
3435 except IgnoringAntiqueTableException as e:
3436 log.warning(IGNORING_ANTIQUE_TABLE_MESSAGE)
3437 resultdict = {
3438 TabletParam.RESULT: escape_newlines(str(e)),
3439 TabletParam.SUCCESS: SUCCESS_CODE,
3440 }
3441 status = "200 OK"
3443 except UserErrorException as e:
3444 log.warning("CLIENT-SIDE SCRIPT ERROR: {}", e)
3445 resultdict = {
3446 TabletParam.SUCCESS: FAILURE_CODE,
3447 TabletParam.ERROR: escape_newlines(str(e)),
3448 }
3449 status = "200 OK"
3451 except ServerErrorException as e:
3452 log.error("SERVER-SIDE SCRIPT ERROR: {}", e)
3453 # rollback? Not sure
3454 resultdict = {
3455 TabletParam.SUCCESS: FAILURE_CODE,
3456 TabletParam.ERROR: escape_newlines(str(e)),
3457 }
3458 status = "503 Database Unavailable: " + str(e)
3460 except Exception as e:
3461 # All other exceptions. May include database write failures.
3462 # Let's return with status '200 OK'; though this seems dumb, it means
3463 # the tablet user will at least see the message.
3464 log.exception("Unhandled exception") # + traceback.format_exc()
3465 resultdict = {
3466 TabletParam.SUCCESS: FAILURE_CODE,
3467 TabletParam.ERROR: escape_newlines(exception_description(e)),
3468 }
3469 status = "200 OK"
3471 # Add session token information
3472 ts = req.tabletsession
3473 resultdict[TabletParam.SESSION_ID] = ts.session_id
3474 resultdict[TabletParam.SESSION_TOKEN] = ts.session_token
3476 # Convert dictionary to text in name-value pair format
3477 txt = "".join(f"{k}:{v}\n" for k, v in resultdict.items())
3479 t1 = time.time()
3480 log.debug("Time in script (s): {t}", t=t1 - t0)
3482 return TextResponse(txt, status=status)