Coverage for cc_modules/client_api.py : 17%

Hot-keys on this page
r m x p toggle line displays
j k next/prev highlighted chunk
0 (zero) top of page
1 (one) first highlighted chunk
1#!/usr/bin/env python
3"""
4camcops_server/cc_modules/client_api.py
6===============================================================================
8 Copyright (C) 2012-2020 Rudolf Cardinal (rudolf@pobox.com).
10 This file is part of CamCOPS.
12 CamCOPS is free software: you can redistribute it and/or modify
13 it under the terms of the GNU General Public License as published by
14 the Free Software Foundation, either version 3 of the License, or
15 (at your option) any later version.
17 CamCOPS is distributed in the hope that it will be useful,
18 but WITHOUT ANY WARRANTY; without even the implied warranty of
19 MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
20 GNU General Public License for more details.
22 You should have received a copy of the GNU General Public License
23 along with CamCOPS. If not, see <https://www.gnu.org/licenses/>.
25===============================================================================
27**Implements the API through which client devices (tablets etc.) upload and
28download data.**
30We use primarily SQLAlchemy Core here (in contrast to the ORM used elsewhere).
32This code is optimized to a degree for speed over clarity, aiming primarily to
33reduce the number of database hits.
35**The overall upload method is as follows**
37Everything that follows refers to records relating to a specific client device
38in the "current" era, only.
40In the preamble, the client:
42- verifies authorization via :func:`op_check_device_registered` and
43 :func:`op_check_upload_user_and_device`;
44- fetches and checks server ID information via :func:`op_get_id_info`;
45- checks its patients are acceptable via :func:`op_validate_patients`;
46- checks which tables are permitted via :func:`op_get_allowed_tables`;
47- performs some internal validity checks.
49Then, in the usual stepwise upload:
51- :func:`op_start_upload`
53 - Rolls back any previous incomplete changes via :func:`rollback_all`.
54 - Creates an upload batch, via :func:`get_batch_details_start_if_needed`.
56- If were are in a preserving/finalizing upload: :func:`op_start_preservation`.
58 - Marks all tables as dirty.
59 - Marks the upload batch as a "preserving" batch.
61- Then call some or all of:
63 - For tables that are empty on the client, :func:`op_upload_empty_tables`.
65 - Current client records are marked as ``_removal_pending``.
66 - Any table that had previous client records is marked as dirty.
67 - If preserving, any table without current records is marked as clean.
69 - For tables that the client wishes to send in one go,
70 :func:`op_upload_table`.
72 - Find current server records.
73 - Use :func:`upload_record_core` to add new records and modify existing
74 ones, and :func:`flag_deleted` to delete ones that weren't on the client.
75 - If any records are new, modified, or deleted, mark the table as dirty.
76 - If preserving and there were no server records in this table, mark the
77 table as clean.
79 - For tables (e.g. BLOBs) that might be too big to send in one go:
81 - client sends PKs to :func:`op_delete_where_key_not`, which "deletes" all
82 other records, via :func:`flag_deleted_where_clientpk_not`.
83 - client sends PK and timestamp values to :func:`op_which_keys_to_send`
84 - server "deletes" records that are not in the list (via
85 :func:`flag_deleted_where_clientpk_not`, which marks the table as dirty
86 if any records were thus modified). Note REDUNDANCY here re
87 :func:`op_delete_where_key_not`.
88 - server tells the client which records are new or need to be updated
89 - client sends each of those via :func:`op_upload_record`
91 - Calls :func`upload_record_core`.
92 - Marks the table as dirty, unless the client erroneously sent an
93 unchanged record.
95- In addition, specific records can be marked as ``_move_off_tablet``.
97 - :func:`upload_record_core` checks this for otherwise "identical" records
98 and applies that flag to the server.
100- When the client's finished, it calls :func:`op_end_upload`.
102 - Calls :func:`commit_all`;
103 - ... which, for all dirty tables, calls :func:`commit_table`;
104 - ... which executes the "add", "remove", and "preserve" functions for the
105 table;
106 - ... and triggers the updating of special server indexes on patient ID
107 numbers and tasks, via :func:`update_indexes`.
108 - At the end, :func:`commit_all` clears the dirty-table list.
110There's a little bit of special code to handle old tablet software, too.
112As of v2.3.0, the function :func:`op_upload_entire_database` does this in one
113step (faster; for use if the network packets are not excessively large).
115- Code relating to this uses ``batchdetails.onestep``.
117**Setup for the upload code**
119- Fire up a CamCOPS client with an empty database, e.g. from the build
120 directory via
122 .. code-block:: bash
124 ./camcops --dbdir ~/tmp/camcops_client_test
126- Fire up a web browser showing both (a) the task list via the index, and (b)
127 the task list without using the index. We'll use this to verify correct
128 indexing. **The two versions of the view should never be different.**
130- Ensure the test client device has no current records (force-finalize if
131 required).
133- Ensure the server's index is proper. Run ``camcops_server reindex`` if
134 required.
136- If required, fire up MySQL with the server database. You may wish to use
137 ``pager less -SFX``, for better display of large tables.
139**Testing the upload code**
141Perform the following steps both (1) with the client forced to the stepwise
142upload method, and (2) with it forced to one-step upload.
144Note that the number of patient ID numbers uploaded (etc.) is ignored below.
146*Single record*
148[Checked for one-step and multi-step upload, 2018-11-21.]
150#. Create a blank ReferrerSatisfactionSurvey (table ``ref_satis_gen``).
151 This has the advantage of being an anonymous single-record task.
153#. Upload/copy.
155 - The server log should show 1 × ref_satis_gen added.
157 - The task lists should show the task as current and incomplete.
159#. Modify it, so it's complete.
161#. Upload/copy.
163 - The server log should show 1 × ref_satis_gen added, 1 × ref_satis_gen
164 modified out.
166 - The task lists should show the task as current and complete.
168#. Upload/move.
170 - The server log should show 2 × ref_satis_gen preserved.
172 - The task lists should show the task as no longer current.
174#. Create another blank one.
176#. Upload/copy.
178#. Modify it so it's complete.
180#. Specifically flag it for preservation (the chequered flags).
182#. Upload/copy.
184 - The server log should show 1 × ref_satis_gen added, 1 × ref_satis_gen
185 modified out, 2 × ref_satis_gen preserved.
187 - The task lists should show the task as complete and no longer current.
189*With a patient*
191[Checked for one-step and multi-step upload, 2018-11-21.]
193#. Create a dummy patient that the server will accept.
195#. Create a Progress Note with location "loc1" and abort its creation, giving
196 an incomplete task.
198#. Create a second Progress Note with location "loc2" and contents "note2".
200#. Create a third Progress Note with location "loc3" and contents "note3".
202#. Upload/copy. Verify. This checks *addition*.
204 - The server log should show 1 × patient added; 3 × progressnote added.
205 (Also however many patientidnum records you chose.)
206 - All three tasks should be "current".
207 - The first should be "incomplete".
209#. Modify the first note by adding contents "note1".
211#. Delete the second note.
213#. Upload/copy. Verify. This checks *modification*, *deletion*,
214 *no-change detection*, and *reindexing*.
216 - The server log should show 1 × progressnote added, 1 × progressnote
217 modified out, 1 × progressnote deleted.
218 - The first note should now appear as complete.
219 - The second should have vanished.
220 - The third should be unchanged.
221 - The two remaining tasks should still be "current".
223#. Delete the contents from the first note again.
225#. Upload/move (or move-keeping-patients; that's only different on the
226 client side). Verify. This checks *preservation (finalizing)* and
227 *reindexing*.
229 - The server log should show 1 × patient preserved; 1 × progressnote added,
230 1 × progressnote modified out, 5 × progressnote preserved.
231 - The two remaining tasks should no longer be "current".
232 - The first should no longer be "complete".
234#. Create a complete "note 4" and an incomplete "note 5".
236#. Upload/copy.
238#. Force-finalize from the server. This tests force-finalizing including
239 reindexing.
241 - The "tasks to finalize" list should have just two tasks in it.
242 - After force-finalizing, the tasks should remain in the index but no
243 longer be marked as current.
245#. Upload/move to get rid of the residual tasks on the client.
247 - The server log should show 1 × patient added, 1 × patient preserved; 2 ×
248 progressnote added, 2 × progressnote preserved.
250*With ancillary tables and BLOBs*
252[Checked for one-step and multi-step upload, 2018-11-21.]
254#. Create a PhotoSequence with text "t1", one photo named "p1" of you holding
255 up one finger vertically, and another photo named "p2" of you holding up
256 two fingers vertically.
258#. Upload/copy.
260 - The server log should show:
262 - blobs: 2 × added;
263 - patient: 1 × added;
264 - photosequence: 1 × added;
265 - photosequence_photos: 2 × added.
267 - The task lists should look sensible.
269#. Clear the second photo and replace it with a photo of you holding up
270 two fingers horizontally.
272#. Upload/copy.
274 - The server log should show:
276 - blobs: 1 × added, 1 × modified out;
277 - photosequence: 1 × added, 1 × modified out;
278 - photosequence_photos: 1 × added, 1 × modified out.
280 - The task lists should look sensible.
282#. Back to two fingers vertically. (This is the fourth photo overall.)
284#. Mark that patient for specific finalization.
286#. Upload/copy.
288 - The server log should show:
290 - blobs: 1 × added, 1 × modified out, 4 × preserved;
291 - patient: 1 × preserved;
292 - photosequence: 1 × added, 1 × modified out, 3 × preserved;
293 - photosequence_photos: 1 × added, 1 × modified out, 4 × preserved.
295 - The tasks should no longer be current.
296 - A fresh "vertical fingers" photo should be visible.
298#. Create another patient and another PhotoSequence with one photo of three
299 fingers.
301#. Upload-copy.
303#. Force-finalize.
305 - Should finalize: 1 × blobs, 1 × patient, 1 × photosequence, 1 ×
306 photosequence_photos.
308#. Upload/move.
310During any MySQL debugging, remember:
312.. code-block:: none
314 -- For better display:
315 pager less -SFX;
317 -- To view relevant parts of the BLOB table without the actual BLOB:
319 SELECT
320 _pk, _group_id, _device_id, _era,
321 _current, _predecessor_pk, _successor_pk,
322 _addition_pending, _when_added_batch_utc, _adding_user_id,
323 _removal_pending, _when_removed_batch_utc, _removing_user_id,
324 _move_off_tablet,
325 _preserving_user_id, _forcibly_preserved,
326 id, tablename, tablepk, fieldname, mimetype, when_last_modified
327 FROM blobs;
329"""
331# =============================================================================
332# Imports
333# =============================================================================
335import logging
336import json
337# from pprint import pformat
338import secrets
339import string
340import time
341from typing import (
342 Any,
343 Dict,
344 Iterable,
345 List,
346 Optional,
347 Sequence,
348 Set,
349 Tuple,
350 TYPE_CHECKING
351)
352from cardinal_pythonlib.datetimefunc import (
353 coerce_to_pendulum,
354 coerce_to_pendulum_date,
355 format_datetime,
356)
357from cardinal_pythonlib.logs import (
358 BraceStyleAdapter,
359)
360from cardinal_pythonlib.pyramid.responses import TextResponse
361from cardinal_pythonlib.sqlalchemy.core_query import (
362 exists_in_table,
363 fetch_all_first_values,
364)
365from cardinal_pythonlib.text import escape_newlines
366from pyramid.httpexceptions import HTTPBadRequest
367from pyramid.view import view_config
368from pyramid.response import Response
369from pyramid.security import NO_PERMISSION_REQUIRED
370from semantic_version import Version
371from sqlalchemy.engine.result import ResultProxy
372from sqlalchemy.exc import IntegrityError
373from sqlalchemy.orm import joinedload
374from sqlalchemy.sql.expression import exists, select, update
375from sqlalchemy.sql.schema import Table
377from camcops_server.cc_modules import cc_audit # avoids "audit" name clash
378from camcops_server.cc_modules.cc_all_models import (
379 CLIENT_TABLE_MAP,
380 RESERVED_FIELDS,
381)
382from camcops_server.cc_modules.cc_blob import Blob
383from camcops_server.cc_modules.cc_cache import cache_region_static, fkg
384from camcops_server.cc_modules.cc_client_api_core import (
385 AllowedTablesFieldNames,
386 BatchDetails,
387 exception_description,
388 ExtraStringFieldNames,
389 fail_server_error,
390 fail_unsupported_operation,
391 fail_user_error,
392 get_server_live_records,
393 IgnoringAntiqueTableException,
394 require_keys,
395 ServerErrorException,
396 ServerRecord,
397 TabletParam,
398 UploadRecordResult,
399 UploadTableChanges,
400 UserErrorException,
401 values_delete_later,
402 values_delete_now,
403 values_preserve_now,
404 WhichKeyToSendInfo,
405)
406from camcops_server.cc_modules.cc_client_api_helpers import (
407 upload_commit_order_sorter,
408)
409from camcops_server.cc_modules.cc_constants import (
410 CLIENT_DATE_FIELD,
411 DateFormat,
412 ERA_NOW,
413 FP_ID_NUM,
414 FP_ID_DESC,
415 FP_ID_SHORT_DESC,
416 MOVE_OFF_TABLET_FIELD,
417 NUMBER_OF_IDNUMS_DEFUNCT, # allowed; for old tablet versions
418 POSSIBLE_SEX_VALUES,
419 TABLET_ID_FIELD,
420)
421from camcops_server.cc_modules.cc_convert import (
422 decode_single_value,
423 decode_values,
424 encode_single_value,
425)
426from camcops_server.cc_modules.cc_db import (
427 FN_ADDING_USER_ID,
428 FN_ADDITION_PENDING,
429 FN_CAMCOPS_VERSION,
430 FN_CURRENT,
431 FN_DEVICE_ID,
432 FN_ERA,
433 FN_GROUP_ID,
434 FN_PK,
435 FN_PREDECESSOR_PK,
436 FN_REMOVAL_PENDING,
437 FN_REMOVING_USER_ID,
438 FN_SUCCESSOR_PK,
439 FN_WHEN_ADDED_BATCH_UTC,
440 FN_WHEN_ADDED_EXACT,
441 FN_WHEN_REMOVED_BATCH_UTC,
442 FN_WHEN_REMOVED_EXACT,
443)
444from camcops_server.cc_modules.cc_device import Device
445from camcops_server.cc_modules.cc_dirtytables import DirtyTable
446from camcops_server.cc_modules.cc_group import Group
447from camcops_server.cc_modules.cc_ipuse import IpUse
448from camcops_server.cc_modules.cc_membership import UserGroupMembership
449from camcops_server.cc_modules.cc_patient import (
450 Patient,
451 is_candidate_patient_valid_for_group,
452 is_candidate_patient_valid_for_restricted_user,
453)
454from camcops_server.cc_modules.cc_patientidnum import (
455 fake_tablet_id_for_patientidnum,
456 PatientIdNum,
457)
458from camcops_server.cc_modules.cc_proquint import (
459 InvalidProquintException,
460 uuid_from_proquint,
461)
462from camcops_server.cc_modules.cc_pyramid import Routes
463from camcops_server.cc_modules.cc_simpleobjects import (
464 BarePatientInfo,
465 IdNumReference,
466)
467from camcops_server.cc_modules.cc_specialnote import SpecialNote
468from camcops_server.cc_modules.cc_task import (
469 all_task_tables_with_min_client_version,
470)
471from camcops_server.cc_modules.cc_taskindex import update_indexes_and_push_exports # noqa
472from camcops_server.cc_modules.cc_user import User
473from camcops_server.cc_modules.cc_validators import (
474 STRING_VALIDATOR_TYPE,
475 validate_anything,
476 validate_email,
477)
478from camcops_server.cc_modules.cc_version import (
479 CAMCOPS_SERVER_VERSION_STRING,
480 MINIMUM_TABLET_VERSION,
481)
483if TYPE_CHECKING:
484 from camcops_server.cc_modules.cc_request import CamcopsRequest
486log = BraceStyleAdapter(logging.getLogger(__name__))
489# =============================================================================
490# Constants
491# =============================================================================
493COPE_WITH_DELETED_PATIENT_DESCRIPTIONS = True
494# ... as of client 2.0.0, ID descriptions are no longer duplicated.
495# As of server 2.0.0, the fields still exist in the database, but the reporting
496# and consistency check has been removed. In the next version of the server,
497# the fields will be removed, and then the server should cope with old clients,
498# at least for a while.
500DUPLICATE_FAILED = "Failed to duplicate record"
501INSERT_FAILED = "Failed to insert record"
503# REGEX_INVALID_TABLE_FIELD_CHARS = re.compile("[^a-zA-Z0-9_]")
504# ... the ^ within the [] means the expression will match any character NOT in
505# the specified range
507DEVICE_STORED_VAR_TABLENAME_DEFUNCT = "storedvars"
508# ... old table, no longer in use, that Titanium clients used to upload.
509# We recognize and ignore it now so that old clients can still work.
511SILENTLY_IGNORE_TABLENAMES = [DEVICE_STORED_VAR_TABLENAME_DEFUNCT]
513IGNORING_ANTIQUE_TABLE_MESSAGE = (
514 "Ignoring user request to upload antique/defunct table, but reporting "
515 "success to the client"
516)
518SUCCESS_MSG = "Success"
519SUCCESS_CODE = "1"
520FAILURE_CODE = "0"
522DEBUG_UPLOAD = False
525# =============================================================================
526# Quasi-constants
527# =============================================================================
529DB_JSON_DECODER = json.JSONDecoder() # just a plain one
530PATIENT_INFO_JSON_DECODER = json.JSONDecoder() # just a plain one
533# =============================================================================
534# Cached information
535# =============================================================================
537@cache_region_static.cache_on_arguments(function_key_generator=fkg)
538def all_tables_with_min_client_version() -> Dict[str, Version]:
539 """
540 For all tables that the client might upload to, return a mapping from the
541 table name to the corresponding minimum client version.
542 """
543 d = all_task_tables_with_min_client_version()
544 d[Blob.__tablename__] = MINIMUM_TABLET_VERSION
545 d[Patient.__tablename__] = MINIMUM_TABLET_VERSION
546 d[PatientIdNum.__tablename__] = MINIMUM_TABLET_VERSION
547 return d
550# =============================================================================
551# Validators
552# =============================================================================
554def ensure_valid_table_name(req: "CamcopsRequest", tablename: str) -> None:
555 """
556 Ensures a table name:
558 - doesn't contain bad characters,
559 - isn't a reserved table that the user is prohibited from accessing, and
560 - is a valid table name that's in the database.
562 Raises :exc:`UserErrorException` upon failure.
564 - 2017-10-08: shortcut to all that: it's OK if it's listed as a valid
565 client table.
566 - 2018-01-16 (v2.2.0): check also that client version is OK
567 """
568 if tablename not in CLIENT_TABLE_MAP:
569 fail_user_error(f"Invalid client table name: {tablename}")
570 tables_versions = all_tables_with_min_client_version()
571 assert tablename in tables_versions
572 client_version = req.tabletsession.tablet_version_ver
573 minimum_client_version = tables_versions[tablename]
574 if client_version < minimum_client_version:
575 fail_user_error(
576 f"Client CamCOPS version {client_version} is less than the "
577 f"version ({minimum_client_version}) "
578 f"required to handle table {tablename}"
579 )
582def ensure_valid_field_name(table: Table, fieldname: str) -> None:
583 """
584 Ensures a field name contains only valid characters, and isn't a
585 reserved fieldname that the user isn't allowed to access.
587 Raises :exc:`UserErrorException` upon failure.
589 - 2017-10-08: shortcut: it's OK if it's a column name for a particular
590 table.
591 """
592 # if fieldname in RESERVED_FIELDS:
593 if fieldname.startswith("_"): # all reserved fields start with _
594 # ... but not all fields starting with "_" are reserved; e.g.
595 # "_move_off_tablet" is allowed.
596 if fieldname in RESERVED_FIELDS:
597 fail_user_error(
598 f"Reserved field name for table {table.name!r}: {fieldname!r}")
599 if fieldname not in table.columns.keys():
600 fail_user_error(
601 f"Invalid field name for table {table.name!r}: {fieldname!r}")
602 # Note that the reserved-field check is case-sensitive, but so is the
603 # "present in table" check. So for a malicious uploader trying to use, for
604 # example, "_PK", this would not be picked up as a reserved field (so would
605 # pass that check) but then wouldn't be recognized as a valid field (so
606 # would fail).
609def ensure_string(value: Any, allow_none: bool = True) -> None:
610 """
611 Used when processing JSON information about patients: ensures that a value
612 is a string, or raises.
614 Args:
615 value: value to test
616 allow_none: is ``None`` allowed (not just an empty string)?
617 """
618 if value is None:
619 if allow_none:
620 return # OK
621 else:
622 fail_user_error("Patient JSON contains absent string")
623 if not isinstance(value, str):
624 fail_user_error(f"Patient JSON contains invalid non-string: {value!r}")
627def ensure_valid_patient_json(req: "CamcopsRequest",
628 group: Group,
629 pt_dict: Dict[str, Any]) -> None:
630 """
631 Ensures that the JSON dictionary contains valid patient details (valid for
632 the group into which it's being uploaded), and that (if applicable) this
633 user is allowed to upload this patient.
635 Args:
636 req:
637 the :class:`camcops_server.cc_modules.cc_request.CamcopsRequest`
638 group:
639 the :class:`camcops_server.cc_modules.cc_group.Group` into which
640 the upload is going
641 pt_dict:
642 a JSON dictionary from the client
644 Raises:
645 :exc:`UserErrorException` if invalid
647 """
648 if not isinstance(pt_dict, dict):
649 fail_user_error("Patient JSON is not a dict")
650 if not pt_dict:
651 fail_user_error("Patient JSON is empty")
652 valid_which_idnums = req.valid_which_idnums
653 errors = [] # type: List[str]
654 finalizing = None
655 ptinfo = BarePatientInfo()
656 idnum_types_seen = set() # type: Set[int]
657 for k, v in pt_dict.items():
658 ensure_string(k, allow_none=False)
660 if k == TabletParam.FORENAME:
661 ensure_string(v)
662 ptinfo.forename = v
664 elif k == TabletParam.SURNAME:
665 ensure_string(v)
666 ptinfo.surname = v
668 elif k == TabletParam.SEX:
669 if v not in POSSIBLE_SEX_VALUES:
670 fail_user_error(f"Bad sex value: {v!r}")
671 ptinfo.sex = v
673 elif k == TabletParam.DOB:
674 ensure_string(v)
675 if v:
676 dob = coerce_to_pendulum_date(v)
677 if dob is None:
678 fail_user_error(f"Invalid DOB: {v!r}")
679 else:
680 dob = None
681 ptinfo.dob = dob
683 elif k == TabletParam.EMAIL:
684 ensure_string(v)
685 if v:
686 try:
687 validate_email(v)
688 except ValueError:
689 fail_user_error(f"Bad e-mail address: {v!r}")
690 ptinfo.email = v
692 elif k == TabletParam.ADDRESS:
693 ensure_string(v)
694 ptinfo.address = v
696 elif k == TabletParam.GP:
697 ensure_string(v)
698 ptinfo.gp = v
700 elif k == TabletParam.OTHER:
701 ensure_string(v)
702 ptinfo.otherdetails = v
704 elif k.startswith(TabletParam.IDNUM_PREFIX):
705 nstr = k[len(TabletParam.IDNUM_PREFIX):]
706 try:
707 which_idnum = int(nstr)
708 except (TypeError, ValueError):
709 fail_user_error(f"Bad idnum key: {k!r}")
710 # noinspection PyUnboundLocalVariable
711 if which_idnum not in valid_which_idnums:
712 fail_user_error(f"Bad ID number type: {which_idnum}")
713 if which_idnum in idnum_types_seen:
714 fail_user_error(f"More than one ID number supplied for ID "
715 f"number type {which_idnum}")
716 idnum_types_seen.add(which_idnum)
717 if v is not None and not isinstance(v, int):
718 fail_user_error(f"Bad ID number value: {v!r}")
719 idref = IdNumReference(which_idnum, v)
720 if not idref.is_valid():
721 fail_user_error(f"Bad ID number: {idref!r}")
722 ptinfo.add_idnum(idref)
724 elif k == TabletParam.FINALIZING:
725 if not isinstance(v, bool):
726 fail_user_error(f"Bad {k!r} value: {v!r}")
727 finalizing = v
729 else:
730 fail_user_error(f"Unknown JSON key: {k!r}")
732 if finalizing is None:
733 fail_user_error(f"Missing {TabletParam.FINALIZING!r} JSON key")
735 pt_ok, reason = is_candidate_patient_valid_for_group(
736 ptinfo, group, finalizing)
737 if not pt_ok:
738 errors.append(f"{ptinfo} -> {reason}")
739 pt_ok, reason = is_candidate_patient_valid_for_restricted_user(
740 req, ptinfo)
741 if not pt_ok:
742 errors.append(f"{ptinfo} -> {reason}")
743 if errors:
744 fail_user_error(f"Invalid patient: {' // '.join(errors)}")
747# =============================================================================
748# Extracting information from the POST request
749# =============================================================================
751def get_str_var(
752 req: "CamcopsRequest",
753 var: str,
754 mandatory: bool = True,
755 validator: STRING_VALIDATOR_TYPE = validate_anything) \
756 -> Optional[str]:
757 """
758 Retrieves a string variable from the CamcopsRequest.
760 By default this performs no validation (because, for example, these strings
761 can contain SQL-encoded data or JSON), but there are a number of subsequent
762 operation-specific validation steps.
764 Args:
765 req: the :class:`camcops_server.cc_modules.cc_request.CamcopsRequest`
766 var: name of variable to retrieve
767 mandatory: if ``True``, raise an exception if the variable is missing
768 validator: validator function to use
770 Returns:
771 value
773 Raises:
774 :exc:`UserErrorException` if the variable was mandatory and
775 no value was provided
776 """
777 try:
778 val = req.get_str_param(var, default=None, validator=validator)
779 if mandatory and val is None:
780 fail_user_error(f"Must provide the variable: {var}")
781 return val
782 except HTTPBadRequest as e: # failed the validator
783 fail_user_error(str(e))
786def get_int_var(req: "CamcopsRequest", var: str) -> int:
787 """
788 Retrieves an integer variable from the CamcopsRequest.
790 Args:
791 req: the :class:`camcops_server.cc_modules.cc_request.CamcopsRequest`
792 var: name of variable to retrieve
794 Returns:
795 value
797 Raises:
798 :exc:`UserErrorException` if no value was provided, or if it wasn't an
799 integer
800 """
801 s = get_str_var(req, var, mandatory=True)
802 try:
803 return int(s)
804 except (TypeError, ValueError):
805 fail_user_error(f"Variable {var} is not a valid integer; was {s!r}")
808def get_bool_int_var(req: "CamcopsRequest", var: str) -> bool:
809 """
810 Retrieves a Boolean variable (encoded as an integer) from the
811 CamcopsRequest. Zero represents false; nonzero represents true.
813 Args:
814 req: the :class:`camcops_server.cc_modules.cc_request.CamcopsRequest`
815 var: name of variable to retrieve
817 Returns:
818 value
820 Raises:
821 :exc:`UserErrorException` if no value was provided, or if it wasn't an
822 integer
823 """
824 num = get_int_var(req, var)
825 return bool(num)
828def get_table_from_req(req: "CamcopsRequest", var: str) -> Table:
829 """
830 Retrieves a table name from a HTTP request, checks it's a valid client
831 table, and returns that table.
833 Args:
834 req: the :class:`camcops_server.cc_modules.cc_request.CamcopsRequest`
835 var: variable name (the variable's should be the table name)
837 Returns:
838 a SQLAlchemy :class:`Table`
840 Raises:
841 :exc:`UserErrorException` if the variable wasn't provided
843 :exc:`IgnoringAntiqueTableException` if the table is one to
844 ignore quietly (requested by an antique client)
845 """
846 tablename = get_str_var(req, var, mandatory=True)
847 if tablename in SILENTLY_IGNORE_TABLENAMES:
848 raise IgnoringAntiqueTableException(f"Ignoring table {tablename}")
849 ensure_valid_table_name(req, tablename)
850 return CLIENT_TABLE_MAP[tablename]
853def get_tables_from_post_var(req: "CamcopsRequest",
854 var: str,
855 mandatory: bool = True) -> List[Table]:
856 """
857 Gets a list of tables from an HTTP request variable, and ensures all are
858 valid.
860 Args:
861 req: the :class:`camcops_server.cc_modules.cc_request.CamcopsRequest`
862 var: name of variable to retrieve
863 mandatory: if ``True``, raise an exception if the variable is missing
865 Returns:
866 a list of SQLAlchemy :class:`Table` objects
868 Raises:
869 :exc:`UserErrorException` if the variable was mandatory and
870 no value was provided, or if one or more tables was not valid
871 """
872 cstables = get_str_var(req, var, mandatory=mandatory)
873 if not cstables:
874 return []
875 # can't have any commas in table names, so it's OK to use a simple
876 # split() command
877 tablenames = [x.strip() for x in cstables.split(",")]
878 tables = [] # type: List[Table]
879 for tn in tablenames:
880 if tn in SILENTLY_IGNORE_TABLENAMES:
881 log.warning(IGNORING_ANTIQUE_TABLE_MESSAGE)
882 continue
883 ensure_valid_table_name(req, tn)
884 tables.append(CLIENT_TABLE_MAP[tn])
885 return tables
888def get_single_field_from_post_var(req: "CamcopsRequest",
889 table: Table,
890 var: str,
891 mandatory: bool = True) -> str:
892 """
893 Retrieves a field (column) name from a the request and checks it's not a
894 bad fieldname for the specified table.
896 Args:
897 req: the :class:`camcops_server.cc_modules.cc_request.CamcopsRequest`
898 table: SQLAlchemy :class:`Table` in which the column should exist
899 var: name of variable to retrieve
900 mandatory: if ``True``, raise an exception if the variable is missing
902 Returns:
903 the field (column) name
905 Raises:
906 :exc:`UserErrorException` if the variable was mandatory and
907 no value was provided, or if the field was not valid for the specified
908 table
909 """
910 field = get_str_var(req, var, mandatory=mandatory)
911 ensure_valid_field_name(table, field)
912 return field
915def get_fields_from_post_var(
916 req: "CamcopsRequest",
917 table: Table,
918 var: str,
919 mandatory: bool = True,
920 allowed_nonexistent_fields: List[str] = None) -> List[str]:
921 """
922 Get a comma-separated list of field names from a request and checks that
923 all are acceptable. Returns a list of fieldnames.
925 Args:
926 req: the :class:`camcops_server.cc_modules.cc_request.CamcopsRequest`
927 table: SQLAlchemy :class:`Table` in which the columns should exist
928 var: name of variable to retrieve
929 mandatory: if ``True``, raise an exception if the variable is missing
930 allowed_nonexistent_fields: fields that are allowed to be in the
931 upload but not in the database (special exemptions!)
933 Returns:
934 a list of the field (column) names
936 Raises:
937 :exc:`UserErrorException` if the variable was mandatory and
938 no value was provided, or if any field was not valid for the specified
939 table
940 """
941 csfields = get_str_var(req, var, mandatory=mandatory)
942 if not csfields:
943 return []
944 allowed_nonexistent_fields = allowed_nonexistent_fields or [] # type: List[str] # noqa
945 # can't have any commas in fields, so it's OK to use a simple
946 # split() command
947 fields = [x.strip() for x in csfields.split(",")]
948 for f in fields:
949 if f in allowed_nonexistent_fields:
950 continue
951 ensure_valid_field_name(table, f)
952 return fields
955def get_values_from_post_var(req: "CamcopsRequest",
956 var: str,
957 mandatory: bool = True) -> List[Any]:
958 """
959 Retrieves a list of values from a CSV-separated list of SQL values
960 stored in a CGI form (including e.g. NULL, numbers, quoted strings, and
961 special handling for base-64/hex-encoded BLOBs.)
963 Args:
964 req: the :class:`camcops_server.cc_modules.cc_request.CamcopsRequest`
965 var: name of variable to retrieve
966 mandatory: if ``True``, raise an exception if the variable is missing
967 """
968 csvalues = get_str_var(req, var, mandatory=mandatory)
969 if not csvalues:
970 return []
971 return decode_values(csvalues)
974def get_fields_and_values(req: "CamcopsRequest",
975 table: Table,
976 fields_var: str,
977 values_var: str,
978 mandatory: bool = True) -> Dict[str, Any]:
979 """
980 Gets fieldnames and matching values from two variables in a request.
982 See :func:`get_fields_from_post_var`, :func:`get_values_from_post_var`.
984 Args:
985 req: the :class:`camcops_server.cc_modules.cc_request.CamcopsRequest`
986 table: SQLAlchemy :class:`Table` in which the columns should exist
987 fields_var: name of CSV "column names" variable to retrieve
988 values_var: name of CSV "corresponding values" variable to retrieve
989 mandatory: if ``True``, raise an exception if the variable is missing
991 Returns:
992 a dictionary mapping column names to decoded values
994 Raises:
995 :exc:`UserErrorException` if the variable was mandatory and
996 no value was provided, or if any field was not valid for the specified
997 table
998 """
999 fields = get_fields_from_post_var(req, table, fields_var,
1000 mandatory=mandatory)
1001 values = get_values_from_post_var(req, values_var, mandatory=mandatory)
1002 if len(fields) != len(values):
1003 fail_user_error(
1004 f"Number of fields ({len(fields)}) doesn't match number of values "
1005 f"({len(values)})"
1006 )
1007 return dict(list(zip(fields, values)))
1010def get_json_from_post_var(req: "CamcopsRequest", key: str,
1011 decoder: json.JSONDecoder = None,
1012 mandatory: bool = True) -> Any:
1013 """
1014 Returns a Python object from a JSON-encoded value.
1016 Args:
1017 req: the :class:`camcops_server.cc_modules.cc_request.CamcopsRequest`
1018 key: the name of the variable to retrieve
1019 decoder: the JSON decoder object to use; if ``None``, a default is
1020 created
1021 mandatory: if ``True``, raise an exception if the variable is missing
1023 Returns:
1024 Python object, e.g. a list of values, or ``None`` if the object is
1025 invalid and not mandatory
1027 Raises:
1028 :exc:`UserErrorException` if the variable was mandatory and
1029 no value was provided or the value was invalid JSON
1030 """
1031 decoder = decoder or json.JSONDecoder()
1032 j = get_str_var(req, key, mandatory=mandatory) # may raise
1033 if not j: # missing but not mandatory
1034 return None
1035 try:
1036 return decoder.decode(j)
1037 except json.JSONDecodeError:
1038 msg = f"Bad JSON for key {key!r}"
1039 if mandatory:
1040 fail_user_error(msg)
1041 else:
1042 log.warning(msg)
1043 return None
1046# =============================================================================
1047# Sending stuff to the client
1048# =============================================================================
1050def get_server_id_info(req: "CamcopsRequest") -> Dict[str, str]:
1051 """
1052 Returns a reply for the tablet, as a variable-to-value dictionary, giving
1053 details of the server.
1054 """
1055 group = Group.get_group_by_id(req.dbsession, req.user.upload_group_id)
1056 reply = {
1057 TabletParam.DATABASE_TITLE: req.database_title,
1058 TabletParam.ID_POLICY_UPLOAD: group.upload_policy or "",
1059 TabletParam.ID_POLICY_FINALIZE: group.finalize_policy or "",
1060 TabletParam.SERVER_CAMCOPS_VERSION: CAMCOPS_SERVER_VERSION_STRING,
1061 }
1062 for iddef in req.idnum_definitions:
1063 n = iddef.which_idnum
1064 nstr = str(n)
1065 reply[TabletParam.ID_DESCRIPTION_PREFIX + nstr] = \
1066 iddef.description or ""
1067 reply[TabletParam.ID_SHORT_DESCRIPTION_PREFIX + nstr] = \
1068 iddef.short_description or ""
1069 reply[TabletParam.ID_VALIDATION_METHOD_PREFIX + nstr] = \
1070 iddef.validation_method or ""
1071 return reply
1074def get_select_reply(fields: Sequence[str],
1075 rows: Sequence[Sequence[Any]]) -> Dict[str, str]:
1076 """
1077 Formats the result of a ``SELECT`` query for the client as a dictionary
1078 reply.
1080 Args:
1081 fields: list of field names
1082 rows: list of rows, where each row is a list of values in the same
1083 order as ``fields``
1085 Returns:
1087 a dictionary of the format:
1089 .. code-block:: none
1091 {
1092 "nfields": NUMBER_OF_FIELDS,
1093 "fields": FIELDNAMES_AS_CSV,
1094 "nrecords": NUMBER_OF_RECORDS,
1095 "record0": VALUES_AS_CSV_LIST_OF_ENCODED_SQL_VALUES,
1096 ...
1097 "record{nrecords - 1}": VALUES_AS_CSV_LIST_OF_ENCODED_SQL_VALUES
1098 }
1100 The final reply to the server is then formatted as text as per
1101 :func:`client_api`.
1103 """ # noqa
1104 nrecords = len(rows)
1105 reply = {
1106 TabletParam.NFIELDS: len(fields),
1107 TabletParam.FIELDS: ",".join(fields),
1108 TabletParam.NRECORDS: nrecords,
1109 }
1110 for r in range(nrecords):
1111 row = rows[r]
1112 encodedvalues = [] # type: List[str]
1113 for val in row:
1114 encodedvalues.append(encode_single_value(val))
1115 reply[TabletParam.RECORD_PREFIX + str(r)] = ",".join(encodedvalues)
1116 return reply
1119# =============================================================================
1120# CamCOPS table reading functions
1121# =============================================================================
1123def record_exists(req: "CamcopsRequest",
1124 table: Table,
1125 clientpk_name: str,
1126 clientpk_value: Any) -> ServerRecord:
1127 """
1128 Checks if a record exists, using the device's perspective of a
1129 table/client PK combination.
1131 Args:
1132 req: the :class:`camcops_server.cc_modules.cc_request.CamcopsRequest`
1133 table: an SQLAlchemy :class:`Table`
1134 clientpk_name: the column name of the client's PK
1135 clientpk_value: the client's PK value
1137 Returns:
1138 a :class:`ServerRecord` with the required information
1140 """
1141 query = (
1142 select([
1143 table.c[FN_PK], # server PK
1144 table.c[CLIENT_DATE_FIELD], # when last modified (on the server)
1145 table.c[MOVE_OFF_TABLET_FIELD] # move_off_tablet
1146 ])
1147 .where(table.c[FN_DEVICE_ID] == req.tabletsession.device_id)
1148 .where(table.c[FN_CURRENT])
1149 .where(table.c[FN_ERA] == ERA_NOW)
1150 .where(table.c[clientpk_name] == clientpk_value)
1151 )
1152 row = req.dbsession.execute(query).fetchone()
1153 if not row:
1154 return ServerRecord(clientpk_value, False)
1155 server_pk, server_when, move_off_tablet = row
1156 return ServerRecord(clientpk_value, True, server_pk, server_when,
1157 move_off_tablet)
1158 # Consider a warning/failure if we have >1 row meeting these criteria.
1159 # Not currently checked for.
1162def client_pks_that_exist(req: "CamcopsRequest",
1163 table: Table,
1164 clientpk_name: str,
1165 clientpk_values: List[int]) \
1166 -> Dict[int, ServerRecord]:
1167 """
1168 Searches for client PK values (for this device, current, and 'now')
1169 matching the input list.
1171 Args:
1172 req: the :class:`camcops_server.cc_modules.cc_request.CamcopsRequest`
1173 table: an SQLAlchemy :class:`Table`
1174 clientpk_name: the column name of the client's PK
1175 clientpk_values: a list of the client's PK values
1177 Returns:
1178 a dictionary mapping client_pk to a :class:`ServerRecord` objects, for
1179 those records that match
1180 """
1181 query = (
1182 select([
1183 table.c[FN_PK], # server PK
1184 table.c[clientpk_name], # client PK
1185 table.c[CLIENT_DATE_FIELD], # when last modified (on the server)
1186 table.c[MOVE_OFF_TABLET_FIELD] # move_off_tablet
1187 ])
1188 .where(table.c[FN_DEVICE_ID] == req.tabletsession.device_id)
1189 .where(table.c[FN_CURRENT])
1190 .where(table.c[FN_ERA] == ERA_NOW)
1191 .where(table.c[clientpk_name].in_(clientpk_values))
1192 )
1193 rows = req.dbsession.execute(query)
1194 d = {} # type: Dict[int, ServerRecord]
1195 for server_pk, client_pk, server_when, move_off_tablet in rows:
1196 d[client_pk] = ServerRecord(client_pk, True, server_pk, server_when,
1197 move_off_tablet)
1198 return d
1201def get_all_predecessor_pks(req: "CamcopsRequest",
1202 table: Table,
1203 last_pk: int,
1204 include_last: bool = True) -> List[int]:
1205 """
1206 Retrieves the PKs of all records that are predecessors of the specified one
1208 Args:
1209 req: the :class:`camcops_server.cc_modules.cc_request.CamcopsRequest`
1210 table: an SQLAlchemy :class:`Table`
1211 last_pk: the PK to start with, and work backwards
1212 include_last: include ``last_pk`` in the list
1214 Returns:
1215 the PKs
1217 """
1218 dbsession = req.dbsession
1219 pks = [] # type: List[int]
1220 if include_last:
1221 pks.append(last_pk)
1222 current_pk = last_pk
1223 finished = False
1224 while not finished:
1225 next_pk = dbsession.execute(
1226 select([table.c[FN_PREDECESSOR_PK]])
1227 .where(table.c[FN_PK] == current_pk)
1228 ).scalar() # type: Optional[int]
1229 if next_pk is None:
1230 finished = True
1231 else:
1232 pks.append(next_pk)
1233 current_pk = next_pk
1234 return sorted(pks)
1237# =============================================================================
1238# Record modification functions
1239# =============================================================================
1241def flag_deleted(req: "CamcopsRequest",
1242 batchdetails: BatchDetails,
1243 table: Table,
1244 pklist: Iterable[int]) -> None:
1245 """
1246 Marks record(s) as deleted, specified by a list of server PKs within a
1247 table. (Note: "deleted" means "deleted with no successor", not "modified
1248 and replaced by a successor record".)
1249 """
1250 if batchdetails.onestep:
1251 values = values_delete_now(req, batchdetails)
1252 else:
1253 values = values_delete_later()
1254 req.dbsession.execute(
1255 update(table)
1256 .where(table.c[FN_PK].in_(pklist))
1257 .values(values)
1258 )
1261def flag_all_records_deleted(req: "CamcopsRequest",
1262 table: Table) -> int:
1263 """
1264 Marks all records in a table as deleted (that are current and in the
1265 current era).
1267 Returns the number of rows affected.
1268 """
1269 rp = req.dbsession.execute(
1270 update(table)
1271 .where(table.c[FN_DEVICE_ID] == req.tabletsession.device_id)
1272 .where(table.c[FN_CURRENT])
1273 .where(table.c[FN_ERA] == ERA_NOW)
1274 .values(values_delete_later())
1275 ) # type: ResultProxy
1276 return rp.rowcount
1277 # https://docs.sqlalchemy.org/en/latest/core/connections.html?highlight=rowcount#sqlalchemy.engine.ResultProxy.rowcount # noqa
1280def flag_deleted_where_clientpk_not(req: "CamcopsRequest",
1281 table: Table,
1282 clientpk_name: str,
1283 clientpk_values: Sequence[Any]) -> None:
1284 """
1285 Marks for deletion all current/current-era records for a device, within a
1286 specific table, defined by a list of client-side PK values (and the name of
1287 the client-side PK column).
1288 """
1289 rp = req.dbsession.execute(
1290 update(table)
1291 .where(table.c[FN_DEVICE_ID] == req.tabletsession.device_id)
1292 .where(table.c[FN_CURRENT])
1293 .where(table.c[FN_ERA] == ERA_NOW)
1294 .where(table.c[clientpk_name].notin_(clientpk_values))
1295 .values(values_delete_later())
1296 ) # type: ResultProxy
1297 if rp.rowcount > 0:
1298 mark_table_dirty(req, table)
1299 # ... but if we are preserving, do NOT mark this table as clean; there may
1300 # be other records that still require preserving.
1303def flag_modified(req: "CamcopsRequest",
1304 batchdetails: BatchDetails,
1305 table: Table,
1306 pk: int,
1307 successor_pk: int) -> None:
1308 """
1309 Marks a record as old, storing its successor's details.
1311 Args:
1312 req: the :class:`camcops_server.cc_modules.cc_request.CamcopsRequest`
1313 batchdetails: the :class:`BatchDetails`
1314 table: SQLAlchemy :class:`Table`
1315 pk: server PK of the record to mark as old
1316 successor_pk: server PK of its successor
1317 """
1318 if batchdetails.onestep:
1319 req.dbsession.execute(
1320 update(table)
1321 .where(table.c[FN_PK] == pk)
1322 .values({
1323 FN_CURRENT: 0,
1324 FN_REMOVAL_PENDING: 0,
1325 FN_SUCCESSOR_PK: successor_pk,
1326 FN_REMOVING_USER_ID: req.user_id,
1327 FN_WHEN_REMOVED_EXACT: req.now,
1328 FN_WHEN_REMOVED_BATCH_UTC: batchdetails.batchtime,
1329 })
1330 )
1331 else:
1332 req.dbsession.execute(
1333 update(table)
1334 .where(table.c[FN_PK] == pk)
1335 .values({
1336 FN_REMOVAL_PENDING: 1,
1337 FN_SUCCESSOR_PK: successor_pk
1338 })
1339 )
1342def flag_multiple_records_for_preservation(
1343 req: "CamcopsRequest",
1344 batchdetails: BatchDetails,
1345 table: Table,
1346 pks_to_preserve: List[int]) -> None:
1347 """
1348 Low-level function to mark records for preservation by server PK.
1349 Does not concern itself with the predecessor chain (for which, see
1350 :func:`flag_record_for_preservation`).
1352 Args:
1353 req: the :class:`camcops_server.cc_modules.cc_request.CamcopsRequest`
1354 batchdetails: the :class:`BatchDetails`
1355 table: SQLAlchemy :class:`Table`
1356 pks_to_preserve: server PK of the records to mark as preserved
1357 """
1358 if batchdetails.onestep:
1359 req.dbsession.execute(
1360 update(table)
1361 .where(table.c[FN_PK].in_(pks_to_preserve))
1362 .values(values_preserve_now(req, batchdetails))
1363 )
1364 # Also any associated special notes:
1365 new_era = batchdetails.new_era
1366 # noinspection PyUnresolvedReferences
1367 req.dbsession.execute(
1368 update(SpecialNote.__table__)
1369 .where(SpecialNote.basetable == table.name)
1370 .where(SpecialNote.device_id == req.tabletsession.device_id)
1371 .where(SpecialNote.era == ERA_NOW)
1372 .where(exists().select_from(table)
1373 .where(table.c[TABLET_ID_FIELD] == SpecialNote.task_id)
1374 .where(table.c[FN_DEVICE_ID] == SpecialNote.device_id)
1375 .where(table.c[FN_ERA] == new_era))
1376 # ^^^^^^^^^^^^^^^^^^^^^^^^^^
1377 # This bit restricts to records being preserved.
1378 .values(era=new_era)
1379 )
1380 else:
1381 req.dbsession.execute(
1382 update(table)
1383 .where(table.c[FN_PK].in_(pks_to_preserve))
1384 .values({
1385 MOVE_OFF_TABLET_FIELD: 1
1386 })
1387 )
1390def flag_record_for_preservation(req: "CamcopsRequest",
1391 batchdetails: BatchDetails,
1392 table: Table,
1393 pk: int) -> List[int]:
1394 """
1395 Marks a record for preservation (moving off the tablet, changing its
1396 era details).
1398 2018-11-18: works back through the predecessor chain too, fixing an old
1399 bug.
1401 Args:
1402 req: the :class:`camcops_server.cc_modules.cc_request.CamcopsRequest`
1403 batchdetails: the :class:`BatchDetails`
1404 table: SQLAlchemy :class:`Table`
1405 pk: server PK of the record to mark
1407 Returns:
1408 list: all PKs being preserved
1409 """
1410 pks_to_preserve = get_all_predecessor_pks(req, table, pk)
1411 flag_multiple_records_for_preservation(req, batchdetails, table,
1412 pks_to_preserve)
1413 return pks_to_preserve
1416def preserve_all(req: "CamcopsRequest",
1417 batchdetails: BatchDetails,
1418 table: Table) -> None:
1419 """
1420 Preserves all records in a table for a device, including non-current ones.
1422 Args:
1423 req: the :class:`camcops_server.cc_modules.cc_request.CamcopsRequest`
1424 batchdetails: the :class:`BatchDetails`
1425 table: SQLAlchemy :class:`Table`
1426 """
1427 device_id = req.tabletsession.device_id
1428 req.dbsession.execute(
1429 update(table)
1430 .where(table.c[FN_DEVICE_ID] == device_id)
1431 .where(table.c[FN_ERA] == ERA_NOW)
1432 .values(values_preserve_now(req, batchdetails))
1433 )
1436# =============================================================================
1437# Upload helper functions
1438# =============================================================================
1440def process_upload_record_special(req: "CamcopsRequest",
1441 batchdetails: BatchDetails,
1442 table: Table,
1443 valuedict: Dict[str, Any]) -> None:
1444 """
1445 Special processing function for upload, in which we inspect the data.
1446 Called by :func:`upload_record_core`.
1448 1. Handles old clients with ID information in the patient table, etc.
1449 (Note: this can be IGNORED for any client using
1450 :func:`op_upload_entire_database`, as these are newer.)
1452 2. Validates ID numbers.
1454 Args:
1455 req: the :class:`camcops_server.cc_modules.cc_request.CamcopsRequest`
1456 batchdetails: the :class:`BatchDetails`
1457 table: an SQLAlchemy :class:`Table`
1458 valuedict: a dictionary of {colname: value} pairs from the client.
1459 May be modified.
1460 """
1461 ts = req.tabletsession
1462 tablename = table.name
1464 if tablename == Patient.__tablename__:
1465 # ---------------------------------------------------------------------
1466 # Deal with old tablets that had ID numbers in a less flexible format.
1467 # ---------------------------------------------------------------------
1468 if ts.cope_with_deleted_patient_descriptors:
1469 # Old tablets (pre-2.0.0) will upload copies of the ID
1470 # descriptions with the patient. To cope with that, we
1471 # remove those here:
1472 for n in range(1, NUMBER_OF_IDNUMS_DEFUNCT + 1):
1473 nstr = str(n)
1474 fn_desc = FP_ID_DESC + nstr
1475 fn_shortdesc = FP_ID_SHORT_DESC + nstr
1476 valuedict.pop(fn_desc, None) # remove item, if exists
1477 valuedict.pop(fn_shortdesc, None)
1479 if ts.cope_with_old_idnums:
1480 # Insert records into the new ID number table from the old
1481 # patient table:
1482 for which_idnum in range(1, NUMBER_OF_IDNUMS_DEFUNCT + 1):
1483 nstr = str(which_idnum)
1484 fn_idnum = FP_ID_NUM + nstr
1485 idnum_value = valuedict.pop(fn_idnum, None)
1486 # ... and remove it from our new Patient record
1487 patient_id = valuedict.get("id", None)
1488 if idnum_value is None or patient_id is None:
1489 continue
1490 # noinspection PyUnresolvedReferences
1491 mark_table_dirty(req, PatientIdNum.__table__)
1492 client_date_value = coerce_to_pendulum(valuedict[CLIENT_DATE_FIELD]) # noqa
1493 # noinspection PyUnresolvedReferences
1494 upload_record_core(
1495 req=req,
1496 batchdetails=batchdetails,
1497 table=PatientIdNum.__table__,
1498 clientpk_name='id',
1499 valuedict={
1500 'id': fake_tablet_id_for_patientidnum(
1501 patient_id=patient_id,
1502 which_idnum=which_idnum
1503 ), # ... guarantees a pseudo client PK
1504 'patient_id': patient_id,
1505 'which_idnum': which_idnum,
1506 'idnum_value': idnum_value,
1507 CLIENT_DATE_FIELD: client_date_value,
1508 MOVE_OFF_TABLET_FIELD: valuedict[MOVE_OFF_TABLET_FIELD], # noqa
1509 }
1510 )
1511 # Now, how to deal with deletion, i.e. records missing from the
1512 # tablet? See our caller, op_upload_table(), which has a special
1513 # handler for this.
1514 #
1515 # Note that op_upload_record() is/was only used for BLOBs, so we
1516 # don't have to worry about special processing for that aspect
1517 # here; also, that method handles deletion in a different way.
1519 elif tablename == PatientIdNum.__tablename__:
1520 # ---------------------------------------------------------------------
1521 # Validate ID numbers.
1522 # ---------------------------------------------------------------------
1523 which_idnum = valuedict.get("which_idnum", None)
1524 if which_idnum not in req.valid_which_idnums:
1525 fail_user_error(f"No such ID number type: {which_idnum}")
1526 idnum_value = valuedict.get("idnum_value", None)
1527 if not req.is_idnum_valid(which_idnum, idnum_value):
1528 why_invalid = req.why_idnum_invalid(which_idnum, idnum_value)
1529 fail_user_error(
1530 f"For ID type {which_idnum}, ID number {idnum_value} is "
1531 f"invalid: {why_invalid}")
1534def upload_record_core(
1535 req: "CamcopsRequest",
1536 batchdetails: BatchDetails,
1537 table: Table,
1538 clientpk_name: str,
1539 valuedict: Dict[str, Any],
1540 server_live_current_records: List[ServerRecord] = None) \
1541 -> UploadRecordResult:
1542 """
1543 Uploads a record. Deals with IDENTICAL, NEW, and MODIFIED records.
1545 Used by :func:`upload_table` and :func:`upload_record`.
1547 Args:
1548 req: the :class:`camcops_server.cc_modules.cc_request.CamcopsRequest`
1549 batchdetails: the :class:`BatchDetails`
1550 table: an SQLAlchemy :class:`Table`
1551 clientpk_name: the column name of the client's PK
1552 valuedict: a dictionary of {colname: value} pairs from the client
1553 server_live_current_records: list of :class:`ServerRecord` objects for
1554 the active records on the server for this client, in this table
1556 Returns:
1557 a :class:`UploadRecordResult` object
1558 """
1559 require_keys(valuedict, [clientpk_name, CLIENT_DATE_FIELD,
1560 MOVE_OFF_TABLET_FIELD])
1561 clientpk_value = valuedict[clientpk_name]
1563 if server_live_current_records:
1564 # All server records for this table/device/era have been prefetched.
1565 serverrec = next((r for r in server_live_current_records
1566 if r.client_pk == clientpk_value), None)
1567 if serverrec is None:
1568 serverrec = ServerRecord(clientpk_value, False)
1569 else:
1570 # Look up this record specifically.
1571 serverrec = record_exists(req, table, clientpk_name, clientpk_value)
1573 if DEBUG_UPLOAD:
1574 log.debug("upload_record_core: {}, {}", table.name, serverrec)
1576 oldserverpk = serverrec.server_pk
1577 urr = UploadRecordResult(
1578 oldserverpk=oldserverpk,
1579 specifically_marked_for_preservation=bool(valuedict[MOVE_OFF_TABLET_FIELD]), # noqa
1580 dirty=True
1581 )
1582 if serverrec.exists:
1583 # There's an existing record, which is either identical or not.
1584 client_date_value = coerce_to_pendulum(valuedict[CLIENT_DATE_FIELD])
1585 if serverrec.server_when == client_date_value:
1586 # The existing record is identical.
1587 # No action needed unless MOVE_OFF_TABLET_FIELDNAME is set.
1588 if not urr.specifically_marked_for_preservation:
1589 urr.dirty = False
1590 else:
1591 # The existing record is different. We need a logical UPDATE, but
1592 # maintaining an audit trail.
1593 process_upload_record_special(req, batchdetails, table, valuedict)
1594 urr.newserverpk = insert_record(req, batchdetails, table,
1595 valuedict, oldserverpk)
1596 flag_modified(req, batchdetails,
1597 table, oldserverpk, urr.newserverpk)
1598 else:
1599 # The record is NEW. We need to INSERT it.
1600 process_upload_record_special(req, batchdetails, table, valuedict)
1601 urr.newserverpk = insert_record(req, batchdetails, table,
1602 valuedict, None)
1603 if urr.specifically_marked_for_preservation:
1604 preservation_pks = flag_record_for_preservation(req, batchdetails,
1605 table, urr.latest_pk)
1606 urr.note_specifically_marked_preservation_pks(preservation_pks)
1608 if DEBUG_UPLOAD:
1609 log.debug("upload_record_core: {}, {!r}", table.name, urr)
1610 return urr
1613def insert_record(req: "CamcopsRequest",
1614 batchdetails: BatchDetails,
1615 table: Table,
1616 valuedict: Dict[str, Any],
1617 predecessor_pk: Optional[int]) -> int:
1618 """
1619 Inserts a record, or raises an exception if that fails.
1621 Args:
1622 req: the :class:`camcops_server.cc_modules.cc_request.CamcopsRequest`
1623 batchdetails: the :class:`BatchDetails`
1624 table: an SQLAlchemy :class:`Table`
1625 valuedict: a dictionary of {colname: value} pairs from the client
1626 predecessor_pk: an optional server PK of the record's predecessor
1628 Returns:
1629 the server PK of the new record
1630 """
1631 ts = req.tabletsession
1632 valuedict.update({
1633 FN_DEVICE_ID: ts.device_id,
1634 FN_ERA: ERA_NOW,
1635 FN_REMOVAL_PENDING: 0,
1636 FN_PREDECESSOR_PK: predecessor_pk,
1637 FN_CAMCOPS_VERSION: ts.tablet_version_str,
1638 FN_GROUP_ID: req.user.upload_group_id,
1639 })
1640 if batchdetails.onestep:
1641 valuedict.update({
1642 FN_CURRENT: 1,
1643 FN_ADDITION_PENDING: 0,
1644 FN_ADDING_USER_ID: req.user_id,
1645 FN_WHEN_ADDED_EXACT: req.now,
1646 FN_WHEN_ADDED_BATCH_UTC: batchdetails.batchtime,
1647 })
1648 else:
1649 valuedict.update({
1650 FN_CURRENT: 0,
1651 FN_ADDITION_PENDING: 1,
1652 })
1653 rp = req.dbsession.execute(
1654 table.insert().values(valuedict)
1655 ) # type: ResultProxy
1656 inserted_pks = rp.inserted_primary_key
1657 assert(isinstance(inserted_pks, list) and len(inserted_pks) == 1)
1658 return inserted_pks[0]
1661def audit_upload(req: "CamcopsRequest",
1662 changes: List[UploadTableChanges]) -> None:
1663 """
1664 Writes audit information for an upload.
1666 Args:
1667 req: the :class:`camcops_server.cc_modules.cc_request.CamcopsRequest`
1668 changes: a list of :class:`UploadTableChanges` objects, one per table
1669 """
1670 msg = (
1671 f"Upload from device {req.tabletsession.device_id}, "
1672 f"username {req.tabletsession.username!r}: "
1673 )
1674 changes = [x for x in changes if x.any_changes]
1675 if changes:
1676 changes.sort(key=lambda x: x.tablename)
1677 msg += ", ".join(x.description() for x in changes)
1678 else:
1679 msg += "No changes"
1680 log.info("audit_upload: {}", msg)
1681 audit(req, msg)
1684# =============================================================================
1685# Batch (atomic) upload and preserving
1686# =============================================================================
1688def get_batch_details(req: "CamcopsRequest") -> BatchDetails:
1689 """
1690 Returns the :class:`BatchDetails` for the current upload. If none exists,
1691 a new batch is created and returned.
1693 SIDE EFFECT: if the username is different from the username that started
1694 a previous upload batch for this device, we restart the upload batch (thus
1695 rolling back previous pending changes).
1697 Raises:
1698 :exc:`camcops_server.cc_modules.cc_client_api_core.ServerErrorException`
1699 if the device doesn't exist
1700 """
1701 device_id = req.tabletsession.device_id
1702 # noinspection PyUnresolvedReferences
1703 query = (
1704 select([Device.ongoing_upload_batch_utc,
1705 Device.uploading_user_id,
1706 Device.currently_preserving])
1707 .select_from(Device.__table__)
1708 .where(Device.id == device_id)
1709 )
1710 row = req.dbsession.execute(query).fetchone()
1711 if not row:
1712 fail_server_error(f"Device {device_id} missing from Device table") # will raise # noqa
1713 upload_batch_utc, uploading_user_id, currently_preserving = row
1714 if not upload_batch_utc or uploading_user_id != req.user_id:
1715 # SIDE EFFECT: if the username changes, we restart (and thus roll back
1716 # previous pending changes)
1717 start_device_upload_batch(req)
1718 return BatchDetails(req.now_utc, False)
1719 return BatchDetails(upload_batch_utc, currently_preserving)
1722def start_device_upload_batch(req: "CamcopsRequest") -> None:
1723 """
1724 Starts an upload batch for a device.
1725 """
1726 rollback_all(req)
1727 # noinspection PyUnresolvedReferences
1728 req.dbsession.execute(
1729 update(Device.__table__)
1730 .where(Device.id == req.tabletsession.device_id)
1731 .values(last_upload_batch_utc=req.now_utc,
1732 ongoing_upload_batch_utc=req.now_utc,
1733 uploading_user_id=req.tabletsession.user_id)
1734 )
1737def _clear_ongoing_upload_batch_details(req: "CamcopsRequest") -> None:
1738 """
1739 Clears upload batch details from the Device table.
1741 Args:
1742 req: the :class:`camcops_server.cc_modules.cc_request.CamcopsRequest`
1743 """
1744 # noinspection PyUnresolvedReferences
1745 req.dbsession.execute(
1746 update(Device.__table__)
1747 .where(Device.id == req.tabletsession.device_id)
1748 .values(ongoing_upload_batch_utc=None,
1749 uploading_user_id=None,
1750 currently_preserving=0)
1751 )
1754def end_device_upload_batch(req: "CamcopsRequest",
1755 batchdetails: BatchDetails) -> None:
1756 """
1757 Ends an upload batch, committing all changes made thus far.
1759 Args:
1760 req: the :class:`camcops_server.cc_modules.cc_request.CamcopsRequest`
1761 batchdetails: the :class:`BatchDetails`
1762 """
1763 commit_all(req, batchdetails)
1764 _clear_ongoing_upload_batch_details(req)
1767def clear_device_upload_batch(req: "CamcopsRequest") -> None:
1768 """
1769 Ensures there is nothing pending. Rools back previous changes. Wipes any
1770 ongoing batch details.
1772 Args:
1773 req: the :class:`camcops_server.cc_modules.cc_request.CamcopsRequest`
1774 """
1775 rollback_all(req)
1776 _clear_ongoing_upload_batch_details(req)
1779def start_preserving(req: "CamcopsRequest") -> None:
1780 """
1781 Starts preservation (the process of moving records from the NOW era to
1782 an older era, so they can be removed safely from the tablet).
1784 Called by :func:`op_start_preservation`.
1786 In this situation, we start by assuming that ALL tables are "dirty",
1787 because they may have live records from a previous upload.
1788 """
1789 # noinspection PyUnresolvedReferences
1790 req.dbsession.execute(
1791 update(Device.__table__)
1792 .where(Device.id == req.tabletsession.device_id)
1793 .values(currently_preserving=1)
1794 )
1795 mark_all_tables_dirty(req)
1798def mark_table_dirty(req: "CamcopsRequest", table: Table) -> None:
1799 """
1800 Marks a table as having been modified during the current upload.
1801 """
1802 tablename = table.name
1803 device_id = req.tabletsession.device_id
1804 dbsession = req.dbsession
1805 # noinspection PyUnresolvedReferences
1806 table_already_dirty = exists_in_table(
1807 dbsession,
1808 DirtyTable.__table__,
1809 DirtyTable.device_id == device_id,
1810 DirtyTable.tablename == tablename
1811 )
1812 if not table_already_dirty:
1813 # noinspection PyUnresolvedReferences
1814 dbsession.execute(
1815 DirtyTable.__table__.insert()
1816 .values(device_id=device_id,
1817 tablename=tablename)
1818 )
1821def mark_tables_dirty(req: "CamcopsRequest", tables: List[Table]) -> None:
1822 """
1823 Marks multiple tables as dirty.
1824 """
1825 if not tables:
1826 return
1827 device_id = req.tabletsession.device_id
1828 tablenames = [t.name for t in tables]
1829 # Delete first
1830 # noinspection PyUnresolvedReferences
1831 req.dbsession.execute(
1832 DirtyTable.__table__.delete()
1833 .where(DirtyTable.device_id == device_id)
1834 .where(DirtyTable.tablename.in_(tablenames))
1835 )
1836 # Then insert
1837 insert_values = [
1838 {"device_id": device_id, "tablename": tn}
1839 for tn in tablenames
1840 ]
1841 # noinspection PyUnresolvedReferences
1842 req.dbsession.execute(
1843 DirtyTable.__table__.insert(),
1844 insert_values
1845 )
1848def mark_all_tables_dirty(req: "CamcopsRequest") -> None:
1849 """
1850 If we are preserving, we assume that all tables are "dirty" (require work
1851 when we complete the upload) unless we specifically mark them clean.
1852 """
1853 device_id = req.tabletsession.device_id
1854 # Delete first
1855 # noinspection PyUnresolvedReferences
1856 req.dbsession.execute(
1857 DirtyTable.__table__.delete()
1858 .where(DirtyTable.device_id == device_id)
1859 )
1860 # Now insert
1861 # https://docs.sqlalchemy.org/en/latest/core/tutorial.html#execute-multiple
1862 all_client_tablenames = list(CLIENT_TABLE_MAP.keys())
1863 insert_values = [
1864 {"device_id": device_id, "tablename": tn}
1865 for tn in all_client_tablenames
1866 ]
1867 # noinspection PyUnresolvedReferences
1868 req.dbsession.execute(
1869 DirtyTable.__table__.insert(),
1870 insert_values
1871 )
1874def mark_table_clean(req: "CamcopsRequest", table: Table) -> None:
1875 """
1876 Marks a table as being clean: that is,
1878 - the table has been scanned during the current upload
1879 - there is nothing to do (either from the current upload, OR A PREVIOUS
1880 UPLOAD).
1881 """
1882 tablename = table.name
1883 device_id = req.tabletsession.device_id
1884 # noinspection PyUnresolvedReferences
1885 req.dbsession.execute(
1886 DirtyTable.__table__.delete()
1887 .where(DirtyTable.device_id == device_id)
1888 .where(DirtyTable.tablename == tablename)
1889 )
1892def mark_tables_clean(req: "CamcopsRequest", tables: List[Table]) -> None:
1893 """
1894 Marks multiple tables as clean.
1895 """
1896 if not tables:
1897 return
1898 device_id = req.tabletsession.device_id
1899 tablenames = [t.name for t in tables]
1900 # Delete first
1901 # noinspection PyUnresolvedReferences
1902 req.dbsession.execute(
1903 DirtyTable.__table__.delete()
1904 .where(DirtyTable.device_id == device_id)
1905 .where(DirtyTable.tablename.in_(tablenames))
1906 )
1909def get_dirty_tables(req: "CamcopsRequest") -> List[Table]:
1910 """
1911 Returns tables marked as dirty for this device. (See
1912 :func:`mark_table_dirty`.)
1913 """
1914 query = (
1915 select([DirtyTable.tablename])
1916 .where(DirtyTable.device_id == req.tabletsession.device_id)
1917 )
1918 tablenames = fetch_all_first_values(req.dbsession, query)
1919 return [CLIENT_TABLE_MAP[tn] for tn in tablenames]
1922def commit_all(req: "CamcopsRequest", batchdetails: BatchDetails) -> None:
1923 """
1924 Commits additions, removals, and preservations for all tables.
1926 Args:
1927 req: the :class:`camcops_server.cc_modules.cc_request.CamcopsRequest`
1928 batchdetails: the :class:`BatchDetails`
1929 """
1930 tables = get_dirty_tables(req)
1931 # log.debug("Dirty tables: {}", list(t.name for t in tables))
1932 tables.sort(key=upload_commit_order_sorter)
1934 changelist = [] # type: List[UploadTableChanges]
1935 for table in tables:
1936 auditinfo = commit_table(req, batchdetails, table, clear_dirty=False)
1937 changelist.append(auditinfo)
1939 if batchdetails.preserving:
1940 # Also preserve/finalize any corresponding special notes (2015-02-01),
1941 # but all in one go (2018-11-13).
1942 # noinspection PyUnresolvedReferences
1943 req.dbsession.execute(
1944 update(SpecialNote.__table__)
1945 .where(SpecialNote.device_id == req.tabletsession.device_id)
1946 .where(SpecialNote.era == ERA_NOW)
1947 .values(era=batchdetails.new_era)
1948 )
1950 clear_dirty_tables(req)
1951 audit_upload(req, changelist)
1953 # Performance 2018-11-13:
1954 # - start at 2.407 s
1955 # - remove final temptable clearance and COUNT(*): 1.626 to 2.118 s
1956 # - IN clause using Python literal not temptable: 1.18 to 1.905 s
1957 # - minor tidy: 1.075 to 1.65
1958 # - remove ORDER BY from task indexing: 1.093 to 1.607
1959 # - optimize special note code won't affect this: 1.076 to 1.617
1960 # At this point, entire upload process ~5s.
1961 # - big difference from commit_table() query optimization
1962 # - huge difference from being more careful with mark_table_dirty()
1963 # - further table scanning optimizations: fewer queries
1964 # Overall upload down to ~2.4s
1967def commit_table(req: "CamcopsRequest",
1968 batchdetails: BatchDetails,
1969 table: Table,
1970 clear_dirty: bool = True) -> UploadTableChanges:
1971 """
1972 Commits additions, removals, and preservations for one table.
1974 Should ONLY be called by :func:`commit_all`.
1976 Also updates task indexes.
1978 Args:
1979 req: the :class:`camcops_server.cc_modules.cc_request.CamcopsRequest`
1980 batchdetails: the :class:`BatchDetails`
1981 table: SQLAlchemy :class:`Table`
1982 clear_dirty: remove the table from the record of dirty tables for
1983 this device? (If called from :func:`commit_all`, this should be
1984 ``False``, since it's faster to clear all dirty tables for the
1985 device simultaneously than one-by-one.)
1987 Returns:
1988 an :class:`UploadTableChanges` object
1989 """
1991 # Tried storing PKs in temporary tables, rather than using an IN clause
1992 # with Python values, as per
1993 # https://www.xaprb.com/blog/2006/06/28/why-large-in-clauses-are-problematic/ # noqa
1994 # However, it was slow.
1995 # We can gain a lot of efficiency (empirically) by:
1996 # - Storing PKs in Python
1997 # - Only performing updates when we need to
1998 # - Using a single query per table to get "add/remove/preserve" PKs
2000 # -------------------------------------------------------------------------
2001 # Helpful temporary variables
2002 # -------------------------------------------------------------------------
2003 user_id = req.user_id
2004 device_id = req.tabletsession.device_id
2005 exacttime = req.now
2006 dbsession = req.dbsession
2007 tablename = table.name
2008 batchtime = batchdetails.batchtime
2009 preserving = batchdetails.preserving
2011 # -------------------------------------------------------------------------
2012 # Fetch addition, removal, preservation, current PKs in a single query
2013 # -------------------------------------------------------------------------
2014 tablechanges = UploadTableChanges(table)
2015 serverrecs = get_server_live_records(req, device_id, table,
2016 current_only=False)
2017 for sr in serverrecs:
2018 tablechanges.note_serverrec(sr, preserving=preserving)
2020 # -------------------------------------------------------------------------
2021 # Additions
2022 # -------------------------------------------------------------------------
2023 # Update the records we're adding
2024 addition_pks = tablechanges.addition_pks
2025 if addition_pks:
2026 # log.debug("commit_table: {}, adding server PKs {}",
2027 # tablename, addition_pks)
2028 dbsession.execute(
2029 update(table)
2030 .where(table.c[FN_PK].in_(addition_pks))
2031 .values({
2032 FN_CURRENT: 1,
2033 FN_ADDITION_PENDING: 0,
2034 FN_ADDING_USER_ID: user_id,
2035 FN_WHEN_ADDED_EXACT: exacttime,
2036 FN_WHEN_ADDED_BATCH_UTC: batchtime
2037 })
2038 )
2040 # -------------------------------------------------------------------------
2041 # Removals
2042 # -------------------------------------------------------------------------
2043 # Update the records we're removing
2044 removal_pks = tablechanges.removal_pks
2045 if removal_pks:
2046 # log.debug("commit_table: {}, removing server PKs {}",
2047 # tablename, removal_pks)
2048 dbsession.execute(
2049 update(table)
2050 .where(table.c[FN_PK].in_(removal_pks))
2051 .values(values_delete_now(req, batchdetails))
2052 )
2054 # -------------------------------------------------------------------------
2055 # Preservation
2056 # -------------------------------------------------------------------------
2057 # Preserve necessary records
2058 preservation_pks = tablechanges.preservation_pks
2059 if preservation_pks:
2060 # log.debug("commit_table: {}, preserving server PKs {}",
2061 # tablename, preservation_pks)
2062 new_era = batchdetails.new_era
2063 dbsession.execute(
2064 update(table)
2065 .where(table.c[FN_PK].in_(preservation_pks))
2066 .values(values_preserve_now(req, batchdetails))
2067 )
2068 if not preserving:
2069 # Also preserve/finalize any corresponding special notes
2070 # (2015-02-01), just for records being specifically preserved. If
2071 # we are preserving, this step happens in one go in commit_all()
2072 # (2018-11-13).
2073 # noinspection PyUnresolvedReferences
2074 dbsession.execute(
2075 update(SpecialNote.__table__)
2076 .where(SpecialNote.basetable == tablename)
2077 .where(SpecialNote.device_id == device_id)
2078 .where(SpecialNote.era == ERA_NOW)
2079 .where(exists().select_from(table)
2080 .where(table.c[TABLET_ID_FIELD] == SpecialNote.task_id)
2081 .where(table.c[FN_DEVICE_ID] == SpecialNote.device_id)
2082 .where(table.c[FN_ERA] == new_era))
2083 # ^^^^^^^^^^^^^^^^^^^^^^^^^^
2084 # This bit restricts to records being preserved.
2085 .values(era=new_era)
2086 )
2088 # -------------------------------------------------------------------------
2089 # Update special indexes
2090 # -------------------------------------------------------------------------
2091 update_indexes_and_push_exports(req, batchdetails, tablechanges)
2093 # -------------------------------------------------------------------------
2094 # Remove individually from list of dirty tables?
2095 # -------------------------------------------------------------------------
2096 if clear_dirty:
2097 # noinspection PyUnresolvedReferences
2098 dbsession.execute(
2099 DirtyTable.__table__.delete()
2100 .where(DirtyTable.device_id == device_id)
2101 .where(DirtyTable.tablename == tablename)
2102 )
2103 # ... otherwise a call to clear_dirty_tables() must be made.
2105 if DEBUG_UPLOAD:
2106 log.debug("commit_table: {}", tablechanges)
2108 return tablechanges
2111def rollback_all(req: "CamcopsRequest") -> None:
2112 """
2113 Rolls back all pending changes for a device.
2114 """
2115 tables = get_dirty_tables(req)
2116 for table in tables:
2117 rollback_table(req, table)
2118 clear_dirty_tables(req)
2121def rollback_table(req: "CamcopsRequest", table: Table) -> None:
2122 """
2123 Rolls back changes for an individual table for a device.
2124 """
2125 device_id = req.tabletsession.device_id
2126 # Pending additions
2127 req.dbsession.execute(
2128 table.delete()
2129 .where(table.c[FN_DEVICE_ID] == device_id)
2130 .where(table.c[FN_ADDITION_PENDING])
2131 )
2132 # Pending deletions
2133 req.dbsession.execute(
2134 update(table)
2135 .where(table.c[FN_DEVICE_ID] == device_id)
2136 .where(table.c[FN_REMOVAL_PENDING])
2137 .values({
2138 FN_REMOVAL_PENDING: 0,
2139 FN_WHEN_ADDED_EXACT: None,
2140 FN_WHEN_REMOVED_BATCH_UTC: None,
2141 FN_REMOVING_USER_ID: None,
2142 FN_SUCCESSOR_PK: None
2143 })
2144 )
2145 # Record-specific preservation (set by flag_record_for_preservation())
2146 req.dbsession.execute(
2147 update(table)
2148 .where(table.c[FN_DEVICE_ID] == device_id)
2149 .values({
2150 MOVE_OFF_TABLET_FIELD: 0
2151 })
2152 )
2155def clear_dirty_tables(req: "CamcopsRequest") -> None:
2156 """
2157 Clears the dirty-table list for a device.
2158 """
2159 device_id = req.tabletsession.device_id
2160 # noinspection PyUnresolvedReferences
2161 req.dbsession.execute(
2162 DirtyTable.__table__.delete()
2163 .where(DirtyTable.device_id == device_id)
2164 )
2167# =============================================================================
2168# Additional helper functions for one-step upload
2169# =============================================================================
2171def process_table_for_onestep_upload(
2172 req: "CamcopsRequest",
2173 batchdetails: BatchDetails,
2174 table: Table,
2175 clientpk_name: str,
2176 rows: List[Dict[str, Any]]) -> UploadTableChanges:
2177 """
2178 Performs all upload steps for a table.
2180 Note that we arrive here in a specific and safe table order; search for
2181 :func:`camcops_server.cc_modules.cc_client_api_helpers.upload_commit_order_sorter`.
2183 Args:
2184 req: the :class:`camcops_server.cc_modules.cc_request.CamcopsRequest`
2185 batchdetails: the :class:`BatchDetails`
2186 table: an SQLAlchemy :class:`Table`
2187 clientpk_name: the name of the PK field on the client
2188 rows: a list of rows, where each row is a dictionary mapping field
2189 (column) names to values (those values being encoded as SQL-style
2190 literals in our extended syntax)
2192 Returns:
2193 an :class:`UploadTableChanges` object
2194 """ # noqa
2195 serverrecs = get_server_live_records(
2196 req, req.tabletsession.device_id, table, clientpk_name,
2197 current_only=False)
2198 servercurrentrecs = [r for r in serverrecs if r.current]
2199 if rows and not clientpk_name:
2200 fail_user_error(f"Client-side PK name not specified by client for "
2201 f"non-empty table {table.name!r}")
2202 tablechanges = UploadTableChanges(table)
2203 server_pks_uploaded = [] # type: List[int]
2204 for row in rows:
2205 valuedict = {k: decode_single_value(v) for k, v in row.items()}
2206 urr = upload_record_core(req, batchdetails, table,
2207 clientpk_name, valuedict,
2208 server_live_current_records=servercurrentrecs)
2209 # ... handles addition, modification, preservation, special processing
2210 # But we also make a note of these for indexing:
2211 if urr.oldserverpk is not None:
2212 server_pks_uploaded.append(urr.oldserverpk)
2213 tablechanges.note_urr(urr,
2214 preserving_new_records=batchdetails.preserving)
2215 # Which leaves:
2216 # (*) Deletion (where no record was uploaded at all)
2217 server_pks_for_deletion = [r.server_pk for r in servercurrentrecs
2218 if r.server_pk not in server_pks_uploaded]
2219 if server_pks_for_deletion:
2220 flag_deleted(req, batchdetails, table, server_pks_for_deletion)
2221 tablechanges.note_removal_deleted_pks(server_pks_for_deletion)
2223 # Preserving all records not specifically processed above, too
2224 if batchdetails.preserving:
2225 # Preserve all, including noncurrent:
2226 preserve_all(req, batchdetails, table)
2227 # Note other preserved records, for indexing:
2228 tablechanges.note_preservation_pks(r.server_pk for r in serverrecs)
2230 # (*) Indexing (and push exports)
2231 update_indexes_and_push_exports(req, batchdetails, tablechanges)
2233 if DEBUG_UPLOAD:
2234 log.debug("process_table_for_onestep_upload: {}", tablechanges)
2236 return tablechanges
2239# =============================================================================
2240# Audit functions
2241# =============================================================================
2243def audit(req: "CamcopsRequest",
2244 details: str,
2245 patient_server_pk: int = None,
2246 tablename: str = None,
2247 server_pk: int = None) -> None:
2248 """
2249 Audit something.
2250 """
2251 # Add parameters and pass on:
2252 cc_audit.audit(
2253 req=req,
2254 details=details,
2255 patient_server_pk=patient_server_pk,
2256 table=tablename,
2257 server_pk=server_pk,
2258 device_id=req.tabletsession.device_id, # added
2259 remote_addr=req.remote_addr, # added
2260 user_id=req.user_id, # added
2261 from_console=False, # added
2262 from_dbclient=True # added
2263 )
2266# =============================================================================
2267# Helper functions for single-user mode
2268# =============================================================================
2270def make_single_user_mode_username(client_device_name: str,
2271 patient_pk: int) -> str:
2272 """
2273 Returns the username for single-user mode.
2274 """
2275 return f"user-{client_device_name}-{patient_pk}"
2278def json_patient_info(patient: Patient) -> str:
2279 """
2280 Converts patient details to a string representation of a JSON list (one
2281 patient) containing a single JSON dictionary (detailing that patient), with
2282 keys/formats known to the client.
2284 (One item list to be consistent with patients uploaded from the tablet.)
2286 Args:
2287 patient: :class:`camcops_server.cc_modules.cc_patient.Patient`
2288 """
2289 patient_dict = {
2290 TabletParam.SURNAME: patient.surname,
2291 TabletParam.FORENAME: patient.forename,
2292 TabletParam.SEX: patient.sex,
2293 TabletParam.DOB: format_datetime(patient.dob,
2294 DateFormat.ISO8601_DATE_ONLY),
2295 TabletParam.EMAIL: patient.email,
2296 TabletParam.ADDRESS: patient.address,
2297 TabletParam.GP: patient.gp,
2298 TabletParam.OTHER: patient.other,
2299 }
2300 for idnum in patient.idnums:
2301 key = f"{TabletParam.IDNUM_PREFIX}{idnum.which_idnum}"
2302 patient_dict[key] = idnum.idnum_value
2303 # One item list to be consistent with patients uploaded from the tablet
2304 return json.dumps([patient_dict])
2307def get_single_server_patient(req: "CamcopsRequest") -> Patient:
2308 """
2309 Returns the patient identified by the proquint access key present in this
2310 request, or raises.
2311 """
2312 _ = req.gettext
2314 patient_proquint = get_str_var(req, TabletParam.PATIENT_PROQUINT)
2315 assert patient_proquint is not None # For type checker
2317 try:
2318 uuid_obj = uuid_from_proquint(patient_proquint)
2319 except InvalidProquintException:
2320 # Checksum failed or characters in wrong place
2321 # We'll do the same validation on the client so in theory
2322 # should never get here
2323 fail_user_error(
2324 _(
2325 "There is no patient with access key '{access_key}'. "
2326 "Have you entered the key correctly?"
2327 ).format(access_key=patient_proquint)
2328 )
2330 server_device = Device.get_server_device(req.dbsession)
2332 # noinspection PyUnboundLocalVariable,PyProtectedMember
2333 patient = req.dbsession.query(Patient).filter(
2334 Patient.uuid == uuid_obj,
2335 Patient._device_id == server_device.id,
2336 Patient._era == ERA_NOW,
2337 Patient._current == True # noqa: E712
2338 ).options(joinedload(Patient.task_schedules)).one_or_none()
2340 if patient is None:
2341 fail_user_error(
2342 _(
2343 "There is no patient with access key '{access_key}'. "
2344 "Have you entered the key correctly?"
2345 ).format(access_key=patient_proquint)
2346 )
2348 if not patient.idnums:
2349 # In theory should never happen. The patient must be created with at
2350 # least one ID number. We did see this once in testing (possibly when
2351 # a patient created on a device was registered)
2352 _ = req.gettext
2353 fail_server_error(_("Patient has no ID numbers"))
2355 return patient
2358def get_or_create_single_user(req: "CamcopsRequest",
2359 name: str,
2360 patient: Patient) -> Tuple[User, str]:
2361 """
2362 Creates a user for a patient (who's using single-user mode).
2364 The user is associated (via its name) with the combination of a client
2365 device and a patient. (If a device is re-registered to another patient, the
2366 username will change.)
2368 If the username already exists, then since we can't look up the password
2369 (it's irreversibly encrypted), we will set it afresh.
2371 - Why is a user associated with a patient? So we can enforce that the user
2372 can upload only data relating to that patient.
2374 - Why is a user associated with a device?
2376 - If it is: then two users (e.g. "Device1-Bob" and "Device2-Bob") can
2377 independently work with the same patient. This will be highly
2378 confusing (mainly because it will allow "double" copies of tasks to be
2379 created, though only by manually entering things twice).
2381 - If it isn't (e.g. user "Bob"): then, because registering the patient on
2382 Device2 will reset the password for the user, registering a new device
2383 for a patient will "take over" from a previous device. That has some
2384 potential for data loss if work was in progress (incomplete tasks won't
2385 be uploadable any more, and re-registering [to fix the password on the
2386 first device] would delete data).
2388 - Since some confusion is better than some data loss, we associate users
2389 with a device/patient combination.
2391 Args:
2392 req:
2393 a :class:`camcops_server.cc_modules.cc_request.CamcopsRequest`
2394 name:
2395 username
2396 patient:
2397 associated :class:`camcops_server.cc_modules.cc_patient.Patient`,
2398 which also tells us the group in which to place this user
2400 Returns:
2401 tuple: :class:`camcops_server.cc_modules.cc_user.User`, password
2403 """
2404 dbsession = req.dbsession
2405 password = random_password()
2406 group = patient.group
2407 assert group is not None # for type checker
2409 user = User.get_user_by_name(dbsession, name)
2410 creating_new_user = user is None
2411 if creating_new_user:
2412 # Create a fresh user.
2413 user = User(username=name)
2414 user.upload_group = group
2415 user.auto_generated = True
2416 user.superuser = False # should be redundant!
2417 # noinspection PyProtectedMember
2418 user.single_patient_pk = patient._pk
2419 user.set_password(req, password)
2420 if creating_new_user:
2421 dbsession.add(user)
2422 # As the username is based on a UUID, we're pretty sure another
2423 # request won't have created the same user, otherwise we'd need
2424 # to catch IntegrityError
2425 dbsession.flush()
2427 membership = UserGroupMembership(
2428 user_id=user.id,
2429 group_id=group.id,
2430 )
2431 membership.may_register_devices = True
2432 membership.may_upload = True
2433 user.user_group_memberships = [membership] # ... only these permissions
2435 return user, password
2438def random_password(length: int = 32) -> str:
2439 """
2440 Create a random password.
2441 """
2442 # Not trying anything clever with distributions of letters, digits etc
2443 characters = string.ascii_letters + string.digits + string.punctuation
2444 # We use secrets.choice() rather than random.choices() as it's better
2445 # for security/cryptography purposes.
2446 return "".join(secrets.choice(characters) for _ in range(length))
2449def get_task_schedules(req: "CamcopsRequest",
2450 patient: Patient) -> str:
2451 """
2452 Gets a JSON string representation of the task schedules for a specified
2453 patient.
2454 """
2455 dbsession = req.dbsession
2457 schedules = []
2459 for pts in patient.task_schedules:
2460 if pts.start_datetime is None:
2461 # Minutes granularity so we are consistent with the form
2462 pts.start_datetime = req.now_utc.replace(second=0, microsecond=0)
2463 dbsession.add(pts)
2465 items = []
2467 for task_info in pts.get_list_of_scheduled_tasks(req):
2468 due_from = task_info.start_datetime.to_iso8601_string()
2469 due_by = task_info.end_datetime.to_iso8601_string()
2471 complete = False
2472 when_completed = None
2473 task = task_info.task
2474 if task:
2475 complete = task.is_complete()
2476 if complete and task.when_last_modified:
2477 when_completed = task.when_last_modified.to_iso8601_string() # noqa
2479 if pts.settings is not None:
2480 settings = pts.settings.get(task_info.tablename, {})
2481 else:
2482 settings = {}
2484 items.append({
2485 TabletParam.TABLE: task_info.tablename,
2486 TabletParam.ANONYMOUS: task_info.is_anonymous,
2487 TabletParam.SETTINGS: settings,
2488 TabletParam.DUE_FROM: due_from,
2489 TabletParam.DUE_BY: due_by,
2490 TabletParam.COMPLETE: complete,
2491 TabletParam.WHEN_COMPLETED: when_completed,
2492 })
2494 schedules.append({
2495 TabletParam.TASK_SCHEDULE_NAME: pts.task_schedule.name,
2496 TabletParam.TASK_SCHEDULE_ITEMS: items,
2497 })
2499 return json.dumps(schedules)
2502# =============================================================================
2503# Action processors: allowed to any user
2504# =============================================================================
2505# If they return None, the framework uses the operation name as the reply in
2506# the success message. Not returning anything is the same as returning None.
2507# Authentication is performed in advance of these.
2509def op_check_device_registered(req: "CamcopsRequest") -> None:
2510 """
2511 Check that a device is registered, or raise
2512 :exc:`UserErrorException`.
2513 """
2514 req.tabletsession.ensure_device_registered()
2517def op_register_patient(req: "CamcopsRequest") -> Dict[str, Any]:
2518 """
2519 Registers a patient. That is, the client provides an access key. If all
2520 is well, the server returns details of that patient, as well as key
2521 server parameters, plus (if required) the username/password to use.
2522 """
2523 # -------------------------------------------------------------------------
2524 # Patient details
2525 # -------------------------------------------------------------------------
2526 patient = get_single_server_patient(req) # may fail/raise
2527 patient_info = json_patient_info(patient)
2528 reply_dict = {
2529 TabletParam.PATIENT_INFO: patient_info,
2530 }
2532 # -------------------------------------------------------------------------
2533 # Username/password
2534 # -------------------------------------------------------------------------
2535 client_device_name = get_str_var(req, TabletParam.DEVICE)
2536 # noinspection PyProtectedMember
2537 user_name = make_single_user_mode_username(client_device_name, patient._pk)
2538 user, password = get_or_create_single_user(req, user_name, patient)
2539 reply_dict[TabletParam.USER] = user.username
2540 reply_dict[TabletParam.PASSWORD] = password
2542 # -------------------------------------------------------------------------
2543 # Intellectual property settings
2544 # -------------------------------------------------------------------------
2545 ip_use = patient.group.ip_use or IpUse()
2546 # ... if the group doesn't have an associated ip_use object, use defaults
2547 ip_dict = {
2548 TabletParam.IP_USE_COMMERCIAL: int(ip_use.commercial),
2549 TabletParam.IP_USE_CLINICAL: int(ip_use.clinical),
2550 TabletParam.IP_USE_EDUCATIONAL: int(ip_use.educational),
2551 TabletParam.IP_USE_RESEARCH: int(ip_use.research),
2552 }
2553 reply_dict[TabletParam.IP_USE_INFO] = json.dumps(ip_dict)
2555 return reply_dict
2558# =============================================================================
2559# Action processors that require REGISTRATION privilege
2560# =============================================================================
2562def op_register_device(req: "CamcopsRequest") -> Dict[str, Any]:
2563 """
2564 Register a device with the server.
2566 Returns:
2567 server information dictionary (from :func:`get_server_id_info`)
2568 """
2569 dbsession = req.dbsession
2570 ts = req.tabletsession
2571 device_friendly_name = get_str_var(req, TabletParam.DEVICE_FRIENDLY_NAME,
2572 mandatory=False)
2573 # noinspection PyUnresolvedReferences
2574 device_exists = exists_in_table(
2575 dbsession,
2576 Device.__table__,
2577 Device.name == ts.device_name
2578 )
2579 if device_exists:
2580 # device already registered, but accept re-registration
2581 # noinspection PyUnresolvedReferences
2582 dbsession.execute(
2583 update(Device.__table__)
2584 .where(Device.name == ts.device_name)
2585 .values(friendly_name=device_friendly_name,
2586 camcops_version=ts.tablet_version_str,
2587 registered_by_user_id=req.user_id,
2588 when_registered_utc=req.now_utc)
2589 )
2590 else:
2591 # new registration
2592 try:
2593 # noinspection PyUnresolvedReferences
2594 dbsession.execute(
2595 Device.__table__.insert()
2596 .values(name=ts.device_name,
2597 friendly_name=device_friendly_name,
2598 camcops_version=ts.tablet_version_str,
2599 registered_by_user_id=req.user_id,
2600 when_registered_utc=req.now_utc)
2601 )
2602 except IntegrityError:
2603 fail_user_error(INSERT_FAILED)
2605 ts.reload_device()
2606 audit(
2607 req,
2608 f"register, device_id={ts.device_id}, "
2609 f"friendly_name={device_friendly_name}",
2610 tablename=Device.__tablename__
2611 )
2612 return get_server_id_info(req)
2615def op_get_extra_strings(req: "CamcopsRequest") -> Dict[str, str]:
2616 """
2617 Fetch all local extra strings from the server.
2619 Returns:
2620 a SELECT-style reply (see :func:`get_select_reply`) for the
2621 extra-string table
2622 """
2623 fields = [ExtraStringFieldNames.TASK,
2624 ExtraStringFieldNames.NAME,
2625 ExtraStringFieldNames.LANGUAGE,
2626 ExtraStringFieldNames.VALUE]
2627 rows = req.get_all_extra_strings()
2628 reply = get_select_reply(fields, rows)
2629 audit(req, "get_extra_strings")
2630 return reply
2633# noinspection PyUnusedLocal
2634def op_get_allowed_tables(req: "CamcopsRequest") -> Dict[str, str]:
2635 """
2636 Returns the names of all possible tables on the server, each paired with
2637 the minimum client (tablet) version that will be accepted for each table.
2638 (Typically these are all the same as the minimum global tablet version.)
2640 Uses the SELECT-like syntax (see :func:`get_select_reply`).
2641 """
2642 tables_versions = all_tables_with_min_client_version()
2643 fields = [AllowedTablesFieldNames.TABLENAME,
2644 AllowedTablesFieldNames.MIN_CLIENT_VERSION]
2645 rows = [[k, str(v)] for k, v in tables_versions.items()]
2646 reply = get_select_reply(fields, rows)
2647 audit(req, "get_allowed_tables")
2648 return reply
2651def op_get_task_schedules(req: "CamcopsRequest") -> Dict[str, str]:
2652 """
2653 Return details of the task schedules for the patient associated with
2654 this request, for single-user mode. Also returns details of the single
2655 patient, in case that's changed.
2656 """
2657 patient = get_single_server_patient(req)
2658 patient_info = json_patient_info(patient)
2659 task_schedules = get_task_schedules(req, patient)
2660 return {
2661 TabletParam.PATIENT_INFO: patient_info,
2662 TabletParam.TASK_SCHEDULES: task_schedules,
2663 }
2666# =============================================================================
2667# Action processors that require UPLOAD privilege
2668# =============================================================================
2670# noinspection PyUnusedLocal
2671def op_check_upload_user_and_device(req: "CamcopsRequest") -> None:
2672 """
2673 Stub function for the operation to check that a user is valid.
2675 To get this far, the user has to be valid, so this function doesn't
2676 actually have to do anything.
2677 """
2678 pass # don't need to do anything!
2681# noinspection PyUnusedLocal
2682def op_get_id_info(req: "CamcopsRequest") -> Dict[str, Any]:
2683 """
2684 Fetch server ID information; see :func:`get_server_id_info`.
2685 """
2686 return get_server_id_info(req)
2689def op_start_upload(req: "CamcopsRequest") -> None:
2690 """
2691 Begin an upload.
2692 """
2693 start_device_upload_batch(req)
2696def op_end_upload(req: "CamcopsRequest") -> None:
2697 """
2698 Ends an upload and commits changes.
2699 """
2700 batchdetails = get_batch_details(req)
2701 # ensure it's the same user finishing as starting!
2702 end_device_upload_batch(req, batchdetails)
2705def op_upload_table(req: "CamcopsRequest") -> str:
2706 """
2707 Upload a table.
2709 Incoming information in the POST request includes a CSV list of fields, a
2710 count of the number of records being provided, and a set of variables named
2711 ``record0`` ... ``record{nrecords - 1}``, each containing a CSV list of
2712 SQL-encoded values.
2714 Typically used for smaller tables, i.e. most except for BLOBs.
2715 """
2716 table = get_table_from_req(req, TabletParam.TABLE)
2718 allowed_nonexistent_fields = [] # type: List[str]
2719 # noinspection PyUnresolvedReferences
2720 if req.tabletsession.cope_with_old_idnums and table == Patient.__table__:
2721 for x in range(1, NUMBER_OF_IDNUMS_DEFUNCT + 1):
2722 allowed_nonexistent_fields.extend([
2723 FP_ID_NUM + str(x),
2724 FP_ID_DESC + str(x),
2725 FP_ID_SHORT_DESC + str(x)
2726 ])
2728 fields = get_fields_from_post_var(
2729 req, table, TabletParam.FIELDS,
2730 allowed_nonexistent_fields=allowed_nonexistent_fields)
2731 nrecords = get_int_var(req, TabletParam.NRECORDS)
2733 nfields = len(fields)
2734 if nfields < 1:
2735 fail_user_error(
2736 f"{TabletParam.FIELDS}={nfields}: can't be less than 1")
2737 if nrecords < 0:
2738 fail_user_error(
2739 f"{TabletParam.NRECORDS}={nrecords}: can't be less than 0")
2741 batchdetails = get_batch_details(req)
2743 ts = req.tabletsession
2744 if ts.explicit_pkname_for_upload_table: # q.v.
2745 # New client: tells us the PK name explicitly.
2746 clientpk_name = get_single_field_from_post_var(req, table,
2747 TabletParam.PKNAME)
2748 else:
2749 # Old client. Either (a) old Titanium client, in which the client PK
2750 # is in fields[0], or (b) an early C++ client, in which there was no
2751 # guaranteed order (and no explicit PK name was sent). However, in
2752 # either case, the client PK name was (is) always "id".
2753 clientpk_name = TABLET_ID_FIELD
2754 ensure_valid_field_name(table, clientpk_name)
2755 server_pks_uploaded = [] # type: List[int]
2756 n_new = 0
2757 n_modified = 0
2758 n_identical = 0
2759 dirty = False
2760 serverrecs = get_server_live_records(req, ts.device_id, table,
2761 clientpk_name=clientpk_name,
2762 current_only=True)
2763 for r in range(nrecords):
2764 recname = TabletParam.RECORD_PREFIX + str(r)
2765 values = get_values_from_post_var(req, recname)
2766 nvalues = len(values)
2767 if nvalues != nfields:
2768 errmsg = (
2769 f"Number of fields in field list ({nfields}) doesn't match "
2770 f"number of values in record {r} ({nvalues})"
2771 )
2772 log.warning(errmsg + f"\nfields: {fields!r}\nvalues: {values!r}")
2773 fail_user_error(errmsg)
2774 valuedict = dict(zip(fields, values))
2775 # log.debug("table {!r}, record {}: {!r}", table.name, r, valuedict)
2776 # CORE: CALLS upload_record_core
2777 urr = upload_record_core(
2778 req, batchdetails, table, clientpk_name, valuedict,
2779 server_live_current_records=serverrecs)
2780 if urr.oldserverpk is not None: # was an existing record
2781 server_pks_uploaded.append(urr.oldserverpk)
2782 if urr.newserverpk is None:
2783 n_identical += 1
2784 else:
2785 n_modified += 1
2786 else: # entirely new
2787 n_new += 1
2788 if urr.dirty:
2789 dirty = True
2791 # Now deal with any ABSENT (not in uploaded data set) conditions.
2792 server_pks_for_deletion = [r.server_pk for r in serverrecs
2793 if r.server_pk not in server_pks_uploaded]
2794 # Note that "deletion" means "end of the line"; records that are modified
2795 # and replaced were handled by upload_record_core().
2796 n_deleted = len(server_pks_for_deletion)
2797 if n_deleted > 0:
2798 flag_deleted(req, batchdetails, table, server_pks_for_deletion)
2800 # Set dirty/clean status
2801 if (dirty or n_new > 0 or n_modified > 0 or n_deleted > 0 or
2802 any(sr.move_off_tablet for sr in serverrecs)):
2803 # ... checks on n_new and n_modified are redundant; dirty will be True
2804 mark_table_dirty(req, table)
2805 elif batchdetails.preserving and not serverrecs:
2806 # We've scanned this table, and there would be no work to do to
2807 # preserve records from previous uploads.
2808 mark_table_clean(req, table)
2810 # Special for old tablets:
2811 # noinspection PyUnresolvedReferences
2812 if req.tabletsession.cope_with_old_idnums and table == Patient.__table__:
2813 # noinspection PyUnresolvedReferences
2814 mark_table_dirty(req, PatientIdNum.__table__)
2815 # Mark patient ID numbers for deletion if their parent Patient is
2816 # similarly being marked for deletion
2817 # noinspection PyUnresolvedReferences,PyProtectedMember
2818 req.dbsession.execute(
2819 update(PatientIdNum.__table__)
2820 .where(PatientIdNum._device_id == Patient._device_id)
2821 .where(PatientIdNum._era == ERA_NOW)
2822 .where(PatientIdNum.patient_id == Patient.id)
2823 .where(Patient._pk.in_(server_pks_for_deletion))
2824 .where(Patient._era == ERA_NOW) # shouldn't be in doubt!
2825 .values(_removal_pending=1,
2826 _successor_pk=None)
2827 )
2829 # Auditing occurs at commit_all.
2830 log.info("Upload successful; {n} records uploaded to table {t} "
2831 "({new} new, {mod} modified, {i} identical, {nd} deleted)",
2832 n=nrecords, t=table.name, new=n_new, mod=n_modified,
2833 i=n_identical, nd=n_deleted)
2834 return f"Table {table.name} upload successful"
2837def op_upload_record(req: "CamcopsRequest") -> str:
2838 """
2839 Upload an individual record. (Typically used for BLOBs.)
2840 Incoming POST information includes a CSV list of fields and a CSV list of
2841 values.
2842 """
2843 batchdetails = get_batch_details(req)
2844 table = get_table_from_req(req, TabletParam.TABLE)
2845 clientpk_name = get_single_field_from_post_var(req, table,
2846 TabletParam.PKNAME)
2847 valuedict = get_fields_and_values(req, table,
2848 TabletParam.FIELDS, TabletParam.VALUES)
2849 urr = upload_record_core(
2850 req, batchdetails, table, clientpk_name, valuedict
2851 )
2852 if urr.dirty:
2853 mark_table_dirty(req, table)
2854 if urr.oldserverpk is None:
2855 log.info("upload-insert")
2856 return "UPLOAD-INSERT"
2857 else:
2858 if urr.newserverpk is None:
2859 log.info("upload-update: skipping existing record")
2860 else:
2861 log.info("upload-update")
2862 return "UPLOAD-UPDATE"
2863 # Auditing occurs at commit_all.
2866def op_upload_empty_tables(req: "CamcopsRequest") -> str:
2867 """
2868 The tablet supplies a list of tables that are empty at its end, and we
2869 will 'wipe' all appropriate tables; this reduces the number of HTTP
2870 requests.
2871 """
2872 tables = get_tables_from_post_var(req, TabletParam.TABLES)
2873 batchdetails = get_batch_details(req)
2874 to_dirty = [] # type: List[Table]
2875 to_clean = [] # type: List[Table]
2876 for table in tables:
2877 nrows_affected = flag_all_records_deleted(req, table)
2878 if nrows_affected > 0:
2879 to_dirty.append(table)
2880 elif batchdetails.preserving:
2881 # There are no records in the current era for this device.
2882 to_clean.append(table)
2883 # In the fewest number of queries:
2884 mark_tables_dirty(req, to_dirty)
2885 mark_tables_clean(req, to_clean)
2886 log.info("upload_empty_tables")
2887 # Auditing occurs at commit_all.
2888 return "UPLOAD-EMPTY-TABLES"
2891def op_start_preservation(req: "CamcopsRequest") -> str:
2892 """
2893 Marks this upload batch as one in which all records will be preserved
2894 (i.e. moved from NOW-era to an older era, so they can be deleted safely
2895 from the tablet).
2897 Without this, individual records can still be marked for preservation if
2898 their MOVE_OFF_TABLET_FIELD field (``_move_off_tablet``) is set; see
2899 :func:`upload_record` and the functions it calls.
2900 """
2901 get_batch_details(req)
2902 start_preserving(req)
2903 log.info("start_preservation successful")
2904 # Auditing occurs at commit_all.
2905 return "STARTPRESERVATION"
2908def op_delete_where_key_not(req: "CamcopsRequest") -> str:
2909 """
2910 Marks records for deletion, for a device/table, where the client PK
2911 is not in a specified list.
2912 """
2913 table = get_table_from_req(req, TabletParam.TABLE)
2914 clientpk_name = get_single_field_from_post_var(
2915 req, table, TabletParam.PKNAME)
2916 clientpk_values = get_values_from_post_var(req, TabletParam.PKVALUES)
2918 get_batch_details(req)
2919 flag_deleted_where_clientpk_not(req, table, clientpk_name, clientpk_values)
2920 # Auditing occurs at commit_all.
2921 # log.info("delete_where_key_not successful; table {} trimmed", table)
2922 return "Trimmed"
2925def op_which_keys_to_send(req: "CamcopsRequest") -> str:
2926 """
2927 Intended use: "For my device, and a specified table, here are my client-
2928 side PKs (as a CSV list), and the modification dates for each corresponding
2929 record (as a CSV list). Please tell me which records have mismatching dates
2930 on the server, i.e. those that I need to re-upload."
2932 Used particularly for BLOBs, to reduce traffic, i.e. so we don't have to
2933 send a lot of BLOBs.
2935 Note new ``TabletParam.MOVE_OFF_TABLET_VALUES`` parameter in server v2.3.0,
2936 with bugfix for pre-2.3.0 clients that won't send this; see changelog.
2937 """
2938 # -------------------------------------------------------------------------
2939 # Get details
2940 # -------------------------------------------------------------------------
2941 try:
2942 table = get_table_from_req(req, TabletParam.TABLE)
2943 except IgnoringAntiqueTableException:
2944 raise IgnoringAntiqueTableException("")
2945 clientpk_name = get_single_field_from_post_var(req, table,
2946 TabletParam.PKNAME)
2947 clientpk_values = get_values_from_post_var(req, TabletParam.PKVALUES,
2948 mandatory=False)
2949 # ... should be autoconverted to int, but we check below
2950 client_dates = get_values_from_post_var(req, TabletParam.DATEVALUES,
2951 mandatory=False)
2952 # ... will be in string format
2954 npkvalues = len(clientpk_values)
2955 ndatevalues = len(client_dates)
2956 if npkvalues != ndatevalues:
2957 fail_user_error(
2958 f"Number of PK values ({npkvalues}) doesn't match number of dates "
2959 f"({ndatevalues})")
2961 # v2.3.0:
2962 move_off_tablet_values = [] # type: List[int] # for type checker
2963 if req.has_param(TabletParam.MOVE_OFF_TABLET_VALUES):
2964 client_reports_move_off_tablet = True
2965 move_off_tablet_values = get_values_from_post_var(
2966 req, TabletParam.MOVE_OFF_TABLET_VALUES, mandatory=True)
2967 # ... should be autoconverted to int
2968 n_motv = len(move_off_tablet_values)
2969 if n_motv != npkvalues:
2970 fail_user_error(
2971 f"Number of move-off-tablet values ({n_motv}) doesn't match "
2972 f"number of PKs ({npkvalues})")
2973 try:
2974 move_off_tablet_values = [bool(x) for x in move_off_tablet_values]
2975 except (TypeError, ValueError):
2976 fail_user_error(
2977 f"Bad move-off-tablet values: {move_off_tablet_values!r}")
2978 else:
2979 client_reports_move_off_tablet = False
2980 log.warning(
2981 "op_which_keys_to_send: old client not reporting "
2982 "{}; requesting all records",
2983 TabletParam.MOVE_OFF_TABLET_VALUES
2984 )
2986 clientinfo = [] # type: List[WhichKeyToSendInfo]
2988 for i in range(npkvalues):
2989 cpkv = clientpk_values[i]
2990 if not isinstance(cpkv, int):
2991 fail_user_error(f"Bad (non-integer) client PK: {cpkv!r}")
2992 dt = None # for type checker
2993 try:
2994 dt = coerce_to_pendulum(client_dates[i])
2995 if dt is None:
2996 fail_user_error(f"Missing date/time for client PK {cpkv}")
2997 except ValueError:
2998 fail_user_error(f"Bad date/time: {client_dates[i]!r}")
2999 clientinfo.append(WhichKeyToSendInfo(
3000 client_pk=cpkv,
3001 client_when=dt,
3002 client_move_off_tablet=(
3003 move_off_tablet_values[i]
3004 if client_reports_move_off_tablet else False
3005 )
3006 ))
3008 # -------------------------------------------------------------------------
3009 # Work out the answer
3010 # -------------------------------------------------------------------------
3011 batchdetails = get_batch_details(req)
3013 # 1. The client sends us all its PKs. So "delete" anything not in that
3014 # list.
3015 flag_deleted_where_clientpk_not(req, table, clientpk_name, clientpk_values)
3017 # 2. See which ones are new or updates.
3018 client_pks_needed = [] # type: List[int]
3019 client_pk_to_serverrec = client_pks_that_exist(
3020 req, table, clientpk_name, clientpk_values)
3021 for wk in clientinfo:
3022 if client_reports_move_off_tablet:
3023 if wk.client_pk not in client_pk_to_serverrec:
3024 # New on the client; we want it
3025 client_pks_needed.append(wk.client_pk)
3026 else:
3027 # We know about some version of this client record.
3028 serverrec = client_pk_to_serverrec[wk.client_pk]
3029 if serverrec.server_when != wk.client_when:
3030 # Modified on the client; we want it
3031 client_pks_needed.append(wk.client_pk)
3032 elif serverrec.move_off_tablet != wk.client_move_off_tablet:
3033 # Not modified on the client. But it is being preserved.
3034 # We don't need to ask the client for it again, but we do
3035 # need to mark the preservation.
3036 flag_record_for_preservation(req, batchdetails, table,
3037 serverrec.server_pk)
3039 else:
3040 # Client hasn't told us about the _move_off_tablet flag. Always
3041 # request the record (workaround potential bug in old clients).
3042 client_pks_needed.append(wk.client_pk)
3044 # Success
3045 pk_csv_list = ",".join([str(x) for x in client_pks_needed if x is not None]) # noqa
3046 # log.info("which_keys_to_send successful: table {}", table.name)
3047 return pk_csv_list
3050def op_validate_patients(req: "CamcopsRequest") -> str:
3051 """
3052 As of v2.3.0, the client can use this command to validate patients against
3053 arbitrary server criteria -- definitely the upload/finalize ID policies,
3054 but potentially also other criteria of the server's (like matching against
3055 a bank of predefined patients).
3057 Compare ``NetworkManager::getPatientInfoJson()`` on the client.
3059 There is a slight weakness with respect to "single-patient" users, in that
3060 the client *asks* if the patients are OK (rather than the server
3061 *enforcing* that they are OK, via hooks into :func:`op_upload_table`,
3062 :func:`op_upload_record`, :func:`op_upload_entire_database` -- made more
3063 complex because ID numbers are not uploaded to the same table...). In
3064 principle, the weakness is that a user could (a) crack their assigned
3065 password and (b) rework the CamCOPS client, in order to upload "bad"
3066 patient data into their assigned group.
3068 todo:
3069 address this by having the server *require* patient validation for
3070 all uploads?
3072 """
3073 pt_json_list = get_json_from_post_var(req, TabletParam.PATIENT_INFO,
3074 decoder=PATIENT_INFO_JSON_DECODER,
3075 mandatory=True)
3076 if not isinstance(pt_json_list, list):
3077 fail_user_error("Top-level JSON is not a list")
3078 group = Group.get_group_by_id(req.dbsession, req.user.upload_group_id)
3079 for pt_dict in pt_json_list:
3080 ensure_valid_patient_json(req, group, pt_dict)
3081 return SUCCESS_MSG
3084def op_upload_entire_database(req: "CamcopsRequest") -> str:
3085 """
3086 Perform a one-step upload of the entire database.
3088 - From v2.3.0.
3089 - Therefore, we do not have to cope with old-style ID numbers.
3090 """
3091 # Roll back and clear any outstanding changes
3092 clear_device_upload_batch(req)
3094 # Fetch the JSON, with sanity checks
3095 preserving = get_bool_int_var(req, TabletParam.FINALIZING)
3096 pknameinfo = get_json_from_post_var(
3097 req, TabletParam.PKNAMEINFO, decoder=DB_JSON_DECODER, mandatory=True)
3098 if not isinstance(pknameinfo, dict):
3099 fail_user_error("PK name info JSON is not a dict")
3100 dbdata = get_json_from_post_var(
3101 req, TabletParam.DBDATA, decoder=DB_JSON_DECODER, mandatory=True)
3102 if not isinstance(dbdata, dict):
3103 fail_user_error("Database data JSON is not a dict")
3105 # Sanity checks
3106 dbdata_tablenames = sorted(dbdata.keys())
3107 pkinfo_tablenames = sorted(pknameinfo.keys())
3108 if pkinfo_tablenames != dbdata_tablenames:
3109 fail_user_error("Table names don't match from (1) DB data (2) PK info")
3110 duff_tablenames = sorted(list(set(dbdata_tablenames) -
3111 set(CLIENT_TABLE_MAP.keys())))
3112 if duff_tablenames:
3113 fail_user_error(
3114 f"Attempt to upload nonexistent tables: {duff_tablenames!r}")
3116 # Perform the upload
3117 batchdetails = BatchDetails(req.now_utc, preserving=preserving,
3118 onestep=True) # NB special "onestep" option
3119 # Process the tables in a certain order:
3120 tables = sorted(CLIENT_TABLE_MAP.values(),
3121 key=upload_commit_order_sorter)
3122 changelist = [] # type: List[UploadTableChanges]
3123 for table in tables:
3124 clientpk_name = pknameinfo.get(table.name, "")
3125 rows = dbdata.get(table.name, [])
3126 tablechanges = process_table_for_onestep_upload(
3127 req, batchdetails, table, clientpk_name, rows)
3128 changelist.append(tablechanges)
3130 # Audit
3131 audit_upload(req, changelist)
3133 # Done
3134 return SUCCESS_MSG
3137# =============================================================================
3138# Action maps
3139# =============================================================================
3141class Operations:
3142 """
3143 Constants giving the name of operations (commands) accepted by this API.
3144 """
3145 CHECK_DEVICE_REGISTERED = "check_device_registered"
3146 CHECK_UPLOAD_USER_DEVICE = "check_upload_user_and_device"
3147 DELETE_WHERE_KEY_NOT = "delete_where_key_not"
3148 END_UPLOAD = "end_upload"
3149 GET_ALLOWED_TABLES = "get_allowed_tables" # v2.2.0
3150 GET_EXTRA_STRINGS = "get_extra_strings"
3151 GET_ID_INFO = "get_id_info"
3152 GET_TASK_SCHEDULES = "get_task_schedules" # v2.4.0
3153 REGISTER = "register"
3154 REGISTER_PATIENT = "register_patient" # v2.4.0
3155 START_PRESERVATION = "start_preservation"
3156 START_UPLOAD = "start_upload"
3157 UPLOAD_EMPTY_TABLES = "upload_empty_tables"
3158 UPLOAD_ENTIRE_DATABASE = "upload_entire_database" # v2.3.0
3159 UPLOAD_RECORD = "upload_record"
3160 UPLOAD_TABLE = "upload_table"
3161 VALIDATE_PATIENTS = "validate_patients" # v2.3.0
3162 WHICH_KEYS_TO_SEND = "which_keys_to_send"
3165OPERATIONS_ANYONE = {
3166 Operations.CHECK_DEVICE_REGISTERED: op_check_device_registered,
3167 # Anyone can register a patient provided they have the right unique code
3168 Operations.REGISTER_PATIENT: op_register_patient,
3169}
3170OPERATIONS_REGISTRATION = {
3171 Operations.GET_ALLOWED_TABLES: op_get_allowed_tables, # v2.2.0
3172 Operations.GET_EXTRA_STRINGS: op_get_extra_strings,
3173 Operations.GET_TASK_SCHEDULES: op_get_task_schedules,
3174 Operations.REGISTER: op_register_device,
3175}
3176OPERATIONS_UPLOAD = {
3177 Operations.CHECK_UPLOAD_USER_DEVICE: op_check_upload_user_and_device,
3178 Operations.DELETE_WHERE_KEY_NOT: op_delete_where_key_not,
3179 Operations.END_UPLOAD: op_end_upload,
3180 Operations.GET_ID_INFO: op_get_id_info,
3181 Operations.START_PRESERVATION: op_start_preservation,
3182 Operations.START_UPLOAD: op_start_upload,
3183 Operations.UPLOAD_EMPTY_TABLES: op_upload_empty_tables,
3184 Operations.UPLOAD_ENTIRE_DATABASE: op_upload_entire_database,
3185 Operations.UPLOAD_RECORD: op_upload_record,
3186 Operations.UPLOAD_TABLE: op_upload_table,
3187 Operations.VALIDATE_PATIENTS: op_validate_patients, # v2.3.0
3188 Operations.WHICH_KEYS_TO_SEND: op_which_keys_to_send,
3189}
3192# =============================================================================
3193# Client API main functions
3194# =============================================================================
3196def main_client_api(req: "CamcopsRequest") -> Dict[str, str]:
3197 """
3198 Main HTTP processor.
3200 For success, returns a dictionary to send (will use status '200 OK')
3201 For failure, raises an exception.
3202 """
3203 # log.info("CamCOPS database script starting at {}",
3204 # format_datetime(req.now, DateFormat.ISO8601))
3205 ts = req.tabletsession
3206 fn = None
3208 if ts.operation in OPERATIONS_ANYONE:
3209 fn = OPERATIONS_ANYONE.get(ts.operation)
3211 elif ts.operation in OPERATIONS_REGISTRATION:
3212 ts.ensure_valid_user_for_device_registration()
3213 fn = OPERATIONS_REGISTRATION.get(ts.operation)
3215 elif ts.operation in OPERATIONS_UPLOAD:
3216 ts.ensure_valid_device_and_user_for_uploading()
3217 fn = OPERATIONS_UPLOAD.get(ts.operation)
3219 if not fn:
3220 fail_unsupported_operation(ts.operation)
3221 result = fn(req)
3222 if result is None:
3223 # generic success
3224 result = {TabletParam.RESULT: ts.operation}
3225 elif not isinstance(result, dict):
3226 # convert strings (etc.) to a dictionary
3227 result = {TabletParam.RESULT: result}
3228 return result
3231@view_config(route_name=Routes.CLIENT_API, permission=NO_PERMISSION_REQUIRED)
3232@view_config(route_name=Routes.CLIENT_API_ALIAS,
3233 permission=NO_PERMISSION_REQUIRED)
3234def client_api(req: "CamcopsRequest") -> Response:
3235 """
3236 View for client API. All tablet interaction comes through here.
3237 Wraps :func:`main_client_api`.
3239 Internally, replies are managed as dictionaries.
3240 For the final reply, the dictionary is converted to text in this format:
3242 .. code-block:: none
3244 k1:v1
3245 k2:v2
3246 k3:v3
3247 ...
3248 """
3249 # log.debug("{!r}", req.environ)
3250 # log.debug("{!r}", req.params)
3251 t0 = time.time() # in seconds
3253 try:
3254 resultdict = main_client_api(req)
3255 resultdict[TabletParam.SUCCESS] = SUCCESS_CODE
3256 status = '200 OK'
3258 except IgnoringAntiqueTableException as e:
3259 log.warning(IGNORING_ANTIQUE_TABLE_MESSAGE)
3260 resultdict = {
3261 TabletParam.RESULT: escape_newlines(str(e)),
3262 TabletParam.SUCCESS: SUCCESS_CODE,
3263 }
3264 status = '200 OK'
3266 except UserErrorException as e:
3267 log.warning("CLIENT-SIDE SCRIPT ERROR: {}", e)
3268 resultdict = {
3269 TabletParam.SUCCESS: FAILURE_CODE,
3270 TabletParam.ERROR: escape_newlines(str(e))
3271 }
3272 status = '200 OK'
3274 except ServerErrorException as e:
3275 log.error("SERVER-SIDE SCRIPT ERROR: {}", e)
3276 # rollback? Not sure
3277 resultdict = {
3278 TabletParam.SUCCESS: FAILURE_CODE,
3279 TabletParam.ERROR: escape_newlines(str(e))
3280 }
3281 status = "503 Database Unavailable: " + str(e)
3283 except Exception as e:
3284 # All other exceptions. May include database write failures.
3285 # Let's return with status '200 OK'; though this seems dumb, it means
3286 # the tablet user will at least see the message.
3287 log.exception("Unhandled exception") # + traceback.format_exc()
3288 resultdict = {
3289 TabletParam.SUCCESS: FAILURE_CODE,
3290 TabletParam.ERROR: escape_newlines(exception_description(e))
3291 }
3292 status = '200 OK'
3294 # Add session token information
3295 ts = req.tabletsession
3296 resultdict[TabletParam.SESSION_ID] = ts.session_id
3297 resultdict[TabletParam.SESSION_TOKEN] = ts.session_token
3299 # Convert dictionary to text in name-value pair format
3300 txt = "".join(f"{k}:{v}\n" for k, v in resultdict.items())
3302 t1 = time.time()
3303 log.debug("Time in script (s): {t}", t=t1 - t0)
3305 return TextResponse(txt, status=status)
3308# =============================================================================
3309# Unit tests
3310# =============================================================================
3312TEST_NHS_NUMBER = 4887211163 # generated at random
3315def get_reply_dict_from_response(response: Response) -> Dict[str, str]:
3316 """
3317 For unit testing: convert the text in a :class:`Response` back to a
3318 dictionary, so we can check it was correct.
3319 """
3320 txt = str(response)
3321 d = {} # type: Dict[str, str]
3322 # Format is: "200 OK\r\n<other headers>\r\n\r\n<content>"
3323 # There's a blank line between the heads and the body.
3324 http_gap = "\r\n\r\n"
3325 camcops_linesplit = "\n"
3326 camcops_k_v_sep = ":"
3327 try:
3328 start_of_content = txt.index(http_gap) + len(http_gap)
3329 txt = txt[start_of_content:]
3330 for line in txt.split(camcops_linesplit):
3331 if not line:
3332 continue
3333 colon_pos = line.index(camcops_k_v_sep)
3334 key = line[:colon_pos]
3335 value = line[colon_pos + len(camcops_k_v_sep):]
3336 key = key.strip()
3337 value = value.strip()
3338 d[key] = value
3339 return d
3340 except ValueError:
3341 return {}