Coverage for cc_modules/client_api.py: 18%

927 statements  

« prev     ^ index     » next       coverage.py v6.5.0, created at 2022-11-08 23:14 +0000

1#!/usr/bin/env python 

2 

3""" 

4camcops_server/cc_modules/client_api.py 

5 

6=============================================================================== 

7 

8 Copyright (C) 2012, University of Cambridge, Department of Psychiatry. 

9 Created by Rudolf Cardinal (rnc1001@cam.ac.uk). 

10 

11 This file is part of CamCOPS. 

12 

13 CamCOPS is free software: you can redistribute it and/or modify 

14 it under the terms of the GNU General Public License as published by 

15 the Free Software Foundation, either version 3 of the License, or 

16 (at your option) any later version. 

17 

18 CamCOPS is distributed in the hope that it will be useful, 

19 but WITHOUT ANY WARRANTY; without even the implied warranty of 

20 MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the 

21 GNU General Public License for more details. 

22 

23 You should have received a copy of the GNU General Public License 

24 along with CamCOPS. If not, see <https://www.gnu.org/licenses/>. 

25 

26=============================================================================== 

27 

28**Implements the API through which client devices (tablets etc.) upload and 

29download data.** 

30 

31We use primarily SQLAlchemy Core here (in contrast to the ORM used elsewhere). 

32 

33This code is optimized to a degree for speed over clarity, aiming primarily to 

34reduce the number of database hits. 

35 

36**The overall upload method is as follows** 

37 

38Everything that follows refers to records relating to a specific client device 

39in the "current" era, only. 

40 

41In the preamble, the client: 

42 

43- verifies authorization via :func:`op_check_device_registered` and 

44 :func:`op_check_upload_user_and_device`; 

45- fetches and checks server ID information via :func:`op_get_id_info`; 

46- checks its patients are acceptable via :func:`op_validate_patients`; 

47- checks which tables are permitted via :func:`op_get_allowed_tables`; 

48- performs some internal validity checks. 

49 

50Then, in the usual stepwise upload: 

51 

52- :func:`op_start_upload` 

53 

54 - Rolls back any previous incomplete changes via :func:`rollback_all`. 

55 - Creates an upload batch, via :func:`get_batch_details_start_if_needed`. 

56 

57- If were are in a preserving/finalizing upload: :func:`op_start_preservation`. 

58 

59 - Marks all tables as dirty. 

60 - Marks the upload batch as a "preserving" batch. 

61 

62- Then call some or all of: 

63 

64 - For tables that are empty on the client, :func:`op_upload_empty_tables`. 

65 

66 - Current client records are marked as ``_removal_pending``. 

67 - Any table that had previous client records is marked as dirty. 

68 - If preserving, any table without current records is marked as clean. 

69 

70 - For tables that the client wishes to send in one go, 

71 :func:`op_upload_table`. 

72 

73 - Find current server records. 

74 - Use :func:`upload_record_core` to add new records and modify existing 

75 ones, and :func:`flag_deleted` to delete ones that weren't on the client. 

76 - If any records are new, modified, or deleted, mark the table as dirty. 

77 - If preserving and there were no server records in this table, mark the 

78 table as clean. 

79 

80 - For tables (e.g. BLOBs) that might be too big to send in one go: 

81 

82 - client sends PKs to :func:`op_delete_where_key_not`, which "deletes" all 

83 other records, via :func:`flag_deleted_where_clientpk_not`. 

84 - client sends PK and timestamp values to :func:`op_which_keys_to_send` 

85 - server "deletes" records that are not in the list (via 

86 :func:`flag_deleted_where_clientpk_not`, which marks the table as dirty 

87 if any records were thus modified). Note REDUNDANCY here re 

88 :func:`op_delete_where_key_not`. 

89 - server tells the client which records are new or need to be updated 

90 - client sends each of those via :func:`op_upload_record` 

91 

92 - Calls :func`upload_record_core`. 

93 - Marks the table as dirty, unless the client erroneously sent an 

94 unchanged record. 

95 

96- In addition, specific records can be marked as ``_move_off_tablet``. 

97 

98 - :func:`upload_record_core` checks this for otherwise "identical" records 

99 and applies that flag to the server. 

100 

101- When the client's finished, it calls :func:`op_end_upload`. 

102 

103 - Calls :func:`commit_all`; 

104 - ... which, for all dirty tables, calls :func:`commit_table`; 

105 - ... which executes the "add", "remove", and "preserve" functions for the 

106 table; 

107 - ... and triggers the updating of special server indexes on patient ID 

108 numbers and tasks, via :func:`update_indexes`. 

109 - At the end, :func:`commit_all` clears the dirty-table list. 

110 

111There's a little bit of special code to handle old tablet software, too. 

112 

113As of v2.3.0, the function :func:`op_upload_entire_database` does this in one 

114step (faster; for use if the network packets are not excessively large). 

115 

116- Code relating to this uses ``batchdetails.onestep``. 

117 

118**Setup for the upload code** 

119 

120- Fire up a CamCOPS client with an empty database, e.g. from the build 

121 directory via 

122 

123 .. code-block:: bash 

124 

125 ./camcops --dbdir ~/tmp/camcops_client_test 

126 

127- Fire up a web browser showing both (a) the task list via the index, and (b) 

128 the task list without using the index. We'll use this to verify correct 

129 indexing. **The two versions of the view should never be different.** 

130 

131- Ensure the test client device has no current records (force-finalize if 

132 required). 

133 

134- Ensure the server's index is proper. Run ``camcops_server reindex`` if 

135 required. 

136 

137- If required, fire up MySQL with the server database. You may wish to use 

138 ``pager less -SFX``, for better display of large tables. 

139 

140**Testing the upload code** 

141 

142Perform the following steps both (1) with the client forced to the stepwise 

143upload method, and (2) with it forced to one-step upload. 

144 

145Note that the number of patient ID numbers uploaded (etc.) is ignored below. 

146 

147*Single record* 

148 

149[Checked for one-step and multi-step upload, 2018-11-21.] 

150 

151#. Create a blank ReferrerSatisfactionSurvey (table ``ref_satis_gen``). 

152 This has the advantage of being an anonymous single-record task. 

153 

154#. Upload/copy. 

155 

156 - The server log should show 1 × ref_satis_gen added. 

157 

158 - The task lists should show the task as current and incomplete. 

159 

160#. Modify it, so it's complete. 

161 

162#. Upload/copy. 

163 

164 - The server log should show 1 × ref_satis_gen added, 1 × ref_satis_gen 

165 modified out. 

166 

167 - The task lists should show the task as current and complete. 

168 

169#. Upload/move. 

170 

171 - The server log should show 2 × ref_satis_gen preserved. 

172 

173 - The task lists should show the task as no longer current. 

174 

175#. Create another blank one. 

176 

177#. Upload/copy. 

178 

179#. Modify it so it's complete. 

180 

181#. Specifically flag it for preservation (the chequered flags). 

182 

183#. Upload/copy. 

184 

185 - The server log should show 1 × ref_satis_gen added, 1 × ref_satis_gen 

186 modified out, 2 × ref_satis_gen preserved. 

187 

188 - The task lists should show the task as complete and no longer current. 

189 

190*With a patient* 

191 

192[Checked for one-step and multi-step upload, 2018-11-21.] 

193 

194#. Create a dummy patient that the server will accept. 

195 

196#. Create a Progress Note with location "loc1" and abort its creation, giving 

197 an incomplete task. 

198 

199#. Create a second Progress Note with location "loc2" and contents "note2". 

200 

201#. Create a third Progress Note with location "loc3" and contents "note3". 

202 

203#. Upload/copy. Verify. This checks *addition*. 

204 

205 - The server log should show 1 × patient added; 3 × progressnote added. 

206 (Also however many patientidnum records you chose.) 

207 - All three tasks should be "current". 

208 - The first should be "incomplete". 

209 

210#. Modify the first note by adding contents "note1". 

211 

212#. Delete the second note. 

213 

214#. Upload/copy. Verify. This checks *modification*, *deletion*, 

215 *no-change detection*, and *reindexing*. 

216 

217 - The server log should show 1 × progressnote added, 1 × progressnote 

218 modified out, 1 × progressnote deleted. 

219 - The first note should now appear as complete. 

220 - The second should have vanished. 

221 - The third should be unchanged. 

222 - The two remaining tasks should still be "current". 

223 

224#. Delete the contents from the first note again. 

225 

226#. Upload/move (or move-keeping-patients; that's only different on the 

227 client side). Verify. This checks *preservation (finalizing)* and 

228 *reindexing*. 

229 

230 - The server log should show 1 × patient preserved; 1 × progressnote added, 

231 1 × progressnote modified out, 5 × progressnote preserved. 

232 - The two remaining tasks should no longer be "current". 

233 - The first should no longer be "complete". 

234 

235#. Create a complete "note 4" and an incomplete "note 5". 

236 

237#. Upload/copy. 

238 

239#. Force-finalize from the server. This tests force-finalizing including 

240 reindexing. 

241 

242 - The "tasks to finalize" list should have just two tasks in it. 

243 - After force-finalizing, the tasks should remain in the index but no 

244 longer be marked as current. 

245 

246#. Upload/move to get rid of the residual tasks on the client. 

247 

248 - The server log should show 1 × patient added, 1 × patient preserved; 2 × 

249 progressnote added, 2 × progressnote preserved. 

250 

251*With ancillary tables and BLOBs* 

252 

253[Checked for one-step and multi-step upload, 2018-11-21.] 

254 

255#. Create a PhotoSequence with text "t1", one photo named "p1" of you holding 

256 up one finger vertically, and another photo named "p2" of you holding up 

257 two fingers vertically. 

258 

259#. Upload/copy. 

260 

261 - The server log should show: 

262 

263 - blobs: 2 × added; 

264 - patient: 1 × added; 

265 - photosequence: 1 × added; 

266 - photosequence_photos: 2 × added. 

267 

268 - The task lists should look sensible. 

269 

270#. Clear the second photo and replace it with a photo of you holding up 

271 two fingers horizontally. 

272 

273#. Upload/copy. 

274 

275 - The server log should show: 

276 

277 - blobs: 1 × added, 1 × modified out; 

278 - photosequence: 1 × added, 1 × modified out; 

279 - photosequence_photos: 1 × added, 1 × modified out. 

280 

281 - The task lists should look sensible. 

282 

283#. Back to two fingers vertically. (This is the fourth photo overall.) 

284 

285#. Mark that patient for specific finalization. 

286 

287#. Upload/copy. 

288 

289 - The server log should show: 

290 

291 - blobs: 1 × added, 1 × modified out, 4 × preserved; 

292 - patient: 1 × preserved; 

293 - photosequence: 1 × added, 1 × modified out, 3 × preserved; 

294 - photosequence_photos: 1 × added, 1 × modified out, 4 × preserved. 

295 

296 - The tasks should no longer be current. 

297 - A fresh "vertical fingers" photo should be visible. 

298 

299#. Create another patient and another PhotoSequence with one photo of three 

300 fingers. 

301 

302#. Upload-copy. 

303 

304#. Force-finalize. 

305 

306 - Should finalize: 1 × blobs, 1 × patient, 1 × photosequence, 1 × 

307 photosequence_photos. 

308 

309#. Upload/move. 

310 

311During any MySQL debugging, remember: 

312 

313.. code-block:: none 

314 

315 -- For better display: 

316 pager less -SFX; 

317 

318 -- To view relevant parts of the BLOB table without the actual BLOB: 

319 

320 SELECT 

321 _pk, _group_id, _device_id, _era, 

322 _current, _predecessor_pk, _successor_pk, 

323 _addition_pending, _when_added_batch_utc, _adding_user_id, 

324 _removal_pending, _when_removed_batch_utc, _removing_user_id, 

325 _move_off_tablet, 

326 _preserving_user_id, _forcibly_preserved, 

327 id, tablename, tablepk, fieldname, mimetype, when_last_modified 

328 FROM blobs; 

329 

330""" 

331 

332# ============================================================================= 

333# Imports 

334# ============================================================================= 

335 

336import logging 

337import json 

338 

339# from pprint import pformat 

340import secrets 

341import string 

342import time 

343from typing import ( 

344 Any, 

345 Dict, 

346 Iterable, 

347 List, 

348 Optional, 

349 Sequence, 

350 Set, 

351 Tuple, 

352 TYPE_CHECKING, 

353) 

354from cardinal_pythonlib.datetimefunc import ( 

355 coerce_to_pendulum, 

356 coerce_to_pendulum_date, 

357 format_datetime, 

358) 

359from cardinal_pythonlib.httpconst import HttpMethod 

360from cardinal_pythonlib.logs import BraceStyleAdapter 

361from cardinal_pythonlib.pyramid.responses import TextResponse 

362from cardinal_pythonlib.sqlalchemy.core_query import ( 

363 exists_in_table, 

364 fetch_all_first_values, 

365) 

366from cardinal_pythonlib.text import escape_newlines 

367from pyramid.httpexceptions import HTTPBadRequest 

368from pyramid.view import view_config 

369from pyramid.response import Response 

370from pyramid.security import NO_PERMISSION_REQUIRED 

371from semantic_version import Version 

372from sqlalchemy.engine.result import ResultProxy 

373from sqlalchemy.exc import IntegrityError 

374from sqlalchemy.orm import joinedload 

375from sqlalchemy.sql.expression import exists, select, update 

376from sqlalchemy.sql.schema import Table 

377 

378from camcops_server.cc_modules import cc_audit # avoids "audit" name clash 

379from camcops_server.cc_modules.cc_all_models import CLIENT_TABLE_MAP 

380from camcops_server.cc_modules.cc_blob import Blob 

381from camcops_server.cc_modules.cc_cache import cache_region_static, fkg 

382from camcops_server.cc_modules.cc_client_api_core import ( 

383 AllowedTablesFieldNames, 

384 BatchDetails, 

385 exception_description, 

386 ExtraStringFieldNames, 

387 fail_server_error, 

388 fail_unsupported_operation, 

389 fail_user_error, 

390 get_server_live_records, 

391 IgnoringAntiqueTableException, 

392 require_keys, 

393 ServerErrorException, 

394 ServerRecord, 

395 TabletParam, 

396 UploadRecordResult, 

397 UploadTableChanges, 

398 UserErrorException, 

399 values_delete_later, 

400 values_delete_now, 

401 values_preserve_now, 

402 WhichKeyToSendInfo, 

403) 

404from camcops_server.cc_modules.cc_client_api_helpers import ( 

405 upload_commit_order_sorter, 

406) 

407from camcops_server.cc_modules.cc_constants import ( 

408 CLIENT_DATE_FIELD, 

409 DateFormat, 

410 ERA_NOW, 

411 FP_ID_NUM, 

412 FP_ID_DESC, 

413 FP_ID_SHORT_DESC, 

414 MOVE_OFF_TABLET_FIELD, 

415 NUMBER_OF_IDNUMS_DEFUNCT, # allowed; for old tablet versions 

416 POSSIBLE_SEX_VALUES, 

417 TABLET_ID_FIELD, 

418) 

419from camcops_server.cc_modules.cc_convert import ( 

420 decode_single_value, 

421 decode_values, 

422 encode_single_value, 

423) 

424from camcops_server.cc_modules.cc_db import ( 

425 FN_ADDING_USER_ID, 

426 FN_ADDITION_PENDING, 

427 FN_CAMCOPS_VERSION, 

428 FN_CURRENT, 

429 FN_DEVICE_ID, 

430 FN_ERA, 

431 FN_GROUP_ID, 

432 FN_PK, 

433 FN_PREDECESSOR_PK, 

434 FN_REMOVAL_PENDING, 

435 FN_REMOVING_USER_ID, 

436 FN_SUCCESSOR_PK, 

437 FN_WHEN_ADDED_BATCH_UTC, 

438 FN_WHEN_ADDED_EXACT, 

439 FN_WHEN_REMOVED_BATCH_UTC, 

440 FN_WHEN_REMOVED_EXACT, 

441 RESERVED_FIELDS, 

442) 

443from camcops_server.cc_modules.cc_device import Device 

444from camcops_server.cc_modules.cc_dirtytables import DirtyTable 

445from camcops_server.cc_modules.cc_group import Group 

446from camcops_server.cc_modules.cc_ipuse import IpUse 

447from camcops_server.cc_modules.cc_membership import UserGroupMembership 

448from camcops_server.cc_modules.cc_patient import ( 

449 Patient, 

450 is_candidate_patient_valid_for_group, 

451 is_candidate_patient_valid_for_restricted_user, 

452) 

453from camcops_server.cc_modules.cc_patientidnum import ( 

454 fake_tablet_id_for_patientidnum, 

455 PatientIdNum, 

456) 

457from camcops_server.cc_modules.cc_proquint import ( 

458 InvalidProquintException, 

459 uuid_from_proquint, 

460) 

461from camcops_server.cc_modules.cc_pyramid import Routes 

462from camcops_server.cc_modules.cc_simpleobjects import ( 

463 BarePatientInfo, 

464 IdNumReference, 

465) 

466from camcops_server.cc_modules.cc_specialnote import SpecialNote 

467from camcops_server.cc_modules.cc_task import ( 

468 all_task_tables_with_min_client_version, 

469) 

470from camcops_server.cc_modules.cc_taskindex import ( 

471 update_indexes_and_push_exports, 

472) 

473from camcops_server.cc_modules.cc_user import User 

474from camcops_server.cc_modules.cc_validators import ( 

475 STRING_VALIDATOR_TYPE, 

476 validate_anything, 

477 validate_email, 

478) 

479from camcops_server.cc_modules.cc_version import ( 

480 CAMCOPS_SERVER_VERSION_STRING, 

481 MINIMUM_TABLET_VERSION, 

482) 

483 

484if TYPE_CHECKING: 

485 from camcops_server.cc_modules.cc_request import CamcopsRequest 

486 

487log = BraceStyleAdapter(logging.getLogger(__name__)) 

488 

489 

490# ============================================================================= 

491# Constants 

492# ============================================================================= 

493 

494COPE_WITH_DELETED_PATIENT_DESCRIPTIONS = True 

495# ... as of client 2.0.0, ID descriptions are no longer duplicated. 

496# As of server 2.0.0, the fields still exist in the database, but the reporting 

497# and consistency check has been removed. In the next version of the server, 

498# the fields will be removed, and then the server should cope with old clients, 

499# at least for a while. 

500 

501DUPLICATE_FAILED = "Failed to duplicate record" 

502INSERT_FAILED = "Failed to insert record" 

503 

504# REGEX_INVALID_TABLE_FIELD_CHARS = re.compile("[^a-zA-Z0-9_]") 

505# ... the ^ within the [] means the expression will match any character NOT in 

506# the specified range 

507 

508DEVICE_STORED_VAR_TABLENAME_DEFUNCT = "storedvars" 

509# ... old table, no longer in use, that Titanium clients used to upload. 

510# We recognize and ignore it now so that old clients can still work. 

511 

512SILENTLY_IGNORE_TABLENAMES = [DEVICE_STORED_VAR_TABLENAME_DEFUNCT] 

513 

514IGNORING_ANTIQUE_TABLE_MESSAGE = ( 

515 "Ignoring user request to upload antique/defunct table, but reporting " 

516 "success to the client" 

517) 

518 

519SUCCESS_MSG = "Success" 

520SUCCESS_CODE = "1" 

521FAILURE_CODE = "0" 

522 

523DEBUG_UPLOAD = False 

524 

525 

526# ============================================================================= 

527# Quasi-constants 

528# ============================================================================= 

529 

530DB_JSON_DECODER = json.JSONDecoder() # just a plain one 

531PATIENT_INFO_JSON_DECODER = json.JSONDecoder() # just a plain one 

532 

533 

534# ============================================================================= 

535# Cached information 

536# ============================================================================= 

537 

538 

539@cache_region_static.cache_on_arguments(function_key_generator=fkg) 

540def all_tables_with_min_client_version() -> Dict[str, Version]: 

541 """ 

542 For all tables that the client might upload to, return a mapping from the 

543 table name to the corresponding minimum client version. 

544 """ 

545 d = all_task_tables_with_min_client_version() 

546 d[Blob.__tablename__] = MINIMUM_TABLET_VERSION 

547 d[Patient.__tablename__] = MINIMUM_TABLET_VERSION 

548 d[PatientIdNum.__tablename__] = MINIMUM_TABLET_VERSION 

549 return d 

550 

551 

552# ============================================================================= 

553# Validators 

554# ============================================================================= 

555 

556 

557def ensure_valid_table_name(req: "CamcopsRequest", tablename: str) -> None: 

558 """ 

559 Ensures a table name: 

560 

561 - doesn't contain bad characters, 

562 - isn't a reserved table that the user is prohibited from accessing, and 

563 - is a valid table name that's in the database. 

564 

565 Raises :exc:`UserErrorException` upon failure. 

566 

567 - 2017-10-08: shortcut to all that: it's OK if it's listed as a valid 

568 client table. 

569 - 2018-01-16 (v2.2.0): check also that client version is OK 

570 """ 

571 if tablename not in CLIENT_TABLE_MAP: 

572 fail_user_error(f"Invalid client table name: {tablename}") 

573 tables_versions = all_tables_with_min_client_version() 

574 assert tablename in tables_versions 

575 client_version = req.tabletsession.tablet_version_ver 

576 minimum_client_version = tables_versions[tablename] 

577 if client_version < minimum_client_version: 

578 fail_user_error( 

579 f"Client CamCOPS version {client_version} is less than the " 

580 f"version ({minimum_client_version}) " 

581 f"required to handle table {tablename}" 

582 ) 

583 

584 

585def ensure_valid_field_name(table: Table, fieldname: str) -> None: 

586 """ 

587 Ensures a field name contains only valid characters, and isn't a 

588 reserved fieldname that the user isn't allowed to access. 

589 

590 Raises :exc:`UserErrorException` upon failure. 

591 

592 - 2017-10-08: shortcut: it's OK if it's a column name for a particular 

593 table. 

594 """ 

595 if fieldname.startswith("_"): # all reserved fields start with _ 

596 # ... but not all fields starting with "_" are reserved; e.g. 

597 # "_move_off_tablet" is allowed. 

598 if fieldname in RESERVED_FIELDS: 

599 fail_user_error( 

600 f"Reserved field name for table {table.name!r}: {fieldname!r}" 

601 ) 

602 if fieldname not in table.columns.keys(): 

603 fail_user_error( 

604 f"Invalid field name for table {table.name!r}: {fieldname!r}" 

605 ) 

606 # Note that the reserved-field check is case-sensitive, but so is the 

607 # "present in table" check. So for a malicious uploader trying to use, for 

608 # example, "_PK", this would not be picked up as a reserved field (so would 

609 # pass that check) but then wouldn't be recognized as a valid field (so 

610 # would fail). 

611 

612 

613def ensure_string(value: Any, allow_none: bool = True) -> None: 

614 """ 

615 Used when processing JSON information about patients: ensures that a value 

616 is a string, or raises. 

617 

618 Args: 

619 value: value to test 

620 allow_none: is ``None`` allowed (not just an empty string)? 

621 """ 

622 if value is None: 

623 if allow_none: 

624 return # OK 

625 else: 

626 fail_user_error("Patient JSON contains absent string") 

627 if not isinstance(value, str): 

628 fail_user_error(f"Patient JSON contains invalid non-string: {value!r}") 

629 

630 

631def ensure_valid_patient_json( 

632 req: "CamcopsRequest", group: Group, pt_dict: Dict[str, Any] 

633) -> None: 

634 """ 

635 Ensures that the JSON dictionary contains valid patient details (valid for 

636 the group into which it's being uploaded), and that (if applicable) this 

637 user is allowed to upload this patient. 

638 

639 Args: 

640 req: 

641 the :class:`camcops_server.cc_modules.cc_request.CamcopsRequest` 

642 group: 

643 the :class:`camcops_server.cc_modules.cc_group.Group` into which 

644 the upload is going 

645 pt_dict: 

646 a JSON dictionary from the client 

647 

648 Raises: 

649 :exc:`UserErrorException` if invalid 

650 

651 """ 

652 if not isinstance(pt_dict, dict): 

653 fail_user_error("Patient JSON is not a dict") 

654 if not pt_dict: 

655 fail_user_error("Patient JSON is empty") 

656 valid_which_idnums = req.valid_which_idnums 

657 errors = [] # type: List[str] 

658 finalizing = None 

659 ptinfo = BarePatientInfo() 

660 idnum_types_seen = set() # type: Set[int] 

661 for k, v in pt_dict.items(): 

662 ensure_string(k, allow_none=False) 

663 

664 if k == TabletParam.FORENAME: 

665 ensure_string(v) 

666 ptinfo.forename = v 

667 

668 elif k == TabletParam.SURNAME: 

669 ensure_string(v) 

670 ptinfo.surname = v 

671 

672 elif k == TabletParam.SEX: 

673 if v not in POSSIBLE_SEX_VALUES: 

674 fail_user_error(f"Bad sex value: {v!r}") 

675 ptinfo.sex = v 

676 

677 elif k == TabletParam.DOB: 

678 ensure_string(v) 

679 if v: 

680 dob = coerce_to_pendulum_date(v) 

681 if dob is None: 

682 fail_user_error(f"Invalid DOB: {v!r}") 

683 else: 

684 dob = None 

685 ptinfo.dob = dob 

686 

687 elif k == TabletParam.EMAIL: 

688 ensure_string(v) 

689 if v: 

690 try: 

691 validate_email(v) 

692 except ValueError: 

693 fail_user_error(f"Bad e-mail address: {v!r}") 

694 ptinfo.email = v 

695 

696 elif k == TabletParam.ADDRESS: 

697 ensure_string(v) 

698 ptinfo.address = v 

699 

700 elif k == TabletParam.GP: 

701 ensure_string(v) 

702 ptinfo.gp = v 

703 

704 elif k == TabletParam.OTHER: 

705 ensure_string(v) 

706 ptinfo.otherdetails = v 

707 

708 elif k.startswith(TabletParam.IDNUM_PREFIX): 

709 nstr = k[len(TabletParam.IDNUM_PREFIX) :] # noqa: E203 

710 try: 

711 which_idnum = int(nstr) 

712 except (TypeError, ValueError): 

713 fail_user_error(f"Bad idnum key: {k!r}") 

714 # noinspection PyUnboundLocalVariable 

715 if which_idnum not in valid_which_idnums: 

716 fail_user_error(f"Bad ID number type: {which_idnum}") 

717 if which_idnum in idnum_types_seen: 

718 fail_user_error( 

719 f"More than one ID number supplied for ID " 

720 f"number type {which_idnum}" 

721 ) 

722 idnum_types_seen.add(which_idnum) 

723 if v is not None and not isinstance(v, int): 

724 fail_user_error(f"Bad ID number value: {v!r}") 

725 idref = IdNumReference(which_idnum, v) 

726 if not idref.is_valid(): 

727 fail_user_error(f"Bad ID number: {idref!r}") 

728 ptinfo.add_idnum(idref) 

729 

730 elif k == TabletParam.FINALIZING: 

731 if not isinstance(v, bool): 

732 fail_user_error(f"Bad {k!r} value: {v!r}") 

733 finalizing = v 

734 

735 else: 

736 fail_user_error(f"Unknown JSON key: {k!r}") 

737 

738 if finalizing is None: 

739 fail_user_error(f"Missing {TabletParam.FINALIZING!r} JSON key") 

740 

741 pt_ok, reason = is_candidate_patient_valid_for_group( 

742 ptinfo, group, finalizing 

743 ) 

744 if not pt_ok: 

745 errors.append(f"{ptinfo} -> {reason}") 

746 pt_ok, reason = is_candidate_patient_valid_for_restricted_user(req, ptinfo) 

747 if not pt_ok: 

748 errors.append(f"{ptinfo} -> {reason}") 

749 if errors: 

750 fail_user_error(f"Invalid patient: {' // '.join(errors)}") 

751 

752 

753# ============================================================================= 

754# Extracting information from the POST request 

755# ============================================================================= 

756 

757 

758def get_str_var( 

759 req: "CamcopsRequest", 

760 var: str, 

761 mandatory: bool = True, 

762 validator: STRING_VALIDATOR_TYPE = validate_anything, 

763) -> Optional[str]: 

764 """ 

765 Retrieves a string variable from the CamcopsRequest. 

766 

767 By default this performs no validation (because, for example, these strings 

768 can contain SQL-encoded data or JSON), but there are a number of subsequent 

769 operation-specific validation steps. 

770 

771 Args: 

772 req: the :class:`camcops_server.cc_modules.cc_request.CamcopsRequest` 

773 var: name of variable to retrieve 

774 mandatory: if ``True``, raise an exception if the variable is missing 

775 validator: validator function to use 

776 

777 Returns: 

778 value 

779 

780 Raises: 

781 :exc:`UserErrorException` if the variable was mandatory and 

782 no value was provided 

783 """ 

784 try: 

785 val = req.get_str_param(var, default=None, validator=validator) 

786 if mandatory and val is None: 

787 fail_user_error(f"Must provide the variable: {var}") 

788 return val 

789 except HTTPBadRequest as e: # failed the validator 

790 fail_user_error(str(e)) 

791 

792 

793def get_int_var(req: "CamcopsRequest", var: str) -> int: 

794 """ 

795 Retrieves an integer variable from the CamcopsRequest. 

796 

797 Args: 

798 req: the :class:`camcops_server.cc_modules.cc_request.CamcopsRequest` 

799 var: name of variable to retrieve 

800 

801 Returns: 

802 value 

803 

804 Raises: 

805 :exc:`UserErrorException` if no value was provided, or if it wasn't an 

806 integer 

807 """ 

808 s = get_str_var(req, var, mandatory=True) 

809 try: 

810 return int(s) 

811 except (TypeError, ValueError): 

812 fail_user_error(f"Variable {var} is not a valid integer; was {s!r}") 

813 

814 

815def get_bool_int_var(req: "CamcopsRequest", var: str) -> bool: 

816 """ 

817 Retrieves a Boolean variable (encoded as an integer) from the 

818 CamcopsRequest. Zero represents false; nonzero represents true. 

819 

820 Args: 

821 req: the :class:`camcops_server.cc_modules.cc_request.CamcopsRequest` 

822 var: name of variable to retrieve 

823 

824 Returns: 

825 value 

826 

827 Raises: 

828 :exc:`UserErrorException` if no value was provided, or if it wasn't an 

829 integer 

830 """ 

831 num = get_int_var(req, var) 

832 return bool(num) 

833 

834 

835def get_table_from_req(req: "CamcopsRequest", var: str) -> Table: 

836 """ 

837 Retrieves a table name from a HTTP request, checks it's a valid client 

838 table, and returns that table. 

839 

840 Args: 

841 req: the :class:`camcops_server.cc_modules.cc_request.CamcopsRequest` 

842 var: variable name (the variable's should be the table name) 

843 

844 Returns: 

845 a SQLAlchemy :class:`Table` 

846 

847 Raises: 

848 :exc:`UserErrorException` if the variable wasn't provided 

849 

850 :exc:`IgnoringAntiqueTableException` if the table is one to 

851 ignore quietly (requested by an antique client) 

852 """ 

853 tablename = get_str_var(req, var, mandatory=True) 

854 if tablename in SILENTLY_IGNORE_TABLENAMES: 

855 raise IgnoringAntiqueTableException(f"Ignoring table {tablename}") 

856 ensure_valid_table_name(req, tablename) 

857 return CLIENT_TABLE_MAP[tablename] 

858 

859 

860def get_tables_from_post_var( 

861 req: "CamcopsRequest", var: str, mandatory: bool = True 

862) -> List[Table]: 

863 """ 

864 Gets a list of tables from an HTTP request variable, and ensures all are 

865 valid. 

866 

867 Args: 

868 req: the :class:`camcops_server.cc_modules.cc_request.CamcopsRequest` 

869 var: name of variable to retrieve 

870 mandatory: if ``True``, raise an exception if the variable is missing 

871 

872 Returns: 

873 a list of SQLAlchemy :class:`Table` objects 

874 

875 Raises: 

876 :exc:`UserErrorException` if the variable was mandatory and 

877 no value was provided, or if one or more tables was not valid 

878 """ 

879 cstables = get_str_var(req, var, mandatory=mandatory) 

880 if not cstables: 

881 return [] 

882 # can't have any commas in table names, so it's OK to use a simple 

883 # split() command 

884 tablenames = [x.strip() for x in cstables.split(",")] 

885 tables = [] # type: List[Table] 

886 for tn in tablenames: 

887 if tn in SILENTLY_IGNORE_TABLENAMES: 

888 log.warning(IGNORING_ANTIQUE_TABLE_MESSAGE) 

889 continue 

890 ensure_valid_table_name(req, tn) 

891 tables.append(CLIENT_TABLE_MAP[tn]) 

892 return tables 

893 

894 

895def get_single_field_from_post_var( 

896 req: "CamcopsRequest", table: Table, var: str, mandatory: bool = True 

897) -> str: 

898 """ 

899 Retrieves a field (column) name from a the request and checks it's not a 

900 bad fieldname for the specified table. 

901 

902 Args: 

903 req: the :class:`camcops_server.cc_modules.cc_request.CamcopsRequest` 

904 table: SQLAlchemy :class:`Table` in which the column should exist 

905 var: name of variable to retrieve 

906 mandatory: if ``True``, raise an exception if the variable is missing 

907 

908 Returns: 

909 the field (column) name 

910 

911 Raises: 

912 :exc:`UserErrorException` if the variable was mandatory and 

913 no value was provided, or if the field was not valid for the specified 

914 table 

915 """ 

916 field = get_str_var(req, var, mandatory=mandatory) 

917 ensure_valid_field_name(table, field) 

918 return field 

919 

920 

921def get_fields_from_post_var( 

922 req: "CamcopsRequest", 

923 table: Table, 

924 var: str, 

925 mandatory: bool = True, 

926 allowed_nonexistent_fields: List[str] = None, 

927) -> List[str]: 

928 """ 

929 Get a comma-separated list of field names from a request and checks that 

930 all are acceptable. Returns a list of fieldnames. 

931 

932 Args: 

933 req: the :class:`camcops_server.cc_modules.cc_request.CamcopsRequest` 

934 table: SQLAlchemy :class:`Table` in which the columns should exist 

935 var: name of variable to retrieve 

936 mandatory: if ``True``, raise an exception if the variable is missing 

937 allowed_nonexistent_fields: fields that are allowed to be in the 

938 upload but not in the database (special exemptions!) 

939 

940 Returns: 

941 a list of the field (column) names 

942 

943 Raises: 

944 :exc:`UserErrorException` if the variable was mandatory and 

945 no value was provided, or if any field was not valid for the specified 

946 table 

947 """ 

948 csfields = get_str_var(req, var, mandatory=mandatory) 

949 if not csfields: 

950 return [] 

951 allowed_nonexistent_fields = ( 

952 allowed_nonexistent_fields or [] 

953 ) # type: List[str] # noqa 

954 # can't have any commas in fields, so it's OK to use a simple 

955 # split() command 

956 fields = [x.strip() for x in csfields.split(",")] 

957 for f in fields: 

958 if f in allowed_nonexistent_fields: 

959 continue 

960 ensure_valid_field_name(table, f) 

961 return fields 

962 

963 

964def get_values_from_post_var( 

965 req: "CamcopsRequest", var: str, mandatory: bool = True 

966) -> List[Any]: 

967 """ 

968 Retrieves a list of values from a CSV-separated list of SQL values 

969 stored in a CGI form (including e.g. NULL, numbers, quoted strings, and 

970 special handling for base-64/hex-encoded BLOBs.) 

971 

972 Args: 

973 req: the :class:`camcops_server.cc_modules.cc_request.CamcopsRequest` 

974 var: name of variable to retrieve 

975 mandatory: if ``True``, raise an exception if the variable is missing 

976 """ 

977 csvalues = get_str_var(req, var, mandatory=mandatory) 

978 if not csvalues: 

979 return [] 

980 return decode_values(csvalues) 

981 

982 

983def get_fields_and_values( 

984 req: "CamcopsRequest", 

985 table: Table, 

986 fields_var: str, 

987 values_var: str, 

988 mandatory: bool = True, 

989) -> Dict[str, Any]: 

990 """ 

991 Gets fieldnames and matching values from two variables in a request. 

992 

993 See :func:`get_fields_from_post_var`, :func:`get_values_from_post_var`. 

994 

995 Args: 

996 req: the :class:`camcops_server.cc_modules.cc_request.CamcopsRequest` 

997 table: SQLAlchemy :class:`Table` in which the columns should exist 

998 fields_var: name of CSV "column names" variable to retrieve 

999 values_var: name of CSV "corresponding values" variable to retrieve 

1000 mandatory: if ``True``, raise an exception if the variable is missing 

1001 

1002 Returns: 

1003 a dictionary mapping column names to decoded values 

1004 

1005 Raises: 

1006 :exc:`UserErrorException` if the variable was mandatory and 

1007 no value was provided, or if any field was not valid for the specified 

1008 table 

1009 """ 

1010 fields = get_fields_from_post_var( 

1011 req, table, fields_var, mandatory=mandatory 

1012 ) 

1013 values = get_values_from_post_var(req, values_var, mandatory=mandatory) 

1014 if len(fields) != len(values): 

1015 fail_user_error( 

1016 f"Number of fields ({len(fields)}) doesn't match number of values " 

1017 f"({len(values)})" 

1018 ) 

1019 return dict(list(zip(fields, values))) 

1020 

1021 

1022def get_json_from_post_var( 

1023 req: "CamcopsRequest", 

1024 key: str, 

1025 decoder: json.JSONDecoder = None, 

1026 mandatory: bool = True, 

1027) -> Any: 

1028 """ 

1029 Returns a Python object from a JSON-encoded value. 

1030 

1031 Args: 

1032 req: the :class:`camcops_server.cc_modules.cc_request.CamcopsRequest` 

1033 key: the name of the variable to retrieve 

1034 decoder: the JSON decoder object to use; if ``None``, a default is 

1035 created 

1036 mandatory: if ``True``, raise an exception if the variable is missing 

1037 

1038 Returns: 

1039 Python object, e.g. a list of values, or ``None`` if the object is 

1040 invalid and not mandatory 

1041 

1042 Raises: 

1043 :exc:`UserErrorException` if the variable was mandatory and 

1044 no value was provided or the value was invalid JSON 

1045 """ 

1046 decoder = decoder or json.JSONDecoder() 

1047 j = get_str_var(req, key, mandatory=mandatory) # may raise 

1048 if not j: # missing but not mandatory 

1049 return None 

1050 try: 

1051 return decoder.decode(j) 

1052 except json.JSONDecodeError: 

1053 msg = f"Bad JSON for key {key!r}" 

1054 if mandatory: 

1055 fail_user_error(msg) 

1056 else: 

1057 log.warning(msg) 

1058 return None 

1059 

1060 

1061# ============================================================================= 

1062# Sending stuff to the client 

1063# ============================================================================= 

1064 

1065 

1066def get_server_id_info(req: "CamcopsRequest") -> Dict[str, str]: 

1067 """ 

1068 Returns a reply for the tablet, as a variable-to-value dictionary, giving 

1069 details of the server. 

1070 """ 

1071 group = Group.get_group_by_id(req.dbsession, req.user.upload_group_id) 

1072 reply = { 

1073 TabletParam.DATABASE_TITLE: req.database_title, 

1074 TabletParam.ID_POLICY_UPLOAD: group.upload_policy or "", 

1075 TabletParam.ID_POLICY_FINALIZE: group.finalize_policy or "", 

1076 TabletParam.SERVER_CAMCOPS_VERSION: CAMCOPS_SERVER_VERSION_STRING, 

1077 } 

1078 for iddef in req.idnum_definitions: 

1079 n = iddef.which_idnum 

1080 nstr = str(n) 

1081 reply[TabletParam.ID_DESCRIPTION_PREFIX + nstr] = ( 

1082 iddef.description or "" 

1083 ) 

1084 reply[TabletParam.ID_SHORT_DESCRIPTION_PREFIX + nstr] = ( 

1085 iddef.short_description or "" 

1086 ) 

1087 reply[TabletParam.ID_VALIDATION_METHOD_PREFIX + nstr] = ( 

1088 iddef.validation_method or "" 

1089 ) 

1090 return reply 

1091 

1092 

1093def get_select_reply( 

1094 fields: Sequence[str], rows: Sequence[Sequence[Any]] 

1095) -> Dict[str, str]: 

1096 """ 

1097 Formats the result of a ``SELECT`` query for the client as a dictionary 

1098 reply. 

1099 

1100 Args: 

1101 fields: list of field names 

1102 rows: list of rows, where each row is a list of values in the same 

1103 order as ``fields`` 

1104 

1105 Returns: 

1106 

1107 a dictionary of the format: 

1108 

1109 .. code-block:: none 

1110 

1111 { 

1112 "nfields": NUMBER_OF_FIELDS, 

1113 "fields": FIELDNAMES_AS_CSV, 

1114 "nrecords": NUMBER_OF_RECORDS, 

1115 "record0": VALUES_AS_CSV_LIST_OF_ENCODED_SQL_VALUES, 

1116 ... 

1117 "record{nrecords - 1}": VALUES_AS_CSV_LIST_OF_ENCODED_SQL_VALUES 

1118 } 

1119 

1120 The final reply to the server is then formatted as text as per 

1121 :func:`client_api`. 

1122 

1123 """ # noqa 

1124 nrecords = len(rows) 

1125 reply = { 

1126 TabletParam.NFIELDS: len(fields), 

1127 TabletParam.FIELDS: ",".join(fields), 

1128 TabletParam.NRECORDS: nrecords, 

1129 } 

1130 for r in range(nrecords): 

1131 row = rows[r] 

1132 encodedvalues = [] # type: List[str] 

1133 for val in row: 

1134 encodedvalues.append(encode_single_value(val)) 

1135 reply[TabletParam.RECORD_PREFIX + str(r)] = ",".join(encodedvalues) 

1136 return reply 

1137 

1138 

1139# ============================================================================= 

1140# CamCOPS table reading functions 

1141# ============================================================================= 

1142 

1143 

1144def record_exists( 

1145 req: "CamcopsRequest", 

1146 table: Table, 

1147 clientpk_name: str, 

1148 clientpk_value: Any, 

1149) -> ServerRecord: 

1150 """ 

1151 Checks if a record exists, using the device's perspective of a 

1152 table/client PK combination. 

1153 

1154 Args: 

1155 req: the :class:`camcops_server.cc_modules.cc_request.CamcopsRequest` 

1156 table: an SQLAlchemy :class:`Table` 

1157 clientpk_name: the column name of the client's PK 

1158 clientpk_value: the client's PK value 

1159 

1160 Returns: 

1161 a :class:`ServerRecord` with the required information 

1162 

1163 """ 

1164 query = ( 

1165 select( 

1166 [ 

1167 table.c[FN_PK], # server PK 

1168 table.c[ 

1169 CLIENT_DATE_FIELD 

1170 ], # when last modified (on the server) 

1171 table.c[MOVE_OFF_TABLET_FIELD], # move_off_tablet 

1172 ] 

1173 ) 

1174 .where(table.c[FN_DEVICE_ID] == req.tabletsession.device_id) 

1175 .where(table.c[FN_CURRENT]) 

1176 .where(table.c[FN_ERA] == ERA_NOW) 

1177 .where(table.c[clientpk_name] == clientpk_value) 

1178 ) 

1179 row = req.dbsession.execute(query).fetchone() 

1180 if not row: 

1181 return ServerRecord(clientpk_value, False) 

1182 server_pk, server_when, move_off_tablet = row 

1183 return ServerRecord( 

1184 clientpk_value, True, server_pk, server_when, move_off_tablet 

1185 ) 

1186 # Consider a warning/failure if we have >1 row meeting these criteria. 

1187 # Not currently checked for. 

1188 

1189 

1190def client_pks_that_exist( 

1191 req: "CamcopsRequest", 

1192 table: Table, 

1193 clientpk_name: str, 

1194 clientpk_values: List[int], 

1195) -> Dict[int, ServerRecord]: 

1196 """ 

1197 Searches for client PK values (for this device, current, and 'now') 

1198 matching the input list. 

1199 

1200 Args: 

1201 req: the :class:`camcops_server.cc_modules.cc_request.CamcopsRequest` 

1202 table: an SQLAlchemy :class:`Table` 

1203 clientpk_name: the column name of the client's PK 

1204 clientpk_values: a list of the client's PK values 

1205 

1206 Returns: 

1207 a dictionary mapping client_pk to a :class:`ServerRecord` objects, for 

1208 those records that match 

1209 """ 

1210 query = ( 

1211 select( 

1212 [ 

1213 table.c[FN_PK], # server PK 

1214 table.c[clientpk_name], # client PK 

1215 table.c[ 

1216 CLIENT_DATE_FIELD 

1217 ], # when last modified (on the server) 

1218 table.c[MOVE_OFF_TABLET_FIELD], # move_off_tablet 

1219 ] 

1220 ) 

1221 .where(table.c[FN_DEVICE_ID] == req.tabletsession.device_id) 

1222 .where(table.c[FN_CURRENT]) 

1223 .where(table.c[FN_ERA] == ERA_NOW) 

1224 .where(table.c[clientpk_name].in_(clientpk_values)) 

1225 ) 

1226 rows = req.dbsession.execute(query) 

1227 d = {} # type: Dict[int, ServerRecord] 

1228 for server_pk, client_pk, server_when, move_off_tablet in rows: 

1229 d[client_pk] = ServerRecord( 

1230 client_pk, True, server_pk, server_when, move_off_tablet 

1231 ) 

1232 return d 

1233 

1234 

1235def get_all_predecessor_pks( 

1236 req: "CamcopsRequest", 

1237 table: Table, 

1238 last_pk: int, 

1239 include_last: bool = True, 

1240) -> List[int]: 

1241 """ 

1242 Retrieves the PKs of all records that are predecessors of the specified one 

1243 

1244 Args: 

1245 req: the :class:`camcops_server.cc_modules.cc_request.CamcopsRequest` 

1246 table: an SQLAlchemy :class:`Table` 

1247 last_pk: the PK to start with, and work backwards 

1248 include_last: include ``last_pk`` in the list 

1249 

1250 Returns: 

1251 the PKs 

1252 

1253 """ 

1254 dbsession = req.dbsession 

1255 pks = [] # type: List[int] 

1256 if include_last: 

1257 pks.append(last_pk) 

1258 current_pk = last_pk 

1259 finished = False 

1260 while not finished: 

1261 next_pk = dbsession.execute( 

1262 select([table.c[FN_PREDECESSOR_PK]]).where( 

1263 table.c[FN_PK] == current_pk 

1264 ) 

1265 ).scalar() # type: Optional[int] 

1266 if next_pk is None: 

1267 finished = True 

1268 else: 

1269 pks.append(next_pk) 

1270 current_pk = next_pk 

1271 return sorted(pks) 

1272 

1273 

1274# ============================================================================= 

1275# Record modification functions 

1276# ============================================================================= 

1277 

1278 

1279def flag_deleted( 

1280 req: "CamcopsRequest", 

1281 batchdetails: BatchDetails, 

1282 table: Table, 

1283 pklist: Iterable[int], 

1284) -> None: 

1285 """ 

1286 Marks record(s) as deleted, specified by a list of server PKs within a 

1287 table. (Note: "deleted" means "deleted with no successor", not "modified 

1288 and replaced by a successor record".) 

1289 """ 

1290 if batchdetails.onestep: 

1291 values = values_delete_now(req, batchdetails) 

1292 else: 

1293 values = values_delete_later() 

1294 req.dbsession.execute( 

1295 update(table).where(table.c[FN_PK].in_(pklist)).values(values) 

1296 ) 

1297 

1298 

1299def flag_all_records_deleted(req: "CamcopsRequest", table: Table) -> int: 

1300 """ 

1301 Marks all records in a table as deleted (that are current and in the 

1302 current era). 

1303 

1304 Returns the number of rows affected. 

1305 """ 

1306 rp = req.dbsession.execute( 

1307 update(table) 

1308 .where(table.c[FN_DEVICE_ID] == req.tabletsession.device_id) 

1309 .where(table.c[FN_CURRENT]) 

1310 .where(table.c[FN_ERA] == ERA_NOW) 

1311 .values(values_delete_later()) 

1312 ) # type: ResultProxy 

1313 return rp.rowcount 

1314 # https://docs.sqlalchemy.org/en/latest/core/connections.html?highlight=rowcount#sqlalchemy.engine.ResultProxy.rowcount # noqa 

1315 

1316 

1317def flag_deleted_where_clientpk_not( 

1318 req: "CamcopsRequest", 

1319 table: Table, 

1320 clientpk_name: str, 

1321 clientpk_values: Sequence[Any], 

1322) -> None: 

1323 """ 

1324 Marks for deletion all current/current-era records for a device, within a 

1325 specific table, defined by a list of client-side PK values (and the name of 

1326 the client-side PK column). 

1327 """ 

1328 rp = req.dbsession.execute( 

1329 update(table) 

1330 .where(table.c[FN_DEVICE_ID] == req.tabletsession.device_id) 

1331 .where(table.c[FN_CURRENT]) 

1332 .where(table.c[FN_ERA] == ERA_NOW) 

1333 .where(table.c[clientpk_name].notin_(clientpk_values)) 

1334 .values(values_delete_later()) 

1335 ) # type: ResultProxy 

1336 if rp.rowcount > 0: 

1337 mark_table_dirty(req, table) 

1338 # ... but if we are preserving, do NOT mark this table as clean; there may 

1339 # be other records that still require preserving. 

1340 

1341 

1342def flag_modified( 

1343 req: "CamcopsRequest", 

1344 batchdetails: BatchDetails, 

1345 table: Table, 

1346 pk: int, 

1347 successor_pk: int, 

1348) -> None: 

1349 """ 

1350 Marks a record as old, storing its successor's details. 

1351 

1352 Args: 

1353 req: the :class:`camcops_server.cc_modules.cc_request.CamcopsRequest` 

1354 batchdetails: the :class:`BatchDetails` 

1355 table: SQLAlchemy :class:`Table` 

1356 pk: server PK of the record to mark as old 

1357 successor_pk: server PK of its successor 

1358 """ 

1359 if batchdetails.onestep: 

1360 req.dbsession.execute( 

1361 update(table) 

1362 .where(table.c[FN_PK] == pk) 

1363 .values( 

1364 { 

1365 FN_CURRENT: 0, 

1366 FN_REMOVAL_PENDING: 0, 

1367 FN_SUCCESSOR_PK: successor_pk, 

1368 FN_REMOVING_USER_ID: req.user_id, 

1369 FN_WHEN_REMOVED_EXACT: req.now, 

1370 FN_WHEN_REMOVED_BATCH_UTC: batchdetails.batchtime, 

1371 } 

1372 ) 

1373 ) 

1374 else: 

1375 req.dbsession.execute( 

1376 update(table) 

1377 .where(table.c[FN_PK] == pk) 

1378 .values({FN_REMOVAL_PENDING: 1, FN_SUCCESSOR_PK: successor_pk}) 

1379 ) 

1380 

1381 

1382def flag_multiple_records_for_preservation( 

1383 req: "CamcopsRequest", 

1384 batchdetails: BatchDetails, 

1385 table: Table, 

1386 pks_to_preserve: List[int], 

1387) -> None: 

1388 """ 

1389 Low-level function to mark records for preservation by server PK. 

1390 Does not concern itself with the predecessor chain (for which, see 

1391 :func:`flag_record_for_preservation`). 

1392 

1393 Args: 

1394 req: the :class:`camcops_server.cc_modules.cc_request.CamcopsRequest` 

1395 batchdetails: the :class:`BatchDetails` 

1396 table: SQLAlchemy :class:`Table` 

1397 pks_to_preserve: server PK of the records to mark as preserved 

1398 """ 

1399 if batchdetails.onestep: 

1400 req.dbsession.execute( 

1401 update(table) 

1402 .where(table.c[FN_PK].in_(pks_to_preserve)) 

1403 .values(values_preserve_now(req, batchdetails)) 

1404 ) 

1405 # Also any associated special notes: 

1406 new_era = batchdetails.new_era 

1407 # noinspection PyUnresolvedReferences 

1408 req.dbsession.execute( 

1409 update(SpecialNote.__table__) 

1410 .where(SpecialNote.basetable == table.name) 

1411 .where(SpecialNote.device_id == req.tabletsession.device_id) 

1412 .where(SpecialNote.era == ERA_NOW) 

1413 .where( 

1414 exists() 

1415 .select_from(table) 

1416 .where(table.c[TABLET_ID_FIELD] == SpecialNote.task_id) 

1417 .where(table.c[FN_DEVICE_ID] == SpecialNote.device_id) 

1418 .where(table.c[FN_ERA] == new_era) 

1419 ) 

1420 # ^^^^^^^^^^^^^^^^^^^^^^^^^^ 

1421 # This bit restricts to records being preserved. 

1422 .values(era=new_era) 

1423 ) 

1424 else: 

1425 req.dbsession.execute( 

1426 update(table) 

1427 .where(table.c[FN_PK].in_(pks_to_preserve)) 

1428 .values({MOVE_OFF_TABLET_FIELD: 1}) 

1429 ) 

1430 

1431 

1432def flag_record_for_preservation( 

1433 req: "CamcopsRequest", batchdetails: BatchDetails, table: Table, pk: int 

1434) -> List[int]: 

1435 """ 

1436 Marks a record for preservation (moving off the tablet, changing its 

1437 era details). 

1438 

1439 2018-11-18: works back through the predecessor chain too, fixing an old 

1440 bug. 

1441 

1442 Args: 

1443 req: the :class:`camcops_server.cc_modules.cc_request.CamcopsRequest` 

1444 batchdetails: the :class:`BatchDetails` 

1445 table: SQLAlchemy :class:`Table` 

1446 pk: server PK of the record to mark 

1447 

1448 Returns: 

1449 list: all PKs being preserved 

1450 """ 

1451 pks_to_preserve = get_all_predecessor_pks(req, table, pk) 

1452 flag_multiple_records_for_preservation( 

1453 req, batchdetails, table, pks_to_preserve 

1454 ) 

1455 return pks_to_preserve 

1456 

1457 

1458def preserve_all( 

1459 req: "CamcopsRequest", batchdetails: BatchDetails, table: Table 

1460) -> None: 

1461 """ 

1462 Preserves all records in a table for a device, including non-current ones. 

1463 

1464 Args: 

1465 req: the :class:`camcops_server.cc_modules.cc_request.CamcopsRequest` 

1466 batchdetails: the :class:`BatchDetails` 

1467 table: SQLAlchemy :class:`Table` 

1468 """ 

1469 device_id = req.tabletsession.device_id 

1470 req.dbsession.execute( 

1471 update(table) 

1472 .where(table.c[FN_DEVICE_ID] == device_id) 

1473 .where(table.c[FN_ERA] == ERA_NOW) 

1474 .values(values_preserve_now(req, batchdetails)) 

1475 ) 

1476 

1477 

1478# ============================================================================= 

1479# Upload helper functions 

1480# ============================================================================= 

1481 

1482 

1483def process_upload_record_special( 

1484 req: "CamcopsRequest", 

1485 batchdetails: BatchDetails, 

1486 table: Table, 

1487 valuedict: Dict[str, Any], 

1488) -> None: 

1489 """ 

1490 Special processing function for upload, in which we inspect the data. 

1491 Called by :func:`upload_record_core`. 

1492 

1493 1. Handles old clients with ID information in the patient table, etc. 

1494 (Note: this can be IGNORED for any client using 

1495 :func:`op_upload_entire_database`, as these are newer.) 

1496 

1497 2. Validates ID numbers. 

1498 

1499 Args: 

1500 req: the :class:`camcops_server.cc_modules.cc_request.CamcopsRequest` 

1501 batchdetails: the :class:`BatchDetails` 

1502 table: an SQLAlchemy :class:`Table` 

1503 valuedict: a dictionary of {colname: value} pairs from the client. 

1504 May be modified. 

1505 """ 

1506 ts = req.tabletsession 

1507 tablename = table.name 

1508 

1509 if tablename == Patient.__tablename__: 

1510 # --------------------------------------------------------------------- 

1511 # Deal with old tablets that had ID numbers in a less flexible format. 

1512 # --------------------------------------------------------------------- 

1513 if ts.cope_with_deleted_patient_descriptors: 

1514 # Old tablets (pre-2.0.0) will upload copies of the ID 

1515 # descriptions with the patient. To cope with that, we 

1516 # remove those here: 

1517 for n in range(1, NUMBER_OF_IDNUMS_DEFUNCT + 1): 

1518 nstr = str(n) 

1519 fn_desc = FP_ID_DESC + nstr 

1520 fn_shortdesc = FP_ID_SHORT_DESC + nstr 

1521 valuedict.pop(fn_desc, None) # remove item, if exists 

1522 valuedict.pop(fn_shortdesc, None) 

1523 

1524 if ts.cope_with_old_idnums: 

1525 # Insert records into the new ID number table from the old 

1526 # patient table: 

1527 for which_idnum in range(1, NUMBER_OF_IDNUMS_DEFUNCT + 1): 

1528 nstr = str(which_idnum) 

1529 fn_idnum = FP_ID_NUM + nstr 

1530 idnum_value = valuedict.pop(fn_idnum, None) 

1531 # ... and remove it from our new Patient record 

1532 patient_id = valuedict.get("id", None) 

1533 if idnum_value is None or patient_id is None: 

1534 continue 

1535 # noinspection PyUnresolvedReferences 

1536 mark_table_dirty(req, PatientIdNum.__table__) 

1537 client_date_value = coerce_to_pendulum( 

1538 valuedict[CLIENT_DATE_FIELD] 

1539 ) 

1540 # noinspection PyUnresolvedReferences 

1541 upload_record_core( 

1542 req=req, 

1543 batchdetails=batchdetails, 

1544 table=PatientIdNum.__table__, 

1545 clientpk_name="id", 

1546 valuedict={ 

1547 "id": fake_tablet_id_for_patientidnum( 

1548 patient_id=patient_id, which_idnum=which_idnum 

1549 ), # ... guarantees a pseudo client PK 

1550 "patient_id": patient_id, 

1551 "which_idnum": which_idnum, 

1552 "idnum_value": idnum_value, 

1553 CLIENT_DATE_FIELD: client_date_value, 

1554 MOVE_OFF_TABLET_FIELD: valuedict[ 

1555 MOVE_OFF_TABLET_FIELD 

1556 ], # noqa 

1557 }, 

1558 ) 

1559 # Now, how to deal with deletion, i.e. records missing from the 

1560 # tablet? See our caller, op_upload_table(), which has a special 

1561 # handler for this. 

1562 # 

1563 # Note that op_upload_record() is/was only used for BLOBs, so we 

1564 # don't have to worry about special processing for that aspect 

1565 # here; also, that method handles deletion in a different way. 

1566 

1567 elif tablename == PatientIdNum.__tablename__: 

1568 # --------------------------------------------------------------------- 

1569 # Validate ID numbers. 

1570 # --------------------------------------------------------------------- 

1571 which_idnum = valuedict.get("which_idnum", None) 

1572 if which_idnum not in req.valid_which_idnums: 

1573 fail_user_error(f"No such ID number type: {which_idnum}") 

1574 idnum_value = valuedict.get("idnum_value", None) 

1575 if not req.is_idnum_valid(which_idnum, idnum_value): 

1576 why_invalid = req.why_idnum_invalid(which_idnum, idnum_value) 

1577 fail_user_error( 

1578 f"For ID type {which_idnum}, ID number {idnum_value} is " 

1579 f"invalid: {why_invalid}" 

1580 ) 

1581 

1582 

1583def upload_record_core( 

1584 req: "CamcopsRequest", 

1585 batchdetails: BatchDetails, 

1586 table: Table, 

1587 clientpk_name: str, 

1588 valuedict: Dict[str, Any], 

1589 server_live_current_records: List[ServerRecord] = None, 

1590) -> UploadRecordResult: 

1591 """ 

1592 Uploads a record. Deals with IDENTICAL, NEW, and MODIFIED records. 

1593 

1594 Used by :func:`upload_table` and :func:`upload_record`. 

1595 

1596 Args: 

1597 req: the :class:`camcops_server.cc_modules.cc_request.CamcopsRequest` 

1598 batchdetails: the :class:`BatchDetails` 

1599 table: an SQLAlchemy :class:`Table` 

1600 clientpk_name: the column name of the client's PK 

1601 valuedict: a dictionary of {colname: value} pairs from the client 

1602 server_live_current_records: list of :class:`ServerRecord` objects for 

1603 the active records on the server for this client, in this table 

1604 

1605 Returns: 

1606 a :class:`UploadRecordResult` object 

1607 """ 

1608 require_keys( 

1609 valuedict, [clientpk_name, CLIENT_DATE_FIELD, MOVE_OFF_TABLET_FIELD] 

1610 ) 

1611 clientpk_value = valuedict[clientpk_name] 

1612 

1613 if server_live_current_records: 

1614 # All server records for this table/device/era have been prefetched. 

1615 serverrec = next( 

1616 ( 

1617 r 

1618 for r in server_live_current_records 

1619 if r.client_pk == clientpk_value 

1620 ), 

1621 None, 

1622 ) 

1623 if serverrec is None: 

1624 serverrec = ServerRecord(clientpk_value, False) 

1625 else: 

1626 # Look up this record specifically. 

1627 serverrec = record_exists(req, table, clientpk_name, clientpk_value) 

1628 

1629 if DEBUG_UPLOAD: 

1630 log.debug("upload_record_core: {}, {}", table.name, serverrec) 

1631 

1632 oldserverpk = serverrec.server_pk 

1633 urr = UploadRecordResult( 

1634 oldserverpk=oldserverpk, 

1635 specifically_marked_for_preservation=bool( 

1636 valuedict[MOVE_OFF_TABLET_FIELD] 

1637 ), 

1638 dirty=True, 

1639 ) 

1640 if serverrec.exists: 

1641 # There's an existing record, which is either identical or not. 

1642 client_date_value = coerce_to_pendulum(valuedict[CLIENT_DATE_FIELD]) 

1643 if serverrec.server_when == client_date_value: 

1644 # The existing record is identical. 

1645 # No action needed unless MOVE_OFF_TABLET_FIELDNAME is set. 

1646 if not urr.specifically_marked_for_preservation: 

1647 urr.dirty = False 

1648 else: 

1649 # The existing record is different. We need a logical UPDATE, but 

1650 # maintaining an audit trail. 

1651 process_upload_record_special(req, batchdetails, table, valuedict) 

1652 urr.newserverpk = insert_record( 

1653 req, batchdetails, table, valuedict, oldserverpk 

1654 ) 

1655 flag_modified( 

1656 req, batchdetails, table, oldserverpk, urr.newserverpk 

1657 ) 

1658 else: 

1659 # The record is NEW. We need to INSERT it. 

1660 process_upload_record_special(req, batchdetails, table, valuedict) 

1661 urr.newserverpk = insert_record( 

1662 req, batchdetails, table, valuedict, None 

1663 ) 

1664 if urr.specifically_marked_for_preservation: 

1665 preservation_pks = flag_record_for_preservation( 

1666 req, batchdetails, table, urr.latest_pk 

1667 ) 

1668 urr.note_specifically_marked_preservation_pks(preservation_pks) 

1669 

1670 if DEBUG_UPLOAD: 

1671 log.debug("upload_record_core: {}, {!r}", table.name, urr) 

1672 return urr 

1673 

1674 

1675def insert_record( 

1676 req: "CamcopsRequest", 

1677 batchdetails: BatchDetails, 

1678 table: Table, 

1679 valuedict: Dict[str, Any], 

1680 predecessor_pk: Optional[int], 

1681) -> int: 

1682 """ 

1683 Inserts a record, or raises an exception if that fails. 

1684 

1685 Args: 

1686 req: the :class:`camcops_server.cc_modules.cc_request.CamcopsRequest` 

1687 batchdetails: the :class:`BatchDetails` 

1688 table: an SQLAlchemy :class:`Table` 

1689 valuedict: a dictionary of {colname: value} pairs from the client 

1690 predecessor_pk: an optional server PK of the record's predecessor 

1691 

1692 Returns: 

1693 the server PK of the new record 

1694 """ 

1695 ts = req.tabletsession 

1696 valuedict.update( 

1697 { 

1698 FN_DEVICE_ID: ts.device_id, 

1699 FN_ERA: ERA_NOW, 

1700 FN_REMOVAL_PENDING: 0, 

1701 FN_PREDECESSOR_PK: predecessor_pk, 

1702 FN_CAMCOPS_VERSION: ts.tablet_version_str, 

1703 FN_GROUP_ID: req.user.upload_group_id, 

1704 } 

1705 ) 

1706 if batchdetails.onestep: 

1707 valuedict.update( 

1708 { 

1709 FN_CURRENT: 1, 

1710 FN_ADDITION_PENDING: 0, 

1711 FN_ADDING_USER_ID: req.user_id, 

1712 FN_WHEN_ADDED_EXACT: req.now, 

1713 FN_WHEN_ADDED_BATCH_UTC: batchdetails.batchtime, 

1714 } 

1715 ) 

1716 else: 

1717 valuedict.update({FN_CURRENT: 0, FN_ADDITION_PENDING: 1}) 

1718 rp = req.dbsession.execute( 

1719 table.insert().values(valuedict) 

1720 ) # type: ResultProxy 

1721 inserted_pks = rp.inserted_primary_key 

1722 assert isinstance(inserted_pks, list) and len(inserted_pks) == 1 

1723 return inserted_pks[0] 

1724 

1725 

1726def audit_upload( 

1727 req: "CamcopsRequest", changes: List[UploadTableChanges] 

1728) -> None: 

1729 """ 

1730 Writes audit information for an upload. 

1731 

1732 Args: 

1733 req: the :class:`camcops_server.cc_modules.cc_request.CamcopsRequest` 

1734 changes: a list of :class:`UploadTableChanges` objects, one per table 

1735 """ 

1736 msg = ( 

1737 f"Upload from device {req.tabletsession.device_id}, " 

1738 f"username {req.tabletsession.username!r}: " 

1739 ) 

1740 changes = [x for x in changes if x.any_changes] 

1741 if changes: 

1742 changes.sort(key=lambda x: x.tablename) 

1743 msg += ", ".join(x.description() for x in changes) 

1744 else: 

1745 msg += "No changes" 

1746 log.info("audit_upload: {}", msg) 

1747 audit(req, msg) 

1748 

1749 

1750# ============================================================================= 

1751# Batch (atomic) upload and preserving 

1752# ============================================================================= 

1753 

1754 

1755def get_batch_details(req: "CamcopsRequest") -> BatchDetails: 

1756 """ 

1757 Returns the :class:`BatchDetails` for the current upload. If none exists, 

1758 a new batch is created and returned. 

1759 

1760 SIDE EFFECT: if the username is different from the username that started 

1761 a previous upload batch for this device, we restart the upload batch (thus 

1762 rolling back previous pending changes). 

1763 

1764 Raises: 

1765 :exc:`camcops_server.cc_modules.cc_client_api_core.ServerErrorException` 

1766 if the device doesn't exist 

1767 """ 

1768 device_id = req.tabletsession.device_id 

1769 # noinspection PyUnresolvedReferences 

1770 query = ( 

1771 select( 

1772 [ 

1773 Device.ongoing_upload_batch_utc, 

1774 Device.uploading_user_id, 

1775 Device.currently_preserving, 

1776 ] 

1777 ) 

1778 .select_from(Device.__table__) 

1779 .where(Device.id == device_id) 

1780 ) 

1781 row = req.dbsession.execute(query).fetchone() 

1782 if not row: 

1783 fail_server_error( 

1784 f"Device {device_id} missing from Device table" 

1785 ) # will raise # noqa 

1786 upload_batch_utc, uploading_user_id, currently_preserving = row 

1787 if not upload_batch_utc or uploading_user_id != req.user_id: 

1788 # SIDE EFFECT: if the username changes, we restart (and thus roll back 

1789 # previous pending changes) 

1790 start_device_upload_batch(req) 

1791 return BatchDetails(req.now_utc, False) 

1792 return BatchDetails(upload_batch_utc, currently_preserving) 

1793 

1794 

1795def start_device_upload_batch(req: "CamcopsRequest") -> None: 

1796 """ 

1797 Starts an upload batch for a device. 

1798 """ 

1799 rollback_all(req) 

1800 # noinspection PyUnresolvedReferences 

1801 req.dbsession.execute( 

1802 update(Device.__table__) 

1803 .where(Device.id == req.tabletsession.device_id) 

1804 .values( 

1805 last_upload_batch_utc=req.now_utc, 

1806 ongoing_upload_batch_utc=req.now_utc, 

1807 uploading_user_id=req.tabletsession.user_id, 

1808 ) 

1809 ) 

1810 

1811 

1812def _clear_ongoing_upload_batch_details(req: "CamcopsRequest") -> None: 

1813 """ 

1814 Clears upload batch details from the Device table. 

1815 

1816 Args: 

1817 req: the :class:`camcops_server.cc_modules.cc_request.CamcopsRequest` 

1818 """ 

1819 # noinspection PyUnresolvedReferences 

1820 req.dbsession.execute( 

1821 update(Device.__table__) 

1822 .where(Device.id == req.tabletsession.device_id) 

1823 .values( 

1824 ongoing_upload_batch_utc=None, 

1825 uploading_user_id=None, 

1826 currently_preserving=0, 

1827 ) 

1828 ) 

1829 

1830 

1831def end_device_upload_batch( 

1832 req: "CamcopsRequest", batchdetails: BatchDetails 

1833) -> None: 

1834 """ 

1835 Ends an upload batch, committing all changes made thus far. 

1836 

1837 Args: 

1838 req: the :class:`camcops_server.cc_modules.cc_request.CamcopsRequest` 

1839 batchdetails: the :class:`BatchDetails` 

1840 """ 

1841 commit_all(req, batchdetails) 

1842 _clear_ongoing_upload_batch_details(req) 

1843 

1844 

1845def clear_device_upload_batch(req: "CamcopsRequest") -> None: 

1846 """ 

1847 Ensures there is nothing pending. Rools back previous changes. Wipes any 

1848 ongoing batch details. 

1849 

1850 Args: 

1851 req: the :class:`camcops_server.cc_modules.cc_request.CamcopsRequest` 

1852 """ 

1853 rollback_all(req) 

1854 _clear_ongoing_upload_batch_details(req) 

1855 

1856 

1857def start_preserving(req: "CamcopsRequest") -> None: 

1858 """ 

1859 Starts preservation (the process of moving records from the NOW era to 

1860 an older era, so they can be removed safely from the tablet). 

1861 

1862 Called by :func:`op_start_preservation`. 

1863 

1864 In this situation, we start by assuming that ALL tables are "dirty", 

1865 because they may have live records from a previous upload. 

1866 """ 

1867 # noinspection PyUnresolvedReferences 

1868 req.dbsession.execute( 

1869 update(Device.__table__) 

1870 .where(Device.id == req.tabletsession.device_id) 

1871 .values(currently_preserving=1) 

1872 ) 

1873 mark_all_tables_dirty(req) 

1874 

1875 

1876def mark_table_dirty(req: "CamcopsRequest", table: Table) -> None: 

1877 """ 

1878 Marks a table as having been modified during the current upload. 

1879 """ 

1880 tablename = table.name 

1881 device_id = req.tabletsession.device_id 

1882 dbsession = req.dbsession 

1883 # noinspection PyUnresolvedReferences 

1884 table_already_dirty = exists_in_table( 

1885 dbsession, 

1886 DirtyTable.__table__, 

1887 DirtyTable.device_id == device_id, 

1888 DirtyTable.tablename == tablename, 

1889 ) 

1890 if not table_already_dirty: 

1891 # noinspection PyUnresolvedReferences 

1892 dbsession.execute( 

1893 DirtyTable.__table__.insert().values( 

1894 device_id=device_id, tablename=tablename 

1895 ) 

1896 ) 

1897 

1898 

1899def mark_tables_dirty(req: "CamcopsRequest", tables: List[Table]) -> None: 

1900 """ 

1901 Marks multiple tables as dirty. 

1902 """ 

1903 if not tables: 

1904 return 

1905 device_id = req.tabletsession.device_id 

1906 tablenames = [t.name for t in tables] 

1907 # Delete first 

1908 # noinspection PyUnresolvedReferences 

1909 req.dbsession.execute( 

1910 DirtyTable.__table__.delete() 

1911 .where(DirtyTable.device_id == device_id) 

1912 .where(DirtyTable.tablename.in_(tablenames)) 

1913 ) 

1914 # Then insert 

1915 insert_values = [ 

1916 {"device_id": device_id, "tablename": tn} for tn in tablenames 

1917 ] 

1918 # noinspection PyUnresolvedReferences 

1919 req.dbsession.execute(DirtyTable.__table__.insert(), insert_values) 

1920 

1921 

1922def mark_all_tables_dirty(req: "CamcopsRequest") -> None: 

1923 """ 

1924 If we are preserving, we assume that all tables are "dirty" (require work 

1925 when we complete the upload) unless we specifically mark them clean. 

1926 """ 

1927 device_id = req.tabletsession.device_id 

1928 # Delete first 

1929 # noinspection PyUnresolvedReferences 

1930 req.dbsession.execute( 

1931 DirtyTable.__table__.delete().where(DirtyTable.device_id == device_id) 

1932 ) 

1933 # Now insert 

1934 # https://docs.sqlalchemy.org/en/latest/core/tutorial.html#execute-multiple 

1935 all_client_tablenames = list(CLIENT_TABLE_MAP.keys()) 

1936 insert_values = [ 

1937 {"device_id": device_id, "tablename": tn} 

1938 for tn in all_client_tablenames 

1939 ] 

1940 # noinspection PyUnresolvedReferences 

1941 req.dbsession.execute(DirtyTable.__table__.insert(), insert_values) 

1942 

1943 

1944def mark_table_clean(req: "CamcopsRequest", table: Table) -> None: 

1945 """ 

1946 Marks a table as being clean: that is, 

1947 

1948 - the table has been scanned during the current upload 

1949 - there is nothing to do (either from the current upload, OR A PREVIOUS 

1950 UPLOAD). 

1951 """ 

1952 tablename = table.name 

1953 device_id = req.tabletsession.device_id 

1954 # noinspection PyUnresolvedReferences 

1955 req.dbsession.execute( 

1956 DirtyTable.__table__.delete() 

1957 .where(DirtyTable.device_id == device_id) 

1958 .where(DirtyTable.tablename == tablename) 

1959 ) 

1960 

1961 

1962def mark_tables_clean(req: "CamcopsRequest", tables: List[Table]) -> None: 

1963 """ 

1964 Marks multiple tables as clean. 

1965 """ 

1966 if not tables: 

1967 return 

1968 device_id = req.tabletsession.device_id 

1969 tablenames = [t.name for t in tables] 

1970 # Delete first 

1971 # noinspection PyUnresolvedReferences 

1972 req.dbsession.execute( 

1973 DirtyTable.__table__.delete() 

1974 .where(DirtyTable.device_id == device_id) 

1975 .where(DirtyTable.tablename.in_(tablenames)) 

1976 ) 

1977 

1978 

1979def get_dirty_tables(req: "CamcopsRequest") -> List[Table]: 

1980 """ 

1981 Returns tables marked as dirty for this device. (See 

1982 :func:`mark_table_dirty`.) 

1983 """ 

1984 query = select([DirtyTable.tablename]).where( 

1985 DirtyTable.device_id == req.tabletsession.device_id 

1986 ) 

1987 tablenames = fetch_all_first_values(req.dbsession, query) 

1988 return [CLIENT_TABLE_MAP[tn] for tn in tablenames] 

1989 

1990 

1991def commit_all(req: "CamcopsRequest", batchdetails: BatchDetails) -> None: 

1992 """ 

1993 Commits additions, removals, and preservations for all tables. 

1994 

1995 Args: 

1996 req: the :class:`camcops_server.cc_modules.cc_request.CamcopsRequest` 

1997 batchdetails: the :class:`BatchDetails` 

1998 """ 

1999 tables = get_dirty_tables(req) 

2000 # log.debug("Dirty tables: {}", list(t.name for t in tables)) 

2001 tables.sort(key=upload_commit_order_sorter) 

2002 

2003 changelist = [] # type: List[UploadTableChanges] 

2004 for table in tables: 

2005 auditinfo = commit_table(req, batchdetails, table, clear_dirty=False) 

2006 changelist.append(auditinfo) 

2007 

2008 if batchdetails.preserving: 

2009 # Also preserve/finalize any corresponding special notes (2015-02-01), 

2010 # but all in one go (2018-11-13). 

2011 # noinspection PyUnresolvedReferences 

2012 req.dbsession.execute( 

2013 update(SpecialNote.__table__) 

2014 .where(SpecialNote.device_id == req.tabletsession.device_id) 

2015 .where(SpecialNote.era == ERA_NOW) 

2016 .values(era=batchdetails.new_era) 

2017 ) 

2018 

2019 clear_dirty_tables(req) 

2020 audit_upload(req, changelist) 

2021 

2022 # Performance 2018-11-13: 

2023 # - start at 2.407 s 

2024 # - remove final temptable clearance and COUNT(*): 1.626 to 2.118 s 

2025 # - IN clause using Python literal not temptable: 1.18 to 1.905 s 

2026 # - minor tidy: 1.075 to 1.65 

2027 # - remove ORDER BY from task indexing: 1.093 to 1.607 

2028 # - optimize special note code won't affect this: 1.076 to 1.617 

2029 # At this point, entire upload process ~5s. 

2030 # - big difference from commit_table() query optimization 

2031 # - huge difference from being more careful with mark_table_dirty() 

2032 # - further table scanning optimizations: fewer queries 

2033 # Overall upload down to ~2.4s 

2034 

2035 

2036def commit_table( 

2037 req: "CamcopsRequest", 

2038 batchdetails: BatchDetails, 

2039 table: Table, 

2040 clear_dirty: bool = True, 

2041) -> UploadTableChanges: 

2042 """ 

2043 Commits additions, removals, and preservations for one table. 

2044 

2045 Should ONLY be called by :func:`commit_all`. 

2046 

2047 Also updates task indexes. 

2048 

2049 Args: 

2050 req: the :class:`camcops_server.cc_modules.cc_request.CamcopsRequest` 

2051 batchdetails: the :class:`BatchDetails` 

2052 table: SQLAlchemy :class:`Table` 

2053 clear_dirty: remove the table from the record of dirty tables for 

2054 this device? (If called from :func:`commit_all`, this should be 

2055 ``False``, since it's faster to clear all dirty tables for the 

2056 device simultaneously than one-by-one.) 

2057 

2058 Returns: 

2059 an :class:`UploadTableChanges` object 

2060 """ 

2061 

2062 # Tried storing PKs in temporary tables, rather than using an IN clause 

2063 # with Python values, as per 

2064 # https://www.xaprb.com/blog/2006/06/28/why-large-in-clauses-are-problematic/ # noqa 

2065 # However, it was slow. 

2066 # We can gain a lot of efficiency (empirically) by: 

2067 # - Storing PKs in Python 

2068 # - Only performing updates when we need to 

2069 # - Using a single query per table to get "add/remove/preserve" PKs 

2070 

2071 # ------------------------------------------------------------------------- 

2072 # Helpful temporary variables 

2073 # ------------------------------------------------------------------------- 

2074 user_id = req.user_id 

2075 device_id = req.tabletsession.device_id 

2076 exacttime = req.now 

2077 dbsession = req.dbsession 

2078 tablename = table.name 

2079 batchtime = batchdetails.batchtime 

2080 preserving = batchdetails.preserving 

2081 

2082 # ------------------------------------------------------------------------- 

2083 # Fetch addition, removal, preservation, current PKs in a single query 

2084 # ------------------------------------------------------------------------- 

2085 tablechanges = UploadTableChanges(table) 

2086 serverrecs = get_server_live_records( 

2087 req, device_id, table, current_only=False 

2088 ) 

2089 for sr in serverrecs: 

2090 tablechanges.note_serverrec(sr, preserving=preserving) 

2091 

2092 # ------------------------------------------------------------------------- 

2093 # Additions 

2094 # ------------------------------------------------------------------------- 

2095 # Update the records we're adding 

2096 addition_pks = tablechanges.addition_pks 

2097 if addition_pks: 

2098 # log.debug("commit_table: {}, adding server PKs {}", 

2099 # tablename, addition_pks) 

2100 dbsession.execute( 

2101 update(table) 

2102 .where(table.c[FN_PK].in_(addition_pks)) 

2103 .values( 

2104 { 

2105 FN_CURRENT: 1, 

2106 FN_ADDITION_PENDING: 0, 

2107 FN_ADDING_USER_ID: user_id, 

2108 FN_WHEN_ADDED_EXACT: exacttime, 

2109 FN_WHEN_ADDED_BATCH_UTC: batchtime, 

2110 } 

2111 ) 

2112 ) 

2113 

2114 # ------------------------------------------------------------------------- 

2115 # Removals 

2116 # ------------------------------------------------------------------------- 

2117 # Update the records we're removing 

2118 removal_pks = tablechanges.removal_pks 

2119 if removal_pks: 

2120 # log.debug("commit_table: {}, removing server PKs {}", 

2121 # tablename, removal_pks) 

2122 dbsession.execute( 

2123 update(table) 

2124 .where(table.c[FN_PK].in_(removal_pks)) 

2125 .values(values_delete_now(req, batchdetails)) 

2126 ) 

2127 

2128 # ------------------------------------------------------------------------- 

2129 # Preservation 

2130 # ------------------------------------------------------------------------- 

2131 # Preserve necessary records 

2132 preservation_pks = tablechanges.preservation_pks 

2133 if preservation_pks: 

2134 # log.debug("commit_table: {}, preserving server PKs {}", 

2135 # tablename, preservation_pks) 

2136 new_era = batchdetails.new_era 

2137 dbsession.execute( 

2138 update(table) 

2139 .where(table.c[FN_PK].in_(preservation_pks)) 

2140 .values(values_preserve_now(req, batchdetails)) 

2141 ) 

2142 if not preserving: 

2143 # Also preserve/finalize any corresponding special notes 

2144 # (2015-02-01), just for records being specifically preserved. If 

2145 # we are preserving, this step happens in one go in commit_all() 

2146 # (2018-11-13). 

2147 # noinspection PyUnresolvedReferences 

2148 dbsession.execute( 

2149 update(SpecialNote.__table__) 

2150 .where(SpecialNote.basetable == tablename) 

2151 .where(SpecialNote.device_id == device_id) 

2152 .where(SpecialNote.era == ERA_NOW) 

2153 .where( 

2154 exists() 

2155 .select_from(table) 

2156 .where(table.c[TABLET_ID_FIELD] == SpecialNote.task_id) 

2157 .where(table.c[FN_DEVICE_ID] == SpecialNote.device_id) 

2158 .where(table.c[FN_ERA] == new_era) 

2159 ) 

2160 # ^^^^^^^^^^^^^^^^^^^^^^^^^^ 

2161 # This bit restricts to records being preserved. 

2162 .values(era=new_era) 

2163 ) 

2164 

2165 # ------------------------------------------------------------------------- 

2166 # Update special indexes 

2167 # ------------------------------------------------------------------------- 

2168 update_indexes_and_push_exports(req, batchdetails, tablechanges) 

2169 

2170 # ------------------------------------------------------------------------- 

2171 # Remove individually from list of dirty tables? 

2172 # ------------------------------------------------------------------------- 

2173 if clear_dirty: 

2174 # noinspection PyUnresolvedReferences 

2175 dbsession.execute( 

2176 DirtyTable.__table__.delete() 

2177 .where(DirtyTable.device_id == device_id) 

2178 .where(DirtyTable.tablename == tablename) 

2179 ) 

2180 # ... otherwise a call to clear_dirty_tables() must be made. 

2181 

2182 if DEBUG_UPLOAD: 

2183 log.debug("commit_table: {}", tablechanges) 

2184 

2185 return tablechanges 

2186 

2187 

2188def rollback_all(req: "CamcopsRequest") -> None: 

2189 """ 

2190 Rolls back all pending changes for a device. 

2191 """ 

2192 tables = get_dirty_tables(req) 

2193 for table in tables: 

2194 rollback_table(req, table) 

2195 clear_dirty_tables(req) 

2196 

2197 

2198def rollback_table(req: "CamcopsRequest", table: Table) -> None: 

2199 """ 

2200 Rolls back changes for an individual table for a device. 

2201 """ 

2202 device_id = req.tabletsession.device_id 

2203 # Pending additions 

2204 req.dbsession.execute( 

2205 table.delete() 

2206 .where(table.c[FN_DEVICE_ID] == device_id) 

2207 .where(table.c[FN_ADDITION_PENDING]) 

2208 ) 

2209 # Pending deletions 

2210 req.dbsession.execute( 

2211 update(table) 

2212 .where(table.c[FN_DEVICE_ID] == device_id) 

2213 .where(table.c[FN_REMOVAL_PENDING]) 

2214 .values( 

2215 { 

2216 FN_REMOVAL_PENDING: 0, 

2217 FN_WHEN_ADDED_EXACT: None, 

2218 FN_WHEN_REMOVED_BATCH_UTC: None, 

2219 FN_REMOVING_USER_ID: None, 

2220 FN_SUCCESSOR_PK: None, 

2221 } 

2222 ) 

2223 ) 

2224 # Record-specific preservation (set by flag_record_for_preservation()) 

2225 req.dbsession.execute( 

2226 update(table) 

2227 .where(table.c[FN_DEVICE_ID] == device_id) 

2228 .values({MOVE_OFF_TABLET_FIELD: 0}) 

2229 ) 

2230 

2231 

2232def clear_dirty_tables(req: "CamcopsRequest") -> None: 

2233 """ 

2234 Clears the dirty-table list for a device. 

2235 """ 

2236 device_id = req.tabletsession.device_id 

2237 # noinspection PyUnresolvedReferences 

2238 req.dbsession.execute( 

2239 DirtyTable.__table__.delete().where(DirtyTable.device_id == device_id) 

2240 ) 

2241 

2242 

2243# ============================================================================= 

2244# Additional helper functions for one-step upload 

2245# ============================================================================= 

2246 

2247 

2248def process_table_for_onestep_upload( 

2249 req: "CamcopsRequest", 

2250 batchdetails: BatchDetails, 

2251 table: Table, 

2252 clientpk_name: str, 

2253 rows: List[Dict[str, Any]], 

2254) -> UploadTableChanges: 

2255 """ 

2256 Performs all upload steps for a table. 

2257 

2258 Note that we arrive here in a specific and safe table order; search for 

2259 :func:`camcops_server.cc_modules.cc_client_api_helpers.upload_commit_order_sorter`. 

2260 

2261 Args: 

2262 req: the :class:`camcops_server.cc_modules.cc_request.CamcopsRequest` 

2263 batchdetails: the :class:`BatchDetails` 

2264 table: an SQLAlchemy :class:`Table` 

2265 clientpk_name: the name of the PK field on the client 

2266 rows: a list of rows, where each row is a dictionary mapping field 

2267 (column) names to values (those values being encoded as SQL-style 

2268 literals in our extended syntax) 

2269 

2270 Returns: 

2271 an :class:`UploadTableChanges` object 

2272 """ # noqa 

2273 serverrecs = get_server_live_records( 

2274 req, 

2275 req.tabletsession.device_id, 

2276 table, 

2277 clientpk_name, 

2278 current_only=False, 

2279 ) 

2280 servercurrentrecs = [r for r in serverrecs if r.current] 

2281 if rows and not clientpk_name: 

2282 fail_user_error( 

2283 f"Client-side PK name not specified by client for " 

2284 f"non-empty table {table.name!r}" 

2285 ) 

2286 tablechanges = UploadTableChanges(table) 

2287 server_pks_uploaded = [] # type: List[int] 

2288 for row in rows: 

2289 valuedict = {k: decode_single_value(v) for k, v in row.items()} 

2290 urr = upload_record_core( 

2291 req, 

2292 batchdetails, 

2293 table, 

2294 clientpk_name, 

2295 valuedict, 

2296 server_live_current_records=servercurrentrecs, 

2297 ) 

2298 # ... handles addition, modification, preservation, special processing 

2299 # But we also make a note of these for indexing: 

2300 if urr.oldserverpk is not None: 

2301 server_pks_uploaded.append(urr.oldserverpk) 

2302 tablechanges.note_urr( 

2303 urr, preserving_new_records=batchdetails.preserving 

2304 ) 

2305 # Which leaves: 

2306 # (*) Deletion (where no record was uploaded at all) 

2307 server_pks_for_deletion = [ 

2308 r.server_pk 

2309 for r in servercurrentrecs 

2310 if r.server_pk not in server_pks_uploaded 

2311 ] 

2312 if server_pks_for_deletion: 

2313 flag_deleted(req, batchdetails, table, server_pks_for_deletion) 

2314 tablechanges.note_removal_deleted_pks(server_pks_for_deletion) 

2315 

2316 # Preserving all records not specifically processed above, too 

2317 if batchdetails.preserving: 

2318 # Preserve all, including noncurrent: 

2319 preserve_all(req, batchdetails, table) 

2320 # Note other preserved records, for indexing: 

2321 tablechanges.note_preservation_pks(r.server_pk for r in serverrecs) 

2322 

2323 # (*) Indexing (and push exports) 

2324 update_indexes_and_push_exports(req, batchdetails, tablechanges) 

2325 

2326 if DEBUG_UPLOAD: 

2327 log.debug("process_table_for_onestep_upload: {}", tablechanges) 

2328 

2329 return tablechanges 

2330 

2331 

2332# ============================================================================= 

2333# Audit functions 

2334# ============================================================================= 

2335 

2336 

2337def audit( 

2338 req: "CamcopsRequest", 

2339 details: str, 

2340 patient_server_pk: int = None, 

2341 tablename: str = None, 

2342 server_pk: int = None, 

2343) -> None: 

2344 """ 

2345 Audit something. 

2346 """ 

2347 # Add parameters and pass on: 

2348 cc_audit.audit( 

2349 req=req, 

2350 details=details, 

2351 patient_server_pk=patient_server_pk, 

2352 table=tablename, 

2353 server_pk=server_pk, 

2354 device_id=req.tabletsession.device_id, # added 

2355 remote_addr=req.remote_addr, # added 

2356 user_id=req.user_id, # added 

2357 from_console=False, # added 

2358 from_dbclient=True, # added 

2359 ) 

2360 

2361 

2362# ============================================================================= 

2363# Helper functions for single-user mode 

2364# ============================================================================= 

2365 

2366 

2367def make_single_user_mode_username( 

2368 client_device_name: str, patient_pk: int 

2369) -> str: 

2370 """ 

2371 Returns the username for single-user mode. 

2372 """ 

2373 return f"user-{client_device_name}-{patient_pk}" 

2374 

2375 

2376def json_patient_info(patient: Patient) -> str: 

2377 """ 

2378 Converts patient details to a string representation of a JSON list (one 

2379 patient) containing a single JSON dictionary (detailing that patient), with 

2380 keys/formats known to the client. 

2381 

2382 (One item list to be consistent with patients uploaded from the tablet.) 

2383 

2384 Args: 

2385 patient: :class:`camcops_server.cc_modules.cc_patient.Patient` 

2386 """ 

2387 patient_dict = { 

2388 TabletParam.SURNAME: patient.surname, 

2389 TabletParam.FORENAME: patient.forename, 

2390 TabletParam.SEX: patient.sex, 

2391 TabletParam.DOB: format_datetime( 

2392 patient.dob, DateFormat.ISO8601_DATE_ONLY 

2393 ), 

2394 TabletParam.EMAIL: patient.email, 

2395 TabletParam.ADDRESS: patient.address, 

2396 TabletParam.GP: patient.gp, 

2397 TabletParam.OTHER: patient.other, 

2398 } 

2399 for idnum in patient.idnums: 

2400 key = f"{TabletParam.IDNUM_PREFIX}{idnum.which_idnum}" 

2401 patient_dict[key] = idnum.idnum_value 

2402 # One item list to be consistent with patients uploaded from the tablet 

2403 return json.dumps([patient_dict]) 

2404 

2405 

2406def get_single_server_patient(req: "CamcopsRequest") -> Patient: 

2407 """ 

2408 Returns the patient identified by the proquint access key present in this 

2409 request, or raises. 

2410 """ 

2411 _ = req.gettext 

2412 

2413 patient_proquint = get_str_var(req, TabletParam.PATIENT_PROQUINT) 

2414 assert patient_proquint is not None # For type checker 

2415 

2416 try: 

2417 uuid_obj = uuid_from_proquint(patient_proquint) 

2418 except InvalidProquintException: 

2419 # Checksum failed or characters in wrong place 

2420 # We'll do the same validation on the client so in theory 

2421 # should never get here 

2422 fail_user_error( 

2423 _( 

2424 "There is no patient with access key '{access_key}'. " 

2425 "Have you entered the key correctly?" 

2426 ).format(access_key=patient_proquint) 

2427 ) 

2428 

2429 server_device = Device.get_server_device(req.dbsession) 

2430 

2431 # noinspection PyUnboundLocalVariable,PyProtectedMember 

2432 patient = ( 

2433 req.dbsession.query(Patient) 

2434 .filter( 

2435 Patient.uuid == uuid_obj, 

2436 Patient._device_id == server_device.id, 

2437 Patient._era == ERA_NOW, 

2438 Patient._current == True, # noqa: E712 

2439 ) 

2440 .options(joinedload(Patient.task_schedules)) 

2441 .one_or_none() 

2442 ) 

2443 

2444 if patient is None: 

2445 fail_user_error( 

2446 _( 

2447 "There is no patient with access key '{access_key}'. " 

2448 "Have you entered the key correctly?" 

2449 ).format(access_key=patient_proquint) 

2450 ) 

2451 

2452 if not patient.idnums: 

2453 # In theory should never happen. The patient must be created with at 

2454 # least one ID number. We did see this once in testing (possibly when 

2455 # a patient created on a device was registered) 

2456 _ = req.gettext 

2457 fail_server_error(_("Patient has no ID numbers")) 

2458 

2459 return patient 

2460 

2461 

2462def get_or_create_single_user( 

2463 req: "CamcopsRequest", name: str, patient: Patient 

2464) -> Tuple[User, str]: 

2465 """ 

2466 Creates a user for a patient (who's using single-user mode). 

2467 

2468 The user is associated (via its name) with the combination of a client 

2469 device and a patient. (If a device is re-registered to another patient, the 

2470 username will change.) 

2471 

2472 If the username already exists, then since we can't look up the password 

2473 (it's irreversibly encrypted), we will set it afresh. 

2474 

2475 - Why is a user associated with a patient? So we can enforce that the user 

2476 can upload only data relating to that patient. 

2477 

2478 - Why is a user associated with a device? 

2479 

2480 - If it is: then two users (e.g. "Device1-Bob" and "Device2-Bob") can 

2481 independently work with the same patient. This will be highly 

2482 confusing (mainly because it will allow "double" copies of tasks to be 

2483 created, though only by manually entering things twice). 

2484 

2485 - If it isn't (e.g. user "Bob"): then, because registering the patient on 

2486 Device2 will reset the password for the user, registering a new device 

2487 for a patient will "take over" from a previous device. That has some 

2488 potential for data loss if work was in progress (incomplete tasks won't 

2489 be uploadable any more, and re-registering [to fix the password on the 

2490 first device] would delete data). 

2491 

2492 - Since some confusion is better than some data loss, we associate users 

2493 with a device/patient combination. 

2494 

2495 Args: 

2496 req: 

2497 a :class:`camcops_server.cc_modules.cc_request.CamcopsRequest` 

2498 name: 

2499 username 

2500 patient: 

2501 associated :class:`camcops_server.cc_modules.cc_patient.Patient`, 

2502 which also tells us the group in which to place this user 

2503 

2504 Returns: 

2505 tuple: :class:`camcops_server.cc_modules.cc_user.User`, password 

2506 

2507 """ 

2508 dbsession = req.dbsession 

2509 password = random_password() 

2510 group = patient.group 

2511 assert group is not None # for type checker 

2512 

2513 user = User.get_user_by_name(dbsession, name) 

2514 creating_new_user = user is None 

2515 if creating_new_user: 

2516 # Create a fresh user. 

2517 user = User(username=name) 

2518 user.upload_group = group 

2519 user.auto_generated = True 

2520 user.superuser = False # should be redundant! 

2521 # noinspection PyProtectedMember 

2522 user.single_patient_pk = patient._pk 

2523 user.set_password(req, password) 

2524 if creating_new_user: 

2525 dbsession.add(user) 

2526 # As the username is based on a UUID, we're pretty sure another 

2527 # request won't have created the same user, otherwise we'd need 

2528 # to catch IntegrityError 

2529 dbsession.flush() 

2530 

2531 membership = UserGroupMembership(user_id=user.id, group_id=group.id) 

2532 membership.may_register_devices = True 

2533 membership.may_upload = True 

2534 user.user_group_memberships = [membership] # ... only these permissions 

2535 

2536 return user, password 

2537 

2538 

2539def random_password(length: int = 32) -> str: 

2540 """ 

2541 Create a random password. 

2542 """ 

2543 # Not trying anything clever with distributions of letters, digits etc 

2544 characters = string.ascii_letters + string.digits + string.punctuation 

2545 # We use secrets.choice() rather than random.choices() as it's better 

2546 # for security/cryptography purposes. 

2547 return "".join(secrets.choice(characters) for _ in range(length)) 

2548 

2549 

2550def get_task_schedules(req: "CamcopsRequest", patient: Patient) -> str: 

2551 """ 

2552 Gets a JSON string representation of the task schedules for a specified 

2553 patient. 

2554 """ 

2555 dbsession = req.dbsession 

2556 

2557 schedules = [] 

2558 

2559 for pts in patient.task_schedules: 

2560 if pts.start_datetime is None: 

2561 # Minutes granularity so we are consistent with the form 

2562 pts.start_datetime = req.now_utc.replace(second=0, microsecond=0) 

2563 dbsession.add(pts) 

2564 

2565 items = [] 

2566 

2567 for task_info in pts.get_list_of_scheduled_tasks(req): 

2568 due_from = task_info.start_datetime.to_iso8601_string() 

2569 due_by = task_info.end_datetime.to_iso8601_string() 

2570 

2571 complete = False 

2572 when_completed = None 

2573 task = task_info.task 

2574 if task: 

2575 complete = task.is_complete() 

2576 if complete and task.when_last_modified: 

2577 when_completed = ( 

2578 task.when_last_modified.to_iso8601_string() 

2579 ) 

2580 

2581 if pts.settings is not None: 

2582 settings = pts.settings.get(task_info.tablename, {}) 

2583 else: 

2584 settings = {} 

2585 

2586 items.append( 

2587 { 

2588 TabletParam.TABLE: task_info.tablename, 

2589 TabletParam.ANONYMOUS: task_info.is_anonymous, 

2590 TabletParam.SETTINGS: settings, 

2591 TabletParam.DUE_FROM: due_from, 

2592 TabletParam.DUE_BY: due_by, 

2593 TabletParam.COMPLETE: complete, 

2594 TabletParam.WHEN_COMPLETED: when_completed, 

2595 } 

2596 ) 

2597 

2598 schedules.append( 

2599 { 

2600 TabletParam.TASK_SCHEDULE_NAME: pts.task_schedule.name, 

2601 TabletParam.TASK_SCHEDULE_ITEMS: items, 

2602 } 

2603 ) 

2604 

2605 return json.dumps(schedules) 

2606 

2607 

2608# ============================================================================= 

2609# Action processors: allowed to any user 

2610# ============================================================================= 

2611# If they return None, the framework uses the operation name as the reply in 

2612# the success message. Not returning anything is the same as returning None. 

2613# Authentication is performed in advance of these. 

2614 

2615 

2616def op_check_device_registered(req: "CamcopsRequest") -> None: 

2617 """ 

2618 Check that a device is registered, or raise 

2619 :exc:`UserErrorException`. 

2620 """ 

2621 req.tabletsession.ensure_device_registered() 

2622 

2623 

2624def op_register_patient(req: "CamcopsRequest") -> Dict[str, Any]: 

2625 """ 

2626 Registers a patient. That is, the client provides an access key. If all 

2627 is well, the server returns details of that patient, as well as key 

2628 server parameters, plus (if required) the username/password to use. 

2629 """ 

2630 # ------------------------------------------------------------------------- 

2631 # Patient details 

2632 # ------------------------------------------------------------------------- 

2633 patient = get_single_server_patient(req) # may fail/raise 

2634 patient_info = json_patient_info(patient) 

2635 reply_dict = {TabletParam.PATIENT_INFO: patient_info} 

2636 

2637 # ------------------------------------------------------------------------- 

2638 # Username/password 

2639 # ------------------------------------------------------------------------- 

2640 client_device_name = get_str_var(req, TabletParam.DEVICE) 

2641 # noinspection PyProtectedMember 

2642 user_name = make_single_user_mode_username(client_device_name, patient._pk) 

2643 user, password = get_or_create_single_user(req, user_name, patient) 

2644 reply_dict[TabletParam.USER] = user.username 

2645 reply_dict[TabletParam.PASSWORD] = password 

2646 

2647 # ------------------------------------------------------------------------- 

2648 # Intellectual property settings 

2649 # ------------------------------------------------------------------------- 

2650 ip_use = patient.group.ip_use or IpUse() 

2651 # ... if the group doesn't have an associated ip_use object, use defaults 

2652 ip_dict = { 

2653 TabletParam.IP_USE_COMMERCIAL: int(ip_use.commercial), 

2654 TabletParam.IP_USE_CLINICAL: int(ip_use.clinical), 

2655 TabletParam.IP_USE_EDUCATIONAL: int(ip_use.educational), 

2656 TabletParam.IP_USE_RESEARCH: int(ip_use.research), 

2657 } 

2658 reply_dict[TabletParam.IP_USE_INFO] = json.dumps(ip_dict) 

2659 

2660 return reply_dict 

2661 

2662 

2663# ============================================================================= 

2664# Action processors that require REGISTRATION privilege 

2665# ============================================================================= 

2666 

2667 

2668def op_register_device(req: "CamcopsRequest") -> Dict[str, Any]: 

2669 """ 

2670 Register a device with the server. 

2671 

2672 Returns: 

2673 server information dictionary (from :func:`get_server_id_info`) 

2674 """ 

2675 dbsession = req.dbsession 

2676 ts = req.tabletsession 

2677 device_friendly_name = get_str_var( 

2678 req, TabletParam.DEVICE_FRIENDLY_NAME, mandatory=False 

2679 ) 

2680 # noinspection PyUnresolvedReferences 

2681 device_exists = exists_in_table( 

2682 dbsession, Device.__table__, Device.name == ts.device_name 

2683 ) 

2684 if device_exists: 

2685 # device already registered, but accept re-registration 

2686 # noinspection PyUnresolvedReferences 

2687 dbsession.execute( 

2688 update(Device.__table__) 

2689 .where(Device.name == ts.device_name) 

2690 .values( 

2691 friendly_name=device_friendly_name, 

2692 camcops_version=ts.tablet_version_str, 

2693 registered_by_user_id=req.user_id, 

2694 when_registered_utc=req.now_utc, 

2695 ) 

2696 ) 

2697 else: 

2698 # new registration 

2699 try: 

2700 # noinspection PyUnresolvedReferences 

2701 dbsession.execute( 

2702 Device.__table__.insert().values( 

2703 name=ts.device_name, 

2704 friendly_name=device_friendly_name, 

2705 camcops_version=ts.tablet_version_str, 

2706 registered_by_user_id=req.user_id, 

2707 when_registered_utc=req.now_utc, 

2708 ) 

2709 ) 

2710 except IntegrityError: 

2711 fail_user_error(INSERT_FAILED) 

2712 

2713 ts.reload_device() 

2714 audit( 

2715 req, 

2716 f"register, device_id={ts.device_id}, " 

2717 f"friendly_name={device_friendly_name}", 

2718 tablename=Device.__tablename__, 

2719 ) 

2720 return get_server_id_info(req) 

2721 

2722 

2723def op_get_extra_strings(req: "CamcopsRequest") -> Dict[str, str]: 

2724 """ 

2725 Fetch all local extra strings from the server. 

2726 

2727 Returns: 

2728 a SELECT-style reply (see :func:`get_select_reply`) for the 

2729 extra-string table 

2730 """ 

2731 fields = [ 

2732 ExtraStringFieldNames.TASK, 

2733 ExtraStringFieldNames.NAME, 

2734 ExtraStringFieldNames.LANGUAGE, 

2735 ExtraStringFieldNames.VALUE, 

2736 ] 

2737 rows = req.get_all_extra_strings() 

2738 reply = get_select_reply(fields, rows) 

2739 audit(req, "get_extra_strings") 

2740 return reply 

2741 

2742 

2743# noinspection PyUnusedLocal 

2744def op_get_allowed_tables(req: "CamcopsRequest") -> Dict[str, str]: 

2745 """ 

2746 Returns the names of all possible tables on the server, each paired with 

2747 the minimum client (tablet) version that will be accepted for each table. 

2748 (Typically these are all the same as the minimum global tablet version.) 

2749 

2750 Uses the SELECT-like syntax (see :func:`get_select_reply`). 

2751 """ 

2752 tables_versions = all_tables_with_min_client_version() 

2753 fields = [ 

2754 AllowedTablesFieldNames.TABLENAME, 

2755 AllowedTablesFieldNames.MIN_CLIENT_VERSION, 

2756 ] 

2757 rows = [[k, str(v)] for k, v in tables_versions.items()] 

2758 reply = get_select_reply(fields, rows) 

2759 audit(req, "get_allowed_tables") 

2760 return reply 

2761 

2762 

2763def op_get_task_schedules(req: "CamcopsRequest") -> Dict[str, str]: 

2764 """ 

2765 Return details of the task schedules for the patient associated with 

2766 this request, for single-user mode. Also returns details of the single 

2767 patient, in case that's changed. 

2768 """ 

2769 patient = get_single_server_patient(req) 

2770 patient_info = json_patient_info(patient) 

2771 task_schedules = get_task_schedules(req, patient) 

2772 return { 

2773 TabletParam.PATIENT_INFO: patient_info, 

2774 TabletParam.TASK_SCHEDULES: task_schedules, 

2775 } 

2776 

2777 

2778# ============================================================================= 

2779# Action processors that require UPLOAD privilege 

2780# ============================================================================= 

2781 

2782# noinspection PyUnusedLocal 

2783def op_check_upload_user_and_device(req: "CamcopsRequest") -> None: 

2784 """ 

2785 Stub function for the operation to check that a user is valid. 

2786 

2787 To get this far, the user has to be valid, so this function doesn't 

2788 actually have to do anything. 

2789 """ 

2790 pass # don't need to do anything! 

2791 

2792 

2793# noinspection PyUnusedLocal 

2794def op_get_id_info(req: "CamcopsRequest") -> Dict[str, Any]: 

2795 """ 

2796 Fetch server ID information; see :func:`get_server_id_info`. 

2797 """ 

2798 return get_server_id_info(req) 

2799 

2800 

2801def op_start_upload(req: "CamcopsRequest") -> None: 

2802 """ 

2803 Begin an upload. 

2804 """ 

2805 start_device_upload_batch(req) 

2806 

2807 

2808def op_end_upload(req: "CamcopsRequest") -> None: 

2809 """ 

2810 Ends an upload and commits changes. 

2811 """ 

2812 batchdetails = get_batch_details(req) 

2813 # ensure it's the same user finishing as starting! 

2814 end_device_upload_batch(req, batchdetails) 

2815 

2816 

2817def op_upload_table(req: "CamcopsRequest") -> str: 

2818 """ 

2819 Upload a table. 

2820 

2821 Incoming information in the POST request includes a CSV list of fields, a 

2822 count of the number of records being provided, and a set of variables named 

2823 ``record0`` ... ``record{nrecords - 1}``, each containing a CSV list of 

2824 SQL-encoded values. 

2825 

2826 Typically used for smaller tables, i.e. most except for BLOBs. 

2827 """ 

2828 table = get_table_from_req(req, TabletParam.TABLE) 

2829 

2830 allowed_nonexistent_fields = [] # type: List[str] 

2831 # noinspection PyUnresolvedReferences 

2832 if req.tabletsession.cope_with_old_idnums and table == Patient.__table__: 

2833 for x in range(1, NUMBER_OF_IDNUMS_DEFUNCT + 1): 

2834 allowed_nonexistent_fields.extend( 

2835 [ 

2836 FP_ID_NUM + str(x), 

2837 FP_ID_DESC + str(x), 

2838 FP_ID_SHORT_DESC + str(x), 

2839 ] 

2840 ) 

2841 

2842 fields = get_fields_from_post_var( 

2843 req, 

2844 table, 

2845 TabletParam.FIELDS, 

2846 allowed_nonexistent_fields=allowed_nonexistent_fields, 

2847 ) 

2848 nrecords = get_int_var(req, TabletParam.NRECORDS) 

2849 

2850 nfields = len(fields) 

2851 if nfields < 1: 

2852 fail_user_error( 

2853 f"{TabletParam.FIELDS}={nfields}: can't be less than 1" 

2854 ) 

2855 if nrecords < 0: 

2856 fail_user_error( 

2857 f"{TabletParam.NRECORDS}={nrecords}: can't be less than 0" 

2858 ) 

2859 

2860 batchdetails = get_batch_details(req) 

2861 

2862 ts = req.tabletsession 

2863 if ts.explicit_pkname_for_upload_table: # q.v. 

2864 # New client: tells us the PK name explicitly. 

2865 clientpk_name = get_single_field_from_post_var( 

2866 req, table, TabletParam.PKNAME 

2867 ) 

2868 else: 

2869 # Old client. Either (a) old Titanium client, in which the client PK 

2870 # is in fields[0], or (b) an early C++ client, in which there was no 

2871 # guaranteed order (and no explicit PK name was sent). However, in 

2872 # either case, the client PK name was (is) always "id". 

2873 clientpk_name = TABLET_ID_FIELD 

2874 ensure_valid_field_name(table, clientpk_name) 

2875 server_pks_uploaded = [] # type: List[int] 

2876 n_new = 0 

2877 n_modified = 0 

2878 n_identical = 0 

2879 dirty = False 

2880 serverrecs = get_server_live_records( 

2881 req, 

2882 ts.device_id, 

2883 table, 

2884 clientpk_name=clientpk_name, 

2885 current_only=True, 

2886 ) 

2887 for r in range(nrecords): 

2888 recname = TabletParam.RECORD_PREFIX + str(r) 

2889 values = get_values_from_post_var(req, recname) 

2890 nvalues = len(values) 

2891 if nvalues != nfields: 

2892 errmsg = ( 

2893 f"Number of fields in field list ({nfields}) doesn't match " 

2894 f"number of values in record {r} ({nvalues})" 

2895 ) 

2896 log.warning(errmsg + f"\nfields: {fields!r}\nvalues: {values!r}") 

2897 fail_user_error(errmsg) 

2898 valuedict = dict(zip(fields, values)) 

2899 # log.debug("table {!r}, record {}: {!r}", table.name, r, valuedict) 

2900 # CORE: CALLS upload_record_core 

2901 urr = upload_record_core( 

2902 req, 

2903 batchdetails, 

2904 table, 

2905 clientpk_name, 

2906 valuedict, 

2907 server_live_current_records=serverrecs, 

2908 ) 

2909 if urr.oldserverpk is not None: # was an existing record 

2910 server_pks_uploaded.append(urr.oldserverpk) 

2911 if urr.newserverpk is None: 

2912 n_identical += 1 

2913 else: 

2914 n_modified += 1 

2915 else: # entirely new 

2916 n_new += 1 

2917 if urr.dirty: 

2918 dirty = True 

2919 

2920 # Now deal with any ABSENT (not in uploaded data set) conditions. 

2921 server_pks_for_deletion = [ 

2922 r.server_pk 

2923 for r in serverrecs 

2924 if r.server_pk not in server_pks_uploaded 

2925 ] 

2926 # Note that "deletion" means "end of the line"; records that are modified 

2927 # and replaced were handled by upload_record_core(). 

2928 n_deleted = len(server_pks_for_deletion) 

2929 if n_deleted > 0: 

2930 flag_deleted(req, batchdetails, table, server_pks_for_deletion) 

2931 

2932 # Set dirty/clean status 

2933 if ( 

2934 dirty 

2935 or n_new > 0 

2936 or n_modified > 0 

2937 or n_deleted > 0 

2938 or any(sr.move_off_tablet for sr in serverrecs) 

2939 ): 

2940 # ... checks on n_new and n_modified are redundant; dirty will be True 

2941 mark_table_dirty(req, table) 

2942 elif batchdetails.preserving and not serverrecs: 

2943 # We've scanned this table, and there would be no work to do to 

2944 # preserve records from previous uploads. 

2945 mark_table_clean(req, table) 

2946 

2947 # Special for old tablets: 

2948 # noinspection PyUnresolvedReferences 

2949 if req.tabletsession.cope_with_old_idnums and table == Patient.__table__: 

2950 # noinspection PyUnresolvedReferences 

2951 mark_table_dirty(req, PatientIdNum.__table__) 

2952 # Mark patient ID numbers for deletion if their parent Patient is 

2953 # similarly being marked for deletion 

2954 # noinspection PyUnresolvedReferences,PyProtectedMember 

2955 req.dbsession.execute( 

2956 update(PatientIdNum.__table__) 

2957 .where(PatientIdNum._device_id == Patient._device_id) 

2958 .where(PatientIdNum._era == ERA_NOW) 

2959 .where(PatientIdNum.patient_id == Patient.id) 

2960 .where(Patient._pk.in_(server_pks_for_deletion)) 

2961 .where(Patient._era == ERA_NOW) # shouldn't be in doubt! 

2962 .values(_removal_pending=1, _successor_pk=None) 

2963 ) 

2964 

2965 # Auditing occurs at commit_all. 

2966 log.info( 

2967 "Upload successful; {n} records uploaded to table {t} " 

2968 "({new} new, {mod} modified, {i} identical, {nd} deleted)", 

2969 n=nrecords, 

2970 t=table.name, 

2971 new=n_new, 

2972 mod=n_modified, 

2973 i=n_identical, 

2974 nd=n_deleted, 

2975 ) 

2976 return f"Table {table.name} upload successful" 

2977 

2978 

2979def op_upload_record(req: "CamcopsRequest") -> str: 

2980 """ 

2981 Upload an individual record. (Typically used for BLOBs.) 

2982 Incoming POST information includes a CSV list of fields and a CSV list of 

2983 values. 

2984 """ 

2985 batchdetails = get_batch_details(req) 

2986 table = get_table_from_req(req, TabletParam.TABLE) 

2987 clientpk_name = get_single_field_from_post_var( 

2988 req, table, TabletParam.PKNAME 

2989 ) 

2990 valuedict = get_fields_and_values( 

2991 req, table, TabletParam.FIELDS, TabletParam.VALUES 

2992 ) 

2993 urr = upload_record_core( 

2994 req, batchdetails, table, clientpk_name, valuedict 

2995 ) 

2996 if urr.dirty: 

2997 mark_table_dirty(req, table) 

2998 if urr.oldserverpk is None: 

2999 log.info("upload-insert") 

3000 return "UPLOAD-INSERT" 

3001 else: 

3002 if urr.newserverpk is None: 

3003 log.info("upload-update: skipping existing record") 

3004 else: 

3005 log.info("upload-update") 

3006 return "UPLOAD-UPDATE" 

3007 # Auditing occurs at commit_all. 

3008 

3009 

3010def op_upload_empty_tables(req: "CamcopsRequest") -> str: 

3011 """ 

3012 The tablet supplies a list of tables that are empty at its end, and we 

3013 will 'wipe' all appropriate tables; this reduces the number of HTTP 

3014 requests. 

3015 """ 

3016 tables = get_tables_from_post_var(req, TabletParam.TABLES) 

3017 batchdetails = get_batch_details(req) 

3018 to_dirty = [] # type: List[Table] 

3019 to_clean = [] # type: List[Table] 

3020 for table in tables: 

3021 nrows_affected = flag_all_records_deleted(req, table) 

3022 if nrows_affected > 0: 

3023 to_dirty.append(table) 

3024 elif batchdetails.preserving: 

3025 # There are no records in the current era for this device. 

3026 to_clean.append(table) 

3027 # In the fewest number of queries: 

3028 mark_tables_dirty(req, to_dirty) 

3029 mark_tables_clean(req, to_clean) 

3030 log.info("upload_empty_tables") 

3031 # Auditing occurs at commit_all. 

3032 return "UPLOAD-EMPTY-TABLES" 

3033 

3034 

3035def op_start_preservation(req: "CamcopsRequest") -> str: 

3036 """ 

3037 Marks this upload batch as one in which all records will be preserved 

3038 (i.e. moved from NOW-era to an older era, so they can be deleted safely 

3039 from the tablet). 

3040 

3041 Without this, individual records can still be marked for preservation if 

3042 their MOVE_OFF_TABLET_FIELD field (``_move_off_tablet``) is set; see 

3043 :func:`upload_record` and the functions it calls. 

3044 """ 

3045 get_batch_details(req) 

3046 start_preserving(req) 

3047 log.info("start_preservation successful") 

3048 # Auditing occurs at commit_all. 

3049 return "STARTPRESERVATION" 

3050 

3051 

3052def op_delete_where_key_not(req: "CamcopsRequest") -> str: 

3053 """ 

3054 Marks records for deletion, for a device/table, where the client PK 

3055 is not in a specified list. 

3056 """ 

3057 table = get_table_from_req(req, TabletParam.TABLE) 

3058 clientpk_name = get_single_field_from_post_var( 

3059 req, table, TabletParam.PKNAME 

3060 ) 

3061 clientpk_values = get_values_from_post_var(req, TabletParam.PKVALUES) 

3062 

3063 get_batch_details(req) 

3064 flag_deleted_where_clientpk_not(req, table, clientpk_name, clientpk_values) 

3065 # Auditing occurs at commit_all. 

3066 # log.info("delete_where_key_not successful; table {} trimmed", table) 

3067 return "Trimmed" 

3068 

3069 

3070def op_which_keys_to_send(req: "CamcopsRequest") -> str: 

3071 """ 

3072 Intended use: "For my device, and a specified table, here are my client- 

3073 side PKs (as a CSV list), and the modification dates for each corresponding 

3074 record (as a CSV list). Please tell me which records have mismatching dates 

3075 on the server, i.e. those that I need to re-upload." 

3076 

3077 Used particularly for BLOBs, to reduce traffic, i.e. so we don't have to 

3078 send a lot of BLOBs. 

3079 

3080 Note new ``TabletParam.MOVE_OFF_TABLET_VALUES`` parameter in server v2.3.0, 

3081 with bugfix for pre-2.3.0 clients that won't send this; see changelog. 

3082 """ 

3083 # ------------------------------------------------------------------------- 

3084 # Get details 

3085 # ------------------------------------------------------------------------- 

3086 try: 

3087 table = get_table_from_req(req, TabletParam.TABLE) 

3088 except IgnoringAntiqueTableException: 

3089 raise IgnoringAntiqueTableException("") 

3090 clientpk_name = get_single_field_from_post_var( 

3091 req, table, TabletParam.PKNAME 

3092 ) 

3093 clientpk_values = get_values_from_post_var( 

3094 req, TabletParam.PKVALUES, mandatory=False 

3095 ) 

3096 # ... should be autoconverted to int, but we check below 

3097 client_dates = get_values_from_post_var( 

3098 req, TabletParam.DATEVALUES, mandatory=False 

3099 ) 

3100 # ... will be in string format 

3101 

3102 npkvalues = len(clientpk_values) 

3103 ndatevalues = len(client_dates) 

3104 if npkvalues != ndatevalues: 

3105 fail_user_error( 

3106 f"Number of PK values ({npkvalues}) doesn't match number of dates " 

3107 f"({ndatevalues})" 

3108 ) 

3109 

3110 # v2.3.0: 

3111 move_off_tablet_values = [] # type: List[int] # for type checker 

3112 if req.has_param(TabletParam.MOVE_OFF_TABLET_VALUES): 

3113 client_reports_move_off_tablet = True 

3114 move_off_tablet_values = get_values_from_post_var( 

3115 req, TabletParam.MOVE_OFF_TABLET_VALUES, mandatory=True 

3116 ) 

3117 # ... should be autoconverted to int 

3118 n_motv = len(move_off_tablet_values) 

3119 if n_motv != npkvalues: 

3120 fail_user_error( 

3121 f"Number of move-off-tablet values ({n_motv}) doesn't match " 

3122 f"number of PKs ({npkvalues})" 

3123 ) 

3124 try: 

3125 move_off_tablet_values = [bool(x) for x in move_off_tablet_values] 

3126 except (TypeError, ValueError): 

3127 fail_user_error( 

3128 f"Bad move-off-tablet values: {move_off_tablet_values!r}" 

3129 ) 

3130 else: 

3131 client_reports_move_off_tablet = False 

3132 log.warning( 

3133 "op_which_keys_to_send: old client not reporting " 

3134 "{}; requesting all records", 

3135 TabletParam.MOVE_OFF_TABLET_VALUES, 

3136 ) 

3137 

3138 clientinfo = [] # type: List[WhichKeyToSendInfo] 

3139 

3140 for i in range(npkvalues): 

3141 cpkv = clientpk_values[i] 

3142 if not isinstance(cpkv, int): 

3143 fail_user_error(f"Bad (non-integer) client PK: {cpkv!r}") 

3144 dt = None # for type checker 

3145 try: 

3146 dt = coerce_to_pendulum(client_dates[i]) 

3147 if dt is None: 

3148 fail_user_error(f"Missing date/time for client PK {cpkv}") 

3149 except ValueError: 

3150 fail_user_error(f"Bad date/time: {client_dates[i]!r}") 

3151 clientinfo.append( 

3152 WhichKeyToSendInfo( 

3153 client_pk=cpkv, 

3154 client_when=dt, 

3155 client_move_off_tablet=( 

3156 move_off_tablet_values[i] 

3157 if client_reports_move_off_tablet 

3158 else False 

3159 ), 

3160 ) 

3161 ) 

3162 

3163 # ------------------------------------------------------------------------- 

3164 # Work out the answer 

3165 # ------------------------------------------------------------------------- 

3166 batchdetails = get_batch_details(req) 

3167 

3168 # 1. The client sends us all its PKs. So "delete" anything not in that 

3169 # list. 

3170 flag_deleted_where_clientpk_not(req, table, clientpk_name, clientpk_values) 

3171 

3172 # 2. See which ones are new or updates. 

3173 client_pks_needed = [] # type: List[int] 

3174 client_pk_to_serverrec = client_pks_that_exist( 

3175 req, table, clientpk_name, clientpk_values 

3176 ) 

3177 for wk in clientinfo: 

3178 if client_reports_move_off_tablet: 

3179 if wk.client_pk not in client_pk_to_serverrec: 

3180 # New on the client; we want it 

3181 client_pks_needed.append(wk.client_pk) 

3182 else: 

3183 # We know about some version of this client record. 

3184 serverrec = client_pk_to_serverrec[wk.client_pk] 

3185 if serverrec.server_when != wk.client_when: 

3186 # Modified on the client; we want it 

3187 client_pks_needed.append(wk.client_pk) 

3188 elif serverrec.move_off_tablet != wk.client_move_off_tablet: 

3189 # Not modified on the client. But it is being preserved. 

3190 # We don't need to ask the client for it again, but we do 

3191 # need to mark the preservation. 

3192 flag_record_for_preservation( 

3193 req, batchdetails, table, serverrec.server_pk 

3194 ) 

3195 

3196 else: 

3197 # Client hasn't told us about the _move_off_tablet flag. Always 

3198 # request the record (workaround potential bug in old clients). 

3199 client_pks_needed.append(wk.client_pk) 

3200 

3201 # Success 

3202 pk_csv_list = ",".join( 

3203 [str(x) for x in client_pks_needed if x is not None] 

3204 ) 

3205 # log.info("which_keys_to_send successful: table {}", table.name) 

3206 return pk_csv_list 

3207 

3208 

3209def op_validate_patients(req: "CamcopsRequest") -> str: 

3210 """ 

3211 As of v2.3.0, the client can use this command to validate patients against 

3212 arbitrary server criteria -- definitely the upload/finalize ID policies, 

3213 but potentially also other criteria of the server's (like matching against 

3214 a bank of predefined patients). 

3215 

3216 Compare ``NetworkManager::getPatientInfoJson()`` on the client. 

3217 

3218 There is a slight weakness with respect to "single-patient" users, in that 

3219 the client *asks* if the patients are OK (rather than the server 

3220 *enforcing* that they are OK, via hooks into :func:`op_upload_table`, 

3221 :func:`op_upload_record`, :func:`op_upload_entire_database` -- made more 

3222 complex because ID numbers are not uploaded to the same table...). In 

3223 principle, the weakness is that a user could (a) crack their assigned 

3224 password and (b) rework the CamCOPS client, in order to upload "bad" 

3225 patient data into their assigned group. 

3226 

3227 todo: 

3228 address this by having the server *require* patient validation for 

3229 all uploads? 

3230 

3231 """ 

3232 pt_json_list = get_json_from_post_var( 

3233 req, 

3234 TabletParam.PATIENT_INFO, 

3235 decoder=PATIENT_INFO_JSON_DECODER, 

3236 mandatory=True, 

3237 ) 

3238 if not isinstance(pt_json_list, list): 

3239 fail_user_error("Top-level JSON is not a list") 

3240 group = Group.get_group_by_id(req.dbsession, req.user.upload_group_id) 

3241 for pt_dict in pt_json_list: 

3242 ensure_valid_patient_json(req, group, pt_dict) 

3243 return SUCCESS_MSG 

3244 

3245 

3246def op_upload_entire_database(req: "CamcopsRequest") -> str: 

3247 """ 

3248 Perform a one-step upload of the entire database. 

3249 

3250 - From v2.3.0. 

3251 - Therefore, we do not have to cope with old-style ID numbers. 

3252 """ 

3253 # Roll back and clear any outstanding changes 

3254 clear_device_upload_batch(req) 

3255 

3256 # Fetch the JSON, with sanity checks 

3257 preserving = get_bool_int_var(req, TabletParam.FINALIZING) 

3258 pknameinfo = get_json_from_post_var( 

3259 req, TabletParam.PKNAMEINFO, decoder=DB_JSON_DECODER, mandatory=True 

3260 ) 

3261 if not isinstance(pknameinfo, dict): 

3262 fail_user_error("PK name info JSON is not a dict") 

3263 dbdata = get_json_from_post_var( 

3264 req, TabletParam.DBDATA, decoder=DB_JSON_DECODER, mandatory=True 

3265 ) 

3266 if not isinstance(dbdata, dict): 

3267 fail_user_error("Database data JSON is not a dict") 

3268 

3269 # Sanity checks 

3270 dbdata_tablenames = sorted(dbdata.keys()) 

3271 pkinfo_tablenames = sorted(pknameinfo.keys()) 

3272 if pkinfo_tablenames != dbdata_tablenames: 

3273 fail_user_error("Table names don't match from (1) DB data (2) PK info") 

3274 duff_tablenames = sorted( 

3275 list(set(dbdata_tablenames) - set(CLIENT_TABLE_MAP.keys())) 

3276 ) 

3277 if duff_tablenames: 

3278 fail_user_error( 

3279 f"Attempt to upload nonexistent tables: {duff_tablenames!r}" 

3280 ) 

3281 

3282 # Perform the upload 

3283 batchdetails = BatchDetails( 

3284 req.now_utc, preserving=preserving, onestep=True 

3285 ) # NB special "onestep" option 

3286 # Process the tables in a certain order: 

3287 tables = sorted(CLIENT_TABLE_MAP.values(), key=upload_commit_order_sorter) 

3288 changelist = [] # type: List[UploadTableChanges] 

3289 for table in tables: 

3290 clientpk_name = pknameinfo.get(table.name, "") 

3291 rows = dbdata.get(table.name, []) 

3292 tablechanges = process_table_for_onestep_upload( 

3293 req, batchdetails, table, clientpk_name, rows 

3294 ) 

3295 changelist.append(tablechanges) 

3296 

3297 # Audit 

3298 audit_upload(req, changelist) 

3299 

3300 # Done 

3301 return SUCCESS_MSG 

3302 

3303 

3304# ============================================================================= 

3305# Action maps 

3306# ============================================================================= 

3307 

3308 

3309class Operations: 

3310 """ 

3311 Constants giving the name of operations (commands) accepted by this API. 

3312 """ 

3313 

3314 CHECK_DEVICE_REGISTERED = "check_device_registered" 

3315 CHECK_UPLOAD_USER_DEVICE = "check_upload_user_and_device" 

3316 DELETE_WHERE_KEY_NOT = "delete_where_key_not" 

3317 END_UPLOAD = "end_upload" 

3318 GET_ALLOWED_TABLES = "get_allowed_tables" # v2.2.0 

3319 GET_EXTRA_STRINGS = "get_extra_strings" 

3320 GET_ID_INFO = "get_id_info" 

3321 GET_TASK_SCHEDULES = "get_task_schedules" # v2.4.0 

3322 REGISTER = "register" 

3323 REGISTER_PATIENT = "register_patient" # v2.4.0 

3324 START_PRESERVATION = "start_preservation" 

3325 START_UPLOAD = "start_upload" 

3326 UPLOAD_EMPTY_TABLES = "upload_empty_tables" 

3327 UPLOAD_ENTIRE_DATABASE = "upload_entire_database" # v2.3.0 

3328 UPLOAD_RECORD = "upload_record" 

3329 UPLOAD_TABLE = "upload_table" 

3330 VALIDATE_PATIENTS = "validate_patients" # v2.3.0 

3331 WHICH_KEYS_TO_SEND = "which_keys_to_send" 

3332 

3333 

3334OPERATIONS_ANYONE = { 

3335 Operations.CHECK_DEVICE_REGISTERED: op_check_device_registered, 

3336 # Anyone can register a patient provided they have the right unique code 

3337 Operations.REGISTER_PATIENT: op_register_patient, 

3338} 

3339OPERATIONS_REGISTRATION = { 

3340 Operations.GET_ALLOWED_TABLES: op_get_allowed_tables, # v2.2.0 

3341 Operations.GET_EXTRA_STRINGS: op_get_extra_strings, 

3342 Operations.GET_TASK_SCHEDULES: op_get_task_schedules, 

3343 Operations.REGISTER: op_register_device, 

3344} 

3345OPERATIONS_UPLOAD = { 

3346 Operations.CHECK_UPLOAD_USER_DEVICE: op_check_upload_user_and_device, 

3347 Operations.DELETE_WHERE_KEY_NOT: op_delete_where_key_not, 

3348 Operations.END_UPLOAD: op_end_upload, 

3349 Operations.GET_ID_INFO: op_get_id_info, 

3350 Operations.START_PRESERVATION: op_start_preservation, 

3351 Operations.START_UPLOAD: op_start_upload, 

3352 Operations.UPLOAD_EMPTY_TABLES: op_upload_empty_tables, 

3353 Operations.UPLOAD_ENTIRE_DATABASE: op_upload_entire_database, 

3354 Operations.UPLOAD_RECORD: op_upload_record, 

3355 Operations.UPLOAD_TABLE: op_upload_table, 

3356 Operations.VALIDATE_PATIENTS: op_validate_patients, # v2.3.0 

3357 Operations.WHICH_KEYS_TO_SEND: op_which_keys_to_send, 

3358} 

3359 

3360 

3361# ============================================================================= 

3362# Client API main functions 

3363# ============================================================================= 

3364 

3365 

3366def main_client_api(req: "CamcopsRequest") -> Dict[str, str]: 

3367 """ 

3368 Main HTTP processor. 

3369 

3370 For success, returns a dictionary to send (will use status '200 OK') 

3371 For failure, raises an exception. 

3372 """ 

3373 # log.info("CamCOPS database script starting at {}", 

3374 # format_datetime(req.now, DateFormat.ISO8601)) 

3375 ts = req.tabletsession 

3376 fn = None 

3377 

3378 if ts.operation in OPERATIONS_ANYONE: 

3379 fn = OPERATIONS_ANYONE.get(ts.operation) 

3380 

3381 elif ts.operation in OPERATIONS_REGISTRATION: 

3382 ts.ensure_valid_user_for_device_registration() 

3383 fn = OPERATIONS_REGISTRATION.get(ts.operation) 

3384 

3385 elif ts.operation in OPERATIONS_UPLOAD: 

3386 ts.ensure_valid_device_and_user_for_uploading() 

3387 fn = OPERATIONS_UPLOAD.get(ts.operation) 

3388 

3389 if not fn: 

3390 fail_unsupported_operation(ts.operation) 

3391 result = fn(req) 

3392 if result is None: 

3393 # generic success 

3394 result = {TabletParam.RESULT: ts.operation} 

3395 elif not isinstance(result, dict): 

3396 # convert strings (etc.) to a dictionary 

3397 result = {TabletParam.RESULT: result} 

3398 return result 

3399 

3400 

3401@view_config( 

3402 route_name=Routes.CLIENT_API, 

3403 request_method=HttpMethod.POST, 

3404 permission=NO_PERMISSION_REQUIRED, 

3405) 

3406@view_config( 

3407 route_name=Routes.CLIENT_API_ALIAS, 

3408 request_method=HttpMethod.POST, 

3409 permission=NO_PERMISSION_REQUIRED, 

3410) 

3411def client_api(req: "CamcopsRequest") -> Response: 

3412 """ 

3413 View for client API. All tablet interaction comes through here. 

3414 Wraps :func:`main_client_api`. 

3415 

3416 Internally, replies are managed as dictionaries. 

3417 For the final reply, the dictionary is converted to text in this format: 

3418 

3419 .. code-block:: none 

3420 

3421 k1:v1 

3422 k2:v2 

3423 k3:v3 

3424 ... 

3425 """ 

3426 # log.debug("{!r}", req.environ) 

3427 # log.debug("{!r}", req.params) 

3428 t0 = time.time() # in seconds 

3429 

3430 try: 

3431 resultdict = main_client_api(req) 

3432 resultdict[TabletParam.SUCCESS] = SUCCESS_CODE 

3433 status = "200 OK" 

3434 

3435 except IgnoringAntiqueTableException as e: 

3436 log.warning(IGNORING_ANTIQUE_TABLE_MESSAGE) 

3437 resultdict = { 

3438 TabletParam.RESULT: escape_newlines(str(e)), 

3439 TabletParam.SUCCESS: SUCCESS_CODE, 

3440 } 

3441 status = "200 OK" 

3442 

3443 except UserErrorException as e: 

3444 log.warning("CLIENT-SIDE SCRIPT ERROR: {}", e) 

3445 resultdict = { 

3446 TabletParam.SUCCESS: FAILURE_CODE, 

3447 TabletParam.ERROR: escape_newlines(str(e)), 

3448 } 

3449 status = "200 OK" 

3450 

3451 except ServerErrorException as e: 

3452 log.error("SERVER-SIDE SCRIPT ERROR: {}", e) 

3453 # rollback? Not sure 

3454 resultdict = { 

3455 TabletParam.SUCCESS: FAILURE_CODE, 

3456 TabletParam.ERROR: escape_newlines(str(e)), 

3457 } 

3458 status = "503 Database Unavailable: " + str(e) 

3459 

3460 except Exception as e: 

3461 # All other exceptions. May include database write failures. 

3462 # Let's return with status '200 OK'; though this seems dumb, it means 

3463 # the tablet user will at least see the message. 

3464 log.exception("Unhandled exception") # + traceback.format_exc() 

3465 resultdict = { 

3466 TabletParam.SUCCESS: FAILURE_CODE, 

3467 TabletParam.ERROR: escape_newlines(exception_description(e)), 

3468 } 

3469 status = "200 OK" 

3470 

3471 # Add session token information 

3472 ts = req.tabletsession 

3473 resultdict[TabletParam.SESSION_ID] = ts.session_id 

3474 resultdict[TabletParam.SESSION_TOKEN] = ts.session_token 

3475 

3476 # Convert dictionary to text in name-value pair format 

3477 txt = "".join(f"{k}:{v}\n" for k, v in resultdict.items()) 

3478 

3479 t1 = time.time() 

3480 log.debug("Time in script (s): {t}", t=t1 - t0) 

3481 

3482 return TextResponse(txt, status=status)