Hide keyboard shortcuts

Hot-keys on this page

r m x p   toggle line displays

j k   next/prev highlighted chunk

0   (zero) top of page

1   (one) first highlighted chunk

1#!/usr/bin/env python 

2 

3""" 

4camcops_server/cc_modules/client_api.py 

5 

6=============================================================================== 

7 

8 Copyright (C) 2012-2020 Rudolf Cardinal (rudolf@pobox.com). 

9 

10 This file is part of CamCOPS. 

11 

12 CamCOPS is free software: you can redistribute it and/or modify 

13 it under the terms of the GNU General Public License as published by 

14 the Free Software Foundation, either version 3 of the License, or 

15 (at your option) any later version. 

16 

17 CamCOPS is distributed in the hope that it will be useful, 

18 but WITHOUT ANY WARRANTY; without even the implied warranty of 

19 MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the 

20 GNU General Public License for more details. 

21 

22 You should have received a copy of the GNU General Public License 

23 along with CamCOPS. If not, see <https://www.gnu.org/licenses/>. 

24 

25=============================================================================== 

26 

27**Implements the API through which client devices (tablets etc.) upload and 

28download data.** 

29 

30We use primarily SQLAlchemy Core here (in contrast to the ORM used elsewhere). 

31 

32This code is optimized to a degree for speed over clarity, aiming primarily to 

33reduce the number of database hits. 

34 

35**The overall upload method is as follows** 

36 

37Everything that follows refers to records relating to a specific client device 

38in the "current" era, only. 

39 

40In the preamble, the client: 

41 

42- verifies authorization via :func:`op_check_device_registered` and 

43 :func:`op_check_upload_user_and_device`; 

44- fetches and checks server ID information via :func:`op_get_id_info`; 

45- checks its patients are acceptable via :func:`op_validate_patients`; 

46- checks which tables are permitted via :func:`op_get_allowed_tables`; 

47- performs some internal validity checks. 

48 

49Then, in the usual stepwise upload: 

50 

51- :func:`op_start_upload` 

52 

53 - Rolls back any previous incomplete changes via :func:`rollback_all`. 

54 - Creates an upload batch, via :func:`get_batch_details_start_if_needed`. 

55 

56- If were are in a preserving/finalizing upload: :func:`op_start_preservation`. 

57 

58 - Marks all tables as dirty. 

59 - Marks the upload batch as a "preserving" batch. 

60 

61- Then call some or all of: 

62 

63 - For tables that are empty on the client, :func:`op_upload_empty_tables`. 

64 

65 - Current client records are marked as ``_removal_pending``. 

66 - Any table that had previous client records is marked as dirty. 

67 - If preserving, any table without current records is marked as clean. 

68 

69 - For tables that the client wishes to send in one go, 

70 :func:`op_upload_table`. 

71 

72 - Find current server records. 

73 - Use :func:`upload_record_core` to add new records and modify existing 

74 ones, and :func:`flag_deleted` to delete ones that weren't on the client. 

75 - If any records are new, modified, or deleted, mark the table as dirty. 

76 - If preserving and there were no server records in this table, mark the 

77 table as clean. 

78 

79 - For tables (e.g. BLOBs) that might be too big to send in one go: 

80 

81 - client sends PKs to :func:`op_delete_where_key_not`, which "deletes" all 

82 other records, via :func:`flag_deleted_where_clientpk_not`. 

83 - client sends PK and timestamp values to :func:`op_which_keys_to_send` 

84 - server "deletes" records that are not in the list (via 

85 :func:`flag_deleted_where_clientpk_not`, which marks the table as dirty 

86 if any records were thus modified). Note REDUNDANCY here re 

87 :func:`op_delete_where_key_not`. 

88 - server tells the client which records are new or need to be updated 

89 - client sends each of those via :func:`op_upload_record` 

90 

91 - Calls :func`upload_record_core`. 

92 - Marks the table as dirty, unless the client erroneously sent an 

93 unchanged record. 

94 

95- In addition, specific records can be marked as ``_move_off_tablet``. 

96 

97 - :func:`upload_record_core` checks this for otherwise "identical" records 

98 and applies that flag to the server. 

99 

100- When the client's finished, it calls :func:`op_end_upload`. 

101 

102 - Calls :func:`commit_all`; 

103 - ... which, for all dirty tables, calls :func:`commit_table`; 

104 - ... which executes the "add", "remove", and "preserve" functions for the 

105 table; 

106 - ... and triggers the updating of special server indexes on patient ID 

107 numbers and tasks, via :func:`update_indexes`. 

108 - At the end, :func:`commit_all` clears the dirty-table list. 

109 

110There's a little bit of special code to handle old tablet software, too. 

111 

112As of v2.3.0, the function :func:`op_upload_entire_database` does this in one 

113step (faster; for use if the network packets are not excessively large). 

114 

115- Code relating to this uses ``batchdetails.onestep``. 

116 

117**Setup for the upload code** 

118 

119- Fire up a CamCOPS client with an empty database, e.g. from the build 

120 directory via 

121 

122 .. code-block:: bash 

123 

124 ./camcops --dbdir ~/tmp/camcops_client_test 

125 

126- Fire up a web browser showing both (a) the task list via the index, and (b) 

127 the task list without using the index. We'll use this to verify correct 

128 indexing. **The two versions of the view should never be different.** 

129 

130- Ensure the test client device has no current records (force-finalize if 

131 required). 

132 

133- Ensure the server's index is proper. Run ``camcops_server reindex`` if 

134 required. 

135 

136- If required, fire up MySQL with the server database. You may wish to use 

137 ``pager less -SFX``, for better display of large tables. 

138 

139**Testing the upload code** 

140 

141Perform the following steps both (1) with the client forced to the stepwise 

142upload method, and (2) with it forced to one-step upload. 

143 

144Note that the number of patient ID numbers uploaded (etc.) is ignored below. 

145 

146*Single record* 

147 

148[Checked for one-step and multi-step upload, 2018-11-21.] 

149 

150#. Create a blank ReferrerSatisfactionSurvey (table ``ref_satis_gen``). 

151 This has the advantage of being an anonymous single-record task. 

152 

153#. Upload/copy. 

154 

155 - The server log should show 1 × ref_satis_gen added. 

156 

157 - The task lists should show the task as current and incomplete. 

158 

159#. Modify it, so it's complete. 

160 

161#. Upload/copy. 

162 

163 - The server log should show 1 × ref_satis_gen added, 1 × ref_satis_gen 

164 modified out. 

165 

166 - The task lists should show the task as current and complete. 

167 

168#. Upload/move. 

169 

170 - The server log should show 2 × ref_satis_gen preserved. 

171 

172 - The task lists should show the task as no longer current. 

173 

174#. Create another blank one. 

175 

176#. Upload/copy. 

177 

178#. Modify it so it's complete. 

179 

180#. Specifically flag it for preservation (the chequered flags). 

181 

182#. Upload/copy. 

183 

184 - The server log should show 1 × ref_satis_gen added, 1 × ref_satis_gen 

185 modified out, 2 × ref_satis_gen preserved. 

186 

187 - The task lists should show the task as complete and no longer current. 

188 

189*With a patient* 

190 

191[Checked for one-step and multi-step upload, 2018-11-21.] 

192 

193#. Create a dummy patient that the server will accept. 

194 

195#. Create a Progress Note with location "loc1" and abort its creation, giving 

196 an incomplete task. 

197 

198#. Create a second Progress Note with location "loc2" and contents "note2". 

199 

200#. Create a third Progress Note with location "loc3" and contents "note3". 

201 

202#. Upload/copy. Verify. This checks *addition*. 

203 

204 - The server log should show 1 × patient added; 3 × progressnote added. 

205 (Also however many patientidnum records you chose.) 

206 - All three tasks should be "current". 

207 - The first should be "incomplete". 

208 

209#. Modify the first note by adding contents "note1". 

210 

211#. Delete the second note. 

212 

213#. Upload/copy. Verify. This checks *modification*, *deletion*, 

214 *no-change detection*, and *reindexing*. 

215 

216 - The server log should show 1 × progressnote added, 1 × progressnote 

217 modified out, 1 × progressnote deleted. 

218 - The first note should now appear as complete. 

219 - The second should have vanished. 

220 - The third should be unchanged. 

221 - The two remaining tasks should still be "current". 

222 

223#. Delete the contents from the first note again. 

224 

225#. Upload/move (or move-keeping-patients; that's only different on the 

226 client side). Verify. This checks *preservation (finalizing)* and 

227 *reindexing*. 

228 

229 - The server log should show 1 × patient preserved; 1 × progressnote added, 

230 1 × progressnote modified out, 5 × progressnote preserved. 

231 - The two remaining tasks should no longer be "current". 

232 - The first should no longer be "complete". 

233 

234#. Create a complete "note 4" and an incomplete "note 5". 

235 

236#. Upload/copy. 

237 

238#. Force-finalize from the server. This tests force-finalizing including 

239 reindexing. 

240 

241 - The "tasks to finalize" list should have just two tasks in it. 

242 - After force-finalizing, the tasks should remain in the index but no 

243 longer be marked as current. 

244 

245#. Upload/move to get rid of the residual tasks on the client. 

246 

247 - The server log should show 1 × patient added, 1 × patient preserved; 2 × 

248 progressnote added, 2 × progressnote preserved. 

249 

250*With ancillary tables and BLOBs* 

251 

252[Checked for one-step and multi-step upload, 2018-11-21.] 

253 

254#. Create a PhotoSequence with text "t1", one photo named "p1" of you holding 

255 up one finger vertically, and another photo named "p2" of you holding up 

256 two fingers vertically. 

257 

258#. Upload/copy. 

259 

260 - The server log should show: 

261 

262 - blobs: 2 × added; 

263 - patient: 1 × added; 

264 - photosequence: 1 × added; 

265 - photosequence_photos: 2 × added. 

266 

267 - The task lists should look sensible. 

268 

269#. Clear the second photo and replace it with a photo of you holding up 

270 two fingers horizontally. 

271 

272#. Upload/copy. 

273 

274 - The server log should show: 

275 

276 - blobs: 1 × added, 1 × modified out; 

277 - photosequence: 1 × added, 1 × modified out; 

278 - photosequence_photos: 1 × added, 1 × modified out. 

279 

280 - The task lists should look sensible. 

281 

282#. Back to two fingers vertically. (This is the fourth photo overall.) 

283 

284#. Mark that patient for specific finalization. 

285 

286#. Upload/copy. 

287 

288 - The server log should show: 

289 

290 - blobs: 1 × added, 1 × modified out, 4 × preserved; 

291 - patient: 1 × preserved; 

292 - photosequence: 1 × added, 1 × modified out, 3 × preserved; 

293 - photosequence_photos: 1 × added, 1 × modified out, 4 × preserved. 

294 

295 - The tasks should no longer be current. 

296 - A fresh "vertical fingers" photo should be visible. 

297 

298#. Create another patient and another PhotoSequence with one photo of three 

299 fingers. 

300 

301#. Upload-copy. 

302 

303#. Force-finalize. 

304 

305 - Should finalize: 1 × blobs, 1 × patient, 1 × photosequence, 1 × 

306 photosequence_photos. 

307 

308#. Upload/move. 

309 

310During any MySQL debugging, remember: 

311 

312.. code-block:: none 

313 

314 -- For better display: 

315 pager less -SFX; 

316 

317 -- To view relevant parts of the BLOB table without the actual BLOB: 

318 

319 SELECT 

320 _pk, _group_id, _device_id, _era, 

321 _current, _predecessor_pk, _successor_pk, 

322 _addition_pending, _when_added_batch_utc, _adding_user_id, 

323 _removal_pending, _when_removed_batch_utc, _removing_user_id, 

324 _move_off_tablet, 

325 _preserving_user_id, _forcibly_preserved, 

326 id, tablename, tablepk, fieldname, mimetype, when_last_modified 

327 FROM blobs; 

328 

329""" 

330 

331# ============================================================================= 

332# Imports 

333# ============================================================================= 

334 

335import logging 

336import json 

337# from pprint import pformat 

338import secrets 

339import string 

340import time 

341from typing import ( 

342 Any, 

343 Dict, 

344 Iterable, 

345 List, 

346 Optional, 

347 Sequence, 

348 Set, 

349 Tuple, 

350 TYPE_CHECKING 

351) 

352from cardinal_pythonlib.datetimefunc import ( 

353 coerce_to_pendulum, 

354 coerce_to_pendulum_date, 

355 format_datetime, 

356) 

357from cardinal_pythonlib.logs import ( 

358 BraceStyleAdapter, 

359) 

360from cardinal_pythonlib.pyramid.responses import TextResponse 

361from cardinal_pythonlib.sqlalchemy.core_query import ( 

362 exists_in_table, 

363 fetch_all_first_values, 

364) 

365from cardinal_pythonlib.text import escape_newlines 

366from pyramid.httpexceptions import HTTPBadRequest 

367from pyramid.view import view_config 

368from pyramid.response import Response 

369from pyramid.security import NO_PERMISSION_REQUIRED 

370from semantic_version import Version 

371from sqlalchemy.engine.result import ResultProxy 

372from sqlalchemy.exc import IntegrityError 

373from sqlalchemy.orm import joinedload 

374from sqlalchemy.sql.expression import exists, select, update 

375from sqlalchemy.sql.schema import Table 

376 

377from camcops_server.cc_modules import cc_audit # avoids "audit" name clash 

378from camcops_server.cc_modules.cc_all_models import ( 

379 CLIENT_TABLE_MAP, 

380 RESERVED_FIELDS, 

381) 

382from camcops_server.cc_modules.cc_blob import Blob 

383from camcops_server.cc_modules.cc_cache import cache_region_static, fkg 

384from camcops_server.cc_modules.cc_client_api_core import ( 

385 AllowedTablesFieldNames, 

386 BatchDetails, 

387 exception_description, 

388 ExtraStringFieldNames, 

389 fail_server_error, 

390 fail_unsupported_operation, 

391 fail_user_error, 

392 get_server_live_records, 

393 IgnoringAntiqueTableException, 

394 require_keys, 

395 ServerErrorException, 

396 ServerRecord, 

397 TabletParam, 

398 UploadRecordResult, 

399 UploadTableChanges, 

400 UserErrorException, 

401 values_delete_later, 

402 values_delete_now, 

403 values_preserve_now, 

404 WhichKeyToSendInfo, 

405) 

406from camcops_server.cc_modules.cc_client_api_helpers import ( 

407 upload_commit_order_sorter, 

408) 

409from camcops_server.cc_modules.cc_constants import ( 

410 CLIENT_DATE_FIELD, 

411 DateFormat, 

412 ERA_NOW, 

413 FP_ID_NUM, 

414 FP_ID_DESC, 

415 FP_ID_SHORT_DESC, 

416 MOVE_OFF_TABLET_FIELD, 

417 NUMBER_OF_IDNUMS_DEFUNCT, # allowed; for old tablet versions 

418 POSSIBLE_SEX_VALUES, 

419 TABLET_ID_FIELD, 

420) 

421from camcops_server.cc_modules.cc_convert import ( 

422 decode_single_value, 

423 decode_values, 

424 encode_single_value, 

425) 

426from camcops_server.cc_modules.cc_db import ( 

427 FN_ADDING_USER_ID, 

428 FN_ADDITION_PENDING, 

429 FN_CAMCOPS_VERSION, 

430 FN_CURRENT, 

431 FN_DEVICE_ID, 

432 FN_ERA, 

433 FN_GROUP_ID, 

434 FN_PK, 

435 FN_PREDECESSOR_PK, 

436 FN_REMOVAL_PENDING, 

437 FN_REMOVING_USER_ID, 

438 FN_SUCCESSOR_PK, 

439 FN_WHEN_ADDED_BATCH_UTC, 

440 FN_WHEN_ADDED_EXACT, 

441 FN_WHEN_REMOVED_BATCH_UTC, 

442 FN_WHEN_REMOVED_EXACT, 

443) 

444from camcops_server.cc_modules.cc_device import Device 

445from camcops_server.cc_modules.cc_dirtytables import DirtyTable 

446from camcops_server.cc_modules.cc_group import Group 

447from camcops_server.cc_modules.cc_ipuse import IpUse 

448from camcops_server.cc_modules.cc_membership import UserGroupMembership 

449from camcops_server.cc_modules.cc_patient import ( 

450 Patient, 

451 is_candidate_patient_valid_for_group, 

452 is_candidate_patient_valid_for_restricted_user, 

453) 

454from camcops_server.cc_modules.cc_patientidnum import ( 

455 fake_tablet_id_for_patientidnum, 

456 PatientIdNum, 

457) 

458from camcops_server.cc_modules.cc_proquint import ( 

459 InvalidProquintException, 

460 uuid_from_proquint, 

461) 

462from camcops_server.cc_modules.cc_pyramid import Routes 

463from camcops_server.cc_modules.cc_simpleobjects import ( 

464 BarePatientInfo, 

465 IdNumReference, 

466) 

467from camcops_server.cc_modules.cc_specialnote import SpecialNote 

468from camcops_server.cc_modules.cc_task import ( 

469 all_task_tables_with_min_client_version, 

470) 

471from camcops_server.cc_modules.cc_taskindex import update_indexes_and_push_exports # noqa 

472from camcops_server.cc_modules.cc_user import User 

473from camcops_server.cc_modules.cc_validators import ( 

474 STRING_VALIDATOR_TYPE, 

475 validate_anything, 

476 validate_email, 

477) 

478from camcops_server.cc_modules.cc_version import ( 

479 CAMCOPS_SERVER_VERSION_STRING, 

480 MINIMUM_TABLET_VERSION, 

481) 

482 

483if TYPE_CHECKING: 

484 from camcops_server.cc_modules.cc_request import CamcopsRequest 

485 

486log = BraceStyleAdapter(logging.getLogger(__name__)) 

487 

488 

489# ============================================================================= 

490# Constants 

491# ============================================================================= 

492 

493COPE_WITH_DELETED_PATIENT_DESCRIPTIONS = True 

494# ... as of client 2.0.0, ID descriptions are no longer duplicated. 

495# As of server 2.0.0, the fields still exist in the database, but the reporting 

496# and consistency check has been removed. In the next version of the server, 

497# the fields will be removed, and then the server should cope with old clients, 

498# at least for a while. 

499 

500DUPLICATE_FAILED = "Failed to duplicate record" 

501INSERT_FAILED = "Failed to insert record" 

502 

503# REGEX_INVALID_TABLE_FIELD_CHARS = re.compile("[^a-zA-Z0-9_]") 

504# ... the ^ within the [] means the expression will match any character NOT in 

505# the specified range 

506 

507DEVICE_STORED_VAR_TABLENAME_DEFUNCT = "storedvars" 

508# ... old table, no longer in use, that Titanium clients used to upload. 

509# We recognize and ignore it now so that old clients can still work. 

510 

511SILENTLY_IGNORE_TABLENAMES = [DEVICE_STORED_VAR_TABLENAME_DEFUNCT] 

512 

513IGNORING_ANTIQUE_TABLE_MESSAGE = ( 

514 "Ignoring user request to upload antique/defunct table, but reporting " 

515 "success to the client" 

516) 

517 

518SUCCESS_MSG = "Success" 

519SUCCESS_CODE = "1" 

520FAILURE_CODE = "0" 

521 

522DEBUG_UPLOAD = False 

523 

524 

525# ============================================================================= 

526# Quasi-constants 

527# ============================================================================= 

528 

529DB_JSON_DECODER = json.JSONDecoder() # just a plain one 

530PATIENT_INFO_JSON_DECODER = json.JSONDecoder() # just a plain one 

531 

532 

533# ============================================================================= 

534# Cached information 

535# ============================================================================= 

536 

537@cache_region_static.cache_on_arguments(function_key_generator=fkg) 

538def all_tables_with_min_client_version() -> Dict[str, Version]: 

539 """ 

540 For all tables that the client might upload to, return a mapping from the 

541 table name to the corresponding minimum client version. 

542 """ 

543 d = all_task_tables_with_min_client_version() 

544 d[Blob.__tablename__] = MINIMUM_TABLET_VERSION 

545 d[Patient.__tablename__] = MINIMUM_TABLET_VERSION 

546 d[PatientIdNum.__tablename__] = MINIMUM_TABLET_VERSION 

547 return d 

548 

549 

550# ============================================================================= 

551# Validators 

552# ============================================================================= 

553 

554def ensure_valid_table_name(req: "CamcopsRequest", tablename: str) -> None: 

555 """ 

556 Ensures a table name: 

557 

558 - doesn't contain bad characters, 

559 - isn't a reserved table that the user is prohibited from accessing, and 

560 - is a valid table name that's in the database. 

561 

562 Raises :exc:`UserErrorException` upon failure. 

563 

564 - 2017-10-08: shortcut to all that: it's OK if it's listed as a valid 

565 client table. 

566 - 2018-01-16 (v2.2.0): check also that client version is OK 

567 """ 

568 if tablename not in CLIENT_TABLE_MAP: 

569 fail_user_error(f"Invalid client table name: {tablename}") 

570 tables_versions = all_tables_with_min_client_version() 

571 assert tablename in tables_versions 

572 client_version = req.tabletsession.tablet_version_ver 

573 minimum_client_version = tables_versions[tablename] 

574 if client_version < minimum_client_version: 

575 fail_user_error( 

576 f"Client CamCOPS version {client_version} is less than the " 

577 f"version ({minimum_client_version}) " 

578 f"required to handle table {tablename}" 

579 ) 

580 

581 

582def ensure_valid_field_name(table: Table, fieldname: str) -> None: 

583 """ 

584 Ensures a field name contains only valid characters, and isn't a 

585 reserved fieldname that the user isn't allowed to access. 

586 

587 Raises :exc:`UserErrorException` upon failure. 

588 

589 - 2017-10-08: shortcut: it's OK if it's a column name for a particular 

590 table. 

591 """ 

592 # if fieldname in RESERVED_FIELDS: 

593 if fieldname.startswith("_"): # all reserved fields start with _ 

594 # ... but not all fields starting with "_" are reserved; e.g. 

595 # "_move_off_tablet" is allowed. 

596 if fieldname in RESERVED_FIELDS: 

597 fail_user_error( 

598 f"Reserved field name for table {table.name!r}: {fieldname!r}") 

599 if fieldname not in table.columns.keys(): 

600 fail_user_error( 

601 f"Invalid field name for table {table.name!r}: {fieldname!r}") 

602 # Note that the reserved-field check is case-sensitive, but so is the 

603 # "present in table" check. So for a malicious uploader trying to use, for 

604 # example, "_PK", this would not be picked up as a reserved field (so would 

605 # pass that check) but then wouldn't be recognized as a valid field (so 

606 # would fail). 

607 

608 

609def ensure_string(value: Any, allow_none: bool = True) -> None: 

610 """ 

611 Used when processing JSON information about patients: ensures that a value 

612 is a string, or raises. 

613 

614 Args: 

615 value: value to test 

616 allow_none: is ``None`` allowed (not just an empty string)? 

617 """ 

618 if value is None: 

619 if allow_none: 

620 return # OK 

621 else: 

622 fail_user_error("Patient JSON contains absent string") 

623 if not isinstance(value, str): 

624 fail_user_error(f"Patient JSON contains invalid non-string: {value!r}") 

625 

626 

627def ensure_valid_patient_json(req: "CamcopsRequest", 

628 group: Group, 

629 pt_dict: Dict[str, Any]) -> None: 

630 """ 

631 Ensures that the JSON dictionary contains valid patient details (valid for 

632 the group into which it's being uploaded), and that (if applicable) this 

633 user is allowed to upload this patient. 

634 

635 Args: 

636 req: 

637 the :class:`camcops_server.cc_modules.cc_request.CamcopsRequest` 

638 group: 

639 the :class:`camcops_server.cc_modules.cc_group.Group` into which 

640 the upload is going 

641 pt_dict: 

642 a JSON dictionary from the client 

643 

644 Raises: 

645 :exc:`UserErrorException` if invalid 

646 

647 """ 

648 if not isinstance(pt_dict, dict): 

649 fail_user_error("Patient JSON is not a dict") 

650 if not pt_dict: 

651 fail_user_error("Patient JSON is empty") 

652 valid_which_idnums = req.valid_which_idnums 

653 errors = [] # type: List[str] 

654 finalizing = None 

655 ptinfo = BarePatientInfo() 

656 idnum_types_seen = set() # type: Set[int] 

657 for k, v in pt_dict.items(): 

658 ensure_string(k, allow_none=False) 

659 

660 if k == TabletParam.FORENAME: 

661 ensure_string(v) 

662 ptinfo.forename = v 

663 

664 elif k == TabletParam.SURNAME: 

665 ensure_string(v) 

666 ptinfo.surname = v 

667 

668 elif k == TabletParam.SEX: 

669 if v not in POSSIBLE_SEX_VALUES: 

670 fail_user_error(f"Bad sex value: {v!r}") 

671 ptinfo.sex = v 

672 

673 elif k == TabletParam.DOB: 

674 ensure_string(v) 

675 if v: 

676 dob = coerce_to_pendulum_date(v) 

677 if dob is None: 

678 fail_user_error(f"Invalid DOB: {v!r}") 

679 else: 

680 dob = None 

681 ptinfo.dob = dob 

682 

683 elif k == TabletParam.EMAIL: 

684 ensure_string(v) 

685 if v: 

686 try: 

687 validate_email(v) 

688 except ValueError: 

689 fail_user_error(f"Bad e-mail address: {v!r}") 

690 ptinfo.email = v 

691 

692 elif k == TabletParam.ADDRESS: 

693 ensure_string(v) 

694 ptinfo.address = v 

695 

696 elif k == TabletParam.GP: 

697 ensure_string(v) 

698 ptinfo.gp = v 

699 

700 elif k == TabletParam.OTHER: 

701 ensure_string(v) 

702 ptinfo.otherdetails = v 

703 

704 elif k.startswith(TabletParam.IDNUM_PREFIX): 

705 nstr = k[len(TabletParam.IDNUM_PREFIX):] 

706 try: 

707 which_idnum = int(nstr) 

708 except (TypeError, ValueError): 

709 fail_user_error(f"Bad idnum key: {k!r}") 

710 # noinspection PyUnboundLocalVariable 

711 if which_idnum not in valid_which_idnums: 

712 fail_user_error(f"Bad ID number type: {which_idnum}") 

713 if which_idnum in idnum_types_seen: 

714 fail_user_error(f"More than one ID number supplied for ID " 

715 f"number type {which_idnum}") 

716 idnum_types_seen.add(which_idnum) 

717 if v is not None and not isinstance(v, int): 

718 fail_user_error(f"Bad ID number value: {v!r}") 

719 idref = IdNumReference(which_idnum, v) 

720 if not idref.is_valid(): 

721 fail_user_error(f"Bad ID number: {idref!r}") 

722 ptinfo.add_idnum(idref) 

723 

724 elif k == TabletParam.FINALIZING: 

725 if not isinstance(v, bool): 

726 fail_user_error(f"Bad {k!r} value: {v!r}") 

727 finalizing = v 

728 

729 else: 

730 fail_user_error(f"Unknown JSON key: {k!r}") 

731 

732 if finalizing is None: 

733 fail_user_error(f"Missing {TabletParam.FINALIZING!r} JSON key") 

734 

735 pt_ok, reason = is_candidate_patient_valid_for_group( 

736 ptinfo, group, finalizing) 

737 if not pt_ok: 

738 errors.append(f"{ptinfo} -> {reason}") 

739 pt_ok, reason = is_candidate_patient_valid_for_restricted_user( 

740 req, ptinfo) 

741 if not pt_ok: 

742 errors.append(f"{ptinfo} -> {reason}") 

743 if errors: 

744 fail_user_error(f"Invalid patient: {' // '.join(errors)}") 

745 

746 

747# ============================================================================= 

748# Extracting information from the POST request 

749# ============================================================================= 

750 

751def get_str_var( 

752 req: "CamcopsRequest", 

753 var: str, 

754 mandatory: bool = True, 

755 validator: STRING_VALIDATOR_TYPE = validate_anything) \ 

756 -> Optional[str]: 

757 """ 

758 Retrieves a string variable from the CamcopsRequest. 

759 

760 By default this performs no validation (because, for example, these strings 

761 can contain SQL-encoded data or JSON), but there are a number of subsequent 

762 operation-specific validation steps. 

763 

764 Args: 

765 req: the :class:`camcops_server.cc_modules.cc_request.CamcopsRequest` 

766 var: name of variable to retrieve 

767 mandatory: if ``True``, raise an exception if the variable is missing 

768 validator: validator function to use 

769 

770 Returns: 

771 value 

772 

773 Raises: 

774 :exc:`UserErrorException` if the variable was mandatory and 

775 no value was provided 

776 """ 

777 try: 

778 val = req.get_str_param(var, default=None, validator=validator) 

779 if mandatory and val is None: 

780 fail_user_error(f"Must provide the variable: {var}") 

781 return val 

782 except HTTPBadRequest as e: # failed the validator 

783 fail_user_error(str(e)) 

784 

785 

786def get_int_var(req: "CamcopsRequest", var: str) -> int: 

787 """ 

788 Retrieves an integer variable from the CamcopsRequest. 

789 

790 Args: 

791 req: the :class:`camcops_server.cc_modules.cc_request.CamcopsRequest` 

792 var: name of variable to retrieve 

793 

794 Returns: 

795 value 

796 

797 Raises: 

798 :exc:`UserErrorException` if no value was provided, or if it wasn't an 

799 integer 

800 """ 

801 s = get_str_var(req, var, mandatory=True) 

802 try: 

803 return int(s) 

804 except (TypeError, ValueError): 

805 fail_user_error(f"Variable {var} is not a valid integer; was {s!r}") 

806 

807 

808def get_bool_int_var(req: "CamcopsRequest", var: str) -> bool: 

809 """ 

810 Retrieves a Boolean variable (encoded as an integer) from the 

811 CamcopsRequest. Zero represents false; nonzero represents true. 

812 

813 Args: 

814 req: the :class:`camcops_server.cc_modules.cc_request.CamcopsRequest` 

815 var: name of variable to retrieve 

816 

817 Returns: 

818 value 

819 

820 Raises: 

821 :exc:`UserErrorException` if no value was provided, or if it wasn't an 

822 integer 

823 """ 

824 num = get_int_var(req, var) 

825 return bool(num) 

826 

827 

828def get_table_from_req(req: "CamcopsRequest", var: str) -> Table: 

829 """ 

830 Retrieves a table name from a HTTP request, checks it's a valid client 

831 table, and returns that table. 

832 

833 Args: 

834 req: the :class:`camcops_server.cc_modules.cc_request.CamcopsRequest` 

835 var: variable name (the variable's should be the table name) 

836 

837 Returns: 

838 a SQLAlchemy :class:`Table` 

839 

840 Raises: 

841 :exc:`UserErrorException` if the variable wasn't provided 

842 

843 :exc:`IgnoringAntiqueTableException` if the table is one to 

844 ignore quietly (requested by an antique client) 

845 """ 

846 tablename = get_str_var(req, var, mandatory=True) 

847 if tablename in SILENTLY_IGNORE_TABLENAMES: 

848 raise IgnoringAntiqueTableException(f"Ignoring table {tablename}") 

849 ensure_valid_table_name(req, tablename) 

850 return CLIENT_TABLE_MAP[tablename] 

851 

852 

853def get_tables_from_post_var(req: "CamcopsRequest", 

854 var: str, 

855 mandatory: bool = True) -> List[Table]: 

856 """ 

857 Gets a list of tables from an HTTP request variable, and ensures all are 

858 valid. 

859 

860 Args: 

861 req: the :class:`camcops_server.cc_modules.cc_request.CamcopsRequest` 

862 var: name of variable to retrieve 

863 mandatory: if ``True``, raise an exception if the variable is missing 

864 

865 Returns: 

866 a list of SQLAlchemy :class:`Table` objects 

867 

868 Raises: 

869 :exc:`UserErrorException` if the variable was mandatory and 

870 no value was provided, or if one or more tables was not valid 

871 """ 

872 cstables = get_str_var(req, var, mandatory=mandatory) 

873 if not cstables: 

874 return [] 

875 # can't have any commas in table names, so it's OK to use a simple 

876 # split() command 

877 tablenames = [x.strip() for x in cstables.split(",")] 

878 tables = [] # type: List[Table] 

879 for tn in tablenames: 

880 if tn in SILENTLY_IGNORE_TABLENAMES: 

881 log.warning(IGNORING_ANTIQUE_TABLE_MESSAGE) 

882 continue 

883 ensure_valid_table_name(req, tn) 

884 tables.append(CLIENT_TABLE_MAP[tn]) 

885 return tables 

886 

887 

888def get_single_field_from_post_var(req: "CamcopsRequest", 

889 table: Table, 

890 var: str, 

891 mandatory: bool = True) -> str: 

892 """ 

893 Retrieves a field (column) name from a the request and checks it's not a 

894 bad fieldname for the specified table. 

895 

896 Args: 

897 req: the :class:`camcops_server.cc_modules.cc_request.CamcopsRequest` 

898 table: SQLAlchemy :class:`Table` in which the column should exist 

899 var: name of variable to retrieve 

900 mandatory: if ``True``, raise an exception if the variable is missing 

901 

902 Returns: 

903 the field (column) name 

904 

905 Raises: 

906 :exc:`UserErrorException` if the variable was mandatory and 

907 no value was provided, or if the field was not valid for the specified 

908 table 

909 """ 

910 field = get_str_var(req, var, mandatory=mandatory) 

911 ensure_valid_field_name(table, field) 

912 return field 

913 

914 

915def get_fields_from_post_var( 

916 req: "CamcopsRequest", 

917 table: Table, 

918 var: str, 

919 mandatory: bool = True, 

920 allowed_nonexistent_fields: List[str] = None) -> List[str]: 

921 """ 

922 Get a comma-separated list of field names from a request and checks that 

923 all are acceptable. Returns a list of fieldnames. 

924 

925 Args: 

926 req: the :class:`camcops_server.cc_modules.cc_request.CamcopsRequest` 

927 table: SQLAlchemy :class:`Table` in which the columns should exist 

928 var: name of variable to retrieve 

929 mandatory: if ``True``, raise an exception if the variable is missing 

930 allowed_nonexistent_fields: fields that are allowed to be in the 

931 upload but not in the database (special exemptions!) 

932 

933 Returns: 

934 a list of the field (column) names 

935 

936 Raises: 

937 :exc:`UserErrorException` if the variable was mandatory and 

938 no value was provided, or if any field was not valid for the specified 

939 table 

940 """ 

941 csfields = get_str_var(req, var, mandatory=mandatory) 

942 if not csfields: 

943 return [] 

944 allowed_nonexistent_fields = allowed_nonexistent_fields or [] # type: List[str] # noqa 

945 # can't have any commas in fields, so it's OK to use a simple 

946 # split() command 

947 fields = [x.strip() for x in csfields.split(",")] 

948 for f in fields: 

949 if f in allowed_nonexistent_fields: 

950 continue 

951 ensure_valid_field_name(table, f) 

952 return fields 

953 

954 

955def get_values_from_post_var(req: "CamcopsRequest", 

956 var: str, 

957 mandatory: bool = True) -> List[Any]: 

958 """ 

959 Retrieves a list of values from a CSV-separated list of SQL values 

960 stored in a CGI form (including e.g. NULL, numbers, quoted strings, and 

961 special handling for base-64/hex-encoded BLOBs.) 

962 

963 Args: 

964 req: the :class:`camcops_server.cc_modules.cc_request.CamcopsRequest` 

965 var: name of variable to retrieve 

966 mandatory: if ``True``, raise an exception if the variable is missing 

967 """ 

968 csvalues = get_str_var(req, var, mandatory=mandatory) 

969 if not csvalues: 

970 return [] 

971 return decode_values(csvalues) 

972 

973 

974def get_fields_and_values(req: "CamcopsRequest", 

975 table: Table, 

976 fields_var: str, 

977 values_var: str, 

978 mandatory: bool = True) -> Dict[str, Any]: 

979 """ 

980 Gets fieldnames and matching values from two variables in a request. 

981 

982 See :func:`get_fields_from_post_var`, :func:`get_values_from_post_var`. 

983 

984 Args: 

985 req: the :class:`camcops_server.cc_modules.cc_request.CamcopsRequest` 

986 table: SQLAlchemy :class:`Table` in which the columns should exist 

987 fields_var: name of CSV "column names" variable to retrieve 

988 values_var: name of CSV "corresponding values" variable to retrieve 

989 mandatory: if ``True``, raise an exception if the variable is missing 

990 

991 Returns: 

992 a dictionary mapping column names to decoded values 

993 

994 Raises: 

995 :exc:`UserErrorException` if the variable was mandatory and 

996 no value was provided, or if any field was not valid for the specified 

997 table 

998 """ 

999 fields = get_fields_from_post_var(req, table, fields_var, 

1000 mandatory=mandatory) 

1001 values = get_values_from_post_var(req, values_var, mandatory=mandatory) 

1002 if len(fields) != len(values): 

1003 fail_user_error( 

1004 f"Number of fields ({len(fields)}) doesn't match number of values " 

1005 f"({len(values)})" 

1006 ) 

1007 return dict(list(zip(fields, values))) 

1008 

1009 

1010def get_json_from_post_var(req: "CamcopsRequest", key: str, 

1011 decoder: json.JSONDecoder = None, 

1012 mandatory: bool = True) -> Any: 

1013 """ 

1014 Returns a Python object from a JSON-encoded value. 

1015 

1016 Args: 

1017 req: the :class:`camcops_server.cc_modules.cc_request.CamcopsRequest` 

1018 key: the name of the variable to retrieve 

1019 decoder: the JSON decoder object to use; if ``None``, a default is 

1020 created 

1021 mandatory: if ``True``, raise an exception if the variable is missing 

1022 

1023 Returns: 

1024 Python object, e.g. a list of values, or ``None`` if the object is 

1025 invalid and not mandatory 

1026 

1027 Raises: 

1028 :exc:`UserErrorException` if the variable was mandatory and 

1029 no value was provided or the value was invalid JSON 

1030 """ 

1031 decoder = decoder or json.JSONDecoder() 

1032 j = get_str_var(req, key, mandatory=mandatory) # may raise 

1033 if not j: # missing but not mandatory 

1034 return None 

1035 try: 

1036 return decoder.decode(j) 

1037 except json.JSONDecodeError: 

1038 msg = f"Bad JSON for key {key!r}" 

1039 if mandatory: 

1040 fail_user_error(msg) 

1041 else: 

1042 log.warning(msg) 

1043 return None 

1044 

1045 

1046# ============================================================================= 

1047# Sending stuff to the client 

1048# ============================================================================= 

1049 

1050def get_server_id_info(req: "CamcopsRequest") -> Dict[str, str]: 

1051 """ 

1052 Returns a reply for the tablet, as a variable-to-value dictionary, giving 

1053 details of the server. 

1054 """ 

1055 group = Group.get_group_by_id(req.dbsession, req.user.upload_group_id) 

1056 reply = { 

1057 TabletParam.DATABASE_TITLE: req.database_title, 

1058 TabletParam.ID_POLICY_UPLOAD: group.upload_policy or "", 

1059 TabletParam.ID_POLICY_FINALIZE: group.finalize_policy or "", 

1060 TabletParam.SERVER_CAMCOPS_VERSION: CAMCOPS_SERVER_VERSION_STRING, 

1061 } 

1062 for iddef in req.idnum_definitions: 

1063 n = iddef.which_idnum 

1064 nstr = str(n) 

1065 reply[TabletParam.ID_DESCRIPTION_PREFIX + nstr] = \ 

1066 iddef.description or "" 

1067 reply[TabletParam.ID_SHORT_DESCRIPTION_PREFIX + nstr] = \ 

1068 iddef.short_description or "" 

1069 reply[TabletParam.ID_VALIDATION_METHOD_PREFIX + nstr] = \ 

1070 iddef.validation_method or "" 

1071 return reply 

1072 

1073 

1074def get_select_reply(fields: Sequence[str], 

1075 rows: Sequence[Sequence[Any]]) -> Dict[str, str]: 

1076 """ 

1077 Formats the result of a ``SELECT`` query for the client as a dictionary 

1078 reply. 

1079 

1080 Args: 

1081 fields: list of field names 

1082 rows: list of rows, where each row is a list of values in the same 

1083 order as ``fields`` 

1084 

1085 Returns: 

1086 

1087 a dictionary of the format: 

1088 

1089 .. code-block:: none 

1090 

1091 { 

1092 "nfields": NUMBER_OF_FIELDS, 

1093 "fields": FIELDNAMES_AS_CSV, 

1094 "nrecords": NUMBER_OF_RECORDS, 

1095 "record0": VALUES_AS_CSV_LIST_OF_ENCODED_SQL_VALUES, 

1096 ... 

1097 "record{nrecords - 1}": VALUES_AS_CSV_LIST_OF_ENCODED_SQL_VALUES 

1098 } 

1099 

1100 The final reply to the server is then formatted as text as per 

1101 :func:`client_api`. 

1102 

1103 """ # noqa 

1104 nrecords = len(rows) 

1105 reply = { 

1106 TabletParam.NFIELDS: len(fields), 

1107 TabletParam.FIELDS: ",".join(fields), 

1108 TabletParam.NRECORDS: nrecords, 

1109 } 

1110 for r in range(nrecords): 

1111 row = rows[r] 

1112 encodedvalues = [] # type: List[str] 

1113 for val in row: 

1114 encodedvalues.append(encode_single_value(val)) 

1115 reply[TabletParam.RECORD_PREFIX + str(r)] = ",".join(encodedvalues) 

1116 return reply 

1117 

1118 

1119# ============================================================================= 

1120# CamCOPS table reading functions 

1121# ============================================================================= 

1122 

1123def record_exists(req: "CamcopsRequest", 

1124 table: Table, 

1125 clientpk_name: str, 

1126 clientpk_value: Any) -> ServerRecord: 

1127 """ 

1128 Checks if a record exists, using the device's perspective of a 

1129 table/client PK combination. 

1130 

1131 Args: 

1132 req: the :class:`camcops_server.cc_modules.cc_request.CamcopsRequest` 

1133 table: an SQLAlchemy :class:`Table` 

1134 clientpk_name: the column name of the client's PK 

1135 clientpk_value: the client's PK value 

1136 

1137 Returns: 

1138 a :class:`ServerRecord` with the required information 

1139 

1140 """ 

1141 query = ( 

1142 select([ 

1143 table.c[FN_PK], # server PK 

1144 table.c[CLIENT_DATE_FIELD], # when last modified (on the server) 

1145 table.c[MOVE_OFF_TABLET_FIELD] # move_off_tablet 

1146 ]) 

1147 .where(table.c[FN_DEVICE_ID] == req.tabletsession.device_id) 

1148 .where(table.c[FN_CURRENT]) 

1149 .where(table.c[FN_ERA] == ERA_NOW) 

1150 .where(table.c[clientpk_name] == clientpk_value) 

1151 ) 

1152 row = req.dbsession.execute(query).fetchone() 

1153 if not row: 

1154 return ServerRecord(clientpk_value, False) 

1155 server_pk, server_when, move_off_tablet = row 

1156 return ServerRecord(clientpk_value, True, server_pk, server_when, 

1157 move_off_tablet) 

1158 # Consider a warning/failure if we have >1 row meeting these criteria. 

1159 # Not currently checked for. 

1160 

1161 

1162def client_pks_that_exist(req: "CamcopsRequest", 

1163 table: Table, 

1164 clientpk_name: str, 

1165 clientpk_values: List[int]) \ 

1166 -> Dict[int, ServerRecord]: 

1167 """ 

1168 Searches for client PK values (for this device, current, and 'now') 

1169 matching the input list. 

1170 

1171 Args: 

1172 req: the :class:`camcops_server.cc_modules.cc_request.CamcopsRequest` 

1173 table: an SQLAlchemy :class:`Table` 

1174 clientpk_name: the column name of the client's PK 

1175 clientpk_values: a list of the client's PK values 

1176 

1177 Returns: 

1178 a dictionary mapping client_pk to a :class:`ServerRecord` objects, for 

1179 those records that match 

1180 """ 

1181 query = ( 

1182 select([ 

1183 table.c[FN_PK], # server PK 

1184 table.c[clientpk_name], # client PK 

1185 table.c[CLIENT_DATE_FIELD], # when last modified (on the server) 

1186 table.c[MOVE_OFF_TABLET_FIELD] # move_off_tablet 

1187 ]) 

1188 .where(table.c[FN_DEVICE_ID] == req.tabletsession.device_id) 

1189 .where(table.c[FN_CURRENT]) 

1190 .where(table.c[FN_ERA] == ERA_NOW) 

1191 .where(table.c[clientpk_name].in_(clientpk_values)) 

1192 ) 

1193 rows = req.dbsession.execute(query) 

1194 d = {} # type: Dict[int, ServerRecord] 

1195 for server_pk, client_pk, server_when, move_off_tablet in rows: 

1196 d[client_pk] = ServerRecord(client_pk, True, server_pk, server_when, 

1197 move_off_tablet) 

1198 return d 

1199 

1200 

1201def get_all_predecessor_pks(req: "CamcopsRequest", 

1202 table: Table, 

1203 last_pk: int, 

1204 include_last: bool = True) -> List[int]: 

1205 """ 

1206 Retrieves the PKs of all records that are predecessors of the specified one 

1207 

1208 Args: 

1209 req: the :class:`camcops_server.cc_modules.cc_request.CamcopsRequest` 

1210 table: an SQLAlchemy :class:`Table` 

1211 last_pk: the PK to start with, and work backwards 

1212 include_last: include ``last_pk`` in the list 

1213 

1214 Returns: 

1215 the PKs 

1216 

1217 """ 

1218 dbsession = req.dbsession 

1219 pks = [] # type: List[int] 

1220 if include_last: 

1221 pks.append(last_pk) 

1222 current_pk = last_pk 

1223 finished = False 

1224 while not finished: 

1225 next_pk = dbsession.execute( 

1226 select([table.c[FN_PREDECESSOR_PK]]) 

1227 .where(table.c[FN_PK] == current_pk) 

1228 ).scalar() # type: Optional[int] 

1229 if next_pk is None: 

1230 finished = True 

1231 else: 

1232 pks.append(next_pk) 

1233 current_pk = next_pk 

1234 return sorted(pks) 

1235 

1236 

1237# ============================================================================= 

1238# Record modification functions 

1239# ============================================================================= 

1240 

1241def flag_deleted(req: "CamcopsRequest", 

1242 batchdetails: BatchDetails, 

1243 table: Table, 

1244 pklist: Iterable[int]) -> None: 

1245 """ 

1246 Marks record(s) as deleted, specified by a list of server PKs within a 

1247 table. (Note: "deleted" means "deleted with no successor", not "modified 

1248 and replaced by a successor record".) 

1249 """ 

1250 if batchdetails.onestep: 

1251 values = values_delete_now(req, batchdetails) 

1252 else: 

1253 values = values_delete_later() 

1254 req.dbsession.execute( 

1255 update(table) 

1256 .where(table.c[FN_PK].in_(pklist)) 

1257 .values(values) 

1258 ) 

1259 

1260 

1261def flag_all_records_deleted(req: "CamcopsRequest", 

1262 table: Table) -> int: 

1263 """ 

1264 Marks all records in a table as deleted (that are current and in the 

1265 current era). 

1266 

1267 Returns the number of rows affected. 

1268 """ 

1269 rp = req.dbsession.execute( 

1270 update(table) 

1271 .where(table.c[FN_DEVICE_ID] == req.tabletsession.device_id) 

1272 .where(table.c[FN_CURRENT]) 

1273 .where(table.c[FN_ERA] == ERA_NOW) 

1274 .values(values_delete_later()) 

1275 ) # type: ResultProxy 

1276 return rp.rowcount 

1277 # https://docs.sqlalchemy.org/en/latest/core/connections.html?highlight=rowcount#sqlalchemy.engine.ResultProxy.rowcount # noqa 

1278 

1279 

1280def flag_deleted_where_clientpk_not(req: "CamcopsRequest", 

1281 table: Table, 

1282 clientpk_name: str, 

1283 clientpk_values: Sequence[Any]) -> None: 

1284 """ 

1285 Marks for deletion all current/current-era records for a device, within a 

1286 specific table, defined by a list of client-side PK values (and the name of 

1287 the client-side PK column). 

1288 """ 

1289 rp = req.dbsession.execute( 

1290 update(table) 

1291 .where(table.c[FN_DEVICE_ID] == req.tabletsession.device_id) 

1292 .where(table.c[FN_CURRENT]) 

1293 .where(table.c[FN_ERA] == ERA_NOW) 

1294 .where(table.c[clientpk_name].notin_(clientpk_values)) 

1295 .values(values_delete_later()) 

1296 ) # type: ResultProxy 

1297 if rp.rowcount > 0: 

1298 mark_table_dirty(req, table) 

1299 # ... but if we are preserving, do NOT mark this table as clean; there may 

1300 # be other records that still require preserving. 

1301 

1302 

1303def flag_modified(req: "CamcopsRequest", 

1304 batchdetails: BatchDetails, 

1305 table: Table, 

1306 pk: int, 

1307 successor_pk: int) -> None: 

1308 """ 

1309 Marks a record as old, storing its successor's details. 

1310 

1311 Args: 

1312 req: the :class:`camcops_server.cc_modules.cc_request.CamcopsRequest` 

1313 batchdetails: the :class:`BatchDetails` 

1314 table: SQLAlchemy :class:`Table` 

1315 pk: server PK of the record to mark as old 

1316 successor_pk: server PK of its successor 

1317 """ 

1318 if batchdetails.onestep: 

1319 req.dbsession.execute( 

1320 update(table) 

1321 .where(table.c[FN_PK] == pk) 

1322 .values({ 

1323 FN_CURRENT: 0, 

1324 FN_REMOVAL_PENDING: 0, 

1325 FN_SUCCESSOR_PK: successor_pk, 

1326 FN_REMOVING_USER_ID: req.user_id, 

1327 FN_WHEN_REMOVED_EXACT: req.now, 

1328 FN_WHEN_REMOVED_BATCH_UTC: batchdetails.batchtime, 

1329 }) 

1330 ) 

1331 else: 

1332 req.dbsession.execute( 

1333 update(table) 

1334 .where(table.c[FN_PK] == pk) 

1335 .values({ 

1336 FN_REMOVAL_PENDING: 1, 

1337 FN_SUCCESSOR_PK: successor_pk 

1338 }) 

1339 ) 

1340 

1341 

1342def flag_multiple_records_for_preservation( 

1343 req: "CamcopsRequest", 

1344 batchdetails: BatchDetails, 

1345 table: Table, 

1346 pks_to_preserve: List[int]) -> None: 

1347 """ 

1348 Low-level function to mark records for preservation by server PK. 

1349 Does not concern itself with the predecessor chain (for which, see 

1350 :func:`flag_record_for_preservation`). 

1351 

1352 Args: 

1353 req: the :class:`camcops_server.cc_modules.cc_request.CamcopsRequest` 

1354 batchdetails: the :class:`BatchDetails` 

1355 table: SQLAlchemy :class:`Table` 

1356 pks_to_preserve: server PK of the records to mark as preserved 

1357 """ 

1358 if batchdetails.onestep: 

1359 req.dbsession.execute( 

1360 update(table) 

1361 .where(table.c[FN_PK].in_(pks_to_preserve)) 

1362 .values(values_preserve_now(req, batchdetails)) 

1363 ) 

1364 # Also any associated special notes: 

1365 new_era = batchdetails.new_era 

1366 # noinspection PyUnresolvedReferences 

1367 req.dbsession.execute( 

1368 update(SpecialNote.__table__) 

1369 .where(SpecialNote.basetable == table.name) 

1370 .where(SpecialNote.device_id == req.tabletsession.device_id) 

1371 .where(SpecialNote.era == ERA_NOW) 

1372 .where(exists().select_from(table) 

1373 .where(table.c[TABLET_ID_FIELD] == SpecialNote.task_id) 

1374 .where(table.c[FN_DEVICE_ID] == SpecialNote.device_id) 

1375 .where(table.c[FN_ERA] == new_era)) 

1376 # ^^^^^^^^^^^^^^^^^^^^^^^^^^ 

1377 # This bit restricts to records being preserved. 

1378 .values(era=new_era) 

1379 ) 

1380 else: 

1381 req.dbsession.execute( 

1382 update(table) 

1383 .where(table.c[FN_PK].in_(pks_to_preserve)) 

1384 .values({ 

1385 MOVE_OFF_TABLET_FIELD: 1 

1386 }) 

1387 ) 

1388 

1389 

1390def flag_record_for_preservation(req: "CamcopsRequest", 

1391 batchdetails: BatchDetails, 

1392 table: Table, 

1393 pk: int) -> List[int]: 

1394 """ 

1395 Marks a record for preservation (moving off the tablet, changing its 

1396 era details). 

1397 

1398 2018-11-18: works back through the predecessor chain too, fixing an old 

1399 bug. 

1400 

1401 Args: 

1402 req: the :class:`camcops_server.cc_modules.cc_request.CamcopsRequest` 

1403 batchdetails: the :class:`BatchDetails` 

1404 table: SQLAlchemy :class:`Table` 

1405 pk: server PK of the record to mark 

1406 

1407 Returns: 

1408 list: all PKs being preserved 

1409 """ 

1410 pks_to_preserve = get_all_predecessor_pks(req, table, pk) 

1411 flag_multiple_records_for_preservation(req, batchdetails, table, 

1412 pks_to_preserve) 

1413 return pks_to_preserve 

1414 

1415 

1416def preserve_all(req: "CamcopsRequest", 

1417 batchdetails: BatchDetails, 

1418 table: Table) -> None: 

1419 """ 

1420 Preserves all records in a table for a device, including non-current ones. 

1421 

1422 Args: 

1423 req: the :class:`camcops_server.cc_modules.cc_request.CamcopsRequest` 

1424 batchdetails: the :class:`BatchDetails` 

1425 table: SQLAlchemy :class:`Table` 

1426 """ 

1427 device_id = req.tabletsession.device_id 

1428 req.dbsession.execute( 

1429 update(table) 

1430 .where(table.c[FN_DEVICE_ID] == device_id) 

1431 .where(table.c[FN_ERA] == ERA_NOW) 

1432 .values(values_preserve_now(req, batchdetails)) 

1433 ) 

1434 

1435 

1436# ============================================================================= 

1437# Upload helper functions 

1438# ============================================================================= 

1439 

1440def process_upload_record_special(req: "CamcopsRequest", 

1441 batchdetails: BatchDetails, 

1442 table: Table, 

1443 valuedict: Dict[str, Any]) -> None: 

1444 """ 

1445 Special processing function for upload, in which we inspect the data. 

1446 Called by :func:`upload_record_core`. 

1447 

1448 1. Handles old clients with ID information in the patient table, etc. 

1449 (Note: this can be IGNORED for any client using 

1450 :func:`op_upload_entire_database`, as these are newer.) 

1451 

1452 2. Validates ID numbers. 

1453 

1454 Args: 

1455 req: the :class:`camcops_server.cc_modules.cc_request.CamcopsRequest` 

1456 batchdetails: the :class:`BatchDetails` 

1457 table: an SQLAlchemy :class:`Table` 

1458 valuedict: a dictionary of {colname: value} pairs from the client. 

1459 May be modified. 

1460 """ 

1461 ts = req.tabletsession 

1462 tablename = table.name 

1463 

1464 if tablename == Patient.__tablename__: 

1465 # --------------------------------------------------------------------- 

1466 # Deal with old tablets that had ID numbers in a less flexible format. 

1467 # --------------------------------------------------------------------- 

1468 if ts.cope_with_deleted_patient_descriptors: 

1469 # Old tablets (pre-2.0.0) will upload copies of the ID 

1470 # descriptions with the patient. To cope with that, we 

1471 # remove those here: 

1472 for n in range(1, NUMBER_OF_IDNUMS_DEFUNCT + 1): 

1473 nstr = str(n) 

1474 fn_desc = FP_ID_DESC + nstr 

1475 fn_shortdesc = FP_ID_SHORT_DESC + nstr 

1476 valuedict.pop(fn_desc, None) # remove item, if exists 

1477 valuedict.pop(fn_shortdesc, None) 

1478 

1479 if ts.cope_with_old_idnums: 

1480 # Insert records into the new ID number table from the old 

1481 # patient table: 

1482 for which_idnum in range(1, NUMBER_OF_IDNUMS_DEFUNCT + 1): 

1483 nstr = str(which_idnum) 

1484 fn_idnum = FP_ID_NUM + nstr 

1485 idnum_value = valuedict.pop(fn_idnum, None) 

1486 # ... and remove it from our new Patient record 

1487 patient_id = valuedict.get("id", None) 

1488 if idnum_value is None or patient_id is None: 

1489 continue 

1490 # noinspection PyUnresolvedReferences 

1491 mark_table_dirty(req, PatientIdNum.__table__) 

1492 client_date_value = coerce_to_pendulum(valuedict[CLIENT_DATE_FIELD]) # noqa 

1493 # noinspection PyUnresolvedReferences 

1494 upload_record_core( 

1495 req=req, 

1496 batchdetails=batchdetails, 

1497 table=PatientIdNum.__table__, 

1498 clientpk_name='id', 

1499 valuedict={ 

1500 'id': fake_tablet_id_for_patientidnum( 

1501 patient_id=patient_id, 

1502 which_idnum=which_idnum 

1503 ), # ... guarantees a pseudo client PK 

1504 'patient_id': patient_id, 

1505 'which_idnum': which_idnum, 

1506 'idnum_value': idnum_value, 

1507 CLIENT_DATE_FIELD: client_date_value, 

1508 MOVE_OFF_TABLET_FIELD: valuedict[MOVE_OFF_TABLET_FIELD], # noqa 

1509 } 

1510 ) 

1511 # Now, how to deal with deletion, i.e. records missing from the 

1512 # tablet? See our caller, op_upload_table(), which has a special 

1513 # handler for this. 

1514 # 

1515 # Note that op_upload_record() is/was only used for BLOBs, so we 

1516 # don't have to worry about special processing for that aspect 

1517 # here; also, that method handles deletion in a different way. 

1518 

1519 elif tablename == PatientIdNum.__tablename__: 

1520 # --------------------------------------------------------------------- 

1521 # Validate ID numbers. 

1522 # --------------------------------------------------------------------- 

1523 which_idnum = valuedict.get("which_idnum", None) 

1524 if which_idnum not in req.valid_which_idnums: 

1525 fail_user_error(f"No such ID number type: {which_idnum}") 

1526 idnum_value = valuedict.get("idnum_value", None) 

1527 if not req.is_idnum_valid(which_idnum, idnum_value): 

1528 why_invalid = req.why_idnum_invalid(which_idnum, idnum_value) 

1529 fail_user_error( 

1530 f"For ID type {which_idnum}, ID number {idnum_value} is " 

1531 f"invalid: {why_invalid}") 

1532 

1533 

1534def upload_record_core( 

1535 req: "CamcopsRequest", 

1536 batchdetails: BatchDetails, 

1537 table: Table, 

1538 clientpk_name: str, 

1539 valuedict: Dict[str, Any], 

1540 server_live_current_records: List[ServerRecord] = None) \ 

1541 -> UploadRecordResult: 

1542 """ 

1543 Uploads a record. Deals with IDENTICAL, NEW, and MODIFIED records. 

1544 

1545 Used by :func:`upload_table` and :func:`upload_record`. 

1546 

1547 Args: 

1548 req: the :class:`camcops_server.cc_modules.cc_request.CamcopsRequest` 

1549 batchdetails: the :class:`BatchDetails` 

1550 table: an SQLAlchemy :class:`Table` 

1551 clientpk_name: the column name of the client's PK 

1552 valuedict: a dictionary of {colname: value} pairs from the client 

1553 server_live_current_records: list of :class:`ServerRecord` objects for 

1554 the active records on the server for this client, in this table 

1555 

1556 Returns: 

1557 a :class:`UploadRecordResult` object 

1558 """ 

1559 require_keys(valuedict, [clientpk_name, CLIENT_DATE_FIELD, 

1560 MOVE_OFF_TABLET_FIELD]) 

1561 clientpk_value = valuedict[clientpk_name] 

1562 

1563 if server_live_current_records: 

1564 # All server records for this table/device/era have been prefetched. 

1565 serverrec = next((r for r in server_live_current_records 

1566 if r.client_pk == clientpk_value), None) 

1567 if serverrec is None: 

1568 serverrec = ServerRecord(clientpk_value, False) 

1569 else: 

1570 # Look up this record specifically. 

1571 serverrec = record_exists(req, table, clientpk_name, clientpk_value) 

1572 

1573 if DEBUG_UPLOAD: 

1574 log.debug("upload_record_core: {}, {}", table.name, serverrec) 

1575 

1576 oldserverpk = serverrec.server_pk 

1577 urr = UploadRecordResult( 

1578 oldserverpk=oldserverpk, 

1579 specifically_marked_for_preservation=bool(valuedict[MOVE_OFF_TABLET_FIELD]), # noqa 

1580 dirty=True 

1581 ) 

1582 if serverrec.exists: 

1583 # There's an existing record, which is either identical or not. 

1584 client_date_value = coerce_to_pendulum(valuedict[CLIENT_DATE_FIELD]) 

1585 if serverrec.server_when == client_date_value: 

1586 # The existing record is identical. 

1587 # No action needed unless MOVE_OFF_TABLET_FIELDNAME is set. 

1588 if not urr.specifically_marked_for_preservation: 

1589 urr.dirty = False 

1590 else: 

1591 # The existing record is different. We need a logical UPDATE, but 

1592 # maintaining an audit trail. 

1593 process_upload_record_special(req, batchdetails, table, valuedict) 

1594 urr.newserverpk = insert_record(req, batchdetails, table, 

1595 valuedict, oldserverpk) 

1596 flag_modified(req, batchdetails, 

1597 table, oldserverpk, urr.newserverpk) 

1598 else: 

1599 # The record is NEW. We need to INSERT it. 

1600 process_upload_record_special(req, batchdetails, table, valuedict) 

1601 urr.newserverpk = insert_record(req, batchdetails, table, 

1602 valuedict, None) 

1603 if urr.specifically_marked_for_preservation: 

1604 preservation_pks = flag_record_for_preservation(req, batchdetails, 

1605 table, urr.latest_pk) 

1606 urr.note_specifically_marked_preservation_pks(preservation_pks) 

1607 

1608 if DEBUG_UPLOAD: 

1609 log.debug("upload_record_core: {}, {!r}", table.name, urr) 

1610 return urr 

1611 

1612 

1613def insert_record(req: "CamcopsRequest", 

1614 batchdetails: BatchDetails, 

1615 table: Table, 

1616 valuedict: Dict[str, Any], 

1617 predecessor_pk: Optional[int]) -> int: 

1618 """ 

1619 Inserts a record, or raises an exception if that fails. 

1620 

1621 Args: 

1622 req: the :class:`camcops_server.cc_modules.cc_request.CamcopsRequest` 

1623 batchdetails: the :class:`BatchDetails` 

1624 table: an SQLAlchemy :class:`Table` 

1625 valuedict: a dictionary of {colname: value} pairs from the client 

1626 predecessor_pk: an optional server PK of the record's predecessor 

1627 

1628 Returns: 

1629 the server PK of the new record 

1630 """ 

1631 ts = req.tabletsession 

1632 valuedict.update({ 

1633 FN_DEVICE_ID: ts.device_id, 

1634 FN_ERA: ERA_NOW, 

1635 FN_REMOVAL_PENDING: 0, 

1636 FN_PREDECESSOR_PK: predecessor_pk, 

1637 FN_CAMCOPS_VERSION: ts.tablet_version_str, 

1638 FN_GROUP_ID: req.user.upload_group_id, 

1639 }) 

1640 if batchdetails.onestep: 

1641 valuedict.update({ 

1642 FN_CURRENT: 1, 

1643 FN_ADDITION_PENDING: 0, 

1644 FN_ADDING_USER_ID: req.user_id, 

1645 FN_WHEN_ADDED_EXACT: req.now, 

1646 FN_WHEN_ADDED_BATCH_UTC: batchdetails.batchtime, 

1647 }) 

1648 else: 

1649 valuedict.update({ 

1650 FN_CURRENT: 0, 

1651 FN_ADDITION_PENDING: 1, 

1652 }) 

1653 rp = req.dbsession.execute( 

1654 table.insert().values(valuedict) 

1655 ) # type: ResultProxy 

1656 inserted_pks = rp.inserted_primary_key 

1657 assert(isinstance(inserted_pks, list) and len(inserted_pks) == 1) 

1658 return inserted_pks[0] 

1659 

1660 

1661def audit_upload(req: "CamcopsRequest", 

1662 changes: List[UploadTableChanges]) -> None: 

1663 """ 

1664 Writes audit information for an upload. 

1665 

1666 Args: 

1667 req: the :class:`camcops_server.cc_modules.cc_request.CamcopsRequest` 

1668 changes: a list of :class:`UploadTableChanges` objects, one per table 

1669 """ 

1670 msg = ( 

1671 f"Upload from device {req.tabletsession.device_id}, " 

1672 f"username {req.tabletsession.username!r}: " 

1673 ) 

1674 changes = [x for x in changes if x.any_changes] 

1675 if changes: 

1676 changes.sort(key=lambda x: x.tablename) 

1677 msg += ", ".join(x.description() for x in changes) 

1678 else: 

1679 msg += "No changes" 

1680 log.info("audit_upload: {}", msg) 

1681 audit(req, msg) 

1682 

1683 

1684# ============================================================================= 

1685# Batch (atomic) upload and preserving 

1686# ============================================================================= 

1687 

1688def get_batch_details(req: "CamcopsRequest") -> BatchDetails: 

1689 """ 

1690 Returns the :class:`BatchDetails` for the current upload. If none exists, 

1691 a new batch is created and returned. 

1692 

1693 SIDE EFFECT: if the username is different from the username that started 

1694 a previous upload batch for this device, we restart the upload batch (thus 

1695 rolling back previous pending changes). 

1696 

1697 Raises: 

1698 :exc:`camcops_server.cc_modules.cc_client_api_core.ServerErrorException` 

1699 if the device doesn't exist 

1700 """ 

1701 device_id = req.tabletsession.device_id 

1702 # noinspection PyUnresolvedReferences 

1703 query = ( 

1704 select([Device.ongoing_upload_batch_utc, 

1705 Device.uploading_user_id, 

1706 Device.currently_preserving]) 

1707 .select_from(Device.__table__) 

1708 .where(Device.id == device_id) 

1709 ) 

1710 row = req.dbsession.execute(query).fetchone() 

1711 if not row: 

1712 fail_server_error(f"Device {device_id} missing from Device table") # will raise # noqa 

1713 upload_batch_utc, uploading_user_id, currently_preserving = row 

1714 if not upload_batch_utc or uploading_user_id != req.user_id: 

1715 # SIDE EFFECT: if the username changes, we restart (and thus roll back 

1716 # previous pending changes) 

1717 start_device_upload_batch(req) 

1718 return BatchDetails(req.now_utc, False) 

1719 return BatchDetails(upload_batch_utc, currently_preserving) 

1720 

1721 

1722def start_device_upload_batch(req: "CamcopsRequest") -> None: 

1723 """ 

1724 Starts an upload batch for a device. 

1725 """ 

1726 rollback_all(req) 

1727 # noinspection PyUnresolvedReferences 

1728 req.dbsession.execute( 

1729 update(Device.__table__) 

1730 .where(Device.id == req.tabletsession.device_id) 

1731 .values(last_upload_batch_utc=req.now_utc, 

1732 ongoing_upload_batch_utc=req.now_utc, 

1733 uploading_user_id=req.tabletsession.user_id) 

1734 ) 

1735 

1736 

1737def _clear_ongoing_upload_batch_details(req: "CamcopsRequest") -> None: 

1738 """ 

1739 Clears upload batch details from the Device table. 

1740 

1741 Args: 

1742 req: the :class:`camcops_server.cc_modules.cc_request.CamcopsRequest` 

1743 """ 

1744 # noinspection PyUnresolvedReferences 

1745 req.dbsession.execute( 

1746 update(Device.__table__) 

1747 .where(Device.id == req.tabletsession.device_id) 

1748 .values(ongoing_upload_batch_utc=None, 

1749 uploading_user_id=None, 

1750 currently_preserving=0) 

1751 ) 

1752 

1753 

1754def end_device_upload_batch(req: "CamcopsRequest", 

1755 batchdetails: BatchDetails) -> None: 

1756 """ 

1757 Ends an upload batch, committing all changes made thus far. 

1758 

1759 Args: 

1760 req: the :class:`camcops_server.cc_modules.cc_request.CamcopsRequest` 

1761 batchdetails: the :class:`BatchDetails` 

1762 """ 

1763 commit_all(req, batchdetails) 

1764 _clear_ongoing_upload_batch_details(req) 

1765 

1766 

1767def clear_device_upload_batch(req: "CamcopsRequest") -> None: 

1768 """ 

1769 Ensures there is nothing pending. Rools back previous changes. Wipes any 

1770 ongoing batch details. 

1771 

1772 Args: 

1773 req: the :class:`camcops_server.cc_modules.cc_request.CamcopsRequest` 

1774 """ 

1775 rollback_all(req) 

1776 _clear_ongoing_upload_batch_details(req) 

1777 

1778 

1779def start_preserving(req: "CamcopsRequest") -> None: 

1780 """ 

1781 Starts preservation (the process of moving records from the NOW era to 

1782 an older era, so they can be removed safely from the tablet). 

1783 

1784 Called by :func:`op_start_preservation`. 

1785 

1786 In this situation, we start by assuming that ALL tables are "dirty", 

1787 because they may have live records from a previous upload. 

1788 """ 

1789 # noinspection PyUnresolvedReferences 

1790 req.dbsession.execute( 

1791 update(Device.__table__) 

1792 .where(Device.id == req.tabletsession.device_id) 

1793 .values(currently_preserving=1) 

1794 ) 

1795 mark_all_tables_dirty(req) 

1796 

1797 

1798def mark_table_dirty(req: "CamcopsRequest", table: Table) -> None: 

1799 """ 

1800 Marks a table as having been modified during the current upload. 

1801 """ 

1802 tablename = table.name 

1803 device_id = req.tabletsession.device_id 

1804 dbsession = req.dbsession 

1805 # noinspection PyUnresolvedReferences 

1806 table_already_dirty = exists_in_table( 

1807 dbsession, 

1808 DirtyTable.__table__, 

1809 DirtyTable.device_id == device_id, 

1810 DirtyTable.tablename == tablename 

1811 ) 

1812 if not table_already_dirty: 

1813 # noinspection PyUnresolvedReferences 

1814 dbsession.execute( 

1815 DirtyTable.__table__.insert() 

1816 .values(device_id=device_id, 

1817 tablename=tablename) 

1818 ) 

1819 

1820 

1821def mark_tables_dirty(req: "CamcopsRequest", tables: List[Table]) -> None: 

1822 """ 

1823 Marks multiple tables as dirty. 

1824 """ 

1825 if not tables: 

1826 return 

1827 device_id = req.tabletsession.device_id 

1828 tablenames = [t.name for t in tables] 

1829 # Delete first 

1830 # noinspection PyUnresolvedReferences 

1831 req.dbsession.execute( 

1832 DirtyTable.__table__.delete() 

1833 .where(DirtyTable.device_id == device_id) 

1834 .where(DirtyTable.tablename.in_(tablenames)) 

1835 ) 

1836 # Then insert 

1837 insert_values = [ 

1838 {"device_id": device_id, "tablename": tn} 

1839 for tn in tablenames 

1840 ] 

1841 # noinspection PyUnresolvedReferences 

1842 req.dbsession.execute( 

1843 DirtyTable.__table__.insert(), 

1844 insert_values 

1845 ) 

1846 

1847 

1848def mark_all_tables_dirty(req: "CamcopsRequest") -> None: 

1849 """ 

1850 If we are preserving, we assume that all tables are "dirty" (require work 

1851 when we complete the upload) unless we specifically mark them clean. 

1852 """ 

1853 device_id = req.tabletsession.device_id 

1854 # Delete first 

1855 # noinspection PyUnresolvedReferences 

1856 req.dbsession.execute( 

1857 DirtyTable.__table__.delete() 

1858 .where(DirtyTable.device_id == device_id) 

1859 ) 

1860 # Now insert 

1861 # https://docs.sqlalchemy.org/en/latest/core/tutorial.html#execute-multiple 

1862 all_client_tablenames = list(CLIENT_TABLE_MAP.keys()) 

1863 insert_values = [ 

1864 {"device_id": device_id, "tablename": tn} 

1865 for tn in all_client_tablenames 

1866 ] 

1867 # noinspection PyUnresolvedReferences 

1868 req.dbsession.execute( 

1869 DirtyTable.__table__.insert(), 

1870 insert_values 

1871 ) 

1872 

1873 

1874def mark_table_clean(req: "CamcopsRequest", table: Table) -> None: 

1875 """ 

1876 Marks a table as being clean: that is, 

1877 

1878 - the table has been scanned during the current upload 

1879 - there is nothing to do (either from the current upload, OR A PREVIOUS 

1880 UPLOAD). 

1881 """ 

1882 tablename = table.name 

1883 device_id = req.tabletsession.device_id 

1884 # noinspection PyUnresolvedReferences 

1885 req.dbsession.execute( 

1886 DirtyTable.__table__.delete() 

1887 .where(DirtyTable.device_id == device_id) 

1888 .where(DirtyTable.tablename == tablename) 

1889 ) 

1890 

1891 

1892def mark_tables_clean(req: "CamcopsRequest", tables: List[Table]) -> None: 

1893 """ 

1894 Marks multiple tables as clean. 

1895 """ 

1896 if not tables: 

1897 return 

1898 device_id = req.tabletsession.device_id 

1899 tablenames = [t.name for t in tables] 

1900 # Delete first 

1901 # noinspection PyUnresolvedReferences 

1902 req.dbsession.execute( 

1903 DirtyTable.__table__.delete() 

1904 .where(DirtyTable.device_id == device_id) 

1905 .where(DirtyTable.tablename.in_(tablenames)) 

1906 ) 

1907 

1908 

1909def get_dirty_tables(req: "CamcopsRequest") -> List[Table]: 

1910 """ 

1911 Returns tables marked as dirty for this device. (See 

1912 :func:`mark_table_dirty`.) 

1913 """ 

1914 query = ( 

1915 select([DirtyTable.tablename]) 

1916 .where(DirtyTable.device_id == req.tabletsession.device_id) 

1917 ) 

1918 tablenames = fetch_all_first_values(req.dbsession, query) 

1919 return [CLIENT_TABLE_MAP[tn] for tn in tablenames] 

1920 

1921 

1922def commit_all(req: "CamcopsRequest", batchdetails: BatchDetails) -> None: 

1923 """ 

1924 Commits additions, removals, and preservations for all tables. 

1925 

1926 Args: 

1927 req: the :class:`camcops_server.cc_modules.cc_request.CamcopsRequest` 

1928 batchdetails: the :class:`BatchDetails` 

1929 """ 

1930 tables = get_dirty_tables(req) 

1931 # log.debug("Dirty tables: {}", list(t.name for t in tables)) 

1932 tables.sort(key=upload_commit_order_sorter) 

1933 

1934 changelist = [] # type: List[UploadTableChanges] 

1935 for table in tables: 

1936 auditinfo = commit_table(req, batchdetails, table, clear_dirty=False) 

1937 changelist.append(auditinfo) 

1938 

1939 if batchdetails.preserving: 

1940 # Also preserve/finalize any corresponding special notes (2015-02-01), 

1941 # but all in one go (2018-11-13). 

1942 # noinspection PyUnresolvedReferences 

1943 req.dbsession.execute( 

1944 update(SpecialNote.__table__) 

1945 .where(SpecialNote.device_id == req.tabletsession.device_id) 

1946 .where(SpecialNote.era == ERA_NOW) 

1947 .values(era=batchdetails.new_era) 

1948 ) 

1949 

1950 clear_dirty_tables(req) 

1951 audit_upload(req, changelist) 

1952 

1953 # Performance 2018-11-13: 

1954 # - start at 2.407 s 

1955 # - remove final temptable clearance and COUNT(*): 1.626 to 2.118 s 

1956 # - IN clause using Python literal not temptable: 1.18 to 1.905 s 

1957 # - minor tidy: 1.075 to 1.65 

1958 # - remove ORDER BY from task indexing: 1.093 to 1.607 

1959 # - optimize special note code won't affect this: 1.076 to 1.617 

1960 # At this point, entire upload process ~5s. 

1961 # - big difference from commit_table() query optimization 

1962 # - huge difference from being more careful with mark_table_dirty() 

1963 # - further table scanning optimizations: fewer queries 

1964 # Overall upload down to ~2.4s 

1965 

1966 

1967def commit_table(req: "CamcopsRequest", 

1968 batchdetails: BatchDetails, 

1969 table: Table, 

1970 clear_dirty: bool = True) -> UploadTableChanges: 

1971 """ 

1972 Commits additions, removals, and preservations for one table. 

1973 

1974 Should ONLY be called by :func:`commit_all`. 

1975 

1976 Also updates task indexes. 

1977 

1978 Args: 

1979 req: the :class:`camcops_server.cc_modules.cc_request.CamcopsRequest` 

1980 batchdetails: the :class:`BatchDetails` 

1981 table: SQLAlchemy :class:`Table` 

1982 clear_dirty: remove the table from the record of dirty tables for 

1983 this device? (If called from :func:`commit_all`, this should be 

1984 ``False``, since it's faster to clear all dirty tables for the 

1985 device simultaneously than one-by-one.) 

1986 

1987 Returns: 

1988 an :class:`UploadTableChanges` object 

1989 """ 

1990 

1991 # Tried storing PKs in temporary tables, rather than using an IN clause 

1992 # with Python values, as per 

1993 # https://www.xaprb.com/blog/2006/06/28/why-large-in-clauses-are-problematic/ # noqa 

1994 # However, it was slow. 

1995 # We can gain a lot of efficiency (empirically) by: 

1996 # - Storing PKs in Python 

1997 # - Only performing updates when we need to 

1998 # - Using a single query per table to get "add/remove/preserve" PKs 

1999 

2000 # ------------------------------------------------------------------------- 

2001 # Helpful temporary variables 

2002 # ------------------------------------------------------------------------- 

2003 user_id = req.user_id 

2004 device_id = req.tabletsession.device_id 

2005 exacttime = req.now 

2006 dbsession = req.dbsession 

2007 tablename = table.name 

2008 batchtime = batchdetails.batchtime 

2009 preserving = batchdetails.preserving 

2010 

2011 # ------------------------------------------------------------------------- 

2012 # Fetch addition, removal, preservation, current PKs in a single query 

2013 # ------------------------------------------------------------------------- 

2014 tablechanges = UploadTableChanges(table) 

2015 serverrecs = get_server_live_records(req, device_id, table, 

2016 current_only=False) 

2017 for sr in serverrecs: 

2018 tablechanges.note_serverrec(sr, preserving=preserving) 

2019 

2020 # ------------------------------------------------------------------------- 

2021 # Additions 

2022 # ------------------------------------------------------------------------- 

2023 # Update the records we're adding 

2024 addition_pks = tablechanges.addition_pks 

2025 if addition_pks: 

2026 # log.debug("commit_table: {}, adding server PKs {}", 

2027 # tablename, addition_pks) 

2028 dbsession.execute( 

2029 update(table) 

2030 .where(table.c[FN_PK].in_(addition_pks)) 

2031 .values({ 

2032 FN_CURRENT: 1, 

2033 FN_ADDITION_PENDING: 0, 

2034 FN_ADDING_USER_ID: user_id, 

2035 FN_WHEN_ADDED_EXACT: exacttime, 

2036 FN_WHEN_ADDED_BATCH_UTC: batchtime 

2037 }) 

2038 ) 

2039 

2040 # ------------------------------------------------------------------------- 

2041 # Removals 

2042 # ------------------------------------------------------------------------- 

2043 # Update the records we're removing 

2044 removal_pks = tablechanges.removal_pks 

2045 if removal_pks: 

2046 # log.debug("commit_table: {}, removing server PKs {}", 

2047 # tablename, removal_pks) 

2048 dbsession.execute( 

2049 update(table) 

2050 .where(table.c[FN_PK].in_(removal_pks)) 

2051 .values(values_delete_now(req, batchdetails)) 

2052 ) 

2053 

2054 # ------------------------------------------------------------------------- 

2055 # Preservation 

2056 # ------------------------------------------------------------------------- 

2057 # Preserve necessary records 

2058 preservation_pks = tablechanges.preservation_pks 

2059 if preservation_pks: 

2060 # log.debug("commit_table: {}, preserving server PKs {}", 

2061 # tablename, preservation_pks) 

2062 new_era = batchdetails.new_era 

2063 dbsession.execute( 

2064 update(table) 

2065 .where(table.c[FN_PK].in_(preservation_pks)) 

2066 .values(values_preserve_now(req, batchdetails)) 

2067 ) 

2068 if not preserving: 

2069 # Also preserve/finalize any corresponding special notes 

2070 # (2015-02-01), just for records being specifically preserved. If 

2071 # we are preserving, this step happens in one go in commit_all() 

2072 # (2018-11-13). 

2073 # noinspection PyUnresolvedReferences 

2074 dbsession.execute( 

2075 update(SpecialNote.__table__) 

2076 .where(SpecialNote.basetable == tablename) 

2077 .where(SpecialNote.device_id == device_id) 

2078 .where(SpecialNote.era == ERA_NOW) 

2079 .where(exists().select_from(table) 

2080 .where(table.c[TABLET_ID_FIELD] == SpecialNote.task_id) 

2081 .where(table.c[FN_DEVICE_ID] == SpecialNote.device_id) 

2082 .where(table.c[FN_ERA] == new_era)) 

2083 # ^^^^^^^^^^^^^^^^^^^^^^^^^^ 

2084 # This bit restricts to records being preserved. 

2085 .values(era=new_era) 

2086 ) 

2087 

2088 # ------------------------------------------------------------------------- 

2089 # Update special indexes 

2090 # ------------------------------------------------------------------------- 

2091 update_indexes_and_push_exports(req, batchdetails, tablechanges) 

2092 

2093 # ------------------------------------------------------------------------- 

2094 # Remove individually from list of dirty tables? 

2095 # ------------------------------------------------------------------------- 

2096 if clear_dirty: 

2097 # noinspection PyUnresolvedReferences 

2098 dbsession.execute( 

2099 DirtyTable.__table__.delete() 

2100 .where(DirtyTable.device_id == device_id) 

2101 .where(DirtyTable.tablename == tablename) 

2102 ) 

2103 # ... otherwise a call to clear_dirty_tables() must be made. 

2104 

2105 if DEBUG_UPLOAD: 

2106 log.debug("commit_table: {}", tablechanges) 

2107 

2108 return tablechanges 

2109 

2110 

2111def rollback_all(req: "CamcopsRequest") -> None: 

2112 """ 

2113 Rolls back all pending changes for a device. 

2114 """ 

2115 tables = get_dirty_tables(req) 

2116 for table in tables: 

2117 rollback_table(req, table) 

2118 clear_dirty_tables(req) 

2119 

2120 

2121def rollback_table(req: "CamcopsRequest", table: Table) -> None: 

2122 """ 

2123 Rolls back changes for an individual table for a device. 

2124 """ 

2125 device_id = req.tabletsession.device_id 

2126 # Pending additions 

2127 req.dbsession.execute( 

2128 table.delete() 

2129 .where(table.c[FN_DEVICE_ID] == device_id) 

2130 .where(table.c[FN_ADDITION_PENDING]) 

2131 ) 

2132 # Pending deletions 

2133 req.dbsession.execute( 

2134 update(table) 

2135 .where(table.c[FN_DEVICE_ID] == device_id) 

2136 .where(table.c[FN_REMOVAL_PENDING]) 

2137 .values({ 

2138 FN_REMOVAL_PENDING: 0, 

2139 FN_WHEN_ADDED_EXACT: None, 

2140 FN_WHEN_REMOVED_BATCH_UTC: None, 

2141 FN_REMOVING_USER_ID: None, 

2142 FN_SUCCESSOR_PK: None 

2143 }) 

2144 ) 

2145 # Record-specific preservation (set by flag_record_for_preservation()) 

2146 req.dbsession.execute( 

2147 update(table) 

2148 .where(table.c[FN_DEVICE_ID] == device_id) 

2149 .values({ 

2150 MOVE_OFF_TABLET_FIELD: 0 

2151 }) 

2152 ) 

2153 

2154 

2155def clear_dirty_tables(req: "CamcopsRequest") -> None: 

2156 """ 

2157 Clears the dirty-table list for a device. 

2158 """ 

2159 device_id = req.tabletsession.device_id 

2160 # noinspection PyUnresolvedReferences 

2161 req.dbsession.execute( 

2162 DirtyTable.__table__.delete() 

2163 .where(DirtyTable.device_id == device_id) 

2164 ) 

2165 

2166 

2167# ============================================================================= 

2168# Additional helper functions for one-step upload 

2169# ============================================================================= 

2170 

2171def process_table_for_onestep_upload( 

2172 req: "CamcopsRequest", 

2173 batchdetails: BatchDetails, 

2174 table: Table, 

2175 clientpk_name: str, 

2176 rows: List[Dict[str, Any]]) -> UploadTableChanges: 

2177 """ 

2178 Performs all upload steps for a table. 

2179 

2180 Note that we arrive here in a specific and safe table order; search for 

2181 :func:`camcops_server.cc_modules.cc_client_api_helpers.upload_commit_order_sorter`. 

2182 

2183 Args: 

2184 req: the :class:`camcops_server.cc_modules.cc_request.CamcopsRequest` 

2185 batchdetails: the :class:`BatchDetails` 

2186 table: an SQLAlchemy :class:`Table` 

2187 clientpk_name: the name of the PK field on the client 

2188 rows: a list of rows, where each row is a dictionary mapping field 

2189 (column) names to values (those values being encoded as SQL-style 

2190 literals in our extended syntax) 

2191 

2192 Returns: 

2193 an :class:`UploadTableChanges` object 

2194 """ # noqa 

2195 serverrecs = get_server_live_records( 

2196 req, req.tabletsession.device_id, table, clientpk_name, 

2197 current_only=False) 

2198 servercurrentrecs = [r for r in serverrecs if r.current] 

2199 if rows and not clientpk_name: 

2200 fail_user_error(f"Client-side PK name not specified by client for " 

2201 f"non-empty table {table.name!r}") 

2202 tablechanges = UploadTableChanges(table) 

2203 server_pks_uploaded = [] # type: List[int] 

2204 for row in rows: 

2205 valuedict = {k: decode_single_value(v) for k, v in row.items()} 

2206 urr = upload_record_core(req, batchdetails, table, 

2207 clientpk_name, valuedict, 

2208 server_live_current_records=servercurrentrecs) 

2209 # ... handles addition, modification, preservation, special processing 

2210 # But we also make a note of these for indexing: 

2211 if urr.oldserverpk is not None: 

2212 server_pks_uploaded.append(urr.oldserverpk) 

2213 tablechanges.note_urr(urr, 

2214 preserving_new_records=batchdetails.preserving) 

2215 # Which leaves: 

2216 # (*) Deletion (where no record was uploaded at all) 

2217 server_pks_for_deletion = [r.server_pk for r in servercurrentrecs 

2218 if r.server_pk not in server_pks_uploaded] 

2219 if server_pks_for_deletion: 

2220 flag_deleted(req, batchdetails, table, server_pks_for_deletion) 

2221 tablechanges.note_removal_deleted_pks(server_pks_for_deletion) 

2222 

2223 # Preserving all records not specifically processed above, too 

2224 if batchdetails.preserving: 

2225 # Preserve all, including noncurrent: 

2226 preserve_all(req, batchdetails, table) 

2227 # Note other preserved records, for indexing: 

2228 tablechanges.note_preservation_pks(r.server_pk for r in serverrecs) 

2229 

2230 # (*) Indexing (and push exports) 

2231 update_indexes_and_push_exports(req, batchdetails, tablechanges) 

2232 

2233 if DEBUG_UPLOAD: 

2234 log.debug("process_table_for_onestep_upload: {}", tablechanges) 

2235 

2236 return tablechanges 

2237 

2238 

2239# ============================================================================= 

2240# Audit functions 

2241# ============================================================================= 

2242 

2243def audit(req: "CamcopsRequest", 

2244 details: str, 

2245 patient_server_pk: int = None, 

2246 tablename: str = None, 

2247 server_pk: int = None) -> None: 

2248 """ 

2249 Audit something. 

2250 """ 

2251 # Add parameters and pass on: 

2252 cc_audit.audit( 

2253 req=req, 

2254 details=details, 

2255 patient_server_pk=patient_server_pk, 

2256 table=tablename, 

2257 server_pk=server_pk, 

2258 device_id=req.tabletsession.device_id, # added 

2259 remote_addr=req.remote_addr, # added 

2260 user_id=req.user_id, # added 

2261 from_console=False, # added 

2262 from_dbclient=True # added 

2263 ) 

2264 

2265 

2266# ============================================================================= 

2267# Helper functions for single-user mode 

2268# ============================================================================= 

2269 

2270def make_single_user_mode_username(client_device_name: str, 

2271 patient_pk: int) -> str: 

2272 """ 

2273 Returns the username for single-user mode. 

2274 """ 

2275 return f"user-{client_device_name}-{patient_pk}" 

2276 

2277 

2278def json_patient_info(patient: Patient) -> str: 

2279 """ 

2280 Converts patient details to a string representation of a JSON list (one 

2281 patient) containing a single JSON dictionary (detailing that patient), with 

2282 keys/formats known to the client. 

2283 

2284 (One item list to be consistent with patients uploaded from the tablet.) 

2285 

2286 Args: 

2287 patient: :class:`camcops_server.cc_modules.cc_patient.Patient` 

2288 """ 

2289 patient_dict = { 

2290 TabletParam.SURNAME: patient.surname, 

2291 TabletParam.FORENAME: patient.forename, 

2292 TabletParam.SEX: patient.sex, 

2293 TabletParam.DOB: format_datetime(patient.dob, 

2294 DateFormat.ISO8601_DATE_ONLY), 

2295 TabletParam.EMAIL: patient.email, 

2296 TabletParam.ADDRESS: patient.address, 

2297 TabletParam.GP: patient.gp, 

2298 TabletParam.OTHER: patient.other, 

2299 } 

2300 for idnum in patient.idnums: 

2301 key = f"{TabletParam.IDNUM_PREFIX}{idnum.which_idnum}" 

2302 patient_dict[key] = idnum.idnum_value 

2303 # One item list to be consistent with patients uploaded from the tablet 

2304 return json.dumps([patient_dict]) 

2305 

2306 

2307def get_single_server_patient(req: "CamcopsRequest") -> Patient: 

2308 """ 

2309 Returns the patient identified by the proquint access key present in this 

2310 request, or raises. 

2311 """ 

2312 _ = req.gettext 

2313 

2314 patient_proquint = get_str_var(req, TabletParam.PATIENT_PROQUINT) 

2315 assert patient_proquint is not None # For type checker 

2316 

2317 try: 

2318 uuid_obj = uuid_from_proquint(patient_proquint) 

2319 except InvalidProquintException: 

2320 # Checksum failed or characters in wrong place 

2321 # We'll do the same validation on the client so in theory 

2322 # should never get here 

2323 fail_user_error( 

2324 _( 

2325 "There is no patient with access key '{access_key}'. " 

2326 "Have you entered the key correctly?" 

2327 ).format(access_key=patient_proquint) 

2328 ) 

2329 

2330 server_device = Device.get_server_device(req.dbsession) 

2331 

2332 # noinspection PyUnboundLocalVariable,PyProtectedMember 

2333 patient = req.dbsession.query(Patient).filter( 

2334 Patient.uuid == uuid_obj, 

2335 Patient._device_id == server_device.id, 

2336 Patient._era == ERA_NOW, 

2337 Patient._current == True # noqa: E712 

2338 ).options(joinedload(Patient.task_schedules)).one_or_none() 

2339 

2340 if patient is None: 

2341 fail_user_error( 

2342 _( 

2343 "There is no patient with access key '{access_key}'. " 

2344 "Have you entered the key correctly?" 

2345 ).format(access_key=patient_proquint) 

2346 ) 

2347 

2348 if not patient.idnums: 

2349 # In theory should never happen. The patient must be created with at 

2350 # least one ID number. We did see this once in testing (possibly when 

2351 # a patient created on a device was registered) 

2352 _ = req.gettext 

2353 fail_server_error(_("Patient has no ID numbers")) 

2354 

2355 return patient 

2356 

2357 

2358def get_or_create_single_user(req: "CamcopsRequest", 

2359 name: str, 

2360 patient: Patient) -> Tuple[User, str]: 

2361 """ 

2362 Creates a user for a patient (who's using single-user mode). 

2363 

2364 The user is associated (via its name) with the combination of a client 

2365 device and a patient. (If a device is re-registered to another patient, the 

2366 username will change.) 

2367 

2368 If the username already exists, then since we can't look up the password 

2369 (it's irreversibly encrypted), we will set it afresh. 

2370 

2371 - Why is a user associated with a patient? So we can enforce that the user 

2372 can upload only data relating to that patient. 

2373 

2374 - Why is a user associated with a device? 

2375 

2376 - If it is: then two users (e.g. "Device1-Bob" and "Device2-Bob") can 

2377 independently work with the same patient. This will be highly 

2378 confusing (mainly because it will allow "double" copies of tasks to be 

2379 created, though only by manually entering things twice). 

2380 

2381 - If it isn't (e.g. user "Bob"): then, because registering the patient on 

2382 Device2 will reset the password for the user, registering a new device 

2383 for a patient will "take over" from a previous device. That has some 

2384 potential for data loss if work was in progress (incomplete tasks won't 

2385 be uploadable any more, and re-registering [to fix the password on the 

2386 first device] would delete data). 

2387 

2388 - Since some confusion is better than some data loss, we associate users 

2389 with a device/patient combination. 

2390 

2391 Args: 

2392 req: 

2393 a :class:`camcops_server.cc_modules.cc_request.CamcopsRequest` 

2394 name: 

2395 username 

2396 patient: 

2397 associated :class:`camcops_server.cc_modules.cc_patient.Patient`, 

2398 which also tells us the group in which to place this user 

2399 

2400 Returns: 

2401 tuple: :class:`camcops_server.cc_modules.cc_user.User`, password 

2402 

2403 """ 

2404 dbsession = req.dbsession 

2405 password = random_password() 

2406 group = patient.group 

2407 assert group is not None # for type checker 

2408 

2409 user = User.get_user_by_name(dbsession, name) 

2410 creating_new_user = user is None 

2411 if creating_new_user: 

2412 # Create a fresh user. 

2413 user = User(username=name) 

2414 user.upload_group = group 

2415 user.auto_generated = True 

2416 user.superuser = False # should be redundant! 

2417 # noinspection PyProtectedMember 

2418 user.single_patient_pk = patient._pk 

2419 user.set_password(req, password) 

2420 if creating_new_user: 

2421 dbsession.add(user) 

2422 # As the username is based on a UUID, we're pretty sure another 

2423 # request won't have created the same user, otherwise we'd need 

2424 # to catch IntegrityError 

2425 dbsession.flush() 

2426 

2427 membership = UserGroupMembership( 

2428 user_id=user.id, 

2429 group_id=group.id, 

2430 ) 

2431 membership.may_register_devices = True 

2432 membership.may_upload = True 

2433 user.user_group_memberships = [membership] # ... only these permissions 

2434 

2435 return user, password 

2436 

2437 

2438def random_password(length: int = 32) -> str: 

2439 """ 

2440 Create a random password. 

2441 """ 

2442 # Not trying anything clever with distributions of letters, digits etc 

2443 characters = string.ascii_letters + string.digits + string.punctuation 

2444 # We use secrets.choice() rather than random.choices() as it's better 

2445 # for security/cryptography purposes. 

2446 return "".join(secrets.choice(characters) for _ in range(length)) 

2447 

2448 

2449def get_task_schedules(req: "CamcopsRequest", 

2450 patient: Patient) -> str: 

2451 """ 

2452 Gets a JSON string representation of the task schedules for a specified 

2453 patient. 

2454 """ 

2455 dbsession = req.dbsession 

2456 

2457 schedules = [] 

2458 

2459 for pts in patient.task_schedules: 

2460 if pts.start_datetime is None: 

2461 # Minutes granularity so we are consistent with the form 

2462 pts.start_datetime = req.now_utc.replace(second=0, microsecond=0) 

2463 dbsession.add(pts) 

2464 

2465 items = [] 

2466 

2467 for task_info in pts.get_list_of_scheduled_tasks(req): 

2468 due_from = task_info.start_datetime.to_iso8601_string() 

2469 due_by = task_info.end_datetime.to_iso8601_string() 

2470 

2471 complete = False 

2472 when_completed = None 

2473 task = task_info.task 

2474 if task: 

2475 complete = task.is_complete() 

2476 if complete and task.when_last_modified: 

2477 when_completed = task.when_last_modified.to_iso8601_string() # noqa 

2478 

2479 if pts.settings is not None: 

2480 settings = pts.settings.get(task_info.tablename, {}) 

2481 else: 

2482 settings = {} 

2483 

2484 items.append({ 

2485 TabletParam.TABLE: task_info.tablename, 

2486 TabletParam.ANONYMOUS: task_info.is_anonymous, 

2487 TabletParam.SETTINGS: settings, 

2488 TabletParam.DUE_FROM: due_from, 

2489 TabletParam.DUE_BY: due_by, 

2490 TabletParam.COMPLETE: complete, 

2491 TabletParam.WHEN_COMPLETED: when_completed, 

2492 }) 

2493 

2494 schedules.append({ 

2495 TabletParam.TASK_SCHEDULE_NAME: pts.task_schedule.name, 

2496 TabletParam.TASK_SCHEDULE_ITEMS: items, 

2497 }) 

2498 

2499 return json.dumps(schedules) 

2500 

2501 

2502# ============================================================================= 

2503# Action processors: allowed to any user 

2504# ============================================================================= 

2505# If they return None, the framework uses the operation name as the reply in 

2506# the success message. Not returning anything is the same as returning None. 

2507# Authentication is performed in advance of these. 

2508 

2509def op_check_device_registered(req: "CamcopsRequest") -> None: 

2510 """ 

2511 Check that a device is registered, or raise 

2512 :exc:`UserErrorException`. 

2513 """ 

2514 req.tabletsession.ensure_device_registered() 

2515 

2516 

2517def op_register_patient(req: "CamcopsRequest") -> Dict[str, Any]: 

2518 """ 

2519 Registers a patient. That is, the client provides an access key. If all 

2520 is well, the server returns details of that patient, as well as key 

2521 server parameters, plus (if required) the username/password to use. 

2522 """ 

2523 # ------------------------------------------------------------------------- 

2524 # Patient details 

2525 # ------------------------------------------------------------------------- 

2526 patient = get_single_server_patient(req) # may fail/raise 

2527 patient_info = json_patient_info(patient) 

2528 reply_dict = { 

2529 TabletParam.PATIENT_INFO: patient_info, 

2530 } 

2531 

2532 # ------------------------------------------------------------------------- 

2533 # Username/password 

2534 # ------------------------------------------------------------------------- 

2535 client_device_name = get_str_var(req, TabletParam.DEVICE) 

2536 # noinspection PyProtectedMember 

2537 user_name = make_single_user_mode_username(client_device_name, patient._pk) 

2538 user, password = get_or_create_single_user(req, user_name, patient) 

2539 reply_dict[TabletParam.USER] = user.username 

2540 reply_dict[TabletParam.PASSWORD] = password 

2541 

2542 # ------------------------------------------------------------------------- 

2543 # Intellectual property settings 

2544 # ------------------------------------------------------------------------- 

2545 ip_use = patient.group.ip_use or IpUse() 

2546 # ... if the group doesn't have an associated ip_use object, use defaults 

2547 ip_dict = { 

2548 TabletParam.IP_USE_COMMERCIAL: int(ip_use.commercial), 

2549 TabletParam.IP_USE_CLINICAL: int(ip_use.clinical), 

2550 TabletParam.IP_USE_EDUCATIONAL: int(ip_use.educational), 

2551 TabletParam.IP_USE_RESEARCH: int(ip_use.research), 

2552 } 

2553 reply_dict[TabletParam.IP_USE_INFO] = json.dumps(ip_dict) 

2554 

2555 return reply_dict 

2556 

2557 

2558# ============================================================================= 

2559# Action processors that require REGISTRATION privilege 

2560# ============================================================================= 

2561 

2562def op_register_device(req: "CamcopsRequest") -> Dict[str, Any]: 

2563 """ 

2564 Register a device with the server. 

2565 

2566 Returns: 

2567 server information dictionary (from :func:`get_server_id_info`) 

2568 """ 

2569 dbsession = req.dbsession 

2570 ts = req.tabletsession 

2571 device_friendly_name = get_str_var(req, TabletParam.DEVICE_FRIENDLY_NAME, 

2572 mandatory=False) 

2573 # noinspection PyUnresolvedReferences 

2574 device_exists = exists_in_table( 

2575 dbsession, 

2576 Device.__table__, 

2577 Device.name == ts.device_name 

2578 ) 

2579 if device_exists: 

2580 # device already registered, but accept re-registration 

2581 # noinspection PyUnresolvedReferences 

2582 dbsession.execute( 

2583 update(Device.__table__) 

2584 .where(Device.name == ts.device_name) 

2585 .values(friendly_name=device_friendly_name, 

2586 camcops_version=ts.tablet_version_str, 

2587 registered_by_user_id=req.user_id, 

2588 when_registered_utc=req.now_utc) 

2589 ) 

2590 else: 

2591 # new registration 

2592 try: 

2593 # noinspection PyUnresolvedReferences 

2594 dbsession.execute( 

2595 Device.__table__.insert() 

2596 .values(name=ts.device_name, 

2597 friendly_name=device_friendly_name, 

2598 camcops_version=ts.tablet_version_str, 

2599 registered_by_user_id=req.user_id, 

2600 when_registered_utc=req.now_utc) 

2601 ) 

2602 except IntegrityError: 

2603 fail_user_error(INSERT_FAILED) 

2604 

2605 ts.reload_device() 

2606 audit( 

2607 req, 

2608 f"register, device_id={ts.device_id}, " 

2609 f"friendly_name={device_friendly_name}", 

2610 tablename=Device.__tablename__ 

2611 ) 

2612 return get_server_id_info(req) 

2613 

2614 

2615def op_get_extra_strings(req: "CamcopsRequest") -> Dict[str, str]: 

2616 """ 

2617 Fetch all local extra strings from the server. 

2618 

2619 Returns: 

2620 a SELECT-style reply (see :func:`get_select_reply`) for the 

2621 extra-string table 

2622 """ 

2623 fields = [ExtraStringFieldNames.TASK, 

2624 ExtraStringFieldNames.NAME, 

2625 ExtraStringFieldNames.LANGUAGE, 

2626 ExtraStringFieldNames.VALUE] 

2627 rows = req.get_all_extra_strings() 

2628 reply = get_select_reply(fields, rows) 

2629 audit(req, "get_extra_strings") 

2630 return reply 

2631 

2632 

2633# noinspection PyUnusedLocal 

2634def op_get_allowed_tables(req: "CamcopsRequest") -> Dict[str, str]: 

2635 """ 

2636 Returns the names of all possible tables on the server, each paired with 

2637 the minimum client (tablet) version that will be accepted for each table. 

2638 (Typically these are all the same as the minimum global tablet version.) 

2639 

2640 Uses the SELECT-like syntax (see :func:`get_select_reply`). 

2641 """ 

2642 tables_versions = all_tables_with_min_client_version() 

2643 fields = [AllowedTablesFieldNames.TABLENAME, 

2644 AllowedTablesFieldNames.MIN_CLIENT_VERSION] 

2645 rows = [[k, str(v)] for k, v in tables_versions.items()] 

2646 reply = get_select_reply(fields, rows) 

2647 audit(req, "get_allowed_tables") 

2648 return reply 

2649 

2650 

2651def op_get_task_schedules(req: "CamcopsRequest") -> Dict[str, str]: 

2652 """ 

2653 Return details of the task schedules for the patient associated with 

2654 this request, for single-user mode. Also returns details of the single 

2655 patient, in case that's changed. 

2656 """ 

2657 patient = get_single_server_patient(req) 

2658 patient_info = json_patient_info(patient) 

2659 task_schedules = get_task_schedules(req, patient) 

2660 return { 

2661 TabletParam.PATIENT_INFO: patient_info, 

2662 TabletParam.TASK_SCHEDULES: task_schedules, 

2663 } 

2664 

2665 

2666# ============================================================================= 

2667# Action processors that require UPLOAD privilege 

2668# ============================================================================= 

2669 

2670# noinspection PyUnusedLocal 

2671def op_check_upload_user_and_device(req: "CamcopsRequest") -> None: 

2672 """ 

2673 Stub function for the operation to check that a user is valid. 

2674 

2675 To get this far, the user has to be valid, so this function doesn't 

2676 actually have to do anything. 

2677 """ 

2678 pass # don't need to do anything! 

2679 

2680 

2681# noinspection PyUnusedLocal 

2682def op_get_id_info(req: "CamcopsRequest") -> Dict[str, Any]: 

2683 """ 

2684 Fetch server ID information; see :func:`get_server_id_info`. 

2685 """ 

2686 return get_server_id_info(req) 

2687 

2688 

2689def op_start_upload(req: "CamcopsRequest") -> None: 

2690 """ 

2691 Begin an upload. 

2692 """ 

2693 start_device_upload_batch(req) 

2694 

2695 

2696def op_end_upload(req: "CamcopsRequest") -> None: 

2697 """ 

2698 Ends an upload and commits changes. 

2699 """ 

2700 batchdetails = get_batch_details(req) 

2701 # ensure it's the same user finishing as starting! 

2702 end_device_upload_batch(req, batchdetails) 

2703 

2704 

2705def op_upload_table(req: "CamcopsRequest") -> str: 

2706 """ 

2707 Upload a table. 

2708 

2709 Incoming information in the POST request includes a CSV list of fields, a 

2710 count of the number of records being provided, and a set of variables named 

2711 ``record0`` ... ``record{nrecords - 1}``, each containing a CSV list of 

2712 SQL-encoded values. 

2713 

2714 Typically used for smaller tables, i.e. most except for BLOBs. 

2715 """ 

2716 table = get_table_from_req(req, TabletParam.TABLE) 

2717 

2718 allowed_nonexistent_fields = [] # type: List[str] 

2719 # noinspection PyUnresolvedReferences 

2720 if req.tabletsession.cope_with_old_idnums and table == Patient.__table__: 

2721 for x in range(1, NUMBER_OF_IDNUMS_DEFUNCT + 1): 

2722 allowed_nonexistent_fields.extend([ 

2723 FP_ID_NUM + str(x), 

2724 FP_ID_DESC + str(x), 

2725 FP_ID_SHORT_DESC + str(x) 

2726 ]) 

2727 

2728 fields = get_fields_from_post_var( 

2729 req, table, TabletParam.FIELDS, 

2730 allowed_nonexistent_fields=allowed_nonexistent_fields) 

2731 nrecords = get_int_var(req, TabletParam.NRECORDS) 

2732 

2733 nfields = len(fields) 

2734 if nfields < 1: 

2735 fail_user_error( 

2736 f"{TabletParam.FIELDS}={nfields}: can't be less than 1") 

2737 if nrecords < 0: 

2738 fail_user_error( 

2739 f"{TabletParam.NRECORDS}={nrecords}: can't be less than 0") 

2740 

2741 batchdetails = get_batch_details(req) 

2742 

2743 ts = req.tabletsession 

2744 if ts.explicit_pkname_for_upload_table: # q.v. 

2745 # New client: tells us the PK name explicitly. 

2746 clientpk_name = get_single_field_from_post_var(req, table, 

2747 TabletParam.PKNAME) 

2748 else: 

2749 # Old client. Either (a) old Titanium client, in which the client PK 

2750 # is in fields[0], or (b) an early C++ client, in which there was no 

2751 # guaranteed order (and no explicit PK name was sent). However, in 

2752 # either case, the client PK name was (is) always "id". 

2753 clientpk_name = TABLET_ID_FIELD 

2754 ensure_valid_field_name(table, clientpk_name) 

2755 server_pks_uploaded = [] # type: List[int] 

2756 n_new = 0 

2757 n_modified = 0 

2758 n_identical = 0 

2759 dirty = False 

2760 serverrecs = get_server_live_records(req, ts.device_id, table, 

2761 clientpk_name=clientpk_name, 

2762 current_only=True) 

2763 for r in range(nrecords): 

2764 recname = TabletParam.RECORD_PREFIX + str(r) 

2765 values = get_values_from_post_var(req, recname) 

2766 nvalues = len(values) 

2767 if nvalues != nfields: 

2768 errmsg = ( 

2769 f"Number of fields in field list ({nfields}) doesn't match " 

2770 f"number of values in record {r} ({nvalues})" 

2771 ) 

2772 log.warning(errmsg + f"\nfields: {fields!r}\nvalues: {values!r}") 

2773 fail_user_error(errmsg) 

2774 valuedict = dict(zip(fields, values)) 

2775 # log.debug("table {!r}, record {}: {!r}", table.name, r, valuedict) 

2776 # CORE: CALLS upload_record_core 

2777 urr = upload_record_core( 

2778 req, batchdetails, table, clientpk_name, valuedict, 

2779 server_live_current_records=serverrecs) 

2780 if urr.oldserverpk is not None: # was an existing record 

2781 server_pks_uploaded.append(urr.oldserverpk) 

2782 if urr.newserverpk is None: 

2783 n_identical += 1 

2784 else: 

2785 n_modified += 1 

2786 else: # entirely new 

2787 n_new += 1 

2788 if urr.dirty: 

2789 dirty = True 

2790 

2791 # Now deal with any ABSENT (not in uploaded data set) conditions. 

2792 server_pks_for_deletion = [r.server_pk for r in serverrecs 

2793 if r.server_pk not in server_pks_uploaded] 

2794 # Note that "deletion" means "end of the line"; records that are modified 

2795 # and replaced were handled by upload_record_core(). 

2796 n_deleted = len(server_pks_for_deletion) 

2797 if n_deleted > 0: 

2798 flag_deleted(req, batchdetails, table, server_pks_for_deletion) 

2799 

2800 # Set dirty/clean status 

2801 if (dirty or n_new > 0 or n_modified > 0 or n_deleted > 0 or 

2802 any(sr.move_off_tablet for sr in serverrecs)): 

2803 # ... checks on n_new and n_modified are redundant; dirty will be True 

2804 mark_table_dirty(req, table) 

2805 elif batchdetails.preserving and not serverrecs: 

2806 # We've scanned this table, and there would be no work to do to 

2807 # preserve records from previous uploads. 

2808 mark_table_clean(req, table) 

2809 

2810 # Special for old tablets: 

2811 # noinspection PyUnresolvedReferences 

2812 if req.tabletsession.cope_with_old_idnums and table == Patient.__table__: 

2813 # noinspection PyUnresolvedReferences 

2814 mark_table_dirty(req, PatientIdNum.__table__) 

2815 # Mark patient ID numbers for deletion if their parent Patient is 

2816 # similarly being marked for deletion 

2817 # noinspection PyUnresolvedReferences,PyProtectedMember 

2818 req.dbsession.execute( 

2819 update(PatientIdNum.__table__) 

2820 .where(PatientIdNum._device_id == Patient._device_id) 

2821 .where(PatientIdNum._era == ERA_NOW) 

2822 .where(PatientIdNum.patient_id == Patient.id) 

2823 .where(Patient._pk.in_(server_pks_for_deletion)) 

2824 .where(Patient._era == ERA_NOW) # shouldn't be in doubt! 

2825 .values(_removal_pending=1, 

2826 _successor_pk=None) 

2827 ) 

2828 

2829 # Auditing occurs at commit_all. 

2830 log.info("Upload successful; {n} records uploaded to table {t} " 

2831 "({new} new, {mod} modified, {i} identical, {nd} deleted)", 

2832 n=nrecords, t=table.name, new=n_new, mod=n_modified, 

2833 i=n_identical, nd=n_deleted) 

2834 return f"Table {table.name} upload successful" 

2835 

2836 

2837def op_upload_record(req: "CamcopsRequest") -> str: 

2838 """ 

2839 Upload an individual record. (Typically used for BLOBs.) 

2840 Incoming POST information includes a CSV list of fields and a CSV list of 

2841 values. 

2842 """ 

2843 batchdetails = get_batch_details(req) 

2844 table = get_table_from_req(req, TabletParam.TABLE) 

2845 clientpk_name = get_single_field_from_post_var(req, table, 

2846 TabletParam.PKNAME) 

2847 valuedict = get_fields_and_values(req, table, 

2848 TabletParam.FIELDS, TabletParam.VALUES) 

2849 urr = upload_record_core( 

2850 req, batchdetails, table, clientpk_name, valuedict 

2851 ) 

2852 if urr.dirty: 

2853 mark_table_dirty(req, table) 

2854 if urr.oldserverpk is None: 

2855 log.info("upload-insert") 

2856 return "UPLOAD-INSERT" 

2857 else: 

2858 if urr.newserverpk is None: 

2859 log.info("upload-update: skipping existing record") 

2860 else: 

2861 log.info("upload-update") 

2862 return "UPLOAD-UPDATE" 

2863 # Auditing occurs at commit_all. 

2864 

2865 

2866def op_upload_empty_tables(req: "CamcopsRequest") -> str: 

2867 """ 

2868 The tablet supplies a list of tables that are empty at its end, and we 

2869 will 'wipe' all appropriate tables; this reduces the number of HTTP 

2870 requests. 

2871 """ 

2872 tables = get_tables_from_post_var(req, TabletParam.TABLES) 

2873 batchdetails = get_batch_details(req) 

2874 to_dirty = [] # type: List[Table] 

2875 to_clean = [] # type: List[Table] 

2876 for table in tables: 

2877 nrows_affected = flag_all_records_deleted(req, table) 

2878 if nrows_affected > 0: 

2879 to_dirty.append(table) 

2880 elif batchdetails.preserving: 

2881 # There are no records in the current era for this device. 

2882 to_clean.append(table) 

2883 # In the fewest number of queries: 

2884 mark_tables_dirty(req, to_dirty) 

2885 mark_tables_clean(req, to_clean) 

2886 log.info("upload_empty_tables") 

2887 # Auditing occurs at commit_all. 

2888 return "UPLOAD-EMPTY-TABLES" 

2889 

2890 

2891def op_start_preservation(req: "CamcopsRequest") -> str: 

2892 """ 

2893 Marks this upload batch as one in which all records will be preserved 

2894 (i.e. moved from NOW-era to an older era, so they can be deleted safely 

2895 from the tablet). 

2896 

2897 Without this, individual records can still be marked for preservation if 

2898 their MOVE_OFF_TABLET_FIELD field (``_move_off_tablet``) is set; see 

2899 :func:`upload_record` and the functions it calls. 

2900 """ 

2901 get_batch_details(req) 

2902 start_preserving(req) 

2903 log.info("start_preservation successful") 

2904 # Auditing occurs at commit_all. 

2905 return "STARTPRESERVATION" 

2906 

2907 

2908def op_delete_where_key_not(req: "CamcopsRequest") -> str: 

2909 """ 

2910 Marks records for deletion, for a device/table, where the client PK 

2911 is not in a specified list. 

2912 """ 

2913 table = get_table_from_req(req, TabletParam.TABLE) 

2914 clientpk_name = get_single_field_from_post_var( 

2915 req, table, TabletParam.PKNAME) 

2916 clientpk_values = get_values_from_post_var(req, TabletParam.PKVALUES) 

2917 

2918 get_batch_details(req) 

2919 flag_deleted_where_clientpk_not(req, table, clientpk_name, clientpk_values) 

2920 # Auditing occurs at commit_all. 

2921 # log.info("delete_where_key_not successful; table {} trimmed", table) 

2922 return "Trimmed" 

2923 

2924 

2925def op_which_keys_to_send(req: "CamcopsRequest") -> str: 

2926 """ 

2927 Intended use: "For my device, and a specified table, here are my client- 

2928 side PKs (as a CSV list), and the modification dates for each corresponding 

2929 record (as a CSV list). Please tell me which records have mismatching dates 

2930 on the server, i.e. those that I need to re-upload." 

2931 

2932 Used particularly for BLOBs, to reduce traffic, i.e. so we don't have to 

2933 send a lot of BLOBs. 

2934 

2935 Note new ``TabletParam.MOVE_OFF_TABLET_VALUES`` parameter in server v2.3.0, 

2936 with bugfix for pre-2.3.0 clients that won't send this; see changelog. 

2937 """ 

2938 # ------------------------------------------------------------------------- 

2939 # Get details 

2940 # ------------------------------------------------------------------------- 

2941 try: 

2942 table = get_table_from_req(req, TabletParam.TABLE) 

2943 except IgnoringAntiqueTableException: 

2944 raise IgnoringAntiqueTableException("") 

2945 clientpk_name = get_single_field_from_post_var(req, table, 

2946 TabletParam.PKNAME) 

2947 clientpk_values = get_values_from_post_var(req, TabletParam.PKVALUES, 

2948 mandatory=False) 

2949 # ... should be autoconverted to int, but we check below 

2950 client_dates = get_values_from_post_var(req, TabletParam.DATEVALUES, 

2951 mandatory=False) 

2952 # ... will be in string format 

2953 

2954 npkvalues = len(clientpk_values) 

2955 ndatevalues = len(client_dates) 

2956 if npkvalues != ndatevalues: 

2957 fail_user_error( 

2958 f"Number of PK values ({npkvalues}) doesn't match number of dates " 

2959 f"({ndatevalues})") 

2960 

2961 # v2.3.0: 

2962 move_off_tablet_values = [] # type: List[int] # for type checker 

2963 if req.has_param(TabletParam.MOVE_OFF_TABLET_VALUES): 

2964 client_reports_move_off_tablet = True 

2965 move_off_tablet_values = get_values_from_post_var( 

2966 req, TabletParam.MOVE_OFF_TABLET_VALUES, mandatory=True) 

2967 # ... should be autoconverted to int 

2968 n_motv = len(move_off_tablet_values) 

2969 if n_motv != npkvalues: 

2970 fail_user_error( 

2971 f"Number of move-off-tablet values ({n_motv}) doesn't match " 

2972 f"number of PKs ({npkvalues})") 

2973 try: 

2974 move_off_tablet_values = [bool(x) for x in move_off_tablet_values] 

2975 except (TypeError, ValueError): 

2976 fail_user_error( 

2977 f"Bad move-off-tablet values: {move_off_tablet_values!r}") 

2978 else: 

2979 client_reports_move_off_tablet = False 

2980 log.warning( 

2981 "op_which_keys_to_send: old client not reporting " 

2982 "{}; requesting all records", 

2983 TabletParam.MOVE_OFF_TABLET_VALUES 

2984 ) 

2985 

2986 clientinfo = [] # type: List[WhichKeyToSendInfo] 

2987 

2988 for i in range(npkvalues): 

2989 cpkv = clientpk_values[i] 

2990 if not isinstance(cpkv, int): 

2991 fail_user_error(f"Bad (non-integer) client PK: {cpkv!r}") 

2992 dt = None # for type checker 

2993 try: 

2994 dt = coerce_to_pendulum(client_dates[i]) 

2995 if dt is None: 

2996 fail_user_error(f"Missing date/time for client PK {cpkv}") 

2997 except ValueError: 

2998 fail_user_error(f"Bad date/time: {client_dates[i]!r}") 

2999 clientinfo.append(WhichKeyToSendInfo( 

3000 client_pk=cpkv, 

3001 client_when=dt, 

3002 client_move_off_tablet=( 

3003 move_off_tablet_values[i] 

3004 if client_reports_move_off_tablet else False 

3005 ) 

3006 )) 

3007 

3008 # ------------------------------------------------------------------------- 

3009 # Work out the answer 

3010 # ------------------------------------------------------------------------- 

3011 batchdetails = get_batch_details(req) 

3012 

3013 # 1. The client sends us all its PKs. So "delete" anything not in that 

3014 # list. 

3015 flag_deleted_where_clientpk_not(req, table, clientpk_name, clientpk_values) 

3016 

3017 # 2. See which ones are new or updates. 

3018 client_pks_needed = [] # type: List[int] 

3019 client_pk_to_serverrec = client_pks_that_exist( 

3020 req, table, clientpk_name, clientpk_values) 

3021 for wk in clientinfo: 

3022 if client_reports_move_off_tablet: 

3023 if wk.client_pk not in client_pk_to_serverrec: 

3024 # New on the client; we want it 

3025 client_pks_needed.append(wk.client_pk) 

3026 else: 

3027 # We know about some version of this client record. 

3028 serverrec = client_pk_to_serverrec[wk.client_pk] 

3029 if serverrec.server_when != wk.client_when: 

3030 # Modified on the client; we want it 

3031 client_pks_needed.append(wk.client_pk) 

3032 elif serverrec.move_off_tablet != wk.client_move_off_tablet: 

3033 # Not modified on the client. But it is being preserved. 

3034 # We don't need to ask the client for it again, but we do 

3035 # need to mark the preservation. 

3036 flag_record_for_preservation(req, batchdetails, table, 

3037 serverrec.server_pk) 

3038 

3039 else: 

3040 # Client hasn't told us about the _move_off_tablet flag. Always 

3041 # request the record (workaround potential bug in old clients). 

3042 client_pks_needed.append(wk.client_pk) 

3043 

3044 # Success 

3045 pk_csv_list = ",".join([str(x) for x in client_pks_needed if x is not None]) # noqa 

3046 # log.info("which_keys_to_send successful: table {}", table.name) 

3047 return pk_csv_list 

3048 

3049 

3050def op_validate_patients(req: "CamcopsRequest") -> str: 

3051 """ 

3052 As of v2.3.0, the client can use this command to validate patients against 

3053 arbitrary server criteria -- definitely the upload/finalize ID policies, 

3054 but potentially also other criteria of the server's (like matching against 

3055 a bank of predefined patients). 

3056 

3057 Compare ``NetworkManager::getPatientInfoJson()`` on the client. 

3058 

3059 There is a slight weakness with respect to "single-patient" users, in that 

3060 the client *asks* if the patients are OK (rather than the server 

3061 *enforcing* that they are OK, via hooks into :func:`op_upload_table`, 

3062 :func:`op_upload_record`, :func:`op_upload_entire_database` -- made more 

3063 complex because ID numbers are not uploaded to the same table...). In 

3064 principle, the weakness is that a user could (a) crack their assigned 

3065 password and (b) rework the CamCOPS client, in order to upload "bad" 

3066 patient data into their assigned group. 

3067 

3068 todo: 

3069 address this by having the server *require* patient validation for 

3070 all uploads? 

3071 

3072 """ 

3073 pt_json_list = get_json_from_post_var(req, TabletParam.PATIENT_INFO, 

3074 decoder=PATIENT_INFO_JSON_DECODER, 

3075 mandatory=True) 

3076 if not isinstance(pt_json_list, list): 

3077 fail_user_error("Top-level JSON is not a list") 

3078 group = Group.get_group_by_id(req.dbsession, req.user.upload_group_id) 

3079 for pt_dict in pt_json_list: 

3080 ensure_valid_patient_json(req, group, pt_dict) 

3081 return SUCCESS_MSG 

3082 

3083 

3084def op_upload_entire_database(req: "CamcopsRequest") -> str: 

3085 """ 

3086 Perform a one-step upload of the entire database. 

3087 

3088 - From v2.3.0. 

3089 - Therefore, we do not have to cope with old-style ID numbers. 

3090 """ 

3091 # Roll back and clear any outstanding changes 

3092 clear_device_upload_batch(req) 

3093 

3094 # Fetch the JSON, with sanity checks 

3095 preserving = get_bool_int_var(req, TabletParam.FINALIZING) 

3096 pknameinfo = get_json_from_post_var( 

3097 req, TabletParam.PKNAMEINFO, decoder=DB_JSON_DECODER, mandatory=True) 

3098 if not isinstance(pknameinfo, dict): 

3099 fail_user_error("PK name info JSON is not a dict") 

3100 dbdata = get_json_from_post_var( 

3101 req, TabletParam.DBDATA, decoder=DB_JSON_DECODER, mandatory=True) 

3102 if not isinstance(dbdata, dict): 

3103 fail_user_error("Database data JSON is not a dict") 

3104 

3105 # Sanity checks 

3106 dbdata_tablenames = sorted(dbdata.keys()) 

3107 pkinfo_tablenames = sorted(pknameinfo.keys()) 

3108 if pkinfo_tablenames != dbdata_tablenames: 

3109 fail_user_error("Table names don't match from (1) DB data (2) PK info") 

3110 duff_tablenames = sorted(list(set(dbdata_tablenames) - 

3111 set(CLIENT_TABLE_MAP.keys()))) 

3112 if duff_tablenames: 

3113 fail_user_error( 

3114 f"Attempt to upload nonexistent tables: {duff_tablenames!r}") 

3115 

3116 # Perform the upload 

3117 batchdetails = BatchDetails(req.now_utc, preserving=preserving, 

3118 onestep=True) # NB special "onestep" option 

3119 # Process the tables in a certain order: 

3120 tables = sorted(CLIENT_TABLE_MAP.values(), 

3121 key=upload_commit_order_sorter) 

3122 changelist = [] # type: List[UploadTableChanges] 

3123 for table in tables: 

3124 clientpk_name = pknameinfo.get(table.name, "") 

3125 rows = dbdata.get(table.name, []) 

3126 tablechanges = process_table_for_onestep_upload( 

3127 req, batchdetails, table, clientpk_name, rows) 

3128 changelist.append(tablechanges) 

3129 

3130 # Audit 

3131 audit_upload(req, changelist) 

3132 

3133 # Done 

3134 return SUCCESS_MSG 

3135 

3136 

3137# ============================================================================= 

3138# Action maps 

3139# ============================================================================= 

3140 

3141class Operations: 

3142 """ 

3143 Constants giving the name of operations (commands) accepted by this API. 

3144 """ 

3145 CHECK_DEVICE_REGISTERED = "check_device_registered" 

3146 CHECK_UPLOAD_USER_DEVICE = "check_upload_user_and_device" 

3147 DELETE_WHERE_KEY_NOT = "delete_where_key_not" 

3148 END_UPLOAD = "end_upload" 

3149 GET_ALLOWED_TABLES = "get_allowed_tables" # v2.2.0 

3150 GET_EXTRA_STRINGS = "get_extra_strings" 

3151 GET_ID_INFO = "get_id_info" 

3152 GET_TASK_SCHEDULES = "get_task_schedules" # v2.4.0 

3153 REGISTER = "register" 

3154 REGISTER_PATIENT = "register_patient" # v2.4.0 

3155 START_PRESERVATION = "start_preservation" 

3156 START_UPLOAD = "start_upload" 

3157 UPLOAD_EMPTY_TABLES = "upload_empty_tables" 

3158 UPLOAD_ENTIRE_DATABASE = "upload_entire_database" # v2.3.0 

3159 UPLOAD_RECORD = "upload_record" 

3160 UPLOAD_TABLE = "upload_table" 

3161 VALIDATE_PATIENTS = "validate_patients" # v2.3.0 

3162 WHICH_KEYS_TO_SEND = "which_keys_to_send" 

3163 

3164 

3165OPERATIONS_ANYONE = { 

3166 Operations.CHECK_DEVICE_REGISTERED: op_check_device_registered, 

3167 # Anyone can register a patient provided they have the right unique code 

3168 Operations.REGISTER_PATIENT: op_register_patient, 

3169} 

3170OPERATIONS_REGISTRATION = { 

3171 Operations.GET_ALLOWED_TABLES: op_get_allowed_tables, # v2.2.0 

3172 Operations.GET_EXTRA_STRINGS: op_get_extra_strings, 

3173 Operations.GET_TASK_SCHEDULES: op_get_task_schedules, 

3174 Operations.REGISTER: op_register_device, 

3175} 

3176OPERATIONS_UPLOAD = { 

3177 Operations.CHECK_UPLOAD_USER_DEVICE: op_check_upload_user_and_device, 

3178 Operations.DELETE_WHERE_KEY_NOT: op_delete_where_key_not, 

3179 Operations.END_UPLOAD: op_end_upload, 

3180 Operations.GET_ID_INFO: op_get_id_info, 

3181 Operations.START_PRESERVATION: op_start_preservation, 

3182 Operations.START_UPLOAD: op_start_upload, 

3183 Operations.UPLOAD_EMPTY_TABLES: op_upload_empty_tables, 

3184 Operations.UPLOAD_ENTIRE_DATABASE: op_upload_entire_database, 

3185 Operations.UPLOAD_RECORD: op_upload_record, 

3186 Operations.UPLOAD_TABLE: op_upload_table, 

3187 Operations.VALIDATE_PATIENTS: op_validate_patients, # v2.3.0 

3188 Operations.WHICH_KEYS_TO_SEND: op_which_keys_to_send, 

3189} 

3190 

3191 

3192# ============================================================================= 

3193# Client API main functions 

3194# ============================================================================= 

3195 

3196def main_client_api(req: "CamcopsRequest") -> Dict[str, str]: 

3197 """ 

3198 Main HTTP processor. 

3199 

3200 For success, returns a dictionary to send (will use status '200 OK') 

3201 For failure, raises an exception. 

3202 """ 

3203 # log.info("CamCOPS database script starting at {}", 

3204 # format_datetime(req.now, DateFormat.ISO8601)) 

3205 ts = req.tabletsession 

3206 fn = None 

3207 

3208 if ts.operation in OPERATIONS_ANYONE: 

3209 fn = OPERATIONS_ANYONE.get(ts.operation) 

3210 

3211 elif ts.operation in OPERATIONS_REGISTRATION: 

3212 ts.ensure_valid_user_for_device_registration() 

3213 fn = OPERATIONS_REGISTRATION.get(ts.operation) 

3214 

3215 elif ts.operation in OPERATIONS_UPLOAD: 

3216 ts.ensure_valid_device_and_user_for_uploading() 

3217 fn = OPERATIONS_UPLOAD.get(ts.operation) 

3218 

3219 if not fn: 

3220 fail_unsupported_operation(ts.operation) 

3221 result = fn(req) 

3222 if result is None: 

3223 # generic success 

3224 result = {TabletParam.RESULT: ts.operation} 

3225 elif not isinstance(result, dict): 

3226 # convert strings (etc.) to a dictionary 

3227 result = {TabletParam.RESULT: result} 

3228 return result 

3229 

3230 

3231@view_config(route_name=Routes.CLIENT_API, request_method="POST", 

3232 permission=NO_PERMISSION_REQUIRED) 

3233@view_config(route_name=Routes.CLIENT_API_ALIAS, request_method="POST", 

3234 permission=NO_PERMISSION_REQUIRED) 

3235def client_api(req: "CamcopsRequest") -> Response: 

3236 """ 

3237 View for client API. All tablet interaction comes through here. 

3238 Wraps :func:`main_client_api`. 

3239 

3240 Internally, replies are managed as dictionaries. 

3241 For the final reply, the dictionary is converted to text in this format: 

3242 

3243 .. code-block:: none 

3244 

3245 k1:v1 

3246 k2:v2 

3247 k3:v3 

3248 ... 

3249 """ 

3250 # log.debug("{!r}", req.environ) 

3251 # log.debug("{!r}", req.params) 

3252 t0 = time.time() # in seconds 

3253 

3254 try: 

3255 resultdict = main_client_api(req) 

3256 resultdict[TabletParam.SUCCESS] = SUCCESS_CODE 

3257 status = '200 OK' 

3258 

3259 except IgnoringAntiqueTableException as e: 

3260 log.warning(IGNORING_ANTIQUE_TABLE_MESSAGE) 

3261 resultdict = { 

3262 TabletParam.RESULT: escape_newlines(str(e)), 

3263 TabletParam.SUCCESS: SUCCESS_CODE, 

3264 } 

3265 status = '200 OK' 

3266 

3267 except UserErrorException as e: 

3268 log.warning("CLIENT-SIDE SCRIPT ERROR: {}", e) 

3269 resultdict = { 

3270 TabletParam.SUCCESS: FAILURE_CODE, 

3271 TabletParam.ERROR: escape_newlines(str(e)) 

3272 } 

3273 status = '200 OK' 

3274 

3275 except ServerErrorException as e: 

3276 log.error("SERVER-SIDE SCRIPT ERROR: {}", e) 

3277 # rollback? Not sure 

3278 resultdict = { 

3279 TabletParam.SUCCESS: FAILURE_CODE, 

3280 TabletParam.ERROR: escape_newlines(str(e)) 

3281 } 

3282 status = "503 Database Unavailable: " + str(e) 

3283 

3284 except Exception as e: 

3285 # All other exceptions. May include database write failures. 

3286 # Let's return with status '200 OK'; though this seems dumb, it means 

3287 # the tablet user will at least see the message. 

3288 log.exception("Unhandled exception") # + traceback.format_exc() 

3289 resultdict = { 

3290 TabletParam.SUCCESS: FAILURE_CODE, 

3291 TabletParam.ERROR: escape_newlines(exception_description(e)) 

3292 } 

3293 status = '200 OK' 

3294 

3295 # Add session token information 

3296 ts = req.tabletsession 

3297 resultdict[TabletParam.SESSION_ID] = ts.session_id 

3298 resultdict[TabletParam.SESSION_TOKEN] = ts.session_token 

3299 

3300 # Convert dictionary to text in name-value pair format 

3301 txt = "".join(f"{k}:{v}\n" for k, v in resultdict.items()) 

3302 

3303 t1 = time.time() 

3304 log.debug("Time in script (s): {t}", t=t1 - t0) 

3305 

3306 return TextResponse(txt, status=status) 

3307 

3308 

3309# ============================================================================= 

3310# Unit tests 

3311# ============================================================================= 

3312 

3313TEST_NHS_NUMBER = 4887211163 # generated at random 

3314 

3315 

3316def get_reply_dict_from_response(response: Response) -> Dict[str, str]: 

3317 """ 

3318 For unit testing: convert the text in a :class:`Response` back to a 

3319 dictionary, so we can check it was correct. 

3320 """ 

3321 txt = str(response) 

3322 d = {} # type: Dict[str, str] 

3323 # Format is: "200 OK\r\n<other headers>\r\n\r\n<content>" 

3324 # There's a blank line between the heads and the body. 

3325 http_gap = "\r\n\r\n" 

3326 camcops_linesplit = "\n" 

3327 camcops_k_v_sep = ":" 

3328 try: 

3329 start_of_content = txt.index(http_gap) + len(http_gap) 

3330 txt = txt[start_of_content:] 

3331 for line in txt.split(camcops_linesplit): 

3332 if not line: 

3333 continue 

3334 colon_pos = line.index(camcops_k_v_sep) 

3335 key = line[:colon_pos] 

3336 value = line[colon_pos + len(camcops_k_v_sep):] 

3337 key = key.strip() 

3338 value = value.strip() 

3339 d[key] = value 

3340 return d 

3341 except ValueError: 

3342 return {}