# default_exp html

⚠️ This writing is a work in progress.⚠️

Please read everything found on the mainpage before continuing; disclaimer and all.

Binder Binder Binder Open Source Love svg3

NPM License Active Python Versions GitHub last commit

GitHub stars GitHub watchers GitHub forks GitHub followers

Tweet Twitter Follow

About this Tutorial:

note: There is a lot of quoting and paraphrasing on this page in particular. Be sure to check the links you want to learn more.

We can update it in-drive and then rerun the results in near-real-time.

Several post-render functions will be used to reduce network requests.

I intend to hook this up to our webpack scripts for all the benefits.

Config Env

!pip install marko !python -m pip install nbdev#export import re from IPython.display import clear_output import marko#export from nbdev import export2html from nbdev.export2html import read_nb, get_metadata, clean_exports from nbdev.export import split_flags_and_code, check_re_multifrom IPython.core.interactiveshell import InteractiveShell InteractiveShell.ast_node_interactivity = "all"

FNS

These scripts will convert an nb to an html doc. NBdev and Marko among other libraries are used.

#export # 1st fn. passes each cell to decision fn. def convertNb(nb): return [cleanCell(c) for c in nb['cells']] # 2nd fn. returns text or passes cell to 'code cell' processor def cleanCell(cell): if cell.cell_type == 'markdown': source = cell.source if ( re.search('
', source) ): return source.replace('\r\n', "source") return marko.convert(source) else: return processCode(cell) # 3rd fn. Calls getFlags, processSource, processOutput def processCode(cell): x = [cell.source] flags = getFlags(x[0]) x[0] = processSource( x[0], flags ) if ( len(cell.outputs) ): for o in cell.outputs: x.append( processOutput(o, flags) ) return x#export def runit(fname='index.ipynb'): # fname = 'index.ipynb' nb = read_nb(fname) meta_jekyll = get_metadata(nb['cells']) outp = convertNb(nb) txt = ''.join( [''.join(c) for c in outp] ) return txt

Misc Notes

Unsorted notes from previous nb

"nb.keys()"'nb.keys()'
  • display_data = ['text/html', 'text/plain']
  • stream
  • execute_result = ['image/jpeg', 'text/plain']
  • error

%html

  • {'output_type': 'display_data', 'data': {'text/html': <script console.log('hi')\n script>", 'text/plain': '<IPython.core.display.HTML object>'} }

raw markdown

  • {'output_type': 'stream', 'text': 'Thisis text \n'}

display( airbnb.head() ); Image(filename='test.jpg')

  • {'output_type': 'display_data', 'data': {'text/html': div>oooAKKKKAP/Z</div', 'text/plain': '<IPython.core.display.Image object>'}

Convert FN's

This next bit will run the conversion on a single file.

import IPython import requests from google.colab import output t= """ outp = runit() v = 0 if not 'v' in globals() else v+1 # page = f"index{v}.html" page = f"index.html" print(page) f = open(page, "a") f.write( ''.join( [''.join(c) for c in outp] ) ) f.close() # posted = requests.get( 'https://charleskarpati.com/version.php', params=[('v', v)], ).json() # IPython.display.HTML( outp ) """

Serve Files with Ngrok

You can test on the Colab by using a flask server which can be a bit tricky.

from flask_ngrok import run_with_ngrok from flask import Flask from numpy import random import requests app = Flask(__name__) run_with_ngrok(app) #starts ngrok when the app is run def getHtml(filepath): with open(filepath, 'r') as file: data = file.read().replace('\n', '') return data @app.route("/") def index(): # Get version number using the localstorage. # v = requests.get( 'https://charleskarpati.com/version.php', ).json() # location = f'index{v}.html' # print('\n \n LOCATION: ', location) # return getHtml(location) outp = runit('BniaScripts.ipynb') resp = ''.join( [''.join(c) for c in outp] ) resp = resp.replace('', '') return resp @app.route("/about") def about(): return getHtml('lines.html') app.run()

Working with GDrive Directories, Github, NBDev

Connect to Drive

If you have content on your Google Drive, you can publish entire directorys.

#hide !pip install nbdev !pip install flask_ngrok from google.colab import drive drive.mount('/content/drive') %cd /content/drive/My Drive/'Software Development Documents'/dataplayInstalling collected packages: flask-ngrok Successfully installed flask-ngrok-0.0.25 Mounted at /content/drive /content/drive/My Drive/Software Development Documents/dataplay project = 'dataguide' # 'dataguide' 'datalabs' 'dataplay' 'VitalSigns', 'DevelopersDocumentation' %cd ../{project}/ /content/drive/My Drive/Software Development Documents/dataguide import requests from IPython.core.display import display, HTML

SETUP

addToHeader = """
""" def addAfterNav(title): return f"""
""" addToFooter = """
"""navTop = """
"""nav = navTop + navMid + navBottom

Run one

lsbuild/ dist/ LICENSE MANIFEST.in README.md setup.py CONTRIBUTING.md docs/ Makefile notebooks/ settings.ini %cd notebooks/content/drive/My Drive/Software Development Documents/dataguide/notebooks ls01_Github.ipynb 07_Javascript_Packages.ipynb 02_Colabs.ipynb 08_Cloud_Functions.ipynb 03_Shell_Basics.ipynb 09_Browser_Extensions.ipynb 04_APIs.ipynb images/ 05_Webscraping.ipynb index.ipynb 06_Nbdev.ipynb from flask_ngrok import run_with_ngrok from flask import Flask from numpy import random import requests header = requests.get('https://bniajfi.org/wp-json/wp/v2/header') header = header.content.decode('utf-8')[:-4] header= header.replace('/wp-content/uploads/2021/02/bnia_text.png', 'https://bniajfi.org/wp-content/uploads/2021/02/bnia_text.png') header = header + addToHeader + nav footer = requests.get('https://bniajfi.org/wp-json/wp/v2/footer') footer = addToFooter+footer.content.decode('utf-8')[:-4] app = Flask(__name__) run_with_ngrok(app) #starts ngrok when the app is run o = """ def getHtml(filepath): with open(filepath, 'r') as file: data = file.read().replace('\n', '') return data """ @app.route("/") def index(): def template(title, x): return f'{header}{title}{x}{footer}' def replace(txt): txt = txt.replace('', '') return txt # Get version number using the localstorage. # v = requests.get( 'https://charleskarpati.com/version.php', ).json() # location = f'index{v}.html' # print('\n \n LOCATION: ', location) # return getHtml(location) infilename = 'index.ipynb' outp = runit(infilename) title = "".join(re.findall('[a-zA-Z][^a-zA-Z]*', 'index.ipynb'[:-6])) resp = ''.join( [''.join(c) for c in outp] ) resp = replace(resp) resp = template( addAfterNav(title) , resp) return resp @app.route("/about") def about(): return getHtml('lines.html') app.run()

Run all

lsCONTRIBUTING.md index_files/ notebooks/ VitalSigns/ dist/ LICENSE README.md VitalSigns.egg-info/ docker-compose.yml Makefile settings.ini docs/ MANIFEST.in setup.py !git rm -r docs -ffatal: pathspec 'docs' did not match any files !rm docs -rmkdir docslsbuild/ dataplay.egg-info/ LICENSE notebooks/ setup.py CONTRIBUTING.md dist/ Makefile README.md dataplay/ docs/ MANIFEST.in settings.ini # ! nbdev_build_docs --fname '' # Makes the readme.md import os def template(title, x): return f'{header}{title}{x}{footer}' def replace(txt): txt = txt.replace('', '') return txt # traverse whole directory for file in os.listdir(r'notebooks'): # check the extension of files if (file.endswith('.ipynb') ): # print whole path of files print(os.path.join('notebooks', file)) title = "".join(re.findall('[a-zA-Z][^a-zA-Z]*', file[:-6])) page = f"./docs/{title}.html" outp = runit(os.path.join('notebooks', file)) resp = ''.join( [''.join(c) for c in outp] ) resp = replace(resp) resp = template( addAfterNav(title) , resp) try: f = open(page, "a") except: f = open(page[1:], "a") f.write( resp ) f.close() # send it to my websesrver for saving.notebooks/00_core.ipynb 47228notebooks/02_TidyAddr.ipynb 48656notebooks/04_Create_Acs_Indicators_Original.ipynb 205133notebooks/05_Create_Acs_Indicators.ipynb 71310notebooks/93_Create_BCountyVitalSigns.ipynb 66977notebooks/92_VitalSigns_Acs_Create_Indicators.ipynb 51859notebooks/91_VitalSigns_Acs_Download.ipynb 54318notebooks/HUD.ipynb 50452notebooks/Infousa.ipynb 107591notebooks/Mdprop_Permits_Fore_Vac Vio.ipynb 98239notebooks/TreeBaltimore_TreePlnt.ipynb 50254notebooks/RBIntel.ipynb 67751notebooks/TidyAddr_EnochPratt.ipynb 48912notebooks/01_Close Crawl.ipynb 89734notebooks/90_ACS_Explore_and_Download.ipynb 77529notebooks/03_convertVSSheetForWPUpload.ipynb 797836notebooks/BCPSS_Schools.ipynb 153919notebooks/Bidbalt_TaxSale_Taxlien.ipynb 56303notebooks/BPD_P1_Visualizations.ipynb 52476notebooks/BPD_P1.ipynb 64742notebooks/CitiStat_Dirtyst_Clogged.ipynb 60010notebooks/City_Liquor_Board_Permits.ipynb 53206notebooks/CityFinance_histaxhometaxowntax.ipynb 59689notebooks/DHR_SNAP_TANF_Create.ipynb 57453notebooks/EnochPratt_Libcard.ipynb 53077notebooks/FARES_Salpr_Shome.ipynb 60734notebooks/FDIC_Banks.ipynb 50265notebooks/index.ipynb 60910

Publishing a new Repo

#step 1 ! git init ! git config --global user.email "bniajfi@gmail.com" ! git config --global user.name "bniajfi"Reinitialized existing Git repository in /content/drive/My Drive/Software Development Documents/datalabs/.git/ #step 2 ! git add * ! git commit -m "Second commit for styling" ! git branch -M main # https://github.com/settings/tokens# step 3 token = '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' username = 'BNIA' print(f'https://{token}@github.com/{username}/{project}.git') ! git remote remove origin ! git remote add origin 'https://{token}@github.com/{username}/{project}.git' #! git push -u origin main !git push 'https://{token}@github.com/{username}/{project}.git'
Pypi
lsCONTRIBUTING.md docs/ Makefile notebooks/ settings.ini dist/ LICENSE MANIFEST.in README.md setup.py ! pip install twine ! nbdev_bump_version ! make pypilsbuild/ dist/ Makefile README.md CONTRIBUTING.md docs/ MANIFEST.in settings.ini datalabs.egg-info/ LICENSE notebooks/ setup.py #hide # https://nbdev.fast.ai/tutorial.html#Add-in-notebook-export-cell # https://nbdev.fast.ai/sync#nbdev_update_lib # first. builds the .py files from from .ipynbs # !nbdev_build_lib # --fname filename.ipynb # second. Push .py changes back to their original .ipynbs # !nbdev_update_lib # sometimes. Update .ipynb import statements if the .py filename.classname changes. # !relimport2name # nbdev_build_docs builds the documentation from the notebooks # !nbdev_build_docs --force_all True --mk_readme True

The “src refspec master does not match any” error occurs if you have forgotten to add the files you have changed to a commit and try to push those changes to a remote repository before you make the first commit in your repository.

'https://docs.github.com/en/github/importing-your-projects-to-github/importing-source-code-to-github/adding-an-existing-project-to-github-using-the-command-line' 'https://stackoverflow.com/questions/61424599/error-when-pushing-files-to-git-in-colab-fatal-could-not-read-username-for-ht' 'https://stackoverflow.com/questions/492558/removing-multiple-files-from-a-git-repo-that-have-already-been-deleted-from-disk' 'https://github.community/t/error-refname-refs-heads-master-not-found/144737/13' 'https://github.com/settings/tokens' 'https://github.com/settings/tokens' 'https://stackoverflow.com/questions/61424599/error-when-pushing-files-to-git-in-colab-fatal-could-not-read-username-for-ht'

image.png

headerPHPEndsLike = """
null """headerDoesntHave = """
class="span8 content" """footerDoesntHave = """
<- end content div
<- end section
<- end container """footerPHPStartsLike = """

SEARCH

"""

SEARCH

CONNECT WITH US

DONATE

Help us keep this resource free and available to the public. Donate now!

Donate to BNIA-JFI

CONTACT US

Baltimore Neighborhood Indicators Alliance
The Jacob France Institute
1420 N. Charles Street, Baltimore, MD 21201
410-837-4377 | bnia-jfi@ubalt.edu

Baltimore Neighborhood Indicators Alliance - © 2018