Tutorial - start here¶
Well, this module does actually nothing.
It’s a cute_mongo_forms module that servers as a template for python(3) projects:

Some features:
- Autogenerated documentation
- Python packaging
- Development and production downloads from git
For the impatient¶
To start developing your own project ASAP, do the following:
Let’s assume your development python packages are in ~/python3_packages. Your pythonpath is $PYTHONPATH. Your name is Janne Jantunen and you have decided to create a new package named cute_mongo_forms. Proceed like this:
cd ~/python3_packages
git clone https://github.com/elsampsa/cute_mongo_forms
mv cute_mongo_forms cute_mongo_forms
cd cute_mongo_forms
./reinit.bash
./setauthor.bash "Janne Jantunen"
./setver.bash "0.1"
ln -s $PWD/cute_mongo_forms $PYTHONPATH/
Mod the documentation by editing .rst files in the “docs/” directory. The idea is that you hide/delete this page (“tutorial.rst”) from your own module.
After this you can re-compile and check the documentation with
cd docs
./compile.bash
firefox index.html
If you want to push your newly minted python project into a git repo, do:
git init
git remote add origin https://[your-personal-git-repository]/cute_mongo_forms.git
git add *
git commit -m "initial commit"
git push -u origin master
Things to keep in mind:
- The name of your package (that is also the name of your python module) should not contain “weird characters”, i.e. “.,-” etc.
- Remember to add git tags with (here we create tag “v1.2”)
git tag v1.2 -m "cool version v1.2" git push origin --tags
- ..now production users can install with pip(3) certain revisions of the software (see here)
Dependencies¶
Keep up with your dependencies!
Edit the setup.py and there, the “install_requires” field. This way, when installing your package with pip3, depencencies are automagically resolved. Keep them up to date also in the file docs/snippets/requirements.txt.
Files¶
This page has been produced with the file “docs/tutorial.rst”. Go ahead and open it in your editor. Open also “index.rst”. Edit them accordingly. Said that, let’s take a closer look at the directory structure:
cute_mongo_forms/ : This is a “scaffold” directory, a necessary evil for packaging
README.md Readme file in online markdown format (nice for github). Play around with online markdown here LICENSE.* License text (say, LICENSE.LGPL3, LICENSE.MIT, etc.) CHANGELOG Recent changes in the code setup.py Python setup script for creating packages MANIFEST.in used by setup.py changestr.bash Helper script to change strings in your package setver.bash Helper script to change the version of your package setauthor.bash Helper script to change the author of the package reinit.bash Reinitializes the package name cleanpy.bash Removes __pycache__ directories and .pyc files; Do maybe before “git add .”
cute_mongo_forms/docs/ : Documentation and autogenerated documentation lives here
index.html Redirects to the autogenerated html [don’t touch] index.rst The main index start from here tutorial.rst The file you are staring at the moment (or html version of it) intro.rst Introduction of the module requirements.rst What the user needs in order to use this module examples.rst Copy-pastable examples for the api user submodules.rst Documentation generated automatically from source code. Just add a new entry here for each submodule, don’t touch otherwise license.rst Copyright and License authors.rst Who is the maintainer & author compile.bash Run this always after modifying your documents / source code [don’t edit] clean.bash Clean autogenerated documentation [don’t edit] conf.py Sphinx configuration file [maybe don’t edit] .nojekyll Dummy file. This is needed for the on-line documentation to work with GitHub [don’t touch] snippets/ Example snippet source files and scripts for generating pages from them. Edit also the file requirements.txt here. generated/ Auto-generated documents [don’t touch]
cute_mongo_forms/cute_mongo_forms : This is where the actual python module lives
__init__.py Python module initialization greeters/ Submodule data/ Static data the module needs and that is included in the python package
The scaffolding “double-directory” cute_mongo_forms/cute_mongo_forms structure might seem inconvenient, but it is necessary for packaging. Here is what we just did in the For the impatient section:
- Keep “scaffolded” python modules in, say “~/python3_packages”
- Link the python module directories to your $PYTHONPATH
It’s time to start documenting! Edit the files “docs/*.rst”. Here are some nice tips for using Sphinx and here are some more
To create and to recreate the docs (after changing the code, etc.), do (in the “docs/” directory):
./compile.bash
To see your documentation, launch
firefox index.html
Online autodocumentation¶
Github users
After creating the git repo, create also a site for your project like this: Settings => GitHub pages => Source : master branch / docs folder => press Save. Now your documentation is online in github! (don’t forget to include the “docs/” folder into git).
Gitlab users
Using a generic service that uses a gitlab server? No problem. There is a simple hack to put the autodocs online.
- In gitlab, create a wiki page for your project. Write there the words “Hello world”. Next, clone the wiki repository (not the repository, just the wiki repository) with that button on the right that says “clone repository”.
- Edit your projects “docs/compile.bash”. Uncomment the options for Gitlab and set the directory of your wiki repository correctly. Run “./compile.bash”. Now the whole documentation tree has been copied to your wiki repo directory.
- In the wiki repo directory, open “home.md” in an editor. Modify it to look like this:
For documentation [click here](_build/html/index.html)
- You still need to use git to add and push all the files to the wiki repository online (there are instructions in “compile.bash”)
Organizing large python projects¶
Let’s say we have a python project that consists of “entities” and those consist of submodules. Imagine a web-service of some sort with frontend, backend and some machine_learning behind them:
- macro_project
- frontend module
- submodule1submodule2
- backend module
- submodule1submodule2
- machine_learning module
- submodule1submodule2
Here are some possibilities:
- Clusterfuck
- Keep everything in the same repository. Avoid defining API interfaces. Everybody can access all code.
- We have all done this and it’s not a good idea..
- Each module in a separate repository
- Each module should have a maintainer and a well-defined API interface
- Development
- Keep the cloned local git repositories at ~/python3_packages (as explained above)
- Production
- You can use pip(3) to install and update the packages directly from the git repository - convenient if the development/production cycle is fast
- When installed with pip(3) there is no fiddling with paths and symlinks, it just works (if you’re interested in details, pip(3) installs it under “~/.local/lib”)
- See how this is managed in the Getting started section
Code Organization¶
So, let’s get back to this tutorial. This is a module that has a single submodule called “greeters” that is organized as follows:
- greeters
- __init__.pybase.pyfancy.py
The idea is that “base.py” has some base class definitions that are used bu “fancy.py” to create derived classes. __init__.py has been tweaked so that user can import with
from cute_mongo_forms.greeters import UberFancyHelloWorld
instead of the cumbersome
from cute_mongo_forms.fancy.greeters import UberFancyHelloWorld
(also, the api user should not be bothered with the internal inheritance, etc.)
So, I suggest you open in your editor all files of the “greeters” submodule and study them a bit.
Quick testing¶
Each module file under “greeters” works also as a stand-alone test. Go to the “greeters” directory and try (either ipython or ipython3):
ipython3
%run fancy.py
Or just (either python or python3):
python3 fancy.py
This is not proper testing, though. It’s quick’n’dirty testing during development. Separate testing programs for systematic testing are a good idea to include in your module package.
Packaging¶
If you have done everything as instructed here, creating a distributable python package can be done as follows:
cd ~/python3_packages/cute_mongo_forms
python3 setup.py sdist
Your distributable python package is now in directory “dist/”. You can install it with:
pip3 install --upgrade dist/cute_mongo_forms-version.tar.gz
The setup.py script automatically finds and includes python packages to the distribution package. In “MANIFEST.in” we also tell it to include the complete “docs/” directory and an auxiliary file from the “greeters” submodule. See “setup.py” for more instructions.
Polish your setup script with the following cycle:
rm dist/*
python3 setup.py sdist
pip3 install --upgrade --verbose dist/cute_mongo_forms-version.tar.gz
The verbose option is nice to see any problems, with say, your post-installing script defined in setup.py. Once you have got rid of all the errors, you can be sure that it works also when people install your python package directly from git using pip(3).
When testing the pip(3) installation directly from git, use virtualenv to see that you got the dependencies right. First, create a virtualenv:
virtualenv --no-site-packages -p python3 test
Then let’s use that virtualenv (we clean up PYTHONPATH so that this is truly an isolated installation) to test your production system:
cd test
source bin/activate
export PYTHONPATH=
pip3 install --upgrade git+git://[your-personal-git-repository]/cute_mongo_forms
See here how the end-user would be using your python module directly from git.
To exit from virtualenv, use:
deactivate
Examples¶
If you are creating big-time examples of how to use your module, the best idea is to create a separate repository for them, in this case it would be called “cute_mongo_forms_examples”.
For a bit smaller example project you might opt for an examples submodule, i.e. for “cute_mongo_forms.examples”
Here we are considering example snippets, i.e. small and compact example programs to get the api user exited. Here is an example:
The following python code can be downloaded from [here]
Check out directory “docs/snippets/”. It serves as a collection for small example programs. Open and edit “form_snippets.bash” there. You can then run it. Remember to recompile the documentation once you’ve run that script.
Miscellaneous¶
- Check out in the source code, how I init and check a large number of parameters in the constructor
- It is also a good idea to add “exceptions.py” under “greeters”. This way you can quickly see what custom exceptions must be handled.