Jam.py Application Design Tips contents

Jam.py Application Design Tips

Welcome to Jam.py! If you are new to Jam.py or application development, this is the place to find some tips.

How the documentation is organised

This Documentation follows official Jam.py documentation, keeping similar concepts and looks:

Development Checklist topic touches on some development principles. We also mention on differences between Django and Jam.py.

Application Design topic discusses Jam.py design terminology, authentication thoughts, etc.

“How-to” guides, here you’ll find short answers to “How do I….?” types of questions.

Development Checklist

Here we discuss some Jam.py Development environment issues and Python ecosystem.

Development Checklist

This list is inspired by the classic article by Joel Spolsky entitled The Joel Test 12 Steps to Better Code.

Python version

Since Jam.py is a Python Web Framework, we need Python installed on the target Operation System. Get the latest version of Python at https://www.python.org/downloads/ or with your operating system’s package manager.

Python on Windows

If you are just starting with Jam.py and using Windows, you may find How to install Jam.py on Windows useful.

Using Python Virtual Environments

Virtual environment is strongly encouraged for any Python development. When compiling Python from source, we would normally install all binaries in some folder, so we can use different Python versions for different projects.

This is particularly true with the Apache Web server and mod_wsgi module, when the module is compiled against specific Python version, and loaded as per Apache procedure.

Standard Python library is used to create a virtual environment. After creation, the new environment must be “sourced”, and all packages installed with pip:

source ~/Downloads/py3.9/bin/activate
pip install jam.py
pip install pandas

The above command will install jam.py and pandas packages into ~/Downloads/py3.9/lib/python3.9/site-packages/ directory. We can have as many virtual environments as needed.

Note

It is advisable to have Production and Development environments.

Using the Source Control

Using Source Control is encouraged with Jam.py Export functionality. Since all Jam.py code is stored in the internal database, it is possible to only Source Control the admin.sqlite database, or it’s content by exporting the tables.

However, for the static files, for example JavaScript libraries, or CSS files, the best approach would be using Export which fetches all objects into one compressed file. It is possible to create an automated Export, for example:

Automated config backup

The Application Builder Export file will not contain the Application database content or structure. Because everything in Jam.py is developed inside the Application Builder, it is not possible to create different branch for the Application only. This is the main difference comparing to Django.

Unit Testing

Jam.py version higher than v5 is using pytest and Chai/Mocha. It is also possible to use Selenium IDE for the browser.

Please visit posted video Jam.py Application Builder automated testing with pytest and Mocha/Chai, and Simple CRM with jam.py and Selenium IDE in 2.45minutes!.

All necessary tools and libraries for testing should be installed in the same Python virtual environment where Jam.py libraries are.

Continuous Integration (CI)

It is possible to use Continuous Integration as we would with Django.

Generating Documentation

Sphinx is used as an defacto Python standard. It is very simple to start with using what is already provided with Jam.py.

As a bare minimum, the below files and folders are copied from Jam.py Docs directory into a new directory to create new Documantation:

conf.py       index.txt  Makefile    README.md
contents.txt  make.bat   prepare.py  set_scale.py

intro:
checklist.txt  index.txt

_static:
favicon.ico  jquery.js

_templates:
jamdocs

Modify all files for your preference and build the documentation. For example, the below command will build one single html file:

$  make singlehtml

For more professional look, the latexpdf option might be used.

$  make latexpdf

Note

Latex libraries are quite large and the installation is on the OS level. Sphinx can be installed inside Python virtual environment.

Limited introduction to the tool

Jam.py is using jam-project.py with no options to create a new project, and server.py to run the project on default port 8080, optionally providing the port number.

Django is using a few different commands, for example manage.py with options, or django-admin with options, etc.

Hence, there are no options to run management commands from the code in Jam.py as in Django.

After running the server.py command, everything is continued in the Web browser, e.g.:

http://127.0.0.1:8080/builder.html

Debugging

Since Jam.py is mostly JavaScript based, most of Debugging work is done via browser debug console. It is possible to debug the Server side Python code with print command as usual.

Profiling

It is strongly suggested not to run production Applications with the server.py command only. Instead, a proper Web server should be configured. Most of the Web server speed benefits are from compressed content sent from the Web server to the User browser, which is particularly true for CSS and JavaScript files. In addition, Jam.py has a special static folder where all images files and documents are remaining, and this files are served by the Web server bypassing the Apache mod_wsgi or similar Python interface for other Web servers. Serving the static might be possible with a separate Web server, similar to Django tactics. Jam.py has no utility as collectstatic Django command, and rsync might be used in this scenario.

Containers

To use a container with Jam.py is easy. However, the decision has to be made if the Application will use Reports based on LibreOffice (LO), since packaging complete LO might be prohibitive due to a container size. If not using LO, but plain CSV export or other mechanisms for reporting, the container size is small and fast to build. Special consideration needs to be made about hosting the Application database and separate Jam.py admin.sqlite and langs.sqlite internal database.

Please refer to more info in here: Use external database for admin e langs

It is also possible to run Jam.py Application as an Azure Web Application, or AWS Functions, hence server less.

Please refer to more info in here: Azure Deployment

Choosing the Web Server

Jam.py is providing a lightweight internal Web server for the Development/Testing, just like Django. This means Jam.py is extremely portable and the Application can be shipped as is, with just jam folder from Jam.py distribution copied into the Application folder.

Apache Web Server and mod_wsgi

Adapted from Django Docs

Django Docs: If you want to use Jam.py on a production site, use Apache with mod_wsgi. mod_wsgi operates in one of two modes: embedded mode or daemon mode. In embedded mode, mod_wsgi is similar to mod_perl – it embeds Python within Apache and loads Python code into memory when the server starts. Code stays in memory throughout the life of an Apache process, which leads to significant performance gains over other server arrangements. In daemon mode, mod_wsgi spawns an independent daemon process that handles requests. The daemon process can run as a different user than the web server, possibly leading to improved security. The daemon process can be restarted without restarting the entire Apache web server, possibly making refreshing your codebase more seamless. Consult the mod_wsgi documentation to determine which mode is right for your setup. Make sure you have Apache installed with the mod_wsgi module activated. Jam.py will work with any version of Apache that supports mod_wsgi.

If you can’t use mod_wsgi for some reason, fear not: Jam.py supports many other deployment options. It works very well with nginx. Additionally, Jam.py follows the WSGI spec (PEP 3333), which allows it to run on a variety of server platforms.

mod_wsgi is tightly coupled with Python and quite often shipped as the default Python version installed on the Operation System. When developing Application in an Python Virtual Environment, for example on the Developer’s computer, it is possible that Python version does not match the mod_wsgi version activated in Apache due to Python mismatch. To solve any problems with Python differences, it is suggested to install mod_wsgi for Apache from the Virtual Environment which matches the Development/Test environment.

IIS Web Server

Using IIS Web with FastCGI is supported. Please visit JamPy deployment on Microsoft IIS for more information.

CPanel

Using CPanel is supported. Please visit Success with cPanel v82.0.12 and Jam.py for more information.

Choosing the Database

Similar to Django, Jam.py attempts to support as many features as possible for supported databases. However, not all database backends are alike, and Jam.py design decisions were made on which features to support.

As contrary to Django, Jam.py has no models, classes, subclasses and attributes. Since Jam.py is exclusively using Application Builder, there is no code to develop model to database table relationship, or table fields specified as class attributes.

Adapted from Django Docs

Django Docs: Jam.py supports many different database servers and is officially supported with PostgreSQL, MariaDB, MySQL, MSSQL, Oracle, Firebird and SQLite.

If you are developing a small project or something you don’t plan to deploy in a production environment, SQLite is generally the best option as it doesn’t require running a separate server. However, SQLite has many differences from other databases, so if you are working on something substantial, it’s recommended to develop with the same database that you plan on using in production.

In addition to a database backend, you’ll need to make sure your Python database bindings are installed.

  • If you’re using PostgreSQL, you’ll need the psycopg2 package.

  • If you’re using MySQL or MariaDB, you’ll need a mysqlclient for Python 2.x or pymysql for Python 3.x.

  • If you’re using Oracle, you’ll need a copy of cx_Oracle.

Even though Jam.py supports all databases from the above, there is no guarantee that some specific and/or propriety database functionality is supported. Here we name a few.

Database triggers

Database triggers are specific to the database vendor and Jam.py does not support creation of triggers within the Application Builder. It is absolutely possible to use triggers, however, when moving the Application into the ie. Production, the Application Export file will not contain any information about the triggers and they need to be recreated manually.

Database views

The database views are specific to the vendor too, and Jam.py does not support it for now. The same applies as for database triggers.

Database indexes

Indexes creation/deletion is supported with Jam.py. The indexes information is stored in the Application Export file if the indexes were created by Application Builder interface. Not all vendor specific index functionality is supported, ie. Oracle function based index, etc. The Primary Key creation will result in creating an index. When working with legacy databases, meaning the tables were Imported and not created by the Application Builder, Jam.py does not import indexes information. This might lead into lost indexes if moving the Application to different environment by Jam.py Export/Import utility.

Database sequences

The database sequences are supported and Jam.py is providing an interface to use the sequence generator. Not all sequence generators can be used as this is specific to the database vendor. The Export file does not store the sequence definition, just the name of the sequence used.

Application Design

Here we discuss some Application Design terminology.

Getting Started

The first thing to understand is the Jam.py terminology: the difference between Catalogs (Catalogues), Journals and Details Groups.

Excellent Article by Marco Fioretti @ Linux Magazine:

https://www.linux-magazine.com/Issues/2020/241/Jam.py

We will reference the above article:

Every Jam.py interface, or project, is structured as a tree of tasks that are organized in groups. You may create custom groups if you like; however, at least for your first Jam.py projects, I strongly suggest that you stick with the default task tree, which consists of four groups called Catalogs, Journals, Details, and Reports.

Catalogs (Catalogues)

Note

Catalogs are tables that store actual records, like the inventory of a store or the students of some school.

Catalogs are groups of tables that can exist on it’s own, as for example in popular Spreadsheets software. Normally, the Spreadsheet is used for automated calculations. Since Jam.py is not a Spreadsheets software, all calculations for table rows is JavaScript code developed by the User/Developer within the Application Builder Client Module for an table. When some data is updated, deleted or added to the table, Jam.py can execute calculations or any other server task in the background by the code developed within the Server Module. Or, any calculations needed within the browser can be executed by the JavaScript within the Client Module.

Once tables are created in Catalogs, they cant be moved to some other group that easily. There is a special utility that does that. In a newer Jam.py version, moving the tables enabled by default, no need for any utility.

Journals

Note

Journals store information about events or transactions pertaining to the records in the Catalogs – such as invoices or purchase orders.

Journals are groups of tables that depend on some other tables, namely Catalogs and Details tables. This are the Master tables in database Master-Details scenario, or General Ledger tables, for example in Accounting. Journals table should contain at least one Details table, and if it does not, it probably is a Catalogs table and not a Journals table. One Journals table can contain as many Details tables as needed. Journals table is using Catalogs tables as a source for data lookups, for example a Customer data like Name, Surname, if this data exist in the Catalogs Customers table.

The same calculations and location principles apply as for Catalogs.

Details

Note

Details store detailed information pertaining to the records in the Journals – such as invoices items details or purchase orders items details.

Details are group of tables used only by Journals tables. This are special tables with extended functionality by the default, since extra fields are added every time an Details table is created. The extra fields are used to identify the owner record in the Journals tables. This enables Jam.py to find information much quicker than not using extra fields. It is possible to use Details with no extra fields to find exactly the same information by the code. For example, when Importing tables from some Legacy system, there would be no such extra field. The solution is to use code for some otherwise missing functionality, which is enabled automatically if used native Jam.py Details.

The same calculations and location principles apply as above.

Reports

Note

Reports are reports based on OpenDocument templates, more specifically Calcs ods files.

Professional look of reports are developed visually in LibreOffice (LO) Calcs. It is similar look and feel as with any other Spreadsheet software. The LO enables us to use graphics, for example as the Company Logo, or it is even possible to insert images from the database itself.

The LibreOffice must be installed on the server OS level which is hosting Jam.py. The containerised LO images do exist on the Internet, if needed to use for the Jam.py container.

Need more Groups?

Any other Group item not covered by the above can be created. For instance, on official Jam.py Demo Application, there is an Item called Dashboard, which is a table within the Analytics Group. The Dashboard is used for exactly that, presenting some statistics as graphs. Quite similar to any Spreadsheet software, except the JavaScript code is needed to create those graphs. Analytics is just storing the Dashboard as a placeholder, since not related to any other Item Groups.

Note

It is possible to rename any existing Item, or delete or make them invisible when accessing the Application.

Wrapping up

Now that we know the Jam.py terminology, it is good to start thinking about what we want to do. We might completely ignore the default Item Groups and make some new ones. Or, we can have only one single Group and create everything in there. However, that would not be nicely organized, due to:

  1. Item Groups are represented by the drop-down menu automatically. Hence, there might be a need for HR, Payroll, General Ledger and Assets Groups, to name a few for a Financial System. The drop-down menu would contain many items (tables), for each Group.

  2. Item Groups are Ordered in any order. In each Group, the items are also ordered.

  3. All of the above happens with no programming.

Journal/Detail (or Master/Child) scenario

As mentioned, Details Group is used to quickly identify what the Detail (Child), is for the Journal (Master), tables. However, there is more in that. Jam.py can automatically join two or more tables to Master as Details only if tables are available in Details Group. Hence, if there is no Details group, with no tables in it, Jam.py can not build the Master/Child relation automatically.

Forms, Buttons and other user interaction items

The Form is a basic type of interaction with the User. All User interaction is based on JavaScript. Which means some knowledge of JS programming might be needed if, and only if, there is a need to extend provided Jam.py functionality out of the box. Forms are automatically created for any table and default buttons are added. The Form look depends on the template created in the index.html file. However, since Jam.py v5 is a SPA (Single Page Application), all Forms will look a like, if there is only one template defined.

To add a button on some Form presenting the data, please consult the official documentation. The buttons can be disabled, enabled, not visible at all due to User Role, or some other conditions, etc.

Authentication Decision

There are a number of Design decisions to make, namely what is the Authentication method the Application will use, the need for Users Registration and password reset, or forgotten password mechanism, default Language or translations, Business Logic and what not.

Built in Authentication

Jam.py has a built in Authentication based on Users and Roles tables within admin.sqlite database. This method works fine for smaller Applications or Proof of Concept (PoC). However, for larger implementation it might be needed to extend the Users/Roles table for the a Application specific requirements. There might be also an requirement to store the two mentioned tables outside the admin.sqlite database for improved security by using some other database than SQLite3. To copy Users/Roles from a built in database to some other Application (e.g. from Development to Test and Prod), export/import of SYS_USERS and SYS_ROLES tables would be needed. All Administrator accounts accessing the Builder interface use built in database.

Non built in Authentication

To extend the Authentication with the Application database and Users/Roles tables is easy. Please consult the official Jam.py Documentation how to do that. The benefits for doing it is the ability to use the database backup or export, which will contain all information specific to the Application, in one or more backup/export files. There is no need to export/import SYS_USERS/SYS_ROLES tables, unless there are a large number of Administrator accounts who are accessing the Builder interface.

With non built in Authentication, the Application can, as an option, use a custom Python password hashing within the Task/Server Module. It is also possible to develop custom password complexity, or password lifetime, which does not exist with build in method.

External Authentication

LDAP (Active Directory) Authentication

LDAP is supported by Jam.py v5. LDAP or Active Directory Authentication is fairly straight forward, and it is possible to use a specific LDAP (AD), branch (or AD Groups), where the Jam.py Users exist. The Application will still use built in or non built in database for storing Users/Roles.

Note

The Python Business Logic (BL) can be developed within the Task/Server Module.

For example, if the User does not exist in the Application database, it can be created with Python BL and some Role can be assigned for the User. Otherwise, the User accessing the Application should be created first manually in the Application with the specific Role assigned. It is worth mentioning that LDAP or AD is using password to Authenticate the User, and it does not contain Roles information in the LDAP tree, unless LDAP is extended to contain Roles for Jam.py.

Please refer to more info in here: https://groups.google.com/g/jam-py/c/Q2iqvSA1zTM/m/o8HKxhvSCgAJ

Note

It is worth using SSL with all connection to LDAP or AD.

SAML or SSO Authentication

In addition to LDAP, SAML will be supported in newer major versions. SAML depends on the DNS service and the SSL Certificates issued for the Application name. It is a large implementation for mostly commercial environment.

Please visit Demo Application in here: http://auth.jampyapplicationbuilder.com

OAuth2, OpenID or SiteMinder (CA) Authentication

Similar to SAML, it is possible to develop other Authentication mechanisms. SiteMinder is supported by Apache.

MFA or Two Factor Authentication

Instead of using LDAP or SAML, the Application might consume Multi Factor Authentication, with Google Authenticator, FreeOTP or similar software.

Please refer to more info in here: https://groups.google.com/g/jam-py/c/Nisyemcx6Vc/m/d6m7A4WSDQAJ

User Registration Form

Instead of manually creating the Users, regardless of Authentication method, the Application might use a Registration Form for initial User creation. the default Role might be assigned for the User by Business Logic (BL).

Please consult the official Jam.py Documentation how to create a Registration Form.

Forgotten Password Method

For the Application Authenticated Users it might be worth to provide the Forgotten Password method by email. Jam.py already has Change Password method explained in the official documentation. However, Forgotten Password method depends on multitude dependencies, namely sending emails from the Application, and SSL, to name a few.

Please refer to more info in here: https://groups.google.com/g/jam-py/c/SYn2R0ILy74/m/Y5YMzUsSCAAJ

“How-to” guides

Here you’ll find short answers to “How do I….?” types of questions.

How to install Jam.py on Windows

This document will guide you through installing Python 3.8 and Jam.py on Windows. It also provides instructions for setting up a virtual environment, which makes it easier to work on Python projects. This is meant as a beginner’s guide for users working on Jam.py projects and does not reflect how Jam.py should be installed when developing patches for Jam.py itself.

The steps in this guide have been tested with Windows 10. In other versions, the steps would be similar. You will need to be familiar with using the Windows command prompt.

Install Python

Jam.py is a Python web framework, thus requiring Python to be installed on your machine. At the time of writing, Python 3.8 is the latest version.

To install Python on your machine go to https://www.python.org/downloads/. The website should offer you a download button for the latest Python version. Download the executable installer and run it. Check the boxes next to “Install launcher for all users (recommended)” then click “Install Now”.

After installation, open the command prompt and check that the Python version matches the version you installed by executing:

...\> py --version

See also

For more details, see python:using/windows documentation.

About pip

pip is a package manager for Python and is included by default with the Python installer. It helps to install and uninstall Python packages (such as Jam.py!). For the rest of the installation, we’ll use pip to install Python packages from the command line.

Setting up a virtual environment

It is best practice to provide a dedicated environment for each Jam.py project you create. There are many options to manage environments and packages within the Python ecosystem, some of which are recommended in the Python documentation. Python itself comes with venv for managing environments which we will use for this guide.

To create a virtual environment for your project, open a new command prompt, navigate to the folder where you want to create your project and then enter the following:

...\> py -m venv project-name

This will create a folder called ‘project-name’ if it does not already exist and set up the virtual environment. To activate the environment, run:

...\> project-name\Scripts\activate.bat

The virtual environment will be activated and you’ll see “(project-name)” next to the command prompt to designate that. Each time you start a new command prompt, you’ll need to activate the environment again.

Install Jam.py

Jam.py can be installed easily using pip within your virtual environment.

In the command prompt, ensure your virtual environment is active, and execute the following command:

...\> py -m pip install jam.py

This will download and install the latest Jam.py release.

After the installation has completed, you can verify your Jam.py installation by executing pip list in the command prompt.

Common pitfalls

  • There might be a problem with the file association in Windows. Check if there is more than one environment variable set for running Python scripts in PATH. This usually occurs when there is more than one Python version installed.

  • If you are connecting to the internet behind a proxy, there might be problems in running the command py -m pip install Jam.py. Set the environment variables for proxy configuration in the command prompt as follows:

    ...\> set http_proxy=http://username:[email protected]:proxyport
    ...\> set https_proxy=https://username:[email protected]:proxyport
    
  • In general, Jam.py assumes that UTF-8 encoding is used for I/O. This may cause problems if your system is set to use a different encoding. Recent versions of Python allow setting the PYTHONUTF8 environment variable in order to force a UTF-8 encoding. Windows 10 also provides a system-wide setting by checking Use Unicode UTF-8 for worldwide language support in Language ‣ Administrative Language Settings ‣ Change system locale in system settings.