GovReady-Q Documentation¶
Note
This documentation set is part of GovReady v0.9.x. In places, it may still refer to v0.8.6 software features rather than v0.9.x features. Thank you for your patience while we update the documentation to apply to the v0.9.x software. Pull requests are very welcome!
The GovReady-Q Compliance Server is an open source GRC platform for highly automated, user-friendly, self-service compliance assessments and documentation. It’s perfect for DevSecOps.
GovReady-Q solves the painful compliance bottleneck of needing months to authorize applications that deploy and redeploy in minutes.
The code is open source, and available on GitHub.
Attention
GovReady-Q is in Beta. Suggested for DevSecOps early adopters needing Compliance-as-Code.
About GovReady-Q¶
Why GovReady-Q?¶
Everything about developing and deploying software is accelerating…except for compliance. Why? Because:
maintaining written documentation is too slow,
pondering how jargon-laden control guidance applies is too hard,
there’s little reuse, and no compliance documentation supply chain.
To stop needing months to authorize systems that deploy in minutes, assessments and authorizations need to be assembled from vetted, pre-fabricated components sourced from the same software supply chain with which we assemble applications.
How GovReady-Q Accelerates Compliance¶
GovReady accelerates compliance through component-centric guidance, pre-written documentation, and collaboration.
When you use or install GovReady-Q, you gain access to a marketplace of small, self-service compliance apps written by peers and vendors that map system components to security controls and guide you step-by-step through assessments and documentation.
Apps map components to security/compliance controls¶
As you and your teammates collaboratively answer questions, the compliance apps work with GovReady-Q to store your data in a relational database and automagically generate and maintain your compliance artifacts for auditors.
GovReady-Q’s contribution to Compliance-as-Code is the data abstractions for shareable, reusable, and customizable packages—Compliance Apps—to map the relationship between a system component and a set of controls. The approach is innovative, yet familiar. Compliance Apps:
enable a hub/marketplace for community contributions;
extend inherited controls model to each system component;
enable modern, user-friendly experiences;
support agile, iterative workflows.
Attention
GovReady-Q software is “Beta” software best suited for
early adopters needing faster compliance for DevSecOps.
GovReady-Q Philosophy¶
Compliance is not security. Compliance scales security.
Compliance is a technique humans have developed for enabling trust in systems that are too large and complex for individuals to assess trustworthiness. Compliance scales participation, attestation and verification of recommended practices.
We love security and innovation and believe they enable each other.
We believe security and compliance are standard, not premium add-ons.
We view compliance as a by-product of a well-instrumented process.
We value ease-of-use to increase adoption.
We value automation to increase consistency.
We see virtualization and DevOps enabling massive gains in security and compliance.
GovReady-Q Features¶
Easy-to-use, beautiful questionnaires
Jargon-free approach to security controls and compliance
Step-by-step guidance through assessments
Compliance-as-Code approach to creating compliance documentation
Discuss questions and answers in the tool instead of in email
Support for rich, clear multi-media and multi-channel communication
RESTful Automation API to integrate with DevOps pipeline and existing agents
Innovative, reusable “Compliance Apps” model
Friendly Open Source license so you can start now
Using Hosted GovReady-Q¶
There’s nothing to install. Q.GovReady.com is the hosted version of GovReady-Q.
Visit Q.GovReady.com
Fill out the form “About your organization” and “About you” to create your account
Don’t worry about the Service Levels – everything’s available to everyone during the Beta phase
We’ll contact you to help you get started
Attention
We will help you get up and running during the current
Beta phase of the project while we make getting started easier.
The hosted version is an excellent solution if have one project/system you are trying to get through NIST SP 800-53 or NIST SP 800-171 compliance, or you are have just trying to pull together a few specific compliance documents like your Privacy Policy or Rules of Behavior. The hosted service is operated by GovReady PBC, the company behind GovReady-Q Compliance Server.
If you have questions about the hosted version, email info@govready.com.
Downloading GovReady-Q¶
D ownloading |
Where |
|---|---|
Current Release on Docker |
|
Nightly Build on Docker |
|
Clone the GitHub repo |
Finding Compliance Apps¶
Compliance Apps are GovReady-Q modular, shareable, reusable, data packages mapping the relationship between system components and security controls. See Understanding Compliance Apps for a more detailed description.
For Hosted Version¶
When using the Hosted Version of GovReady-Q, GovReady PBC manages the Compliance Apps available to your organization. Send email to info@govready.com to request changes.
For Local Installs¶
The docker and downloaded version of GovReady-Q automatically loads a small set of example Compliance Apps. Compliance Apps are published in collections known as “AppSources” (e.g., repos). Here are a few:
You can can show and hide compliance apps from the Django administration
page at main.localhost:8000/admin/guidedmodules/appsource/.
Creating Your Own Compliance Apps¶
To get started writing your own Compliance Apps see: Creating Compliance Apps.
Documentation¶
The official GovReady-Q documentation is maintained at govready-q.readthedocs.io.
Support¶
Commercial support for GovReady-Q is provided by GovReady PBC. Email info@govready.com.
Sign up for Security Notifications email list at GovReady Security Alerts.
Reporting Bugs & Issues¶
Please file bug reports on our GitHub issue. When reporting a bug, please include as much information as possible. This includes:
Install type: Hosted, Local, Docker, etc
URL
Action taken
Expected result
Actual result
Screenshot (if relevant)
License / Credits¶
This repository is licensed under the GNU GPL v3.
Emoji icons by http://emojione.com/developers/.
Generic server icon by Stock Image Folio from Noun Project.
About GovReady PBC¶
GovReady PBC is a Public Benefit Corporation whose mission is to lower the cost of innovation in digital services to citizens. GovReady’s innovative self-service IT compliance tool GovReady-Q was developed as part of an R&D contract to automate and lower the cost of cyber security compliance from the Department of Homeland Security, Science and Technology Directorate, Cyber Security Division. GovReady PBC is based in the greater Washington, DC metro area.
Installation Guide¶
This document will guide you through the GovReady-Q installation process.
Setting up GovReady-Q involves installing the GovReady-Q server, a database, a WSGI server to run the Python code, and optionally connecting GovReady-Q to a mailserver to send invitations and notifications.
Note
This is a note.
Contents
System Requirements¶
GovReady-Q is a Python 3.6 and higher, Django 2.x application with a relational database back-end.
Software Requirements¶
Required Software Packages (partial list) |
|---|
(GovReady-Q application) |
Python 3.6 |
Django 2.2 |
Jinja 2.x |
uwsgi 2.x |
unzip |
graphviz |
libmagic |
pandoc |
supervisor |
Optional Software Packages (partial list) |
|---|
Wkhtmltopdf |
Git 2.x |
Supported Databases |
|---|
Postgres 9.4 (psycopg2 2.7.5 adapter) |
Mysql 7.6 and higher (mysqlclient 1.3.12 adapter) |
SQLite 3.x |
Recommended Database |
|---|
Postgres 9.4 (psycopg2 2.7.5 adapter) |
SMTP Mail Server (for sending email notifications and receiving comments via email) |
|---|
Any SMTP mail server (MTA) supporting STARTTLS connections. |
For a more detailed list of software dependencies and requirements see: * https://github.com/GovReady/govready-q/blob/master/requirements.in * https://github.com/GovReady/govready-q/blob/master/requirements.txt * https://github.com/GovReady/govready-q/blob/master/requirements_mysql.in * https://github.com/GovReady/govready-q/blob/master/requirements_mysql.txt * https://github.com/GovReady/govready-q/blob/master/Vagrantfile
Hardware Requirements¶
Minimum Hardware |
|---|
Single server to host both multi-tenant GovReady-Q application and Database |
Linux-compatible hardware |
2GB RAM |
10 GB storage (for database) |
Recommended Hardware |
|---|
2 servers: 1 for multi-tenant GovReady-Q application; 1 for Database Server |
Linux-compatible hardware (64 bit architecture; FIPS-140-2 validated cryptographic module) |
8GB RAM for each server |
100 GB storage (for database server) |
Installing GovReady-Q server¶
The GovReady-Q server can be installed on any Unix-like operating system. It is most commonly installed on Linux.
The primary differences between OS installations are package names and the OS’s install package command.
Once the required OS packages have been installed, installing GovReady-Q server is identical.
Consult the table below and choose how to proceed:
Operating system |
Version |
CentOS / RHEL |
|
Ubuntu |
|
macOS |
|
Docker (Unix) |
|
Docker GovReady+Wazuh (Unix) |
<govready-q_server_wazuh_docker> |
Contents
CentOS / RHEL 7 from sources¶
This guide describes how to install the GovReady-Q server for CentOS 7 or greater from source code.
Note
Instructions applicable for RHEL 7 and CentOS 7. Tested on a CentOS 7.8.2003 Docker image.
Installing required OS packages¶
GovReady-Q requires Python 3.6 or higher and several Linux packages to provide full functionality. Execute the following commands:
# enable IUS repository
sudo yum install https://centos7.iuscommunity.org/ius-release.rpm
sudo yum update
# install dependencies
sudo yum install \
python36u python36u-pip \
unzip git2u jq nmap-ncat \
graphviz pandoc
# to generate thumbnails and PDFs for export, you must install wkhtmltopdf
# WARNING: wkhtmltopdf can expose you to security risks. For more information,
# search the web for "wkhtmltopdf Server-Side Request Forgery"
read -p "Are you sure (yes/no)? " ; if [ "$REPLY" = "yes" ]; then sudo yum install xorg-x11-server-Xvfb wkhtmltopdf ; fi
GovReady-Q calls out to git to fetch apps from git repositories, but
that requires git version 2 or later because of the use of the
GIT_SSH_COMMAND environment variable. The stock git on RHEL is version 1.
Switch it to version 2+ by using the IUS package:
# if necessary, remove any git currently installed
sudo yum remove git
# install git2u
sudo yum install git2u
Upgrading pip on RHEL 7¶
Upgrade pip because the RHEL package version is out of date (we need
>=9.1 to properly process hashes in requirements.txt)
pip3 install --upgrade pip
Installing GovReady-Q¶
Clone the GovReady source code and install packages.
# clone GovReady-Q
git clone https://github.com/govready/govready-q
cd govready-q
# install Python 3 packages
pip3 install --user -r requirements.txt
# install Bootstrap and other vendor resources locally
# (sudo needed only for the embedded 'yum install' command)
sudo ./fetch-vendor-resources.sh
Setting up GovReady-Q¶
Run the final setup commands to initialize a local SQLite database in local/db.sqlite to make sure everything is OK so far.
The following warning message is expected and okay:
WARNING: Specified PDF generator is not supported. Setting generator to ‘off’.
# run database migrations (sqlite3 database used by default)
python3 manage.py migrate
# load a few critical modules
python3 manage.py load_modules
# create superuser with initial account
python3 manage.py first_run
Starting GovReady-Q¶
# run the server
python3 manage.py runserver 0.0.0.0:8000
Visit your GovReady-Q site in your web browser at:
It is not necessary to specify a port. GovReady-Q will read the local/environment.json file to determine host name and port.
# run the server
python3 manage.py runserver
Note
Depending on host configuration both python3 and python commands will work.
GovReady-Q can run on ports other than 8000. Port 8000 is selected for convenience.
GovReady-Q defaults to localhost:8000 when launched with python manage.py runserver.
Tested on a ``CentOS 7.8.2003 Docker image <https://hub.docker.com/_/centos>``__ on 2020-05-04.
(Optional) Installing Postgres, MySQL¶
GovReady-Q can optionally be configured to work with Postgress or MySQL database engines instead of the default SQLITE3.
# optional install of postgres and/or mysql
sudo yum install postgresql mysql-devel
# if you intend to use optional configurations, such as the MySQL adapter, you
# may need to run additional `pip3 install` commands, such as:
pip3 install --user -r requirements_mysql.txt
Creating “environment.json” configuration file¶
GovReady uses a configuration file stored at local/environment.json.
See Environment Settings for a complete list of variables you can configure.
Create a file there and include values like these:
{
"debug": false,
"host": "localhost:8000",
"https": false,
"secret-key": "...something here..."
}
(Optional) Enabling PDF export¶
To activate PDF and thumbnail generation, add gr-pdf-generator and
gr-img-generator environment variables to your
local/environment.json configuration file:
{
...
"gr-pdf-generator": "wkhtmltopdf",
"gr-img-generator": "wkhtmltopdf",
...
}
(Optional) Deployment utilities¶
Sample apache.conf, supervisor.ini, and update.sh files can
be found in the source code directory deployment/rhel.
(Optional) Creating a dedicated GovReady UNIX user¶
You may find it useful to create a user specifically for GovReady-Q. Do this before installing GovReady-Q.
# Create user.
useradd govready-q -c "govready-q"
# Change permissions so that the webserver can read static files.
chmod a+rx /home/govready-q
Ubuntu from sources¶
This guide describes how to install the GovReady-Q server for Ubuntu 20.04 or greater from source code.
Note
Instructions applicable for Ubuntu 20.04 LTS (Focal Fossa) Tested on an Ubuntu focal-20200423 Docker image.
Installing required OS packages¶
GovReady-Q requires Python 3.6 or higher and several Linux packages to provide full functionality. Execute the following commands:
# update package list
sudo apt-get update
# install dependencies
DEBIAN_FRONTEND=noninteractive \
sudo -E apt-get install \
unzip git curl \
python3 python3-pip \
python3-yaml \
graphviz pandoc \
language-pack-en-base language-pack-en
# to generate thumbnails and PDFs for export, you must install wkhtmltopdf
# WARNING: wkhtmltopdf can expose you to security risks. For more information,
# search the web for "wkhtmltopdf Server-Side Request Forgery"
read -p "Are you sure (yes/no)? " ; if [ "$REPLY" = "yes" ]; then sudo apt-get install wkhtmltopdf ; fi
Installing GovReady-Q¶
Clone the GovReady source code and install packages.
# clone GovReady-Q
git clone https://github.com/govready/govready-q
cd govready-q
# install Python 3 packages
pip3 install --user -r requirements.txt
# install Bootstrap and other vendor resources locally
./fetch-vendor-resources.sh
Setting up GovReady-Q¶
Run the final setup commands to initialize a local SQLite database in local/db.sqlite to make sure everything is OK so far.
The following warning message is expected and okay:
WARNING: Specified PDF generator is not supported. Setting generator to ‘off’.
# run database migrations (sqlite3 database used by default)
python3 manage.py migrate
# load a few critical modules
python3 manage.py load_modules
# create superuser with initial account
python3 manage.py first_run
Starting GovReady-Q¶
# run the server
python3 manage.py runserver 0.0.0.0:8000
Visit your GovReady-Q site in your web browser at:
It is not necessary to specify a port. GovReady-Q will read the local/environment.json file to determine host name and port.
# run the server
python3 manage.py runserver
Note
Depending on host configuration both python3 and python commands will work.
GovReady-Q can run on ports other than 8000. Port 8000 is selected for convenience.
GovReady-Q defaults to localhost:8000 when launched with python manage.py runserver.
Tested on an Ubuntu focal-20200423 Docker image.
(Optional) Installing Postgres, MySQL¶
GovReady-Q can optionally be configured to work with Postgress or MySQL database engines instead of the default SQLITE3.
# optional install of postgres and/or mysql
sudo apt-get install postgresql mysql-server mysql-client
# if you intend to use optional configurations, such as the MySQL adapter, you
# may need to run additional `pip3 install` commands, such as:
pip3 install --user -r requirements_mysql.txt
Creating “environment.json” configuration file¶
GovReady uses a configuration file stored at local/environment.json.
See Environment Settings for a complete list of variables you can configure.
Create a file there and include values like these:
{
"debug": false,
"host": "localhost:8000",
"https": false,
"secret-key": "...something here..."
}
(Optional) Enabling PDF export¶
To activate PDF and thumbnail generation, add gr-pdf-generator and
gr-img-generator environment variables to your
local/environment.json configuration file:
{
...
"gr-pdf-generator": "wkhtmltopdf",
"gr-img-generator": "wkhtmltopdf",
...
}
(Optional) Deployment utilities¶
Sample nginx.conf, supervisor.confg, and update.sh files can
be found in the source code directory deployment/ubuntu.
(Optional) Creating a dedicated GovReady UNIX user¶
You may find it useful to create a user specifically for GovReady-Q. Do this before installing GovReady-Q.
# Create user.
useradd govready-q -m -c "govready-q"
# Change permissions so that the webserver can read static files.
chmod a+rx /home/govready-q
macOS from sources¶
This guide describes how to install the GovReady-Q server for macOS 10.10 or greater from source code.
Note
These instructions have not been recently tested.
Installing required OS packages¶
GovReady-Q requires Python 3.6 or higher to run and several Unix packages to provide full functionality. Install the Homebrew package manager (https://brew.sh) to easily install Unix packages on macOS. Homebrew will install all packages in your userspace and not change native macOS Python or other libraries. Execute the following commands:
# install Homebrew package manager
/usr/bin/ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)"
Now install Python3 and the required Unix packages.
# install dependencies using brew
brew install python3
# install other packages:
brew install unzip graphviz pandoc selenium-server-standalone magic libmagic
# to generate thumbnails and PDFs for export, you must install wkhtmltopdf
# WARNING: wkhtmltopdf can expose you to security risks. For more information,
# search the web for "wkhtmltopdf Server-Side Request Forgery"
read -p "Are you sure (yes/no)? " ; if [ "$REPLY" = "yes" ]; then brew cask install wkhtmltopdf ; fi
Installing GovReady-Q¶
Clone the GovReady source code and install packages.
# clone GovReady-Q
git clone https://github.com/govready/govready-q
cd govready-q
# install Python 3 packages
pip3 install --user -r requirements.txt
# install Bootstrap and other vendor resources locally
# (sudo needed only for the embedded 'yum install' command)
sudo ./fetch-vendor-resources.sh
Setting up GovReady-Q¶
Run the final setup commands to initialize a local SQLite database in local/db.sqlite to make sure everything is OK so far.
The following warning message is expected and okay:
WARNING: Specified PDF generator is not supported. Setting generator to ‘off’.
# run database migrations (sqlite3 database used by default)
python3 manage.py migrate
# load a few critical modules
python3 manage.py load_modules
# create superuser with initial account
python3 manage.py first_run
Starting GovReady-Q¶
# run the server
python3 manage.py runserver 0.0.0.0:8000
Visit your GovReady-Q site in your web browser at:
It is not necessary to specify a port. GovReady-Q will read the local/environment.json file to determine host name and port.
# run the server
python3 manage.py runserver
Note
Depending on host configuration both python3 and python commands will work.
GovReady-Q can run on ports other than 8000. Port 8000 is selected for convenience.
GovReady-Q defaults to localhost:8000 when launched with python manage.py runserver.
Tested on a ``CentOS 7.8.2003 Docker image <https://hub.docker.com/_/centos>``__ on 2020-05-04.
(Optional) Installing Postgres, MySQL¶
GovReady-Q can optionally be configured to work with Postgress or MySQL database engines instead of the default SQLITE3.
# optional install of postgres and/or mysql
sudo yum install postgresql mysql-devel
# if you intend to use optional configurations, such as the MySQL adapter, you
# may need to run additional `pip3 install` commands, such as:
pip3 install --user -r requirements_mysql.txt
Creating “environment.json” configuration file¶
GovReady uses a configuration file stored at local/environment.json.
See Environment Settings for a complete list of variables you can configure.
Create a file there and include values like these:
{
"debug": false,
"host": "localhost:8000",
"https": false,
"secret-key": "...something here..."
}
(Optional) Enabling PDF export¶
To activate PDF and thumbnail generation, add gr-pdf-generator and
gr-img-generator environment variables to your
local/environment.json configuration file:
{
...
"gr-pdf-generator": "wkhtmltopdf",
"gr-img-generator": "`wkhtmltopdf",
...
}
(Optional) Deployment utilities¶
Sample apache.conf, superviser.ini, and update.sh files can
be found in the source code directory deployment/rhel.
(Optional) Creating a dedicated GovReady UNIX user¶
You may find it useful to create a user specifically for GovReady-Q. Do this before installing GovReady-Q.
# Create user.
useradd govready-q -c "govready-q"
# Change permissions so that the webserver can read static files.
chmod a+rx /home/govready-q
Docker¶
This guide describes how to install the GovReady-Q server using Docker.
Installing Docker¶
Installing with Docker
Make sure you first install Docker (https://docs.docker.com/engine/installation/) and, if appropriate, grant non-root users access to run Docker containers (https://docs.docker.com/engine/installation/linux/linux-postinstall/#manage-docker-as-a-non-root-user) (or else use sudo when invoking Docker below).
Installing GovReady-Q server¶
Start
# Run the docker container in detached mode
docker container run --name govready-q --detach -p 8000:8000 govready/govready-q
# Create admin account and organization data
docker container exec -it govready-q first_run
# Stop, start container (when needed)
docker container stop govready-q
docker container start govready-q
# View logs - useful if site does not appear
docker container logs govready-q
# To destroy the container and all user data entered into Q
docker container rm -f govready-q
Visit your GovReady-Q site in your web browser at:
GovReady-Q and Wazuh via Docker Compose¶
This guide describes how to install GovReady-Q and Wazuh endpoint assessment platform together using Docker Compose to launch multiple containers.
The containers that are run are:
govready-q: The GovReady-Q server, that assists teams through the compliance process. wazuh: It runs the Wazuh manager, Wazuh API and Filebeat (for integration with Elastic Stack) wazuh-kibana: Provides a web user interface to browse through alerts data. It includes Wazuh plugin for Kibana, that allows you to visualize agents configuration and status. wazuh-nginx: Proxies the Kibana container, adding HTTPS (via self-signed SSL certificate) and Basic authentication. wazuh-elasticsearch: An Elasticsearch container (working as a single-node cluster) using Elastic Stack Docker images. Be aware to increase the vm.max_map_count setting, as it’s detailed in the Wazuh documentation.
Use [Docker Compose](https://docs.docker.com/compose/) to manage the multi-container app.
Docker Compose commands are similar to, but different from, regular Docker commands. Read the Docker Compose docs for more details.
Installing Docker¶
Installing with Docker
Make sure you first install Docker (https://docs.docker.com/engine/installation/) and, if appropriate, grant non-root users access to run Docker containers (https://docs.docker.com/engine/installation/linux/linux-postinstall/#manage-docker-as-a-non-root-user) (or else use sudo when invoking Docker below).
Installing GovReady-Q server¶
Start
# To start docker containers in detached mode
docker-compose up -d
# Create admin account and organization data
docker container exec -it govready-q first_run
# Stop, start container (when needed)
docker container stop govready-q
docker container start govready-q
# View logs - useful if site does not appear
docker container logs govready-q
# To stop and remove the containers (and delete user data entered into Q if there no persistent database exists)
docker-compose down
Visit your GovReady-Q site in your web browser at:
Notes and Common Issues¶
Multiple containers will be created, one for each “service” (as they’re called in Docker Compose).
You can check the status of the containers:
`bash
docker-compose ps
`
For additional information on wazuh containers, including configuration information, see: [https://github.com/wazuh/wazuh-docker](https://github.com/wazuh/wazuh-docker)
Deploying GovReady-Q for Development¶
This page provides additional tips for installing and run GovReady-Q in a mode suitable for making and testing changes to the software (i.e., in a Dev environment).
Quickstart¶
For local development, there is a quickstart script available to speed up environment setup. After installing system requirements through your package manager, run the following four commands in order to set up GovReady-Q in a new directory:
git clone https://github.com/govready/govready-q
cd govready-q
virtualenv -p python3 env
./quickstart.sh
This will set up a local/environment.json file suitable for a dev
environment; set up local dependencies; and run the assorted initial
manage.py commands (migrate, load_modules, first_run, etc.).
Additionally, it can run common post-installation steps, based on user
input.
The quickstart.sh script is set up to take user input, and is
expected to be run interactively.
Creating local/environment.json file¶
When you first run GovReady-Q with python manage.py runserver,
you’ll be prompted to copy some JSON data into a file at
local/environment.json like this:
{
"debug": true,
"host": "localhost:8000",
"https": false,
"secret-key": "...something here..."
}
This file is important for persisting login sessions, and you can provide other Q settings in this file.
Invitations on local systems¶
You will probably want to try the invite feature at some point. The debug server is configured to dump all outbound emails to the console. So if you “invite” others to join you within the application, you’ll need to go to the console to get the invitation acceptance link.
Updating the source code¶
To update the source code from this repository you can git pull. You
then may need to re-run some of the setup commands:
# if you set up virtualenv
git pull
source env/bin/activate
pip3 install -r requirements.txt
./fetch-vendor-resources.sh
python3 manage.py migrate
python3 manage.py load_modules
# if you did NOT set up virtualenv
git pull
pip3 install --user -r requirements.txt
./fetch-vendor-resources.sh
python3 manage.py migrate
python3 manage.py load_modules
Deploying GovReady-Q in Production environments¶
These instructions assume that GovReady-Q is installed by the user
govready-q, in the directory /home/govready-q/govready-q/.
To verify that this is the case, run the following command, and check
whether GovReady-Q responds to HTTP requests (on localhost:8000 by
default).
cd /home/govready-q/govready-q/ && python3 manage.py runserver
If GovReady-Q is installed successfully, proceed with the rest of these configuration instructions. If it doesn’t, see OS-specific install instructions.
Set basic configuration variables¶
Create a file named local/environment.json (ensure it is not
world-readable) that contains site configuration in JSON, with some
recommended settings:
{
"debug": false,
"host": "webserver.hostname.com",
"https": true,
"secret-key": "generate random string using e.g. https://www.miniwebtool.com/django-secret-key-generator/",
"static": "/home/govready-q/public_html/static"
}
Because of host header checking, to test the site again using
python3 manage.py runserver you will need to visit it using
webserver.hostname.com and not localhost. (Be sure to replace
webserver.hostname.com with your hostname.)
Remember to Define Your host¶
The DisallowedHost…Invalid HTTP_HOST header…You may need to add ‘’ to
ALLOWED_HOSTS is a common error received when first trying to get
GovReady-Q running on a server at a specific domain. The error indicates
the domain you are trying to visit is not white listed in Django’s
special ALLOWED_HOST variable.
For security, Django requires white listing your server’s domain(s) in
the ALLOWED_HOST variable. Ordinarily this is hardcoded into the
settings.py file. GovReady-Q allows the ALLOWED_HOST to be set
by the host environment settings so the values can be passed at
runtime.
hostmust be defined, or GovReady-Q will default value tolocalhost
Setting up the Database Server¶
For production deployment, it is recommended to use dedicated database software, rather than SQLite.
The recommended database is PostgreSQL - see instructions on setting up Q with PostgreSQL
Setting up a Webserver¶
It’s recommended to run a dedicated webserver software, such as Apache or Nginx, as a reverse proxy in front of the Q application (running through uWSGI). To read how to do this, see instructions on setting up Q with a reverse proxy webserver.
Creating the First User¶
Create the initial user and a “main” organization using:
python3 manage.py first_run
You should now be able to log into GovReady-Q using the user created in this section.
You can also use the Django admin to create organizations.
Other Configuration Settings¶
Set up email by adding to local/environment.json:
"admins": [["Your Name", "you@company.com"]],
"email": {
"host": "smtp.server.com", "port": "587", "user": "...", "pw": "....",
"domain": "webserver.hostname.com"
},
"mailgun_api_key": "...",
Updating Deployment¶
When there are changes to the GovReady-Q software, pull new sources and restart processes with:
# replace $DISTRO with an appropriate value.
# Currently-supported options include "rhel" and "ubuntu"
sudo -iu govready-q /home/govready-q/govready-q/deployment/$DISTRO/update.sh
As root, you can also restart just the Python/Django process:
sudo supervisorctl restart all
But this won’t do a full update so don’t normally do that (it won’t restart the separate notifications process or generate static assets, etc.).
Setting up a Database for Production Workloads¶
The preferred production database for Q is PostgreSQL, but MySQL/MariaDB is also supported.
SQLite is supported in development environments, but not recommended for production use.
Setting Up Postgres¶
These instructions assume a separate database server and webapp server.
On the database server¶
On the database server, install Postgres. If using RHEL, CentOS, or similar:
yum install postgresql-server postgresql-contrib
postgresql-setup initdb
In /var/lib/pgsql/data/postgresql.conf, enable TLS connections by
changing the ssl option to
ssl = on
and enable remote connections by binding to all interfaces:
listen_addresses = '*'
Enable remote connections to the database only from the webapp server
and only encrypted with TLS by editing
/var/lib/pgsql/data/pg_hba.conf and adding the line (replacing the
hostname with the hostname of the Q webapp server):
hostssl all all webserver.hostname.com md5
Generate a self-signed certificate (replace db.govready-q.internal
with the database server’s hostname if possible):
openssl req -new -newkey rsa:2048 -days 365 -nodes -x509 -keyout /var/lib/pgsql/data/server.key -out /var/lib/pgsql/data/server.crt -subj '/CN=db.govready-q.internal'
chmod 600 /var/lib/pgsql/data/server.{key,crt}
chown postgres.postgres /var/lib/pgsql/data/server.{key,crt}
Copy the certificate to the webapp server so that the webapp server can make trusted connections to the database server:
cat /var/lib/pgsql/data/server.crt
# place on webapp server at /home/govready-q/pgsql.crt
Then restart the database:
service postgresql restart
Then set up the user and database (both named govready_q):
sudo -iu postgres createuser -P govready_q
# paste a long random password
sudo -iu postgres createdb govready_q
Postgres’s default permissions automatically grant users access to a database of the same name.
And if necessary, open the Postgres port:
firewall-cmd --zone=public --add-port=5432/tcp --permanent
firewall-cmd --reload
On the webapp server¶
On the webapp server, now check that secure connections can be made:
psql "postgresql://govready_q@dbserver.hostname.com/govready_q?sslmode=verify-full&sslrootcert=/home/govready-q/pgsql.crt"
(It should fail if the TLS certificate file is not provided, if sslmode
is set to disable, if a different user or database is given, or if
the wrong password is given.)
Then in our GovReady-Q local/environment.json file, configure the
database (replace THEPASSWORDHERE) by setting the following key:
"db": "postgresql://govready_q:THEPASSWORDHERE@dbserver.hostname.com/govready_q?sslmode=verify-full&sslrootcert=/home/govready-q/pgsql.crt",
Then initialize the database content:
python3 manage.py migrate
python3 manage.py load_modules
And generate static files:
python3 manage.py collectstatic
Configuring a Reverse Proxy Webserver for Production Use¶
Setting Up Apache & uWSGI¶
Install Apache 2.x with SSL (back to being root):
yum install httpd mod_ssl
Copy the Apache config into place:
cp /home/govready-q/govready-q/deployment/rhel/apache.conf /etc/httpd/conf.d/govready-q.conf
And then edit the file replacing q.govready.com and
*.govready.com with your hostnames.
If you don’t have a TLS certificate ready to use, create a self-signed
certificate (replacing webserver.hostname.com with your hostname):
openssl req -new -newkey rsa:2048 -days 365 -nodes -x509 -keyout /home/govready-q/ssl_certificate.key -out /home/govready-q/ssl_certificate.crt -subj '/CN=webserver.hostname.com'
chmod 600 /home/govready-q/ssl_certificate.{key,crt}
chown apache.apache /home/govready-q/ssl_certificate.{key,crt}
If SELinux is enabled (sestatus shows SELinux status: enabled),
grant the Apache process access to these files as well as the site’s
static files:
chcon -v -R --type=httpd_sys_content_t /home/govready-q/govready-q/deployment/rhel/apache.conf /home/govready-q/ssl_certificate.{key,crt} /home/govready-q/public_html
and grant Apache permission to make network connections so that it can connect to the Python/uwsgi backend running GovReady-Q:
setsebool httpd_can_network_connect true
Install supervisor which will keep the Python/Django process running
and symlink our supervisor config into place:
yum install supervisor
ln -s /home/govready-q/govready-q/deployment/rhel/supervisor.ini /etc/supervisord.d/govready-q.ini
Restart services:
service supervisord restart
service httpd restart
And if necessary open the web ports:
firewall-cmd --zone=public --add-port=80/tcp --permanent
firewall-cmd --zone=public --add-port=443/tcp --permanent
firewall-cmd --reload
GovReady-Q should now be running and accessible at your domain name. Follow the instructions in the main README.md for creating your first organization.
Setting up an HTTPS Certificate¶
The instructions above created a self-signed certificate to get the
website up and running. To use Let’s Encrypt to automatically provision
a real certificate, install and run certbot:
yum install -y python-certbot-apache
certbot --apache -d webserver.hostname.com
# and follow the prompts
Then set it to automatically renew certificates as needed:
# edit root's crontab
crontab -e
# insert at end:
30 2 * * * /usr/bin/certbot renew >> /var/log/le-renew.log
Setting up Nginx¶
Configure nginx to use nginx.conf in the govready-q directory:
# Turn off nginx's default website.
rm -f /etc/nginx/sites-enabled/default
# Put in our nginx site config.
ln -sf /home/govready-q/govready-q/deployment/ubuntu/nginx.conf /etc/nginx/sites-enabled/yourdomain.com
service nginx restart
The nginx conf file assumes a certificate chain and private key are
present at /etc/ssl/local/ssl_certificate.crt/key.
Environment Settings¶
Available Configuration Settings¶
The following environment variables are used to configure your
GovReady-Q instance and can be configured through your local
local/environment.json or passed in as environmental parameters when
launching a container with GovReady-Q.
admins: used to configure a display point of contact “Administrator” on site and unrelated to the configuration of actual administrators configured in the database.branding: used for `custom branding`_.db: if supplied, this is the DB connection used. See `DB configuration`_.debug: should befalseor absent in production environments. If set totrue, turns on certain debug/development settings.email: used to configure access to a mail server for sending and receiving email. Object has the following format:{"host": "smtp.server.com", "port": "587", "user": "...", "pw": "....", "domain": "webserver.hostname.com"}. See `Configuring email`_ and `Other Configuration Settings`_. },.govready_cms_api_auth: used to store API key to interact with GovReady’s CMS agent and dashboard. Not relevant to most users. See `GovReady-CMS-API`_.host: this is the domain name (HTTPHostheader) used for root-level GovReady-Q pages. See alsoorganization-parent-domain, which is used to construct organization subdomains (if using a different base domain).https: set to true to generate HTTPS headers and urls when site is running behind a proxy terminating HTTPS connections. See `Configuring a Reverse Proxy Webserver for Production Use`_.mailgun_api_key: used to hold API key for using mailgun to send/receive emails.memcached: if setting is true, enable a memcached cache using the default host/portorganization-parent-domain: this is the domain name (HTTPHostheader) suffix used for organization-specific GovReady-Q pages.organization-seen-anonymously: show list of all created organizations to an anonymous (e.g., not signed-in) user on home page if set to true.secret-key- used to make instance more secure by contributing a salt value to generating various random strings and hashes. Do not share.gr-pdf-generator- specifies the library/process used to generate PDFs, options are off and wkhtmltopdf and default is None.gr-img-generator- specifies the library/process used to generate images and thumbnails, options are off and wkhtmltopdf and default is None.single-organization: used to enforce single organization mode with “main” as subdomain of default organization instead of multi-tenant with different subdomains for different organizations.static: used to prepend a root path to the default/static/URL path for accessing static assets.syslog: used to set the host and port of a syslog-compatible log message sink. (Default: None.)trust-user-authentication-headers: used to activate reverse proxy authentication. See `Proxy Authentication Server`_.
Production Deployment Environment Settings¶
A production system deployment may need to set more options in
local/environment.json. Here are recommended settings:
{
"debug": false,
"admins": [["Name", "email@domain.com"], ...],
"host": "q.<yourdomain>.com",
"https": true,
"secret-key": "something random here",
"static": "/root/public_html"
}
Enterprise Single-Sign On Environment Settings¶
GovReady-Q supports Enterprise Login via IAM (Identity and Access Management). In this configuration, GovReady-Q is deployed behind a reverse proxy that authenticates users and passes the authenticated user’s username and email address in HTTP headers.
To activate reverse proxy authentication, add the header names used by
the proxy to your local/environment.json, e.g.:
{
"trust-user-authentication-headers": {
"username": "X-Authenticated-User-Username",
"email": "X-Authenticated-User-Email"
},
}
GovReady-Q must be run at a private address that cannot be accessed except through the proxy server. The proxy server must be configured to proxy to GovReady-Q’s private address. See Enterprise Single-Sign On / Login for additional details.
PDF and Image Generation Environment Settings¶
GovReady-Q optionally supports generating PDFs and custom thumbnails for
uploaded files using wkhtmltopdf and wkhtmltoimage. Admins must
make sure the wkhtmltopdf library is installed properly for operating
system being used.
GovReady-Q PDF generation and thumbnails are turned off by default for security reasons.
PDF generator library wkhtmltopdf has security issues wherein individuals could add
HTML references such as links or file references inside the documents
they are creating which the PDF Generator blindly interprets. This leads
to SSRF (Server Side Request Forgery) in which data is retrieved from
server and added to PDF by the PDF Generator. An issue also exists
with the sub-dependency of libxslt before CentOS 8.x raising CVE vulnerability
with scanners. For these reasons, PDF Generation is being a configurable setting.
To activate PDF and thumbnail generation, add gr-pdf-generator and
gr-img-generator environmental parameters to your local/environment.json, e.g.:
{
...
"gr-pdf-generator": "wkhtmltopdf",
"gr-img-generator": "`wkhtmltopdf",
...
}
Custom Branding Environment Settings¶
You may override the templates and stylesheets that are used for
GovReady-Q’s branding by adding a new key named branding that is the
name of an installed Django app Python module (i.e. created using
manage.py startapp) that holds templates and static files. See
Applying Custom Organization Branding.
Enterprise Single-Sign On / Login¶
Proxy Authentication Server¶
GovReady-Q can be deployed behind a reverse proxy that authenticates users and passes the authenticated user’s username and email address in HTTP headers. In this configuration:
The user points their browser to the reverse proxy authentication server.
The proxy authenticates users and proxies the request to GovReady-Q if and only if the user is authenticated and authorized to access GovReady-Q. The proxy passes the user’s username and email address in HTTP headers of the proxy’s choosing.
GovReady-Q will create a user account for a new user or treat the user as logged in as soon as the user requests a page. Therefore, there is no sign-up or log-in process within GovReady-Q when a proxy authentication server is used.
All other authentication methods to GovReady-Q are disabled when proxy authentication is enabled. Therefore you should ensure that the Django admin’s username matches the admin’s username as provided by the proxy server. Otherwise, you will lose access to the admin page. However, if there is a mismatch, you may disable proxy authentication, log in to the Django admin with your admin username and password, and change your admin username to match the username sent by the proxy server.
GovReady-Q must be run at a private address that cannot be accessed except through the proxy server.
To activate reverse proxy authentication, add the header names used by
the proxy to your local/environment.json, e.g.:
{
"trust-user-authentication-headers": {
"username": "X-Authenticated-User-Username",
"email": "X-Authenticated-User-Email"
},
}
The proxy server must be configured to proxy to GovReady-Q’s private
address. But the host and https settings in GovReady-Q’s
local/environment.json file must reflect the host and protocol used
in the URL the end user uses to access GovReady-Q. They do not need
to match the address that the proxy server uses to reach the GovReady-Q
server.
Per the Django Documentation on authentication using REMOTE_USER, you must be sure that your proxy server always sets or strips the special username and email headers, including headers that normalize to the same Django key (in particular headers with underscores), from the client request and does not permit an end-user to submit a fake (or “spoofed”) header value.
We have an example reverse proxy authentication server at https://github.com/GovReady/govready-q/tree/master/tools/simple_iam_proxy_server which can be used for debugging purposes.
Applying Custom Organization Branding¶
The look and feel of GovReady-Q can be customized a bit by overriding the Django templates that are used to construct the site’s pages and by serving additional static assets.
Custom branding can contain static assets (such as a logo image) and HTML template overrides. Branding is packaged into a directory with a particular directory layout and some Python boilerplate code that allows GovReady-Q to find the branding files. The directory is placed inside the main GovReady-Q directory, and an application setting is used to activate it.
Before setting out to create custom branding, make sure you have GovReady-Q set up for development on your workstation. You’ll need a working setup of GovReady-Q to create the branding directory and to test your changes.
Creating the branding directory¶
Custom branding is packaged inside what Django confusingly calls an application, but it is just a packaged sub-component of a website. To create a new branding package directory, change to the directory where you have GovReady-Q set up. Then run:
python3 manage.py startapp sample_branding
This command creates a new directory called sample_branding with
Python boilerplate code to make it a valid Django “application.”
Make directories for storing the custom static assets and templates:
mkdir sample_branding/static
mkdir sample_branding/templates
Activate the branding package¶
Next, let your development installation of GovReady-Q know that you want
to use the custom branding package. In your local/environment.json
file, add a setting named branding and set it to the name of the
custom branding package directory.
"branding": "sample_branding",
See Environment Settings for more information
about the local/environment.json file. Note that for the file to be
valid JSON the last setting cannot have a trailing comma.
Overriding templates¶
Any of the templates that make up GovReady-Q’s frontend can be overridden. The full list of templates can be browsed in GovReady-Q’s GitHub repository at
https://github.com/GovReady/govready-q/tree/master/templates
Start by trying to override the navbar.html template, which is
inserted at the top of every page. Use your favorite text editor to
create a file at sample_branding/templates/navbar.html. Copy the
content of GovReady-Q’s stock navbar.html from
https://github.com/GovReady/govready-q/blob/master/templates/navbar.html
into it. (GitHub’s “Raw” button is handy for getting a clean version to
save or copy/paste.)
At the bottom of the file, add some custom HTML, such as:
<div>
<b>Welcome to my organization’s custom site!</b>
</div>
Start GovReady-Q on your workstation (see the development docs) and visit a page. You should see your new content below the navbar at the top of every page.
Adding custom CSS¶
You can also add a custom CSS stylesheet to your branded GovReady-Q by taking the following steps:
Add the CSS file as a static asset.
Insert a
<link rel="stylesheet" href="...">tag into the<head>section of each page’s HTML by overriding thehead.htmltemplate.
To create the static asset, make a new file named
sample_branding/static/custom.css. Let’s say you want to make the
background color of each page red. The file should contain:
body {
background: red !important;
}
Then override the head.html template. GovReady-Q’s base for
head.html is empty — its purpose is only to allow you to add to the
<head> element. So create a new file at
sample_branding/templates/head.html and put in it:
{% load static %}
<link rel="stylesheet" href="{% static "custom.css" %}">
See the Django documentation for static
files for
more information about the static template tag.
Open any page in your locally running GovReady-Q and you should see that the background color of every page has changed.
Keeping your templates up to date¶
With each new released version of GovReady-Q, there is the possibility that the stock templates have changed. Some changes may require you to re-engineer your template overrides to preserve functionality.
Creating a custom Docker image¶
If your organization is deploying GovReady-Q using Docker, you will need to embed your custom branding package within a Docker image. You have two options:
Modify GovReady-Q’s stock Dockerfile, i.e. the one in GovReady-Q’s source code, to add and activate your branding package and then build your own GovReady-Q Docker image from the GovReady-Q source files that you cloned from GitHub.
Make your own Dockerfile that uses a released GovReady-Q image as its parent image and adds to it just the steps needed to add and activate your branding package.
Creating your own Dockerfile that uses a released GovReady-Q image as its parent image¶
We recommend method 2. To create your own Dockerfile that uses a
released GovReady-Q image as its parent image, create a new
Dockerfile in your branding package directory, e.g. a new file named
Dockerfile in the sample_branding directory you created earlier.
Then choose which parent image you will use from the available
GovReady-Q tags.
Each tag corresponds to a release version. Your Dockerfile begins with a
FROM line that combines govready/govready-q: with the tag name
you choose. In this example we use the latest tag which is an alias
for the most recent version of GovReady-Q:
FROM govready/govready-q:latest
The subsequent commands in your Dockerfile configures the container, picking up where the parent image’s Dockerfile leaves off. For more information about the parent image, refer to GovReady-Q’s Dockerfile on GitHub.
Your Dockerfile’s next step is to add your branding package into the
image in a directory named branding:
RUN mkdir branding
COPY . branding
Finally, you’ll need some commands to adjust permissions, to activate the branding package when GovReady-Q starts, and to prepare the static assets to be served. The complete Dockerfile should look like this:
# Build an image on top of the stock GovReady-Q image.
FROM govready/govready-q:latest
# The parent Dockerfile ends with 'USER application' to run the
# container as a non-privileged user. But we need to go back to
# root to add additional files and then switch back to the non-
# root user at the end.
USER root
# Copy our public app files into place.
RUN mkdir branding
COPY . branding
# Activate the branding package. The environment variable is read
# by dockerfile_exec.sh in the GovReady-Q parent image. And modifying
# /tmp/environment.json is necessary at this step so that collectstatic
# picks it up below.
ENV BRANDING branding
RUN sed -i "s/}/,\"branding\": \"branding\" }/" /tmp/environment.json
# Flatten static files. The base image did it once, but we may have
# added new static files so we must do it again.
RUN python3.6 manage.py collectstatic --noinput
# Run the container's process zero as this user --- see above.
USER application
# Check that everything looks good.
RUN python3.6 manage.py check
Finally you can build and test your custom image.
Building your docker image¶
If you were in the GovReady-Q sources directory, move into your branding package directory:
cd sample_branding
Then fetch the parent image and build your image:
docker image pull govready/govready-q:latest
docker image build --tag myorg/govready-q-branded:latest .
(Substitute the right tag depending on the tag you chose for the
FROM line in your Dockerfile.)
Test that your image works by launching a new container based on your image:
docker container run --rm -it -p 127.0.0.1:8000:8000 myorg/govready-q-branded:latest
Once GovReady-Q is running in the container, visit it at
http://localhost:8000. Use CTRL+C in the console to terminate and
destroy the test container running your image.
For more about running GovReady-Q with Docker, see Deploying with Docker.
Multi-Container GovReady-Q and NGINX via Docker Compose¶
Overview¶
This directory contains configuration files that run two Docker containers, one for GovReady-Q and the other for NGINX, as a multi-container app. NGINX is used in a reverse proxy configuration, to handle incoming HTTP and HTTPS requests, which it then passes to GovReady-Q.
Use Docker Compose to manage the multi-container app.
Docker Compose commands are similar to, but different from, regular Docker commands. Read the Docker Compose docs for more details.
Set Up A Docker Host¶
Workstation¶
Install Docker and Docker Compose on your workstation.
On Mac and Windows, Docker Compose is included as part of the Docker install.
On Linux, after installing Docker, install Docker Compose. Also, you may want to grant non-root users access to run Docker containers.
Docker Machine¶
Docker Machine can be used to set up Docker host on either a local or cloud server. Once you have configured your shell to connect to a Docker host set up by Docker Machine, the Docker Compose commands you need to use will be the same as if you were using the Docker engine running on your workstation as the Docker host.
Get This Kit¶
Get the files by cloning the GovReady-Q repository.
git clone https://github.com/GovReady/govready-q.git
cd govready-q/deployment/docker-compose-nginx/
Make sure you are in the ``docker-compose-nginx`` directory.
Any docker-compose commands will need the docker-compose.yml
file to know which containers to operate on.
SSL/TLS Certificates for HTTPS¶
There are self-signed certs including in the nginx directory. They
are copied into the nginx container for nginx to refer to.
Self-signed certs are sufficient to allow GovReady-Q and NGINX to work together with your browser. However, you will get a security exception notice from your browser, and you will have to approve the “unsafe” exception to proceed.
To use real certs issued against a CA your browser will recognize, you
can replace the cert.pem and key.pem files and issue the
docker-compose build file, or you can mount a data volume with your
certs in it to /etc/pki/tls/certs/ .
Later versions of this project may include more documentation about the volume method, or other ways to include certs, such as Let’s Encrypt.
Build Images¶
You need to build images whenever you make changes to the Dockerfiles or
nginx config file. If you don’t make changes, though, the command in
the next section will automatically build the images.
To build the images:
docker-compose build
Run GovReady-Q + NGINX Multi-container App¶
To start the containers:
docker-compose up -d
Using the -d detaches the containers and runs them in the
background.
If you prefer, you can omit -d, and then output will be printed to
your console window. If you hit ^C, the containers will shut down
gracefully. If you hit ^C^C they will be terminated immediately.
Two containers will be created, one for each “service” (as they’re called in Docker Compose).
Docker Compose gives these containers names like
docker-compose-nginx_govready-q_1 and
docker-compose-nginx_nginx_1. These are three-part names, with the
parts separated by underscores. docker-compose-nginx comes from the
name of this project (the directory it’s in). The second element is the
service name (govready-q or nginx). The third element is a
serial number (ascending from 1) for multiple instances of the same
service. The docker-compose.yml file here only specifies one
instance, so the number will always be 1.
You can check the status of the containers:
docker-compose ps
Specify Parameters¶
Before starting the containers, you can specify which GovReady-Q image to use, which database host to use, and the hostname of the Docker host. It’s important to specify the correct hostname if you are using real TLS certs.
Set these environment variables (sample values provided, replace with your own values):
export GOVREADY_Q_HOST=ec2-nnn-nnn-nnn-nnn.us-east-1.compute.amazonaws.com
export GOVREADY_Q_DBURL=postgres://govready_q:my_private_password@grq-002.cog63arfw9bib.us-east-1.rds.amazonaws.com/govready_q
export GOVREADY_Q_IMAGENAME=govready/govready-q-0.9.0
After setting the variables, continue with the “Run GovReady-Q + NGINX Multi-container App” section above.
If you don’t set environment variables, these defaults are used:
export GOVREADY_Q_HOST=test.example.com
export GOVREADY_Q_DBURL=
export GOVREADY_Q_IMAGENAME=govready/govready-q
When no DBURL is specified, GovReady-Q uses an internal sqlite database.
Check Logs From A Container¶
Check the logs by specifying the service name:
docker-compose logs govready-q
docker-compose logs nginx
GovReady-Q Is Up¶
GovReady-Q will boot up, and be ready to answer web requests in 20-30 seconds.
It will answer HTTP on the standard port, 80, and HTTPS on the standard port, 443.
Visit https://localhost/. (Or http://localhost, which will be
redirected to https by nginx.)
The default hostname used for this project is test.example.com. To
check it, put this entry in your /etc/hosts file:
127.0.0.1 test.example.com
When you have /etc/hosts set up, visit https://test.example.com/
Execute A Script In A Container¶
You can exec a script inside one of the containers by specifying the
service name. Unlike normal docker, you do not specify -it to
make the exec interactive.
Here we are executing the first_run script inside the govready-q
service/container.
docker-compose exec govready-q first_run
Permissions¶
This document describes the user permissions model in Q.
What Q tracks¶
GovReady-Q tracks the following major entities
Users - individuals with logins to an installed instance of Q
Organizations - entities, e.g., companies, around with data in Q is segmented
Folders - collections of Projects
Projects - instantiations of Compliance Apps
Membership - associating individual users with organizations and systems
Tasks/Modules - coherent grouping of questions and educational content
Questions/Answers - specific snippet of content within a Task
Templates - drives automatic generation of artifacts, supporting variable substitution
Users¶
Global user data¶
A User is authenticated by a unique username and a hashed password.
Each User has one or more email addresses associated with their account and notification email settings.
User accounts — i.e. the fields above — are global to a Q deployment. They are not segmented by Organization. For instance, if a User has been created for one Organization, they sign in — and not sign up — into other Organizations. Changing a User’s password changes it for all Organizations. (Of course, a User will not be authorized to access all Organizations. See the next section on Organizations.)
Segmented user data¶
Each User additionally has a unique “organization user profile” associated with each organization to support the same User having different roles at different organizations. (This user profile is an “account project” that holds additional User information including a User’s full name, profile photo, etc. In short, each User has a different profile in each Organization.
The profile information can be seen by all other members of the Organization because it is used in the history of question answers, notifications, discussions, and many other places.
(The User is the only “member” of their account projects (see Project membership below), which means they are the only User who can edit the information.)
System staff¶
Users marked as staff in the Django admin can see Q Analytics.
Portfolios¶
Portfolios are used to segment Projects and the data contained within them to create data isolation at a logical level.
Access to Portfolios¶
Each Portfolio begins with an ‘owner’ creating the portfolio. When any user signs up, a portfolio is automatically created with the title as the users username and assigning ownership to the user. An owner of a portfolio can grant other users access to their porfolio. If a user has access to a portfolio, they will have access to the projects within that portfolio. If an owner of a portfolio removes a users access to a portfolio, they will lose access to the projects within that portfolio.
Folders¶
A Folder is a collection of one or more Projects (see below) within the same Organization.
Folder permissions are based in part on Project permissions:
A user can see that a Project is a part of a Folder if the user has read access on the Project.
A user can add Projects to a Folder or rename a Folder if the user is an administrator of any Project within the Folder or is an administrator of the Folder itself. These users may not be able to see all Projects within the folder if they do not have read access to those Projects, but they will be told how many Projects they can’t see in the Folder.
There is no separate “read” permission on a Folder. A Folder can be seen just when a user has read access on a Project within it or is an administrator of the Folder itself.
Projects¶
Each time an app is started from the Compliance Store, a new Project is created. A Project represents the instantiated app and is comprised of a collection of Tasks. Every Project belongs to exactly one Organization.
Membership¶
Projects have zero or more Users who are members and zero or more Users who are administrators.
Read Access¶
Any access to a Project requires read access, which is granted if any of the following are true:
They are a member or administrator of the Project.
They have read access to any Task in the project.
They are a guest participant in a Discussion within the Project.
(This is a subset of the requirements for membership in an Organization, therefore read access to a Project guarantees membership in the Organization it belongs to.)
Operations¶
Project members can begin Tasks listed on Project pages, either by adding apps from the Compliance Store or starting Tasks for modules contained in the Project’s app. Project members can also invite guests to discussions.
Only administrators can send invitations to add new project members, import and export Project data, and delete Projects.
The Tasks (the questions and answers) within a Project are further restricted (see Tasks below).
New Projects¶
Any member of an Organization can create a new Project within that Organization by starting a new Compliance App and becomes the Project’s first administrator.
When a User creates a new Project, they are offered Compliance Apps from
AppSources whose Available to all option is checked and from
AppSources that don’t have ‘Available to all’ checked but explicitly
list the Organization in its ‘Available to orgs’ list.
New Projects are added into a new or existing Folder (for existing folders, see Folder permissions above).
Tasks¶
A Task is a set of questions and answers. Tasks represent the state of a Project — each Project has a root Task — as well as the state of all the modules started within the Project.
Each Task belongs to exactly one Project. Each Project has exactly one root Task.
A Task has an editor, which is the User who has primary responsibility for completing the Task.
A User has both read and write access to a Task if any of the following are true:
They are the editor of the Task.
They are a member or administrator of the Project that the Task belongs to.
A User with read access can see the Task on the page for the Project that it belongs to and can see all of its questions, answers, and outputs and can start a Discussion on questions.
A user can also see a particular question within a Task (and its answers and some Task metadata, but not other questions or Task outputs) if they are a guest in a Discussion on that question.
A User with write access to a Task can answer questions within the Task (which sometimes involves starting new Tasks which they become the editor of), invite other users to become the Task’s new editor, and delete/undelete the Task (although there is no UI for that currently).
Authoring Compliance Apps¶
Understanding Compliance Apps¶
Compliance apps map IT System components to compliance controls. A “component” can be any part of a system that contributes to its operation including organizational processes.
Compliance apps collect and assess information about one or more system components and translate that information to compliance documentation.
Apps map components to security/compliance controls¶
Compliance apps can collect information about a system component from people (via web-based questionnaires) and from system components (via an Automation API).
Compliance Apps are Collections of Modules¶
A Compliance app is a collection of “modules” for gathering information. A module is a collection of questions and output documents. A module can have just questions and output documents, just output documents and no questions, or both questions and output documents.
An “app” is a collection of “modules,” one of which must be named “app.” Modules are linear sequence of questions presented to users that produces zero or more output documents. Modules are stored in YAML files. Output documents of various types are supported such as markdown, HTML, and YAML. (See Modules, Questions, and Documents for documentation on writing modules.)
The typical user experience will be to first pick a “top level” app from the compliance catalog representative of IT System, then pick the “component” apps that represent the specific components of the IT System, and then iteratively complete the questions within the component apps modules.
Technically speaking, a top level app is a module containing questions whose answers are other apps.
The below diagram depicts an exploded view of the relationships between a top level app to a component app to modules and questions.
App Structure¶
Each app is defined by a set of YAML files and asset files stored in the following directory structure:
app_name
├── README.md
├── app.yaml
├── assets
│ ├── app.png
│ ├── image_one.yaml
│ ├── image_two.yaml
│ └── ...
├── module_one.yaml
├── module_two.yaml
└── ...
By convention, each app is required to have app.yaml file which
holds metadata for displaying the app in the compliance apps catalog,
such as its title and description, and an assets/app.png graphic
which displays as the app’s icon. app.yaml also holds the top-level
module questions which define the layout of the app’s main screen once
it is started by the user. The contents of README.md are also
displayed in the apps catalog.
Other module YAML file may be includes in the app as well, as needed.
The assets subdirectory can contain any static assets that will be
served when showing the app’s modules for the user, such as images
included in document templates. A file typically named app.png in
the assets directory is the app’s icon, which is displayed when browsing
the app catalog as well as when the app is used within another app, if
icon: app.png is specified in app.yaml.
App YAML¶
The app.yaml file that exists in every app serves two purposes:
It includes app catalog information, i.e. metadata, that will be shown in the app directory, such as the app’s short and long description, version number, vendor, etc.
It also defines a module (see Modules, Questions, and Documents) which defines the top-level layout of the app. The module may only contain questions whose type are
moduleormodule-set.
The app.yaml file looks like this:
App structure¶
id: app
title: My App
type: project
icon: app.png # refers to file in app's assets directory
protocol: # for inner apps only
- globally_unique_protocol_name
catalog:
categories:
- Category Name
- Another category name
vendor: GovReady PBC
vendor_url: https://www.govready.com
status: Operational
version: 0.6
version-name: First Release
source_url: https://github.com/GovReady/govready-app-example
description:
short: |
One-line description of the app here, using Markdown.
long: |
Long description of the app here only if README.md is
not present.
It can be multiple paragraphs and is Markdown.
recommended_for:
- key_short: Org
value: Medium
- key_short: Tech
value: Drupal
- key_short: Role
value: Dev
questions:
- id: item1
title: Do A Thing
type: module
module-id: module1 # refers to module1.yaml within this app
tab: TabName
group: GroupName
... more questions here ...
output:
- tab: TabName
format: markdown
template: |
This (optional) content will appear at the top of the TabName tab.
The questions in the app YAML file can only be of type module and
module-set. The questions can specify a module-id to refer to
another module within the same app or a protocol to allow the user
to choose any app that has a matching protocol value set at the top
level of the YAML file. See Modules, Questions, and
Documents for details on these question types.
A module YAML structure is identical to app.yaml structure but
without the catalog details section.
Hide “Question Skip” Buttons¶
As of version 0.8.6, the “I don’t know” and “It doesn’t apply” buttons to skip questions can be hidden from users.
We recommend you never use this feature.
This feature was added to support the use case of complex, legacy questionnaire assessments that (1) cannot be changed easily and (2) provide a better user experience when users get stuck and start a discussion instead of later learning that skipping a question caused them to miss many other questions.
Some background is useful. GovReady-Q was designed for users to love easily and quickly answering as few questions as possible to generate the information that organizational processes need. Users love the option to skip questions. Skipped questions empower users to move fast, answer what they can, and iteratively complete work. That gets information to you quickly and reduces cycle time. Skipped questions also provide instant feedback that a user doesn’t have information readily available. Frequently skipped questions indicate stumped users, and the need to rethink the question or use multiple questions and interstitials to better guide users.
We’ve all experienced the frustration of not understanding what is being asked of us, not knowing whom to ask for help, or knowing the question asked does not apply. This frustration turns galling and Kafkaesque when the party demanding compliance confounds our ability to comply.
Fight-or-flight response kicks in when users feel trapped. Users start to avoid your process or combat it. They can (unfairly) transfer their frustration onto your role, or worse, onto you. Instead of helping to pull your colleagues into your process, they feel you are pushing them away.
So if you really, really need to hide the skip buttons to make the
experience easier for your users, add a hidden-buttons array key to
the app.yaml file and list the skip buttons to hide. The example
snippet below hides just the “I don’t know button”.
id: app
title: My App
type: project
icon: app.png # refers to file in app's assets directory
protocol: # for inner apps only
- globally_unique_protocol_name
hidden-buttons:
- no-idea
The possible YAML array values for the buttons are no-idea (“I have
no idea”), not-applicable (“It doesn’t apply”), not-now-button
(“I’ll come back”), and not-sure-button (“Unsure”).
NOTE: As of version 0.8.6, the “I’ll come back” and “Unsure” functionality been removed from the UI for all cases because of poor user experience but are preserved for legacy data and potential future use with a better UI.
Top Level Apps¶
Apps that describe the required components of a compliant IT system are considered “Top Level” apps. Each question in a Top Level app specifies a type of compliance app (e.g., a compliance app “protocol”) that is needed to represent that component.
Adding Apps to GovReady-Q Deployments¶
Separating compliance apps from the compliance server enables a much richer ecosystem and virtuous cycle of innovation than having everything embedded exclusively within the compliance server. A GovReady-Q deployment can pull app and module content from local directories and git repositories. An organization using GovReady-Q can freely mix compliance apps from third parties with private compliance apps located only on their network.
Compliance apps are very much like modular plugins that customize the compliance server to the unique system and components of the organization.
This leaves the need to specify which compliance apps are available to a
compliance server deployment. This specification of available apps is
known as an “app source” and is done with a JSON “spec” file entered in
the AppSource model via the Django admin interface.
The process is currently a bit clumsy with terminology that reflects the software’s evolution toward the app concept. Nevertheless, the approach provides flexibility of sourcing apps from local file systems and public and private git repositories. And each source specifies a virtual filesystem from which one or more top level apps and compliance apps can be found located.
The below screenshot of the AppSource module in the Django admin
interfaces shows the JSON “spec” file.
Screenshot of AppSource from GovReady-Q Django admin interface¶
The AppSource module also contains fields to indicate to which
subdomains of the deployment the source’s apps are available.
App Source virtual filesystem layout¶
Whether the source is a local directory or a git repository, the source must have a directory layout in which each app is stored in its own directory. (The directory name becomes an internal name for the app.) For instance:
app1/app.yaml
app1/...other_app1_files
app2/app.yaml
app2/...other_app2_files
...
Updating modules¶
After making changes to modules or AppSources for system modules (like
account settings), run python3 manage.py load_modules to pull the
modules from the sources into the database. This only updates system
modules.
Other modules that have already been started as apps will not be updated. Each time you make a change to an app, you can reload changes using the app authoring tool in GovReady-Q.
Compliance App Authoring Tutorial¶
This is a step-by-step guide to creating compliance apps using the Docker version of the GovReady-Q Compliance Server.
In this guide you will learn how to:
Start and configure the Docker version of GovReady-Q
Create a compliance app
Edit a compliance app’s YAML files
Edit a compliance app using GovReady-Q’s authoring tools
Deploy the app to a production instance of GovReady-Q and storing apps in a source code version control repository
Step 1: Prepare your local environment¶
Create a folder on your workstation¶
GovReady-Q compliance apps are generally developed in an off-line development environment, usually on the app developer’s macOS or Linux workstation — any environment that can run Docker. In this environment, the compliance app data files will be stored in a local directory. This guide assumes the use of a local workstation for development and discusses production deployment at the end.
(Once the apps are ready to be published to the rest of the organization, the apps can be uploaded to a git repository, such as GitHub or an on-premise equivalent. The production instance of GovReady-Q will typically read compliance apps from the git repository directly and not from a local disk.)
On the development workstation, create a folder to hold GovReady’s install script, the GovReady-Q database (in development, Sqlite is used), and the compliance apps that you will be authoring. The folder can be anywhere:
mkdir /path/to/dev_directory
cd /path/to/dev_directory
Install Docker¶
If you haven’t already done so install
Docker on the
workstation and, if appropriate, grant non-root users access to run
Docker
containers
(or else use sudo when invoking Docker below).
Step 2: Install the GovReady-Q Compliance Server, Docker version¶
Starting the Docker container¶
Next download GovReady’s
docker_container_run.sh script. This
script simplifies passing various settings to create and configure the
govready-q docker container that we will use for local development.
wget https://raw.githubusercontent.com/GovReady/govready-q/master/deployment/docker/docker_container_run.sh
chmod +x docker_container_run.sh
docker_container_run.sh supports a variety of advanced
configuration settings via command
line parameters. The ones we care about for developing compliance apps
are:
--sqlitedb /path/to/govready-q-database.sqlite, which sets an absolute path to a Sqlite database that holds all persistent information across container runs--appsdevdir /path/to/apps, which sets an absolute path to the directory in which app YAML files will be developed--relaunch, which removes any existinggovready-qDocker container if one is running
Download and start GovReady-Q:
./docker_container_run.sh --sqlitedb `pwd`/database.sqlite --appsdevdir `pwd`/apps --relaunch
Note that pwd is used to ensure the paths are absolute.
The script will download the govready/govready-q
image from the Docker
Hub, which could take a few minutes. It will then start a new Docker
container named govready-q and will launch the Q source code within
it.
When the container is launched it will let you know the URL to visit:
GovReady-Q has been started!
Container Name: govready-q
Container ID: d99e8ac2d6a761cfd7be7f94bd01d5f7115efd66714064f7b1f0f6c09b74c269
URL: http://localhost:8000
(You can change the hostname and port by adding
e.g. --address q.company.com:8010.)
It takes about 15 seconds for the GovReady-Q server to be ready. Open the URL (e.g. http://localhost:8000) and reload a few times until the GovReady-Q Compliance Server becomes available:
First run of GovReady-Q¶
Setting up your organization and administrative user¶
Now that the GovReady-Q Compliance Server is running, create an administrative account and an organization. Run the following command and answer the prompts:
docker container exec -it govready-q first_run
Your prompt and reply will look something like this:
Installed 2 object(s) from 1 fixture(s)
Let's create your first Q user. This user will have superuser privileges in the Q administrative interface.
Username: admin
Email address: admin@mycompany.com
Password:
Password (again):
Superuser created successfully.
Let's create your Q organization.
Organization Name: The Company, Inc.
Now return to your browser, reload the page, and notice the company name has updated:
First run of GovReady-Q with the organization configured¶
You can now sign in with the administrative username and password you created.
First run after logging in¶
Congratulations! You’ve installed GovReady-Q Compliance Server configured for local development of compliance apps!
Step 3: Creating a compliance app¶
Creating the app¶
In this section we’ll create our first compliance app. The app will
appear in the compliance apps catalog in GovReady-Q. Click
Add other app in your browser to go to the compliance apps catalog.
Compliance apps catalog¶
Let’s create our first compliance app! Use the command below:
docker container exec -it govready-q ./manage.py compliance_app host myfirstapp
The output will be:
Created new app in AppSource host at /usr/src/app/q-files/myfirstapp
The path shown in the output is a path within the container’s
filesystem, which is inaccessible from the workstation. The actual path
is inside the path given to the --appsdevdir command line argument
previously. If you followed our steps above exactly, you can see the
app’s files in your apps folder:
$ ls -l apps/myfirstapp
-rw-r--r-- 1 root root 664 Oct 25 11:43 app.yaml
drwxr-xr-x 2 root root 4096 Oct 25 11:43 assets
-rw-r--r-- 1 root root 449 Oct 25 11:43 example.yaml
Head back to your browser and reload the compliance apps catalog page.
Your new app may not appear because the catalog is cached. To clear the cache, restart the container:
docker container restart govready-q
After a few moments the container will be back up. Reload the compliance app catalog page. You should now see your app if you scroll to the end:
Compliance apps catalog with your first app¶
The development directory on the workstation now holds:
.
├── apps
│ └── myfirstapp
│ ├── app.yaml
│ ├── assets
│ │ └── app.png
│ └── example.yaml
├── database.sqlite
└── docker_container_run.sh
(More information about the structure of the app directory can be found in Understanding Compliance Apps.)
Editing app catalog metadata¶
Open apps/myfirstapp/app.yaml in a text editor. Edit the short
description and add some text describing the app you are building:
description:
short: |
Achieve compliance for our organization's systems.
Since this file was created by Docker, which is running as root, the file will be owned by root. You may need to use sudo to edit this file.
Reload the container to clear the app catalog cache:
docker container restart govready-q
And then reload the catalog page in your browser to see your description
beneath myfirstapp. You can also edit the app’s title and other
catalog metadata, including the app’s icon in
apps/myfirstapp/assets/app.png.
Compliance apps catalog after metadata change¶
Step 4: Edit the compliance app’s YAML files¶
Start the app¶
In your browser, click on the myfirstapp entry’s Add button in the
app catalog.
The new app¶
About editing the app¶
We can edit our new compliance app by editing its app.yaml and
example.yaml files on disk in our favorite text editor (described in
this section) or with GovReady-Q’s built-in authoring tools (described
in the next section).
After each edit to the compliance apps files on disk, it may be
necessary to restart the Docker container if you modified app catalog
metadata (as you did above with docker container restart) or start a
new instance of the compliance app from the compliance apps catalog page
in your browser, if you modified the app’s questions and output
templates.
GovReady-Q purposely does not automatically recognize changes to compliance apps on disk until a new instance of the app is selected or a reload command (described below) is issued. This ensures previously loaded versions of the compliance app correctly maintain data entered by end-users.
Editing the app’s main page¶
The opening screen of the app is determined by the questions section
of the app.yaml file:
questions:
- id: q1
title: Example Module
type: module
module-id: example
The new app has a single question labeled by the title
Example Module, as you see in the YAML and in your browser. When the
user clicks Example Module in the browser, they will start a new module
defined by the YAML file referenced in the module-id data, in this
case example.yaml.
Edit the title to:
title: Start Compliance
As described above, reloading the page in the browser will not show the change. This is by design. Since you are developing an app on your local filesystem, the GovReady-Q authoring tools are available.
Click Authoring Tool in the right column, and then click
Reload App from local filesystem. (Alternatively, you could return
to the compliance app catalog page and add the app again.)
Module authoring tools¶
Note how Start Compliance now appears in the browser.
Reloaded app¶
Editing the app’s first module¶
Click Start Compliance. This begins the app’s module defined in
example.yaml. The example module contains a single sample question:
The new app’s sample question¶
Open example.yaml and see that the question’s type, prompt, and
choices are defined in the YAML file’s question’s section:
questions:
- id: q1
title: What is your favorite science fiction franchise?
prompt: What is your favorite science fiction franchise?
type: choice
choices:
- key: startrek
text: Star Trek
- key: starwars
text: Star Wars
- key: lordoftherings
text: Lord of the Rings
- key: other
text: Other
Change the prompt or choices.
(As with app.yaml, since this file was created by Docker the file
will be owned by root. You may need to use sudo to edit this file.)
As described above, reloading the page in the browser will not show the
change. This is by design. Go back to the main app page, click
Authoring Tool and then Reload App from local filesystem, and
then go back to the Start Compliance page.
Your changes are now seen in your browser.
The revised question¶
More information about the file format of modules can be found in Modules, Questions, and Documents.
Step 5: Edit a compliance app using GovReady-Q’s authoring tools¶
About the authoring tools¶
It is also possible to edit a compliance app’s questions without leaving your browser. When editing the compliance app via GovReady-Q’s built-in authoring tools, you will immediately see the changes in the instance of the compliance app you are editing without having to reload it. The changes are also immediately written to the files on disk.
GovReady-Q’s built-in authoring tools will let you edit and add questions, but currently won’t let you change the name of the description of the app in catalog. You will still need to edit those details in the compliance app YAML files stored on disk, as described above.
Editing a question¶
A blue pencil icon will appear at the top right of module questions when the authoring tools are available. Click the pencil icon for the sample question. The question editor will pop up:
Question authoring tool¶
This is a much easier way of editing questions! Try editing this
question. After clicking Save Changes, look in your text editor to
see that the changes have been immediately saved to example.yaml.
Adding questions¶
It is also possible to add questions. In order to add a question, all of
the existing questions must be answered. Answer the sample question, or
click Skip.
You’ll see an Add Question button on the module review page:
Add question button¶
Try out the Add Question button now. It will create a new text question. Use the blue pencil icon to change the question’s prompt and choices.
You have now seen how to create and edit an app!
Step 6: Deploy the app to a production instance of GovReady-Q¶
Adding apps to a git repository¶
Your workstation’s instance of GovReady-Q has been configured to load apps from the local filesystem. Your organization’s production instance of GovReady-Q can be configured similarly, but more likely it will be configured to load apps from a remote git repository.
Create a new git repository in your source code control system and push
your apps directory to the repository. The repository’s root
directory should contain a directory named myfirstapp:
repository root
└── myfirstapp
├── app.yaml
├── assets
│ └── app.png
└── example.yaml
If you have an existing source code control system containing apps in
this layout, consider checking out the repository locally so that it is
in the same path provided to the --appsdevdir argument to
docker_container_run.sh. If your repository is in a different layout
or if you are using multiple repositories to store compliance apps, see
below.
Configuring a production system to load apps from the git repository¶
On the production GovReady-Q instance, log into the Django admin at
https://production-q/admin. Add a new App Source.
Set its Slug to a short name for the repository, composed of
letters, numbers, and underscores, such as mygitrepo.
If your git repository is public or accessible over an https: URL¶
If your git repository is accessible over an https: URL (such as a public GitHub repository), change the Source Type to Git Repository over HTTPS and paste the URL into the URL field. The other fields can be left blank. Here’s what that looks like:
App Source for a public git repository¶
If your git repository is private¶
If your git repository is private and accessible instead using an SSH URL (typically git@github.com:organization/repository.git) and an SSH public/private keypair, such as with GitHub or GitLab deploy keys, then first create a new SSH key for your GovReady-Q instance:
ssh-keygen -q -t rsa -b 2048 -N "" -C "_your-repo-name_-deployment-key" -f ./repo_deploy_key
Your GovReady-Q instance will hold the private key half of the newly
generated keypair, and your source code control system will hold the
public key. Back in the Django admin, set the Source Type to Git
Repository over SSH. Paste the SSH URL into the URL field. Then open the
newly generated file repo_deploy_key and paste its contents into the
SSH Key field. The other fields can be left blank. Here’s what that
looks like:
App Source for a private git repository¶
Copy the public key in the newly generated file repo_deploy_key.pub
into the deploy keys section of your source code repository. Here is
what that looks like on GitHub:
Adding a deploy key to GitHub¶
Other information about App Sources¶
As with local development, the production system’s compliance app catalog may be cached. To see new apps, restart the production instance of GovReady-Q.
See App Sources for more information about how to configure your production instance of GovReady-Q to load apps from local filesystem directories, git repositories (including on-prem git repositories), or GitHub.
Advanced setups for development with a repository of apps¶
In this guide we have used the --appsdevdir command to specify a
location in which app YAML files and assets are stored. In a small
setup, all apps could be stored in a subdirectory of the location given
to --appsdevdir. But you may want to separate apps into different
folders, such as if they are divided between folders in a single git
repository or across multiple git repositories, then a more advanced
configuration of GovReady-Q is necessary.
Imagine the following directory structure where two GitHub repositories
are cloned into two separate local directories within apps, and each
has a compliance_apps directory holding its apps:
.
├── apps (`--appsdevdir` directory)
│ ├── repo1
│ │ └── compliance_apps
│ │ ├── myfirstapp
│ │ └── mysecondapp
│ └── repo2
│ └── compliance_apps
│ ├── mythirdapp
│ └── myfourthapp
├── database.sqlite
└── docker_container_run.sh
The default setup from GovReady-Q docker installation only show apps in the compliance app catalog if the app files are located in the immediate subdirectory of path configured to load apps. But we can also tell GovReady-Q to load apps from multiple locations. In this case we will configure GovReady-Q to load apps from two locations:
apps/repo1/compliance_apps
apps/repo2/compliance_apps
Recall that the path given to --appsdevdir is mapped to a path
within the Docker container so that the container can see the YAML files
on the (host) local filesystem. The container sees these directories as
/usr/src/app/q-files/repo1/compliance_apps
/usr/src/app/q-files/repo2/compliance_apps
Log into the Django admin at http://localhost:8000/admin. Add two
new AppSource entries:
For the first, set the Slug to repo1 (or any other label that
will help you distinguish the two repositories), the Source Type to
Local Directory, and the Path to
/usr/src/app/q-files/repo1/compliance_apps. For the second, set the
Slug to repo2, the Source Type to Local Directory, and
the Path to /usr/src/app/q-files/repo2/compliance_apps.
Then restart the container:
docker container restart govready-q
and the apps defined in all of the repositories should be visible in the compliance app catalog.
App Sources¶
GovReady-Q can be configured by an administrator to load compliance apps from one or more sources, which can be local directories or remote git repositories.
When using the Hosted Version of GovReady-Q, GovReady PBC is the administrator. If you are the administrator of an installation of GovReady-Q at your organization, the information below will help you configure the App Sources available to your users.
App Sources are configured in the Django admin at the URL /admin on
your GovReady-Q domain under App Sources:
App Sources list¶
Each App Source points GovReady-Q to a directory or repository of compliance apps.
Example App Source¶
App Source Slug¶
The first App Source field is the Slug. The Slug is a short name you assign to the App Source to distinguish it from other App Sources. The Slug is used to form URLs in GovReady-Q’s compliance apps catalog, so it may only contain letters, numbers, dashes, underscores, and other URL path-safe characters.
App Source Type¶
There are four types of App Sources: local directories, remote git repositories using HTTP which are typically public repositories, remote git repositories using SSH which typically use SSH deploy keys for access, and remote GitHub repositories using a GitHub username and password for access.
Local Directory¶
The Local Directory source type directs GovReady-Q to load compliance apps from a directory on the same machine GovReady-Q is running on. (When deploying with Docker, that’s on the container filesystem unless a path has been mounted to a volume or to the host machine.)
In the Path field, enter the path to a local directory containing
compliance apps. This path is expected to contain a sub-directory for
each compliance app contained in this source. For instance, if you have
this directory layout:
.
└── home
└── user
└── compliance_apps
├── myfirstapp
│ └── app.yaml
└── mysecondapp
└── app.yaml
then your Path would be /home/user/compliance_apps.
The path can be absolute or relative to the path in which GovReady-Q is installed.
Git Repository over HTTPS¶
The Git Repository over HTTPS source type is for git repositories, such as on GitHub or GitLab, that can be cloned using an HTTPS URL. These repositories are typically public, or in an enterprise environment public within your organization’s network.
Paste the HTTPS git clone URL — such as https://github.com/GovReady/govready-apps-dev — into the URL field. Here’s what that looks like:
App Source for a public git repository¶
The other fields can be left blank.
The Path field optionally specifies a sub-directory within the
repository in which the compliance apps are stored if they are not
stored in the root of the repository. For instance if the repository has
a directory layout similar to:
.
└── github.com/organization/repository
└── apps
├── myfirstapp
│ └── app.yaml
└── mysecondapp
└── app.yaml
then set the Path field to apps.
If the compliance apps are not in the repository’s default branch
(i.e. something other than the typical master default branch), then
set the Branch field to the name of the branch to read the
compliance apps from.
You can use HTTPS to access private repositories by placing your username and password or personal access token into the URL, such as:
https://username:password@github.com/GovReady/govready-apps-dev
Since this requires user credentials, it should be avoided for production deployments in favor of using Git Repository over SSH (see below).
Git Repository over SSH¶
If your git repository is private and accessible using an SSH URL (which typically looks like git@github.com:organization/repository.git) and an SSH public/private keypair, such as with GitHub or GitLab deploy keys, then use the Git Repository over SSH source type.
Create a new SSH key for your GovReady-Q instance to be used as a deploy key:
ssh-keygen -q -t rsa -b 2048 -N "" -C "_your-repo-name_-deployment-key" -f ./repo_deploy_key
Your GovReady-Q instance will hold the private key half of the newly generated keypair, and your source code control system will hold the public key.
Back in the Django admin, set the Source Type to Git Repository over
SSH. Paste the git clone SSH URL into the URL field. Then open the newly
generated file repo_deploy_key and paste its contents into the SSH
Key field. Here’s what that looks like:
App Source for a private git repository¶
The other fields can be left blank. Path and Branch can be set
the same as with the Git Repository over HTTPS source type (see above).
Copy the public key in the newly generated file repo_deploy_key.pub
into the deploy keys section of your source code repository. Here is
what that looks like on GitHub:
Adding a deploy key to GitHub¶
Make the key read only by leaving “Allow write access” field unchecked
and click Add the key to save the key.
GitHub Repository using the GitHub API¶
This source type can be used to access private GitHub repositories using a GitHub username and password or a username and personal access token.
Set the Repository field to the organization name and repository
name, separated by a slash, as in the repository’s URL following
github.com/. In Other Parameters, paste a small YAML-formatted
document holding a GitHub username and password or username and personal
access token, formatted as follows:
auth:
user: 'myusername'
pw: 'mypassword'
The other fields can be left blank. Branch can be set the same as
with the Git Repository over HTTPS source type (see above).
Since this source type requires user credentials, it should be avoided for production deployments in favor of using Git Repository over SSH.
Controlling access to apps¶
Controlling which organizations in a Q deployment can access which apps is done via the App Sources table.
The “Available to all” field of App Source, which is on by default, gives all users of all organizations the ability to start an app provided by the App Source.
If the “Available to all” field is unchecked, then only users within white-listed organizations can start apps provided by the App Source. The white-list is a multi-select box on the App Source page.
Removing access to a App Source does not affect any apps that have already been started by a user.
App executable content¶
Apps can contain executable content (some of which is disabled by default):
JavaScript executed by the client browser contained within page HTML, via module template content.
JavaScript executed by the client browser served as a static asset and referenced by a
<script>tag.
Both sources of Javascript execute within the context of pages on the domain that the Q site itself runs on, which means the scripts have access to the page DOM, cookies, localStorage, etc. These scripts must only be enabled if they are trusted for these environments.
Javascript static assets (but not Javascript in module templates - this is a TODO) are therefore disabled by default. (Javascript static assets are disabled by serving them with an incorrect MIME type.)
To enable these scripts, the Trust assets flag must be true on the
App Source that provides the app. This flag must only be true if any
Apps provided by the App Source, including Apps already loaded into Q,
are trusted to have executable content that may have as much client or
server-side access as the Q instance does itself.
Modules, Questions, and Documents YAML Reference¶
A module and its questions are defined in YAML specification files. The schema for the specification files is as follows:
Module¶
Each file is a Module. A Module has the following required fields:
id: module_id
title: Your Title Here
The module_id must match the file name that the YAML is saved into,
without the path or file extension.
Several optional fields can be specified:
type: project
version: 1
instance-name: "Module for {{q1}}"
invitation-message: "Can you tell me about {{question.text}} and let me know when you are done?"
icon: app.png
The type field is set to project just when the Module is to be
offered to users when they start a new Project. (system-project is
used for project-like modules that are system controlled and not offered
to the user.)
The version field is used only to force changes in the specification
to be considered incompatible with any existing user answers (see
Updating Modules).
The instance-name is a template to generate a dynamic title for
in-progress and completed modules. The instance-name is rendered
like other Module documents but it is always specified in text
format (see Documents).
For modules that define the root of an application, icon specifies a
static asset (in the assets directory) to use as an application
icon.
In addition, a Module may have an introduction document (for
projects, the introduction appears at the top of the project page; for
other module types, it appears as an implicit initial interstitial) and
a list of one or more output documents. For example:
introduction:
format: markdown
template: |
Welcome to the module.
This module should take you two minutes.
output:
- title: Document 1
format: markdown
template: |
# Document for {{project}}
Hello! This is the output of the module. You answered {{q1}}.
- title: Document 2
glyphicon: dashboard
format: html
template: |
<h1># Document for {{project}}</h1>
<p>Hello! This is the output of the module. You answered {{q1}}.</p>
The format of documents are described in a later section.
Finally, a Module contains a list of one or more Questions:
questions:
- id: q1
title: Your Favorite Animal
prompt: What's your favorite animal?
type: text
- id: q2
title: What Kind of Animal Is It
prompt: How would you classify this animal?
type: choice
choices:
- key: pet
text: common pet
help: Is this animal a common pet?
- key: wild
text: wild animal
help: Is this animal an undomesticated wild animal?
The schema for questions is documented in a later section.
Additional fields for projects¶
Question Fields¶
The questions of projects are displayed in a layout of tabs and groups within each tab pane. Each question that shows up on a project page should specify its tab and group name (which are also the display strings):
questions:
- id: howto_ssp
title: "SSP 101: What's a System Security Plan"
type: module
module-id: howto_ssp
tab: How To
group: Start Here
icon: ssp.png
icon specifies a static asset (in the assets directory) to use
as an icon for the question. If the question’s type is module and it
is answered and the answer is a Task that has a top-level icon
field, then the answer’s icon is used instead.
Instead of tab and group, placement: action-buttons can be
used instead to show the question in an action bar above the tabs,
rather than in tabs.
Output Document Fields¶
Output documents of a project module that have an id field are used
in the following ways:
They are displayed in the Related Controls page for the project. Add a
titleattribute to set the heading text above the document’s content.They can be accessed from higher-level apps into which this app has been added. In a higher-level app, access the rendered HTML value of the output document as
{{question.output_documents.document_id}}.
When display: top is set on an output document, it is rendered above
the Your Answers section.
Test Answers¶
Projects can provide sets of exemplar answers for use in test scripts. e.g.:
tests:
test1: # <-- test suite ID
description: "Sample data."
answers:
q1:
answers: # <-- answers to sub-task's questions
q1: desktop
q2: My Secure Tool
Documents¶
Documents occur as introduction and output documents of Modules,
and a restricted form of documents also occurs in Question prompts (see
Questions below). A document appearing in the output documents list is
given as:
output:
- id: mydoc
title: Document 1
format: markdown
template: |
Hello!
The id and title fields are generally optional and are used for
output documents only. An id is required to make the document
downloadable. The fields also have special uses in projects (see above).
The format field is described below.
The document can also be stored in a separate file by replacing the document data in the module YAML file with a filename and placing the document properties and template in the named file, as in:
# module.yaml
output:
- mydoc.md
# mydoc.md
id: mydoc
title: Document 1
format: markdown
...
Hello!
When using a separate file, the document properties (id, title,
and format) are given in a YAML block at the top of the file. A line
containing just three dots signifies the end of the YAML block,
separating it from the document template. The document template follows.
Document Format¶
The introduction and output documents of Modules allow a format
to be specified. The document formats are:
markdown— The document is entered in CommonMark (quick guide) in the specification file, but it will be rendered into a richly formatted presentation on screen.html— The document is given in raw HTML, but it will be rendered on screen.text— The document is given in plain text, and it will display as preformatted (fixed-width) text on screen.json,yaml— Experimental.
Additional Markdown Notes¶
Documents specified in markdown format are rendered according to the
CommonMark 0.25 specification.
Note that for some things like tables, it is necessary to insert raw HTML right into the document, which is acceptable CommonMark. To create a table:
<table><thead><th>
Col 1
</th>
<th>
Col 2
</th>
</thead>
<tbody><tr><td>
Some [commonmark](http://www.google.com) within the cell.
</td>
<td>
More *content.*
</td></tr></tbody></table>
Some of the newlines are necessary to get CommonMark to go out of raw HTML mode and back into parsing CommonMark.
Document Templating¶
All document formats are evaluated as Jinja2 templates. That means within your document you can embed special tags that are replaced prior to the document being displayed to the user:
{{ question_id }}will be replaced with the user’s answer to the question whoseidisquestion_id. For choice-type questions, the value is replaced by the choicekey. Use{{ question_id.text }}to get display text. See the question types documentation below for details.{% if question_id == 'value' %}....{% endif %}is a conditional block. The contents inside the block (....) will be included in the output if the condition is true. In this example, the contents inside the block will be included in the output if the user’s answer toquestion_idisvalue.
Output documents and question prompts have access to the user’s answers to questions in question variables. (The introduction document does not have access to the user’s answers because questions have not yet been answered.)
The following information is also available within the output template
for each question as of version v0.8.6:
{{ question_id.not_yet_answered }}Question has not yet been answered.{{ question_id.answered }}Question has an answer either by user or was imputed, but not imputednullor answerednull.{{ question_id.imputed }}Question considered “answered” but no TaskAnswerHistory record exists in the database for question meaning a user didn’t provide the answer.{{ question_id.skipped }}Question has a null answer either because imputednullor the user skipped it.{{ question_id.skipped_by_user }}Question has anullanswer because used a skip button (e.g., question wasn’t imputednull).{{ question_id.skipped_reason }}Question’s indicated reason for skipping (e.g. “I don’t know” or “It doesn’t apply”){{ question_id.unsure }}If question was answered by a user, its unsure flag. (NOTE: Purpose of this flag was to allow users to indicate uncertainty in the answer. Due to usability issues however, this feature is currently hidden.){{ question_id.date_answered }}Question answered date.{{ question_id.reviewed_state }}Question reviewed value.
All documents also have access to the project title as {{project}}.
Project Documents¶
In addition to the output documents described above, a project
module may also have a snippet that defines how a project appears in
the project listing page:
snippet:
format: markdown
template: |
Project {{name}}
Module Assets¶
Modules often make use of assets outside of the YAML file.
Static Assets¶
Static assets such as images can be referenced in module content
(introductions, question prompts, and output documents). These assets
are exposed by the Q web server in its static path. Place static assets
in an assets subdirectory where the module is. When the asset is
referenced in a Markdown document template, its path will be rewritten
to be its public (virtual) path on the web server.
For example, to include an image in a module introduction add the image in the Markdown template:
module.yaml
-----------
...
format: markdown
template: |
...
Place the module and image files at the path:
module.yaml
assets/my_image.png
Private Assets¶
Private assets are other files that are stored with a module but are not exposed by the web server. The directory provides a place to store files for internal use during module development.
Place private assets in a private-assets subdirectory next to the
module YAML file.
Questions¶
Questions have the following required fields:
- id: q1
title: Your Favorite Animal
prompt: What's your favorite animal?
type: text
The question id is used to refer to this Question in other questions
and in the output documents.
The title is used to describe the Question in places where a
long-form prompt would not be appropriate.
The prompt is the text the user is prompted with when presented with
the question. The prompt is rendered like other Module documents but it
is always specified in markdown format (see Documents). The first
line (paragraph) of the prompt is shown in larger, bold type.
A question may have other optional fields that provide the user with other information, such as:
examples:
- example: |
First example.
- example: |
Second example.
reference_text: See NIST SP 800-171 page 102.
Like the prompt, each entry inside examples and the
reference_text are Markdown templates.
Removing a question, changing a question type, and other changes as noted below are incompatible changes (see Updating Modules).
Question Types¶
text¶
This type asks the user for a single line of free-form text. The text cannot be empty.
A placeholder can be specified which places ghosted “placeholder”
text inside the form field when the user has not yet entered anything. A
default value can be specified, instead, which fills in the field
with a value that the user can edit (or not) before submitting the
answer. The placeholder and default fields are rendered like other
Module documents — just like the prompt.
help text can be specified which provides an additional prompt
smaller and below the field input.
Example:
- id: q1
title: Your Favorite Animal
prompt: What's your favorite animal?
type: text
placeholder: enter a type of animal
help: Examples: dog, cat, turtle, lion
In document templates and impute conditions, the value of text
questions is simply the text the user entered.
password¶
This type asks the user for a password. It is the same as the text
question type, except that a password input field is used to mask the
input. help can be specified. placeholder and default are
not allowed.
email-address¶
This type asks the user for an email address. It is the same as the
text question type, except that the value entered must be a valid
email address. placeholder, default, and help can be
specified.
url¶
This type asks the user for a web address (a URL). It is the same as the
text question type, except that the value entered must be a valid
web address. placeholder, default, and help can be
specified. The web address is not checked for existence — only the form
(syntax) of the address is checked.
longtext¶
This type asks the user for free-form text using a large rich text input area that allows for multiple lines of text and some simple formatting. The text cannot be empty.
A default value can be specified, which fills in the field with a
value that the user can edit (or not) before submitting the answer. The
field is rendered like other Module documents — just like the
prompt. It is given in Markdown.
help text can be specified which provides an additional prompt
smaller and below the field input.
In document templates and impute conditions, the value of longtext
questions is the text the user entered, as a string, with rich formatted
represented in CommonMark. In document templates, the text is
automatically converted back to rich formatting.
date¶
This type asks the user for a date.
help text can be specified which provides an additional prompt
smaller and below the field input.
In document templates and impute conditions, the value of date
questions is a text string in YYYY-MM-DD format.
choice¶
This type asks the user to choose one of several options. The options are given as:
choices:
- key: pet
text: common pet
help: Is this animal a common pet?
- key: wild
text: wild animal
help: Is this animal an undomesticated wild animal?
The user must select exactly one choice.
The help text is optional. It is displayed smaller and below each
choice. (Unlike some other question types, there is no help field on
the question as a whole.)
In document templates and impute conditions, the value of choice
questions is the key of the choice selected by the user. Use
questionid.text to access the display text for the choice.
Removing a choice is an incompatible change (see Updating Modules).
yesno¶
This type is the same as choice but with built-in choices for yes
and no. It is the same as a choice question type with these choices:
choices:
- key: yes
text: Yes
- key: no
text: No
The user must choose either yes or no.
multiple-choice¶
The multiple-choice question type is similar to the choice
question type except that:
The user can select multiple choices.
In document templates and impute conditions, the value of
multiple-choicequestions is a list of thekeys of the choices selected by the user. When used bare, this renders as a comma-separated list of keys. One can use the`|lengthfilter <http://jinja.pocoo.org/docs/dev/templates/#length>`__ and{% for ... in ... %}... {% endfor %}loops to access the individual choices the user selected. Usequestionid.textto render a comma-separated list of the display text of the selected choices.minandmaxmay be specified. Ifminis specified, it must be greater than or equal to zero and requires that the user choose at least that many choices. Ifmaxis specified, it must be greater than or equal to one (and ifminis specified, it must be at leastmin) and requires that the user choose at most that number of choices.
Increasing the min or decreasing the max are incompatible
changes (see Updating Modules).
datagrid¶
The datagrid question type is similar to the multiple-choice
question type except that:
An array of fields define columns for tabular data (AKA “datagrid”).
Users can enter as many rows of data as desired.
Results are usually displayed as a table.
minandmaxmay be specified. Ifminis specified, it must be greater than or equal to zero and requires that the user add at least that many rows. Ifmaxis specified, it must be greater than or equal to one (and ifminis specified, it must be at leastmin) and prevents the user from adding more than that number rows.renderkey can be added and set toverticalto force the tabular data to render in a vertical format with the fields listed vertically on the side instead of horizontally. This is good for rendering information about a single, multi-field entry such as an address or Point of Contact.
Increasing the min or decreasing the max are incompatible
changes (see Updating Modules).
integer¶
This question type asks for a numeric, integer input.
If min and max are set, then the value is restricted to that
range. If min is omitted, then negative numbers are allowed!
As with the text question types, placeholder and help text can
also be specified.
In document templates and impute conditions, the value of integer
questions is the numeric value entered by the user.
real¶
This question type asks for a numeric input, allowing for real (floating-point) numbers.
If min and max are set, then the value is restricted to that
range. If min is omitted, then negative numbers are allowed!
As with the text question types, placeholder and help text can
also be specified and in document templates and impute conditions the
value of these questions is the numeric value entered by the user. .
file¶
This question type asks the user to upload a file.
help text can also be specified, as in the text question types.
By default, any type of file is permitted to be uploaded. If the
optional file-type field is set, the uploaded file is validated to
be of a particular type. Supported values for the file-type field
are:
image: Ensures the file is an image. The uploaded file is converted to PNG format internally.
If file-type is image, then some image transformation can be
run, e.g.:
- id: logo
title: Logo
prompt: Upload a logo.
type: file
file-type: image
image:
max-size:
width: 60
height: 60
If image->max-size is given, then the image will be resized prior to
being saved internally so that its width and height do not exceed the
given dimensions.
In document templates and impute conditions, the value of these
questions is a Python dict (JSON object) containing url (a download
URL) and size (in bytes) fields.
module, module-set¶
These question type prompt the user to select another completed module
as the answer to the question. The module-id field specifies the ID
of another module specification. The module question type allows for
a single other module to answer the question. The module-set
question type allows for zero or more other modules to answer the
question.
The module-id field specifies a module ID as it occurs in the id
field of another YAML file in the same application.
Here’s an example of the module question type:
- id: evidence
title: Evidence
type: module
module-id: evidence
prompt: |
Provide evidence of your properly configured firewall, if possible.
impute:
- condition: not(have_other_dmz == 'ad_hoc_dmz')
value: ~
Instead of using module-id, a protocol can be specified instead.
A protocol is a globally unique identifier that apps in the Compliance
Store use to indicate that their questions and output documents meet a
certain criteria (i.e. implement the protocol). When a user attempts to
answer a module or module-set question that uses protocol
instead of module-id, instead of starting a particular named module,
the user instead can start any app from the Compliance Store that
implements the protocol.
For example:
- id: evidence
title: Evidence
type: module
protocol: govready.com/apps/compliance/2017/nist-sp-800-171-r1-ssp
prompt: |
Provide evidence of your properly configured firewall, if possible.
When a user answers this question, they will be redirected to the
Compliance Store but will be offered only apps that implement the
protocol govready.com/apps/compliance/2017/nist-sp-800-171-r1-ssp.
An app implements a protocol by having a protocol: field at the top
level of the app’s YAML specification file with the same value. For
instance, the following app would be offered in the Compliance Store for
this example question:
id: app
title: My App
type: project
protocol: govready.com/apps/compliance/2017/nist-sp-800-171-r1-ssp
Both protocol fields can be either a single string or a list of strings.
When the question protocol value is a list, then only apps which
implement all of the listed protocols will be offered.
Changing the module-id or protocol is considered an incompatible
change (see Updating Modules), and if the referenced Module’s
specification is changed on disk in an incompatible way with existing
user answers, the Module in which the question occurs is also considered
to have an incompatible change. Thus an incompatible change in a module
triggers an incompatible change in any other Module that refers to it
(and so on recursively).
In document templates and impute conditions, the value of module
questions is a dictionary of the answers to that module. For example, if
q5 is the ID of a question whose type is module, then
{{q5.q1}} will provide the answer to q1 within the module the
user selected that answers q5.
interstitial¶
An interstitial question is not really a question at all! The
prompt contains template content, as with other questions, but it is
typically longer content with deeper explanatory text. The user is not
asked to enter any information.
In document templates and impute conditions, the value of
interstitial questions is always a null value.
raw¶
This type is meant for questions that are always imputed (i.e. that are never presented to the user) and where the answer value can be any JSON-serializable Python data structure, as given by the impute value (see Imputing Answers below).
This question type should be avoided if one of the other question types
specifies a more narrow data type. For instance, if the imputed value is
always a string, the text or longtext question types should be
used instead.
Imputing Answers¶
The answer to one question may provide the answer to another. In such cases, the latter question is said to have an imputed value and the user is not asked to answer the question. To impute a value, specify on the question whose value is being imputed:
impute:
- condition: q1 == 'no'
value: don't know
This example says that if the answer to q1 is no, then the
answer to this question is don't know.
The condition is a Jinja2
expression.
Any question can be referred to in the expression (by its id).
Questions are tested on their internal values. For choice and
multiple-choice questions, their values are their keys, not
their label text, and multiple-choice questions are lists of keys.
If condition is omitted, the imputed value is always taken (i.e. the
condition is implicitly met).
The value provided must be a valid value for the question type it is
a part of. For choice questions, the value must be a choice key,
not the label text. For multiple-choice questions, the value must be
a list of keys.
Multiple condition/value blocks can be provided. They are evaluated in order, with the first matching condition taking precedence.
impute:
- condition: q1 == 'no'
value: I don't know.
- condition: q1 == 'yes'
value: I do know.
The value field can be evaluated as a Jinja2
expression,
just like the condition, if value-mode is set to expression.
This can be used to pull forward the answers of previous questions:
impute:
- condition: q1 == 'same-as-q0'
value: q0
value-mode: expression
value-mode can also be template to evaluate the value as a
Jinja2 template, which will yield a text value.
In both conditions and expression-type values, as well as in
documents, the variables you can use are:
ids of questions in the modulequestion_id.subquestion_idto access questions within the tasks that are assigned as answers tomodule-type questionsproject, which gives the project nameproject.question_id,project.question_id.subquestion_id, etc. to access questions within the projectorganization, which gives the organization name
We also have a function to retrieve the URL of a module’s static assets, e.g.:
<script src="{{static_asset_path_for('myscript.js')}}"></script>
Question Order¶
The order in which Questions are asked is determined through an algorithm. The algorithm determines which questions need to be asked before other questions and which need to be asked in order to generate the output documents.
The only Questions that are asked of the user are those that are mentioned in any of the output templates or other Questions that required to be asked before those mentioned Questions can be answered.
If a Question mentions another question in its prompt text or impute conditions, the other question must be answered first. A Question can also list other Questions that should be answered first as:
ask-first:
- q1
- q2
Updating Modules¶
When a Module file specification is changed, the change is considered “compatible” or “incompatible” with existing user answers.
Many changes are “compatible”: Changing the introduction or output documents, question prompts, and adding new questions and choices are all compatible changes. These changes can be made “live” on any existing user answers.
Other changes are “incompatible”: Removing a choice is an incompatible change because a user may have already chosen it. Removing a question is incompatible because it would result in a loss of user data.
When there is an incompatible change in a Module specification, a new iteration of the Module will be stored in the program database but existing user answers will continue to be tied to the previous iteration of the Module specification.
Automation API¶
GovReady Q Compliance Apps can be updated with information gathered from live systems via the GovReady Q API. The benefit of this capability is that the documentation produced by GovReady Compliance Apps, such as System Security Plans (SSPs), can be assembled and updated with actual system data in an automated way.
Overview of the GovReady Q API¶
The GovReady Q API provides read and write access to the information stored in GovReady Q’s question-and-answer data model. It is a RESTful API using HTTP GET and POST requests, API keys that are issued per user, and JSON for request and response data.
Each GovReady Compliance App provides a separate API, and each app’s API is composed of fields for the same information the app would ask an end-user using the Q website in a web browser. The app’s definition of questions to ask the end user (see Modules, Questions, and Documents) also define the data model of the API.
As an example, the screenshot below shows a demonstration of a macOS File Server compliance app. The app asks questions about the hostname of the server and the use of security groups.
macOS File Server Compliance App¶
The answers to these questions can be both read from Q and written to Q using a JSON data structure:
{
"schema": "GovReady Q Project API 1.0",
"project": {
"file_server": {
"hostname": "twiggie",
"login_message": "/etc/issue:\n\n```Ubuntu 16.04.3 LTS \\n \\l```",
"login_message.html": "<p>/etc/issue:</p>\n<p><code>Ubuntu 16.04.3 LTS
\\n \\l</code></p>",
"using_security_groups": "yes",
"using_security_groups.text": "Yes"
"security_groups_description": "There are 17 UNIX groups: `adm` (2 users:
`syslog`, `user`), `audio` (2 users: `speech-dispatcher`, `pulse`), ...",
"security_groups_description.html": "<p>There are 17 UNIX groups: <code>
adm</code> (2 users: <code>syslog</code>, <code>user</code>), ..."
}
}
}
The second question, shown below, is a yes-no question. In the web
browser this question appears as a radio select question with Yes and No
choices. In the JSON data structure, shown above, it is encoded as the
JSON strings "yes" or "no".
macOS File Server Compliance App Question¶
Each compliance app that has been started in GovReady Q and added to a
project folder provides an API Docs page with samples and data
schema documentation:
API Docs Page¶
Using the Compliance API¶
To use a compliance app API, an app must already be started in GovReady Q by adding it to a project folder and its modules must be started (but need not be completed) for them to be accessible from the API.
API keys¶
Every call to the API requires an API key. Each user has three API keys listed on their API Keys page, which can be found in the user drop-down menu on Q: a read-only API key, a read-write API key, and a write-only API key:
The read-only API key gives external tools the ability to see all data values that the associated user can see on Q, but the API key cannot be used to change any data values.
The read-write API key gives external tools the ability to see and make changes to anything the associated user can see and make changes to on Q.
The write-only API key gives external tools the ability to make changes to anything the associated user can make changes to on Q but does not include the ability to see any data values stored in Q. The write-only key is useful in situations where the external tool needs to be able to upload data but does not need to read existing data values.
Getting data from the app using the GET API¶
Project data can be read from the API using an HTTP GET request to a URL of the following pattern:
{site base URL}/api/v1/projects/{project id}/answers
The complete URL can be found on the API Docs page for a compliance
app that has been started and added to a project folder.
An API key must be passed in the HTTP Authorization header. You can
get your API key from the API Keys page, which is found in the site
header menu in the user drop-down.
The API response is a JSON data structure similar to the example above.
The schema of the response object is documented on the app’s
API Docs page. Further information can also be found below.
If you are using an operating system with a command line and the
curl tool, you can try out the API by running:
curl --header "Authorization: {your-api-key}" \
{site base URL}/api/v1/projects/{project id}/answers
Updating data using a POST request with form data¶
There are two types of POST requests that can be used to update app data. In the first type, described in this section, data values are provided as key-value pairs using the regular web browser form submission method. (In the second form, described below, answers are provided using a JSON data structure that is formatted the same as the JSON data structure returned by a GET request, which may be more appropriate when submitting non-textual and non-binary content.)
In each of the key-value pairs submitted in the POST request, the key
is a dotted-path question ID. The key always begins with project.
and is followed by the property names on the path to the question being
updated, according to the JSON data structure, with property names
separated by the . character.
The value of each key-value pair is an answer submitted either as plain text or, for file-type questions, as a binary file. If submitted as plain text and the question expects non-text data, such as a number, the value will be converted. When uploading a binary file, the multipart/form-data content type must be used for the POST request.
As with the GET API, an API key must be passed in the HTTP Authorization header. An API key with write permission must be used. You can get an API key from the API keys page on your Q site.
If you are using an operating system with a command line and the
curl tool, you can try out the API by running:
curl \
--header "Authorization: <i>your-api-key</i>" \
-F project.question.subquestion1=datavalue \
-F project.question.subquestion2=datavalue \
{site base URL}/api/v1/projects/{project id}/answers
For a file upload, use -F @filename.ext. curl’s -d option
can be used in place of -F if none of the fields are file uploads.
Updating data using a POST request with JSON¶
Use a POST request instead a GET request to the same URL to update data stored in the app. Data values to save in the app are included in the request body as JSON in the same format as returned by the GET request.
The POST request body always includes:
{
"schema": "GovReady Q Project API 1.0",
"project": {
...
}
}
Answer data is placed inside the project field.
As with the GET API, an API key must be passed in the HTTP
Authorization header. An API key with write permission must be used.
You can get your API key from the API Keys page, which is found in the
site header menu in the user drop-down.
If you are using an operating system with a command line and the
curl tool, you can try out the API by placing the JSON request data
in a file named data.json and then running:
curl --header "Authorization: {your-api-key}" \
-XPOST --data @data.json --header "Content-Type: application/json" \
{site base URL}/api/v1/projects/{project id}/answers
API Data Schema¶
Each compliance app documents its data schema on its API Docs page, which can be found inside the compliance app after it has been started and added to a project folder.
Each question defined by the app — which it would ask an end-user when in a web browser — is exposed as a field in the JSON data structure. The field types are:
Text, password, email-address, and URL fields: Encoded as a JSON string. Email-address fields must contain valid email addresses. URL fields must contain valid URLs.
Long text fields, which hold multi-paragraph text: Encoded as a JSON string with formatting expressed in CommonMark (i.e. Markdown).
Date fields: Encoded as a JSON string in YYYY-MM-DD format.
Single-choice and yes-no fields: Encoded as a JSON string holding a programmatic identifier for the selected choice. Yes-no fields use the identifiers
yesandno.Multiple-choice fields: Encoded as a JSON array of strings, where each string is a programmatic identifier for a selected choice.
Datagrid fields: Encoded as a JSON array of dictionaries, where each dictionary describes the row of data fields.
Integer and real number fields: Encoded as a JSON number. Integer fields must contain integer values.
File fields: Encoded as a JSON object containing the properties
url(a link to download the file content),type(the MIME type), andsize(the size of the file in bytes).“Module” questions create recursive structures and are encoded as JSON objects. “Module-set” questions are encoded as JSON arrays of JSON objects.
All fields can also hold null, which indicates the question has been
explicitly “skipped.” If a question is unanswered, it does not appear in
the API.
Single-choice, multiple-choice, and yes-no fields also appear in
human-readable form as a second read-only field that uses a .text
suffix in the field’s name.
Long text fields have an HTML display form, in which the CommonMark is
pre-rendered, in a parallel field with a .html suffix in the field’s
name. These fields cannot be used in the POST API.
Datagrid fields appear in human-readable form as tabular data.
More information about Q’s data types can be found in Modules, Questions, and Documents.
Data Design Guide¶
The documents in this section describe GovReady-Q’s database design.
Users, Organizations, Projects, Folders, and Invitations¶
The diagram below provides a summary representation of GovReady-Q’s
Django siteapp data model, which handles users, organizations,
projects and folders, and invitations.
Siteapp data model (not all tables represented)¶
The siteapp data model represents users who have membership in different projects. Users must be invited to projects. Projects belong to organizations.
Access control is based on organization and projects. Information cannot be shared across organizations and only limited information can be shared across projects within an organization.
Compliance Apps, Modules, Questions, Tasks, and Answers¶
GovReady-Q is a governance, risk, and compliance platform for creating automated compliance processes ranging from gathering information from persons and computers to generating compliance artifacts.
Information gathering is at the heart of GovReady-Q’s guidedmodules
data model, which handles compliance apps, modules, questions, tasks,
and answers. Eight database tables make up the guidedmodules data
model. The complete data model spans three categories:
Compliance apps, which are reusable packages of questions, business logic, and document templates, are defined by the database tables
AppSource,AppVersion,Module,ModuleQuestion, andModuleAsset. Many compliance apps, and different versions of the same compliance app, can be in use simultaneously.Information submitted by end-users to answer compliance app questions, as well as information submitted through the GovReady-Q API, is stored in the database tables
Task,TaskAnswer, andTaskAnswerHistory.Imputed answers and documents generated by the business logic stored in compliance apps, which are computed on-the-fly.
The tables are described is additional detail below and their relationships are summarized in the following diagram:
Guidedmodules data model (not all tables represented)¶
Compliance Apps (Questions, Business Logic, and Templates)¶
The smallest unit of a compliance app is a Question. Questions come
in different types, such as text, number, and date (full
list). Questions are grouped into questionnaires called
Modules. And Modules are grouped into AppVersion, which are the
versions of a compliance app that are loaded into the GovReady-Q
database. AppVersions are loaded from AppSources, which define how to
load compliance apps from remote sources such as GitHub or an
on-premesis enterprise source control system.
AppVersion, Modules, and Questions define the structure of a
compliance app but do not store any user-submitted content. Separating
structure from content is a common pattern in application design and is
motivated by several GovReady-Q goals:
Compliance apps are reusable and can be easily loaded into different installs of GovReady-Q at different organizations.
Anyone can author compliance apps and they can be kept private or shared publicly.
One type of Question allows the user to choose a complete set of answers to another Module, which allows question answers (i.e. user data) to be accessed from the business logic and templates of not only the Modules the Questions are defined in but also from other Modules and even other compliance apps.
Compliance apps are versioned and apps that have been started by users can be updated in non-destructive ways, preserving answered questionnaires over the course of years,
GovReady-Q administrators will configure GovReady-Q with an
AppSource record for each source of compliance apps that will be
made available to GovReady-Q users. Each AppSource has a “slug,” which
is its unique name on the GovReady-Q installation. An AppSource also
defines a remote location — such as a GitHub repository, local
directory, or on-premise git server — to query for compliance apps,
which will be listed in the app catalog. A typical GovReady-Q
installation might have one AppSource providing compliance apps
published by GovReady PBC and a second AppSource for providing
compliance apps defined by the organization. Each AppSource defines the
remote location as well as local permissions such as which compliance
apps in the store to make available to GovReady-Q users and
Organizations. See App Sources for more
information.
Each time a compliance app is started by a user, an AppSource is
queried for the latest version of the selected compliance app and an
AppVersion is created that holds the complete set of questions,
business logic, and templates defined in that version of the compliance
app. Therefore there will be a separate AppVersion in the database for
every version of every compliance app being used by GovReady-Q users.
Each AppVersion links back to the AppSource it was created from (the
“source” field) and holds the name it was given (the “appname” field).
Each AppVersion brings along with it the Modules and Questions
defined in that version of the compliance app. In other words, Modules
and Questions are specific to a AppVersion. Therefore if two versions of
a compliance app are present in the database, a question that exists in
both versions of the app is represented as two Question records — one
for each version of the app. Similarly, there will be (at least) two
Modules, one for each version of the app.
All compliance apps have at least two Modules. The first module,
whose “module_name” identifier is always app, defines the layout of
the starting page of a compliance app, which can list one or more
Modules to complete:
compliance app sample 1¶
or a collection of modules and slots to start other compliance apps:
compliance app sample 2¶
In both cases, the “cards” that represent modules to answer or slots for
compliance apps to start are defined by Questions in the “app”
Module. When a user starts a module or selects a compliance app,
that is recorded in the database as answering the respective Question
in the “app” Module (more on that below).
Besides listing questions, Modules also define zero or more output
documents. Each output document is generated by combining a template
stored in the Module with user answers.
Similar to Modules, ModuleQuestions have a “key” field that uniquely
identifies them within the Module they are defined in. ModuleQuestions
store the question type (text, date, etc.), the prompt shown to users,
impute conditions (see below), and other metadata.
ModuleAssets store a compliance app’s static assets used by the
app’s templates. These assets often appear as images or other embedded
media in output documents generated by the compliance app.
User Answers (Tasks and Answers)¶
When a user is completing the questions in a compliance app, their
answers are stored in a separate set of database tables distinct from
the tables used to store compliance app questions, business logic, and
templates. The tables that hold answers are Task, TaskAnwser,
and TaskAnswerHistory.
Task and TaskAnswer are parallel tables to Module and
ModuleQuestion and are related to where user answers are stored. A
Task is the instantiation of a Module that a GovReady-Q user or
set of users are completing. A TaskAnswer is the instantiation of a
ModuleQuestion that a GovReady-Q user has answered. All of the Tasks
instantiated together for the same compliance app are related through
the “project” field.
TaskAnswerHistory stores the complete history of user answers
related to a TaskAnswer, i.e. to an instantiated question. The current
answer to a question and its associated metadata are stored in the most
recent TaskAnswerHistory record for a particular TaskAnswer (the one
with the highest “id” value — “id”s are assigned to answers in strictly
increasing order). Therefore only the most recent TaskAnswerHistory
record for a TaskAnswer holds a current answer, and earlier
TaskAnswerHistory records are for audit logging and tracking changes.
TaskAnswerHistory records have a “stored_value” field which holds the user’s answer encoded in JSON, other metadata such as “answered_by” for which user provided the answer, “skipped_reason” and “unsure” which are flags set if the user skipped the question or wants to return to it later, and “reviewed” which holds workflow review state (e.g. if a reviewer marks the answer as approved).
This data model supports GovReady-Q design goals, such as:
Compliance app modules and questions can be assigned to different users to answer.
The answer to questions may change while a complete history of answers are preserved in an immutable record, including preserving past answer metadata such as who answered the question and whether the answer was approved by a reviewer.
Answers are strongly typed: text, numbers, dates, choices, and so on are encoded in a JSON representation that preserves their data type.
All questions can be skipped by storing
nullin “stored_value.”
Imputed Answers and Output Documents¶
Compliance apps hold business logic and templates that are used to “impute” answers to questions and generate output documents, respectively. These computational outputs are not stored in the database. Instead, they are computed on-the-fly by GovReady-Q as they are needed, and the results of the computations are cached so long as they remain valid.
Imputed Answers¶
Imputation uses business logic rules to infer the answer to questions based on previous answers to questions. Imputation is used for a variety of purposes, such as:
Hiding questions that are not applicable based on the answers to previous questions, by imputing
nullas the answer to the question.Pre-answering questions when the answer is known based on the answers to previous questions.
Running business logic computations, such as computing a grade or gap analysis, and storing the result of the computation as the answer to the imputed question.
Questions whose answers are imputed are not asked of the user — the user may never see these questions at all. Some questions are designed to always be imputed to support the execution of business logic rules.
The results of imputation are not stored in the database because they
are computed on-the-fly to ensure that the GovReady-Q always runs the
business logic rules on the most recent, current set of answers to the
questions. As a result, there may be no TaskAnswer or
TaskAnswerHistory records for questions that have been imputed.
In certain circumstances, a question’s answer may be imputed after a user already provided an answer to the question. In such cases, the user’s answer remains in the database and appears in the database as the current answer to the question. However, when visiting GovReady-Q, imputed answers supersede user answers and only the imputed value will be used.
Output Documents¶
Compliance apps produce output documents. Each output document is generated by combining a template with the answers to questions (both user-inputted and imputed). Templates are typically written using Markdown syntax and are displayed in GovReady-Q as HTML documents, but they typically can also be downloaded in other formats such as a Microsoft Word document or PDF.
As with imputation, output documents are generated on-the-fly when they are viewed by GovReady-Q users. The generated documents are not stored in the database because they are computed on-the-fly to ensure that the GovReady-Q always runs the template on the most recent, current set of answers to the Module’s questions. As a result, there is no database table for output documents.
Database Query Examples¶
Example: Find all approved answers to a particular question across users and tasks¶
Scenario: Unix File Server App contains a text-type question named “Hostname”. Many users have finished answering all of the questions in the app. However, our reviewers have only approved some of the answers so far. I want to write an SQL query to return all approved answers to the “Hostname” question.
In this section, we will build up an SQL query to extract the data identified in the scenario. The query will be built progressively over the next several sections to explain the rationale behind the GovReady-Q data model. Some of GovReady-Q’s design choices — including separating the definitions of compliance apps from user-submitted data, as well as recording an immutable history of user answers — are reflected in the SQL queries below. The complete SQL query is shown at the end.
You may prefer to use the GovReady-Q API instead of writing a low-level database query, but this example is illustrative for understanding GovReady-Q’s data model no matter which method you use to query the data.
Find the AppVersions¶
First locate the AppSource “slug” and AppVersion “appname” that
identifies a compliance app in GovReady-Q’s database. Find the app in
the compliance apps catalog and click its Info button:
compliance app¶
The slug and the appname of the compliance app can be found in the URL:
http://mygovreadyq/store/myapps/unix_file_server
In this case the slug is “myapps” and the appname is “unix_file_server”. These two fields identify the compliance app across its versions.
Construct an SQL query to return the numeric IDs of the AppVersions in the database for this compliance app. Each AppVersion may be a different version of the compliance app or a different instance of the same app in use by different users.
SELECT guidedmodules_appversion.id
FROM guidedmodules_appversion
LEFT JOIN guidedmodules_appsource
ON guidedmodules_appsource.id = guidedmodules_appversion.source_id
WHERE guidedmodules_appsource.slug = "myapps"
AND guidedmodules_appversion.appname = "unix_file_server";
This query will be adapted in the next section to find the hostname question.
Find the ModuleQuestions¶
Consult the compliance app source code YAML files to determine the
“module_name” of the Module and “key” of the ModuleQuestion —
which are in the “id” fields in the YAML file.
Module YAML¶
The Module containing the hostname question has “file_server” as its
module_name, and the ModuleQuestion’s key is “hostname.”
Construct a preliminary SQL query to find all of the ModuleQuestion
records for this question:
SELECT guidedmodules_modulequestion.id
FROM guidedmodules_modulequestion
LEFT JOIN guidedmodules_module
ON guidedmodules_module.id = guidedmodules_modulequestion.module_id
WHERE guidedmodules_module.module_name = "file_server"
AND guidedmodules_modulequestion.key = "hostname";
This query might be too broad — it does not restrict the questions to those defined in the Unix File Server compliance app. There might be other compliance apps that use the same module_name and question key. Combine the first two queries to ensure only questions in the Unix File Server app are returned using a LEFT JOIN to bridge the tables:
SELECT guidedmodules_modulequestion.id
FROM guidedmodules_modulequestion
LEFT JOIN guidedmodules_appsource
ON guidedmodules_appsource.id = guidedmodules_appversion.source_id
LEFT JOIN guidedmodules_appversion
ON guidedmodules_appversion.id = guidedmodules_module.app_id
LEFT JOIN guidedmodules_module
ON guidedmodules_module.id = guidedmodules_modulequestion.module_id
WHERE guidedmodules_appsource.slug = "myapps"
AND guidedmodules_appversion.appname = "unix_file_server"
AND guidedmodules_module.module_name = "file_server"
AND guidedmodules_modulequestion.key = "hostname";
We’ll call this query MODULE_QUESTIONS — we’ll use it as a sub-query
in the next step.
Find the history of answers¶
GovReady-Q has been designed so that separate tables contain the
definition of the question and the user-submitted answers to the
question. Each answer is connected to a ModuleQuestion through a
TaskAnswer. Locate the TaskAnswers for the questions:
SELECT guidedmodules_taskanswer.id
FROM guidedmodules_taskanswer
WHERE guidedmodules_taskanswer.question_id IN (MODULE_QUESTIONS);
Replace MODULE_QUESTIONS with the preceding SQL query, inserting it
as a sub-query.
The TaskAnswer table does not hold user answers, however. Answers are
stored in the TaskAnswerHistory table where the complete history of
answers to questions are stored. We’ll now adapt the query to fetch the
history of answers to this question, including some metadata about the
answers, by using a LEFT JOIN to bridge the TaskAnswerHistory table and
the TaskAnswer table:
SELECT guidedmodules_taskanswer.id, answer.stored_value, answer.created, siteapp_user.username
FROM guidedmodules_taskanswerhistory AS answer
LEFT JOIN guidedmodules_taskanswer
ON guidedmodules_taskanswer.id = answer.taskanswer_id
LEFT JOIN siteapp_user
ON siteapp_user.id = answer.answered_by_id
WHERE guidedmodules_taskanswer.question_id IN (MODULE_QUESTIONS);
Here is an example result:
Task Answer |
Stored Value |
Created |
Username |
|---|---|---|---|
10 |
“server1.company.com” |
2018-05-19 20:33 |
user1 |
10 |
“server2.company.com” |
2018-05-20 10:15 |
user1 |
10 |
null |
2018-05-20 10:35 |
user1 |
11 |
“server2.company.com” |
2018-05-19 16:20 |
user2 |
This is the complete history of answers for the “hostname” question in
two separate Tasks, i.e. two instantiations of the compliance app
started by different users. The two instantiations of the question are
identified by their TaskAnswer “id”s, 10 and 11.
The history for TaskAnswer 10 has three rows. Two rows – the first two —
reflect old answers to questions. This indicates the user returned to
the question twice. On the first occasion, the user replaced the
original answer with "server2.company.com". On the second revisit,
the user replaced the original answer with null, clearing the answer
because the user decided they didn’t know the answer or the question
didn’t apply to them.
The second TaskAnswer was answered once.
We’ll adapt this query in the next step to fetch just the current (most recent) answer in each Task.
Find the current answer to each question¶
The current answer for each question is stored in the
TaskAnswerHistory record with the highest “id” for each TaskAnswer.
The IDs in the TaskAnswerHistory table are assigned strictly
sequentially. To determine which TaskAnswerHistory record holds the
current answer, use GROUP BY and max to fetch one
TaskAnswerHistory for each TaskAnswer:
SELECT max(answer.id)
FROM guidedmodules_taskanswerhistory AS answer
LEFT JOIN guidedmodules_taskanswer
ON guidedmodules_taskanswer.id = answer.taskanswer_id
LEFT JOIN siteapp_user
ON siteapp_user.id = answer.answered_by_id
WHERE guidedmodules_taskanswer.question_id IN (MODULE_QUESTIONS)
GROUP BY guidedmodules_taskanswer.id;
id |
|---|
103 |
104 |
This result holds the current answers to the hostname question. We’ll
call this query CURRENT_ANSWERS — we’ll use it as a sub-query in the
next query.
To fetch the answers and metadata but for the current answers, we’ll
query the TaskAnswerHistory table using the CURRENT_ANSWERS
query as a sub-query to identify just the rows that are current answers
to questions:
SELECT taskanswer_id, stored_value, created, username, reviewed
FROM guidedmodules_taskanswerhistory
LEFT JOIN siteapp_user
ON siteapp_user.id = answered_by_id
WHERE guidedmodules_taskanswerhistory.id IN (CURRENT_ANSWERS);
Here is an example result:
Task Answer |
Stored Value |
Created |
Username |
Reviewed |
|---|---|---|---|---|
10 |
null |
2018-05-20 10:35 |
user1 |
0 |
11 |
“server2.company.com” |
2018-05-19 16:20 |
user2 |
2 |
This result holds the current answers to the Unix File Server hostname
question across all instances of the compliance app in the GovReady-Q
installation. Notice that the rows in the previous table that
represented replaced answers to the first TaskAnswer are omitted from
the results in this query and only the current answer for each Task
is included.
The “stored_value” column holds the user’s answer encoded in JSON. In
JSON, text (strings) are enclosed in double quotes. Therefore we know
that the second answer is text. In JSON, null (without double quotes
around it) represents an empty value — in GovReady-Q, that means the
user skipped the question choosing I Don’t Know, It Doesn’t Apply,
or I’ll Come Back.
We’ll modify this query in the next section to filter on the reviewed status of each answer.
Filter on approved answers¶
The “reviewed” field of TaskAnswerHistory stores GovReady-Q’s simple
workflow status of the answer. The values are 0 (not reviewed),
1 (reviewed), and 2 (approved). To select just approved answers,
add a WHERE clause to the previous SQL query:
SELECT taskanswer_id, stored_value, created, username, reviewed
FROM guidedmodules_taskanswerhistory
LEFT JOIN siteapp_user
ON siteapp_user.id = answered_by_id
WHERE guidedmodules_taskanswerhistory.id IN (CURRENT_ANSWERS)
AND reviewed = 2;
The query extracts the answers in a structure similar to the following table:
Task Answer |
Stored Value |
Created |
Username |
Reviewed |
|---|---|---|---|---|
11 |
“server2.company.com” |
2018-05-19 16:20 |
user2 |
2 |
This is the complete query to extract the approved answers to the
hostname question in the Unix File Server compliance app. The query has
been simplified by replacing a sub-query with CURRENT_ANSWERS, which
itself has a sub-query that has been replaced by MODULE_QUESTIONS.
Both sub-queries can be found above.
Discussions¶
The diagram below provides a summary representation of GovReady-Q’s
Django discussion data model that handles discussions, comments, and
invitations.
Discussion data model (not all tables represented)¶
A single discussion can be instantiated and associated to any task (task ~= “question”). A discussion can have multiple comments. Comments can have multiple attachments.
Generating Detailed Data Models¶
Below are instructions to use django-extensions to generate detailed
data models.
# Install django-extensions
# http://django-extensions.readthedocs.io/en/latest/installation_instructions.html
apt install graphviz-dev
pip3 install django-extensions pygraphviz
# Add django-extensions INSTALLED_APPS in siteapp > settings.py
# INSTALLED_APPS = (
# ...
# 'django_extensions',
# )
# examples:
python3 manage.py graph_models -a -g -o my_project_visualized.png
python3 manage.py graph_models -a -o my_project.png
python3 manage.py graph_models -a > my_project.dot
# for a single django app:
python3 manage.py graph_models app1 -o my_project_app1.png
Testing GovReady¶
GovReady includes utilities for automated testing of the GovReady Code, User Interface, and known vulnerabilities in required libraries.
Contents
Automated testing¶
GovReady-Q’s unit tests and integration tests are currently combined. Our integration tests uses Selenium to simulate user interactions with the interface.
To run the integration tests, you’ll also need to install chromedriver:
sudo apt-get install chromium-chromedriver (on Ubuntu)
brew cask install chromedriver (on Mac)
Navigate within your terminal to GovReady-Q top level directory.
Then run the test suite with:
./manage.py test
_NOTE: Depending on your Python3 configuration, you may need to run:_
To selectively run tests from individual modules:
# test rendering of guided modules
./manage.py test guidedmodules
# test general siteapp logic
./manage.py test siteapp
# test discussion functionality
./manage.py test discussion
Or to selectively run tests from individual classes or methods:
# run tests from individual test class
./manage.py test siteapp.tests.GeneralTests
# run tests from individual test method
./manage.py test siteapp.tests.GeneralTests.test_login
To produce a code coverage report, run the tests with coverage:
Code scanning and analysis¶
GovReady-Q is a Python web application written on top of the Django framework and uses a variety of industry standard Javascript libraries. See [Software Requirements](requirements.html#software-requirements) for high level view and the requirement*.txt files for detailed view.
GovReady-Q’s Python application code is found in the *.py files in the following directories and their subdirectories:
discussion/
guidedmodules/
siteapp/
The small manage.py script in the root directory is part of the Django framework. We use bash utilities scripts (*.sh) to automate installation and maintenance tasks of the code base. Python scripts in .circleci directory are used within our Continuous Implementation pipeline.
Simple Static Code Analysis¶
To run a static code analysis with our typical settings:
We use -s on the command-line and nosec in limited places in the source code to disable some checks that are determined after review to be false positives.
Detailed Static and Dynamic Code Analysis¶
We periodically scan GovReady-Q’s code base with more traditional/powerful tools and remediate critical and high vulnerabilities.
To scan GovReady-Q’s codebase, you will need to configure your tools to scan Python code. You are looking for the *.py files across the code base.
To scan or do other penetration tests on the code base, we recommend [deploying GovReady-Q with Docker](deploy_docker.html).
Supply chain and dependency management¶
Our requirements.txt file is designed to work with pip install --require-hashes, which ensures that every installed dependency matches a hash stored in this repository. The option requires that every dependency (including dependencies of dependencies) be listed, pinned to a version number, and paired with a hash. We therefore don’t manually edit requirements.txt. Instead, we place our immediate dependencies in requirements.in and run requirements_txt_updater.sh (which calls pip-tools’s pip-compile command) to update the requirements.txt file for production.
Continuous integration is set up with CircleCI at https://circleci.com/gh/GovReady/govready-q and performs unit tests, integration tests, and security checks on our dependencies.
1. CI runs requirements_txt_checker.sh which ensures requirements.txt is in sync with requirements.in. This script is set up to run against any similar files as well, such as MySQL-specific requirements_mysql.* files.
1. CI checks that there are no known vulnerabilities in the dependencies using [pyup.io](https://pyup.io/).
1. CI checks that all packages are up to date with upstream sources (unless the package and its latest upstream version are listed in requirements_txt_checker_ignoreupdates.txt).
Populating sample data for testing and verification¶
In some cases, you may wish to perform manual testing on an instance of GovReady-Q which has been populated with data. Several Django commands have been added to facilitate this, in the testmocking module. Generated data is intended to be structurally similar to what might be found in a real GovReady-Q instance, but the actual content of the data will often appear machine-generated.
If you wish to get up and running quickly, the following command is recommended:
python3 manage.py add_data --non-interactive
The add_data command will fill in a recommended set of sample data.
Version 0.9.x¶
What’s New in 0.9.x¶
Release 0.9.x (coming Autumn 2019) is a minor release improving the user experience and performance.
Faster loading and launching of Assessments/questionnaires
Simplified install with no subdomains to worry about
Replaces subdomain multi-tenancy with simplified “Portfolios” model
Improved authoring screens
Helpful new start page
Adds and updates portfolio level permissions
Release 0.9.x removes multi-tenancy and serves all pages from the same domain. In the earlier multi-tenancy versions, requests to GovReady-Q came in on subdomains: the subdomain determined which organization to associate with the request, and individuals had to re-login across subdomains. Little value was being realized by actual users for the subdomain-based multi-tenancy. Removing subdomain-based multi-tenancy reduces technical debt simplifies deployment.
This release’s compliance apps catalog now reads from the database rather than constantly rescanning remote repositories and the file system. The app catalog cache is removed since the page loads much faster. Release 0.9.x begins to replace the “compliance app” terminology with the plain language “projects” and “assessment” terminology in end user pages.
This release also introduces a “Portfolios” feature to organize and manage related projects.
For a complete list of changes see the 0.9.x.dev branch CHANGELOG.
Release 0.9.x progress can be found on the 0.9.x.dev and
0.9.x.rc-xxx branches.
Screenshots¶
version 0.8.6 sign in page
version 0.9.x sign in page
version 0.8.6 projects page
version 0.9.x projects page
480px :alt: version 0.8.6 portfolio create page
480px :alt: version 0.9.x portfolio detail page
version 0.8.6 module page
version 0.9.x module page
version 0.8.6 question page
version 0.9.x question page
Sign-in Page¶
0.8.6 (Old Version) + |
0.9.x (New Version) + |
|086_sign_in| + |
|090_sign_in| + |
Projects List Page¶
0.8.6 (Old Version) + |
0.9.x (New Version) + |
|086_projects| + |
|090_projects| + |
Portfolios (New in 0.9.x)¶
0.8.6 (Old Version) + |
0.9.x (New Version) + |
Feature did not exist + |
|090_portfolios_1| + |
Feature did not exist + |
|090_portfolios_2| + |
Module View¶
0.8.6 (Old Version) + |
0.9.x (New Version) + |
|086_modules| + |
|090_modules| + |
Question Page¶
0.8.6 (Old Version) + |
0.9.x (New Version) + |
|086_question| + |
|090_question| + |
Release Date¶
The target release date for 0.9.x is Autumn 2019.
Upgrading to 0.9.x from 0.8.x¶
Backup your database before upgrading to 0.9.x. Release 0.9.x performs database changes that makes rolling back difficult.
Upgrading to 0.9.x from 0.8.x¶
Backup your database before upgrading to 0.9.x. Release 0.9.x performs database changes that makes rolling back difficult.
Adding and Managing “Compliance Apps” in 0.9.x¶
GovReady-Q wll load a small set of example questionnaires as part of the
python manage.py first_run command during installation.
GovReady-Q must be configured by an administrator to load additional
compliance apps (e.g., assessments and questionnaires) from one or more
sources, which can be local directories or remote git repositories. Full
administrative privileges are assigned to original user account created
when executing python manage.py first_run during installation.
App Sources are configured in the Django admin at the URL /admin on
your GovReady-Q domain under App Sources:
App Sources list¶
Each App Source points GovReady-Q to a directory or repository of compliance apps.
Example App Source¶
The first App Source field is the Slug. The Slug is a short name you assign to the App Source to distinguish it from other App Sources. The Slug is used to form URLs in GovReady-Q’s compliance apps catalog, so it may only contain letters, numbers, dashes, underscores, and other URL path-safe characters.
There are four types of App Sources: local directories, remote git repositories using HTTP which are typically public repositories, remote git repositories using SSH which typically use SSH deploy keys for access, and remote GitHub repositories using a GitHub username and password for access.
Local Directory¶
The Local Directory source type directs GovReady-Q to load compliance apps from a directory on the same machine GovReady-Q is running on. (When deploying with Docker, that’s on the container filesystem unless a path has been mounted to a volume or to the host machine.)
In the Path field, enter the path to a local directory containing
compliance apps. This path is expected to contain a sub-directory for
each compliance app contained in this source. For instance, if you have
this directory layout:
.
└── home
└── user
└── compliance_apps
├── myfirstapp
│ └── app.yaml
└── mysecondapp
└── app.yaml
then your Path would be /home/user/compliance_apps.
The path can be absolute or relative to the path in which GovReady-Q is installed.
Git Repository over HTTPS¶
The Git Repository over HTTPS source type is for git repositories, such as on GitHub or GitLab, that can be cloned using an HTTPS URL. These repositories are typically public, or in an enterprise environment public within your organization’s network.
Paste the HTTPS git clone URL — such as https://github.com/GovReady/govready-apps-dev — into the URL field. Here’s what that looks like:
App Source for a public git repository¶
The other fields can be left blank.
The Path field optionally specifies a sub-directory within the
repository in which the compliance apps are stored if they are not
stored in the root of the repository. For instance if the repository has
a directory layout similar to:
.
└── github.com/organization/repository
└── apps
├── myfirstapp
│ └── app.yaml
└── mysecondapp
└── app.yaml
then set the Path field to apps.
If the compliance apps are not in the repository’s default branch
(i.e. something other than the typical master default branch), then
set the Branch field to the name of the branch to read the
compliance apps from.
You can use HTTPS to access private repositories by placing your username and password or personal access token into the URL, such as:
https://username:password@github.com/GovReady/govready-apps-dev
Since this requires user credentials, it should be avoided for production deployments in favor of using Git Repository over SSH (see below).
Git Repository over SSH¶
If your git repository is private and accessible using an SSH URL (which typically looks like git@github.com:organization/repository.git) and an SSH public/private keypair, such as with GitHub or GitLab deploy keys, then use the Git Repository over SSH source type.
Create a new SSH key for your GovReady-Q instance to be used as a deploy key:
ssh-keygen -q -t rsa -b 2048 -N "" -C "_your-repo-name_-deployment-key" -f ./repo_deploy_key
Your GovReady-Q instance will hold the private key half of the newly generated keypair, and your source code control system will hold the public key.
Back in the Django admin, set the Source Type to Git Repository over
SSH. Paste the git clone SSH URL into the URL field. Then open the newly
generated file repo_deploy_key and paste its contents into the SSH
Key field. Here’s what that looks like:
App Source for a private git repository¶
The other fields can be left blank. Path and Branch can be set
the same as with the Git Repository over HTTPS source type (see above).
Copy the public key in the newly generated file repo_deploy_key.pub
into the deploy keys section of your source code repository. Here is
what that looks like on GitHub:
Adding a deploy key to GitHub¶
Make the key read only by leaving “Allow write access” field unchecked
and click Add the key to save the key.
GitHub Repository using the GitHub API¶
This source type can be used to access private GitHub repositories using a GitHub username and password or a username and personal access token.
Set the Repository field to the organization name and repository
name, separated by a slash, as in the repository’s URL following
github.com/. In Other Parameters, paste a small YAML-formatted
document holding a GitHub username and password or username and personal
access token, formatted as follows:
auth:
user: 'myusername'
pw: 'mypassword'
The other fields can be left blank. Branch can be set the same as
with the Git Repository over HTTPS source type (see above).
Since this source type requires user credentials, it should be avoided for production deployments in favor of using Git Repository over SSH.
Compliance Apps¶
Compliance Apps are the actual assessments and questionnaires, the “data packs”, that drive GovReady-Q.
The AppSource admin now lists all of the apps provided by the source and has links to import new app versions into the database and to see the app versions already in the database by version number. When the App Source is defined, additional options appear on the database App Source admin page to selectively add individual projects and assessments from the App Source repository to the be published on GovReady-Q. Any time the individual admin page for an App Source is viewed, the App Source is rescanned and new versions of the apps are displayed to be selectively added to be available to users on GovReady-Q.
screenshot of App Sources list of apps¶
When starting a compliance app (i.e. creating a new project), we no longer have to import the app from the remote repository — instead, we create a new Project and set its root_task to point to a Module in an AppVersion already in the database.
App loading is refactored in a few places. The routines for getting app catalog information from the remote app data are removed since now we only need it for apps already stored in the database.
The AppSource admin’s approved app lists form is removed since adding apps into the database is now an administrative function and the database column for it is dropped.
AppVersion now has a boolean field for whether the instance should be included in the compliance apps catalog for users to start new projects with that app.
Migration Guide for GovReady-Q v0.8.6 to v0.9.x¶
Major Changes between v0.8.6 and v0.9.x¶
Changes GovReady-Q users will encounter in v0.9.x:
Organization subdomains are no longer used - everything is on the main domain.
Individual Questionnaires/Assessments from an App Source must now be explicitly added to the catalog via the Django admin page. More information can be found in the version 0.9.0 section of the GovReady-Q documentation.
Assessments are referred to as “Projects” in the UI.
“Projects” are organized into “Portfolios.” Every Project belongs to exactly one Portfolio.
Users can be added to Portfolios and be granted different permissions in Portfolios via an improved permission model.
v0.8.6 |
v0.9.x |
What migrating does |
|---|---|---|
Subdomains. |
Single domain. |
Single domain is used as per |
Organizations associated with different subdomains. |
Single “master” organization. |
A “Portfolio” is created for each Organization with the same name as the Subdomain. Organizations continue to exist in database, but are not used. |
Users are associated with multiple subdomain organizations. |
All users associated with single instance. |
Preserves users. |
User have different profiles for each subdomain organization. |
User has single profile. |
First profile is kept and can be edited by user. |
User “is staff” and “is super user” set in Django Admin. |
User “is staff” and “is super user” set in Django Admin. |
Roles preserved, see Roles and Permissions in v0.9.x section. |
Started and completed apps. |
Started and completed apps. |
All existing apps/questionnaires are preserved. Each questionnaire becomes associated with the “Portfolio” having the same name as the Organization to which the project was previously associated. |
User signs in under each subdomain. |
User signs in once. |
Subdomain associates are removed. |
Complete list of changes: https://github.com/GovReady/govready-q/blob/0.9.0.dev/CHANGELOG.md.
Migration Process¶
General Guidelines¶
Make sure your live data is backed up, and can be found and restored properly, before, during, and after the migration.
Do test migrations on test servers first, to ensure you understand the process and have worked out any kinks, before working on production servers.
Check for customizations, and preserve or modify them as needed.
Back Up Your Production Data¶
Ensure that you have a backup of your production data, and that it is safe, and will be available for a successful restore.
Read the Documentation¶
Familiarize yourself with the migration process before conducting the first test migration.
Make sure GovReady-Q official documentation is working for you.
Do a Test Migration¶
Use test data that will model and exercise the same features as your production data.
Use a clean install of the version you run in production, with test data installed.
Migrate your customizations to the test platform.
Perform the upgrade to v0.9.x.release.
Test to ensure the upgrade performed properly. Keep notes as you test.
Repeat until you’re comfortable with the process and results.
Optionally have selected end-users sign into the upgraded test instance to perform their own tests.
Distribute a Migration Plan¶
Create a migration plan. Include user impacts, timelines, contingency plans, and technical details (perhaps in a separate technical plan).
Confirm that your colleagues who are responsible, accountable, consulted, and informed about the migration are satisfied with the plan.
Distribute the plan to anybody affected.
When migration starts, communicate to users that migration is starting, and keep a communication line with them open.
Do the Production Migration for Deployments with Source Code¶
Take your GovReady-Q instance offline.
Back up the most recent version of the production database.
Test a restore of most recent version of the database.
Update code to version 0.9.0.release.
Run
pip install -r requirements.txtto get new libraries.Update any template and file customizations in your workstream.
Run
python manage.py migrateto update your database schema.Run
python manage.py runserverto run your updated instance.
Do the Production Migration for Deployments with Docker/Containers¶
Take your GovReady-Q instance offline.
Back up the most recent version of the production database.
Test a restore of most recent version of the database.
Synchronize your container customizations to produce a new version of your container.
Deploy container running version 0.9.0 with environment variable ``DB_BACKED_UP_DO_UPGRADE`` set to “True”. (This special environment variable is required to avoid accidental running of database migrations before database has been backed up.)
Docker will automatically run migrations as part of deployment.
Migration Finalization and Testing¶
Test the new instance.
Back up the newly migrated production database.
Post Migration Clean Up¶
Mark all old Notifications
emailedas True. (v0.9.x notification checks emailed status of notifications and sets emailed to True after email is sent.)Review Help Squad, Reviewers, and Administrators.
The migration converts Organizations to Portfolios.
Help Squad, Reviewers, and Organization admins are only preserved on the “main” organization (e.g., first organization created).
The migration does not modify the Help Squad or Reviewers.
Organization Admins are associated with the “main” organization.
You should review who has these permissions and adjust accordingly after migrating.
Release the production instance to users.
Roles and Permissions in v0.9.x¶
Permission/Role Name |
Description |
What Happens During Migration |
|---|---|---|
Organization > get_who_can_read |
A user can see an Organization if: they have read permission on any Project within the Organization, they are an editor of a Task within a Project within the Organization (but might not otherwise be a Project member), they are a guest in any Discussion on TaskQuestion in a Task in a Project in the Organization. |
A Portfolio is created for every organization that exists. |
projectmembership |
See Project > has_read_priv |
See below. |
Project > has_read_priv (Inverse is Project > get_all_participants) |
Team members + anyone with read privs to a task within this project + anyone who is a guest in discussion within this project. |
See below. |
Project > is_admin |
Person flagged as project admin in ProjectMembership. |
Grant project_delete permission on project and portfolio_owner permission for portfolio for which project is a part |
Project > is_member Project > editor_of task(s)Project > discussion_guest_in discussion(s) |
Various permissions in a project. |
Grant project_view, project_add, project_change, for project of which project is a part; Grant Portfolio_view, portfolio_add, portfolio_change for which project is a part after migration. |
task_editor |
The user that has primary responsibility for completing this Task. |
Grant project_view, project_add, project_change, for project of which task is a part; Grant Portfolio_view, portfolio_add, portfolio_change for which task is a part after migration. |
help_squad |
Receives all discussion messages |
Not modified by migration |
reviewer |
Can set review status of answers. |
Not modified by migration |
superuser |
Django designated superuser. |
Not modified by migration |
Folder permissions |
A folder object exists but is not used. |
Not modified by migration |
portfolio_owner |
Permission on portfolio object, can invite others to portfolio and can make others to portfolio owner |
If user was project_membership and had project_membership admin flag True, user is made portfolio owner. |
Portfolio_view, portfolio_add, portfolio_delete, portfolio_change |
Permissions on Portfolio objects. Currently everyone who has one of these has all of these permissions. |
Every user gets a portfolio with their name for which user is the portfolio_owner. For every organization, a portfolio is created with the same name and associating the organization projects with the portfolio. Users get access to the projects. |
project_view, project_add, project_change, project_delete |
Permissions on Project objects. Currently everyone who has one of these has all of these permissions. |
If user has project_membership on project, user gets project_view, project_add, project_change. If user has project_membership on project and project_membership admin flag True, user also gets project_delete. If user has task object for a project, user gets project_view, project_add, project_change. If user is a guest of a discussion object for a project, user gets project_view, project_add, project_change. |