Installation Guide

This section shows how to install Augur’s Python library from the source. If you don’t have a required dependency, please follow the provided links to install and configure it. .. note:

There are three main issues new developers encounter when first installing Augur:

1. The absence of a `GCC` or `Fortran` compiler; which are required by NumPy and NLTK Python libraries. Look up how to install these compilers for your local operating system. Many times they need to be updated to a more current version.

2. Conflicting versions of Python: The fix is platform-specific. On Mac OS X, multiple versions of Python often have been installed by the OS, brew, Anaconda, or both. The result is some python commands draw from different paths because of how they link in `/usr/local/bin`

3. Multiple, or conflicting versions of PostgreSQL, sometimes due to the absence of a functional `psql` function at the command line.

General Requirements

Backend

Required:

Python 3.9 is not yet supported because TensorFlow, which we use in our machine learning workers, does not yet support Python 3.9.

Our REST API & data collection workers write in Python 3.6. We query the GitHub & GitLab API to collect data about issues, pull requests, contributors, and other information about a repository, so GitLab and GitHub access tokens are required for data collection.

Optional:

The value_worker uses a Go package called scc to run COCOMO calculations. Once you’ve installed Go, follow the appropriate steps for your system to install the scc package.

  • Install gcc OpenMP Support: sudo apt-get install libgomp1 – Ubuntu

The message_insights_worker uses a system-level package called OpenMP. You will need this installed at the system level for that worker to work.

Caching System (Redis)

Message Broker (RabbitMQ)

After installation, you must also set up your rabbitmq instance by running the below commands:

sudo rabbitmqctl add_user augur password123

sudo rabbitmqctl add_vhost augur_vhost

sudo rabbitmqctl set_user_tags augur augurTag

sudo rabbitmqctl set_permissions -p augur_vhost augur ".*" ".*" ".*"

Note

it is important to have a static hostname when using rabbitmq as it uses hostname to communicate with nodes.

Then, start rabbitmq server with .. code-block:: bash

sudo systemctl start rabbitmq.service

If your setup of rabbitmq is successful your broker url should look like this:

broker_url = ‘amqp://augur:password123@localhost:5672/augur_vhost’

During installation you will be prompted for this broker url.

Frontend

If you’re interested in using our visualizations, you can optionally install the frontend dependencies:

We use Vue.js as our frontend web framework and npm as our package manager.

Visualization API calls

On Ubuntu and other Linux flavors: if you want to use the new Augur API Calls that generate downloadable graphics developed in the https://github.com/chaoss/augur-community-reports repository, you need to install the firefox-geckodriver (on Ubuntu or Red Hat Fedora) or geckodriver on Mac OSX, at the system level. This dependency exists because the Bokeh libraries we use for these APIs require a web browser engine.

For Ubuntu you can use:

- which firefox-geckodriver
- if nothing returned, then:
- sudo apt install firefox-geckodriver

For Fedora you can use

- which firefox-geckodriver
- if nothing returned, then:
- sudo dnf install firefox-geckodriver

For Mac OSX you can use:

-  which geckodriver
-  if nothing returned, then:
-  brew install geckodriver

Note

If you have BOTH Firefox-geckodriver AND ChromeDriver installed the visualization API will not work.

We have fully tested with Firefox-gecko driver on Linux platforms, and geckodriver on OSX. If you have ONLY ChromeDriver installed, it will probably work. Open an issue if you have a functioning ChromeDriver implementation.

Installation

Now you’re ready to build! The steps below outline how to create a virtual environment (required) and start the installation process, after which you’ll move on to the next section to configure the workers. The instructions are written in a way that you can follow for your respective Operating System.

Note

Lines that start with a $ denote a command that needs to run in an interactive terminal.

Warning

Do NOT install or run Augur using sudo. It is not required, and using it will inevitably cause some permissions trouble.

For macOS Errata

If you’re running Augur on macOS, we strongly suggest updating your shell’s initialization script in the following:

In a terminal, open the script:

nano .bash_profile

Add the following line to the end of the file:

export OBJC_DISABLE_INITIALIZE_FORK_SAFETY=YES

Save the file and exit. Run this command to reload bash_profile:

source .bash_profile

Check if it is updated:

env

env should contain OBJC_DISABLE_INITIALIZE_FORK_SAFETY.

macOS takes “helpful” measures to prevent Python subprocesses (which Augur uses) from forking cleanly, and setting this environment variable disables these safety measures to restore regular Python functionality.

Warning

If you skip this step, you’ll likely see all housekeeper jobs randomly exiting for no reason, and the Gunicorn server will not behave nicely either. Skip this step at your own risk!

General Augur Installation Steps (Irrespective of Operating System)

  1. Clone the repository and change to the newly-created directory.

$ git clone 'https://github.com/chaoss/augur.git'
$ cd augur/

2. Create a virtual environment in a directory of your choosing. Be sure to use the correct python command for your installation of Python 3: on most systems, this is python3, but yours may differ (you can use python -V or python3 -V to check).

# to create the environment
$ python3 -m venv $HOME/.virtualenvs/augur_env

# to activate the environment
$ source $HOME/.virtualenvs/augur_env/bin/activate

  1. Set AUGUR_DB environment variable with a postgres database connection string (if you have not setup a database yet, refer to database setup) Note: Three terminals will be needed to collect data for augur, and AUGUR_DB needs to be set for 2 out of the 3. If you don’t want to add it to both terminals you can add it permanently in your .bashrc file if running bash, or .zshrc file if in running zsh.

# set postgres database connection string to AUGUR_DB environment variable
# replace <> variables with actual values
$ export AUGUR_DB=postgresql+psycopg2://<user>:<password>@<host>:<port>/<database_name>

  1. Run the install script. This script will:

  • Install Augur’s Python library and application server

  • Install Augur’s schema in the configured database

  • Prompt you for GitHub and GitLab keys

  • Add GitHub and GitLab keys to config table in the database

Note

The install script will also generate an Augur API key for your database at the very end. This key will be automatically inserted into your database and printed to your terminal. It requires to use the repo & repo group creation endpoints, so make sure you save it off somewhere! There is only one key per database.

# run the install script
$ make install

# If you want to develop with Augur, use this command instead
$ make install-dev

If you think something went wrong, check the log files in logs/. If you want to try again, you can use make clean to delete any build files before running make install again.

MacOS users:

If your build fails and in gunicorn.log you see this error: Connection in use: ('0.0.0.0', 5000), that means port 5000 is being used by another process. To solve this issue, go to System Preferences -> Sharing -> Disable Airplay Receiver.

If you want to test new code you have written, you can rebuild Augur using:

$ make rebuild-dev

Note

If you chose to install Augur’s frontend dependencies, you might see a bunch of canvas@1.6.x and canvas-prebuilt@1.6.x errors in the installation logs. These are harmless and caused by a few of our dependencies having optional requirements for old versions of these libraries. If they seem to be causing you trouble, feel free to open an issue.

To enable log parsing for errors, you need to install Elasticsearch and Logstash.

Warning

Please note, that Logstash v7.0 and above have unresolved issues that affect this functionality.

In order to use it in the near future, please download v6.8.

If you use a package manager, it defaults to v7+, so we recommend downloading binary .

This change is tested with Elasticsearch v7.8.0_2 and Logstash v6.8.10.

Once everything installs, you’re ready to configure your data collection workers!