Quickstart
Select installation instructions from those most closely related to the operating system that you use below. Note that Augur’s dependencies do not consistently support python 3.11 at this time. Python 3.8 - Python 3.10 have been tested on each platform.
- Ubuntu 22.x Setup
- Git Platform Requirements (Things to have setup prior to initiating installation.)
- Pre-Requisite Operating System Level Packages
- Git Configuration
- Postgresql Configuration
- Rabbitmq Broker Configuration
- Proxying Augur through Nginx
- Installing and Configuring Augur!
- Post Installation of Augur
- Postgresql Configuration
- Augur Commands
- Starting your Augur Instance
- Accessing Repo Addition and Visualization Front End
- Stopping your Augur Instance
- OSX Setup
- For OSX You Need to make sure to install XCode Command line tools:
- Git Platform Requirements (Things to have setup prior to initiating installation.)
- Pre-Requisite Operating System Level Packages
- Things to start before augur later
- Git Configuration
- Postgresql Configuration
- Installing and Configuring Augur!
- Post Installation of Augur
- (OPTIONAL: NOT FOR DEV: Proxying Augur through Nginx)
- Postgresql Configuration
- Augur Commands
- Starting your Augur Instance
- Accessing Repo Addition and Visualization Front End
- Stopping your Augur Instance
Explanations of Technologies
What does Redis Do?
Redis is used to make the state of data collection jobs visible on an external dashboard, like Flower. Internally, Augur relies on Redis to cache GitHub API Keys, and for OAuth Authentication. Redis is used to maintain awareness of Augur’s internal state.
What does RabbitMQ Do?
Augur is a distributed system. Even on one server, there are many collection processes happening simultaneously. Each job to collect data is put on the RabbitMQ Queue by Augur’s “Main Brain”. Then independent workers pop messages off the RabbitMQ Queue and go collect the data. These tasks then become standalone processes that report their completion or failure states back to the Redis server.
Edit the /etc/redis/redis.conf
file to ensure these parameters
are configured in this way:
supervised systemd
databases 900
maxmemory-samples 10
maxmemory 20GB
NOTE: You may be able to have fewer databases and lower maxmemory settings. This is a function of how many repositories you are collecting data for at a given time. The more repositories you are managing data for, the close to these settings you will need to be.
Consequences : If the settings are too low for Redis, Augur’s maintainer team has observed cases where collection appears to stall. (TEAM: This is a working theory as of 3/10/2023 for Ubuntu 22.x, based on EC2 experiments.)
Possible EC2 Configuration Requirements
With virtualization there may be issues associated with redis-server connections exceeding available memory. In these cases, the following workarounds help to resolve issues.
Specifically, you may find this error in your augur logs:
redis.exceptions.ConnectionError: Error 111 connecting to 127.0.0.1:6379. Connection refused.
INSTALL sudo apt install libhugetlbfs-bin
COMMAND:
hugeadm --thp-never` &&
echo never > /sys/kernel/mm/transparent_hugepage/enabled
sudo vi /etc/rc.local
paste into /etc/rc.local
if test -f /sys/kernel/mm/transparent_hugepage/enabled; then
echo never > /sys/kernel/mm/transparent_hugepage/enabled
fi
EDIT : /etc/default/grub
add the following line:
GRUB_DISABLE_OS_PROBER=true
Postgresql Configuration
Your postgresql instance should optimally allow 1,000 connections:
max_connections = 1000 # (change requires restart)
shared_buffers = 8GB # min 128kB
work_mem = 2GB # min 64kB
Augur will generally hold up to 150 simultaneous connections while collecting data. The 1,000 number is recommended to accommodate both collection and analysis on the same database. Use of PGBouncer or other utility may change these characteristics.
Augur Commands
To access command line options, use augur --help
. To load repos from
GitHub organizations prior to collection, or in other ways, the direct
route is augur db --help
.
Start a Flower Dashboard, which you can use to monitor progress, and
report any failed processes as issues on the Augur GitHub site. The
error rate for tasks is currently 0.04%, and most errors involve
unhandled platform API timeouts. We continue to identify and add fixes
to handle these errors through additional retries. Starting Flower:
(nohup celery -A augur.tasks.init.celery_app.celery_app flower --port=8400 --max-tasks=1000000 &)
NOTE: You can use any open port on your server, and access the dashboard
in a browser with http://servername-or-ip:8400 in the example above
(assuming you have access to that port, and its open on your network.)
Starting your Augur Instance
Start Augur: (nohup augur backend start &)
When data collection is complete you will see only a single task running in your flower Dashboard.
Accessing Repo Addition and Visualization Front End
Your Augur instance will now be available at http://hostname.io:port_number
For example: http://chaoss.tv:5038
Note: Augur will run on port 5000 by default (you probably need to change that in augur_operations.config for OSX)
Stopping your Augur Instance
You can stop augur with augur backend stop
, followed by
augur backend kill
. We recommend waiting 5 minutes between commands
so Augur can shutdown more gently. There is no issue with data integrity
if you issue them seconds apart, its just that stopping is nicer than
killing.
Docker
Make sure docker, and docker compose are both installed
Modify the
environment.txt
file in the root of the repository to include your GitHub and GitLab API keys.If you are already running postgresql on your server you have two choices:
Change the port mappings in the
docker-compose.yml
file to match ports for Postgresql not currently in use.Change to variables in
environment.txt
to include the correct values for your local, non-docker-container database.
sudo docker build -t augur-new -f docker/backend/Dockerfile .
sudo docker compose --env-file ./environment.txt --file docker-compose.yml up
to run the database in a Docker Container orsudo docker compose --env-file ./environment.txt --file docker-compose.yml up
to connect to an already running database.