PYROS

h1. PYROS INSTALLATION FROM THE BEGINNING (for dev only, history of the initial project creation)

{{>toc}}

---

h3. Install MySql (only if necessary)

See in III-Installation
https://projects.irap.omp.eu/projects/pyros/wiki/Wiki#III-INSTALLATION

---

h3. Create the database (from sql script generated with Mysql Workbench)

* All platforms:
<pre>
One liner:
$ mysql -u root < pyros_create.sql

Or :
$ mysql -u root
mysql> create database pyros;
mysql> use pyros;
mysql> source pyros_create.sql;

(
TODO:
mysql> grant all on pyros.* to pyros@localhost identified by ‘pyros’;)
mysql> flush privileges;
)
</pre>

h3. Install Python3.5 (only if necessary)

See in III-Installation
https://projects.irap.omp.eu/projects/pyros/wiki/Wiki#III-INSTALLATION

h3. Create the project structure

<pre>
$ mkdir PYROS
</pre>

Example of a good organization :

1 project = N applis
1 appli = N models
A! L’appli est à côté du projet, PAS DEDANS,
cela facilite la REUTILISATION
(an appli can be part of many projects => reuse)
1 appli = 1 Python module, organized for Django, by default = appli web (but not mandatory)

<pre>

MYPROJECT/
REQUIREMENTS.txt
src/
myproject/
appli1/
appli2/
…
appliN/

public/
static/

private/
venv_py35_pyros/

</pre>

Set needed folders:
<pre>
$ cd PYROS/
$ mkdir private public
$ mkdir public/static
</pre>

---

h3. Create a Python3 virtual environment dedicated to the project (inside the project folder)

See in III-Installation
https://projects.irap.omp.eu/projects/pyros/wiki/Wiki#III-INSTALLATION

h3. Activate the Python virtual environment (from inside the project)

See in III-Installation
https://projects.irap.omp.eu/projects/pyros/wiki/Wiki#III-INSTALLATION

h3. Install the needed Python packages (from within the virtual environment)

See in III-Installation
https://projects.irap.omp.eu/projects/pyros/wiki/Wiki#Install-the-needed-Python-packages-from-within-the-virtual-environment
(cf manual installation)

---

h3. Set Requirements

<pre>
$ pip freeze > REQUIREMENTS.txt
</pre>

h3. Create a Django project named PYROS

<pre>

From inside the project:
$ pwd
.../PYROS/

$ django-admin startproject pyros

Rename the project folder "pyros/" as "src/"

$ mv pyros src

We have then this architecture:

PYROS
├── REQUIREMENTS.txt
├── private
│ └── venv_py35_pyros
├── public
│ └── static
├── src
│ ├── manage.py
│ ├── pyros
│ │ ├── __init__.py
│ │ ├── settings.py
│ │ ├── urls.py
│ │ └── wsgi.py

</pre>

---

h3. Test the project

See in III-Installation
https://projects.irap.omp.eu/projects/pyros/wiki/Wiki#III-INSTALLATION

h3. The Web server

Apache : gère tous les fichiers statiques (images, html…), et délègue les fichiers python au serveur django (par défaut)

Le fichier pyros/urls.py prend le relai pour tout ce qui est django

Le moteur web django sera soit du wsgi soit du unicorn

Par défaut, 1 seul worker, mais on peut en configurer plusieurs, l’idéal étant de faire "nb coeurs + 1"
(le worker maître qui fait le dispatching aux autres)

Frontend : Apache ou Ngininx

Backend : gunicorn (gère facilement des workers) ou uwsgi

<pre>
$ gunicorn pyros.wsgi
(à la place de manage runserver => A EVITER EN PROD)

Ou encore:

$ gunicorn --workers 5 library.wsgi
</pre>

---

h3. Set the database engine as MySql

Edit src/pyros/settings.py

<pre>
DATABASES = {
'default': {
'ENGINE': 'django.db.backends.mysql',
'NAME': 'pyros',
'USER': 'root',
'PASSWORD': ''
}
}
</pre>

---

h3. Import the database into Django (with inspectdb)

From src/ :

<pre>
$ ./manage.py inspectdb > models.py
</pre>

Issue on Mac OS X:
<pre>
Traceback (most recent call last):
File "/Users/epallier/Documents/_W_more/PROJECTS/GFT/SOFT/PYROS/pyros/venv_py35_pyros/lib/python3.5/site-packages/django/db/backends/mysql/base.py", line 25, in <module>
import MySQLdb as Database
File "/Users/epallier/Documents/_W_more/PROJECTS/GFT/SOFT/PYROS/pyros/venv_py35_pyros/lib/python3.5/site-packages/MySQLdb/__init__.py", line 19, in <module>
import _mysql
ImportError: dlopen(/Users/epallier/Documents/_W_more/PROJECTS/GFT/SOFT/PYROS/pyros/venv_py35_pyros/lib/python3.5/site-packages/_mysql.cpython-35m-darwin.so, 2): Library not loaded: libmysqlclient.18.dylib
Referenced from: /Users/epallier/Documents/_W_more/PROJECTS/GFT/SOFT/PYROS/pyros/venv_py35_pyros/lib/python3.5/site-packages/_mysql.cpython-35m-darwin.so
Reason: image not found

=> BOUH !!!

LA SOLUTION EST ICI : http://stackoverflow.com/questions/6383310/python-mysqldb-library-not-loaded-libmysqlclient-18-dylib

Il suffit de faire ceci:

$ sudo mkdir -p /usr/local/lib
$ sudo ln -s /Applications/XAMPP/xamppfiles/lib/libmysql* /usr/local/lib/

Mais on peut aussi faire ceci:

Okay, so the offending file is /Users/epallier/Documents/_W_more/PROJECTS/GFT/SOFT/PYROS/pyros/venv_py35_pyros/lib/python3.5/site-packages/_mysql.cpython-35m-darwin.so

Next, figure out where _mysql.so thinks it should find libmysqlclient.18.dylib:

$ otool -L /Users/epallier/Documents/_W_more/PROJECTS/GFT/SOFT/PYROS/pyros/venv_py35_pyros/lib/python3.5/site-packages/_mysql.cpython-35m-darwin.so
/Users/epallier/Documents/_W_more/PROJECTS/GFT/SOFT/PYROS/pyros/venv_py35_pyros/lib/python3.5/site-packages/_mysql.cpython-35m-darwin.so:
libmysqlclient.18.dylib (compatibility version 18.0.0, current version 18.0.0)
...

So, it's looking for libmysqlclient.18.dylib with no path information, let's fix that:

$ locate libmysqlclient.18.dylib
/Applications/XAMPP/xamppfiles/lib/libmysqlclient.18.dylib
/Library/SystemMigration/History/Migration-68137DFB-CB6A-4FBB-81E2-11BDB5D01E48/QuarantineRoot/usr/lib/libmysqlclient.18.dylib

$ sudo install_name_tool -change libmysqlclient.18.dylib /Applications/XAMPP/xamppfiles/lib/libmysqlclient.18.dylib /Users/epallier/Documents/_W_more/PROJECTS/GFT/SOFT/PYROS/pyros/venv_py35_pyros/lib/python3.5/site-packages/_mysql.cpython-35m-darwin.so

Now _mysql.so knows the full path to the library and everything works, regardless of environment variables.

$ otool -L /Users/epallier/Documents/_W_more/PROJECTS/GFT/SOFT/PYROS/pyros/venv_py35_pyros/lib/python3.5/site-packages/_mysql.cpython-35m-darwin.so
/Users/epallier/Documents/_W_more/PROJECTS/GFT/SOFT/PYROS/pyros/venv_py35_pyros/lib/python3.5/site-packages/_mysql.cpython-35m-darwin.so:
/Applications/XAMPP/xamppfiles/lib/libmysqlclient.18.dylib (compatibility version 18.0.0, current version 18.0.0)
...

$ ./manage.py inspectdb > models.py

=> YES !!!
</pre>

---

h3. Create a Django application named "pyrosapp"

From src/ :

<pre>
$ ./manage.py startapp pyrosapp
</pre>

We obtain this structure:

<pre>

PYROS/
├── REQUIREMENTS.txt
├── private/
│ └── venv_py35_pyros/
├── public/
│ └── static/
├── src/
│ ├── manage.py
│ ├── pyros/
│ │ ├── __init__.py
│ │ ├── __pycache__
│ │ ├── settings.py
│ │ ├── urls.py
│ │ └── wsgi.py
│ └── pyrosapp/
│ ├── __init__.py
│ ├── admin.py
│ ├── apps.py
│ ├── migrations
│ ├── models.py
│ ├── tests.py
│ └── views.py

</pre>

---

h3. Replace the default pyrosapp models.py with the inspectdb generated one

From src/ :

<pre>
$ mv models.py pyrosapp/
</pre>

Add pyrosapp to the project's applications :

Edit src/pyros/settings.py

<pre>
INSTALLED_APPS = [
'django.contrib.admin',
'django.contrib.auth',
'django.contrib.contenttypes',
'django.contrib.sessions',
'django.contrib.messages',
'django.contrib.staticfiles',
'pyrosapp',
]
</pre>

---

h3. Initialize the git repository

*Git global setup:*

<pre>
$ git config --global user.name "Etienne Pallier"
$ git config --global user.email "etienne.pallier@irap.omp.eu"

$ cat ~/.gitconfig
[user]
name = Etienne Pallier
email = epallier@irap.omp.eu
[http]
sslVerify = false
</pre>

*Create a new repository:*

<pre>
$ cd PYROS/

Define files and folders to be ignored:
$ vi .gitignore
.DS_Store
private
__pycache__

$ touch README.md

$ git add README.md

$ git commit -m "first commit"

$ git remote add origin https://gitlab.irap.omp.eu/epallier/pyros.git

$ git push -u origin master

$ git add .

( if you want to be sure to add ALL files:
$ git add -A
)

( if you wanted to remove added files, just type:
$ git reset HEAD
)

$ git commit -m "first full project commit"

$ git push -u origin master
Counting objects: 43, done.
Delta compression using up to 4 threads.
Compressing objects: 100% (41/41), done.
Writing objects: 100% (43/43), 575.13 KiB ö 0 bytes/s, done.
Total 43 (delta 2), reused 0 (delta 0)
To https://gitlab.irap.omp.eu/epallier/pyros.git
9c7128c..64501c9 master -> master
Branch master set up to track remote branch master from origin.

$ git status
On branch master
Your branch is up-to-date with 'origin/master'.
nothing to commit, working directory clean

</pre>

---

h3. Fix and improve the pyrosapp models.py file (generated by inspectdb)

Edit Once models.py file generated, we need to delete the database and create an empty one :

&lt;pre&gt;
$ mysql -u root [-p (if password needed)]

mysql&gt; DROP DATABSE pyros;
mysql&gt; CREATE DATABASE pyros;

(or:
mysql&gt; CREATE SCHEMA IF NOT EXISTS &#x27;pyros&#x27; DEFAULT CHARACTER SET utf8;
)

&lt;/pre&gt;

Then edit pyrosapp/models.py :

* Change 'managed = False' to 'managed = True' for every model

* Change classes names to CamelCase (do not change the 'db_table = ...' lines). *Be careful* : it is needed to change all occurences :

* NrtAlanysis
* ScheduleHistory
* ScientificProgram
* SequenceType
* SiteWatch
* SiteWatchHistory
* StrategyObs
* UserLevel
* WeatherWatch
* WeatherWatchHistory

* Change the deleting mode from 'models.DO_NOTHING' to 'models.CASCADE' for the following foreign keys :

* Image.plan
* Plan.album
* Album.sequence
* Sequence.request

* Change the 'ForeignKey' liaisons to 'OneToOneField' liaisons (just replace ForeignKey by OneToOneField), and change deleting mode to 'models.CASCADE' for the following foreign keys :

* Alert.request
* Detector.device
* Filter.device
* Telescope.device

* We need to redefine many to many relationships for the following classes :

* User - ScientificProgram :

* add 'users = models.ManyToManyField('User')' in ScientificProgram class
* delete UserHasScientificProgram class

* Sequence - ScheduleHistory

* add 'sequences = models.ManyToManyField('Sequence')' in ScheduleHistory class
* delete ScheduleHasSequences class

* For each ForeignKey and ManyToManyField creation in models.py, add the 'related_name=[...]' named parameter, as in the following examples :

<pre>
class Sequence(models.Model):
request = models.ForeignKey(Request, models.CASCADE, related_name="sequences")
sequencetype = models.ForeignKey('SequenceType', models.DO_NOTHING, related_name="sequences")
schedule = models.ForeignKey(Schedule, models.DO_NOTHING, related_name="sequences")
name = models.CharField(max_length=45, blank=True, null=True)
desc = models.TextField(blank=True, null=True)
...

class ScheduleHistory(models.Model):
sequences = models.ManyToManyField('Sequence', related_name='schedulehistorys')
created = models.DateTimeField(blank=True, null=True)
...

</pre>



* Finally apply modifications to the database :

First, we need to delete the database and create an empty one :

<pre>
$ mysql -u root [-p (if password needed)]

mysql> DROP DATABSE pyros;
mysql> CREATE DATABASE pyros;

(or:
mysql> CREATE SCHEMA IF NOT EXISTS 'pyros' DEFAULT CHARACTER SET utf8;
)

</pre>

Then sync the database from Django models:

<pre>
$ pwd
.../PYROS/src
$ python manage.py makemigrations pyrosapp
$ python manage.py migrate
</pre>

---

h3. CONFIGURATION of the Django Back Office (administration interface)

cf https://projects.irap.omp.eu/projects/pyros/wiki/Wiki#IV-CONFIGURATION-of-the-Django-Back-Office-administration-interface