PYROS

h1. PYROS INSTALLATION FROM THE BEGINNING (for dev only, history of the initial project creation)

{{>toc}}

---

h3. Install MySql (only if necessary)

See in III-Installation

https://projects.irap.omp.eu/projects/pyros/wiki/Wiki#III-INSTALLATION

---

h3. Create the database (from sql script generated with Mysql Workbench)

 * All platforms:

<pre>

One liner:

$ mysql -u root < pyros_create.sql

Or :

$ mysql -u root

mysql> create database pyros;

mysql> use pyros;

mysql> source pyros_create.sql;

(

TODO:

mysql> grant all on pyros.* to pyros@localhost identified by ‘pyros’;)

mysql> flush privileges;

)

</pre>

h3. Install Python3.5 (only if necessary)

See in III-Installation

https://projects.irap.omp.eu/projects/pyros/wiki/Wiki#III-INSTALLATION

h3. Create the project structure

<pre>

$ mkdir PYROS

</pre>

Example of a good organization :

1 project = N applis

1 appli = N models

A! L’appli est à côté du projet, PAS DEDANS,

cela facilite la REUTILISATION

(an appli can be part of many projects => reuse)

1 appli = 1 Python module, organized for Django, by default = appli web (but not mandatory)

<pre>

MYPROJECT/

	REQUIREMENTS.txt

	src/

		myproject/

		appli1/

		appli2/

		…

		appliN/

	public/

		static/

	private/

		venv_py35_pyros/

</pre>

Set needed folders:

<pre>

$ cd PYROS/

$ mkdir private public

$ mkdir public/static

</pre>

---

h3. Create a Python3 virtual environment dedicated to the project (inside the project folder)

See in III-Installation

https://projects.irap.omp.eu/projects/pyros/wiki/Wiki#III-INSTALLATION

h3. Activate the Python virtual environment (from inside the project)

See in III-Installation

https://projects.irap.omp.eu/projects/pyros/wiki/Wiki#III-INSTALLATION

h3. Install the needed Python packages (from within the virtual environment)

See in III-Installation

https://projects.irap.omp.eu/projects/pyros/wiki/Wiki#Install-the-needed-Python-packages-from-within-the-virtual-environment

(cf manual installation)

---

h3. Set Requirements

<pre>

$ pip freeze > REQUIREMENTS.txt

</pre>

h3. Create a Django project named PYROS

<pre>

From inside the project:

$ pwd

.../PYROS/

$ django-admin startproject pyros

Rename the project folder "pyros/" as "src/"

$ mv pyros src

We have then this architecture:

PYROS

├── REQUIREMENTS.txt

├── private

│   └── venv_py35_pyros

├── public

│   └── static

├── src

│   ├── manage.py

│   ├── pyros

│   │   ├── __init__.py

│   │   ├── settings.py

│   │   ├── urls.py

│   │   └── wsgi.py

</pre>

---

h3. Test the project

See in III-Installation

https://projects.irap.omp.eu/projects/pyros/wiki/Wiki#III-INSTALLATION

h3. The Web server

Apache : gère tous les fichiers statiques (images, html…), et délègue les fichiers python au serveur django (par défaut)

Le fichier pyros/urls.py prend le relai pour tout ce qui est django

Le moteur web django sera soit du wsgi soit du unicorn

Par défaut, 1 seul worker, mais on peut en configurer plusieurs, l’idéal étant de faire "nb coeurs + 1" 

(le worker maître qui fait le dispatching aux autres)

Frontend : Apache ou Ngininx

Backend : gunicorn (gère facilement des workers) ou uwsgi

<pre>

$ gunicorn pyros.wsgi

(à la place de manage runserver => A EVITER EN PROD)

Ou encore:

$ gunicorn --workers 5 library.wsgi

</pre>

---

h3. Set the database engine as MySql

Edit src/pyros/settings.py

<pre>

DATABASES = {

    'default': {

        'ENGINE': 'django.db.backends.mysql',

        'NAME': 'pyros',

        'USER': 'root',

        'PASSWORD': ''

    }

}

</pre>

---

h3. Import the database into Django (with inspectdb)

From src/ :

<pre>

$ ./manage.py inspectdb > models.py

</pre>

Issue on Mac OS X:

<pre>

Traceback (most recent call last):

  File "/Users/epallier/Documents/_W_more/PROJECTS/GFT/SOFT/PYROS/pyros/venv_py35_pyros/lib/python3.5/site-packages/django/db/backends/mysql/base.py", line 25, in <module>

    import MySQLdb as Database

  File "/Users/epallier/Documents/_W_more/PROJECTS/GFT/SOFT/PYROS/pyros/venv_py35_pyros/lib/python3.5/site-packages/MySQLdb/__init__.py", line 19, in <module>

    import _mysql

ImportError: dlopen(/Users/epallier/Documents/_W_more/PROJECTS/GFT/SOFT/PYROS/pyros/venv_py35_pyros/lib/python3.5/site-packages/_mysql.cpython-35m-darwin.so, 2): Library not loaded: libmysqlclient.18.dylib

  Referenced from: /Users/epallier/Documents/_W_more/PROJECTS/GFT/SOFT/PYROS/pyros/venv_py35_pyros/lib/python3.5/site-packages/_mysql.cpython-35m-darwin.so

  Reason: image not found

=> BOUH !!!

LA SOLUTION EST ICI : http://stackoverflow.com/questions/6383310/python-mysqldb-library-not-loaded-libmysqlclient-18-dylib

Il suffit de faire ceci:

$ sudo mkdir -p /usr/local/lib   

$ sudo ln -s /Applications/XAMPP/xamppfiles/lib/libmysql* /usr/local/lib/

Mais on peut aussi faire ceci:

Okay, so the offending file is /Users/epallier/Documents/_W_more/PROJECTS/GFT/SOFT/PYROS/pyros/venv_py35_pyros/lib/python3.5/site-packages/_mysql.cpython-35m-darwin.so

Next, figure out where _mysql.so thinks it should find libmysqlclient.18.dylib:

$ otool -L /Users/epallier/Documents/_W_more/PROJECTS/GFT/SOFT/PYROS/pyros/venv_py35_pyros/lib/python3.5/site-packages/_mysql.cpython-35m-darwin.so

/Users/epallier/Documents/_W_more/PROJECTS/GFT/SOFT/PYROS/pyros/venv_py35_pyros/lib/python3.5/site-packages/_mysql.cpython-35m-darwin.so:

	libmysqlclient.18.dylib (compatibility version 18.0.0, current version 18.0.0)

...

So, it's looking for libmysqlclient.18.dylib with no path information, let's fix that:

$ locate libmysqlclient.18.dylib

/Applications/XAMPP/xamppfiles/lib/libmysqlclient.18.dylib

/Library/SystemMigration/History/Migration-68137DFB-CB6A-4FBB-81E2-11BDB5D01E48/QuarantineRoot/usr/lib/libmysqlclient.18.dylib

$ sudo install_name_tool -change libmysqlclient.18.dylib /Applications/XAMPP/xamppfiles/lib/libmysqlclient.18.dylib /Users/epallier/Documents/_W_more/PROJECTS/GFT/SOFT/PYROS/pyros/venv_py35_pyros/lib/python3.5/site-packages/_mysql.cpython-35m-darwin.so

Now _mysql.so knows the full path to the library and everything works, regardless of environment variables.

$ otool -L /Users/epallier/Documents/_W_more/PROJECTS/GFT/SOFT/PYROS/pyros/venv_py35_pyros/lib/python3.5/site-packages/_mysql.cpython-35m-darwin.so

/Users/epallier/Documents/_W_more/PROJECTS/GFT/SOFT/PYROS/pyros/venv_py35_pyros/lib/python3.5/site-packages/_mysql.cpython-35m-darwin.so:

	/Applications/XAMPP/xamppfiles/lib/libmysqlclient.18.dylib (compatibility version 18.0.0, current version 18.0.0)

...

$ ./manage.py inspectdb > models.py

=> YES !!!

</pre>

---

h3. Create a Django application named "pyrosapp"

From src/ :

<pre>

$ ./manage.py startapp pyrosapp

</pre>

We obtain this structure:

<pre>

PYROS/

├── REQUIREMENTS.txt

├── private/

│   └── venv_py35_pyros/

├── public/

│   └── static/

├── src/

│   ├── manage.py

│   ├── pyros/

│   │   ├── __init__.py

│   │   ├── __pycache__

│   │   ├── settings.py

│   │   ├── urls.py

│   │   └── wsgi.py

│   └── pyrosapp/

│       ├── __init__.py

│       ├── admin.py

│       ├── apps.py

│       ├── migrations

│       ├── models.py

│       ├── tests.py

│       └── views.py

</pre>

---

h3. Replace the default pyrosapp models.py with the inspectdb generated one

From src/ :

<pre>

$ mv models.py pyrosapp/

</pre>

Add pyrosapp to the project's applications :

Edit src/pyros/settings.py

<pre>

INSTALLED_APPS = [

    'django.contrib.admin',

    'django.contrib.auth',

    'django.contrib.contenttypes',

    'django.contrib.sessions',

    'django.contrib.messages',

    'django.contrib.staticfiles',

    'pyrosapp',

]

</pre>

---

h3. Fix and improve the pyrosapp models.py file (generated by inspectdb)

h3. Initialize the git repository

*Git global setup:*

<pre>

$ git config --global user.name "Etienne Pallier"

$ git config --global user.email "etienne.pallier@irap.omp.eu"

$ cat ~/.gitconfig 

[user]

	name = Etienne Pallier

	email = epallier@irap.omp.eu

[http]

	sslVerify = false

</pre>

*Create a new repository:*

<pre>

$ cd PYROS/

Define files and folders to be ignored:

$ vi .gitignore

.DS_Store

private

__pycache__

$ touch README.md

$ git add README.md

$ git commit -m "first commit"

$ git remote add origin https://gitlab.irap.omp.eu/epallier/pyros.git

$ git push -u origin master

$ git add .

( if you want to be sure to add ALL files: 

$ git add -A

)

( if you wanted to remove added files, just type:

$ git reset HEAD

)

$ git commit -m "first full project commit"

$ git push -u origin master

Counting objects: 43, done.

Delta compression using up to 4 threads.

Compressing objects: 100% (41/41), done.

Writing objects: 100% (43/43), 575.13 KiB ö 0 bytes/s, done.

Total 43 (delta 2), reused 0 (delta 0)

To https://gitlab.irap.omp.eu/epallier/pyros.git

   9c7128c..64501c9  master -> master

Branch master set up to track remote branch master from origin.

$ git status

On branch master

Your branch is up-to-date with 'origin/master'.

nothing to commit, working directory clean

</pre>

---

h3. Fix and improve the pyrosapp models.py file (generated by inspectdb)

Once models.py file generated, we need to delete the database and create an empty one :

<pre>

$ mysql -u root [-p (if password needed)]

mysql> DROP DATABSE pyros

mysql> CREATE SCHEMA IF NOT EXISTS 'pyros' DEFAULT CHARACTER SET utf8;

</pre>

Then edit pyrosapp/models.py :

 * Change 'managed = False' to 'managed = True' for every model

 * Change classes names to CamelCase (do not change the 'db_table = ...' lines). *Be careful* : it is needed to change all occurences :

    * NrtAlanysis

    * ScheduleHistory

    * ScientificProgram

    * SequenceType

    * SiteWatch

    * SiteWatchHistory

    * StrategyObs

    * UserLevel

    * WeatherWatch

    * WeatherWatchHistory

 * Change the deleting mode from 'models.DO_NOTHING' to 'models.CASCADE' for the following foreign keys :

    * Image.plan

    * Plan.album

    * Album.sequence

    * Sequence.request

 * Change the 'ForeignKey' liaisons to 'OneToOneField' liaisons (just replace ForeignKey by OneToOneField), and change deleting mode to 'models.CASCADE' for the following foreign keys :

    * Alert.request

    * Detector.device

    * Filter.device

    * Telescope.device

 * We need to redefine many to many relationships for the following classes :

    * User - ScientificProgram :

        * add 'users = models.ManyToManyField('User')' in ScientificProgram class

        * delete UserHasScientificProgram class

    * Sequence - ScheduleHistory

        * add 'sequences = models.ManyToManyField('Sequence')' in ScheduleHistory class

        * delete ScheduleHasSequences class

 * For each ForeignKey and ManyToManyField creation in models.py, add the 'related_name=[...]' named parameter, as in the following examples :

<pre>

class Sequence(models.Model):

    request = models.ForeignKey(Request, models.CASCADE, related_name="sequences")

    sequencetype = models.ForeignKey('SequenceType', models.DO_NOTHING, related_name="sequences")

    schedule = models.ForeignKey(Schedule, models.DO_NOTHING, related_name="sequences")

    name = models.CharField(max_length=45, blank=True, null=True)

    desc = models.TextField(blank=True, null=True)

    ...

class ScheduleHistory(models.Model):

    sequences = models.ManyToManyField('Sequence', related_name='schedulehistorys')

    created = models.DateTimeField(blank=True, null=True)

    ...

</pre>

 * Finally apply modifications to the database :

<pre>

$ pwd

.../PYROS/src

$ python manage.py makemigrations pyrosapp

$ python manage.py migrate

</pre>

---

h3. CONFIGURATION of the Django Back Office (administration interface)

cf https://projects.irap.omp.eu/projects/pyros/wiki/Wiki#IV-CONFIGURATION-of-the-Django-Back-Office-administration-interface