PYROS

h1. PYROS INSTALLATION FROM THE BEGINNING (for dev only, history of the initial project creation)

h3. Install MySql (only if necessary)

See above in III-Installation

https://projects.irap.omp.eu/projects/pyros/wiki/Wiki#III-INSTALLATION

---

h3. Create the database (from sql script generated with Mysql Workbench)

 * Linux and Mac OS X:

<pre>

One liner:

$ mysql -u root < pyros_create.sql

Or :

$ mysql -u root

mysql> create database pyros;

mysql> use pyros;

mysql> source pyros_create.sql;

(

TODO:

mysql> grant all on pyros.* to pyros@localhost identified by ‘pyros’;)

mysql> flush privileges;

)

</pre>

 * Windows:

<pre>

Download and install the newest version on https://dev.mysql.com/downloads/installer/

Once installed, launch MySQL Installer. Clic on 'Add...' on the right.

In MySQLServers section, choose the newest, then clic on next.

Install and configure the server (just follow the installation guide).

Then launch mysql (via the Windows menu).

See linux section below for the SQL commands to create and fill the database

</pre>

h3. Install Python3.5 (only if necessary)

See above in III-Installation

https://projects.irap.omp.eu/projects/pyros/wiki/Wiki#III-INSTALLATION

h3. Create the project structure

<pre>

$ mkdir PYROS

</pre>

Example of a good organization :

1 project = N applis

1 appli = N models

A! L’appli est à côté du projet, PAS DEDANS,

cela facilite la REUTILISATION

(an appli can be part of many projects => reuse)

1 appli = 1 Python module, organized for Django, by default = appli web (but not mandatory)

<pre>

MYPROJECT/

	REQUIREMENTS.txt

	src/

		myproject/

		appli1/

		appli2/

		…

		appliN/

	public/

		static/

	private/

		venv_py35_pyros/

</pre>

Set needed folders:

<pre>

$ cd PYROS/

$ mkdir private public

$ mkdir public/static

</pre>

---

h3. Create a Python3 virtual environment dedicated to the project (inside the project folder)

See above in III-Installation

https://projects.irap.omp.eu/projects/pyros/wiki/Wiki#III-INSTALLATION

h3. Activate the Python virtual environment (from inside the project)

See above in III-Installation

https://projects.irap.omp.eu/projects/pyros/wiki/Wiki#III-INSTALLATION

h3. Install the needed Python packages (from within the virtual environment)

See above in III-Installation

https://projects.irap.omp.eu/projects/pyros/wiki/Wiki#Install-the-needed-Python-packages-from-within-the-virtual-environment

(cf manual installation)

h3. Create a Django project named PYROS

<pre>

From inside the project:

$ pwd

.../PYROS/

$ django-admin startproject pyros

Rename the project folder "pyros/" as "src/"

$ mv pyros src

We have then this architecture:

PYROS

├── REQUIREMENTS.txt

├── private

│   └── venv_py35_pyros

├── public

│   └── static

├── src

│   ├── manage.py

│   ├── pyros

│   │   ├── __init__.py

│   │   ├── settings.py

│   │   ├── urls.py

│   │   └── wsgi.py

</pre>

---

h3. Test the project

See above in III-Installation

https://projects.irap.omp.eu/projects/pyros/wiki/Wiki#III-INSTALLATION

h3. The Web server

Apache : gère tous les fichiers statiques (images, html…), et délègue les fichiers python au serveur django (par défaut)

Le fichier pyros/urls.py prend le relai pour tout ce qui est django

Le moteur web django sera soit du wsgi soit du unicorn

Par défaut, 1 seul worker, mais on peut en configurer plusieurs, l’idéal étant de faire "nb coeurs + 1" 

(le worker maître qui fait le dispatching aux autres)

Frontend : Apache ou Ngininx

Backend : gunicorn (gère facilement des workers) ou uwsgi

<pre>

$ gunicorn pyros.wsgi

(à la place de manage runserver => A EVITER EN PROD)

Ou encore:

$ gunicorn --workers 5 library.wsgi

</pre>

---

h3. Set the database engine as MySql

Edit src/pyros/settings.py

<pre>

DATABASES = {

    'default': {

        'ENGINE': 'django.db.backends.mysql',

        'NAME': 'pyros',

        'USER': 'root',

        'PASSWORD': ''

    }

}

</pre>

---

h3. Import the database into Django (with inspectdb)

From src/ :

<pre>

$ ./manage.py inspectdb > models.py

</pre>

Issue on Mac OS X:

<pre>

Traceback (most recent call last):

  File "/Users/epallier/Documents/_W_more/PROJECTS/GFT/SOFT/PYROS/pyros/venv_py35_pyros/lib/python3.5/site-packages/django/db/backends/mysql/base.py", line 25, in <module>

    import MySQLdb as Database

  File "/Users/epallier/Documents/_W_more/PROJECTS/GFT/SOFT/PYROS/pyros/venv_py35_pyros/lib/python3.5/site-packages/MySQLdb/__init__.py", line 19, in <module>

    import _mysql

ImportError: dlopen(/Users/epallier/Documents/_W_more/PROJECTS/GFT/SOFT/PYROS/pyros/venv_py35_pyros/lib/python3.5/site-packages/_mysql.cpython-35m-darwin.so, 2): Library not loaded: libmysqlclient.18.dylib

  Referenced from: /Users/epallier/Documents/_W_more/PROJECTS/GFT/SOFT/PYROS/pyros/venv_py35_pyros/lib/python3.5/site-packages/_mysql.cpython-35m-darwin.so

  Reason: image not found

=> BOUH !!!

LA SOLUTION EST ICI : http://stackoverflow.com/questions/6383310/python-mysqldb-library-not-loaded-libmysqlclient-18-dylib

Il suffit de faire ceci:

$ sudo mkdir -p /usr/local/lib   

$ sudo ln -s /Applications/XAMPP/xamppfiles/lib/libmysql* /usr/local/lib/

Mais on peut aussi faire ceci:

Okay, so the offending file is /Users/epallier/Documents/_W_more/PROJECTS/GFT/SOFT/PYROS/pyros/venv_py35_pyros/lib/python3.5/site-packages/_mysql.cpython-35m-darwin.so

Next, figure out where _mysql.so thinks it should find libmysqlclient.18.dylib:

$ otool -L /Users/epallier/Documents/_W_more/PROJECTS/GFT/SOFT/PYROS/pyros/venv_py35_pyros/lib/python3.5/site-packages/_mysql.cpython-35m-darwin.so

/Users/epallier/Documents/_W_more/PROJECTS/GFT/SOFT/PYROS/pyros/venv_py35_pyros/lib/python3.5/site-packages/_mysql.cpython-35m-darwin.so:

	libmysqlclient.18.dylib (compatibility version 18.0.0, current version 18.0.0)

...

So, it's looking for libmysqlclient.18.dylib with no path information, let's fix that:

$ locate libmysqlclient.18.dylib

/Applications/XAMPP/xamppfiles/lib/libmysqlclient.18.dylib

/Library/SystemMigration/History/Migration-68137DFB-CB6A-4FBB-81E2-11BDB5D01E48/QuarantineRoot/usr/lib/libmysqlclient.18.dylib

$ sudo install_name_tool -change libmysqlclient.18.dylib /Applications/XAMPP/xamppfiles/lib/libmysqlclient.18.dylib /Users/epallier/Documents/_W_more/PROJECTS/GFT/SOFT/PYROS/pyros/venv_py35_pyros/lib/python3.5/site-packages/_mysql.cpython-35m-darwin.so

Now _mysql.so knows the full path to the library and everything works, regardless of environment variables.

$ otool -L /Users/epallier/Documents/_W_more/PROJECTS/GFT/SOFT/PYROS/pyros/venv_py35_pyros/lib/python3.5/site-packages/_mysql.cpython-35m-darwin.so

/Users/epallier/Documents/_W_more/PROJECTS/GFT/SOFT/PYROS/pyros/venv_py35_pyros/lib/python3.5/site-packages/_mysql.cpython-35m-darwin.so:

	/Applications/XAMPP/xamppfiles/lib/libmysqlclient.18.dylib (compatibility version 18.0.0, current version 18.0.0)

...

$ ./manage.py inspectdb > models.py

=> YES !!!

</pre>

---

h3. Create a Django application named "pyrosapp"

From src/ :

<pre>

$ ./manage.py startapp pyrosapp

</pre>

We obtain this structure:

<pre>

PYROS/

├── REQUIREMENTS.txt

├── private/

│   └── venv_py35_pyros/

├── public/

│   └── static/

├── src/

│   ├── manage.py

│   ├── pyros/

│   │   ├── __init__.py

│   │   ├── __pycache__

│   │   ├── settings.py

│   │   ├── urls.py

│   │   └── wsgi.py

│   └── pyrosapp/

│       ├── __init__.py

│       ├── admin.py

│       ├── apps.py

│       ├── migrations

│       ├── models.py

│       ├── tests.py

│       └── views.py

</pre>

---

h3. Replace the default pyrosapp models.py with the inspectdb generated one

From src/ :

<pre>

$ mv models.py pyrosapp/

</pre>

Add pyrosapp to the project's applications :

Edit src/pyros/settings.py

<pre>

INSTALLED_APPS = [

    'django.contrib.admin',

    'django.contrib.auth',

    'django.contrib.contenttypes',

    'django.contrib.sessions',

    'django.contrib.messages',

    'django.contrib.staticfiles',

    'pyrosapp',

]

</pre>

---

h3. Fix and improve the pyrosapp models.py file (generated by inspectdb)

h3. Set Requirements

<pre>

$ pip freeze > REQUIREMENTS.txt

</pre>

---

h3. Initialize the git repository

*Git global setup:*

<pre>

$ git config --global user.name "Etienne Pallier"

$ git config --global user.email "etienne.pallier@irap.omp.eu"

$ cat ~/.gitconfig 

[user]

	name = Etienne Pallier

	email = epallier@irap.omp.eu

[http]

	sslVerify = false

</pre>

*Create a new repository:*

<pre>

$ cd PYROS/

Define files and folders to be ignored:

$ vi .gitignore

.DS_Store

private

__pycache__

$ touch README.md

$ git add README.md

$ git commit -m "first commit"

$ git remote add origin https://gitlab.irap.omp.eu/epallier/pyros.git

$ git push -u origin master

$ git add .

( if you want to be sure to add ALL files: 

$ git add -A

)

( if you wanted to remove added files, just type:

$ git reset HEAD

)

$ git commit -m "first full project commit"

$ git push -u origin master

Counting objects: 43, done.

Delta compression using up to 4 threads.

Compressing objects: 100% (41/41), done.

Writing objects: 100% (43/43), 575.13 KiB ö 0 bytes/s, done.

Total 43 (delta 2), reused 0 (delta 0)

To https://gitlab.irap.omp.eu/epallier/pyros.git

   9c7128c..64501c9  master -> master

Branch master set up to track remote branch master from origin.

$ git status

On branch master

Your branch is up-to-date with 'origin/master'.

nothing to commit, working directory clean

</pre>

---

h3. Fix and improve the pyrosapp models.py file (generated by inspectdb)

Once models.py file generated, we need to delete the database and create an empty one :

<pre>

$ mysql -u root [-p (if password needed)]

mysql> DROP DATABSE pyros

mysql> CREATE SCHEMA IF NOT EXISTS 'pyros' DEFAULT CHARACTER SET utf8;

</pre>

Then edit pyrosapp/models.py :

 * Change 'managed = False' to 'managed = True' for every model

 * Change classes names to CamelCase (do not change the 'db_table = ...' lines). *Be careful* : it is needed to change all occurences :

    * NrtAlanysis

    * ScheduleHistory

    * ScientificProgram

    * SequenceType

    * SiteWatch

    * SiteWatchHistory

    * StrategyObs

    * UserLevel

    * WeatherWatch

    * WeatherWatchHistory

 * Change the deleting mode from 'models.DO_NOTHING' to 'models.CASCADE' for the following foreign keys :

    * Image.plan

    * Plan.album

    * Album.sequence

    * Sequence.request

 * Change the 'ForeignKey' liaisons to 'OneToOneField' liaisons (just replace ForeignKey by OneToOneField), and change deleting mode to 'models.CASCADE' for the following foreign keys :

    * Alert.request

    * Detector.device

    * Filter.device

    * Telescope.device

 * We need to redefine many to many relationships for the following classes :

    * User - ScientificProgram :

        * add 'users = models.ManyToManyField('User')' in ScientificProgram class

        * delete UserHasScientificProgram class

    * Sequence - ScheduleHistory

        * add 'sequences = models.ManyToManyField('Sequence')' in ScheduleHistory class

        * delete ScheduleHasSequences class

 * For each ForeignKey and ManyToManyField creation in models.py, add the 'related_name=[...]' named parameter, as in the following examples :

<pre>

class Sequence(models.Model):

    request = models.ForeignKey(Request, models.CASCADE, related_name="sequences")

    sequencetype = models.ForeignKey('SequenceType', models.DO_NOTHING, related_name="sequences")

    schedule = models.ForeignKey(Schedule, models.DO_NOTHING, related_name="sequences")

    name = models.CharField(max_length=45, blank=True, null=True)

    desc = models.TextField(blank=True, null=True)

    ...

class ScheduleHistory(models.Model):

    sequences = models.ManyToManyField('Sequence', related_name='schedulehistorys')

    created = models.DateTimeField(blank=True, null=True)

    ...

</pre>

 * Finally apply modifications to the database :

<pre>

$ pwd

.../PYROS/src

$ python manage.py makemigrations pyrosapp

$ python manage.py migrate

</pre>

---

h3. CONFIGURATION of the Django Back Office (administration interface)

cf https://projects.irap.omp.eu/projects/pyros/wiki/Wiki#IV-CONFIGURATION-of-the-Django-Back-Office-administration-interface