PYROS

h1. Technical Documentation for the PYROS project (FGFT-CC)

HOWTO Format Redmine Wiki : http://www.redmine.org/projects/redmine/wiki/FrRedmineWikiFormatting

{{>toc}}

---

h2. %{margin-left:0px; font-weight:bold; font-size:25px;  display:block; color:red;}I - TODO%

 * installation sur windows

 * gitlab

 * séparation des BD Django et Pyros

 * intégration dans Eclipse

 * Intégration des modules Django déjà développés

---

h2. %{margin-left:0px; font-weight:bold; font-size:25px;  display:block; color:red;}I - DATABASE SCHEMA (v0.2.1)%

{{thumbnail(PYROS_PDM_v021.png, size=300, title=Pyros data model)}}

---

h2. %{margin-left:0px; font-weight:bold; font-size:25px;  display:block; color:red;}II - Get the project (from gitlab)%

https://gitlab.irap.omp.eu/epallier/pyros

https://gitlab.irap.omp.eu/epallier/pyros.git

ssh git@gitlab.irap.omp.eu:epallier/pyros.git

h3. Get the project from the terminal

<pre>

git clone https://gitlab.irap.omp.eu/epallier/pyros.git PYROS

(or also : git clone git@gitlab.irap.omp.eu:epallier/pyros.git)

</pre>

If you just want a static copy of the project (without synchronization) just remove the .git/ folder :

<pre>

$ rm -r .git/

</pre>

h3. Get the project from Eclipse

TODO:

h2. %{margin-left:0px; font-weight:bold; font-size:25px;  display:block; color:red;}III - INSTALLATION%

---

h3. Install MySql (only if necessary)

 * Linux Ubuntu

<pre>

$ sudo apt-get install mysql-server

$ sudo apt-get install mysql-client

</pre>

 * Linux CentOS

<pre>

TODO:

$ sudo yum install mysql

...

</pre>

 * Mac OS X

Install XAMPP

(but you could also use the pre-installed Mac OS MySql)

<pre>

TODO:

</pre>

 * Windows

Install XAMPP

<pre>

TODO:

</pre>

---

h3. Install Python3.5 (only if necessary)

 * Mac OS X :

<pre>

1) Installer MacPort

(TODO: doc)

2) Installer le "port" python35

$ sudo port install python35

</pre>

 * Linux (Ubuntu) :

<pre>

sudo add-apt-repository ppa:fkrull/deadsnakes

sudo apt-get update

sudo apt-get install python3.5

sudo pip install virtualenv

</pre>

 * Windows 10 :

<pre>

TODO:

</pre>

---

h3. Create a Python3 virtual environment dedicated to the project (inside the project folder)

<pre>

$ mkdir private/

$ cd private/

$ which python3.5

/opt/local/bin/python3.5

$ virtualenv-3.5 venv_py35_pyros -p /opt/local/bin/python3.5

=> creates a venv_py35_pyros/ folder inside PYROS/private/

</pre>

---

h3. Activate the python virtual environment (from inside the project)

<pre>

$ pwd

.../PYROS/private

$ source ./venv_py35_pyros/bin/activate

$ python -V

Python 3.5.1

$ which pip

.../PYROS/venv_py35_pyros/bin/pip

Upgrade pip to last version available:

$ pip install --upgrade pip

Collecting pip

  Downloading pip-8.1.1-py2.py3-none-any.whl (1.2MB)

Installing collected packages: pip

  Found existing installation: pip 7.1.2

    Uninstalling pip-7.1.2:

      Successfully uninstalled pip-7.1.2

Successfully installed pip-8.1.1

Upgrade wheel to last version available:

$ pip install --upgrade wheel

Collecting wheel

  Downloading wheel-0.29.0-py2.py3-none-any.whl (66kB)

Installing collected packages: wheel

  Found existing installation: wheel 0.24.0

    Uninstalling wheel-0.24.0:

      Successfully uninstalled wheel-0.24.0

Successfully installed wheel-0.29.0

</pre>

h3. Install needed python packages (from within the virtual environment)

First, be sure that the virtual environment is activated:

<pre>

$ python -V

Python 3.5.1

</pre>

 * *Automatic Installation of all packages*

<pre>

$ pip install -r REQUIREMENTS.txt

</pre>

 * *Or, manual installation of each package*

  * *Install Django* :

<pre>

$ pip install django

Collecting django

  Downloading Django-1.9.4-py2.py3-none-any.whl (6.6MB)

Installing collected packages: django

Successfully installed django-1.9.4

$ pip install django-admin-tools

Collecting django-admin-tools

  Downloading django_admin_tools-0.7.2-py2.py3-none-any.whl (289kB)

Installing collected packages: django-admin-tools

Successfully installed django-admin-tools-0.7.2

$ pip install django-debug-toolbar

Collecting django-debug-toolbar

  Downloading django_debug_toolbar-1.4-py2.py3-none-any.whl (212kB)

Requirement already satisfied (use --upgrade to upgrade): Django>=1.7 in ./venv_py35_pyros/lib/python3.5/site-packages (from django-debug-toolbar)

Collecting sqlparse (from django-debug-toolbar)

  Downloading sqlparse-0.1.19.tar.gz (58kB)

Building wheels for collected packages: sqlparse

  Running setup.py bdist_wheel for sqlparse ... done

  Stored in directory: /Users/epallier/Library/Caches/pip/wheels/7b/d4/72/6011bb100dd5fc213164e4bbee13d4e03261dd54ce6a5de6b8

Successfully built sqlparse

Installing collected packages: sqlparse, django-debug-toolbar

Successfully installed django-debug-toolbar-1.4 sqlparse-0.1.19

$ pip install django-extensions

Collecting django-extensions

  Downloading django_extensions-1.6.1-py2.py3-none-any.whl (202kB)

Collecting six>=1.2 (from django-extensions)

  Downloading six-1.10.0-py2.py3-none-any.whl

Installing collected packages: six, django-extensions

Successfully installed django-extensions-1.6.1 six-1.10.0

$ pip install django-suit

Collecting django-suit

  Downloading django-suit-0.2.18.tar.gz (587kB)

Building wheels for collected packages: django-suit

  Running setup.py bdist_wheel for django-suit ... done

  Stored in directory: /Users/epallier/Library/Caches/pip/wheels/12/8b/9a/e02ab0ad9229881638aa040d47d77c8f562999533811927d41

Successfully built django-suit

Installing collected packages: django-suit

Successfully installed django-suit-0.2.18

</pre>

  * *Install the web application server gunicorn (will be used in production instead of the dev django web server)* :

<pre>

$ pip install gunicorn

Collecting gunicorn

  Downloading gunicorn-19.4.5-py2.py3-none-any.whl (112kB)

Installing collected packages: gunicorn

Successfully installed gunicorn-19.4.5

</pre>

  * *Install the python mysql client*:

<pre>

$ pip install mysqlclient

...

</pre>

   * => Issue under Mac OS X:

<pre>

$ pip install mysqlclient

Collecting mysqlclient

  Downloading mysqlclient-1.3.7.tar.gz (79kB)

Building wheels for collected packages: mysqlclient

  Running setup.py bdist_wheel for mysqlclient ... error

…

  ----------------------------------------

  Failed building wheel for mysqlclient

  Running setup.py clean for mysqlclient

Failed to build mysqlclient

Installing collected packages: mysqlclient

  Running setup.py install for mysqlclient ... done

Successfully installed mysqlclient-1.3.7

BOUH !!!

$ pip install --upgrade wheel

Collecting wheel

  Downloading wheel-0.29.0-py2.py3-none-any.whl (66kB)

Installing collected packages: wheel

  Found existing installation: wheel 0.24.0

    Uninstalling wheel-0.24.0:

      Successfully uninstalled wheel-0.24.0

Successfully installed wheel-0.29.0

$ pip uninstall mysqlclient

$ pip install mysqlclient

Collecting mysqlclient

  Using cached mysqlclient-1.3.7.tar.gz

Building wheels for collected packages: mysqlclient

  Running setup.py bdist_wheel for mysqlclient ... done

  Stored in directory: /Users/epallier/Library/Caches/pip/wheels/9b/06/50/d11418c26cf8f2156b13d4363b5afde8e7e75ebb8540d0228d

Successfully built mysqlclient

Installing collected packages: mysqlclient

Successfully installed mysqlclient-1.3.7

YES !!!

</pre>

   * => Issues under Ubuntu:

<pre>

$ pip install mysqlclient

Collecting mysqlclient

  Downloading mysqlclient-1.3.7.tar.gz (79kB)

    100% |████████████████████████████████| 81kB 1.5MB/s

    Complete output from command python setup.py egg_info:

    /bin/sh: 1: mysql_config: not found

    Traceback (most recent call last):

      File "<string>", line 1, in <module>

    [...]

    ----------------------------------------

Command "python setup.py egg_info" failed with error code 1 in /tmp/pip-build-q6j4inuz/mysqlclient/

BOUH !!!

$ sudo apt-get install libmysqlclient-dev

$ pip install mysqlclient

Collecting mysqlclient

  Using cached mysqlclient-1.3.7.tar.gz

Building wheels for collected packages: mysqlclient

  Running setup.py bdist_wheel for mysqlclient ... error

…

    _mysql.c:40:20: fatal error: Python.h: No such file or directory

     #include "Python.h"

                        ^

    compilation terminated.

    error: command 'x86_64-linux-gnu-gcc' failed with exit status 1

    ----------------------------------------

Command "/home/carens_p/pyros/venv_py35_pyros/bin/python3.5 -u -c "import setuptools, tokenize;__file__='/tmp/pip-build-k3klv92j/mysqlclient/setup.py';exec(compile(getattr(tokenize, 'open', open)(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" install --record /tmp/pip-gz242xxs-record/install-record.txt --single-version-externally-managed --compile --install-headers /home/carens_p/pyros/venv_py35_pyros/include/site/python3.5/mysqlclient" failed with error code 1 in /tmp/pip-build-k3klv92j/mysqlclient/

BOUH !!!

$ sudo apt-get install python3.5-dev

$ pip install mysqlclient

YES !!!

</pre>

 * *Set Requirements*

<pre>

$ pip freeze > REQUIREMENTS.txt

</pre>

---

h3. Create Django project pyros

<pre>

From inside the project:

$ pwd

.../PYROS/

$ django-admin startproject pyros

Rename the project folder "pyros/" as "src/"

$ mv pyros src

We have then this architecture:

PYROS

├── REQUIREMENTS.txt

├── private

│   └── venv_py35_pyros

├── public

│   └── static

├── src

│   ├── manage.py

│   ├── pyros

│   │   ├── __init__.py

│   │   ├── settings.py

│   │   ├── urls.py

│   │   └── wsgi.py

</pre>

---

h3. Test the project

<pre>

$ cd src/

$ ./manage.py runserver

(or gunicorn pyros.wsgi)

==> http://localhost:8000

...

...

Ctrl-c

</pre>

---

h3. The Web server

Apache : gère tous les fichiers statiques (images, html…), et délègue les fichiers python au serveur django (par défaut)

Le fichier pyros/urls.py prend le relai pour tout ce qui est django

Le moteur web django sera soit du wsgi soit du unicorn

Par défaut, 1 seul worker, mais on peut en configurer plusieurs, l’idéal étant de faire "nb coeurs + 1" 

(le worker maître qui fait le dispatching aux autres)

Frontend : Apache ou Ngininx

Backend : gunicorn (gère facilement des workers) ou uwsgi

<pre>

$ gunicorn pyros.wsgi

(à la place de manage runserver => A EVITER EN PROD)

Ou encore:

$ gunicorn --workers 5 library.wsgi

</pre>

---

h3. Set Database engine as MySql

Edit src/pyros/settings.py

<pre>

DATABASES = {

    'default': {

        'ENGINE': 'django.db.backends.mysql',

        'NAME': 'pyros',

        'USER': 'root',

        'PASSWORD': ''

    }

}

</pre>

---

h3. Import database into Django (with inspectdb)

From src/ :

<pre>

$ ./manage.py inspectdb > models.py

</pre>

Issue on Mac OS X:

<pre>

Traceback (most recent call last):

  File "/Users/epallier/Documents/_W_more/PROJECTS/GFT/SOFT/PYROS/pyros/venv_py35_pyros/lib/python3.5/site-packages/django/db/backends/mysql/base.py", line 25, in <module>

    import MySQLdb as Database

  File "/Users/epallier/Documents/_W_more/PROJECTS/GFT/SOFT/PYROS/pyros/venv_py35_pyros/lib/python3.5/site-packages/MySQLdb/__init__.py", line 19, in <module>

    import _mysql

ImportError: dlopen(/Users/epallier/Documents/_W_more/PROJECTS/GFT/SOFT/PYROS/pyros/venv_py35_pyros/lib/python3.5/site-packages/_mysql.cpython-35m-darwin.so, 2): Library not loaded: libmysqlclient.18.dylib

  Referenced from: /Users/epallier/Documents/_W_more/PROJECTS/GFT/SOFT/PYROS/pyros/venv_py35_pyros/lib/python3.5/site-packages/_mysql.cpython-35m-darwin.so

  Reason: image not found

=> BOUH !!!

LA SOLUTION EST ICI : http://stackoverflow.com/questions/6383310/python-mysqldb-library-not-loaded-libmysqlclient-18-dylib

Il suffit de faire ceci:

$ sudo mkdir -p /usr/local/lib   

$ sudo ln -s /Applications/XAMPP/xamppfiles/lib/libmysql* /usr/local/lib/

Mais on peut aussi faire ceci:

Okay, so the offending file is /Users/epallier/Documents/_W_more/PROJECTS/GFT/SOFT/PYROS/pyros/venv_py35_pyros/lib/python3.5/site-packages/_mysql.cpython-35m-darwin.so

Next, figure out where _mysql.so thinks it should find libmysqlclient.18.dylib:

$ otool -L /Users/epallier/Documents/_W_more/PROJECTS/GFT/SOFT/PYROS/pyros/venv_py35_pyros/lib/python3.5/site-packages/_mysql.cpython-35m-darwin.so

/Users/epallier/Documents/_W_more/PROJECTS/GFT/SOFT/PYROS/pyros/venv_py35_pyros/lib/python3.5/site-packages/_mysql.cpython-35m-darwin.so:

	libmysqlclient.18.dylib (compatibility version 18.0.0, current version 18.0.0)

...

So, it's looking for libmysqlclient.18.dylib with no path information, let's fix that:

$ locate libmysqlclient.18.dylib

/Applications/XAMPP/xamppfiles/lib/libmysqlclient.18.dylib

/Library/SystemMigration/History/Migration-68137DFB-CB6A-4FBB-81E2-11BDB5D01E48/QuarantineRoot/usr/lib/libmysqlclient.18.dylib

$ sudo install_name_tool -change libmysqlclient.18.dylib /Applications/XAMPP/xamppfiles/lib/libmysqlclient.18.dylib /Users/epallier/Documents/_W_more/PROJECTS/GFT/SOFT/PYROS/pyros/venv_py35_pyros/lib/python3.5/site-packages/_mysql.cpython-35m-darwin.so

Now _mysql.so knows the full path to the library and everything works, regardless of environment variables.

$ otool -L /Users/epallier/Documents/_W_more/PROJECTS/GFT/SOFT/PYROS/pyros/venv_py35_pyros/lib/python3.5/site-packages/_mysql.cpython-35m-darwin.so

/Users/epallier/Documents/_W_more/PROJECTS/GFT/SOFT/PYROS/pyros/venv_py35_pyros/lib/python3.5/site-packages/_mysql.cpython-35m-darwin.so:

	/Applications/XAMPP/xamppfiles/lib/libmysqlclient.18.dylib (compatibility version 18.0.0, current version 18.0.0)

...

$ ./manage.py inspectdb > models.py

=> YES !!!

</pre>

---

h3. Create a Django application pyrosapp

From src/ :

<pre>

$ ./manage.py startapp pyrosapp

</pre>

We obtain this structure:

<pre>

PYROS/

├── REQUIREMENTS.txt

├── private/

│   └── venv_py35_pyros/

├── public/

│   └── static/

├── src/

│   ├── manage.py

│   ├── pyros/

│   │   ├── __init__.py

│   │   ├── __pycache__

│   │   ├── settings.py

│   │   ├── urls.py

│   │   └── wsgi.py

│   └── pyrosapp/

│       ├── __init__.py

│       ├── admin.py

│       ├── apps.py

│       ├── migrations

│       ├── models.py

│       ├── tests.py

│       └── views.py

</pre>

---

h3. Replace the default pyrosapp models.py with the inspectdb generated one

From src/ :

<pre>

$ mv models.py pyrosapp/

</pre>

Add pyrosapp to the project's applications :

Edit src/pyros/settings.py

<pre>

INSTALLED_APPS = [

    'django.contrib.admin',

    'django.contrib.auth',

    'django.contrib.contenttypes',

    'django.contrib.sessions',

    'django.contrib.messages',

    'django.contrib.staticfiles',

    'pyrosapp',

]

</pre>

---

h3. Fix and improve the pyrosapp models.py file (generated by inspectdb)

Once models.py file generated, we need to delete the database and create an empty one :

<pre>

$ mysql -u root [-p (if password needed)]

mysql> DROP DATABSE pyros

mysql> CREATE SCHEMA IF NOT EXISTS 'pyros' DEFAULT CHARACTER SET utf8;

</pre>

Then edit pyrosapp/models.py :

 * Change 'managed = False' to 'managed = True' for every model

 * Change classes names to CamelCase (do not change the 'db_table = ...' lines). *Be careful* : it is needed to change all occurences :

    * NrtAlanysis

    * ScheduleHistory

    * ScientificProgram

    * SequenceType

    * SiteWatch

    * SiteWatchHistory

    * StrategyObs

    * UserLevel

    * WeatherWatch

    * WeatherWatchHistory

 * Change the deleting mode from 'models.DO_NOTHING' to 'models.CASCADE' for the following foreign keys :

    * Image.plan

    * Plan.album

    * Album.sequence

    * Sequence.request

 * Change the 'ForeignKey' liaisons to 'OneToOneField' liaisons (just replace ForeignKey by OneToOneField), and change deleting mode to 'models.CASCADE' for the following foreign keys :

    * Alert.request

    * Detector.device

    * Filter.device

    * Telescope.device

 * We need to redefine many to many relationships for the following classes :

    * User - ScientificProgram :

        * add 'users = models.ManyToManyField('User')' in ScientificProgram class

        * delete UserHasScientificProgram class

    * Sequence - ScheduleHistory

        * add 'sequences = models.ManyToManyField('Sequence')' in ScheduleHistory class

        * delete ScheduleHasSequences class

 * Finally apply modifications to the database :

<pre>

$ pwd

.../PYROS/src

$ python manage.py makemigrations pyrosapp

$ python manage.py migrate

</pre>

---

h2. %{margin-left:0px; font-weight:bold; font-size:25px;  display:block; color:red;}IV - CONFIGURATION of the Django Back Office (administration interface)%

---

h3. Back Office setup

 * Prerequisites in src/pyros/settings.py :

    * INSTALLED_APPS must (at least) contain :

        * django.contrib.admin

        * django.contrib.auth

        * django.contrib.contenttypes

        * django.contrib.sessions

    * MIDDLEWARES must (at least) contain :

        * django.contrib.sessions.middleware.SessionMiddleware

        * django.middleware.common.CommonMiddleware

        * django.contrib.auth.middleware.AuthenticationMiddleware

 * At least one 'python manage.py migrate' must have been executed

 * Create a superuser for the administration :

<pre>

$ python manage.py createsuperuser

</pre>

* For each app of the project, fill the admin.py file :

<pre>

from django.contrib import admin

from app.models import Model1, Model2

admin.site.register(Model1)

admin.site.register(Model2)

</pre>

*Reminder* : each application must be registered in the settings.py INSTALLED_APPS variable.

* For each model in models.py, add a '__str__()' method in order to identify the object on the back office. Example :

<pre>

class UserLevel(models.Model):

    name = models.CharField(max_length=45, blank=True, null=True)

    desc = models.TextField(blank=True, null=True)

    priority = models.IntegerField(blank=True, null=True)

    quota = models.FloatField(blank=True, null=True)

    class Meta:

        managed = True

        db_table = 'userlevel'

    def __str__(self):

        return (str(self.name))

</pre>

*Naming convention* : Use self.name when possible, the creation time/date otherwise. Example :

<pre>

class SiteWatch(models.Model):

    updated = models.DateTimeField(blank=True, null=True)

    lights = models.CharField(max_length=45, blank=True, null=True)

    dome = models.CharField(max_length=45, blank=True, null=True)

    doors = models.CharField(max_length=45, blank=True, null=True)

    temperature = models.FloatField(blank=True, null=True)

    class Meta:

        managed = True

        db_table = 'sitewatch'

    def __str__(self):

        return (str(self.updated))

</pre>

* For each ForeignKey and ManyToManyField creation in models.py, add the 'related_name=[...]' named parameter, as in the following examples :

<pre>

class Sequence(models.Model):

    request = models.ForeignKey(Request, models.CASCADE, related_name="sequences")

    sequencetype = models.ForeignKey('SequenceType', models.DO_NOTHING, related_name="sequences")

    schedule = models.ForeignKey(Schedule, models.DO_NOTHING, related_name="sequences")

    name = models.CharField(max_length=45, blank=True, null=True)

    desc = models.TextField(blank=True, null=True)

    ...

class ScheduleHistory(models.Model):

    sequences = models.ManyToManyField('Sequence', related_name='schedulehistorys')

    created = models.DateTimeField(blank=True, null=True)

    ...

</pre>

h2. %{margin-left:0px; font-weight:bold; font-size:25px;  display:block; color:red;}V - INSTALLATION FROM THE BEGINNING (for dev only)%

h3. How the git repository was created

*Git global setup:*

<pre>

$ git config --global user.name "Etienne Pallier"

$ git config --global user.email "etienne.pallier@irap.omp.eu"

$ cat ~/.gitconfig 

[user]

	name = Etienne Pallier

	email = epallier@irap.omp.eu

[http]

	sslVerify = false

</pre>

*Create a new repository:*

<pre>

$ cd PYROS/

Define files and folders to be ignored:

$ vi .gitignore

.DS_Store

private

__pycache__

$ touch README.md

$ git add README.md

$ git commit -m "first commit"

$ git remote add origin https://gitlab.irap.omp.eu/epallier/pyros.git

$ git push -u origin master

$ git add .

( if you want to be sure to add ALL files: 

$ git add -A

)

( if you wanted to remove added files, just type:

$ git reset HEAD

)

$ git commit -m "first full project commit"

$ git push -u origin master

Counting objects: 43, done.

Delta compression using up to 4 threads.

Compressing objects: 100% (41/41), done.

Writing objects: 100% (43/43), 575.13 KiB ö 0 bytes/s, done.

Total 43 (delta 2), reused 0 (delta 0)

To https://gitlab.irap.omp.eu/epallier/pyros.git

   9c7128c..64501c9  master -> master

Branch master set up to track remote branch master from origin.

$ git status

On branch master

Your branch is up-to-date with 'origin/master'.

nothing to commit, working directory clean

</pre>

h3. Create the database

 * Linux and Mac OS X:

<pre>

One liner:

$ mysql -u root < pyros_create.sql

Or :

$ mysql -u root

mysql> create database pyros;

mysql> use pyros;

mysql> source pyros_create.sql;

(

TODO:

mysql> grant all on pyros.* to pyros@localhost identified by ‘pyros’;)

mysql> flush privileges;

)

</pre>

 * Windows:

<pre>

TODO: Use phpmyadmin ?

</pre>

h3. Create the project structure

<pre>

$ mkdir PYROS

</pre>

Example of a good organization :

1 project = N applis

1 appli = N models

A! L’appli est à côté du projet, PAS DEDANS,

cela facilite la REUTILISATION

(an appli can be part of many projects => reuse)

1 appli = 1 Python module, organized for Django, by default = appli web (but not mandatory)

<pre>

MYPROJECT/

	REQUIREMENTS.txt

	src/

		myproject/

		appli1/

		appli2/

		…

		appliN/

	public/

		static/

	private/

		venv_py35_pyros/

</pre>

Set needed folders:

<pre>

$ cd PYROS/

$ mkdir private public

$ mkdir public/static

</pre>

---

---