History

PYROS INSTALLATION FROM THE BEGINNING (for dev only, history of the initial project creation)¶

PYROS INSTALLATION FROM THE BEGINNING (for dev only, history of the initial project creation)

Install MySql (only if necessary)¶

See in III-Installation
https://projects.irap.omp.eu/projects/pyros/wiki/Wiki#III-INSTALLATION

Create the database (from sql script generated with Mysql Workbench)¶

All platforms:

One liner:
$ mysql -u root < pyros_create.sql

Or :
$ mysql -u root
mysql> create database pyros;
mysql> use pyros;
mysql> source pyros_create.sql;

(
TODO:
mysql> grant all on pyros.* to pyros@localhost identified by ‘pyros’;)
mysql> flush privileges;
)

Install Python3.5 (only if necessary)¶

See in III-Installation
https://projects.irap.omp.eu/projects/pyros/wiki/Wiki#III-INSTALLATION

Create the project structure¶

$ mkdir PYROS

Example of a good organization :

1 project = N applis
1 appli = N models
A! L’appli est à côté du projet, PAS DEDANS,
cela facilite la REUTILISATION
(an appli can be part of many projects => reuse)
1 appli = 1 Python module, organized for Django, by default = appli web (but not mandatory)


MYPROJECT/
    REQUIREMENTS.txt
    src/
        myproject/
        appli1/
        appli2/
        …
        appliN/

    public/
        static/

    private/
        venv_py35_pyros/

Set needed folders:

$ cd PYROS/
$ mkdir private public
$ mkdir public/static

Create a Python3 virtual environment dedicated to the project (inside the project folder)¶

See in III-Installation
https://projects.irap.omp.eu/projects/pyros/wiki/Wiki#III-INSTALLATION

Activate the Python virtual environment (from inside the project)¶

See in III-Installation
https://projects.irap.omp.eu/projects/pyros/wiki/Wiki#III-INSTALLATION

Install the needed Python packages (from within the virtual environment)¶

See in III-Installation
https://projects.irap.omp.eu/projects/pyros/wiki/Wiki#Install-the-needed-Python-packages-from-within-the-virtual-environment
(cf manual installation)

Set Requirements¶

$ pip freeze > REQUIREMENTS.txt

Create a Django project named PYROS¶


From inside the project:
$ pwd
.../PYROS/

$ django-admin startproject pyros

Rename the project folder "pyros/" as "src/" 

$ mv pyros src

We have then this architecture:

PYROS
├── REQUIREMENTS.txt
├── private
│   └── venv_py35_pyros
├── public
│   └── static
├── src
│   ├── manage.py
│   ├── pyros
│   │   ├── __init__.py
│   │   ├── settings.py
│   │   ├── urls.py
│   │   └── wsgi.py

Test the project¶

See in III-Installation
https://projects.irap.omp.eu/projects/pyros/wiki/Wiki#III-INSTALLATION

The Web server¶

Apache : gère tous les fichiers statiques (images, html…), et délègue les fichiers python au serveur django (par défaut)

Le fichier pyros/urls.py prend le relai pour tout ce qui est django

Le moteur web django sera soit du wsgi soit du unicorn

Par défaut, 1 seul worker, mais on peut en configurer plusieurs, l’idéal étant de faire "nb coeurs + 1"
(le worker maître qui fait le dispatching aux autres)

Frontend : Apache ou Ngininx

Backend : gunicorn (gère facilement des workers) ou uwsgi

$ gunicorn pyros.wsgi
(à la place de manage runserver => A EVITER EN PROD)

Ou encore:

$ gunicorn --workers 5 library.wsgi

Set the database engine as MySql¶

Edit src/pyros/settings.py

DATABASES = {
    'default': {
        'ENGINE': 'django.db.backends.mysql',
        'NAME': 'pyros',
        'USER': 'root',
        'PASSWORD': ''
    }
}

Import the database into Django (with inspectdb)¶

From src/ :

$ ./manage.py inspectdb > models.py

Issue on Mac OS X:

Traceback (most recent call last):
  File "/Users/epallier/Documents/_W_more/PROJECTS/GFT/SOFT/PYROS/pyros/venv_py35_pyros/lib/python3.5/site-packages/django/db/backends/mysql/base.py", line 25, in <module>
    import MySQLdb as Database
  File "/Users/epallier/Documents/_W_more/PROJECTS/GFT/SOFT/PYROS/pyros/venv_py35_pyros/lib/python3.5/site-packages/MySQLdb/__init__.py", line 19, in <module>
    import _mysql
ImportError: dlopen(/Users/epallier/Documents/_W_more/PROJECTS/GFT/SOFT/PYROS/pyros/venv_py35_pyros/lib/python3.5/site-packages/_mysql.cpython-35m-darwin.so, 2): Library not loaded: libmysqlclient.18.dylib
  Referenced from: /Users/epallier/Documents/_W_more/PROJECTS/GFT/SOFT/PYROS/pyros/venv_py35_pyros/lib/python3.5/site-packages/_mysql.cpython-35m-darwin.so
  Reason: image not found

=> BOUH !!!

LA SOLUTION EST ICI : http://stackoverflow.com/questions/6383310/python-mysqldb-library-not-loaded-libmysqlclient-18-dylib

Il suffit de faire ceci:

$ sudo mkdir -p /usr/local/lib   
$ sudo ln -s /Applications/XAMPP/xamppfiles/lib/libmysql* /usr/local/lib/

Mais on peut aussi faire ceci:

Okay, so the offending file is /Users/epallier/Documents/_W_more/PROJECTS/GFT/SOFT/PYROS/pyros/venv_py35_pyros/lib/python3.5/site-packages/_mysql.cpython-35m-darwin.so

Next, figure out where _mysql.so thinks it should find libmysqlclient.18.dylib:

$ otool -L /Users/epallier/Documents/_W_more/PROJECTS/GFT/SOFT/PYROS/pyros/venv_py35_pyros/lib/python3.5/site-packages/_mysql.cpython-35m-darwin.so
/Users/epallier/Documents/_W_more/PROJECTS/GFT/SOFT/PYROS/pyros/venv_py35_pyros/lib/python3.5/site-packages/_mysql.cpython-35m-darwin.so:
    libmysqlclient.18.dylib (compatibility version 18.0.0, current version 18.0.0)
...

So, it's looking for libmysqlclient.18.dylib with no path information, let's fix that:

$ locate libmysqlclient.18.dylib
/Applications/XAMPP/xamppfiles/lib/libmysqlclient.18.dylib
/Library/SystemMigration/History/Migration-68137DFB-CB6A-4FBB-81E2-11BDB5D01E48/QuarantineRoot/usr/lib/libmysqlclient.18.dylib

$ sudo install_name_tool -change libmysqlclient.18.dylib /Applications/XAMPP/xamppfiles/lib/libmysqlclient.18.dylib /Users/epallier/Documents/_W_more/PROJECTS/GFT/SOFT/PYROS/pyros/venv_py35_pyros/lib/python3.5/site-packages/_mysql.cpython-35m-darwin.so

Now _mysql.so knows the full path to the library and everything works, regardless of environment variables.

$ otool -L /Users/epallier/Documents/_W_more/PROJECTS/GFT/SOFT/PYROS/pyros/venv_py35_pyros/lib/python3.5/site-packages/_mysql.cpython-35m-darwin.so
/Users/epallier/Documents/_W_more/PROJECTS/GFT/SOFT/PYROS/pyros/venv_py35_pyros/lib/python3.5/site-packages/_mysql.cpython-35m-darwin.so:
    /Applications/XAMPP/xamppfiles/lib/libmysqlclient.18.dylib (compatibility version 18.0.0, current version 18.0.0)
...

$ ./manage.py inspectdb > models.py

=> YES !!!

Create a Django application named "pyrosapp"¶

From src/ :

$ ./manage.py startapp pyrosapp

We obtain this structure:


PYROS/
├── REQUIREMENTS.txt
├── private/
│   └── venv_py35_pyros/
├── public/
│   └── static/
├── src/
│   ├── manage.py
│   ├── pyros/
│   │   ├── __init__.py
│   │   ├── __pycache__
│   │   ├── settings.py
│   │   ├── urls.py
│   │   └── wsgi.py
│   └── pyrosapp/
│       ├── __init__.py
│       ├── admin.py
│       ├── apps.py
│       ├── migrations
│       ├── models.py
│       ├── tests.py
│       └── views.py

Replace the default pyrosapp models.py with the inspectdb generated one¶

From src/ :

$ mv models.py pyrosapp/

Add pyrosapp to the project's applications :

Edit src/pyros/settings.py

INSTALLED_APPS = [
    'django.contrib.admin',
    'django.contrib.auth',
    'django.contrib.contenttypes',
    'django.contrib.sessions',
    'django.contrib.messages',
    'django.contrib.staticfiles',
    'pyrosapp',
]

Initialize the git repository¶

Git global setup:

$ git config --global user.name "Etienne Pallier" 
$ git config --global user.email "etienne.pallier@irap.omp.eu" 

$ cat ~/.gitconfig 
[user]
    name = Etienne Pallier
    email = epallier@irap.omp.eu
[http]
    sslVerify = false

Create a new repository:

$ cd PYROS/

Define files and folders to be ignored:
$ vi .gitignore
.DS_Store
private
__pycache__

$ touch README.md

$ git add README.md

$ git commit -m "first commit" 

$ git remote add origin https://gitlab.irap.omp.eu/epallier/pyros.git

$ git push -u origin master

$ git add .

( if you want to be sure to add ALL files: 
$ git add -A
)

( if you wanted to remove added files, just type:
$ git reset HEAD
)

$ git commit -m "first full project commit" 

$ git push -u origin master
Counting objects: 43, done.
Delta compression using up to 4 threads.
Compressing objects: 100% (41/41), done.
Writing objects: 100% (43/43), 575.13 KiB ö 0 bytes/s, done.
Total 43 (delta 2), reused 0 (delta 0)
To https://gitlab.irap.omp.eu/epallier/pyros.git
   9c7128c..64501c9  master -> master
Branch master set up to track remote branch master from origin.

$ git status
On branch master
Your branch is up-to-date with 'origin/master'.
nothing to commit, working directory clean

Fix and improve the pyrosapp models.py file (generated by inspectdb)¶

Edit pyrosapp/models.py :

Change 'managed = False' to 'managed = True' for every model

Change classes names to CamelCase (do not change the 'db_table = ...' lines). Be careful : it is required to change all occurences :
- NrtAlanysis
- ScheduleHistory
- ScientificProgram
- SequenceType
- SiteWatch
- SiteWatchHistory
- StrategyObs
- UserLevel
- WeatherWatch
- WeatherWatchHistory

Change the deleting mode from 'models.DO_NOTHING' to 'models.CASCADE' for the following foreign keys :
- Image.plan
- Plan.album
- Album.sequence
- Sequence.request

Change the 'ForeignKey' links to 'OneToOneField' links (just replace ForeignKey by OneToOneField), and change deleting mode to 'models.CASCADE' for the following foreign keys :
- Alert.request
- Detector.device
- Filter.device
- Telescope.device

We need to redefine many to many relationships for the following classes :
- User - ScientificProgram :
  - add 'users = models.ManyToManyField('User')' in ScientificProgram class
  - delete UserHasScientificProgram class
- Sequence - ScheduleHistory
  - add 'sequences = models.ManyToManyField('Sequence')' in ScheduleHistory class
  - delete ScheduleHasSequences class

For each ForeignKey and ManyToManyField creation in models.py, add the 'related_name=[...]' named parameter, as in the following examples :

class Sequence(models.Model):
    request = models.ForeignKey(Request, models.CASCADE, related_name="sequences")
    sequencetype = models.ForeignKey('SequenceType', models.DO_NOTHING, related_name="sequences")
    schedule = models.ForeignKey(Schedule, models.DO_NOTHING, related_name="sequences")
    name = models.CharField(max_length=45, blank=True, null=True)
    desc = models.TextField(blank=True, null=True)
    ...

class ScheduleHistory(models.Model):
    sequences = models.ManyToManyField('Sequence', related_name='schedulehistories')
    created = models.DateTimeField(blank=True, null=True)
    ...

Finally apply modifications to the database :

First, we need to delete the database and create an empty one :

$ mysql -u root [-p (if password needed)]

mysql> DROP DATABSE pyros;
mysql> CREATE DATABASE pyros;

(or:
mysql> CREATE SCHEMA IF NOT EXISTS 'pyros' DEFAULT CHARACTER SET utf8;
)

Then sync the database from Django models:

$ pwd
.../PYROS/src
$ python manage.py makemigrations pyrosapp
$ python manage.py migrate

CONFIGURATION of the Django Back Office (administration interface)¶

cf https://projects.irap.omp.eu/projects/pyros/wiki/Wiki#IV-CONFIGURATION-of-the-Django-Back-Office-administration-interface