History

« Previous - Version 152/350 (diff) - Next » - Current version
Etienne Pallier, 03/21/2016 03:03 pm

Technical Documentation for the PYROS project (FGFT-CC)¶

HOWTO Format Redmine Wiki : http://www.redmine.org/projects/redmine/wiki/FrRedmineWikiFormatting

Technical Documentation for the PYROS project (FGFT-CC)

I - TODO¶

installation sur windows

gitlab

séparation des BD Django et Pyros

intégration dans Eclipse

récupération des modules existants

I - DATABASE SCHEMA (v0.2.1)¶

Error executing the thumbnail macro (Attachment PYROS_PDM_v021.png not found)

II - Get the project (from gitlab)¶

https://gitlab.irap.omp.eu/epallier/pyros

III - INSTALLATION¶

If necessary, install MySql¶

Linux Ubuntu


$ sudo apt-get install mysql-server
$ sudo apt-get install mysql-client

Linux CentOS
```
TODO:
$ sudo yum install mysql
...
```

Mac OS X

TODO:
Install XAMPP
(but you could also use the pre-installed Mac OS MySql)

Windows
```
TODO:
```

Create database¶

Linux and Mac OS X:

One liner:
$ mysql -u root < pyros_create.sql

Or :
$ mysql -u root
mysql> create database pyros;
mysql> use pyros;
mysql> source pyros_create.sql;

(
TODO:
mysql> grant all on pyros.* to pyros@localhost identified by ‘pyros’;)
mysql> flush privileges;
)

Windows:
```
TODO: Use phpmyadmin ?
```

Install python3.5¶

Mac OS X :

1) Installer MacPort
(TODO: doc)

2) Installer le "port" python35
$ sudo port install python35

Linux (Ubuntu) :

sudo add-apt-repository ppa:fkrull/deadsnakes
sudo apt-get update
sudo apt-get install python3.5

sudo pip install virtualenv

Get the project (from git)¶

TODO:

$ git http://pyros... PYROS

This creates a PYROS folder containing the project

Create project structure¶

$ mkdir PYROS

Example of a good organization :

1 project = N applis
(but 1 appli can be part of many projects)
1 appli = 1 Python module, organized for Django, by default = appli web (but not mandatory)
(then 1 appli = N models)


MYPROJECT/

    src/
        myproject/
        appli1/
        appli2/
        …
        appliN/

    public/
        static/

    private/
        REQUIREMENTS.txt
        venv_py35_pyros/

Set needed folders:

$ cd PYROS/
$ mkdir private public
$ mkdir public/static

Create virtualenv with python3.5 dedicated to pyros project (inside the project folder)¶


$ cd private/

$ which python3.5
/opt/local/bin/python3.5

$ virtualenv-3.5 venv_py35_pyros -p /opt/local/bin/python3.5
=> creates a venv_py35_pyros/ folder inside PYROS/private/

Activate the python virtual environment (from inside the project)¶


$ pwd
.../PYROS/private

$ source ./venv_py35_pyros/bin/activate

$ python -V
Python 3.5.1

$ which pip
.../PYROS/venv_py35_pyros/bin/pip

Upgrade pip to last version available :
$ pip install --upgrade pip
Collecting pip
  Downloading pip-8.1.1-py2.py3-none-any.whl (1.2MB)
Installing collected packages: pip
  Found existing installation: pip 7.1.2
    Uninstalling pip-7.1.2:
      Successfully uninstalled pip-7.1.2
Successfully installed pip-8.1.1

Install needed python packages (from within the virtual environment)¶

First, be sure that the virtual environment is activated:

$ python -V
Python 3.5.1

Automatic Installation of all packages
```
$ pip install -r REQUIREMENTS.txt
```

Or, manual installation of each package

Install Django :

$ pip install django
Collecting django
  Downloading Django-1.9.4-py2.py3-none-any.whl (6.6MB)
Installing collected packages: django
Successfully installed django-1.9.4

$ pip install django-admin-tools
Collecting django-admin-tools
  Downloading django_admin_tools-0.7.2-py2.py3-none-any.whl (289kB)
Installing collected packages: django-admin-tools
Successfully installed django-admin-tools-0.7.2

$ pip install django-debug-toolbar
Collecting django-debug-toolbar
  Downloading django_debug_toolbar-1.4-py2.py3-none-any.whl (212kB)
Requirement already satisfied (use --upgrade to upgrade): Django>=1.7 in ./venv_py35_pyros/lib/python3.5/site-packages (from django-debug-toolbar)
Collecting sqlparse (from django-debug-toolbar)
  Downloading sqlparse-0.1.19.tar.gz (58kB)
Building wheels for collected packages: sqlparse
  Running setup.py bdist_wheel for sqlparse ... done
  Stored in directory: /Users/epallier/Library/Caches/pip/wheels/7b/d4/72/6011bb100dd5fc213164e4bbee13d4e03261dd54ce6a5de6b8
Successfully built sqlparse
Installing collected packages: sqlparse, django-debug-toolbar
Successfully installed django-debug-toolbar-1.4 sqlparse-0.1.19

$ pip install django-extensions
Collecting django-extensions
  Downloading django_extensions-1.6.1-py2.py3-none-any.whl (202kB)
Collecting six>=1.2 (from django-extensions)
  Downloading six-1.10.0-py2.py3-none-any.whl
Installing collected packages: six, django-extensions
Successfully installed django-extensions-1.6.1 six-1.10.0

$ pip install django-suit
Collecting django-suit
  Downloading django-suit-0.2.18.tar.gz (587kB)
Building wheels for collected packages: django-suit
  Running setup.py bdist_wheel for django-suit ... done
  Stored in directory: /Users/epallier/Library/Caches/pip/wheels/12/8b/9a/e02ab0ad9229881638aa040d47d77c8f562999533811927d41
Successfully built django-suit
Installing collected packages: django-suit
Successfully installed django-suit-0.2.18

Install the web application server gunicorn (will be used in production instead of the dev django web server) :

$ pip install gunicorn
Collecting gunicorn
  Downloading gunicorn-19.4.5-py2.py3-none-any.whl (112kB)
Installing collected packages: gunicorn
Successfully installed gunicorn-19.4.5

Install the python mysql client:

$ pip install mysqlclient
...

=> Issue under Mac OS X:

$ pip install mysqlclient
Collecting mysqlclient
  Downloading mysqlclient-1.3.7.tar.gz (79kB)
Building wheels for collected packages: mysqlclient
  Running setup.py bdist_wheel for mysqlclient ... error
…
  ----------------------------------------
  Failed building wheel for mysqlclient
  Running setup.py clean for mysqlclient
Failed to build mysqlclient
Installing collected packages: mysqlclient
  Running setup.py install for mysqlclient ... done
Successfully installed mysqlclient-1.3.7

BOUH !!!

$ pip install --upgrade wheel
Collecting wheel
  Downloading wheel-0.29.0-py2.py3-none-any.whl (66kB)
Installing collected packages: wheel
  Found existing installation: wheel 0.24.0
    Uninstalling wheel-0.24.0:
      Successfully uninstalled wheel-0.24.0
Successfully installed wheel-0.29.0

$ pip uninstall mysqlclient

$ pip install mysqlclient
Collecting mysqlclient
  Using cached mysqlclient-1.3.7.tar.gz
Building wheels for collected packages: mysqlclient
  Running setup.py bdist_wheel for mysqlclient ... done
  Stored in directory: /Users/epallier/Library/Caches/pip/wheels/9b/06/50/d11418c26cf8f2156b13d4363b5afde8e7e75ebb8540d0228d
Successfully built mysqlclient
Installing collected packages: mysqlclient
Successfully installed mysqlclient-1.3.7

YES !!!

=> Issues under Ubuntu:

$ pip install mysqlclient
Collecting mysqlclient
  Downloading mysqlclient-1.3.7.tar.gz (79kB)
    100% |████████████████████████████████| 81kB 1.5MB/s
    Complete output from command python setup.py egg_info:
    /bin/sh: 1: mysql_config: not found
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
    [...]
    ----------------------------------------
Command "python setup.py egg_info" failed with error code 1 in /tmp/pip-build-q6j4inuz/mysqlclient/

BOUH !!!

$ sudo apt-get install libmysqlclient-dev

$ pip install mysqlclient
Collecting mysqlclient
  Using cached mysqlclient-1.3.7.tar.gz
Building wheels for collected packages: mysqlclient
  Running setup.py bdist_wheel for mysqlclient ... error
…
    _mysql.c:40:20: fatal error: Python.h: No such file or directory
     #include "Python.h" 
                        ^
    compilation terminated.
    error: command 'x86_64-linux-gnu-gcc' failed with exit status 1

    ----------------------------------------
Command "/home/carens_p/pyros/venv_py35_pyros/bin/python3.5 -u -c "import setuptools, tokenize;__file__='/tmp/pip-build-k3klv92j/mysqlclient/setup.py';exec(compile(getattr(tokenize, 'open', open)(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" install --record /tmp/pip-gz242xxs-record/install-record.txt --single-version-externally-managed --compile --install-headers /home/carens_p/pyros/venv_py35_pyros/include/site/python3.5/mysqlclient" failed with error code 1 in /tmp/pip-build-k3klv92j/mysqlclient/

BOUH !!!

$ sudo apt-get install python3.5-dev

$ pip install mysqlclient

YES !!!

Set Requirements

$ pip freeze > REQUIREMENTS.txt

$ mv REQUIREMENTS.txt .../PYROS/private/

Create Django project pyros¶


From inside the project:
$ pwd
.../PYROS/

$ django-admin startproject pyros

Rename the project folder "pyros/" as "src/" 

$ mv pyros src

We have then this architecture:

PYROS
├── private
│   └── REQUIREMENTS.txt
│   └── venv_py35_pyros
├── public
│   └── static
├── src
│   ├── manage.py
│   ├── pyros
│   │   ├── __init__.py
│   │   ├── settings.py
│   │   ├── urls.py
│   │   └── wsgi.py

Test the project¶


$ cd src/

$ ./manage.py runserver
(or gunicorn pyros.wsgi)
==> http://localhost:8000
...
...
Ctrl-c

The Web server¶

Apache : gère tous les fichiers statiques (images, html…), et délègue les fichiers python au serveur django (par défaut)

Le fichier pyros/urls.py prend le relai pour tout ce qui est django

Le moteur web django sera soit du wsgi soit du unicorn

Par défaut, 1 seul worker, mais on peut en configurer plusieurs, l’idéal étant de faire "nb coeurs + 1"
(le worker maître qui fait le dispatching aux autres)

Frontend : Apache ou Ngininx

Backend : gunicorn (gère facilement des workers) ou uwsgi

$ gunicorn pyros.wsgi
(à la place de manage runserver => A EVITER EN PROD)

Ou encore:

$ gunicorn --workers 5 library.wsgi

Set Database engine as MySql¶

Edit src/pyros/settings.py

DATABASES = {
    'default': {
        'ENGINE': 'django.db.backends.mysql',
        'NAME': 'pyros',
        'USER': 'root',
        'PASSWORD': ''
    }
}

Import database into Django (with inspectdb)¶

From src/ :

$ ./manage.py inspectdb > models.py

Issue on Mac OS X:

Traceback (most recent call last):
  File "/Users/epallier/Documents/_W_more/PROJECTS/GFT/SOFT/PYROS/pyros/venv_py35_pyros/lib/python3.5/site-packages/django/db/backends/mysql/base.py", line 25, in <module>
    import MySQLdb as Database
  File "/Users/epallier/Documents/_W_more/PROJECTS/GFT/SOFT/PYROS/pyros/venv_py35_pyros/lib/python3.5/site-packages/MySQLdb/__init__.py", line 19, in <module>
    import _mysql
ImportError: dlopen(/Users/epallier/Documents/_W_more/PROJECTS/GFT/SOFT/PYROS/pyros/venv_py35_pyros/lib/python3.5/site-packages/_mysql.cpython-35m-darwin.so, 2): Library not loaded: libmysqlclient.18.dylib
  Referenced from: /Users/epallier/Documents/_W_more/PROJECTS/GFT/SOFT/PYROS/pyros/venv_py35_pyros/lib/python3.5/site-packages/_mysql.cpython-35m-darwin.so
  Reason: image not found

=> BOUH !!!

LA SOLUTION EST ICI : http://stackoverflow.com/questions/6383310/python-mysqldb-library-not-loaded-libmysqlclient-18-dylib

Il suffit de faire ceci:

$ sudo mkdir -p /usr/local/lib   
$ sudo ln -s /Applications/XAMPP/xamppfiles/lib/libmysql* /usr/local/lib/

Mais on peut aussi faire ceci:

Okay, so the offending file is /Users/epallier/Documents/_W_more/PROJECTS/GFT/SOFT/PYROS/pyros/venv_py35_pyros/lib/python3.5/site-packages/_mysql.cpython-35m-darwin.so

Next, figure out where _mysql.so thinks it should find libmysqlclient.18.dylib:

$ otool -L /Users/epallier/Documents/_W_more/PROJECTS/GFT/SOFT/PYROS/pyros/venv_py35_pyros/lib/python3.5/site-packages/_mysql.cpython-35m-darwin.so
/Users/epallier/Documents/_W_more/PROJECTS/GFT/SOFT/PYROS/pyros/venv_py35_pyros/lib/python3.5/site-packages/_mysql.cpython-35m-darwin.so:
    libmysqlclient.18.dylib (compatibility version 18.0.0, current version 18.0.0)
...

So, it's looking for libmysqlclient.18.dylib with no path information, let's fix that:

$ locate libmysqlclient.18.dylib
/Applications/XAMPP/xamppfiles/lib/libmysqlclient.18.dylib
/Library/SystemMigration/History/Migration-68137DFB-CB6A-4FBB-81E2-11BDB5D01E48/QuarantineRoot/usr/lib/libmysqlclient.18.dylib

$ sudo install_name_tool -change libmysqlclient.18.dylib /Applications/XAMPP/xamppfiles/lib/libmysqlclient.18.dylib /Users/epallier/Documents/_W_more/PROJECTS/GFT/SOFT/PYROS/pyros/venv_py35_pyros/lib/python3.5/site-packages/_mysql.cpython-35m-darwin.so

Now _mysql.so knows the full path to the library and everything works, regardless of environment variables.

$ otool -L /Users/epallier/Documents/_W_more/PROJECTS/GFT/SOFT/PYROS/pyros/venv_py35_pyros/lib/python3.5/site-packages/_mysql.cpython-35m-darwin.so
/Users/epallier/Documents/_W_more/PROJECTS/GFT/SOFT/PYROS/pyros/venv_py35_pyros/lib/python3.5/site-packages/_mysql.cpython-35m-darwin.so:
    /Applications/XAMPP/xamppfiles/lib/libmysqlclient.18.dylib (compatibility version 18.0.0, current version 18.0.0)
...

$ ./manage.py inspectdb > models.py

=> YES !!!

Create a Django application pyrosapp¶

From src/ :

$ ./manage.py startapp pyrosapp

We obtain this structure:


PYROS/
├── private
│   └── REQUIREMENTS.txt
├── public
│   └── static
├── src
│   ├── db.sqlite3
│   ├── manage.py
│   ├── pyros
│   │   ├── __init__.py
│   │   ├── __pycache__
│   │   ├── settings.py
│   │   ├── urls.py
│   │   └── wsgi.py
│   └── pyrosapp
│       ├── __init__.py
│       ├── admin.py
│       ├── apps.py
│       ├── migrations
│       ├── models.py
│       ├── tests.py
│       └── views.py
└── venv_py35_pyros

Replace the default pyrosapp models.py with the inspectdb generated one¶

From src/ :

$ mv models.py pyrosapp/

Add pyrosapp to the project's applications :

Edit src/pyros/settings.py

INSTALLED_APPS = [
    'django.contrib.admin',
    'django.contrib.auth',
    'django.contrib.contenttypes',
    'django.contrib.sessions',
    'django.contrib.messages',
    'django.contrib.staticfiles',
    'pyrosapp',
]

Fix and improve the pyrosapp models.py file (generated by inspectdb)¶

Once models.py file generated, we need to delete the database and create an empty one :

$ mysql -u root [-p (if password needed)]

mysql> DROP DATABSE pyros
mysql> CREATE SCHEMA IF NOT EXISTS 'pyros' DEFAULT CHARACTER SET utf8;

Then edit pyrosapp/models.py :

Change 'managed = False' to 'managed = True' for every model

Change classes names to CamelCase (do not change the 'db_table = ...' lines). Be careful : it is needed to change all occurences :
- NrtAlanysis
- ScheduleHistory
- ScientificProgram
- SequenceType
- SiteWatch
- SiteWatchHistory
- StrategyObs
- WeatherWatch
- WeatherWatchHistory

Change the deleting mode from 'models.DO_NOTHING' to 'models.CASCADE' for the following foreign keys :
- Image.plan
- Plan.album
- Album.sequence
- Sequence.request

Change the 'ForeignKey' liaisons to 'OneToOneField' liaisons (just replace ForeignKey by OneToOneField), and change deleting mode to 'models.CASCADE' for the following foreign keys :
- Alert.request
- Detector.device
- Filter.device
- Telescope.device

We need to redefine many to many relationships for the following classes :
- User - ScientificProgram :
  - add 'users = models.ManyToManyField('User')' in ScientificProgram class
  - delete UserHasScientificProgram class
- Sequence - ScheduleHistory
  - add 'sequences = models.ManyToManyField('Sequence')' in ScheduleHistory class
  - delete ScheduleHasSequences class

Finally apply modifications to the database :

$ pwd
.../PYROS/src
$ python manage.py makemigrations pyrosapp
$ python manage.py migrate

II - CONFIGURATION of the Django Back Office (administration interface)¶

pyros intro.pdf (77.7 KB) Etienne Pallier, 02/06/2019 05:04 pm

pyros specs.pdf (1.8 MB) Etienne Pallier, 02/06/2019 05:04 pm

pyros dev-guide.pdf (649 KB) Etienne Pallier, 02/06/2019 05:04 pm