Wiki
Version 190 (Etienne Pallier, 03/22/2016 10:16 am) → Version 191/350 (Etienne Pallier, 03/22/2016 10:17 am)
h1. Technical Documentation for the PYROS project (FGFT-CC)
HOWTO Format Redmine Wiki : http://www.redmine.org/projects/redmine/wiki/FrRedmineWikiFormatting
{{>toc}}
---
h2. %{margin-left:0px; font-weight:bold; font-size:25px; display:block; color:red;}I - TODO%
* installation sur windows
* gitlab
* séparation des BD Django et Pyros
* intégration dans Eclipse
* Intégration des modules Django déjà développés
---
h2. %{margin-left:0px; font-weight:bold; font-size:25px; display:block; color:red;}I - DATABASE SCHEMA (v0.2.1)%
{{thumbnail(PYROS_PDM_v021.png, size=300, title=Pyros data model)}}
---
h2. %{margin-left:0px; font-weight:bold; font-size:25px; display:block; color:red;}II - Get the project (from gitlab)%
https://gitlab.irap.omp.eu/epallier/pyros
https://gitlab.irap.omp.eu/epallier/pyros.git
ssh git@gitlab.irap.omp.eu:epallier/pyros.git
h3. Get the project from the terminal
<pre>
git clone https://gitlab.irap.omp.eu/epallier/pyros.git PYROS
(or also : git clone git@gitlab.irap.omp.eu:epallier/pyros.git)
</pre>
If you just want a static copy of the project (without synchronization) just remove the .git/ folder :
<pre>
$ rm -r .git/
</pre>
h3. Get the project from Eclipse
TODO:
h2. %{margin-left:0px; font-weight:bold; font-size:25px; display:block; color:red;}III - INSTALLATION%
---
h3. Install MySql (only if necessary)
* Linux Ubuntu
<pre>
$ sudo apt-get install mysql-server
$ sudo apt-get install mysql-client
</pre>
* Linux CentOS
<pre>
TODO:
$ sudo yum install mysql
...
</pre>
* Mac OS X
Install XAMPP
(but you could also use the pre-installed Mac OS MySql)
<pre>
TODO:
</pre>
* Windows
Install XAMPP
<pre>
TODO:
</pre>
---
h3. Install Python3.5 (only if necessary)
* Mac OS X :
<pre>
1) Installer MacPort
(TODO: doc)
2) Installer le "port" python35
$ sudo port install python35
</pre>
* Linux (Ubuntu) :
<pre>
sudo add-apt-repository ppa:fkrull/deadsnakes
sudo apt-get update
sudo apt-get install python3.5
sudo pip install virtualenv
</pre>
* Windows 10 :
<pre>
TODO:
</pre>
---
h3. Get the project (from git)
<pre>
$ git clone https://gitlab.irap.omp.eu/epallier/pyros.git PYROS
</pre>
This creates a PYROS/ folder containing the project (with a .git/ subfolder for synchronization with the git repository)
---
h3. Create virtualenv with python3.5 dedicated to pyros project (inside the project folder)
<pre>
$ cd private/
$ which python3.5
/opt/local/bin/python3.5
$ virtualenv-3.5 venv_py35_pyros -p /opt/local/bin/python3.5
=> creates a venv_py35_pyros/ folder inside PYROS/private/
</pre>
---
h3. Activate the python virtual environment (from inside the project)
<pre>
$ pwd
.../PYROS/private
$ source ./venv_py35_pyros/bin/activate
$ python -V
Python 3.5.1
$ which pip
.../PYROS/venv_py35_pyros/bin/pip
Upgrade pip to last version available :
$ pip install --upgrade pip
Collecting pip
Downloading pip-8.1.1-py2.py3-none-any.whl (1.2MB)
Installing collected packages: pip
Found existing installation: pip 7.1.2
Uninstalling pip-7.1.2:
Successfully uninstalled pip-7.1.2
Successfully installed pip-8.1.1
</pre>
h3. Install needed python packages (from within the virtual environment)
First, be sure that the virtual environment is activated:
<pre>
$ python -V
Python 3.5.1
</pre>
* *Automatic Installation of all packages*
<pre>
$ pip install -r REQUIREMENTS.txt
</pre>
* *Or, manual installation of each package*
* *Install Django* :
<pre>
$ pip install django
Collecting django
Downloading Django-1.9.4-py2.py3-none-any.whl (6.6MB)
Installing collected packages: django
Successfully installed django-1.9.4
$ pip install django-admin-tools
Collecting django-admin-tools
Downloading django_admin_tools-0.7.2-py2.py3-none-any.whl (289kB)
Installing collected packages: django-admin-tools
Successfully installed django-admin-tools-0.7.2
$ pip install django-debug-toolbar
Collecting django-debug-toolbar
Downloading django_debug_toolbar-1.4-py2.py3-none-any.whl (212kB)
Requirement already satisfied (use --upgrade to upgrade): Django>=1.7 in ./venv_py35_pyros/lib/python3.5/site-packages (from django-debug-toolbar)
Collecting sqlparse (from django-debug-toolbar)
Downloading sqlparse-0.1.19.tar.gz (58kB)
Building wheels for collected packages: sqlparse
Running setup.py bdist_wheel for sqlparse ... done
Stored in directory: /Users/epallier/Library/Caches/pip/wheels/7b/d4/72/6011bb100dd5fc213164e4bbee13d4e03261dd54ce6a5de6b8
Successfully built sqlparse
Installing collected packages: sqlparse, django-debug-toolbar
Successfully installed django-debug-toolbar-1.4 sqlparse-0.1.19
$ pip install django-extensions
Collecting django-extensions
Downloading django_extensions-1.6.1-py2.py3-none-any.whl (202kB)
Collecting six>=1.2 (from django-extensions)
Downloading six-1.10.0-py2.py3-none-any.whl
Installing collected packages: six, django-extensions
Successfully installed django-extensions-1.6.1 six-1.10.0
$ pip install django-suit
Collecting django-suit
Downloading django-suit-0.2.18.tar.gz (587kB)
Building wheels for collected packages: django-suit
Running setup.py bdist_wheel for django-suit ... done
Stored in directory: /Users/epallier/Library/Caches/pip/wheels/12/8b/9a/e02ab0ad9229881638aa040d47d77c8f562999533811927d41
Successfully built django-suit
Installing collected packages: django-suit
Successfully installed django-suit-0.2.18
</pre>
* *Install the web application server gunicorn (will be used in production instead of the dev django web server)* :
<pre>
$ pip install gunicorn
Collecting gunicorn
Downloading gunicorn-19.4.5-py2.py3-none-any.whl (112kB)
Installing collected packages: gunicorn
Successfully installed gunicorn-19.4.5
</pre>
* *Install the python mysql client*:
<pre>
$ pip install mysqlclient
...
</pre>
* => Issue under Mac OS X:
<pre>
$ pip install mysqlclient
Collecting mysqlclient
Downloading mysqlclient-1.3.7.tar.gz (79kB)
Building wheels for collected packages: mysqlclient
Running setup.py bdist_wheel for mysqlclient ... error
…
----------------------------------------
Failed building wheel for mysqlclient
Running setup.py clean for mysqlclient
Failed to build mysqlclient
Installing collected packages: mysqlclient
Running setup.py install for mysqlclient ... done
Successfully installed mysqlclient-1.3.7
BOUH !!!
$ pip install --upgrade wheel
Collecting wheel
Downloading wheel-0.29.0-py2.py3-none-any.whl (66kB)
Installing collected packages: wheel
Found existing installation: wheel 0.24.0
Uninstalling wheel-0.24.0:
Successfully uninstalled wheel-0.24.0
Successfully installed wheel-0.29.0
$ pip uninstall mysqlclient
$ pip install mysqlclient
Collecting mysqlclient
Using cached mysqlclient-1.3.7.tar.gz
Building wheels for collected packages: mysqlclient
Running setup.py bdist_wheel for mysqlclient ... done
Stored in directory: /Users/epallier/Library/Caches/pip/wheels/9b/06/50/d11418c26cf8f2156b13d4363b5afde8e7e75ebb8540d0228d
Successfully built mysqlclient
Installing collected packages: mysqlclient
Successfully installed mysqlclient-1.3.7
YES !!!
</pre>
* => Issues under Ubuntu:
<pre>
$ pip install mysqlclient
Collecting mysqlclient
Downloading mysqlclient-1.3.7.tar.gz (79kB)
100% |████████████████████████████████| 81kB 1.5MB/s
Complete output from command python setup.py egg_info:
/bin/sh: 1: mysql_config: not found
Traceback (most recent call last):
File "<string>", line 1, in <module>
[...]
----------------------------------------
Command "python setup.py egg_info" failed with error code 1 in /tmp/pip-build-q6j4inuz/mysqlclient/
BOUH !!!
$ sudo apt-get install libmysqlclient-dev
$ pip install mysqlclient
Collecting mysqlclient
Using cached mysqlclient-1.3.7.tar.gz
Building wheels for collected packages: mysqlclient
Running setup.py bdist_wheel for mysqlclient ... error
…
_mysql.c:40:20: fatal error: Python.h: No such file or directory
#include "Python.h"
^
compilation terminated.
error: command 'x86_64-linux-gnu-gcc' failed with exit status 1
----------------------------------------
Command "/home/carens_p/pyros/venv_py35_pyros/bin/python3.5 -u -c "import setuptools, tokenize;__file__='/tmp/pip-build-k3klv92j/mysqlclient/setup.py';exec(compile(getattr(tokenize, 'open', open)(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" install --record /tmp/pip-gz242xxs-record/install-record.txt --single-version-externally-managed --compile --install-headers /home/carens_p/pyros/venv_py35_pyros/include/site/python3.5/mysqlclient" failed with error code 1 in /tmp/pip-build-k3klv92j/mysqlclient/
BOUH !!!
$ sudo apt-get install python3.5-dev
$ pip install mysqlclient
YES !!!
</pre>
* *Set Requirements*
<pre>
$ pip freeze > REQUIREMENTS.txt
</pre>
---
h3. Create Django project pyros
<pre>
From inside the project:
$ pwd
.../PYROS/
$ django-admin startproject pyros
Rename the project folder "pyros/" as "src/"
$ mv pyros src
We have then this architecture:
PYROS
├── REQUIREMENTS.txt
├── private
│ └── venv_py35_pyros
├── public
│ └── static
├── src
│ ├── manage.py
│ ├── pyros
│ │ ├── __init__.py
│ │ ├── settings.py
│ │ ├── urls.py
│ │ └── wsgi.py
</pre>
---
h3. Test the project
<pre>
$ cd src/
$ ./manage.py runserver
(or gunicorn pyros.wsgi)
==> http://localhost:8000
...
...
Ctrl-c
</pre>
---
h3. The Web server
Apache : gère tous les fichiers statiques (images, html…), et délègue les fichiers python au serveur django (par défaut)
Le fichier pyros/urls.py prend le relai pour tout ce qui est django
Le moteur web django sera soit du wsgi soit du unicorn
Par défaut, 1 seul worker, mais on peut en configurer plusieurs, l’idéal étant de faire "nb coeurs + 1"
(le worker maître qui fait le dispatching aux autres)
Frontend : Apache ou Ngininx
Backend : gunicorn (gère facilement des workers) ou uwsgi
<pre>
$ gunicorn pyros.wsgi
(à la place de manage runserver => A EVITER EN PROD)
Ou encore:
$ gunicorn --workers 5 library.wsgi
</pre>
---
h3. Set Database engine as MySql
Edit src/pyros/settings.py
<pre>
DATABASES = {
'default': {
'ENGINE': 'django.db.backends.mysql',
'NAME': 'pyros',
'USER': 'root',
'PASSWORD': ''
}
}
</pre>
---
h3. Import database into Django (with inspectdb)
From src/ :
<pre>
$ ./manage.py inspectdb > models.py
</pre>
Issue on Mac OS X:
<pre>
Traceback (most recent call last):
File "/Users/epallier/Documents/_W_more/PROJECTS/GFT/SOFT/PYROS/pyros/venv_py35_pyros/lib/python3.5/site-packages/django/db/backends/mysql/base.py", line 25, in <module>
import MySQLdb as Database
File "/Users/epallier/Documents/_W_more/PROJECTS/GFT/SOFT/PYROS/pyros/venv_py35_pyros/lib/python3.5/site-packages/MySQLdb/__init__.py", line 19, in <module>
import _mysql
ImportError: dlopen(/Users/epallier/Documents/_W_more/PROJECTS/GFT/SOFT/PYROS/pyros/venv_py35_pyros/lib/python3.5/site-packages/_mysql.cpython-35m-darwin.so, 2): Library not loaded: libmysqlclient.18.dylib
Referenced from: /Users/epallier/Documents/_W_more/PROJECTS/GFT/SOFT/PYROS/pyros/venv_py35_pyros/lib/python3.5/site-packages/_mysql.cpython-35m-darwin.so
Reason: image not found
=> BOUH !!!
LA SOLUTION EST ICI : http://stackoverflow.com/questions/6383310/python-mysqldb-library-not-loaded-libmysqlclient-18-dylib
Il suffit de faire ceci:
$ sudo mkdir -p /usr/local/lib
$ sudo ln -s /Applications/XAMPP/xamppfiles/lib/libmysql* /usr/local/lib/
Mais on peut aussi faire ceci:
Okay, so the offending file is /Users/epallier/Documents/_W_more/PROJECTS/GFT/SOFT/PYROS/pyros/venv_py35_pyros/lib/python3.5/site-packages/_mysql.cpython-35m-darwin.so
Next, figure out where _mysql.so thinks it should find libmysqlclient.18.dylib:
$ otool -L /Users/epallier/Documents/_W_more/PROJECTS/GFT/SOFT/PYROS/pyros/venv_py35_pyros/lib/python3.5/site-packages/_mysql.cpython-35m-darwin.so
/Users/epallier/Documents/_W_more/PROJECTS/GFT/SOFT/PYROS/pyros/venv_py35_pyros/lib/python3.5/site-packages/_mysql.cpython-35m-darwin.so:
libmysqlclient.18.dylib (compatibility version 18.0.0, current version 18.0.0)
...
So, it's looking for libmysqlclient.18.dylib with no path information, let's fix that:
$ locate libmysqlclient.18.dylib
/Applications/XAMPP/xamppfiles/lib/libmysqlclient.18.dylib
/Library/SystemMigration/History/Migration-68137DFB-CB6A-4FBB-81E2-11BDB5D01E48/QuarantineRoot/usr/lib/libmysqlclient.18.dylib
$ sudo install_name_tool -change libmysqlclient.18.dylib /Applications/XAMPP/xamppfiles/lib/libmysqlclient.18.dylib /Users/epallier/Documents/_W_more/PROJECTS/GFT/SOFT/PYROS/pyros/venv_py35_pyros/lib/python3.5/site-packages/_mysql.cpython-35m-darwin.so
Now _mysql.so knows the full path to the library and everything works, regardless of environment variables.
$ otool -L /Users/epallier/Documents/_W_more/PROJECTS/GFT/SOFT/PYROS/pyros/venv_py35_pyros/lib/python3.5/site-packages/_mysql.cpython-35m-darwin.so
/Users/epallier/Documents/_W_more/PROJECTS/GFT/SOFT/PYROS/pyros/venv_py35_pyros/lib/python3.5/site-packages/_mysql.cpython-35m-darwin.so:
/Applications/XAMPP/xamppfiles/lib/libmysqlclient.18.dylib (compatibility version 18.0.0, current version 18.0.0)
...
$ ./manage.py inspectdb > models.py
=> YES !!!
</pre>
---
h3. Create a Django application pyrosapp
From src/ :
<pre>
$ ./manage.py startapp pyrosapp
</pre>
We obtain this structure:
<pre>
PYROS/
├── REQUIREMENTS.txt
├── private/
│ └── venv_py35_pyros/
├── public/
│ └── static/
├── src/
│ ├── manage.py
│ ├── pyros/
│ │ ├── __init__.py
│ │ ├── __pycache__
│ │ ├── settings.py
│ │ ├── urls.py
│ │ └── wsgi.py
│ └── pyrosapp/
│ ├── __init__.py
│ ├── admin.py
│ ├── apps.py
│ ├── migrations
│ ├── models.py
│ ├── tests.py
│ └── views.py
</pre>
---
h3. Replace the default pyrosapp models.py with the inspectdb generated one
From src/ :
<pre>
$ mv models.py pyrosapp/
</pre>
Add pyrosapp to the project's applications :
Edit src/pyros/settings.py
<pre>
INSTALLED_APPS = [
'django.contrib.admin',
'django.contrib.auth',
'django.contrib.contenttypes',
'django.contrib.sessions',
'django.contrib.messages',
'django.contrib.staticfiles',
'pyrosapp',
]
</pre>
---
h3. Fix and improve the pyrosapp models.py file (generated by inspectdb)
Once models.py file generated, we need to delete the database and create an empty one :
<pre>
$ mysql -u root [-p (if password needed)]
mysql> DROP DATABSE pyros
mysql> CREATE SCHEMA IF NOT EXISTS 'pyros' DEFAULT CHARACTER SET utf8;
</pre>
Then edit pyrosapp/models.py :
* Change 'managed = False' to 'managed = True' for every model
* Change classes names to CamelCase (do not change the 'db_table = ...' lines). *Be careful* : it is needed to change all occurences :
* NrtAlanysis
* ScheduleHistory
* ScientificProgram
* SequenceType
* SiteWatch
* SiteWatchHistory
* StrategyObs
* UserLevel
* WeatherWatch
* WeatherWatchHistory
* Change the deleting mode from 'models.DO_NOTHING' to 'models.CASCADE' for the following foreign keys :
* Image.plan
* Plan.album
* Album.sequence
* Sequence.request
* Change the 'ForeignKey' liaisons to 'OneToOneField' liaisons (just replace ForeignKey by OneToOneField), and change deleting mode to 'models.CASCADE' for the following foreign keys :
* Alert.request
* Detector.device
* Filter.device
* Telescope.device
* We need to redefine many to many relationships for the following classes :
* User - ScientificProgram :
* add 'users = models.ManyToManyField('User')' in ScientificProgram class
* delete UserHasScientificProgram class
* Sequence - ScheduleHistory
* add 'sequences = models.ManyToManyField('Sequence')' in ScheduleHistory class
* delete ScheduleHasSequences class
* Finally apply modifications to the database :
<pre>
$ pwd
.../PYROS/src
$ python manage.py makemigrations pyrosapp
$ python manage.py migrate
</pre>
---
h2. %{margin-left:0px; font-weight:bold; font-size:25px; display:block; color:red;}IV - CONFIGURATION of the Django Back Office (administration interface)%
---
h3. Back Office setup
* Prerequisites in src/pyros/settings.py :
* INSTALLED_APPS must (at least) contain :
* django.contrib.admin
* django.contrib.auth
* django.contrib.contenttypes
* django.contrib.sessions
* MIDDLEWARES must (at least) contain :
* django.contrib.sessions.middleware.SessionMiddleware
* django.middleware.common.CommonMiddleware
* django.contrib.auth.middleware.AuthenticationMiddleware
* At least one 'python manage.py migrate' must have been executed
* Create a superuser for the administration :
<pre>
$ python manage.py createsuperuser
</pre>
* For each app of the project, fill the admin.py file :
<pre>
from django.contrib import admin
from app.models import Model1, Model2
admin.site.register(Model1)
admin.site.register(Model2)
</pre>
*Reminder* : each application must be registered in the settings.py INSTALLED_APPS variable.
* For each model in models.py, add a '__str__()' method in order to identify the object on the back office. Example :
<pre>
class UserLevel(models.Model):
name = models.CharField(max_length=45, blank=True, null=True)
desc = models.TextField(blank=True, null=True)
priority = models.IntegerField(blank=True, null=True)
quota = models.FloatField(blank=True, null=True)
class Meta:
managed = True
db_table = 'userlevel'
def __str__(self):
return (str(self.name))
</pre>
*Naming convention* : Use self.name when possible, the creation time/date otherwise. Example :
<pre>
class SiteWatch(models.Model):
updated = models.DateTimeField(blank=True, null=True)
lights = models.CharField(max_length=45, blank=True, null=True)
dome = models.CharField(max_length=45, blank=True, null=True)
doors = models.CharField(max_length=45, blank=True, null=True)
temperature = models.FloatField(blank=True, null=True)
class Meta:
managed = True
db_table = 'sitewatch'
def __str__(self):
return (str(self.updated))
</pre>
h2. %{margin-left:0px; font-weight:bold; font-size:25px; display:block; color:red;}V - INSTALLATION FROM THE BEGINNING (for dev only)%
h3. How the git repository was created
*Git global setup:*
<pre>
$ git config --global user.name "Etienne Pallier"
$ git config --global user.email "etienne.pallier@irap.omp.eu"
$ cat ~/.gitconfig
[user]
name = Etienne Pallier
email = epallier@irap.omp.eu
[http]
sslVerify = false
</pre>
*Create a new repository:*
<pre>
$ cd PYROS/
Define files and folders to be ignored:
$ vi .gitignore
.DS_Store
private
__pycache__
$ touch README.md
$ git add README.md
$ git commit -m "first commit"
$ git remote add origin https://gitlab.irap.omp.eu/epallier/pyros.git
$ git push -u origin master
$ git add .
( if you want to be sure to add ALL files:
$ git add -A
)
( if you wanted to remove added files, just type:
$ git reset HEAD
)
$ git commit -m "first full project commit"
$ git push -u origin master
Counting objects: 43, done.
Delta compression using up to 4 threads.
Compressing objects: 100% (41/41), done.
Writing objects: 100% (43/43), 575.13 KiB ö 0 bytes/s, done.
Total 43 (delta 2), reused 0 (delta 0)
To https://gitlab.irap.omp.eu/epallier/pyros.git
9c7128c..64501c9 master -> master
Branch master set up to track remote branch master from origin.
$ git status
On branch master
Your branch is up-to-date with 'origin/master'.
nothing to commit, working directory clean
</pre>
h3. Create the database
* Linux and Mac OS X:
<pre>
One liner:
$ mysql -u root < pyros_create.sql
Or :
$ mysql -u root
mysql> create database pyros;
mysql> use pyros;
mysql> source pyros_create.sql;
(
TODO:
mysql> grant all on pyros.* to pyros@localhost identified by ‘pyros’;)
mysql> flush privileges;
)
</pre>
* Windows:
<pre>
TODO: Use phpmyadmin ?
</pre>
h3. Create the project structure
<pre>
$ mkdir PYROS
</pre>
Example of a good organization :
1 project = N applis
1 appli = N models
A! L’appli est à côté du projet, PAS DEDANS,
cela facilite la REUTILISATION
(an appli can be part of many projects => reuse)
1 appli = 1 Python module, organized for Django, by default = appli web (but not mandatory)
<pre>
MYPROJECT/
REQUIREMENTS.txt
src/
myproject/
appli1/
appli2/
…
appliN/
public/
static/
private/
venv_py35_pyros/
</pre>
Set needed folders:
<pre>
$ cd PYROS/
$ mkdir private public
$ mkdir public/static
</pre>
---
---
HOWTO Format Redmine Wiki : http://www.redmine.org/projects/redmine/wiki/FrRedmineWikiFormatting
{{>toc}}
---
h2. %{margin-left:0px; font-weight:bold; font-size:25px; display:block; color:red;}I - TODO%
* installation sur windows
* gitlab
* séparation des BD Django et Pyros
* intégration dans Eclipse
* Intégration des modules Django déjà développés
---
h2. %{margin-left:0px; font-weight:bold; font-size:25px; display:block; color:red;}I - DATABASE SCHEMA (v0.2.1)%
{{thumbnail(PYROS_PDM_v021.png, size=300, title=Pyros data model)}}
---
h2. %{margin-left:0px; font-weight:bold; font-size:25px; display:block; color:red;}II - Get the project (from gitlab)%
https://gitlab.irap.omp.eu/epallier/pyros
https://gitlab.irap.omp.eu/epallier/pyros.git
ssh git@gitlab.irap.omp.eu:epallier/pyros.git
h3. Get the project from the terminal
<pre>
git clone https://gitlab.irap.omp.eu/epallier/pyros.git PYROS
(or also : git clone git@gitlab.irap.omp.eu:epallier/pyros.git)
</pre>
If you just want a static copy of the project (without synchronization) just remove the .git/ folder :
<pre>
$ rm -r .git/
</pre>
h3. Get the project from Eclipse
TODO:
h2. %{margin-left:0px; font-weight:bold; font-size:25px; display:block; color:red;}III - INSTALLATION%
---
h3. Install MySql (only if necessary)
* Linux Ubuntu
<pre>
$ sudo apt-get install mysql-server
$ sudo apt-get install mysql-client
</pre>
* Linux CentOS
<pre>
TODO:
$ sudo yum install mysql
...
</pre>
* Mac OS X
Install XAMPP
(but you could also use the pre-installed Mac OS MySql)
<pre>
TODO:
</pre>
* Windows
Install XAMPP
<pre>
TODO:
</pre>
---
h3. Install Python3.5 (only if necessary)
* Mac OS X :
<pre>
1) Installer MacPort
(TODO: doc)
2) Installer le "port" python35
$ sudo port install python35
</pre>
* Linux (Ubuntu) :
<pre>
sudo add-apt-repository ppa:fkrull/deadsnakes
sudo apt-get update
sudo apt-get install python3.5
sudo pip install virtualenv
</pre>
* Windows 10 :
<pre>
TODO:
</pre>
---
h3. Get the project (from git)
<pre>
$ git clone https://gitlab.irap.omp.eu/epallier/pyros.git PYROS
</pre>
This creates a PYROS/ folder containing the project (with a .git/ subfolder for synchronization with the git repository)
---
h3. Create virtualenv with python3.5 dedicated to pyros project (inside the project folder)
<pre>
$ cd private/
$ which python3.5
/opt/local/bin/python3.5
$ virtualenv-3.5 venv_py35_pyros -p /opt/local/bin/python3.5
=> creates a venv_py35_pyros/ folder inside PYROS/private/
</pre>
---
h3. Activate the python virtual environment (from inside the project)
<pre>
$ pwd
.../PYROS/private
$ source ./venv_py35_pyros/bin/activate
$ python -V
Python 3.5.1
$ which pip
.../PYROS/venv_py35_pyros/bin/pip
Upgrade pip to last version available :
$ pip install --upgrade pip
Collecting pip
Downloading pip-8.1.1-py2.py3-none-any.whl (1.2MB)
Installing collected packages: pip
Found existing installation: pip 7.1.2
Uninstalling pip-7.1.2:
Successfully uninstalled pip-7.1.2
Successfully installed pip-8.1.1
</pre>
h3. Install needed python packages (from within the virtual environment)
First, be sure that the virtual environment is activated:
<pre>
$ python -V
Python 3.5.1
</pre>
* *Automatic Installation of all packages*
<pre>
$ pip install -r REQUIREMENTS.txt
</pre>
* *Or, manual installation of each package*
* *Install Django* :
<pre>
$ pip install django
Collecting django
Downloading Django-1.9.4-py2.py3-none-any.whl (6.6MB)
Installing collected packages: django
Successfully installed django-1.9.4
$ pip install django-admin-tools
Collecting django-admin-tools
Downloading django_admin_tools-0.7.2-py2.py3-none-any.whl (289kB)
Installing collected packages: django-admin-tools
Successfully installed django-admin-tools-0.7.2
$ pip install django-debug-toolbar
Collecting django-debug-toolbar
Downloading django_debug_toolbar-1.4-py2.py3-none-any.whl (212kB)
Requirement already satisfied (use --upgrade to upgrade): Django>=1.7 in ./venv_py35_pyros/lib/python3.5/site-packages (from django-debug-toolbar)
Collecting sqlparse (from django-debug-toolbar)
Downloading sqlparse-0.1.19.tar.gz (58kB)
Building wheels for collected packages: sqlparse
Running setup.py bdist_wheel for sqlparse ... done
Stored in directory: /Users/epallier/Library/Caches/pip/wheels/7b/d4/72/6011bb100dd5fc213164e4bbee13d4e03261dd54ce6a5de6b8
Successfully built sqlparse
Installing collected packages: sqlparse, django-debug-toolbar
Successfully installed django-debug-toolbar-1.4 sqlparse-0.1.19
$ pip install django-extensions
Collecting django-extensions
Downloading django_extensions-1.6.1-py2.py3-none-any.whl (202kB)
Collecting six>=1.2 (from django-extensions)
Downloading six-1.10.0-py2.py3-none-any.whl
Installing collected packages: six, django-extensions
Successfully installed django-extensions-1.6.1 six-1.10.0
$ pip install django-suit
Collecting django-suit
Downloading django-suit-0.2.18.tar.gz (587kB)
Building wheels for collected packages: django-suit
Running setup.py bdist_wheel for django-suit ... done
Stored in directory: /Users/epallier/Library/Caches/pip/wheels/12/8b/9a/e02ab0ad9229881638aa040d47d77c8f562999533811927d41
Successfully built django-suit
Installing collected packages: django-suit
Successfully installed django-suit-0.2.18
</pre>
* *Install the web application server gunicorn (will be used in production instead of the dev django web server)* :
<pre>
$ pip install gunicorn
Collecting gunicorn
Downloading gunicorn-19.4.5-py2.py3-none-any.whl (112kB)
Installing collected packages: gunicorn
Successfully installed gunicorn-19.4.5
</pre>
* *Install the python mysql client*:
<pre>
$ pip install mysqlclient
...
</pre>
* => Issue under Mac OS X:
<pre>
$ pip install mysqlclient
Collecting mysqlclient
Downloading mysqlclient-1.3.7.tar.gz (79kB)
Building wheels for collected packages: mysqlclient
Running setup.py bdist_wheel for mysqlclient ... error
…
----------------------------------------
Failed building wheel for mysqlclient
Running setup.py clean for mysqlclient
Failed to build mysqlclient
Installing collected packages: mysqlclient
Running setup.py install for mysqlclient ... done
Successfully installed mysqlclient-1.3.7
BOUH !!!
$ pip install --upgrade wheel
Collecting wheel
Downloading wheel-0.29.0-py2.py3-none-any.whl (66kB)
Installing collected packages: wheel
Found existing installation: wheel 0.24.0
Uninstalling wheel-0.24.0:
Successfully uninstalled wheel-0.24.0
Successfully installed wheel-0.29.0
$ pip uninstall mysqlclient
$ pip install mysqlclient
Collecting mysqlclient
Using cached mysqlclient-1.3.7.tar.gz
Building wheels for collected packages: mysqlclient
Running setup.py bdist_wheel for mysqlclient ... done
Stored in directory: /Users/epallier/Library/Caches/pip/wheels/9b/06/50/d11418c26cf8f2156b13d4363b5afde8e7e75ebb8540d0228d
Successfully built mysqlclient
Installing collected packages: mysqlclient
Successfully installed mysqlclient-1.3.7
YES !!!
</pre>
* => Issues under Ubuntu:
<pre>
$ pip install mysqlclient
Collecting mysqlclient
Downloading mysqlclient-1.3.7.tar.gz (79kB)
100% |████████████████████████████████| 81kB 1.5MB/s
Complete output from command python setup.py egg_info:
/bin/sh: 1: mysql_config: not found
Traceback (most recent call last):
File "<string>", line 1, in <module>
[...]
----------------------------------------
Command "python setup.py egg_info" failed with error code 1 in /tmp/pip-build-q6j4inuz/mysqlclient/
BOUH !!!
$ sudo apt-get install libmysqlclient-dev
$ pip install mysqlclient
Collecting mysqlclient
Using cached mysqlclient-1.3.7.tar.gz
Building wheels for collected packages: mysqlclient
Running setup.py bdist_wheel for mysqlclient ... error
…
_mysql.c:40:20: fatal error: Python.h: No such file or directory
#include "Python.h"
^
compilation terminated.
error: command 'x86_64-linux-gnu-gcc' failed with exit status 1
----------------------------------------
Command "/home/carens_p/pyros/venv_py35_pyros/bin/python3.5 -u -c "import setuptools, tokenize;__file__='/tmp/pip-build-k3klv92j/mysqlclient/setup.py';exec(compile(getattr(tokenize, 'open', open)(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" install --record /tmp/pip-gz242xxs-record/install-record.txt --single-version-externally-managed --compile --install-headers /home/carens_p/pyros/venv_py35_pyros/include/site/python3.5/mysqlclient" failed with error code 1 in /tmp/pip-build-k3klv92j/mysqlclient/
BOUH !!!
$ sudo apt-get install python3.5-dev
$ pip install mysqlclient
YES !!!
</pre>
* *Set Requirements*
<pre>
$ pip freeze > REQUIREMENTS.txt
</pre>
---
h3. Create Django project pyros
<pre>
From inside the project:
$ pwd
.../PYROS/
$ django-admin startproject pyros
Rename the project folder "pyros/" as "src/"
$ mv pyros src
We have then this architecture:
PYROS
├── REQUIREMENTS.txt
├── private
│ └── venv_py35_pyros
├── public
│ └── static
├── src
│ ├── manage.py
│ ├── pyros
│ │ ├── __init__.py
│ │ ├── settings.py
│ │ ├── urls.py
│ │ └── wsgi.py
</pre>
---
h3. Test the project
<pre>
$ cd src/
$ ./manage.py runserver
(or gunicorn pyros.wsgi)
==> http://localhost:8000
...
...
Ctrl-c
</pre>
---
h3. The Web server
Apache : gère tous les fichiers statiques (images, html…), et délègue les fichiers python au serveur django (par défaut)
Le fichier pyros/urls.py prend le relai pour tout ce qui est django
Le moteur web django sera soit du wsgi soit du unicorn
Par défaut, 1 seul worker, mais on peut en configurer plusieurs, l’idéal étant de faire "nb coeurs + 1"
(le worker maître qui fait le dispatching aux autres)
Frontend : Apache ou Ngininx
Backend : gunicorn (gère facilement des workers) ou uwsgi
<pre>
$ gunicorn pyros.wsgi
(à la place de manage runserver => A EVITER EN PROD)
Ou encore:
$ gunicorn --workers 5 library.wsgi
</pre>
---
h3. Set Database engine as MySql
Edit src/pyros/settings.py
<pre>
DATABASES = {
'default': {
'ENGINE': 'django.db.backends.mysql',
'NAME': 'pyros',
'USER': 'root',
'PASSWORD': ''
}
}
</pre>
---
h3. Import database into Django (with inspectdb)
From src/ :
<pre>
$ ./manage.py inspectdb > models.py
</pre>
Issue on Mac OS X:
<pre>
Traceback (most recent call last):
File "/Users/epallier/Documents/_W_more/PROJECTS/GFT/SOFT/PYROS/pyros/venv_py35_pyros/lib/python3.5/site-packages/django/db/backends/mysql/base.py", line 25, in <module>
import MySQLdb as Database
File "/Users/epallier/Documents/_W_more/PROJECTS/GFT/SOFT/PYROS/pyros/venv_py35_pyros/lib/python3.5/site-packages/MySQLdb/__init__.py", line 19, in <module>
import _mysql
ImportError: dlopen(/Users/epallier/Documents/_W_more/PROJECTS/GFT/SOFT/PYROS/pyros/venv_py35_pyros/lib/python3.5/site-packages/_mysql.cpython-35m-darwin.so, 2): Library not loaded: libmysqlclient.18.dylib
Referenced from: /Users/epallier/Documents/_W_more/PROJECTS/GFT/SOFT/PYROS/pyros/venv_py35_pyros/lib/python3.5/site-packages/_mysql.cpython-35m-darwin.so
Reason: image not found
=> BOUH !!!
LA SOLUTION EST ICI : http://stackoverflow.com/questions/6383310/python-mysqldb-library-not-loaded-libmysqlclient-18-dylib
Il suffit de faire ceci:
$ sudo mkdir -p /usr/local/lib
$ sudo ln -s /Applications/XAMPP/xamppfiles/lib/libmysql* /usr/local/lib/
Mais on peut aussi faire ceci:
Okay, so the offending file is /Users/epallier/Documents/_W_more/PROJECTS/GFT/SOFT/PYROS/pyros/venv_py35_pyros/lib/python3.5/site-packages/_mysql.cpython-35m-darwin.so
Next, figure out where _mysql.so thinks it should find libmysqlclient.18.dylib:
$ otool -L /Users/epallier/Documents/_W_more/PROJECTS/GFT/SOFT/PYROS/pyros/venv_py35_pyros/lib/python3.5/site-packages/_mysql.cpython-35m-darwin.so
/Users/epallier/Documents/_W_more/PROJECTS/GFT/SOFT/PYROS/pyros/venv_py35_pyros/lib/python3.5/site-packages/_mysql.cpython-35m-darwin.so:
libmysqlclient.18.dylib (compatibility version 18.0.0, current version 18.0.0)
...
So, it's looking for libmysqlclient.18.dylib with no path information, let's fix that:
$ locate libmysqlclient.18.dylib
/Applications/XAMPP/xamppfiles/lib/libmysqlclient.18.dylib
/Library/SystemMigration/History/Migration-68137DFB-CB6A-4FBB-81E2-11BDB5D01E48/QuarantineRoot/usr/lib/libmysqlclient.18.dylib
$ sudo install_name_tool -change libmysqlclient.18.dylib /Applications/XAMPP/xamppfiles/lib/libmysqlclient.18.dylib /Users/epallier/Documents/_W_more/PROJECTS/GFT/SOFT/PYROS/pyros/venv_py35_pyros/lib/python3.5/site-packages/_mysql.cpython-35m-darwin.so
Now _mysql.so knows the full path to the library and everything works, regardless of environment variables.
$ otool -L /Users/epallier/Documents/_W_more/PROJECTS/GFT/SOFT/PYROS/pyros/venv_py35_pyros/lib/python3.5/site-packages/_mysql.cpython-35m-darwin.so
/Users/epallier/Documents/_W_more/PROJECTS/GFT/SOFT/PYROS/pyros/venv_py35_pyros/lib/python3.5/site-packages/_mysql.cpython-35m-darwin.so:
/Applications/XAMPP/xamppfiles/lib/libmysqlclient.18.dylib (compatibility version 18.0.0, current version 18.0.0)
...
$ ./manage.py inspectdb > models.py
=> YES !!!
</pre>
---
h3. Create a Django application pyrosapp
From src/ :
<pre>
$ ./manage.py startapp pyrosapp
</pre>
We obtain this structure:
<pre>
PYROS/
├── REQUIREMENTS.txt
├── private/
│ └── venv_py35_pyros/
├── public/
│ └── static/
├── src/
│ ├── manage.py
│ ├── pyros/
│ │ ├── __init__.py
│ │ ├── __pycache__
│ │ ├── settings.py
│ │ ├── urls.py
│ │ └── wsgi.py
│ └── pyrosapp/
│ ├── __init__.py
│ ├── admin.py
│ ├── apps.py
│ ├── migrations
│ ├── models.py
│ ├── tests.py
│ └── views.py
</pre>
---
h3. Replace the default pyrosapp models.py with the inspectdb generated one
From src/ :
<pre>
$ mv models.py pyrosapp/
</pre>
Add pyrosapp to the project's applications :
Edit src/pyros/settings.py
<pre>
INSTALLED_APPS = [
'django.contrib.admin',
'django.contrib.auth',
'django.contrib.contenttypes',
'django.contrib.sessions',
'django.contrib.messages',
'django.contrib.staticfiles',
'pyrosapp',
]
</pre>
---
h3. Fix and improve the pyrosapp models.py file (generated by inspectdb)
Once models.py file generated, we need to delete the database and create an empty one :
<pre>
$ mysql -u root [-p (if password needed)]
mysql> DROP DATABSE pyros
mysql> CREATE SCHEMA IF NOT EXISTS 'pyros' DEFAULT CHARACTER SET utf8;
</pre>
Then edit pyrosapp/models.py :
* Change 'managed = False' to 'managed = True' for every model
* Change classes names to CamelCase (do not change the 'db_table = ...' lines). *Be careful* : it is needed to change all occurences :
* NrtAlanysis
* ScheduleHistory
* ScientificProgram
* SequenceType
* SiteWatch
* SiteWatchHistory
* StrategyObs
* UserLevel
* WeatherWatch
* WeatherWatchHistory
* Change the deleting mode from 'models.DO_NOTHING' to 'models.CASCADE' for the following foreign keys :
* Image.plan
* Plan.album
* Album.sequence
* Sequence.request
* Change the 'ForeignKey' liaisons to 'OneToOneField' liaisons (just replace ForeignKey by OneToOneField), and change deleting mode to 'models.CASCADE' for the following foreign keys :
* Alert.request
* Detector.device
* Filter.device
* Telescope.device
* We need to redefine many to many relationships for the following classes :
* User - ScientificProgram :
* add 'users = models.ManyToManyField('User')' in ScientificProgram class
* delete UserHasScientificProgram class
* Sequence - ScheduleHistory
* add 'sequences = models.ManyToManyField('Sequence')' in ScheduleHistory class
* delete ScheduleHasSequences class
* Finally apply modifications to the database :
<pre>
$ pwd
.../PYROS/src
$ python manage.py makemigrations pyrosapp
$ python manage.py migrate
</pre>
---
h2. %{margin-left:0px; font-weight:bold; font-size:25px; display:block; color:red;}IV - CONFIGURATION of the Django Back Office (administration interface)%
---
h3. Back Office setup
* Prerequisites in src/pyros/settings.py :
* INSTALLED_APPS must (at least) contain :
* django.contrib.admin
* django.contrib.auth
* django.contrib.contenttypes
* django.contrib.sessions
* MIDDLEWARES must (at least) contain :
* django.contrib.sessions.middleware.SessionMiddleware
* django.middleware.common.CommonMiddleware
* django.contrib.auth.middleware.AuthenticationMiddleware
* At least one 'python manage.py migrate' must have been executed
* Create a superuser for the administration :
<pre>
$ python manage.py createsuperuser
</pre>
* For each app of the project, fill the admin.py file :
<pre>
from django.contrib import admin
from app.models import Model1, Model2
admin.site.register(Model1)
admin.site.register(Model2)
</pre>
*Reminder* : each application must be registered in the settings.py INSTALLED_APPS variable.
* For each model in models.py, add a '__str__()' method in order to identify the object on the back office. Example :
<pre>
class UserLevel(models.Model):
name = models.CharField(max_length=45, blank=True, null=True)
desc = models.TextField(blank=True, null=True)
priority = models.IntegerField(blank=True, null=True)
quota = models.FloatField(blank=True, null=True)
class Meta:
managed = True
db_table = 'userlevel'
def __str__(self):
return (str(self.name))
</pre>
*Naming convention* : Use self.name when possible, the creation time/date otherwise. Example :
<pre>
class SiteWatch(models.Model):
updated = models.DateTimeField(blank=True, null=True)
lights = models.CharField(max_length=45, blank=True, null=True)
dome = models.CharField(max_length=45, blank=True, null=True)
doors = models.CharField(max_length=45, blank=True, null=True)
temperature = models.FloatField(blank=True, null=True)
class Meta:
managed = True
db_table = 'sitewatch'
def __str__(self):
return (str(self.updated))
</pre>
h2. %{margin-left:0px; font-weight:bold; font-size:25px; display:block; color:red;}V - INSTALLATION FROM THE BEGINNING (for dev only)%
h3. How the git repository was created
*Git global setup:*
<pre>
$ git config --global user.name "Etienne Pallier"
$ git config --global user.email "etienne.pallier@irap.omp.eu"
$ cat ~/.gitconfig
[user]
name = Etienne Pallier
email = epallier@irap.omp.eu
[http]
sslVerify = false
</pre>
*Create a new repository:*
<pre>
$ cd PYROS/
Define files and folders to be ignored:
$ vi .gitignore
.DS_Store
private
__pycache__
$ touch README.md
$ git add README.md
$ git commit -m "first commit"
$ git remote add origin https://gitlab.irap.omp.eu/epallier/pyros.git
$ git push -u origin master
$ git add .
( if you want to be sure to add ALL files:
$ git add -A
)
( if you wanted to remove added files, just type:
$ git reset HEAD
)
$ git commit -m "first full project commit"
$ git push -u origin master
Counting objects: 43, done.
Delta compression using up to 4 threads.
Compressing objects: 100% (41/41), done.
Writing objects: 100% (43/43), 575.13 KiB ö 0 bytes/s, done.
Total 43 (delta 2), reused 0 (delta 0)
To https://gitlab.irap.omp.eu/epallier/pyros.git
9c7128c..64501c9 master -> master
Branch master set up to track remote branch master from origin.
$ git status
On branch master
Your branch is up-to-date with 'origin/master'.
nothing to commit, working directory clean
</pre>
h3. Create the database
* Linux and Mac OS X:
<pre>
One liner:
$ mysql -u root < pyros_create.sql
Or :
$ mysql -u root
mysql> create database pyros;
mysql> use pyros;
mysql> source pyros_create.sql;
(
TODO:
mysql> grant all on pyros.* to pyros@localhost identified by ‘pyros’;)
mysql> flush privileges;
)
</pre>
* Windows:
<pre>
TODO: Use phpmyadmin ?
</pre>
h3. Create the project structure
<pre>
$ mkdir PYROS
</pre>
Example of a good organization :
1 project = N applis
1 appli = N models
A! L’appli est à côté du projet, PAS DEDANS,
cela facilite la REUTILISATION
(an appli can be part of many projects => reuse)
1 appli = 1 Python module, organized for Django, by default = appli web (but not mandatory)
<pre>
MYPROJECT/
REQUIREMENTS.txt
src/
myproject/
appli1/
appli2/
…
appliN/
public/
static/
private/
venv_py35_pyros/
</pre>
Set needed folders:
<pre>
$ cd PYROS/
$ mkdir private public
$ mkdir public/static
</pre>
---
---