Technical Documentation for the PYROS project (FGFT-CC)¶

HOWTO Format Redmine Wiki : http://www.redmine.org/projects/redmine/wiki/FrRedmineWikiFormatting

• Technical Documentation for the PYROS project (FGFT-CC)
  • I - TODO
  • I - DATABASE SCHEMA (v0.2.1)
  • II - Get the project (from gitlab)
    • Get the project from the terminal
    • Get the project from Eclipse
  • III - INSTALLATION
    • Install MySql (only if necessary)
    • Install Python3.5 (only if necessary)
    • Create a Python3 virtual environment dedicated to the project (inside the project folder)
    • Activate the python virtual environment (from inside the project)
    • Install the needed Python packages (from within the virtual environment)
    • Test the project
  • IV - CONFIGURATION of the Django Back Office (administration interface)
    • Back Office setup
    • Adaptation of the one-to-many and many-to-many display
  • V - INSTALLATION FROM THE BEGINNING (for dev only, history of the initial project creation)
    • Install MySql (only if necessary)
    • Create the database (from sql script generated with Mysql Workbench)
    • Install Python3.5 (only if necessary)
    • Create the project structure
    • Create a Python3 virtual environment dedicated to the project (inside the project folder)
    • Activate the Python virtual environment (from inside the project)
    • Install the needed Python packages (from within the virtual environment)
    • Create a Django project named PYROS
    • Test the project
    • The Web server
    • Set the database engine as MySql
    • Import the database into Django (with inspectdb)
    • Create a Django application named "pyrosapp"
    • Replace the default pyrosapp models.py with the inspectdb generated one
    • Fix and improve the pyrosapp models.py file (generated by inspectdb)
    • Set Requirements
    • Initialize the git repository
    • Fix and improve the pyrosapp models.py file (generated by inspectdb)
    • CONFIGURATION of the Django Back Office (administration interface)

---

I - TODO¶

• installation sur windows

• gitlab

• séparation des BD Django et Pyros

• intégration dans Eclipse

• Intégration des modules Django déjà développés

---

I - DATABASE SCHEMA (v0.2.1)¶

---

II - Get the project (from gitlab)¶

Get the project from the terminal¶

git clone https://gitlab.irap.omp.eu/epallier/pyros.git PYROS

(or also : git clone git@gitlab.irap.omp.eu:epallier/pyros.git)

This creates a PYROS/ folder containing the project (with a .git/ subfolder for synchronization with the git repository)

If you just wanted a static copy of the project (without synchronization) just remove the .git/ folder:

$ rm -r .git/

You should obtain this structure:

PYROS/
├── REQUIREMENTS.txt
├── private/
│   └── venv_py35_pyros/
├── public/
│   └── static/
├── src/
│   ├── manage.py
│   ├── pyros/
│   │   ├── __init__.py
│   │   ├── __pycache__
│   │   ├── settings.py
│   │   ├── urls.py
│   │   └── wsgi.py
│   └── pyrosapp/
│       ├── __init__.py
│       ├── admin.py
│       ├── apps.py
│       ├── migrations
│       ├── models.py
│       ├── tests.py
│       └── views.py

Get the project from Eclipse¶

TODO:

III - INSTALLATION¶

---

Install MySql (only if necessary)¶

• Linux Ubuntu
  

  
  $ sudo apt-get install mysql-server
  $ sudo apt-get install mysql-client
  
  

• Linux CentOS
  

  TODO:
  $ sudo yum install mysql
  ...
  

• Mac OS X
  Install XAMPP
  (but you could also use the pre-installed Mac OS MySql)
  

  TODO:
  

• Windows
  Install XAMPP
  

  TODO:
  

---

Install Python3.5 (only if necessary)¶

• Mac OS X :
  

  
  1) Installer MacPort
  (TODO: doc)
  
  2) Installer le "port" python35
  $ sudo port install python35
  
  

• Linux (Ubuntu) :
  

  sudo add-apt-repository ppa:fkrull/deadsnakes
  sudo apt-get update
  sudo apt-get install python3.5
  
  sudo pip install virtualenv
  

• Windows 10 :
  

  TODO:
  

---

Create a Python3 virtual environment dedicated to the project (inside the project folder)¶

$ mkdir private/

$ cd private/

$ which python3.5
/opt/local/bin/python3.5

$ virtualenv-3.5 venv_py35_pyros -p /opt/local/bin/python3.5
=> creates a venv_py35_pyros/ folder inside PYROS/private/

---

Activate the python virtual environment (from inside the project)¶

$ pwd
.../PYROS/private

$ source ./venv_py35_pyros/bin/activate

$ python -V
Python 3.5.1

$ which pip
.../PYROS/venv_py35_pyros/bin/pip

Upgrade pip to last version available:
$ pip install --upgrade pip
Collecting pip
  Downloading pip-8.1.1-py2.py3-none-any.whl (1.2MB)
Installing collected packages: pip
  Found existing installation: pip 7.1.2
    Uninstalling pip-7.1.2:
      Successfully uninstalled pip-7.1.2
Successfully installed pip-8.1.1

Upgrade wheel to last version available:
$ pip install --upgrade wheel
Collecting wheel
  Downloading wheel-0.29.0-py2.py3-none-any.whl (66kB)
Installing collected packages: wheel
  Found existing installation: wheel 0.24.0
    Uninstalling wheel-0.24.0:
      Successfully uninstalled wheel-0.24.0
Successfully installed wheel-0.29.0

Install the needed Python packages (from within the virtual environment)¶

First, be sure that the virtual environment is activated:

$ python -V
Python 3.5.1

• Automatic Installation of all packages
  

  $ pip install -r REQUIREMENTS.txt
  

• Or, manual installation of each package
  • Install Django :
    

    $ pip install django
    Collecting django
      Downloading Django-1.9.4-py2.py3-none-any.whl (6.6MB)
    Installing collected packages: django
    Successfully installed django-1.9.4
    
    $ pip install django-admin-tools
    Collecting django-admin-tools
      Downloading django_admin_tools-0.7.2-py2.py3-none-any.whl (289kB)
    Installing collected packages: django-admin-tools
    Successfully installed django-admin-tools-0.7.2
    
    $ pip install django-debug-toolbar
    Collecting django-debug-toolbar
      Downloading django_debug_toolbar-1.4-py2.py3-none-any.whl (212kB)
    Requirement already satisfied (use --upgrade to upgrade): Django>=1.7 in ./venv_py35_pyros/lib/python3.5/site-packages (from django-debug-toolbar)
    Collecting sqlparse (from django-debug-toolbar)
      Downloading sqlparse-0.1.19.tar.gz (58kB)
    Building wheels for collected packages: sqlparse
      Running setup.py bdist_wheel for sqlparse ... done
      Stored in directory: /Users/epallier/Library/Caches/pip/wheels/7b/d4/72/6011bb100dd5fc213164e4bbee13d4e03261dd54ce6a5de6b8
    Successfully built sqlparse
    Installing collected packages: sqlparse, django-debug-toolbar
    Successfully installed django-debug-toolbar-1.4 sqlparse-0.1.19
    
    $ pip install django-extensions
    Collecting django-extensions
      Downloading django_extensions-1.6.1-py2.py3-none-any.whl (202kB)
    Collecting six>=1.2 (from django-extensions)
      Downloading six-1.10.0-py2.py3-none-any.whl
    Installing collected packages: six, django-extensions
    Successfully installed django-extensions-1.6.1 six-1.10.0
    
    $ pip install django-suit
    Collecting django-suit
      Downloading django-suit-0.2.18.tar.gz (587kB)
    Building wheels for collected packages: django-suit
      Running setup.py bdist_wheel for django-suit ... done
      Stored in directory: /Users/epallier/Library/Caches/pip/wheels/12/8b/9a/e02ab0ad9229881638aa040d47d77c8f562999533811927d41
    Successfully built django-suit
    Installing collected packages: django-suit
    Successfully installed django-suit-0.2.18
    
    

  • Install the web application server gunicorn (will be used in production instead of the dev django web server) :
    

    $ pip install gunicorn
    Collecting gunicorn
      Downloading gunicorn-19.4.5-py2.py3-none-any.whl (112kB)
    Installing collected packages: gunicorn
    Successfully installed gunicorn-19.4.5
    

  • Install the python mysql client:
    

    $ pip install mysqlclient
    ...
    

    • => Issue under Mac OS X:
      

      $ pip install mysqlclient
      Collecting mysqlclient
        Downloading mysqlclient-1.3.7.tar.gz (79kB)
      Building wheels for collected packages: mysqlclient
        Running setup.py bdist_wheel for mysqlclient ... error
      …
        ----------------------------------------
        Failed building wheel for mysqlclient
        Running setup.py clean for mysqlclient
      Failed to build mysqlclient
      Installing collected packages: mysqlclient
        Running setup.py install for mysqlclient ... done
      Successfully installed mysqlclient-1.3.7
      
      BOUH !!!
      
      $ pip install --upgrade wheel
      Collecting wheel
        Downloading wheel-0.29.0-py2.py3-none-any.whl (66kB)
      Installing collected packages: wheel
        Found existing installation: wheel 0.24.0
          Uninstalling wheel-0.24.0:
            Successfully uninstalled wheel-0.24.0
      Successfully installed wheel-0.29.0
      
      $ pip uninstall mysqlclient
      
      $ pip install mysqlclient
      Collecting mysqlclient
        Using cached mysqlclient-1.3.7.tar.gz
      Building wheels for collected packages: mysqlclient
        Running setup.py bdist_wheel for mysqlclient ... done
        Stored in directory: /Users/epallier/Library/Caches/pip/wheels/9b/06/50/d11418c26cf8f2156b13d4363b5afde8e7e75ebb8540d0228d
      Successfully built mysqlclient
      Installing collected packages: mysqlclient
      Successfully installed mysqlclient-1.3.7
      
      YES !!!
      
      

    • => Issues under Ubuntu:
      

      $ pip install mysqlclient
      Collecting mysqlclient
        Downloading mysqlclient-1.3.7.tar.gz (79kB)
          100% |████████████████████████████████| 81kB 1.5MB/s
          Complete output from command python setup.py egg_info:
          /bin/sh: 1: mysql_config: not found
          Traceback (most recent call last):
            File "<string>", line 1, in <module>
          [...]
          ----------------------------------------
      Command "python setup.py egg_info" failed with error code 1 in /tmp/pip-build-q6j4inuz/mysqlclient/
      
      BOUH !!!
      
      $ sudo apt-get install libmysqlclient-dev
      
      $ pip install mysqlclient
      Collecting mysqlclient
        Using cached mysqlclient-1.3.7.tar.gz
      Building wheels for collected packages: mysqlclient
        Running setup.py bdist_wheel for mysqlclient ... error
      …
          _mysql.c:40:20: fatal error: Python.h: No such file or directory
           #include "Python.h" 
                              ^
          compilation terminated.
          error: command 'x86_64-linux-gnu-gcc' failed with exit status 1
      
          ----------------------------------------
      Command "/home/carens_p/pyros/venv_py35_pyros/bin/python3.5 -u -c "import setuptools, tokenize;__file__='/tmp/pip-build-k3klv92j/mysqlclient/setup.py';exec(compile(getattr(tokenize, 'open', open)(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" install --record /tmp/pip-gz242xxs-record/install-record.txt --single-version-externally-managed --compile --install-headers /home/carens_p/pyros/venv_py35_pyros/include/site/python3.5/mysqlclient" failed with error code 1 in /tmp/pip-build-k3klv92j/mysqlclient/
      
      BOUH !!!
      
      $ sudo apt-get install python3.5-dev
      
      $ pip install mysqlclient
      
      YES !!!
      
      

---

Test the project¶

$ cd src/

$ ./manage.py runserver
(or gunicorn pyros.wsgi)
==> http://localhost:8000
...
...
Ctrl-c

---

IV - CONFIGURATION of the Django Back Office (administration interface)¶

---

Back Office setup¶

• Prerequisites in src/pyros/settings.py :
  • INSTALLED_APPS must (at least) contain :
    • django.contrib.admin
    • django.contrib.auth
    • django.contrib.contenttypes
    • django.contrib.sessions
  • MIDDLEWARES must (at least) contain :
    • django.contrib.sessions.middleware.SessionMiddleware
    • django.middleware.common.CommonMiddleware
    • django.contrib.auth.middleware.AuthenticationMiddleware

• At least one 'python manage.py migrate' must have been executed

• Create a superuser for the administration :

$ python manage.py createsuperuser

• For each app of the project, fill the admin.py file :

from django.contrib import admin
from app.models import Model1, Model2

admin.site.register(Model1)
admin.site.register(Model2)

Reminder : each application must be registered in the settings.py INSTALLED_APPS variable.

• For each model in models.py, add a '__str__()' method in order to identify the object on the back office. Example :

class UserLevel(models.Model):
    name = models.CharField(max_length=45, blank=True, null=True)
    desc = models.TextField(blank=True, null=True)
    priority = models.IntegerField(blank=True, null=True)
    quota = models.FloatField(blank=True, null=True)

    class Meta:
        managed = True
        db_table = 'userlevel'

    def __str__(self):
        return (str(self.name))

Naming convention : Use self.name when possible, the creation time/date otherwise. Example :

class SiteWatch(models.Model):
    updated = models.DateTimeField(blank=True, null=True)
    lights = models.CharField(max_length=45, blank=True, null=True)
    dome = models.CharField(max_length=45, blank=True, null=True)
    doors = models.CharField(max_length=45, blank=True, null=True)
    temperature = models.FloatField(blank=True, null=True)

    class Meta:
        managed = True
        db_table = 'sitewatch'

    def __str__(self):
        return (str(self.updated))

---

Adaptation of the one-to-many and many-to-many display¶

• The one-to-many relationships are the following (One.many format) :
  • Schedule.sequences
  • Request.sequences
  • Sequence.albums
  • Album.plans
  • Plan.images
  • Telescope.detectors
  • Detector.filters
  • NrtAnalysis.images
  • Filter.plans
  • Detector.albums
  • UserLevel.users
  • Country.users
  • ScientificProgram.requests
  • User.requests
  • StrategyObs.alerts
  • SequenceType.sequences

• For each "many", create a new class in admin.py just after the imports, following these examples :

For Schedule.sequences, Request.sequences and Sequentype.sequences, we will need :

class SequenceInline(admin.TabularInline):
    model = Sequence
    fields = ("name",)
    show_change_link = True

For Sequence.albums and Detector.albums, we will need :

class AlbumInline(admin.TabularInline):
    model = Album
    fields = ("name",)
    show_change_link = True

For StrategyObs.alerts, we will need :

class AlertInline(admin.TabularInline):
    model = Alert
    fields = ("request.name",) # there is no 'name' attribute in the Alert model
    show_change_link = True

• For each "One", declare a new class in admin.py, just after the "Inlines" class declaration, as done in the following examples :

For Request.sequences :

class RequestAdmin(admin.ModelAdmin):
    inlines = [
    SequenceInline,
    ]

For Detector.filters and Detector.albums :

class DetectorAdmin(admin.ModelAdmin):
    inlines = [
    FilterInline,
        AlbumInline,
    ]

• The many-to-many relationships are the following :
  • ScientificProgram - User
  • ScheduleHistory - Sequence

• For each many-to-many relationship, declare a new "Inline" class in admin.py just after the imports, like this :

For ScientificProgram - User :

class UserAndSPInline(admin.TabularInline):
    model = ScientificProgram.users.through

For ScheduleHistory - Sequence

class SequenceAndSHInline(admin.TabularInline):
    model = ScheduleHistory.sequences.through

Note : The order in the line "model = ScientificProgram.users.through" is very important : the first model (ScientificProgram) is the one in which is declared the ManyToManyField relationship.

• For each many-to-many relationship, declare two new classes in admin.py, just after the inlines, like in the following examples :

For the ScheduleHistory - Sequence relationship :

class ScheduleHistoryAdmin(admin.ModelAdmin):
    inlines = [
        SequenceAndSHInline,
    ]
    exclude = ('sequences',) # ScheduleHistory declares the ManyToManyField, and we want to replace its display in the back office, so we won't display the default field

class SequenceAdmin(admin.ModelAdmin):
    inlines = [
        AlbumInline,        # This is the Inline for the one-to-many relationship Sequence.albums
        SequenceAndSHInline,                                                                
    ]

For the ScientificProgram - User relationship :

class ScientificProgramAdmin(admin.ModelAdmin):
    inlines = [
        RequestInline,
    UserAndSPInline,                                                                         
    ]
    exclude = ('users',) # Same as ScheduleHistory                                                                         

class UserAdmin(admin.ModelAdmin):
    inlines = [
    RequestInline,   # This is the Inline for the one-to-many relationship User.requests
        UserAndSPInline,                                                                    
    ]

• For each ModelAdmin class in the admin.py, change the registering line

admin.site.register(Album)

to

admin.site.register(Album, AlbumAdmin)

V - INSTALLATION FROM THE BEGINNING (for dev only, history of the initial project creation)¶

Install MySql (only if necessary)¶

See above in III-Installation
https://projects.irap.omp.eu/projects/pyros/wiki/Wiki#III-INSTALLATION

---

Create the database (from sql script generated with Mysql Workbench)¶

• Linux and Mac OS X:
  

  One liner:
  $ mysql -u root < pyros_create.sql
  
  Or :
  $ mysql -u root
  mysql> create database pyros;
  mysql> use pyros;
  mysql> source pyros_create.sql;
  
  (
  TODO:
  mysql> grant all on pyros.* to pyros@localhost identified by ‘pyros’;)
  mysql> flush privileges;
  )
  

• Windows:
  

  TODO: Use phpmyadmin ?
  

Install Python3.5 (only if necessary)¶

See above in III-Installation
https://projects.irap.omp.eu/projects/pyros/wiki/Wiki#III-INSTALLATION

Create the project structure¶

$ mkdir PYROS

Example of a good organization :

1 project = N applis
1 appli = N models
A! L’appli est à côté du projet, PAS DEDANS,
cela facilite la REUTILISATION
(an appli can be part of many projects => reuse)
1 appli = 1 Python module, organized for Django, by default = appli web (but not mandatory)

MYPROJECT/
    REQUIREMENTS.txt
    src/
        myproject/
        appli1/
        appli2/
        …
        appliN/

    public/
        static/

    private/
        venv_py35_pyros/

Set needed folders:

$ cd PYROS/
$ mkdir private public
$ mkdir public/static

---

Create a Python3 virtual environment dedicated to the project (inside the project folder)¶

See above in III-Installation
https://projects.irap.omp.eu/projects/pyros/wiki/Wiki#III-INSTALLATION

Activate the Python virtual environment (from inside the project)¶

See above in III-Installation
https://projects.irap.omp.eu/projects/pyros/wiki/Wiki#III-INSTALLATION

Install the needed Python packages (from within the virtual environment)¶

See above in III-Installation
https://projects.irap.omp.eu/projects/pyros/wiki/Wiki#Install-the-needed-Python-packages-from-within-the-virtual-environment
(cf manual installation)

Create a Django project named PYROS¶

From inside the project:
$ pwd
.../PYROS/

$ django-admin startproject pyros

Rename the project folder "pyros/" as "src/" 

$ mv pyros src

We have then this architecture:

PYROS
├── REQUIREMENTS.txt
├── private
│   └── venv_py35_pyros
├── public
│   └── static
├── src
│   ├── manage.py
│   ├── pyros
│   │   ├── __init__.py
│   │   ├── settings.py
│   │   ├── urls.py
│   │   └── wsgi.py

---

Test the project¶

See above in III-Installation
https://projects.irap.omp.eu/projects/pyros/wiki/Wiki#III-INSTALLATION

The Web server¶

Apache : gère tous les fichiers statiques (images, html…), et délègue les fichiers python au serveur django (par défaut)

Le fichier pyros/urls.py prend le relai pour tout ce qui est django

Le moteur web django sera soit du wsgi soit du unicorn

Par défaut, 1 seul worker, mais on peut en configurer plusieurs, l’idéal étant de faire "nb coeurs + 1"
(le worker maître qui fait le dispatching aux autres)

Frontend : Apache ou Ngininx

Backend : gunicorn (gère facilement des workers) ou uwsgi

$ gunicorn pyros.wsgi
(à la place de manage runserver => A EVITER EN PROD)

Ou encore:

$ gunicorn --workers 5 library.wsgi

---

Set the database engine as MySql¶

Edit src/pyros/settings.py

DATABASES = {
    'default': {
        'ENGINE': 'django.db.backends.mysql',
        'NAME': 'pyros',
        'USER': 'root',
        'PASSWORD': ''
    }
}

---

Import the database into Django (with inspectdb)¶

From src/ :

$ ./manage.py inspectdb > models.py

Issue on Mac OS X:

Traceback (most recent call last):
  File "/Users/epallier/Documents/_W_more/PROJECTS/GFT/SOFT/PYROS/pyros/venv_py35_pyros/lib/python3.5/site-packages/django/db/backends/mysql/base.py", line 25, in <module>
    import MySQLdb as Database
  File "/Users/epallier/Documents/_W_more/PROJECTS/GFT/SOFT/PYROS/pyros/venv_py35_pyros/lib/python3.5/site-packages/MySQLdb/__init__.py", line 19, in <module>
    import _mysql
ImportError: dlopen(/Users/epallier/Documents/_W_more/PROJECTS/GFT/SOFT/PYROS/pyros/venv_py35_pyros/lib/python3.5/site-packages/_mysql.cpython-35m-darwin.so, 2): Library not loaded: libmysqlclient.18.dylib
  Referenced from: /Users/epallier/Documents/_W_more/PROJECTS/GFT/SOFT/PYROS/pyros/venv_py35_pyros/lib/python3.5/site-packages/_mysql.cpython-35m-darwin.so
  Reason: image not found

=> BOUH !!!

LA SOLUTION EST ICI : http://stackoverflow.com/questions/6383310/python-mysqldb-library-not-loaded-libmysqlclient-18-dylib

Il suffit de faire ceci:

$ sudo mkdir -p /usr/local/lib   
$ sudo ln -s /Applications/XAMPP/xamppfiles/lib/libmysql* /usr/local/lib/

Mais on peut aussi faire ceci:

Okay, so the offending file is /Users/epallier/Documents/_W_more/PROJECTS/GFT/SOFT/PYROS/pyros/venv_py35_pyros/lib/python3.5/site-packages/_mysql.cpython-35m-darwin.so

Next, figure out where _mysql.so thinks it should find libmysqlclient.18.dylib:

$ otool -L /Users/epallier/Documents/_W_more/PROJECTS/GFT/SOFT/PYROS/pyros/venv_py35_pyros/lib/python3.5/site-packages/_mysql.cpython-35m-darwin.so
/Users/epallier/Documents/_W_more/PROJECTS/GFT/SOFT/PYROS/pyros/venv_py35_pyros/lib/python3.5/site-packages/_mysql.cpython-35m-darwin.so:
    libmysqlclient.18.dylib (compatibility version 18.0.0, current version 18.0.0)
...

So, it's looking for libmysqlclient.18.dylib with no path information, let's fix that:

$ locate libmysqlclient.18.dylib
/Applications/XAMPP/xamppfiles/lib/libmysqlclient.18.dylib
/Library/SystemMigration/History/Migration-68137DFB-CB6A-4FBB-81E2-11BDB5D01E48/QuarantineRoot/usr/lib/libmysqlclient.18.dylib

$ sudo install_name_tool -change libmysqlclient.18.dylib /Applications/XAMPP/xamppfiles/lib/libmysqlclient.18.dylib /Users/epallier/Documents/_W_more/PROJECTS/GFT/SOFT/PYROS/pyros/venv_py35_pyros/lib/python3.5/site-packages/_mysql.cpython-35m-darwin.so

Now _mysql.so knows the full path to the library and everything works, regardless of environment variables.

$ otool -L /Users/epallier/Documents/_W_more/PROJECTS/GFT/SOFT/PYROS/pyros/venv_py35_pyros/lib/python3.5/site-packages/_mysql.cpython-35m-darwin.so
/Users/epallier/Documents/_W_more/PROJECTS/GFT/SOFT/PYROS/pyros/venv_py35_pyros/lib/python3.5/site-packages/_mysql.cpython-35m-darwin.so:
    /Applications/XAMPP/xamppfiles/lib/libmysqlclient.18.dylib (compatibility version 18.0.0, current version 18.0.0)
...

$ ./manage.py inspectdb > models.py

=> YES !!!

---

Create a Django application named "pyrosapp"¶

From src/ :

$ ./manage.py startapp pyrosapp

We obtain this structure:

PYROS/
├── REQUIREMENTS.txt
├── private/
│   └── venv_py35_pyros/
├── public/
│   └── static/
├── src/
│   ├── manage.py
│   ├── pyros/
│   │   ├── __init__.py
│   │   ├── __pycache__
│   │   ├── settings.py
│   │   ├── urls.py
│   │   └── wsgi.py
│   └── pyrosapp/
│       ├── __init__.py
│       ├── admin.py
│       ├── apps.py
│       ├── migrations
│       ├── models.py
│       ├── tests.py
│       └── views.py

---

Replace the default pyrosapp models.py with the inspectdb generated one¶

From src/ :

$ mv models.py pyrosapp/

Add pyrosapp to the project's applications :

Edit src/pyros/settings.py

INSTALLED_APPS = [
    'django.contrib.admin',
    'django.contrib.auth',
    'django.contrib.contenttypes',
    'django.contrib.sessions',
    'django.contrib.messages',
    'django.contrib.staticfiles',
    'pyrosapp',
]

---

Fix and improve the pyrosapp models.py file (generated by inspectdb)¶

Set Requirements¶

$ pip freeze > REQUIREMENTS.txt

---

Initialize the git repository¶

Git global setup:

$ git config --global user.name "Etienne Pallier" 
$ git config --global user.email "etienne.pallier@irap.omp.eu" 

$ cat ~/.gitconfig 
[user]
    name = Etienne Pallier
    email = epallier@irap.omp.eu
[http]
    sslVerify = false

Create a new repository:

$ cd PYROS/

Define files and folders to be ignored:
$ vi .gitignore
.DS_Store
private
__pycache__

$ touch README.md

$ git add README.md

$ git commit -m "first commit" 

$ git remote add origin https://gitlab.irap.omp.eu/epallier/pyros.git

$ git push -u origin master

$ git add .

( if you want to be sure to add ALL files: 
$ git add -A
)

( if you wanted to remove added files, just type:
$ git reset HEAD
)

$ git commit -m "first full project commit" 

$ git push -u origin master
Counting objects: 43, done.
Delta compression using up to 4 threads.
Compressing objects: 100% (41/41), done.
Writing objects: 100% (43/43), 575.13 KiB ö 0 bytes/s, done.
Total 43 (delta 2), reused 0 (delta 0)
To https://gitlab.irap.omp.eu/epallier/pyros.git
   9c7128c..64501c9  master -> master
Branch master set up to track remote branch master from origin.

$ git status
On branch master
Your branch is up-to-date with 'origin/master'.
nothing to commit, working directory clean

---

Fix and improve the pyrosapp models.py file (generated by inspectdb)¶

Once models.py file generated, we need to delete the database and create an empty one :

$ mysql -u root [-p (if password needed)]

mysql> DROP DATABSE pyros
mysql> CREATE SCHEMA IF NOT EXISTS 'pyros' DEFAULT CHARACTER SET utf8;

Then edit pyrosapp/models.py :

• Change 'managed = False' to 'managed = True' for every model

• Change classes names to CamelCase (do not change the 'db_table = ...' lines). Be careful : it is needed to change all occurences :
  • NrtAlanysis
  • ScheduleHistory
  • ScientificProgram
  • SequenceType
  • SiteWatch
  • SiteWatchHistory
  • StrategyObs
  • UserLevel
  • WeatherWatch
  • WeatherWatchHistory

• Change the deleting mode from 'models.DO_NOTHING' to 'models.CASCADE' for the following foreign keys :
  • Image.plan
  • Plan.album
  • Album.sequence
  • Sequence.request

• Change the 'ForeignKey' liaisons to 'OneToOneField' liaisons (just replace ForeignKey by OneToOneField), and change deleting mode to 'models.CASCADE' for the following foreign keys :
  • Alert.request
  • Detector.device
  • Filter.device
  • Telescope.device

• We need to redefine many to many relationships for the following classes :
  • User - ScientificProgram :
    • add 'users = models.ManyToManyField('User')' in ScientificProgram class
    • delete UserHasScientificProgram class
  • Sequence - ScheduleHistory
    • add 'sequences = models.ManyToManyField('Sequence')' in ScheduleHistory class
    • delete ScheduleHasSequences class

• For each ForeignKey and ManyToManyField creation in models.py, add the 'related_name=[...]' named parameter, as in the following examples :

class Sequence(models.Model):
    request = models.ForeignKey(Request, models.CASCADE, related_name="sequences")
    sequencetype = models.ForeignKey('SequenceType', models.DO_NOTHING, related_name="sequences")
    schedule = models.ForeignKey(Schedule, models.DO_NOTHING, related_name="sequences")
    name = models.CharField(max_length=45, blank=True, null=True)
    desc = models.TextField(blank=True, null=True)
    ...

class ScheduleHistory(models.Model):
    sequences = models.ManyToManyField('Sequence', related_name='schedulehistorys')
    created = models.DateTimeField(blank=True, null=True)
    ...

* Finally apply modifications to the database :

$ pwd
.../PYROS/src
$ python manage.py makemigrations pyrosapp
$ python manage.py migrate

---

CONFIGURATION of the Django Back Office (administration interface)¶

cf https://projects.irap.omp.eu/projects/pyros/wiki/Wiki#IV-CONFIGURATION-of-the-Django-Back-Office-administration-interface