PYROS

h1. Pyros applications

List and details of all the pyros applications.

{{>toc}}

---

h2. %{margin-left:0px; font-weight:bold; font-size:25px;  display:block; color:red;}pyrosapp%

h3. Purpose

 * Contains all the database Models

 * Basic tests in tests.py

 * Backoffice configuration in admin.py

h3. Notes

---

h2. %{margin-left:0px; font-weight:bold; font-size:25px;  display:block; color:red;}dashboard%

h3. Purpose

 * Interface for all external users

 * Leads to displays and actions for all the pyros modules (users, requests, system execution, ...)

h3. Evolution

 * Creating application

    * manage.py startapp dashboard

    * added 'dashboard' in settings.py -> INSTALLED_APPS

    * created a urls.py file in dashboard module

    * added _url(r'^dashboard/', include('dashboard.urls'))_ in pyros/urls.py -> urlpatterns

    * created templates/ and templates/dashboard/ folders in dashboard module

 * Main page

    * added 'home' view in views.py

    * linked 'home' view to 8000/dashboard URL

    * created a template for the homepage in templates/dashboard/ (with bootstrap)

    * created views, views linking (urls.py) and templates for the dashboard modules

    * redirected mainpage buttons to Admin interface (temporary)

 * System page (logs)

    * Retrieve logs from the db

    * Print logs and automatically update every second via an ajax request

h3. Notes

 * The buttons lead to the Admin interface for the moment

 * Added bootstrap3 module (see Installation)

 * Added Django Template Editor (see Eclipse configuration)

h3. TODO

 * Create the backoffice views as the modules are integrated in pyros

 * Think about a system of permissions

---

h2. %{margin-left:0px; font-weight:bold; font-size:25px;  display:block; color:red;}scheduler%

h3. Purpose

 * Creates the planning with the OBSERVABLE sequences

 * Give access to a web page to see the current & old plannings

h3. Evolution

 * Creating application

    * manage.py startapp scheduler

    * added 'scheduler' in settings.py -> INSTALLED_APPS

    * created a urls.py file in scheduler module

    * added _url(r'^scheduler/', include('scheduler.urls'))_ in pyros/urls.py -> urlpatterns

    * created templates/ and templates/scheduler/ folders in scheduler module

 * Model modifications

    * Schedule

        * Remove day_start

        * Remove day_stop

        * Add plan_start

        * Add plan_end

        * Enum system for the status 

    * ScheduleHistory

        * Remove day_start

        * Remove day_stop

        * Add plan_start

        * Add plan_end

    * Sequence

        * Remove exec_start

        * Remove exec_stop

        * Add tsp

        * Add tep

        * Add jd1

        * Add jd2

        * Add deltaTL

        * Add deltaTR

        * Add t_prefered

        * Changed duration from Float to DecimalField (more precise)

        * Add overhead

    * manage.py makemigrations sheduler ; manage.py migrate

 * Creation of Scheduler and Interval classes in models.py

 * Implementation of the Interval class

 * Implementation of the Scheduler's 'make_schedule' function (and children). This is the only entry point for now. This function creates the planning (organizes the observable sequences).

 * Creation of the web interface

    * Added current_schedule.html in template/scheduler folder

    * Created view and url linking to this template (with current planning retrieving)

 * Creation of the simulator

    * Created a second entry point in the Scheduler class (with a few minor adaptations to handle SIMULATION mode)

    * Created a simulator module in the scheduler

        * Added the MyHTMLParser class (easy implementation of HTMLParser)

        * Added Simulator class. It parses a file given in parametr to retrieve sequences and create a schedule

    * Created a second view linked to schedule/simulation to show simulation results

 * Supression du système d'agent et ajout de la tâche (celery) de scheduling

 * Main update 19/05/2016 : overheads & historic system

    * Deleted ScheduleHistory

    * Transformed the Sequence - Schedule relation to ManyToMany

    * Moved the scheduling information into the M2M relationship class (shs = ScheduleHasSequences)

    * Added a static overhead into scheduling (can be set from the calling code, through scheduler.max_overhead = blabla(float))

h3. Tests

 * Some tests in test.py to test the different functionalities of the scheduler. These tests include the simulation.

 * A simulator at localhost:8000/scheduler/simulation, taking sequences from scheduler/sequences_cador.html

h3. Notes

 * Priorities and quotas are default-calculated (for the moment)

 * What is the 'flag' attribute in the Schedule model ?

h3. TODO

 * views

  * Link the main page to the current schedule instead of the simulation page

  * Show user sequences in the schedules (with links)

  * Give accces to old schedules (day / days before / all end-night plannings / all plannings)

  * Give access to the refused sequences of a schedule, and the reasons of rejects

 * scheduler

  * Change the system to determine night start/end (they must be given in parameter, only if first_schedule is True)

  * Store the reasons of rejects (create a new attribute)

  * What is the 'flag' attribute ? (@AK)

  * Do not create the execute_sequence tasks if it's not the night (- 120 seconds)

  * Priority and quota computing

  * Quotas evolution

  * Blank space filling

  * At the end of a scheduling send, it to the IC ?

---

h2. %{margin-left:0px; font-weight:bold; font-size:25px;  display:block; color:red;}alert_manager%

h3. Purpose

 * Listen to VOEvent network

 * Sort interesting events

 * Create requests according to these events

 * Manage the requests created via the alerts

 * Change the observation strategies for alerts

h3. Evolution

 * Creating application

    * See scheduler documentation above

 * Implementation of the agent system

    * Created agent.py whith a "class AlertManagerAgent(Agent):" inside

    * Overriding agent's 'work' and 'shutdown' method

    * Added message(s) to agent.actions_by_message dictionnary, with the functions associated to these messages

    * Added the Agent's launch system in apps.py

 * Implementation of the VOEventListener Thread

    * Created a "class VOEventListener(Thread):" in agent.py

    * Overriding the run method

 * 19/05/2016 : deleted the agent system (deleted from the applications below too)

 * Added some VOEvents in events_to_send

 * Installed Comet (see in [[Technical components]] section)

 * Created 2 celery tasks in tesks.py : alert_listener and change_obs_strategy

 * Added alert_listener launch at server start

 * Implemented alert_listener : it waits for an event file to be created in events_received, then read it and creates a request

 * Added a simulator system ( see section Tests below )

 * Added the strategy_change page

h3. Tests

 * Simulator : 

  * You first need to set settings.SIMULATION to True (in pyros/settings.py)

  * Start celery workers (scripts/start_celery_workers.sh)

  * Start server (manage.py runserver)

  * Go to localhost:8000/alert_manager/simulation, and press the button !

  * This will read into the simulation_sequences file.

 * Test AlertListenerTestsCelery.test_alert_reception :

  * Tests if the alert_listener is working well.

  * +Be careful :+ this will act on the production database, and will start the entire alert workflow

  * To start it : script/celery_test.sh alert_manager

 * Tests TestStrategyChange.* :

  * Test the changing of the observation strategy with errors and different parameters

h3. Notes

 * The request created by alert_listener is hard-coded

h3. TODO

 * Web :

  * Print if there is an alert in progress in the main page

  * Link the alerts to their status and results

 * Connect to a real VOEvent broker

 * Determine the communication with FSC for strategy change

 * VOEvents :

  * Extract the good fields (see AK Q&A 07/01/2016)

  * Fill the request & alerts objects

  * Use strategies to build a request

  * Handle VOEvent updates

  * Be careful to not create 2 alerts for a same GRB, seen by 2 different satellites 

---

h2. %{margin-left:0px; font-weight:bold; font-size:25px;  display:block; color:red;}analyzer%

h3. Purpose

 * Apply calibration files to the images taken by the observation manager

 * Analyze the images produced by the calibrations

h3. Evolution

 * Creating application

    * See scheduler documentation above

 * Implementation of the agent system

    * See alert_manager above

    * 'shutdown' method is not implemented here because the Agent's default one is enough 

 * 

h3. Notes

h3. TODO

---

h2. %{margin-left:0px; font-weight:bold; font-size:25px;  display:block; color:red;}majordome%

h3. Purpose

h3. Evolution

 * Creating application

    * See scheduler documentation above

 * Implementation of the agent system

    * See alert_manager above

 * Implementation of the MajordomeExecutor Thread

    * Created a "class MajordomeExecutor(Thread):" in agent.py

    * Overriding the run method

h3. Notes

h3. TODO

---

h2. %{margin-left:0px; font-weight:bold; font-size:25px;  display:block; color:red;}monitoring%

h3. Purpose

h3. Evolution

 * Creating application

    * See scheduler documentation above

 * Implementation of the agent system

    * See alert_manager above

    * 'shutdown' method is not implemented here because the Agent's default one is enough 

h3. Notes

h3. TODO

---

h2. %{margin-left:0px; font-weight:bold; font-size:25px;  display:block; color:red;}observation_manager%

h3. Purpose

h3. Evolution

 * Creating application

    * See scheduler documentation above

 * Implementation of the agent system

    * See alert_manager above

 * Implementation of the ObservationExecutor Thread

    * Created a "class ObservationExecutor(Thread):" in agent.py

    * Overriding the run method

h3. Notes

h3. TODO

---

h2. %{margin-left:0px; font-weight:bold; font-size:25px;  display:block; color:red;}routine_manager%

h3. Purpose

h3. Evolution

 * Creating application

    * See scheduler documentation above

h3. Notes

h3. TODO

---

h2. %{margin-left:0px; font-weight:bold; font-size:25px;  display:block; color:red;}user_manager%

h3. Purpose

 * Handle user create / delete / login / logout

 * Allow to see user profile and make password recovery

h3. Evolution

 * Creating application

    * See scheduler documentation above

 * Taking and adapting templates from Alexandru project

 * User creation

  * Form in admin.py

  * Functions to verify if email doesn't exist and if passwords match

  * Added check to pass login page if user is already autheticated

  * Restricted access to all pyros views to authenticated users (except from user creation and login) with @login_required

h3. Notes

h3. TODO

 * Password recovery

 * Profile page

 * User validation by administrator / commission

---

h2. %{margin-left:0px; font-weight:bold; font-size:25px;  display:block; color:red;}common%

h3. Purpose

 * Regroups common data and functions shared by applications

h3. Content

 * RequestBuilder - class to build and save in DB the requests

  * You can start a request, then add sequences, albums and plans, given their parent class (ex: you need to give a sequence id to create an album)

  * Each function returns the ID of the element it created (to give it to the children classes)

  * To save a request, you need to use validate_request

h3. Notes

 * Will probably contain the DB models at last ...

h3. TODO