PYROS

h1. Pyros applications

List and details of all the pyros applications.

{{>toc}}

---

h2. %{margin-left:0px; font-weight:bold; font-size:25px;  display:block; color:red;}pyrosapp%

h3. Purpose

 * Contains all the database Models

 * Basic tests in tests.py

 * Backoffice configuration in admin.py

h3. Notes

 * Should be removed, and the models moved in the "common" application

---

h2. %{margin-left:0px; font-weight:bold; font-size:25px;  display:block; color:red;}dashboard%

h3. Purpose

 * Interface for all external users

 * Leads to displays and actions for all the pyros modules (users, requests, system execution, ...)

h3. Evolution

 * Creating application

    * manage.py startapp dashboard

    * added 'dashboard' in settings.py -> INSTALLED_APPS

    * created a urls.py file in dashboard module

    * added _url(r'^dashboard/', include('dashboard.urls'))_ in pyros/urls.py -> urlpatterns

    * created templates/ and templates/dashboard/ folders in dashboard module

 * Main page

    * added 'home' view in views.py

    * linked 'home' view to 8000/dashboard URL

    * created a template for the homepage in templates/dashboard/ (with bootstrap)

    * created views, views linking (urls.py) and templates for the dashboard modules

    * redirected mainpage buttons to Admin interface (temporary)

 * System page (logs)

    * Retrieve logs from the db

    * Print logs and automatically update every second via an ajax request

h3. Notes

 * The buttons lead to the Admin interface for the moment

 * Added bootstrap3 module (see Installation)

 * Added Django Template Editor (see Eclipse configuration)

h3. TODO

 * Create the backoffice views as the modules are integrated in pyros

 * Think about a system of permissions

---

h2. %{margin-left:0px; font-weight:bold; font-size:25px;  display:block; color:red;}scheduler%

h3. Purpose

 * Creates the planning with the OBSERVABLE sequences

 * Give access to a web page to see the current & old plannings

h3. Evolution

 * Creating application

    * manage.py startapp scheduler

    * added 'scheduler' in settings.py -> INSTALLED_APPS

    * created a urls.py file in scheduler module

    * added _url(r'^scheduler/', include('scheduler.urls'))_ in pyros/urls.py -> urlpatterns

    * created templates/ and templates/scheduler/ folders in scheduler module

 * Model modifications

    * Schedule

        * Remove day_start

        * Remove day_stop

        * Add plan_start

        * Add plan_end

        * Enum system for the status 

    * ScheduleHistory

        * Remove day_start

        * Remove day_stop

        * Add plan_start

        * Add plan_end

    * Sequence

        * Remove exec_start

        * Remove exec_stop

        * Add tsp

        * Add tep

        * Add jd1

        * Add jd2

        * Add deltaTL

        * Add deltaTR

        * Add t_prefered

        * Changed duration from Float to DecimalField (more precise)

        * Add overhead

    * manage.py makemigrations sheduler ; manage.py migrate

 * Creation of Scheduler and Interval classes in models.py

 * Implementation of the Interval class

 * Implementation of the Scheduler's 'make_schedule' function (and children). This is the only entry point for now. This function creates the planning (organizes the observable sequences).

 * Creation of the web interface

    * Added current_schedule.html in template/scheduler folder

    * Created view and url linking to this template (with current planning retrieving)

 * Creation of the simulator

    * Created a second entry point in the Scheduler class (with a few minor adaptations to handle SIMULATION mode)

    * Created a simulator module in the scheduler

        * Added the MyHTMLParser class (easy implementation of HTMLParser)

        * Added Simulator class. It parses a file given in parametr to retrieve sequences and create a schedule

    * Created a second view linked to schedule/simulation to show simulation results

 * Supression du système d'agent et ajout de la tâche (celery) de scheduling

 * Main update 19/05/2016 : overheads & historic system

    * Deleted ScheduleHistory

    * Transformed the Sequence - Schedule relation to ManyToMany

    * Moved the scheduling information into the M2M relationship class (shs = ScheduleHasSequences)

    * Added a static overhead into scheduling (can be set from the calling code, through scheduler.max_overhead = blabla(float))

h3. Tests

 * Some tests in test.py to test the different functionalities of the scheduler. These tests include the simulation.

 * A simulator at localhost:8000/scheduler/simulation, taking sequences from scheduler/sequences_cador.html

h3. Notes

 * Priorities and quotas are default-calculated (for the moment)

 * What is the 'flag' attribute in the Schedule model ?

h3. TODO

 * views

  * Link the main page to the current schedule instead of the simulation page

  * Show user sequences in the schedules (with links)

  * Give accces to old schedules (day / days before / all end-night plannings / all plannings)

  * Give access to the refused sequences of a schedule, and the reasons of rejects

 * scheduler

  * Change the system to determine night start/end (they must be given in parameter, only if first_schedule is True)

  * Store the reasons of rejects (create a new attribute, in shs ?)

  * What is the 'flag' attribute ? (@AK)

  * Do not create the execute_sequence tasks if it's not the night (- 120 seconds)

  * Priority and quota computing

  * Quotas evolution

  * Blank space filling

  * At the end of a scheduling send, it to the IC ?

---

h2. %{margin-left:0px; font-weight:bold; font-size:25px;  display:block; color:red;}alert_manager%

h3. Purpose

 * Listen to VOEvent network

 * Sort interesting events

 * Create requests according to these events

 * Manage the requests created via the alerts

 * Change the observation strategies for alerts

h3. Evolution

 * Creating application

    * See scheduler documentation above

 * Implementation of the agent system

    * Created agent.py whith a "class AlertManagerAgent(Agent):" inside

    * Overriding agent's 'work' and 'shutdown' method

    * Added message(s) to agent.actions_by_message dictionnary, with the functions associated to these messages

    * Added the Agent's launch system in apps.py

 * Implementation of the VOEventListener Thread

    * Created a "class VOEventListener(Thread):" in agent.py

    * Overriding the run method

 * 19/05/2016 : deleted the agent system (deleted from the applications below too)

 * Added some VOEvents in events_to_send

 * Installed Comet (see in [[Technical components]] section)

 * Created 2 celery tasks in tesks.py : alert_listener and change_obs_strategy

 * Added alert_listener launch at server start

 * Implemented alert_listener : it waits for an event file to be created in events_received, then read it and creates a request

 * Added a simulator system ( see section Tests below )

 * Added the strategy_change page

 * Removed launching at start : it is now started by the monitoring task

h3. Tests

 * Simulator : 

  * You first need to set settings.SIMULATION to True (in pyros/settings.py)

  * Start celery workers (scripts/start_celery_workers.sh)

  * Start server (manage.py runserver)

  * Go to localhost:8000/alert_manager/simulation, and press the button !

  * This will read into the simulation_sequences file.

 * Test AlertListenerTestsCelery.test_alert_reception :

  * Tests if the alert_listener is working well.

  * +Be careful :+ this will act on the production database, and will start the entire alert workflow

  * To start it : script/celery_test.sh alert_manager

 * Tests TestStrategyChange.* :

  * Test the changing of the observation strategy with errors and different parameters

h3. Notes

 * The request created by alert_listener is hard-coded

h3. TODO

 * Web :

  * Print if there is an alert in progress in the main page

  * Link the alerts to their status and results

 * Connect to a real VOEvent broker

 * Determine the communication with FSC for strategy change

 * VOEvents :

  * Extract the good fields (see AK Q&A 07/01/2016)

  * Fill the request & alerts objects

  * Use strategies to build a request

  * Possibility to change the default strategy

  * Handle VOEvent updates

  * Be careful to not create 2 alerts for a same GRB, seen by 2 different satellites 

---

h2. %{margin-left:0px; font-weight:bold; font-size:25px;  display:block; color:red;}analyzer%

h3. Purpose

 * Apply calibration files to the images taken by the observation manager

 * Analyze the images produced by the calibrations

h3. Evolution

 * Creating application

    * See scheduler documentation above

 * Implemented the skeleton of the module in the task Analysis (call  a function for calibrations then a function for analyses)

h3. Notes

h3. TODO

 * Apply the calibrations in the right function

 * Apply the analyses only if it's a GRB

 * Implement the analyses

 * Send analyses to FSC

---

h2. %{margin-left:0px; font-weight:bold; font-size:25px;  display:block; color:red;}majordome%

h3. Purpose

 * Handles the execution of the sequences in the right order during the night

 * Handles the system start & stop

 * Handles the changes of observation conditions

 * Stops the celery tasks when needed

h3. Evolution

 * Creating application

  * See scheduler documentation above

  * Created all the tasks in tasks.py

  * Created the TaskManager to stop the sequence & plan executions when an alert / routine is received

  * Implemented execute_sequence (not finished)

h3. Notes

h3. TODO

 * TaskManager

  * When a sequence is cancelled, give back the quota to the user

  * In case of alert, do not stop the ongoing plan, and make the instruments abort

 * execute_sequence

  * Add the PLC checks at start (to see if we do the slew)

  * Use the global telescope (instead of creating one here)

  * Give first_schedule as false when a scheduling is launched

  * Remove the default countdown (1, for tests)

 * system_pause

  * Abort the isntruments

  * Stop the execution tasks

 * system_restart

  * Start a scheduling

 * change_obs_conditions

  * Change sequences status (if  needed)

  * If some status changed, re-launch a scheduling

---

h2. %{margin-left:0px; font-weight:bold; font-size:25px;  display:block; color:red;}monitoring%

h3. Purpose

 * It is the only tasks launched at the system's start

 * Make all the starting actions : instruments configuration, connection to the PLC, updating software versions, determine night start/end & start a scheduling

 * Starts the alert_listener

 * Infinite loop to regularly check the instruments and PLC status (and send them to the IC)

 * Handles night start & end

h3. Evolution

 * Creating application

    * See scheduler documentation above

 * System to launch the task at the system's start

 * Implement skeleton of task

 * Implement starting actions (with some empty functions)

 * Main loop to check PLC, instruments and night start / end (not finished)

h3. Notes

h3. TODO

 * views

  * Move the dashboard here

  * Print the instrument status

  * Print PLC informations (with the evolution)

  * In the dashboard screens, put scroll on each screen to see the old logs

 * Monitoring task

  * Uncomment the scheduling at the beginning

  * Implement night start/end computation

  * Initialize communication with the instruments

  * Configure intruments at start

  * Send software versions to the IC

  * Initialize connection with PLC

  * After the starting actions, loop to wait for the instruments configuration to be finished

  * Ask PLC for status

  * Ask filter wheel for status

  * Store the instruments & PLC status

  * Send all status to IC

  * Analyse PLC status (obs conditions, ...)

  * Create tasks of obs condition changes

---

h2. %{margin-left:0px; font-weight:bold; font-size:25px;  display:block; color:red;}observation_manager%

h3. Purpose

 * Give orders the instruments to execute the plans

 * Take the calibration images and computes the 'super' calibration images

h3. Evolution

 * Creating application

    * See scheduler documentation above

 * Created the tasks : execute_plan_(nir/vis) & create_calibrations

 * Implemented the execute_plan tasks with some incomplete functions

h3. Notes

h3. TODO

 * execute_plan :

  * Use the cameras at a global level instead of creating them here (same for the filter wheels)

  * Uncomment the instruments_ready waiting function

  * Uncomment the observation_ending waiting function

  * Try to remove code duplication

  * Determine what needs to be done at the end of an observation

 * create_calibrations :

  * Make the calibration images

  * Generate super images

  * Send them to the IC

---

h2. %{margin-left:0px; font-weight:bold; font-size:25px;  display:block; color:red;}routine_manager%

h3. Purpose

 * Enables astronomers to create routines via the website

 * Import / export of routines (XML)

 * Display observation results for routines

h3. Evolution

 * Creating application

    * See scheduler documentation above

 * Created forms online for routine creation

 * Added import / export

 * Added routine submit / unsubmit

h3. Tests

 * Import fake file

 * Import invalid file

 * Import good file

 * Export incomplete

 * Export valid

h3. Notes

h3. TODO

 * Web

  * Put the goods fields (for coordinates etc)

  * Only propose the objects that matches the conditions (ex: scientific programs of the user only)

  * Do all the needed checks

  * Add automatic computation of JD1/JD2

  * Add checkbox for JD / GD

  * Add options : copy my sequence on x days, and authorise report

  * Add ETC-IS simulation

  * Add help for new users (and for it the first time an account come on the page)

 * Do more checks at unserialization

 * views

  * When saving, do more checks on coordinates, jd1/2 etc

  * Uncomment filter for alerts removing

  * When submitting, use the monitoring to determine sequences status

  * When submitting, modify the first_schedule to False, when scheduling

  * When unsubmitting, uncomment the check for EXED and EXING removing

  * When unsubmitting, uncomment the scheduling and change the first_schedule to False

---

h2. %{margin-left:0px; font-weight:bold; font-size:25px;  display:block; color:red;}user_manager%

h3. Purpose

 * Handle user create / delete / login / logout

 * Allow to see user profile and make password recovery

h3. Evolution

 * Creating application

    * See scheduler documentation above

 * Taking and adapting templates from Alexandru project

 * User creation

  * Form in admin.py

  * Functions to verify if email doesn't exist and if passwords match

  * Added check to pass login page if user is already autheticated

  * Restricted access to all pyros views to authenticated users (except from user creation and login) with @login_required

h3. Tests

 * Creation

 * Login

 * wrong email

 * wrong password

 * logout

h3. Notes

h3. TODO

 * Password recovery

 * Profile page

 * User validation by administrator / commission

 * Handle permissions and access

---

h2. %{margin-left:0px; font-weight:bold; font-size:25px;  display:block; color:red;}common%

h3. Purpose

 * Regroups common data and functions shared by applications

h3. Content

 * RequestBuilder - class to build and save in DB the requests

  * You can start a request, then add sequences, albums and plans, given their parent class (ex: you need to give a sequence id to create an album)

  * Each function returns the ID of the element it created (to give it to the children classes)

  * To save a request, you need to use validate_request

 * Device and children

  * Implementation of devices communication, messages and initialization

  * All classes inherit from the Device class

  * All classes re-implement the list of messages

h3. Tests

 * Test the creation of a full request by the RequestBuilder

h3. Notes

 * Will probably contain the DB models at last ...

h3. TODO

 * Network communication for every instrument

 * Change the 'first_schedule' to False at the end of RequestBuilder.validate()