Pyros applications

Version 52 (Paul Carensac, 07/08/2016 12:29 pm) → Version 53/60 (Paul Carensac, 07/08/2016 12:31 pm)

h1. Pyros applications

List and details of all the pyros applications.

{{>toc}}

---

h2. %{margin-left:0px; font-weight:bold; font-size:25px; display:block; color:red;}pyrosapp%

h3. Purpose

* Contains all the database Models
* Basic tests in tests.py
* Backoffice configuration in admin.py

h3. Notes

---

h2. %{margin-left:0px; font-weight:bold; font-size:25px; display:block; color:red;}dashboard%

h3. Purpose

* Interface for all external users
* Leads to displays and actions for all the pyros modules (users, requests, system execution, ...)

h3. Evolution

* Creating application

* manage.py startapp dashboard
* added 'dashboard' in settings.py -> INSTALLED_APPS
* created a urls.py file in dashboard module
* added _url(r'^dashboard/', include('dashboard.urls'))_ in pyros/urls.py -> urlpatterns
* created templates/ and templates/dashboard/ folders in dashboard module

* Main page

* added 'home' view in views.py
* linked 'home' view to 8000/dashboard URL
* created a template for the homepage in templates/dashboard/ (with bootstrap)
* created views, views linking (urls.py) and templates for the dashboard modules
* redirected mainpage buttons to Admin interface (temporary)

* System page (logs)

* Retrieve logs from the db
* Print logs and automatically update every second via an ajax request

h3. Notes

* The buttons lead to the Admin interface for the moment
* Added bootstrap3 module (see Installation)
* Added Django Template Editor (see Eclipse configuration)

h3. TODO

* Create the backoffice views as the modules are integrated in pyros
* Think about a system of permissions

---

h2. %{margin-left:0px; font-weight:bold; font-size:25px; display:block; color:red;}scheduler%

h3. Purpose

* Creates the planning with the OBSERVABLE sequences
* Give access to a web page to see the current & old plannings

h3. Evolution

* Creating application

* manage.py startapp scheduler
* added 'scheduler' in settings.py -> INSTALLED_APPS
* created a urls.py file in scheduler module
* added _url(r'^scheduler/', include('scheduler.urls'))_ in pyros/urls.py -> urlpatterns
* created templates/ and templates/scheduler/ folders in scheduler module

* Model modifications

* Schedule

* Remove day_start
* Remove day_stop
* Add plan_start
* Add plan_end
* Enum system for the status
* ScheduleHistory

* Remove day_start
* Remove day_stop
* Add plan_start
* Add plan_end
* Sequence

* Remove exec_start
* Remove exec_stop
* Add tsp
* Add tep
* Add jd1
* Add jd2
* Add deltaTL
* Add deltaTR
* Add t_prefered
* Changed duration from Float to DecimalField (more precise)
* Add overhead

* manage.py makemigrations sheduler ; manage.py migrate

* Creation of Scheduler and Interval classes in models.py

* Implementation of the Interval class

* Implementation of the Scheduler's 'make_schedule' function (and children). This is the only entry point for now. This function creates the planning (organizes the observable sequences).

* Creation of the web interface

* Added current_schedule.html in template/scheduler folder
* Created view and url linking to this template (with current planning retrieving)

* Creation of the simulator

* Created a second entry point in the Scheduler class (with a few minor adaptations to handle SIMULATION mode)
* Created a simulator module in the scheduler

* Added the MyHTMLParser class (easy implementation of HTMLParser)
* Added Simulator class. It parses a file given in parametr to retrieve sequences and create a schedule
* Created a second view linked to schedule/simulation to show simulation results

* Supression du système d'agent et ajout de la tâche (celery) de scheduling

* Main update 19/05/2016 : overheads & historic system

* Deleted ScheduleHistory
* Transformed the Sequence - Schedule relation to ManyToMany
* Moved the scheduling information into the M2M relationship class (shs = ScheduleHasSequences)
* Added a static overhead into scheduling (can be set from the calling code, through scheduler.max_overhead = blabla(float))

h3. Tests

* Some tests in test.py to test the different functionalities of the scheduler. These tests include the simulation.
* A simulator at localhost:8000/scheduler/simulation, taking sequences from scheduler/sequences_cador.html

h3. Notes

* Priorities and quotas are default-calculated (for the moment)
* What is the 'flag' attribute in the Schedule model ?

h3. TODO

* views

* Link the main page to the current schedule instead of the simulation page
* Show user sequences in the schedules (with links)
* Give accces to old schedules (day / days before / all end-night plannings / all plannings)
* Give access to the refused sequences of a schedule, and the reasons of rejects

* scheduler

* Change the system to determine night start/end (they must be given in parameter, only if first_schedule is True)
* Store the reasons of rejects (create a new attribute)
* What is the 'flag' attribute ? (@AK)
* Do not create the execute_sequence tasks if it's not the night (- 120 seconds)
* Priority and quota computing
* Quotas evolution
* Blank space filling
* At the end of a scheduling send, it to the IC ?

---

h2. %{margin-left:0px; font-weight:bold; font-size:25px; display:block; color:red;}alert_manager%

h3. Purpose

* Listen to VOEvent network
* Sort interesting events
* Create requests according to these events
* Manage the requests created via the alerts
* Change the observation strategies for alerts

h3. Evolution

* Creating application

* See scheduler documentation above

* Implementation of the agent system

* Created agent.py whith a "class AlertManagerAgent(Agent):" inside
* Overriding agent's 'work' and 'shutdown' method
* Added message(s) to agent.actions_by_message dictionnary, with the functions associated to these messages
* Added the Agent's launch system in apps.py

* Implementation of the VOEventListener Thread

* Created a "class VOEventListener(Thread):" in agent.py
* Overriding the run method

* 19/05/2016 : deleted the agent system (deleted from the applications below too)

* Added some VOEvents in events_to_send

* Installed Comet (see in [[Technical components]] section)

* Created 2 celery tasks in tesks.py : alert_listener and change_obs_strategy

* Added alert_listener launch at server start

* Implemented alert_listener : it waits for an event file to be created in events_received, then read it and creates a request

* Added a simulator system ( see section Tests below )

* Added the strategy_change page

* Removed launching at start : it is now started by the monitoring task

h3. Tests

* Simulator :

* You first need to set settings.SIMULATION to True (in pyros/settings.py)
* Start celery workers (scripts/start_celery_workers.sh)
* Start server (manage.py runserver)
* Go to localhost:8000/alert_manager/simulation, and press the button !
* This will read into the simulation_sequences file.

* Test AlertListenerTestsCelery.test_alert_reception :

* Tests if the alert_listener is working well.
* +Be careful :+ this will act on the production database, and will start the entire alert workflow
* To start it : script/celery_test.sh alert_manager

* Tests TestStrategyChange.* :
* Test the changing of the observation strategy with errors and different parameters

h3. Notes

* The request created by alert_listener is hard-coded

h3. TODO

* Web :

* Print if there is an alert in progress in the main page
* Link the alerts to their status and results

* Connect to a real VOEvent broker

* Determine the communication with FSC for strategy change

* VOEvents :

* Extract the good fields (see AK Q&A 07/01/2016)
* Fill the request & alerts objects
* Use strategies to build a request
* Handle VOEvent updates
* Be careful to not create 2 alerts for a same GRB, seen by 2 different satellites

---

h2. %{margin-left:0px; font-weight:bold; font-size:25px; display:block; color:red;}analyzer%

h3. Purpose

* Apply calibration files to the images taken by the observation manager
* Analyze the images produced by the calibrations

h3. Evolution

* Creating application

* See scheduler documentation above

* Implemented the skeleton of the module in the task Analysis (call a function for calibrations then a function for analyses)

h3. Notes

h3. TODO

* Apply the calibrations in the right function
* Apply the analyses only if it's a GRB
* Implement the analyses
* Send analyses to FSC

---

h2. %{margin-left:0px; font-weight:bold; font-size:25px; display:block; color:red;}majordome%

h3. Purpose

* Handles the execution of the sequences in the right order during the night
* Handles the system start & stop
* Handles the changes of observation conditions
* Stops the celery tasks when needed

h3. Evolution

* Creating application

* See scheduler documentation above

* Created all the tasks in tasks.py

* Created the TaskManager to stop the sequence & plan executions when an alert / routine is received

* Implemented execute_sequence (not finished)

h3. Notes

h3. TODO

* TaskManager

* When a sequence is cancelled, give back the quota to the user
* In case of alert, do not stop the ongoing plan, and make the instruments abort

* execute_sequence

* Add the PLC checks at start (to see if we do the slew)
* Use the global telescope (instead of creating one here)
* Give first_schedule as false when a scheduling is launched
* Remove the default countdown (1, for tests)

* system_pause

* Abort the isntruments
* Stop the execution tasks

* system_restart

* Start a scheduling

* change_obs_conditions

* Change sequences status (if needed)
* If some status changed, re-launch a scheduling

---

h2. %{margin-left:0px; font-weight:bold; font-size:25px; display:block; color:red;}monitoring%

h3. Purpose

* It is the only tasks launched at the system's start
* Make all the starting actions : instruments configuration, connection to the PLC, updating software versions, determine night start/end & start a scheduling
* Starts the alert_listener
* Infinite loop to regularly check the instruments and PLC status (and send them to the IC)
* Handles night start & end

h3. Evolution

* Creating application

* See scheduler documentation above

* System to launch the task at the system's start

* Implement skeleton of task

* Implement starting actions (with some empty functions)

* Main loop to check PLC, instruments and night start / end (not finished)

h3. Notes

h3. TODO

* views

* Move the dashboard here
* Print the instrument status
* Print PLC informations (with the evolution)
* In the dashboard screens, put scroll on each screen to see the old logs

* Monitoring task

* Uncomment the scheduling at the beginning
* Implement night start/end computation
* Initialize communication with the instruments
* Configure intruments at start
* Send software versions to the IC
* Initialize connection with PLC
* After the starting actions, loop to wait for the instruments configuration to be finished
* Ask PLC for status
* Ask filter wheel for status
* Store the instruments & PLC status
* Send all status to IC
* Analyse PLC status (obs conditions, ...)
* Create tasks of obs condition changes

---

h2. %{margin-left:0px; font-weight:bold; font-size:25px; display:block; color:red;}observation_manager%

h3. Purpose

* Give orders the instruments to execute the plans
* Take the calibration images and computes the 'super' calibration images

h3. Evolution

* Creating application

* See scheduler documentation above

* Created the tasks : execute_plan_(nir/vis) & create_calibrations

* Implemented the execute_plan tasks with some incomplete functions

h3. Notes

h3. TODO

* execute_plan :

* Use the cameras at a global level instead of creating them here (same for the filter wheels)
* Uncomment the instruments_ready waiting function
* Uncomment the observation_ending waiting function
* Try to remove code duplication
* Determine what needs to be done at the end of an observation

* create_calibrations :

* Make the calibration images
* Generate super images
* Send them to the IC

---

h2. %{margin-left:0px; font-weight:bold; font-size:25px; display:block; color:red;}routine_manager%

h3. Purpose

* Enables astronomers to create routines via the website
* Import / export of routines (XML)
* Display observation results for routines

h3. Evolution

* Creating application

* See scheduler documentation above

* Created forms online for routine creation

* Added import / export

* Added routine submit / unsubmit

h3. Tests

* Import fake file
* Import invalid file
* Import good file
* Export incomplete
* Export valid

h3. Notes

h3. TODO

* Web

* Put the goods fields (for coordinates etc)
* Only propose the objects that matches the conditions (ex: scientific programs of the user only)
* Do all the needed checks
* Add automatic computation of JD1/JD2
* Add checkbox for JD / GD
* Add options : copy my sequence on x days, and authorise report
* Add ETC-IS simulation
* Add help for new users (and for it the first time an account come on the page)

* Do more checks at unserialization

* views

* When saving, do more checks on coordinates, jd1/2 etc
* Uncomment filter for alerts removing
* When submitting, use the monitoring to determine sequences status
* When submitting, modify the first_schedule to False, when scheduling
* When unsubmitting, uncomment the check for EXED and EXING removing
* When unsubmitting, uncomment the scheduling and change the first_schedule to False

---

h2. %{margin-left:0px; font-weight:bold; font-size:25px; display:block; color:red;}user_manager%

h3. Purpose

* Handle user create / delete / login / logout
* Allow to see user profile and make password recovery

h3. Evolution

* Creating application

* See scheduler documentation above

* Taking and adapting templates from Alexandru project

* User creation

* Form in admin.py
* Functions to verify if email doesn't exist and if passwords match
* Added check to pass login page if user is already autheticated
* Restricted access to all pyros views to authenticated users (except from user creation and login) with @login_required

h3. Tests

* Creation
* Login
* wrong email
* wrong password
* logout

h3.
Notes

h3. TODO

* Password recovery
* Profile page
* User validation by administrator / commission
* Handle permissions and access



---

h2. %{margin-left:0px; font-weight:bold; font-size:25px; display:block; color:red;}common%

h3. Purpose

* Regroups common data and functions shared by applications

h3. Content

* RequestBuilder - class to build and save in DB the requests

* You can start a request, then add sequences, albums and plans, given their parent class (ex: you need to give a sequence id to create an album)
* Each function returns the ID of the element it created (to give it to the children classes)
* To save a request, you need to use validate_request

h3. Notes

* Will probably contain the DB models at last ...

h3. TODO