Pyros applications
Version 47 (Paul Carensac, 07/08/2016 11:56 am) → Version 48/60 (Paul Carensac, 07/08/2016 12:01 pm)
h1. Pyros applications
List and details of all the pyros applications.
{{>toc}}
---
h2. %{margin-left:0px; font-weight:bold; font-size:25px; display:block; color:red;}pyrosapp%
h3. Purpose
* Contains all the database Models
* Basic tests in tests.py
* Backoffice configuration in admin.py
h3. Notes
---
h2. %{margin-left:0px; font-weight:bold; font-size:25px; display:block; color:red;}dashboard%
h3. Purpose
* Interface for all external users
* Leads to displays and actions for all the pyros modules (users, requests, system execution, ...)
h3. Evolution
* Creating application
* manage.py startapp dashboard
* added 'dashboard' in settings.py -> INSTALLED_APPS
* created a urls.py file in dashboard module
* added _url(r'^dashboard/', include('dashboard.urls'))_ in pyros/urls.py -> urlpatterns
* created templates/ and templates/dashboard/ folders in dashboard module
* Main page
* added 'home' view in views.py
* linked 'home' view to 8000/dashboard URL
* created a template for the homepage in templates/dashboard/ (with bootstrap)
* created views, views linking (urls.py) and templates for the dashboard modules
* redirected mainpage buttons to Admin interface (temporary)
* System page (logs)
* Retrieve logs from the db
* Print logs and automatically update every second via an ajax request
h3. Notes
* The buttons lead to the Admin interface for the moment
* Added bootstrap3 module (see Installation)
* Added Django Template Editor (see Eclipse configuration)
h3. TODO
* Create the backoffice views as the modules are integrated in pyros
* Think about a system of permissions
---
h2. %{margin-left:0px; font-weight:bold; font-size:25px; display:block; color:red;}scheduler%
h3. Purpose
* Creates the planning with the OBSERVABLE sequences
* Give access to a web page to see the current & old plannings
h3. Evolution
* Creating application
* manage.py startapp scheduler
* added 'scheduler' in settings.py -> INSTALLED_APPS
* created a urls.py file in scheduler module
* added _url(r'^scheduler/', include('scheduler.urls'))_ in pyros/urls.py -> urlpatterns
* created templates/ and templates/scheduler/ folders in scheduler module
* Model modifications
* Schedule
* Remove day_start
* Remove day_stop
* Add plan_start
* Add plan_end
* Enum system for the status
* ScheduleHistory
* Remove day_start
* Remove day_stop
* Add plan_start
* Add plan_end
* Sequence
* Remove exec_start
* Remove exec_stop
* Add tsp
* Add tep
* Add jd1
* Add jd2
* Add deltaTL
* Add deltaTR
* Add t_prefered
* Changed duration from Float to DecimalField (more precise)
* Add overhead
* manage.py makemigrations sheduler ; manage.py migrate
* Creation of Scheduler and Interval classes in models.py
* Implementation of the Interval class
* Implementation of the Scheduler's 'make_schedule' function (and children). This is the only entry point for now. This function creates the planning (organizes the observable sequences).
* Creation of the web interface
* Added current_schedule.html in template/scheduler folder
* Created view and url linking to this template (with current planning retrieving)
* Creation of the simulator
* Created a second entry point in the Scheduler class (with a few minor adaptations to handle SIMULATION mode)
* Created a simulator module in the scheduler
* Added the MyHTMLParser class (easy implementation of HTMLParser)
* Added Simulator class. It parses a file given in parametr to retrieve sequences and create a schedule
* Created a second view linked to schedule/simulation to show simulation results
* Supression du système d'agent et ajout de la tâche (celery) de scheduling
* Main update 19/05/2016 : overheads & historic system
* Deleted ScheduleHistory
* Transformed the Sequence - Schedule relation to ManyToMany
* Moved the scheduling information into the M2M relationship class (shs = ScheduleHasSequences)
* Added a static overhead into scheduling (can be set from the calling code, through scheduler.max_overhead = blabla(float))
h3. Tests
* Some tests in test.py to test the different functionalities of the scheduler. These tests include the simulation.
* A simulator at localhost:8000/scheduler/simulation, taking sequences from scheduler/sequences_cador.html
h3. Notes
* Priorities and quotas are default-calculated (for the moment)
* What is the 'flag' attribute in the Schedule model ?
h3. TODO
* views
* Link the main page to the current schedule instead of the simulation page
* Show user sequences in the schedules (with links)
* Give accces to old schedules (day / days before / all end-night plannings / all plannings)
* Give access to the refused sequences of a schedule, and the reasons of rejects
* scheduler
* Change the system to determine night start/end (they must be given in parameter, only if first_schedule is True)
* Store the reasons of rejects (create a new attribute)
* What is the 'flag' attribute ? (@AK)
* Do not create the execute_sequence tasks if it's not the night (- 120 seconds)
* Priority and quota computing
* Quotas evolution
* Blank space filling
* At the end of a scheduling send, it to the IC ?
---
h2. %{margin-left:0px; font-weight:bold; font-size:25px; display:block; color:red;}alert_manager%
h3. Purpose
* Listen to VOEvent network
* Sort interesting events
* Create requests according to these events
* Manage the requests created via the alerts
* Change the observation strategies for alerts
h3. Evolution
* Creating application
* See scheduler documentation above
* Implementation of the agent system
* Created agent.py whith a "class AlertManagerAgent(Agent):" inside
* Overriding agent's 'work' and 'shutdown' method
* Added message(s) to agent.actions_by_message dictionnary, with the functions associated to these messages
* Added the Agent's launch system in apps.py
* Implementation of the VOEventListener Thread
* Created a "class VOEventListener(Thread):" in agent.py
* Overriding the run method
* 19/05/2016 : deleted the agent system (deleted from the applications below too)
* Added some VOEvents in events_to_send
* Installed Comet (see in [[Technical components]] section)
* Created 2 celery tasks in tesks.py : alert_listener and change_obs_strategy
* Added alert_listener launch at server start
* Implemented alert_listener : it waits for an event file to be created in events_received, then read it and creates a request
* Added a simulator system ( see section Tests below )
* Added the strategy_change page
* Removed launching at start : it is now started by the monitoring task
h3. Tests
* Simulator :
* You first need to set settings.SIMULATION to True (in pyros/settings.py)
* Start celery workers (scripts/start_celery_workers.sh)
* Start server (manage.py runserver)
* Go to localhost:8000/alert_manager/simulation, and press the button !
* This will read into the simulation_sequences file.
* Test AlertListenerTestsCelery.test_alert_reception :
* Tests if the alert_listener is working well.
* +Be careful :+ this will act on the production database, and will start the entire alert workflow
* To start it : script/celery_test.sh alert_manager
* Tests TestStrategyChange.* :
* Test the changing of the observation strategy with errors and different parameters
h3. Notes
* The request created by alert_listener is hard-coded
h3. TODO
* Web :
* Print if there is an alert in progress in the main page
* Link the alerts to their status and results
* Connect to a real VOEvent broker
* Determine the communication with FSC for strategy change
* VOEvents :
* Extract the good fields (see AK Q&A 07/01/2016)
* Fill the request & alerts objects
* Use strategies to build a request
* Handle VOEvent updates
* Be careful to not create 2 alerts for a same GRB, seen by 2 different satellites
---
h2. %{margin-left:0px; font-weight:bold; font-size:25px; display:block; color:red;}analyzer%
h3. Purpose
* Apply calibration files to the images taken by the observation manager
* Analyze the images produced by the calibrations
h3. Evolution
* Creating application
* See scheduler documentation above
* Implemented the skeleton of the module in the task Analysis (call a function for calibrations then a function for analyses)
h3. Notes
h3. TODO
* Apply the calibrations in the right function
* Apply the analyses only if it's a GRB
* Implement the analyses
* Send analyses to FSC
---
h2. %{margin-left:0px; font-weight:bold; font-size:25px; display:block; color:red;}majordome%
h3. Purpose
* Handles the execution of the sequences in the right order during the night
* Handles the system start & stop
* Handles the changes of observation conditions
* Stops the celery tasks when needed
h3. Evolution
* Creating application
* See scheduler documentation above
* Created all the tasks in tasks.py
* Created the TaskManager to stop the sequence & plan executions when an alert / routine is received
* Implemented execute_sequence (not finished)
h3. Notes
h3. TODO
* TaskManager
* When a sequence is cancelled, give back the quota to the user
* In case of alert, do not stop the ongoing plan, and make the instruments abort
* execute_sequence
* Add the PLC checks at start (to see if we do the slew)
* Use the global telescope (instead of creating one here)
* Give first_schedule as false when a scheduling is launched
* Remove the default countdown (1, for tests)
* system_pause
* Abort the isntruments
* Stop the execution tasks
* system_restart
* Start a scheduling
* change_obs_conditions
* Change sequences status (if needed)
* If some status changed, re-launch a scheduling
---
h2. %{margin-left:0px; font-weight:bold; font-size:25px; display:block; color:red;}monitoring%
h3. Purpose
* It is the only tasks launched at the system's start
* Make all the starting actions : instruments configuration, connection to the PLC, updating software versions, determine night start/end & start a scheduling
* Starts the alert_listener
* Infinite loop to regularly check the instruments and PLC status (and send them to the IC)
* Handles night start & end
h3. Evolution
* Creating application
* See scheduler documentation above
* System to launch the task at the system's start
* Implement skeleton of task
* Implement starting actions (with some empty functions)
* Main loop to check PLC, instruments and night start / end (not finished)
h3. Notes
h3. TODO
* views
* Move the dashboard here
* Print the instrument status
* Print PLC informations (with the evolution)
* In the dashboard screens, put scroll on each screen to see the old logs
* Monitoring task
* Uncomment the scheduling at the beginning
* Implement night start/end computation
* Initialize communication with the instruments
* Configure intruments at start
* Send software versions to the IC
* Initialize connection with PLC
* After the starting actions, loop to wait for the instruments configuration to be finished
* Ask PLC for status
* Ask filter wheel for status
* Store the instruments & PLC status
* Send all status to IC
* Analyse PLC status (obs conditions, ...)
* Create tasks of obs condition changes
---
h2. %{margin-left:0px; font-weight:bold; font-size:25px; display:block; color:red;}observation_manager%
h3. Purpose
h3. Evolution
* Creating application
* See scheduler documentation above
* Implementation of the agent system
* See alert_manager above
* Implementation of the ObservationExecutor Thread
* Created a "class ObservationExecutor(Thread):" in agent.py
* Overriding the run method
h3. Notes
h3. TODO
---
h2. %{margin-left:0px; font-weight:bold; font-size:25px; display:block; color:red;}routine_manager%
h3. Purpose
h3. Evolution
* Creating application
* See scheduler documentation above
h3. Notes
h3. TODO
---
h2. %{margin-left:0px; font-weight:bold; font-size:25px; display:block; color:red;}user_manager%
h3. Purpose
* Handle user create / delete / login / logout
* Allow to see user profile and make password recovery
h3. Evolution
* Creating application
* See scheduler documentation above
* Taking and adapting templates from Alexandru project
* User creation
* Form in admin.py
* Functions to verify if email doesn't exist and if passwords match
* Added check to pass login page if user is already autheticated
* Restricted access to all pyros views to authenticated users (except from user creation and login) with @login_required
h3. Notes
h3. TODO
* Password recovery
* Profile page
* User validation by administrator / commission
---
h2. %{margin-left:0px; font-weight:bold; font-size:25px; display:block; color:red;}common%
h3. Purpose
* Regroups common data and functions shared by applications
h3. Content
* RequestBuilder - class to build and save in DB the requests
* You can start a request, then add sequences, albums and plans, given their parent class (ex: you need to give a sequence id to create an album)
* Each function returns the ID of the element it created (to give it to the children classes)
* To save a request, you need to use validate_request
h3. Notes
* Will probably contain the DB models at last ...
h3. TODO
List and details of all the pyros applications.
{{>toc}}
---
h2. %{margin-left:0px; font-weight:bold; font-size:25px; display:block; color:red;}pyrosapp%
h3. Purpose
* Contains all the database Models
* Basic tests in tests.py
* Backoffice configuration in admin.py
h3. Notes
---
h2. %{margin-left:0px; font-weight:bold; font-size:25px; display:block; color:red;}dashboard%
h3. Purpose
* Interface for all external users
* Leads to displays and actions for all the pyros modules (users, requests, system execution, ...)
h3. Evolution
* Creating application
* manage.py startapp dashboard
* added 'dashboard' in settings.py -> INSTALLED_APPS
* created a urls.py file in dashboard module
* added _url(r'^dashboard/', include('dashboard.urls'))_ in pyros/urls.py -> urlpatterns
* created templates/ and templates/dashboard/ folders in dashboard module
* Main page
* added 'home' view in views.py
* linked 'home' view to 8000/dashboard URL
* created a template for the homepage in templates/dashboard/ (with bootstrap)
* created views, views linking (urls.py) and templates for the dashboard modules
* redirected mainpage buttons to Admin interface (temporary)
* System page (logs)
* Retrieve logs from the db
* Print logs and automatically update every second via an ajax request
h3. Notes
* The buttons lead to the Admin interface for the moment
* Added bootstrap3 module (see Installation)
* Added Django Template Editor (see Eclipse configuration)
h3. TODO
* Create the backoffice views as the modules are integrated in pyros
* Think about a system of permissions
---
h2. %{margin-left:0px; font-weight:bold; font-size:25px; display:block; color:red;}scheduler%
h3. Purpose
* Creates the planning with the OBSERVABLE sequences
* Give access to a web page to see the current & old plannings
h3. Evolution
* Creating application
* manage.py startapp scheduler
* added 'scheduler' in settings.py -> INSTALLED_APPS
* created a urls.py file in scheduler module
* added _url(r'^scheduler/', include('scheduler.urls'))_ in pyros/urls.py -> urlpatterns
* created templates/ and templates/scheduler/ folders in scheduler module
* Model modifications
* Schedule
* Remove day_start
* Remove day_stop
* Add plan_start
* Add plan_end
* Enum system for the status
* ScheduleHistory
* Remove day_start
* Remove day_stop
* Add plan_start
* Add plan_end
* Sequence
* Remove exec_start
* Remove exec_stop
* Add tsp
* Add tep
* Add jd1
* Add jd2
* Add deltaTL
* Add deltaTR
* Add t_prefered
* Changed duration from Float to DecimalField (more precise)
* Add overhead
* manage.py makemigrations sheduler ; manage.py migrate
* Creation of Scheduler and Interval classes in models.py
* Implementation of the Interval class
* Implementation of the Scheduler's 'make_schedule' function (and children). This is the only entry point for now. This function creates the planning (organizes the observable sequences).
* Creation of the web interface
* Added current_schedule.html in template/scheduler folder
* Created view and url linking to this template (with current planning retrieving)
* Creation of the simulator
* Created a second entry point in the Scheduler class (with a few minor adaptations to handle SIMULATION mode)
* Created a simulator module in the scheduler
* Added the MyHTMLParser class (easy implementation of HTMLParser)
* Added Simulator class. It parses a file given in parametr to retrieve sequences and create a schedule
* Created a second view linked to schedule/simulation to show simulation results
* Supression du système d'agent et ajout de la tâche (celery) de scheduling
* Main update 19/05/2016 : overheads & historic system
* Deleted ScheduleHistory
* Transformed the Sequence - Schedule relation to ManyToMany
* Moved the scheduling information into the M2M relationship class (shs = ScheduleHasSequences)
* Added a static overhead into scheduling (can be set from the calling code, through scheduler.max_overhead = blabla(float))
h3. Tests
* Some tests in test.py to test the different functionalities of the scheduler. These tests include the simulation.
* A simulator at localhost:8000/scheduler/simulation, taking sequences from scheduler/sequences_cador.html
h3. Notes
* Priorities and quotas are default-calculated (for the moment)
* What is the 'flag' attribute in the Schedule model ?
h3. TODO
* views
* Link the main page to the current schedule instead of the simulation page
* Show user sequences in the schedules (with links)
* Give accces to old schedules (day / days before / all end-night plannings / all plannings)
* Give access to the refused sequences of a schedule, and the reasons of rejects
* scheduler
* Change the system to determine night start/end (they must be given in parameter, only if first_schedule is True)
* Store the reasons of rejects (create a new attribute)
* What is the 'flag' attribute ? (@AK)
* Do not create the execute_sequence tasks if it's not the night (- 120 seconds)
* Priority and quota computing
* Quotas evolution
* Blank space filling
* At the end of a scheduling send, it to the IC ?
---
h2. %{margin-left:0px; font-weight:bold; font-size:25px; display:block; color:red;}alert_manager%
h3. Purpose
* Listen to VOEvent network
* Sort interesting events
* Create requests according to these events
* Manage the requests created via the alerts
* Change the observation strategies for alerts
h3. Evolution
* Creating application
* See scheduler documentation above
* Implementation of the agent system
* Created agent.py whith a "class AlertManagerAgent(Agent):" inside
* Overriding agent's 'work' and 'shutdown' method
* Added message(s) to agent.actions_by_message dictionnary, with the functions associated to these messages
* Added the Agent's launch system in apps.py
* Implementation of the VOEventListener Thread
* Created a "class VOEventListener(Thread):" in agent.py
* Overriding the run method
* 19/05/2016 : deleted the agent system (deleted from the applications below too)
* Added some VOEvents in events_to_send
* Installed Comet (see in [[Technical components]] section)
* Created 2 celery tasks in tesks.py : alert_listener and change_obs_strategy
* Added alert_listener launch at server start
* Implemented alert_listener : it waits for an event file to be created in events_received, then read it and creates a request
* Added a simulator system ( see section Tests below )
* Added the strategy_change page
* Removed launching at start : it is now started by the monitoring task
h3. Tests
* Simulator :
* You first need to set settings.SIMULATION to True (in pyros/settings.py)
* Start celery workers (scripts/start_celery_workers.sh)
* Start server (manage.py runserver)
* Go to localhost:8000/alert_manager/simulation, and press the button !
* This will read into the simulation_sequences file.
* Test AlertListenerTestsCelery.test_alert_reception :
* Tests if the alert_listener is working well.
* +Be careful :+ this will act on the production database, and will start the entire alert workflow
* To start it : script/celery_test.sh alert_manager
* Tests TestStrategyChange.* :
* Test the changing of the observation strategy with errors and different parameters
h3. Notes
* The request created by alert_listener is hard-coded
h3. TODO
* Web :
* Print if there is an alert in progress in the main page
* Link the alerts to their status and results
* Connect to a real VOEvent broker
* Determine the communication with FSC for strategy change
* VOEvents :
* Extract the good fields (see AK Q&A 07/01/2016)
* Fill the request & alerts objects
* Use strategies to build a request
* Handle VOEvent updates
* Be careful to not create 2 alerts for a same GRB, seen by 2 different satellites
---
h2. %{margin-left:0px; font-weight:bold; font-size:25px; display:block; color:red;}analyzer%
h3. Purpose
* Apply calibration files to the images taken by the observation manager
* Analyze the images produced by the calibrations
h3. Evolution
* Creating application
* See scheduler documentation above
* Implemented the skeleton of the module in the task Analysis (call a function for calibrations then a function for analyses)
h3. Notes
h3. TODO
* Apply the calibrations in the right function
* Apply the analyses only if it's a GRB
* Implement the analyses
* Send analyses to FSC
---
h2. %{margin-left:0px; font-weight:bold; font-size:25px; display:block; color:red;}majordome%
h3. Purpose
* Handles the execution of the sequences in the right order during the night
* Handles the system start & stop
* Handles the changes of observation conditions
* Stops the celery tasks when needed
h3. Evolution
* Creating application
* See scheduler documentation above
* Created all the tasks in tasks.py
* Created the TaskManager to stop the sequence & plan executions when an alert / routine is received
* Implemented execute_sequence (not finished)
h3. Notes
h3. TODO
* TaskManager
* When a sequence is cancelled, give back the quota to the user
* In case of alert, do not stop the ongoing plan, and make the instruments abort
* execute_sequence
* Add the PLC checks at start (to see if we do the slew)
* Use the global telescope (instead of creating one here)
* Give first_schedule as false when a scheduling is launched
* Remove the default countdown (1, for tests)
* system_pause
* Abort the isntruments
* Stop the execution tasks
* system_restart
* Start a scheduling
* change_obs_conditions
* Change sequences status (if needed)
* If some status changed, re-launch a scheduling
---
h2. %{margin-left:0px; font-weight:bold; font-size:25px; display:block; color:red;}monitoring%
h3. Purpose
* It is the only tasks launched at the system's start
* Make all the starting actions : instruments configuration, connection to the PLC, updating software versions, determine night start/end & start a scheduling
* Starts the alert_listener
* Infinite loop to regularly check the instruments and PLC status (and send them to the IC)
* Handles night start & end
h3. Evolution
* Creating application
* See scheduler documentation above
* System to launch the task at the system's start
* Implement skeleton of task
* Implement starting actions (with some empty functions)
* Main loop to check PLC, instruments and night start / end (not finished)
h3. Notes
h3. TODO
* views
* Move the dashboard here
* Print the instrument status
* Print PLC informations (with the evolution)
* In the dashboard screens, put scroll on each screen to see the old logs
* Monitoring task
* Uncomment the scheduling at the beginning
* Implement night start/end computation
* Initialize communication with the instruments
* Configure intruments at start
* Send software versions to the IC
* Initialize connection with PLC
* After the starting actions, loop to wait for the instruments configuration to be finished
* Ask PLC for status
* Ask filter wheel for status
* Store the instruments & PLC status
* Send all status to IC
* Analyse PLC status (obs conditions, ...)
* Create tasks of obs condition changes
---
h2. %{margin-left:0px; font-weight:bold; font-size:25px; display:block; color:red;}observation_manager%
h3. Purpose
h3. Evolution
* Creating application
* See scheduler documentation above
* Implementation of the agent system
* See alert_manager above
* Implementation of the ObservationExecutor Thread
* Created a "class ObservationExecutor(Thread):" in agent.py
* Overriding the run method
h3. Notes
h3. TODO
---
h2. %{margin-left:0px; font-weight:bold; font-size:25px; display:block; color:red;}routine_manager%
h3. Purpose
h3. Evolution
* Creating application
* See scheduler documentation above
h3. Notes
h3. TODO
---
h2. %{margin-left:0px; font-weight:bold; font-size:25px; display:block; color:red;}user_manager%
h3. Purpose
* Handle user create / delete / login / logout
* Allow to see user profile and make password recovery
h3. Evolution
* Creating application
* See scheduler documentation above
* Taking and adapting templates from Alexandru project
* User creation
* Form in admin.py
* Functions to verify if email doesn't exist and if passwords match
* Added check to pass login page if user is already autheticated
* Restricted access to all pyros views to authenticated users (except from user creation and login) with @login_required
h3. Notes
h3. TODO
* Password recovery
* Profile page
* User validation by administrator / commission
---
h2. %{margin-left:0px; font-weight:bold; font-size:25px; display:block; color:red;}common%
h3. Purpose
* Regroups common data and functions shared by applications
h3. Content
* RequestBuilder - class to build and save in DB the requests
* You can start a request, then add sequences, albums and plans, given their parent class (ex: you need to give a sequence id to create an album)
* Each function returns the ID of the element it created (to give it to the children classes)
* To save a request, you need to use validate_request
h3. Notes
* Will probably contain the DB models at last ...
h3. TODO