Pyros applications
Version 35 (Paul Carensac, 07/07/2016 04:39 pm) → Version 36/60 (Paul Carensac, 07/07/2016 05:08 pm)
h1. Pyros applications
List and details of all the pyros applications.
{{>toc}}
---
h2. %{margin-left:0px; font-weight:bold; font-size:25px; display:block; color:red;}pyrosapp%
h3. Purpose
* Contains all the database Models
* Basic tests in tests.py
* Backoffice configuration in admin.py
h3. Notes
* Will be progressively deleted while models being displaced
---
h2. %{margin-left:0px; font-weight:bold; font-size:25px; display:block; color:red;}dashboard%
h3. Purpose
* Interface for all external users
* Leads to displays and actions for all the pyros modules (users, requests, system execution, ...)
h3. Evolution
* Creating application
* manage.py startapp dashboard
* added 'dashboard' in settings.py -> INSTALLED_APPS
* created a urls.py file in dashboard module
* added _url(r'^dashboard/', include('dashboard.urls'))_ in pyros/urls.py -> urlpatterns
* created templates/ and templates/dashboard/ folders in dashboard module
* Main page
* added 'home' view in views.py
* linked 'home' view to 8000/dashboard URL
* created a template for the homepage in templates/dashboard/ (with bootstrap)
* created views, views linking (urls.py) and templates for the dashboard modules
* redirected mainpage buttons to Admin interface (temporary)
* System page (logs)
* Retrieve logs from the db
* Print logs and automatically update every second via an ajax request
h3. Notes
* The buttons lead to the Admin interface for the moment
* Added bootstrap3 module (see Installation)
* Added Django Template Editor (see Eclipse configuration)
h3. TODO
* Create the backoffice views as the modules are integrated in pyros
* Think about a system of permissions
---
h2. %{margin-left:0px; font-weight:bold; font-size:25px; display:block; color:red;}scheduler%
h3. Purpose
* Creates the planning with the OBSERVABLE sequences
* Give acces to a web page to see the current planning
h3. Evolution
* Creating application
* manage.py startapp scheduler
* added 'scheduler' in settings.py -> INSTALLED_APPS
* created a urls.py file in scheduler module
* added _url(r'^scheduler/', include('scheduler.urls'))_ in pyros/urls.py -> urlpatterns
* created templates/ and templates/scheduler/ folders in scheduler module
* Model modifications
* Schedule
* Remove day_start
* Remove day_stop
* Add plan_start
* Add plan_end
* Enum system for the status
* ScheduleHistory
* Remove day_start
* Remove day_stop
* Add plan_start
* Add plan_end
* Sequence
* Remove exec_start
* Remove exec_stop
* Add tsp
* Add tep
* Add jd1
* Add jd2
* Add deltaTL
* Add deltaTR
* Add t_prefered
* Changed duration from Float to DecimalField (more precise)
* Add overhead
* manage.py makemigrations sheduler ; manage.py migrate
* Creation of Scheduler and Interval classes in models.py
* Implementation of the Interval class
* Implementation of the Scheduler's 'make_schedule' function (and children). This is the only entry point for now. This function creates the planning (organizes the observable sequences).
* Creation of the web interface
* Added current_schedule.html in template/scheduler folder
* Created view and url linking to this template (with current planning retrieving)
* Creation of the simulator
* Created a second entry point in the Scheduler class (with a few minor adaptations to handle SIMULATION mode)
* Created a simulator module in the scheduler
* Added the MyHTMLParser class (easy implementation of HTMLParser)
* Added Simulator class. It parses a file given in parametr to retrieve sequences and create a schedule
* Created a second view linked to schedule/simulation to show simulation results
* Supression du système d'agent et ajout de la tâche (celery) de scheduling
* Main update 19/05/2016 : overheads & historic system
* Deleted ScheduleHistory
* Transformed the Sequence - Schedule relation to ManyToMany
* Moved the scheduling information into the M2M relationship class (shs = ScheduleHasSequences)
* Added a static overhead into scheduling (can be set from the calling code, through scheduler.max_overhead = blabla(float))
h3. Tests
* Some tests in test.py to test the different functionalities of the scheduler. These tests include the simulation.
* A simulator at localhost:8000/scheduler/simulation, taking sequences from scheduler/sequences_cador.html
h3. Notes
* Priorities and quotas are default-calculated (for the moment)
* What is the 'flag' attribute in the Schedule model ?
h3. TODO
* Determine plan_start & plan_end
* Priority and quota computing
* Re-scheduling
* Blank space filling
* Change t_prefered handling (place in nearest interval instead of first-before)
---
h2. %{margin-left:0px; font-weight:bold; font-size:25px; display:block; color:red;}alert_manager%
h3. Purpose
* Listen to VOEvent network
* Sort interesting events
* Create requests according to these events
* Manage the requests created via the alerts
* Change the observation strategies for alerts
h3. Evolution
* Creating application
* See scheduler documentation above
* Implementation of the agent system
* Created agent.py whith a "class AlertManagerAgent(Agent):" inside
* Overriding agent's 'work' and 'shutdown' method
* Added message(s) to agent.actions_by_message dictionnary, with the functions associated to these messages
* Added the Agent's launch system in apps.py
* Implementation of the VOEventListener Thread
* Created a "class VOEventListener(Thread):" in agent.py
* Overriding the run method
* 19/05/2016 : deleted the agent system (deleted from the applications below too)
* Added some VOEvents in events_to_send
* Installed Comet (see in [[Technical components]] section)
* Created 2 celery tasks in tesks.py : alert_listener and change_obs_strategy
* Added alert_listener launch at server start
* Implemented alert_listener : it waits for an event file to be created in events_received, then read it and creates a request
* Added a simulator system ( see section Tests below )
* Added the strategy_change page
h3. Tests
* Simulator :
* You first need to set settings.SIMULATION to True (in pyros/settings.py)
* Start celery workers (scripts/start_celery_workers.sh)
* Start server (manage.py runserver)
* Go to localhost:8000/alert_manager/simulation, and press the button !
* This will read into the simulation_sequences file.
* Test AlertListenerTests.test_alert_reception :
* Tests if the alert_listener is working well.
* +Be careful :+ this will act on the production database, and will start the entire alert workflow
* To start it : script/celery_test.sh alert_manager
h3. Notes
* The request created by alert_listener is hard-coded
h3. TODO
* Analysis of VOEvents to create real requests
---
h2. %{margin-left:0px; font-weight:bold; font-size:25px; display:block; color:red;}analyzer%
h3. Purpose
* Apply calibration files to Analyze the images taken by the observation manager
* Analyze the images produced by the calibrations
h3. Evolution
* Creating application
* See scheduler documentation above
* Implementation of the agent system
* See alert_manager above
* 'shutdown' method is not implemented here because the Agent's default one is enough
*
h3. Notes
h3. TODO
---
h2. %{margin-left:0px; font-weight:bold; font-size:25px; display:block; color:red;}majordome%
h3. Purpose
h3. Evolution
* Creating application
* See scheduler documentation above
* Implementation of the agent system
* See alert_manager above
* Implementation of the MajordomeExecutor Thread
* Created a "class MajordomeExecutor(Thread):" in agent.py
* Overriding the run method
h3. Notes
h3. TODO
---
h2. %{margin-left:0px; font-weight:bold; font-size:25px; display:block; color:red;}monitoring%
h3. Purpose
h3. Evolution
* Creating application
* See scheduler documentation above
* Implementation of the agent system
* See alert_manager above
* 'shutdown' method is not implemented here because the Agent's default one is enough
h3. Notes
h3. TODO
---
h2. %{margin-left:0px; font-weight:bold; font-size:25px; display:block; color:red;}observation_manager%
h3. Purpose
h3. Evolution
* Creating application
* See scheduler documentation above
* Implementation of the agent system
* See alert_manager above
* Implementation of the ObservationExecutor Thread
* Created a "class ObservationExecutor(Thread):" in agent.py
* Overriding the run method
h3. Notes
h3. TODO
---
h2. %{margin-left:0px; font-weight:bold; font-size:25px; display:block; color:red;}routine_manager%
h3. Purpose
h3. Evolution
* Creating application
* See scheduler documentation above
h3. Notes
h3. TODO
---
h2. %{margin-left:0px; font-weight:bold; font-size:25px; display:block; color:red;}user_manager%
h3. Purpose
* Handle user create / delete / login / logout
* Allow to see user profile and make password recovery
h3. Evolution
* Creating application
* See scheduler documentation above
* Taking and adapting templates from Alexandru project
* User creation
* Form in admin.py
* Functions to verify if email doesn't exist and if passwords match
* Added check to pass login page if user is already autheticated
* Restricted access to all pyros views to authenticated users (except from user creation and login) with @login_required
h3. Notes
h3. TODO
* Password recovery
* Profile page
* User validation by administrator / commission
---
h2. %{margin-left:0px; font-weight:bold; font-size:25px; display:block; color:red;}common%
h3. Purpose
* Regroups common data and functions shared by applications
h3. Content
* RequestBuilder - class to build and save in DB the requests
* You can start a request, then add sequences, albums and plans, given their parent class (ex: you need to give a sequence id to create an album)
* Each function returns the ID of the element it created (to give it to the children classes)
* To save a request, you need to use validate_request
h3. Notes
* Will probably contain the DB models at last ...
h3. TODO
List and details of all the pyros applications.
{{>toc}}
---
h2. %{margin-left:0px; font-weight:bold; font-size:25px; display:block; color:red;}pyrosapp%
h3. Purpose
* Contains all the database Models
* Basic tests in tests.py
* Backoffice configuration in admin.py
h3. Notes
* Will be progressively deleted while models being displaced
---
h2. %{margin-left:0px; font-weight:bold; font-size:25px; display:block; color:red;}dashboard%
h3. Purpose
* Interface for all external users
* Leads to displays and actions for all the pyros modules (users, requests, system execution, ...)
h3. Evolution
* Creating application
* manage.py startapp dashboard
* added 'dashboard' in settings.py -> INSTALLED_APPS
* created a urls.py file in dashboard module
* added _url(r'^dashboard/', include('dashboard.urls'))_ in pyros/urls.py -> urlpatterns
* created templates/ and templates/dashboard/ folders in dashboard module
* Main page
* added 'home' view in views.py
* linked 'home' view to 8000/dashboard URL
* created a template for the homepage in templates/dashboard/ (with bootstrap)
* created views, views linking (urls.py) and templates for the dashboard modules
* redirected mainpage buttons to Admin interface (temporary)
* System page (logs)
* Retrieve logs from the db
* Print logs and automatically update every second via an ajax request
h3. Notes
* The buttons lead to the Admin interface for the moment
* Added bootstrap3 module (see Installation)
* Added Django Template Editor (see Eclipse configuration)
h3. TODO
* Create the backoffice views as the modules are integrated in pyros
* Think about a system of permissions
---
h2. %{margin-left:0px; font-weight:bold; font-size:25px; display:block; color:red;}scheduler%
h3. Purpose
* Creates the planning with the OBSERVABLE sequences
* Give acces to a web page to see the current planning
h3. Evolution
* Creating application
* manage.py startapp scheduler
* added 'scheduler' in settings.py -> INSTALLED_APPS
* created a urls.py file in scheduler module
* added _url(r'^scheduler/', include('scheduler.urls'))_ in pyros/urls.py -> urlpatterns
* created templates/ and templates/scheduler/ folders in scheduler module
* Model modifications
* Schedule
* Remove day_start
* Remove day_stop
* Add plan_start
* Add plan_end
* Enum system for the status
* ScheduleHistory
* Remove day_start
* Remove day_stop
* Add plan_start
* Add plan_end
* Sequence
* Remove exec_start
* Remove exec_stop
* Add tsp
* Add tep
* Add jd1
* Add jd2
* Add deltaTL
* Add deltaTR
* Add t_prefered
* Changed duration from Float to DecimalField (more precise)
* Add overhead
* manage.py makemigrations sheduler ; manage.py migrate
* Creation of Scheduler and Interval classes in models.py
* Implementation of the Interval class
* Implementation of the Scheduler's 'make_schedule' function (and children). This is the only entry point for now. This function creates the planning (organizes the observable sequences).
* Creation of the web interface
* Added current_schedule.html in template/scheduler folder
* Created view and url linking to this template (with current planning retrieving)
* Creation of the simulator
* Created a second entry point in the Scheduler class (with a few minor adaptations to handle SIMULATION mode)
* Created a simulator module in the scheduler
* Added the MyHTMLParser class (easy implementation of HTMLParser)
* Added Simulator class. It parses a file given in parametr to retrieve sequences and create a schedule
* Created a second view linked to schedule/simulation to show simulation results
* Supression du système d'agent et ajout de la tâche (celery) de scheduling
* Main update 19/05/2016 : overheads & historic system
* Deleted ScheduleHistory
* Transformed the Sequence - Schedule relation to ManyToMany
* Moved the scheduling information into the M2M relationship class (shs = ScheduleHasSequences)
* Added a static overhead into scheduling (can be set from the calling code, through scheduler.max_overhead = blabla(float))
h3. Tests
* Some tests in test.py to test the different functionalities of the scheduler. These tests include the simulation.
* A simulator at localhost:8000/scheduler/simulation, taking sequences from scheduler/sequences_cador.html
h3. Notes
* Priorities and quotas are default-calculated (for the moment)
* What is the 'flag' attribute in the Schedule model ?
h3. TODO
* Determine plan_start & plan_end
* Priority and quota computing
* Re-scheduling
* Blank space filling
* Change t_prefered handling (place in nearest interval instead of first-before)
---
h2. %{margin-left:0px; font-weight:bold; font-size:25px; display:block; color:red;}alert_manager%
h3. Purpose
* Listen to VOEvent network
* Sort interesting events
* Create requests according to these events
* Manage the requests created via the alerts
* Change the observation strategies for alerts
h3. Evolution
* Creating application
* See scheduler documentation above
* Implementation of the agent system
* Created agent.py whith a "class AlertManagerAgent(Agent):" inside
* Overriding agent's 'work' and 'shutdown' method
* Added message(s) to agent.actions_by_message dictionnary, with the functions associated to these messages
* Added the Agent's launch system in apps.py
* Implementation of the VOEventListener Thread
* Created a "class VOEventListener(Thread):" in agent.py
* Overriding the run method
* 19/05/2016 : deleted the agent system (deleted from the applications below too)
* Added some VOEvents in events_to_send
* Installed Comet (see in [[Technical components]] section)
* Created 2 celery tasks in tesks.py : alert_listener and change_obs_strategy
* Added alert_listener launch at server start
* Implemented alert_listener : it waits for an event file to be created in events_received, then read it and creates a request
* Added a simulator system ( see section Tests below )
* Added the strategy_change page
h3. Tests
* Simulator :
* You first need to set settings.SIMULATION to True (in pyros/settings.py)
* Start celery workers (scripts/start_celery_workers.sh)
* Start server (manage.py runserver)
* Go to localhost:8000/alert_manager/simulation, and press the button !
* This will read into the simulation_sequences file.
* Test AlertListenerTests.test_alert_reception :
* Tests if the alert_listener is working well.
* +Be careful :+ this will act on the production database, and will start the entire alert workflow
* To start it : script/celery_test.sh alert_manager
h3. Notes
* The request created by alert_listener is hard-coded
h3. TODO
* Analysis of VOEvents to create real requests
---
h2. %{margin-left:0px; font-weight:bold; font-size:25px; display:block; color:red;}analyzer%
h3. Purpose
* Apply calibration files to Analyze the images taken by the observation manager
* Analyze the images produced by the calibrations
h3. Evolution
* Creating application
* See scheduler documentation above
* Implementation of the agent system
* See alert_manager above
* 'shutdown' method is not implemented here because the Agent's default one is enough
*
h3. Notes
h3. TODO
---
h2. %{margin-left:0px; font-weight:bold; font-size:25px; display:block; color:red;}majordome%
h3. Purpose
h3. Evolution
* Creating application
* See scheduler documentation above
* Implementation of the agent system
* See alert_manager above
* Implementation of the MajordomeExecutor Thread
* Created a "class MajordomeExecutor(Thread):" in agent.py
* Overriding the run method
h3. Notes
h3. TODO
---
h2. %{margin-left:0px; font-weight:bold; font-size:25px; display:block; color:red;}monitoring%
h3. Purpose
h3. Evolution
* Creating application
* See scheduler documentation above
* Implementation of the agent system
* See alert_manager above
* 'shutdown' method is not implemented here because the Agent's default one is enough
h3. Notes
h3. TODO
---
h2. %{margin-left:0px; font-weight:bold; font-size:25px; display:block; color:red;}observation_manager%
h3. Purpose
h3. Evolution
* Creating application
* See scheduler documentation above
* Implementation of the agent system
* See alert_manager above
* Implementation of the ObservationExecutor Thread
* Created a "class ObservationExecutor(Thread):" in agent.py
* Overriding the run method
h3. Notes
h3. TODO
---
h2. %{margin-left:0px; font-weight:bold; font-size:25px; display:block; color:red;}routine_manager%
h3. Purpose
h3. Evolution
* Creating application
* See scheduler documentation above
h3. Notes
h3. TODO
---
h2. %{margin-left:0px; font-weight:bold; font-size:25px; display:block; color:red;}user_manager%
h3. Purpose
* Handle user create / delete / login / logout
* Allow to see user profile and make password recovery
h3. Evolution
* Creating application
* See scheduler documentation above
* Taking and adapting templates from Alexandru project
* User creation
* Form in admin.py
* Functions to verify if email doesn't exist and if passwords match
* Added check to pass login page if user is already autheticated
* Restricted access to all pyros views to authenticated users (except from user creation and login) with @login_required
h3. Notes
h3. TODO
* Password recovery
* Profile page
* User validation by administrator / commission
---
h2. %{margin-left:0px; font-weight:bold; font-size:25px; display:block; color:red;}common%
h3. Purpose
* Regroups common data and functions shared by applications
h3. Content
* RequestBuilder - class to build and save in DB the requests
* You can start a request, then add sequences, albums and plans, given their parent class (ex: you need to give a sequence id to create an album)
* Each function returns the ID of the element it created (to give it to the children classes)
* To save a request, you need to use validate_request
h3. Notes
* Will probably contain the DB models at last ...
h3. TODO