History

Wiki » Project Development »

« Previous - Version 45/60 (diff) - Next » - Current version
Paul Carensac, 07/08/2016 11:42 am

Pyros applications

List and details of all the pyros applications.

pyrosapp

Purpose

  • Contains all the database Models
  • Basic tests in tests.py
  • Backoffice configuration in admin.py

Notes

dashboard

Purpose

  • Interface for all external users
  • Leads to displays and actions for all the pyros modules (users, requests, system execution, ...)

Evolution

  • Creating application
    • manage.py startapp dashboard
    • added 'dashboard' in settings.py -> INSTALLED_APPS
    • created a urls.py file in dashboard module
    • added url(r'^dashboard/', include('dashboard.urls')) in pyros/urls.py -> urlpatterns
    • created templates/ and templates/dashboard/ folders in dashboard module
  • Main page
    • added 'home' view in views.py
    • linked 'home' view to 8000/dashboard URL
    • created a template for the homepage in templates/dashboard/ (with bootstrap)
    • created views, views linking (urls.py) and templates for the dashboard modules
    • redirected mainpage buttons to Admin interface (temporary)
  • System page (logs)
    • Retrieve logs from the db
    • Print logs and automatically update every second via an ajax request

Notes

  • The buttons lead to the Admin interface for the moment
  • Added bootstrap3 module (see Installation)
  • Added Django Template Editor (see Eclipse configuration)

TODO

  • Create the backoffice views as the modules are integrated in pyros
  • Think about a system of permissions

scheduler

Purpose

  • Creates the planning with the OBSERVABLE sequences
  • Give access to a web page to see the current & old plannings

Evolution

  • Creating application
    • manage.py startapp scheduler
    • added 'scheduler' in settings.py -> INSTALLED_APPS
    • created a urls.py file in scheduler module
    • added url(r'^scheduler/', include('scheduler.urls')) in pyros/urls.py -> urlpatterns
    • created templates/ and templates/scheduler/ folders in scheduler module
  • Model modifications
    • Schedule
      • Remove day_start
      • Remove day_stop
      • Add plan_start
      • Add plan_end
      • Enum system for the status
    • ScheduleHistory
      • Remove day_start
      • Remove day_stop
      • Add plan_start
      • Add plan_end
    • Sequence
      • Remove exec_start
      • Remove exec_stop
      • Add tsp
      • Add tep
      • Add jd1
      • Add jd2
      • Add deltaTL
      • Add deltaTR
      • Add t_prefered
      • Changed duration from Float to DecimalField (more precise)
      • Add overhead
    • manage.py makemigrations sheduler ; manage.py migrate
  • Creation of Scheduler and Interval classes in models.py
  • Implementation of the Interval class
  • Implementation of the Scheduler's 'make_schedule' function (and children). This is the only entry point for now. This function creates the planning (organizes the observable sequences).
  • Creation of the web interface
    • Added current_schedule.html in template/scheduler folder
    • Created view and url linking to this template (with current planning retrieving)
  • Creation of the simulator
    • Created a second entry point in the Scheduler class (with a few minor adaptations to handle SIMULATION mode)
    • Created a simulator module in the scheduler
      • Added the MyHTMLParser class (easy implementation of HTMLParser)
      • Added Simulator class. It parses a file given in parametr to retrieve sequences and create a schedule
    • Created a second view linked to schedule/simulation to show simulation results
  • Supression du système d'agent et ajout de la tâche (celery) de scheduling
  • Main update 19/05/2016 : overheads & historic system
    • Deleted ScheduleHistory
    • Transformed the Sequence - Schedule relation to ManyToMany
    • Moved the scheduling information into the M2M relationship class (shs = ScheduleHasSequences)
    • Added a static overhead into scheduling (can be set from the calling code, through scheduler.max_overhead = blabla(float))

Tests

  • Some tests in test.py to test the different functionalities of the scheduler. These tests include the simulation.
  • A simulator at localhost:8000/scheduler/simulation, taking sequences from scheduler/sequences_cador.html

Notes

  • Priorities and quotas are default-calculated (for the moment)
  • What is the 'flag' attribute in the Schedule model ?

TODO

  • views
    • Link the main page to the current schedule instead of the simulation page
    • Show user sequences in the schedules (with links)
    • Give accces to old schedules (day / days before / all end-night plannings / all plannings)
    • Give access to the refused sequences of a schedule, and the reasons of rejects
  • scheduler
    • Change the system to determine night start/end (they must be given in parameter, only if first_schedule is True)
    • Store the reasons of rejects (create a new attribute)
    • What is the 'flag' attribute ? (@AK)
    • Do not create the execute_sequence tasks if it's not the night (- 120 seconds)
    • Priority and quota computing
    • Quotas evolution
    • Blank space filling
    • At the end of a scheduling send, it to the IC ?

alert_manager

Purpose

  • Listen to VOEvent network
  • Sort interesting events
  • Create requests according to these events
  • Manage the requests created via the alerts
  • Change the observation strategies for alerts

Evolution

  • Creating application
    • See scheduler documentation above
  • Implementation of the agent system
    • Created agent.py whith a "class AlertManagerAgent(Agent):" inside
    • Overriding agent's 'work' and 'shutdown' method
    • Added message(s) to agent.actions_by_message dictionnary, with the functions associated to these messages
    • Added the Agent's launch system in apps.py
  • Implementation of the VOEventListener Thread
    • Created a "class VOEventListener(Thread):" in agent.py
    • Overriding the run method
  • 19/05/2016 : deleted the agent system (deleted from the applications below too)
  • Added some VOEvents in events_to_send
  • Created 2 celery tasks in tesks.py : alert_listener and change_obs_strategy
  • Added alert_listener launch at server start
  • Implemented alert_listener : it waits for an event file to be created in events_received, then read it and creates a request
  • Added a simulator system ( see section Tests below )
  • Added the strategy_change page
  • Removed launching at start : it is now started by the monitoring task

Tests

  • Simulator :
    • You first need to set settings.SIMULATION to True (in pyros/settings.py)
    • Start celery workers (scripts/start_celery_workers.sh)
    • Start server (manage.py runserver)
    • Go to localhost:8000/alert_manager/simulation, and press the button !
    • This will read into the simulation_sequences file.
  • Test AlertListenerTestsCelery.test_alert_reception :
    • Tests if the alert_listener is working well.
    • Be careful : this will act on the production database, and will start the entire alert workflow
    • To start it : script/celery_test.sh alert_manager
  • Tests TestStrategyChange.* : * Test the changing of the observation strategy with errors and different parameters

Notes

  • The request created by alert_listener is hard-coded

TODO

  • Web :
    • Print if there is an alert in progress in the main page
    • Link the alerts to their status and results
  • Connect to a real VOEvent broker
  • Determine the communication with FSC for strategy change
  • VOEvents :
    • Extract the good fields (see AK Q&A 07/01/2016)
    • Fill the request & alerts objects
    • Use strategies to build a request
    • Handle VOEvent updates
    • Be careful to not create 2 alerts for a same GRB, seen by 2 different satellites

analyzer

Purpose

  • Apply calibration files to the images taken by the observation manager
  • Analyze the images produced by the calibrations

Evolution

  • Creating application
    • See scheduler documentation above
  • Implemented the skeleton of the module in the task Analysis (call a function for calibrations then a function for analyses)

Notes

TODO

  • Apply the calibrations in the right function
  • Apply the analyses only if it's a GRB
  • Implement the analyses
  • Send analyses to FSC

majordome

Purpose

  • Handles the execution of the sequences in the right order during the night
  • Handles the system start & stop
  • Handles the changes of observation conditions

Evolution

  • Creating application
    • See scheduler documentation above
  • Implementation of the MajordomeExecutor Thread
    • Created a "class MajordomeExecutor(Thread):" in agent.py
    • Overriding the run method

Notes

TODO

monitoring

Purpose

Evolution

  • Creating application
    • See scheduler documentation above
  • Implementation of the agent system
    • See alert_manager above
    • 'shutdown' method is not implemented here because the Agent's default one is enough

Notes

TODO

observation_manager

Purpose

Evolution

  • Creating application
    • See scheduler documentation above
  • Implementation of the agent system
    • See alert_manager above
  • Implementation of the ObservationExecutor Thread
    • Created a "class ObservationExecutor(Thread):" in agent.py
    • Overriding the run method

Notes

TODO

routine_manager

Purpose

Evolution

  • Creating application
    • See scheduler documentation above

Notes

TODO

user_manager

Purpose

  • Handle user create / delete / login / logout
  • Allow to see user profile and make password recovery

Evolution

  • Creating application
    • See scheduler documentation above
  • Taking and adapting templates from Alexandru project
  • User creation
    • Form in admin.py
    • Functions to verify if email doesn't exist and if passwords match
    • Added check to pass login page if user is already autheticated
    • Restricted access to all pyros views to authenticated users (except from user creation and login) with @login_required

Notes

TODO

  • Password recovery
  • Profile page
  • User validation by administrator / commission

common

Purpose

  • Regroups common data and functions shared by applications

Content

  • RequestBuilder - class to build and save in DB the requests
    • You can start a request, then add sequences, albums and plans, given their parent class (ex: you need to give a sequence id to create an album)
    • Each function returns the ID of the element it created (to give it to the children classes)
    • To save a request, you need to use validate_request

Notes

  • Will probably contain the DB models at last ...

TODO