History

Wiki » Project Development »

« Previous - Version 52/60 (diff) - Next » - Current version
Paul Carensac, 07/08/2016 12:29 pm

Pyros applications

List and details of all the pyros applications.

pyrosapp

Purpose

  • Contains all the database Models
  • Basic tests in tests.py
  • Backoffice configuration in admin.py

Notes

dashboard

Purpose

  • Interface for all external users
  • Leads to displays and actions for all the pyros modules (users, requests, system execution, ...)

Evolution

  • Creating application
    • manage.py startapp dashboard
    • added 'dashboard' in settings.py -> INSTALLED_APPS
    • created a urls.py file in dashboard module
    • added url(r'^dashboard/', include('dashboard.urls')) in pyros/urls.py -> urlpatterns
    • created templates/ and templates/dashboard/ folders in dashboard module
  • Main page
    • added 'home' view in views.py
    • linked 'home' view to 8000/dashboard URL
    • created a template for the homepage in templates/dashboard/ (with bootstrap)
    • created views, views linking (urls.py) and templates for the dashboard modules
    • redirected mainpage buttons to Admin interface (temporary)
  • System page (logs)
    • Retrieve logs from the db
    • Print logs and automatically update every second via an ajax request

Notes

  • The buttons lead to the Admin interface for the moment
  • Added bootstrap3 module (see Installation)
  • Added Django Template Editor (see Eclipse configuration)

TODO

  • Create the backoffice views as the modules are integrated in pyros
  • Think about a system of permissions

scheduler

Purpose

  • Creates the planning with the OBSERVABLE sequences
  • Give access to a web page to see the current & old plannings

Evolution

  • Creating application
    • manage.py startapp scheduler
    • added 'scheduler' in settings.py -> INSTALLED_APPS
    • created a urls.py file in scheduler module
    • added url(r'^scheduler/', include('scheduler.urls')) in pyros/urls.py -> urlpatterns
    • created templates/ and templates/scheduler/ folders in scheduler module
  • Model modifications
    • Schedule
      • Remove day_start
      • Remove day_stop
      • Add plan_start
      • Add plan_end
      • Enum system for the status
    • ScheduleHistory
      • Remove day_start
      • Remove day_stop
      • Add plan_start
      • Add plan_end
    • Sequence
      • Remove exec_start
      • Remove exec_stop
      • Add tsp
      • Add tep
      • Add jd1
      • Add jd2
      • Add deltaTL
      • Add deltaTR
      • Add t_prefered
      • Changed duration from Float to DecimalField (more precise)
      • Add overhead
    • manage.py makemigrations sheduler ; manage.py migrate
  • Creation of Scheduler and Interval classes in models.py
  • Implementation of the Interval class
  • Implementation of the Scheduler's 'make_schedule' function (and children). This is the only entry point for now. This function creates the planning (organizes the observable sequences).
  • Creation of the web interface
    • Added current_schedule.html in template/scheduler folder
    • Created view and url linking to this template (with current planning retrieving)
  • Creation of the simulator
    • Created a second entry point in the Scheduler class (with a few minor adaptations to handle SIMULATION mode)
    • Created a simulator module in the scheduler
      • Added the MyHTMLParser class (easy implementation of HTMLParser)
      • Added Simulator class. It parses a file given in parametr to retrieve sequences and create a schedule
    • Created a second view linked to schedule/simulation to show simulation results
  • Supression du système d'agent et ajout de la tâche (celery) de scheduling
  • Main update 19/05/2016 : overheads & historic system
    • Deleted ScheduleHistory
    • Transformed the Sequence - Schedule relation to ManyToMany
    • Moved the scheduling information into the M2M relationship class (shs = ScheduleHasSequences)
    • Added a static overhead into scheduling (can be set from the calling code, through scheduler.max_overhead = blabla(float))

Tests

  • Some tests in test.py to test the different functionalities of the scheduler. These tests include the simulation.
  • A simulator at localhost:8000/scheduler/simulation, taking sequences from scheduler/sequences_cador.html

Notes

  • Priorities and quotas are default-calculated (for the moment)
  • What is the 'flag' attribute in the Schedule model ?

TODO

  • views
    • Link the main page to the current schedule instead of the simulation page
    • Show user sequences in the schedules (with links)
    • Give accces to old schedules (day / days before / all end-night plannings / all plannings)
    • Give access to the refused sequences of a schedule, and the reasons of rejects
  • scheduler
    • Change the system to determine night start/end (they must be given in parameter, only if first_schedule is True)
    • Store the reasons of rejects (create a new attribute)
    • What is the 'flag' attribute ? (@AK)
    • Do not create the execute_sequence tasks if it's not the night (- 120 seconds)
    • Priority and quota computing
    • Quotas evolution
    • Blank space filling
    • At the end of a scheduling send, it to the IC ?

alert_manager

Purpose

  • Listen to VOEvent network
  • Sort interesting events
  • Create requests according to these events
  • Manage the requests created via the alerts
  • Change the observation strategies for alerts

Evolution

  • Creating application
    • See scheduler documentation above
  • Implementation of the agent system
    • Created agent.py whith a "class AlertManagerAgent(Agent):" inside
    • Overriding agent's 'work' and 'shutdown' method
    • Added message(s) to agent.actions_by_message dictionnary, with the functions associated to these messages
    • Added the Agent's launch system in apps.py
  • Implementation of the VOEventListener Thread
    • Created a "class VOEventListener(Thread):" in agent.py
    • Overriding the run method
  • 19/05/2016 : deleted the agent system (deleted from the applications below too)
  • Added some VOEvents in events_to_send
  • Created 2 celery tasks in tesks.py : alert_listener and change_obs_strategy
  • Added alert_listener launch at server start
  • Implemented alert_listener : it waits for an event file to be created in events_received, then read it and creates a request
  • Added a simulator system ( see section Tests below )
  • Added the strategy_change page
  • Removed launching at start : it is now started by the monitoring task

Tests

  • Simulator :
    • You first need to set settings.SIMULATION to True (in pyros/settings.py)
    • Start celery workers (scripts/start_celery_workers.sh)
    • Start server (manage.py runserver)
    • Go to localhost:8000/alert_manager/simulation, and press the button !
    • This will read into the simulation_sequences file.
  • Test AlertListenerTestsCelery.test_alert_reception :
    • Tests if the alert_listener is working well.
    • Be careful : this will act on the production database, and will start the entire alert workflow
    • To start it : script/celery_test.sh alert_manager
  • Tests TestStrategyChange.* : * Test the changing of the observation strategy with errors and different parameters

Notes

  • The request created by alert_listener is hard-coded

TODO

  • Web :
    • Print if there is an alert in progress in the main page
    • Link the alerts to their status and results
  • Connect to a real VOEvent broker
  • Determine the communication with FSC for strategy change
  • VOEvents :
    • Extract the good fields (see AK Q&A 07/01/2016)
    • Fill the request & alerts objects
    • Use strategies to build a request
    • Handle VOEvent updates
    • Be careful to not create 2 alerts for a same GRB, seen by 2 different satellites

analyzer

Purpose

  • Apply calibration files to the images taken by the observation manager
  • Analyze the images produced by the calibrations

Evolution

  • Creating application
    • See scheduler documentation above
  • Implemented the skeleton of the module in the task Analysis (call a function for calibrations then a function for analyses)

Notes

TODO

  • Apply the calibrations in the right function
  • Apply the analyses only if it's a GRB
  • Implement the analyses
  • Send analyses to FSC

majordome

Purpose

  • Handles the execution of the sequences in the right order during the night
  • Handles the system start & stop
  • Handles the changes of observation conditions
  • Stops the celery tasks when needed

Evolution

  • Creating application
    • See scheduler documentation above
    • Created all the tasks in tasks.py
    • Created the TaskManager to stop the sequence & plan executions when an alert / routine is received
    • Implemented execute_sequence (not finished)

Notes

TODO

  • TaskManager
    • When a sequence is cancelled, give back the quota to the user
    • In case of alert, do not stop the ongoing plan, and make the instruments abort
  • execute_sequence
    • Add the PLC checks at start (to see if we do the slew)
    • Use the global telescope (instead of creating one here)
    • Give first_schedule as false when a scheduling is launched
    • Remove the default countdown (1, for tests)
  • system_pause
    • Abort the isntruments
    • Stop the execution tasks
  • system_restart
    • Start a scheduling
  • change_obs_conditions
    • Change sequences status (if needed)
    • If some status changed, re-launch a scheduling

monitoring

Purpose

  • It is the only tasks launched at the system's start
  • Make all the starting actions : instruments configuration, connection to the PLC, updating software versions, determine night start/end & start a scheduling
  • Starts the alert_listener
  • Infinite loop to regularly check the instruments and PLC status (and send them to the IC)
  • Handles night start & end

Evolution

  • Creating application
    • See scheduler documentation above
  • System to launch the task at the system's start
  • Implement skeleton of task
  • Implement starting actions (with some empty functions)
  • Main loop to check PLC, instruments and night start / end (not finished)

Notes

TODO

  • views
    • Move the dashboard here
    • Print the instrument status
    • Print PLC informations (with the evolution)
    • In the dashboard screens, put scroll on each screen to see the old logs
  • Monitoring task
    • Uncomment the scheduling at the beginning
    • Implement night start/end computation
    • Initialize communication with the instruments
    • Configure intruments at start
    • Send software versions to the IC
    • Initialize connection with PLC
    • After the starting actions, loop to wait for the instruments configuration to be finished
    • Ask PLC for status
    • Ask filter wheel for status
    • Store the instruments & PLC status
    • Send all status to IC
    • Analyse PLC status (obs conditions, ...)
    • Create tasks of obs condition changes

observation_manager

Purpose

  • Give orders the instruments to execute the plans
  • Take the calibration images and computes the 'super' calibration images

Evolution

  • Creating application
    • See scheduler documentation above
  • Created the tasks : execute_plan_(nir/vis) & create_calibrations
  • Implemented the execute_plan tasks with some incomplete functions

Notes

TODO

  • execute_plan :
    • Use the cameras at a global level instead of creating them here (same for the filter wheels)
    • Uncomment the instruments_ready waiting function
    • Uncomment the observation_ending waiting function
    • Try to remove code duplication
    • Determine what needs to be done at the end of an observation
  • create_calibrations :
    • Make the calibration images
    • Generate super images
    • Send them to the IC

routine_manager

Purpose

  • Enables astronomers to create routines via the website
  • Import / export of routines (XML)
  • Display observation results for routines

Evolution

  • Creating application
    • See scheduler documentation above
  • Created forms online for routine creation
  • Added import / export
  • Added routine submit / unsubmit

Tests

  • Import fake file
  • Import invalid file
  • Import good file
  • Export incomplete
  • Export valid

Notes

TODO

  • Web
    • Put the goods fields (for coordinates etc)
    • Only propose the objects that matches the conditions (ex: scientific programs of the user only)
    • Do all the needed checks
    • Add automatic computation of JD1/JD2
    • Add checkbox for JD / GD
    • Add options : copy my sequence on x days, and authorise report
    • Add ETC-IS simulation
    • Add help for new users (and for it the first time an account come on the page)
  • Do more checks at unserialization
  • views
    • When saving, do more checks on coordinates, jd1/2 etc
    • Uncomment filter for alerts removing
    • When submitting, use the monitoring to determine sequences status
    • When submitting, modify the first_schedule to False, when scheduling
    • When unsubmitting, uncomment the check for EXED and EXING removing
    • When unsubmitting, uncomment the scheduling and change the first_schedule to False

user_manager

Purpose

  • Handle user create / delete / login / logout
  • Allow to see user profile and make password recovery

Evolution

  • Creating application
    • See scheduler documentation above
  • Taking and adapting templates from Alexandru project
  • User creation
    • Form in admin.py
    • Functions to verify if email doesn't exist and if passwords match
    • Added check to pass login page if user is already autheticated
    • Restricted access to all pyros views to authenticated users (except from user creation and login) with @login_required

Notes

TODO

  • Password recovery
  • Profile page
  • User validation by administrator / commission

common

Purpose

  • Regroups common data and functions shared by applications

Content

  • RequestBuilder - class to build and save in DB the requests
    • You can start a request, then add sequences, albums and plans, given their parent class (ex: you need to give a sequence id to create an album)
    • Each function returns the ID of the element it created (to give it to the children classes)
    • To save a request, you need to use validate_request

Notes

  • Will probably contain the DB models at last ...

TODO