User interface design

Ready for discussion (3/3/16)
Note (3/6/16): I (@jjt) have added @tatianag’s (tg): comments in-line to make the discussion easier to follow.


User console:

Two use cases:
Use cases are now in a Google doc.

  • Select an image region and perform some function on it.
  • Blink two displayed images.

Select an image region and calculate some statistics in it.

Assume that an image is being displayed in one box.

  • User selects a region:

  • Using FF tool.

  • By entering coordinates on the Javascript command line.
    (tg): I think if coordinates are entered into command line, the region still needs to be displayed

  • Send command to FF client side communication channel, by typing this:
    <Command> <region> <Image_source_URL> <Result_box>
    Nominal region format (if rectangle): xleft ,ytop, xright, ybottom.

  • Do we want to be able to enter either image or display pixels?

  • The image URL is only needed if more that one image is being displayed. Is there another way for the server to figure it out?
    (tg):: A user should be providing Image URL or file name only when choosing which image to load into image viewer. After that, the user should be referencing the displayed image by the name (for example, div name passed to makeImageViewer). The client code should keep track of the image, displayed by the given image viewer. In future, it should be able to query FF for this info.

  • If <Result_box> is not passed to the server, how does the client know what to do with a response message? Should the command interface block? I have in mind that some commands might take some time to process.
    (tg):: <Result_box> should not be passed to the server. Javascript code should use Promises for asynchronous calls.

  • Server receives the command.

  • How do we synchronize the list of valid commands? (need configuration files).
    (tg):: Firefly Extension API makes it easy to define context actions when a selection is made. Perhaps your client code should start using this mechanism to define the actions available on a particular selection.

  • Is any validity checking needed? If so, how are errors returned? We may need a separate communication channel, because the client is looking for a “result” in a particular format. (or, maybe, each return format has an error value - seems like a kludge).
    (tg):: If the region is displayed on the image it helps with validation.

  • (For the future) We may have several client and server instantiations. How is the linkage established and maintained? If one server deals with multiple clients, a more complicated communications protocol will be required.
    (tg):: FF is designed to run in multi-server, multi-client environment. In fact we do run it this way in production. The interface should be designed in such a way that all necessary information is in the request.

  • Server Python processes the command.

  • It needs to know what image to process (if URL is not part of the command).
    (tg):: We should separate end-user interface from the internal interface (between Camera Javascript code and FF). Camera Javascript code will know or will be able to query FF for the path of the displayed image. The path to the displayed image file will be passed from Javascript client via FF API to Python server-side code.

  • It may need auxiliary data (e.g., compare image with a template). How will it know where to find this?

  • After processing, server sends the response.

  • Response format is determined by <Command>.
    (tg):: Python server-side process can return any information in JSON format. You can also tell FF to display FITS image or table produced by the server-side Python.

  • (For the future) If multiple clients, how does FF know where to send it?
    (tg):: Firefly is using shared file system and is replicating cache between server nodes. For efficiency, it might be using sticky sessions to make sure the subsequent requests go to same server.

  • Client receives the response and displays the result in <Result_box>

  • If the interface is non-blocking, we need to think about how to associate responses with commands.
    (tg):: As I mentioned, this problem is solved with Javascript Promises, which allow to handle the response whenever it becomes available.

  • Other issues:

  • It may be that the response is an update to an existing graph (e.g., a time history).

  • The user may want to manipulate (e.g., compare) results (perhaps in separate boxes. The data must be kept in retrievable form.

Blink two displayed images

I am choosing one scenario. There might be many ways to do this.

Assume that two images are being displayed in two boxes.

  • I have in mind two commands, Start_blink and Stop_blink. That is, blinking is a persistent state.

  • I am assuming that the user will want to see both images separately during blinking.

  • User selects a the two images, by naming the boxes.

  • Command: Start_blink <image box 1> <image box 2> <display box>

  • The image in <box 1> will appear in <display box>.

  • The client is now in a blinking state, and some behaviors will be different.

  • Each click in <display box> will cause the other image to be displayed there - the display toggles back and forth under user control.

  • While in the blinking state, the user can manipulate the image display properties.

  • Panning, zooming, and other changes of image display properties are automatically applied to both images.

  • The user might want to pan, zoom, etc., the <display box> independently, in order to focus on a small region (e.g., change the contrast) without losing the big images.

  • The user might want to obtain statistics from both images in the region displayed in the <display box>.

  • To leave the blinking state, the user enters this command:
    Stop_blink <display box>.

  • Other issues:

  • Low latency is an important property of the blinking state.

  • Some commands will require two commands to the back end, to modify the display of both images.

  • This behavior is one reason one needs a blinking state.

This is a copy of an email from @dyue2 about @jjt’s diagram:


Your diagram is mostly correct. However, the reality is a little bit more complicated than what you described.

Firefly display is in some way directly linked to firefly-tools for image access and (table data access, however we currently did not use those functions). However, the communication is controlled by the parameter set with javascript.

Firefly display has its own UI, which covers some of the usage cases. However, these UI operation currently is beyond our control.

On the server side, the data access is more nuanced. The firefly tools only provides the information passed from the frontside, and they expect us to have a direct access from python analysis to data access for calculation purpose.

That been said, firefly tools also had access to the data for display, and that was controlled by the FF interface.

So, I feel we could really benefit to have a diagram as you propose, that would make the architecture more manageable. I am sorry that my explanation might not be very straightforward.

The interface between the Javascript UI and display is through various API provided by the Firefly. Currently we can read mouse (pixel position) and draw ds9 region,

Backend FF is a process and python analysis is a subprocess, and I believe the DM will be incorporated into the architecture as a library that will be called by the python process later on. Currently firefly is only a mediator that passes the information the user sent at the frontend and the calculation we performed at the backend, but nothing more. Currently we mainly use the most widely-used “json” format for the communication.

Dingcheng Yue

I have a proposal to this. It is easier to maintain a display stack (using html tag for example) based on firefly’s image viewer. And the operation of start blink and stop blink is equivalent to bubbling the individual viewer to the top of the stack. And it is much easier to control from the outside. And if that is the case, it is easier to imagine the operation including splitting stack and join stack.

Regarding expanded mode. Firefly does have an API for expanded mode: search for ExpandedViewer in this doc.