Guide authors: Wilson Skipper and Emily Moravec
For any suggested changes to this guide please contact ude.o1656673401arn@c1656673401evaro1656673401me1656673401. For any detailed MUSTANG2 questions contact ude.o1656673401arn@n1656673401osamb1656673401 or any of the current MUSTANG2 team members.
If this is your first time observing, PLEASE read all collapsed sections.
This guide assumes at least limited familiarity with Astrid and CLEO. If you are unfamiliar with those, please follow this link for video guides: GBT Observers. You can also access the Observer’s Guide here for written instructions on Astrid and CLEO.
How to use this guide
- If you are Green Bank Staff or on the MUSTANG2 (M2) instrument team and need to start-up, tune, and bias M2, follow section A.
- If you are NOT Green Bank Staff or on the M2 instrument and are logging on to only observe, the only necessary steps for you to do are in section B. However, it’s still recommended that you read through all the steps at least once to have a general understanding.
Additional MUSTANG2 Resources:
- MUSTANG2 Development Page
- OnGBTOps Wiki (for setup)
- ObservingWithTheGBT Wiki (for observing)
- Tuning Results Wiki
- Mustang2 Gui Wiki
- MUSTANG-2 home page
A: Setup – Tuning and Biasing
Detailed M2 instrument team instructions on tuning and biasing that can be used as a reference can be found here: https://safe.nrao.edu/wiki/bin/view/GB/Pennarray/OnGbtOps.
A1.1. Start an observing VNC
A1.2. Ask the operator to put you in the gateway for M2.
A2. Run startup + tuning script
A2.1. Change directories to where the startup + tuning script is.
A2.2. Run script.
where project name is the name of your project and the session number (e.g. AGBT18A _014_01) where the 18A_014 refers to the proposal number (this will be in the e-mail about observing) and the last two digits are the session number. The session number is not the source code from the DSS e-mail. It is very important to get this right or data reduction will fail to pick up the tuning which in turn affects focusing. If you should make a mistake and get it wrong after tuning is done it can be corrected by creating soft links in the same way as you do when changing projects.
Note: It’s important when typing out your session number that the session number have two digits. For example, if this is your 9th session, type out AGBTProjectID_09 NOT AGBTProjectID_9. If you do the latter option then you will not be able to take data correctly.
If you’ve accidentally mislabeled your Project ID and session number, ssh to egret using lmonctrl and type this command:
In this directory you should be able to see your projectID and session number. Simply rename it by typing:
mv <Old_ProjectID_session#> <New_ProjectID_session#>
This will fix any issues with data acquisition.
A2.3. Follow process of script.
A2.3.1. The script will ask you if you really want to do this. Use the arrow keys to select yes.
A2.3.2. Then it will ask you to enter the lmonctrl password on egret (ask if you do not know it).
A2.3.3. Then it will do the following:
- Log into egret and restart the manager
- Telnet into the iboot bar and turn on the roaches, function generator, and HEMTs
- Start a gnome-terminal with tabs running ipython sessions which tune each roach – sometimes gnome-terminal fails in which case it will bring up seperate x-terms
- During tuning it will ssh into each roach every 5 seconds
- After tuning has finished it will bring up the tuning plots
- Then set the manager into observing mode and check if data are flowing – if not it will attempt to fix this.
A2.4. Check the IQ, Flux Ramp, and Phase Response plots output by the script.
See https://safe.nrao.edu/wiki/bin/view/GB/Pennarray/TuningResults for explanations and examples.
Info: Further explanation on tuning
Tuning is kind of like MUSTANG’s equivalent to balancing. It has multiple channels of data and tuning is trying to ensure that the attenuation levels are all roughly equivalent. You do not need to be anywhere specific on the sky nor does MUSTANG need to be in the GBT turret for this step to be done.
Mustang has ~256 channels, ~215 feed horns, of which about ~211 are populated. It has 4 roaches, which are a type of Field Programmable Gate Array (FPGA, which is essentially a specialized CPU for data acquisition and manipulation). These 4 roaches process 64 channels each, adding up to the 256 channels. However, since not all the feed horns are populated, some of the channels in these roaches are already unused.
The IQ plots and many graphs for each roach represent all the different channels for each roach. Since some of those channels are unused, it’s not abnormal to see some channels that don’t look right. As long as the majority look okay, then things are working fine.
A3. Check that data is flowing
Go to the Mustang Manager in CLEO. Click the miscellaneous tab, and click the “Locked” on the bottom left of the window to unlock the regular features, then also unlock advanced features by clicking the “Locked” next to Advanced Features.
Click through the channels and look for:
- The “Frame Cntr” numbers should be changing and not be really low or 0.
- The “Roach Data” numbers changing.
- The Frame and Clock Cntr columns next to the Channel and Roach data should be similar across the 4 roaches (if they finished tuning at the same time).
If there is no data flowing in one or more roaches, you have a few potential solutions:
- You can try resending the channel numbers by going to “Num Chan” -> enter 64 -> press enter.
- Or turn “DataXinit” off then on.
- SSH to the problematic roach(es), e.g. “ssh root@mustangr1-1”.
- If these steps do not solve the problem, you may need to either restart the manager, or worst case, turn off data streaming (zero biases if you notice a problem after biasing the detectors) and power cycle the roaches (in the ibootbar). If after restarting the manager, the problem persists, restart the roaches (in ibootbar). If you restart the ROACHes, you will need to redo the tuning steps.
Note: Be sure to lock the Mustang Manager back when you are done to prevent any accidental miss-clicks. You press the same “Locked” buttons as you did in the beginning of this step, only now they will appear as “unlocked” until you click them again.
A4.1. Intro to Biasing
The MUSTANG2 receiver is a continuum receiver that uses a bolometric thermometer to make its measurements. Essentially, it is a highly sensitive thermometer with a filter for its bandwidth. Therefore, any photons in the bandwidth hitting the receiver will raise the temperature slightly.
It is able to be this sensitive by taking advantage of the science behind superconductors. This can be explained using the graph below:
This graph is for a specific superconductor, but the concept is the same, even if the exact temperature and resistance is different. Don’t pay attention to the numbers, but rather the trends.
As you can see, the material is only superconducting at lower temperatures. Once it gets hot enough, it becomes a regular resistor, with higher resistance with higher temperatures. What the MUSTANG2 receiver takes advantage of is the portion of the graph called the “transition edge,” the area in between the material being a regular resistor and being a superconductor. Here, the resistance changes very rapidly with even a slight change in temperature.
What biasing does, is ensure that each roach, when observing blank sky, is set in such a way that the maximum number of channels are placed at this transition edge, in order to ensure maximum sensitivity of the receiver. You will be seeing graphs for each channel, and the point which the AI is choosing is what it believes to be the transition edge of that graph.
Because we are only able to choose one setting for each roach, hence the attempt to simply maximize the effectiveness of all the channels, usually at the expense of certain channels in that roach.
A4.2. Do biasing.
All previous steps can take place in the 1-hour prep before your allotted observing time. That is, before you are given control of the GBT. However, biasing must be done on blank sky, therefore you must have control of the telescope for this and all subsequent steps (else just seeing the subreflector/ground spill over). You must be on your own account, logged onto titania or ariel, and also have permission to be in the gateway from the operator.
A4.2.1. Navigate to the proper directory to run biasing:
A4.2.2. Configure the appropriate bash profile:
A4.2.3. Run the bias script:
python new_detbiasV3.py <filename>
Where filename is AGBTsemester_projectCode_session. They are typically referred to as det bias files, as det bias is a shortened way of referring to the determined bias. You will see the speed of data coming going quickly and ‘Det Bias’ (in Misc tab) changing. After waiting a while, you will get a set of graphs.
A good set of biases will look like this:
You will get 4 sets of graphs like this, one for each roach. This one is for roach D, or roach 4, as shown in the title.
The solid black lines indicate the AI-decided detbias for each channel. It’s okay to see some of the lines reversed in direction (like in detectors 56 to 59 in this example) however something is wrong with that detector when it doesn’t have that general shape (such as in detector 20-23, or 60-63). Having a couple bad detectors isn’t unheard of, it’s more bothersome if a large percentage of detectors don’t look right.
A4.2.4. Once the plots have been come up and you have inspected them, close plots.
A4.2.5. If you are happy with the plots and values, in the terminal, enter Y to send bias to roaches and anything else to ignore calculated values.
A4.2.6. Make note of what the calculated values are by checking the Bias values in Misc!!! Then if the case that the manager crashes, you know what values to re-enter (see A5).
A5: Entering biases manually
If you have a short observing session, you can manually enter the biases to save some time. To do this, unlock the manager and going one roach at a time, set the DetBias to 5.0, press enter, wait until the blue box shows a DetBias of 5.0, and then enter 1.2 (check with an experienced M2 team member as to what the current reliable value is) into the DetBias. Repeat this step for all 4 of the roaches.
If the manager crashed and you need to re-enter the values that were previously calculated, follow the same process but put in your recorded values.
A6: Crash Mitigation – Restart the Manager
A6.1. Ask the operator to restart the MUSTANG manager using TaskMaster, even if you’ve been told how to do this yourself. Restarting machines through TaskMaster is a responsibility that is supposed to only be held by the operator.
A6.2. When the operator has told you that they have restarted the M2 manager, to your Cleo Mustang Manager screen and in the drop down menu go to Managers→Off and then again to click Managers->On to to turn the manager off and back on. You’ll then want to re-check the daily cycle to make sure that it is turned off.
A6.3. If you’re re-starting the manager before biasing, then you’re done restarting the manager. If you’ve already biased, re-check that the det-biases are what you expected them to be (detailed in step 11), and check that the dataXinit buttons are on (detailed in step 11)
B1.1. Calibrator and Source Preparation – several hours before.
You are expected to have your pointing calibrators, flux calibrators, and science sources planned out at least a few hours before the time of your observation. You can use CLEO’s Scheduler and Skyview to do this (See our tutorial videos on CLEO).
For flux calibrators, find those that are closest to your source. You can use planets or any of the ALMA grid cals listed in the following catalog:
It is efficient to use these flux calibrators as your first OOF source of the night. For OOF sources, you want a bright source that is > 2 Jy and 25 < elevation < 80.
For pointing calibrators, you can find suitable calibrators by going to the Scheduler & SkyView in the upper right-hand corner click “Catalog…” → “Add/Select/DeSelect Catalogs …” → “mustang_pointing” → “Apply”.
The goal is to find a calibrator that is 10-15 deg from your target and > 0.5 Jy (though if have good weather a better choice is something close that is 0.1 Jy). To find a source that is > 0.5 Jy, in the Scheduler & SkyView, go to the box in the right-hand corner that says “Source Intensity Range” and in the “Min” box put 0.5 and hit enter. Now load your science source catalog, enter the time you will be observing in the “UT Date and Time” box, and find a source that is showing and is 10-15 deg from your target.
For your science source catalog, you will either need to make one or a M2 team member has already made one for you.
B1.2: Observing Scripts – several hours before.
Template observing scripts are located in:
If you are creating the scripts for the first time for your project, you will want to copy (a) the scripts 1_m2setup, 2_m2oof, 3_m2quickDaisyOOF, and 4_m2quickDaisyPC and (b) one of the 5_ scripts for your science targets (the radius of the daisy will depend on your science – reach out to the M2 instrument team for guidance). The scripts m2quickDaisy and skydip are extra but can be of use. Read the README for instructions on editing these scripts once you have them in your project directory.
B1.3: Observing Notes.
During observing, you are expected to edit the MUSTANG observing run notes wiki and take notes of what’s occurred throughout the night on this wiki. Create a new page and entry at the bottom of the page by clicking “Edit Wiki text” and follow the naming convention of entry above <AGBTSemester_project-code_session-number>.
If you don’t have permissions yet to edit the wiki and are observing, you can take notes in a text document and email them to the MUSTANG team afterwards to upload to the wiki for you.
B1.4. Observing Prep – shortly before observations.
Open and connect to VNC session on titania or ariel via FastX or tunneling+VNC viewer.
Open an Astrid session and navigate to your corresponding MUSTANG2 project. The MUSTANG2 instrument team should have already populated your Astrid area with appropriate scripts.
The following are suggested CLEO windows to have open during observing:
- Launch → Receivers → Mustang2
- Launch → Status
- Launch → Antenna
- Launch → Observer Tools → Scheduler & Skyview and load in the mustang_pointing and your science target catalogs
B1.5. Observing Pre-flight Checklist
- Make/Find source catalog
- Choose flux calibrator/OOF source
- Choose pointing calibrator
- Make observing log
- Open VNC session with Astrid & Cleo
B2. Observing Procedure
B2.1. Communicate with operator.
A few minutes before your observing start time (say 10 minutes), get on Talk & Draw, tell the operator who you are and what project you are observing for. Also ask who the operator is.
B2.2. Fill in info.
In Astrid under ObservationManagement, go to the Run tab and fill in the Observer and Operator information.
B2.3. Take control.
Once the member of the M2 instrument team has finished biasing and the operator tells you are in the gateway/gives you the go ahead, in Astrid > File > Real time mode … > work online with control of the telescope.
Run the 1_m2setup script in Astrid.
Make sure that you have changed mySrc in 2_m2oof and run the 2_m2OOF script in Astrid.
For the first OOF of the night, you need to have calSeq=True so that a skydip is done as a part of OOFing process. An OOF will take ~20 minutes to run. Check the OOF results in Astrid > DataDisplay > OOF and re-rerun if necessary. For M2, we typically apply the z5 corrections. When the corrections are available, press the green button that reads “After selecting the Zernike solution above, click this green button to send the solutions to the telescope.” Note: sometimes OOF may time out and you will get a red screen if this happens.
Note: While your OOF is running, it is a good time to:
- write the weather conditions from the GbtStatus tab in Astrid in the log (Pyrgeometer – if working, Temperature, Humidity, IR Cloud Cover, and Wind Velocity).
- start the m2gui that is used to check M2 data while observing and check the skydip and that you can see the OOF images
- to start the m2gui execute the following commands in a terminal window:
- see section E3 for using the GUI
- to start the m2gui execute the following commands in a terminal window:
B2.6. Quick daisy on OOF source.
Run the 2_m2quickDaisyOOF script on your OOF/calibrator source (it’s best if you can make your OOF source and your calibrator source the same). Using the m2gui calculate the beam shape (WidthA & WidthB) and peak of the source (Peak_Height) and record these values in the observing log.
B2.7. Quick daisy on pointing calibrator.
Run the 3_m2quickDaisyPC script on your pointing source. Calculate the beam shape (WidthA & WidthB) and peak of the source (Peak_Height) using the m2gui (see Section E3) and record these values in the observing log.
Note: during this initial data acquisition (and to some extent, throughout the night) check your M2 Cleo screen that you opened up in step E1.4.3, and make sure that the numbers are continuing to change with time (if so, the boxes will mostly be blue) if they stop (indicated when the boxes turn lavender) then the Mustang2 manager has crashed, and you’ll need to reset it (detailed in Section C).
B2.8. Take science data.
Take ~30 minutes of science data followed by a quick daisy on your pointing calibrator. Often this is accomplished by running the 5_beauty_rX scheduling blocks in Astrid as many times as needed (or other types of scripts). Each individual “beauty” scan is ~8-9 minutes in length. So if you are submitting individual beauty scans (which 5_beauty_rX are), you can submit 4 of the beauty scripts in a row followed by your pointing calibrator scan.
Note: If you try to look at the science data in the m2gui, make sure you choose the “faint science” option under “source type.”
What is beauty_r2p5 and beauty_r3?
Beauty_r2p5 and Beauty_r3 are the science scans of the observation. The difference between the two is the radius size of the scans in arcminutes (one is 2.5′ and one is 3′ respectively). If you only see Beauty scans, unlabeled otherwise, then they are likely 3′ in diameter.
B2.8. Continue to take science data.
Continue to do ~30 minutes of science data followed by a quick daisy on the pointing calibrator for the rest of the night. Monitor the beam size (WidthA and WidthB) and the Peak_Height using the m2gui to determine if you need to OOF again.
When to OOF?
If the new Peak_Height is down by more than ~15%, or if WidthA and WidthB become very different from one another (indicating that the beam has become overly elliptical) you’ll want to do an OOF. Optional:If you don’t have much observing time left, once the PeakHeight is down by more than 15%, instead of redoing the OOF scan, you can do another m2QuickDaisy on the pointing source to be sure that it is that low, and then do two more Beauty scans until the PeakHeight has gone down by another 15% (so a cumulative 30%).
B3. Checking data with the m2gui
After you have opened the m2gui follow these steps to check the tipping scan, monitor the beam shape (width, widthA, widthB) and peak of calibrators (Peak_Height), or to just check the data.
B3.1 Start-up/Check Tipping Scan
B3.1.1. Go online. Click the “online” button
B3.1.2. Select tipping scan. Under Calibration, click “Select Tip Scan” and choose the most recent scan number from the bottom labeled “tip” under “scan type.” This should be from the beginning of the 4 OOF scans.
B3.1.3. Inspect plots. Many plots will pop up – one for each roach showing the results of the tipping scan for each roach. You can click out of these once they finish unless you are particularly curious about specific roaches. After these plots have been produced, you will see a graph to the right in the main gui window, showing the results of the tip scan – each roach is plotted in black with a fit in green. Check to make sure that it looks reasonable.
Example: A good tipping scan. Below is an example of a reasonable tipping scan. The black lines (one for each roach) should be fairly free of wiggles and the green line (which is the fit) should follow the black lines fairly closely.
Example: A bad tipping scan. Below is what a bad tipping scan looks like:
If the tipping scan doesn’t look right (a lot of wiggles), try running the “skydip” script in Astrid. This reruns the tipping scan without having to redo the whole OOF. If it still looks bad, check the weather conditions in CLEO. The weather might not be good enough to observe. You can also call one of the M2 instrument team and get their advice.
B3.1.4. Check the number of live detectors. At this stage, check the number of live detectors, as well as throughout the night. Record this in the observing log.
In the image below, you can see where to check the number of live detectors:
Generally it’s good to have 170+ live detectors, however it can sometimes be as low as 160 if the tuning step didn’t go very well. If you see this number as low as the 150s or 140s (especially if it’s lower than that, which it shouldn’t be) be sure to contact a M2 team member. You can also try re-tuning (see section A) and hope that that fixes it.
If the tipping scan looks good, continue forward.
B3.2. Checking Calibrator/Beam Parameters
B3.2.1. Make map. To make a map of a calibrator, after you have run a m2quickDaisy on a source, click “Update Scan List” to find the source scan number of the source you just observed then set the “Scan Numbers” to that scan number. Then set “Source Type” to “Calibrator,” and then click “Make Map”.
This will open up an image of the daisy map that you selected.
The map should look something like this:
The maps that the M2 team makes are called daisy scans. This is because they loop many times around a central point, looking somewhat like daisy petals. This emphasizes exposure time on the center of the map, with less exposure on the outside edges of the map, making the center of the map more accurately calibrated. They then use the outside of the map to calibrate the sky temperature and remove these effects in the center of the daisy in later post-processing.
What you see at this stage is an image of the daisy scan. In the center is your calibrator source, visible because it is a bright source. Later, when looking at daisy scans of your science source, it’s very likely that you will only see a flat map in the center because it’s so much more faint.
The units of the color-coding of this map are in Kelvin of the forward beam. The forward beam is calibrated for the estimated sky temperature at that elevation that we gleaned from our tipping scan earlier on in the night. Therefore, the forward beam temperature should hover around zero if everything is calibrated correctly.
The lines drawn on the map designate the beam path of the GBT on the sky relative to your source. As you can see, each loop begins at the source, extends out, and then returns to the source. This is done throughout the space around your source. Because every loop returns to your source, this results in a higher exposure time on your source relative to the rest of the sky. However, because the units are in Kelvin of the forward beam, this does not mean a higher temperature, but instead simply less noise in the map.
B3.2.2. Fit Map. Click “Fit Map”
This will produce the following plots in the gui.
Once you “Fit Map”, the fit parameters will be printed out in your terminal.
The Floating underflow error you see is not a concern.
B3.2.3. Record values. Write down the Peak_Height, WidthA, and WidthB to compare to later pointing scans to monitor the beam and decide if you need to re-OOF.
B3.3. Checking Science Scans
If you would like to make a map of of a science scan(s), you can do so by following the same steps as making a map of a calibrator but under “Source Type” you need to select “Faint Science.”
B3.4. Showing Time Stream
This is optional, but if you would like you can check the time streams (checking how the sky temperature is changing over time) by clicking the “show time stream” button after making your map.
Here’s where the button is:
This is what a good time stream will look like:
This is what a bad time stream will look like:
B4. Changing M2 Projects/Second M2 Project of the Night
If you are observing for an M2 project that is not the first M2 project of the night then before observing you will need to create a link for the tuning so that OOF & data reduction can find the right tuning.
B4.1. Make symlink. Before you begin observing, login to egret and type:
ln -s <old_project_session> <new_project_session>
Where the old project session is the full name of the previous M2 project and the new project session is the second M2 project of the night that you are observing for. Be very careful to put in the right project and session ID or this step will not work and you won’t get any data. You can ask the previous observer for the old project session ID, or look for it by typing:
ls -ltr /home/gbtdata/
Then the last modified file will tell you what the most recent project ID was.
B4.2. Run m2setup. When the observing time for the second project starts, you need run m2setup in Astrid again. This is already outlined in the directions, but some people think they can skip it when changing from another MUSTANG run. This is not the case. It’s very important to still run m2setup at the beginning of your session.
You can possibly possibly skip OOFing at the beginning of this second project. You can ask the previous observer when they last did an OOF and what the progression of the beam was.
If you need to re-OOf, make sure that calSeq=True to get a skydip.
If you do not need to re-OOF, do a stand-alone skydip and change myAz to the Azimuth of whatever your first source will be (calibrator, etc.). The telescope will slew to that Az.
B4.3. Flux calibrator. You’ll also want to still observe your flux calibrator using the m2quickdaisy script. This is another thing people think they can skip, but it makes reduction later more difficult. Check the beam with this flux calibrator.
C: Closing up for the night
C1. Go offline
In Astrid, go from working online to working offline by going to > File > Real time mode … > work offline.
C2. Set detector biases to zero
To set the biases to zero, go to the Mustang Manager in CLEO and click on the miscellaneous tab. Then in the top middle, you will see 4 rows of Det Bias 1-4, corresponding to the 4 roaches.
Unlock the manager and going one roach at a time, type 0 in the left DetBias box, press enter, and wait until the blue box (right DetBias box) shows a DetBias of 0. (Roaches 1-4 correspond to roaches A-D). Repeat this step for all 4 of the roaches.
C3. Turn data transmission off
In your Mustang2 Cleo screen, turn the DataXinit off for all four roaches. You will need to be in gateway AND unlock both the unlock and advanced features unlock buttons to do this.
C4. Turn off components
In VNC session, Gg to http://mustangboot.gbt.nrao.edu and turn off the roaches, HEMTs, and Function Generator by checking those three boxes then go to left of the screen and click ‘Off’ (gray button).
C5. Turn on daily cycle
Back on M2 CLEO window > Housekeeping (unlock) > recheck daily cycle to be on and put autocycle trigger to HE4.
Set the “daily cycle time” = 0.65of a day in UT.
C6. Kill VNC session
Either kill your FastX session or your VNC session via the terminal.
Congratulations! You’re all done! Happy Observing!
- How to use this guide
- A: Setup – Tuning and Biasing
- B: Observing
- C: Closing up for the night