For any suggested changes to this guide please contact email@example.com. For any detailed MUSTANG questions contact firstname.lastname@example.org or any of the current MUSTANG team members.
If this is your first time observing, PLEASE read all collapsed sections.
For conference calls with the MUSTANG team during observations: 215-490-9937
This guide assumes at least limited familiarity with Astrid and CLEO. If you are unfamiliar with those, please follow this link for video guides: GBT Observers
You can also access the Observer’s Guide here for written instructions on Astrid and CLEO.
Note: If you are NOT training/on Green Bank Staff or the MUSTANG2 instrument team, then the only necessary steps for you to know are steps 13 and onward. However, it’s still recommended that you read through all the steps at least once to have a general understanding.
OnGBTOps Wiki (for setup)
ObservingWithTheGBT Wiki (for observing)
Turning on Components
1. Open a vnc session on egret:2
Typically, this will be the always-live session on egret:2. Ask one of the mustang team for the password. You can do this by typing this into any GBO terminal:
2. Open the MUSTANG2 manager
Open CLEO. Under the “launch” window, go to receivers, and open Mustang 2
3. Check that the Array temperature is less than 0.4 Kelvin.
Checking the array temperature ensures that the receiver is cool and that the setup won’t be useless and wonky. This is sometimes the case when MUSTANG has been tipped or if it just finished a cooling cycle.
Details on Tipping
The MUSTANG cryogenics on the Green Bank Telescope are very elevation-dependent. If the telescope is below 35 degrees, for example, and the operator wishes to rotate the turret, they must first raise the telescope above 35 degrees before changing out receivers. This is the case even when MUSTANG is not observing at the time. If this is not done, then the cryogenics will fail on MUSTANG and the receiver will begin to heat up. This would leave the MUSTANG receiver unable to take usable data for several hours. When this occurs, observer’s colloquially say that the MUSTANG receiver is “tipped.”
Details on Cooling
MUSTANG uses Helium 3 and Helium 4 for its cryogenics. As the Helium 3 and Helium 4 run out, the cryogenics have to cycle every 24 hours to ensure that the receiver is always cool. However, when the receiver is in the middle of a cycle, it warms up slightly as the cryogenics are being moved out of the system to refill with new Helium 3 and 4. This cycling takes several hours. This is why, when you’re observing, you want to disable this daily cycling to ensure that it doesn’t try and begin a cycle in the middle of your observation. That would heat up the receiver and make you unable to obtain decent data. The same is the case with setting the Auto Cycle trigger to “None.” The Auto Cycle Trigger is an extra precaution where the receiver will automatically kick off a cycle when it detects that it is out of Helium 4. This in addition to the daily cycling.
4. Disable daily cycle time
Disabling the daily cycle ensures that the receiver won’t start trying to cool itself during your observation. This is done in the MUSTANG manager
5. Set the auto cycle trigger to “None”
Disabling the auto cycling ensures that the receiver won’t start trying to cool itself during your observation. This is done in the MUSTANG manager
6. Turn on the roaches, HEMTs, and FunctionGenerator
6.1 Go to this link on any browser from a computer on the Green Bank network: mustangboot.gbt.nrao.edu
6.2 Log in with the username and password for the MUSTANG bootbar, ask one of the MUSTANG team for this.
6.3 Turn on the Roaches, HEMTs, and FunctionGenerator by checking them and clicking “On”. Leave this browser open for later.
The ibootbar is how the team turns on on the Roaches and its linked components, which are MUSTANG’s equivalent backend to what VEGAS does for the GBT. They are a type of Field Programmable Gate Array (FPGA) which are essentially advanced specialized CPU’s with analog to digital converters.
The above image is what the browser page will look like once you’ve logged in. You’ll want to check next to the Roaches, HEMTs, and FunctionGenerator, and then click the “On” button under the “Control” section. You can see that they are the only things off in this photo.
7. Start The Tuning
7.1 Go back to your vnc session from step 1. Find a terminal and change to the correct directory using this command:
7.2. Start the tuning using this command:
./startM2.bash <AGBTprojectID_session#>Further explanation on tuning
Tuning is kind of like MUSTANG’s equivalent to balancing. They have multiple channels of data and are trying to ensure that the attenuation levels are all roughly equivalent. You do not need to be anywhere specific on the sky nor does MUSTANG need to be in the GBT turret for this step to be done.
Mustang has ~256 channels, ~215 feed horns, of which about ~211 are populated. It has 4 roaches, which are a type of Field Programmable Gate Array (FPGA, which is essentially a specialized CPU for data acquisition and manipulation). These 4 roaches process 64 channels each, adding up to the 256 channels. However, since not all the feed horns are populated, some of the channels in these roaches are already unused.
The IQ plots and many graphs for each roach represent all the different channels for each roach. Since some of those channels are unused, it’s not abnormal to see some channels that don’t look right. As long as the majority look okay, then things are working fine.
Note: It’s important when typing out your session number that the session number have two digits. For example, if this is your 9th session, type out AGBTProjectID_09 NOT AGBTProjectID_9. If you do the latter option then you will not be able to take data correctly.
If you’ve accidentally mislabeled your Project ID and session number, go to your vnc session on egret (see step 1) and open a new terminal. Type this command:
In this directory you should be able to see your projectID and session number. Simply rename it by typing:
mv <Old_ProjectID_session#> <New_ProjectID_session#>
This will fix any issues with data acquisition.
The tuning command will open up 5 windows. One labeled “FG” and the other 4 labeled M1-M4 ascending. This is the window showing the status of the function generator and the 4 roaches respectively. There are 4 roaches, interchangeably labeled 1,2,3, and 4 and A,B,C, and D. Be sure to check each of the 4 Roach windows as they open and make sure that they say that they are “connected” in the messages showing up. Then wait, as the full set up of the roaches takes a few minutes.
You’ll know when this script is done when an ipython text indicator pops up for each of the 4 roach windows, as well as an onslaught of 3 different kinds of graphs explodes on your screen. We will discuss those graphs in the next few steps.
8. Examine The IQ Loops
The first set of plots we’re going to look at are called “IQ loops,” which are also saved in this directory if you want to look at them later:
There will be four sets of loops, one for each roach.
The above image is what good IQ loops look like. If there is one loop that looks bad but the rest look fine, then this is passable and nothing needs to be done.
The graph below shows some examples of what bad IQ loops look like:
More explanation on IQ loops graphs
There will be multiple sets of graphs looking like this, each corresponding to one of the 4 roaches. Which roach it is will be labeled in the title of the graph (the above images’ title is cut off). Each loop corresponds to one detector, and there are 64 detectors for each roach, hence the many loops all corresponding to one roach.
If your IQ loops look bad, in the window for the corresponding Roach, type:
This should redo the loops to hopefully make them look better. You’ll know this is done when you’re able to type commands into that roach window again, but it doesn’t take very long to do.
You can check the plots by typing:
to redo those plots.
9. Examine The IQ Flux ramp response plots
The other two sets of plots are called the IQ flux ramp response plots. There will be multiple of the each plot for each Roach, the roach indicated will be in the title of the plot, numbered 1-4.
These are examples of what these plots look like when things are going well. For the left plot, note that all the detectors have sinusoidal waves.
On the right plot, note that the maximum amplitude is around 10,000. If this is not the case, usually if it’s less than 8000 or so, you should redo the plots. In addition, the lines should be almost straight but slightly curved toward the left like they are in this case, with increasing amplitude. If there is one colored line that’s off, that’s fine, as long as it’s not more than one or two.
These are examples of bad looking plots:
If either of these plots doesn’t look right, then close the current set of plots. Then, in the corresponding roach type:
You can then redo the plots to check if they look right using the command:
9.1 Bad connectivity: low power
If you get a plot like this:
then this can be due to a connectivity issue – not of data transmission, but rather injected power into the readout. There are a few potential fixes, but the best solution seems to be patience. That is, this problem tends to resolve itself (with some nudging) after 30 minutes to 1 hour of the ROACHes being on.
Ways to nudge the misbehaving ROACH:
- Power cycle via the boot bar (just the ROACHes; leave off for ~30 seconds; turn on and wait ~30 seconds before your next action. More than 30 seconds is OK too; it’s just a matter of your patience).
- The above sequence within Python (um1.progAttens(), um1.autoADCrange()) has, on the rare occasion resolved this issue.
- Retune (see “remaining tuning troubles” below).
9.2 Remaining tuning troubles
If for any of the tuning plots, these commands don’t seem to fix the problem, then you can redo the tuning altogether. The command for that is (again type this into each roach window):
And a new set of plots should come up to check.
10. Restart the Manager (Optional Step):
How optional is “optional”?
This step is optional because the manager has, during certain seasons, had a habit of crashing during the first scan after the setup has completed, and it’s been shown that restarting the manager as a part of setup tends to reduce the likelihood of it crashing during the first scan. However, it’s not absolutely necessary, and if one is in a hurry to set up, they can skip this step and take the additional risk that the manager will crash during the first scan. Additionally, if the manager does crash at any point in the night, then it is required to restart the manager to get it up and running again.
10.1. Ask the operator to restart the MUSTANG manager using TaskMaster, even if you’ve been told how to do this yourself. Restarting machines through TaskMaster is a responsibility that is supposed to only be held by the operator.
10.2. When the operator has told you that they have restarted the M2 manager, to your Cleo Mustang Manager screen (detailed in Step 2) and in the drop down menu go to Managers→Off and then again to click Managers->On to to turn the manager off and back on. You’ll then want to re-check the daily cycle to make sure that it is turned off (detailed in step 4).
10.3. If you’re re-starting the manager before biasing, then you’re done restarting the manager. If you’ve already biased, re-check that the det-biases are what you expected them to be (detailed in step 11), and check that the dataXinit buttons are on (detailed in step 11)
Streaming Data; check for packet loss.
11. Turn on dataXinit and Check the High Channels
11.1. Go to the Mustang Manager under CLEO (detailed in Step 2). Click the miscellaneous tab, and click the “Locked” on the bottom left of the window to unlock the regular features, then also unlock advanced features by clicking the “Locked” next to Advanced Features.
11.2. Find the DataXinit Column for the rows labeled A through D. These indicate each of the four roaches. Click the DataXinit button for all four roaches to turn on data acquisition for each respectively.
11.3. Once those are turned on, you’ll want to check the high channels to see if the Roach data is at zero. In the bottom 4 rows, labeled A-D (again, for the 4 roaches respectively) change the “Channel” to 64 for all 4 roaches. Then check the “Roach Data” Column to see if it is zero. If it’s close to zero, even as close as 0.00001, that’s okay, as long as it’s not exactly zero. Also, the Frame and Clock cntr columns next to the Channel and Roach data should be similar across the 4 roaches (if they finished tuning at the same time).
If there is a problem at this step, you have a few potential solutions.
- If there is a connectivity problem (no/little data streaming), you should try to SSH to the problematic ROACH(es), e.g. “ssh mustangr1-1”.
- In tandem with the above, you can turn off and on again the DataXinit.
- If these steps do not solve the problem, you may need to either restart the manager (detailed in step 10), or worst case, turn off data streaming (zero biases if you notice a problem after biasing the detectors, i.e. step 12) and power cycle the ROACHes (step 6 – just the ROACHes – no need to cycle HEMTs and FG). If after restarting the manager, the problem persists, restart the roaches (detailed in steps 6 and 7). If you restart the ROACHes, you will need to redo the tuning steps (detailed in steps 8 and 9).
11.4. Be sure to lock the Mustang Manager back when you are done to prevent any accidental miss-clicks. You press the same “Locked” buttons as you did in the beginning of this step, only now they will appear as “unlocked” until you click them again.
12. Perform Bias and Enter in Determined Bias
The MUSTANG2 receiver is a continuum receiver that uses a bolometric thermometer to make it’s measurements. Essentially, it is a highly sensitive thermometer with a filter for it’s bandwidth. Therefore, any photons in the bandwidth hitting the receiver will raise the temperature slightly.
It is able to be this sensitive by taking advantage of the science behind superconductors. This can be explained using the graph below:
This graph is for a specific superconductor, but the concept is the same, even if the exact temperature and resistance is different. Don’t pay attention to the numbers, but rather the trends.
As you can see, the material is only superconducting at lower temperatures. Once it gets hot enough, it becomes a regular resistor, with higher resistance with higher temperatures. What the MUSTANG2 receiver takes advantage of is the portion of the graph called the “transition edge,” the area in between the material being a regular resistor and being a superconductor. Here, the resistance changes very rapidly with even a slight change in temperature.
What biasing does, is ensure that each roach, when observing blank sky, is set in such a way that the maximum number of channels are placed at this transition edge, in order to ensure maximum sensitivity of the receiver. You will be seeing graphs for each channel, and the point which the AI is choosing is what it believes to be the transition edge of that graph.
Because we are only able to choose one setting for each roach, hence the attempt to simply maximize the effectiveness of all the channels, usually at the expense of certain channels in that roach.
Note: All previous steps can take place in the 1-hour prep before your allotted observing time. That is, before you are given control of the GBT. However, biasing must be done on blank sky, therefore you must have control of the telescope for this and all subsequent steps. You must be on your own account, (not the egret:2 vnc) logged onto titania or ariel, and also have permission to be in the gateway from the operator.
12.1. Run the biasing by typing these commands in the terminal:
Navigate to the proper directory to run biasing:
Configure the appropriate bash profile:
Run the biasing command:
python new_detbiasV3.py --start 3. --stop 0.5 --pulse 5.0 --step 0.025 <filename> 64 64 64 64
Where <filename> is the name of your project, or whatever else you want to call these files. They are typically referred to as det bias files, as det bias is a shortened way of referring to the determined bias. After waiting a while, you will get a set of graphs.
A good set of biases will look like this:
You will get 4 sets of graphs like this, one for each roach. This one is for roach D, or roach 4, as shown in the title.
The solid black lines indicate the AI-decided detbias for each channel. You’ll want to pick an average of what you think the detbias is for all the channels. You do this by picking an average of the x-axis location for all the solid black lines. If you don’t want to use the AI, the transition edge is the zero-slope point of the line for each detector. It’s okay to see some of the lines reversed in direction (like in detectors 56 to 59 in this example) however something is wrong with that detector when it doesn’t have that general shape (such as in detector 20-23, or 60-63). Having a couple bad detectors isn’t unheard of, it’s more bothersome if a large percentage of detectors don’t look right.
It’s better to err on the side of a lower voltage rather than a higher voltage. For example, in this particular graph, most of the solid black lines hover around 1.6-1.7 Volts. I would then pick 1.6 Volts as my determined bias.
12.2. Put in the decided detbias. Go to the Mustang Manager in CLEO (detailed in Step 2) and click on the miscellaneous tab. Then in the top middle, you will see 4 rows of Det Bias 1-4, corresponding to the 4 roaches.
12.3. Unlock the manager (detailed in the beginning of step 11) and going one roach at a time, set the detbias to 5.0, press enter, wait until the blue box shows a detbias of 5.0, and then reset the detbias to the determined number from the graphs for that Roach (Roaches 1-4 correspond to roaches A-D). Repeat this step for all 4 of the roaches.
Note: During observing, you are expected to edit the MUSTANG observing wiki and take notes of what’s occurred throughout the night.
Ask a member of the MUSTANG team to add you to have editing permissions. You can also try emailing the helpdesk at:
The wiki is located in
Where <your_proj_id> is the GBO project ID given to you.
If you’d like to see a list of all the projects in the wiki, go to safe.nrao.edu/wiki/bin/view/GB/Pennarray/ and click “index” in the top left corner of the page.
If you don’t have permissions to edit the wiki and are still observing, you can take notes in a text document and email them to the MUSTANG team afterwards to upload to the wiki for you.
You are also expected to have your pointing source, your calibrator source, and your science sources planned out at least a few hours before the time of your observation. You can use CLEO’s Scheduler and Skyview to do this (See our tutorial videos on CLEO) and upload these catalogs:
For pointing sources:
For your science sources:
13. Setting Up Components
13.1. Open an Astrid session and navigate to your corresponding MUSTANG2 project. The MUSTANG2 instrument team should have already populated your Astrid area with appropriate scripts.
13.2. Open CLEO → receivers → Mustang2
13.3. Run the command for the MUSTANG2 idl by typing in your terminal:
We are opening up the MUSTANG2 GUI, which we can later use for examining our data as it comes in from the GBT.
14. Initial Calibration
The following pale pink step will generally be done by the MUSTANG-2 team, but it written here for completeness.
If the previous run was also MUSTANG and you’re switching projects, there are some extra steps to take before you can begin observing. First, you will need to login to the egret:2 vnc session (detailed in step 1) and type:
ln -s <old_project_session> <new_project_session>
Where the old project session is the full name of the previous MUSTANG project that preceded yours, and the new project session is your own project. Be very careful to put in the right project and session ID or this step will not work and you won’t get any data. You can ask the previous observer for the old project session ID, or look for it by typing:
ls -ltr /home/gbtdata/
You can then look at the most recent project id as the last modified file.
In addition to this, you’ll want to make sure to run setupm2 in Astrid again. This is already outlined in the directions, but some people think they can skip it when changing from another MUSTANG run. This is not the case. It’s very important to still run setupm2 at the beginning of your session.
You’ll also want to still observe your flux calibrator using the m2quickdaisy script, also outlined in the directions. This is another thing people think they can skip, but it makes reduction later more difficult
One thing you can possibly skip is the m2oof script. You can ask the previous observer when they last did an OOF. If it was relatively recently (within 1 or 2 hours ago) then you can move on to an m2quickdaisy of the pointing source first, and follow the steps outlined below (detailed in step 15) to check the beam shape of your pointing source. If the WidthA and WidthB found are both below 10″ (the units given are arcseconds) then it’s probably okay to continue on without performing another m2oof. Then you can go forward with your observation as normal.
14.1. Run the setupm2 script in Astrid
14.2. Run the OOF script in Astrid, check OOF in Astrid and re-rerun if necessary
Note: While your OOF is running, it’s a good time to write the weather conditions in the log.
To edit the wiki, you’ll need permissions from GB computing (email@example.com). If you don’t yet have permissions to edit the wiki, you can keep a text document copy and send it to the MUSTANG-2 team to upload to the wiki.
You’ll want to note these weather conditions from the Astrid Status tab: Pyrgeometer, IR Cloud Cover, Wind Velocity, Humidity, Temperature, and Dew Point
14.3. Run the m2QuickDaisy script on your calibrator source (it’s best if you can make your OOF source and your calibrator source the same)
14.4. Run the m2QuickDaisy script on your pointing source
Note: during this initial data acquisition (and to some extent, throughout the night) check your Mustang2 Cleo screen that you opened up in step 2, and make sure that the numbers are continuing to change with time (if so, the boxes will mostly be blue) if they stop (indicated when the boxes turn lavender) then the Mustang2 manager has crashed, and you’ll need to reset it (detailed in step 10).
15. Checking data
We are going to check this run using the m2gui (opened in step 13)
15.1. Click the “online” button
15.2. Under Calibration, click “Select Tip Scan” and choose the most recent scan number from the bottom labeled “tip” under “scan type.” This should be from the beginning of the 4 OOF scans.
You will see a graph to the right, showing the tip scan. Make sure that it looks reasonable.
This is what a reasonable tipping scan should look like. The green line is the fit and the black line should follow it fairly closely.
This is what a bad tipping scan looks like:
If the tipping scan doesn’t look right, try running the “skydip” script in Astrid. This reruns the tipping scan without having to redo the whole OOF. If it still looks bad, check the weather conditions in CLEO. The weather might not be good enough to observe. You can also call one of the MUSTANG2 instrument team and get their advice.
Note: Make sure you check the number of live detectors at this stage, as well as throughout the night.
You can see in the image below where to check the number of live detectors:
Generally it’s good to have 170+ live detectors, however it can sometimes be as low as 160 if the tuning step didn’t go very well (see steps 7-9 for details on tuning). If you see this number as low as the 150s or 140s (especially if it’s lower than that, which it shouldn’t be) be sure to contact a MUSTANG team member. You can also try re-tuning (detailed at the end of step 9) and hope that that fixes it.
15.3. If the tipping scan looks good, then make sure “source type” is set to “calibrator,” and set the “scan numbers” to the m2quickDaisy pointing scan that you just made. Then click “Make Map”.
This will open up an image of the daisy map that you selected.
The map should look something like this:
The maps that the MUSTANG2 team makes are called daisy scans. This is because they loop many times around a central point, looking somewhat like daisy petals. This emphasizes exposure time on the center of the map, with less exposure on the outside edges of the map, making the center of the map more accurately calibrated. They then use the outside of the map to calibrate the sky temperature and remove these effects in the center of the daisy in later post-processing.
What you see at this stage is an image of the daisy scan. In the center is your calibrator source, visible because it is a bright source. Later, when looking at daisy scans of your science source, it’s very likely that you will only see a flat map in the center because it’s so much more faint.
The units of the color-coding of this map are in Kelvin of the forward beam. The forward beam is calibrated for the estimated sky temperature at that elevation that we gleaned from our tipping scan earlier on in the night.Therefore, the forward beam temperature should hover around zero if everything is calibrated correctly.
The lines drawn on the map designate the beam path of the GBT on the sky relative to your source. As you can see, each loop begins at the source, extends out, and then returns to the source. This is done throughout the space around your source. Because every loop returns to your source, this results in a higher exposure time on your source relative to the rest of the sky. However, because the units are in Kelvin of the forward beam, this does not mean a higher temperature, but instead simply less noise in the map.
15.4. Click “Fit Map”
Fit Map button
In your terminal, a series of information should be put onto your screen.
The Floating underflow error you see is not a concern.
15.5. Write down the Peak_Height, WidthA, and WidthB to compare to later pointing scans.Optional: Showing Time Stream
You can also look at how the sky temperature is changing over time by clicking the “show time stream” button after making your map.
Here’s where the button is:
This is what a good time stream will look like:
This is what a bad time stream will look like:
Take Science Data
16.1. Go into Astrid, you will see scheduling blocks labeled beauty2.5 and beauty3. Run 2 each of beauty2.5 and beauty3, totaling to 4 beauty scans. It’s best if you alternate so you get the most of each in case something happens and you have to stop in the middle of the 4 scans. Make sure when you check these in the m2gui that you check the scan type to “faint science.”
What is Beauty2.5 and Beauty3?
Beauty2.5 and Beauty3 are the science scans of the observation. The difference between the two is the radius size of the scans in arcminutes (one is 2.5′ and one is 3′ respectively). If you only see Beauty scans, unlabeled otherwise, then they are likely 3′ in diameter and you can just run 4 beauty scans in a row.
16.2. Run another m2quickDaisy scan on your pointing source, and then go through the m2gui to check the Peak_Height, WidthA, and WidthB (detailed in step 15)
If the new PeakHeight is down by more than ~15%, or if the WidthA and WidthB scan become very different (indicating that the beam has become overly elliptical) you’ll want to re-do your OOF scan.
Optional: If you’d like, once the PeakHeight is down by more than 15%, instead of redoing the OOF scan, you can do another m2QuickDaisy on the pointing source to be sure that it is that low, and then do two more Beauty scans until the PeakHeight has gone down by another 15% (so a cumulative 30%).
16.3. Redo step 16 for the rest of the night.
Closing up for the night:
Turn Off Components
17.1. In your Mustang2 Cleo screen, turn all the biases back to zero.
To set the biases to zero, go to the Mustang Manager in CLEO (detailed in step 2) and click on the miscellaneous tab. Then in the top middle, you will see 4 rows of Det Bias 1-4, corresponding to the 4 roaches.
Unlock the manager (detailed in the beginning of step 11) and going one roach at a time, set the detbias to 0, press enter, and wait until the blue box shows a detbias of 0. (Roaches 1-4 correspond to roaches A-D). Repeat this step for all 4 of the roaches.
17.2. In your Mustang2 Cleo screen, turn the DataXinit off for all four roaches.
17.3. Go to http://mustangboot.gbt.nrao.edu and turn off the roaches, HEMTs, and Function Generator (turning these on is outlined in step 6, it is the same idea to turn them off as well)
Congratulations! You’re all done! Happy Observing!
- Turning on Components
- Crash Mitigation
- Streaming Data; check for packet loss.
- Closing up for the night: