MUSTANG-2 Startup and Observing Notes

Preamble

Guide authors: Emily Moravec and Wilson Skipper.

For any suggested changes to this guide please contact ude.o1714003201arn@c1714003201evaro1714003201me1714003201. For any detailed MUSTANG2 questions contact ude.o1714003201arn@n1714003201osamb1714003201 or any of the current MUSTANG2 team members.

If this is your first time observing, PLEASE read all collapsed sections.

This guide assumes at least limited familiarity with Astrid and CLEO. If you are unfamiliar with those, please follow this link for video guides: GBT Observers. You can also access the Observer’s Guide here for written instructions on Astrid and CLEO.

How to use this guide

• If you are Green Bank Staff or on the MUSTANG2 (M2) instrument team and need to start-up, tune, and bias M2, follow section A.
• If you are NOT Green Bank Staff or on the M2 instrument and are logging on to only observe, the only necessary steps for you to do are in section B. However, it’s still recommended that you read through all the steps at least once to have a general understanding.

Additional MUSTANG2 Resources:

• MUSTANG2 Development Page
• OnGBTOps Wiki (for setup)
• ObservingWithTheGBT Wiki (for observing)
• Tuning Results Wiki
• Mustang2 Gui Wiki
• MUSTANG-2 home page

A: Setup – Tuning and Biasing

---

Detailed M2 instrument team instructions on tuning and biasing that can be used as a reference can be found here: https://safe.nrao.edu/wiki/bin/view/GB/Pennarray/OnGbtOps.

A1. Preparation

A1.1. Start an observing VNC

A1.2. Ask the operator to put you in the gateway for M2. You don’t have to be in the gateway to tune but you do in order to unlock the M2 manager and enter values.

A2. Run startup + tuning script

A2.1. Check that the array is cool and not cycling.

Go the CLEO M2 housekeeping tab and check that the array temperature is ~400mK. Also check that the “Cycle State” on the left side of the window is IDLE and not in stages 1-5. If it is in a “stage” this means that it is cycling. If either the temperature of the array is warm or it is cycling, the array is not ready and/or not cool enough for tuning yet.

A2.2. Change directories to where the startup + tuning script is.

cd /users/penarray/Public

A2.3. Run script.

./startMUSTANG.bash <project_name>

where project name is the name of your project and the session number (e.g. AGBT18A _014_01) where the 18A_014 refers to the proposal number (this will be in the e-mail about observing) and the last two digits are the session number. The session number is not the source code from the DSS e-mail. It is very important to get this right or data reduction will fail to pick up the tuning which in turn affects focusing. If you should make a mistake and get it wrong after tuning is done it can be corrected by creating soft links in the same way as you do when changing projects.

Note: It’s important when typing out your session number that the session number have two digits. For example, if this is your 9th session, type out AGBTProjectID_09 NOT AGBTProjectID_9. If you do the latter option then you will not be able to take data correctly.

If you’ve accidentally mislabeled your Project ID and session number, ssh to egret using lmonctrl and type this command:

cd /home/gbtlogs/Rcvr_MBA1_5tuning/

In this directory you should be able to see your projectID and session number. Simply rename it by typing:

mv <Old_ProjectID_session#> <New_ProjectID_session#>

This will fix any issues with data acquisition.

A2.4. Follow process of script.

A2.4.1. The script will ask you if you really want to do this. Use the arrow keys to select yes.

A2.4.2. Then it will ask you to enter the lmonctrl password on egret (ask if you do not know it).

A2.4.3. Then it will do the following:

1. Log into egret and restart the manager
2. Telnet into the iboot bar and turn on the roaches, function generator, and HEMTs
3. Start a gnome-terminal with tabs running ipython sessions which tune each roach – sometimes gnome-terminal fails in which case it will bring up seperate x-terms
   • During tuning it will ssh into each roach every 5 seconds
4. After tuning has finished it will bring up the tuning plots
5. Then set the manager into observing mode and check if data are flowing – if not it will attempt to fix this.

A2.5. Check the IQ, Flux Ramp, and Phase Response plots output by the script.

See https://safe.nrao.edu/wiki/bin/view/GB/Pennarray/TuningResults for explanations and examples.

Info: Further explanation on tuning

Tuning is kind of like MUSTANG’s equivalent to balancing. It has multiple channels of data and tuning is trying to ensure that the attenuation levels are all roughly equivalent. You do not need to be anywhere specific on the sky nor does MUSTANG need to be in the GBT turret for this step to be done.

Mustang has ~256 channels, ~215 feed horns, of which about ~211 are populated. It has 4 roaches, which are a type of Field Programmable Gate Array (FPGA, which is essentially a specialized CPU for data acquisition and manipulation). These 4 roaches process 64 channels each, adding up to the 256 channels. However, since not all the feed horns are populated, some of the channels in these roaches are already unused.

The IQ plots and many graphs for each roach represent all the different channels for each roach. Since some of those channels are unused, it’s not abnormal to see some channels that don’t look right. As long as the majority look okay, then things are working fine.

A2.6. Tuning Troubleshooting

If one of the roaches is not awake you may see an error in the tuning process like this:

If one of the roaches will not wake up, first ssh to it:

ssh root@mustangr1-#

where # is the roach number. It may take a while for the ssh to go through (several minutes). Once the ssh goes through, in the ipython session for that roach you can redo the tuning by doing the um1=startDAQ() command. Simply type in um1= and do the up arrow to find the command; you are looking for a command that looks like um1=startDAQ(rootdir=rootdir, SaveDir=SaveDir, project=proj, doVNA=False, logLevel=”DEBUG”, tuneKwds=tuneKwds,…) If that does not work, you will have to do the whole tuning process again using the same project code and session number.

A3. Check that data is flowing

Go to the Mustang Manager in CLEO. Click the miscellaneous tab, and click the “Locked” on the bottom left of the window to unlock the regular features, then also unlock advanced features by clicking the “Locked” next to Advanced Features.

Click through the channels and look for:

1. The “Frame Cntr” numbers should be changing and not be really low or 0.
2. The “Roach Data” numbers changing.
3. The Frame and Clock Cntr columns next to the Channel and Roach data should be similar across the 4 roaches (if they finished tuning at the same time).

If there is no data flowing in one or more roaches, you have a few potential solutions:

• You can try resending the channel numbers by going to “Num Chan” -> enter 64 -> press enter.
• Or turn “DataXinit” off then on.
• SSH to the problematic roach(es), e.g. “ssh root@mustangr1-1”.
• If these steps do not solve the problem, you may need to either restart the manager, or worst case, turn off data streaming (zero biases if you notice a problem after biasing the detectors) and power cycle the roaches (in the ibootbar). If after restarting the manager, the problem persists, restart the roaches (in ibootbar). If you restart the ROACHes, you will need to redo the tuning steps.

Note: Be sure to lock the Mustang Manager back when you are done to prevent any accidental miss-clicks. You press the same “Locked” buttons as you did in the beginning of this step, only now they will appear as “unlocked” until you click them again.

A4. Biasing

A4.1. Intro to Biasing

The MUSTANG2 receiver is a continuum receiver that uses a bolometric thermometer to make its measurements. Essentially, it is a highly sensitive thermometer with a filter for its bandwidth. Therefore, any photons in the bandwidth hitting the receiver will raise the temperature slightly.

It is able to be this sensitive by taking advantage of the science behind superconductors. This can be explained using the graph below:

This graph is for a specific superconductor, but the concept is the same, even if the exact temperature and resistance is different. Don’t pay attention to the numbers, but rather the trends.

As you can see, the material is only superconducting at lower temperatures. Once it gets hot enough, it becomes a regular resistor, with higher resistance with higher temperatures. What the MUSTANG2 receiver takes advantage of is the portion of the graph called the “transition edge,” the area in between the material being a regular resistor and being a superconductor. Here, the resistance changes very rapidly with even a slight change in temperature.

What biasing does, is ensure that each roach, when observing blank sky, is set in such a way that the maximum number of channels are placed at this transition edge, in order to ensure maximum sensitivity of the receiver. You will be seeing graphs for each channel, and the point which the AI is choosing is what it believes to be the transition edge of that graph.

Because we are only able to choose one setting for each roach, hence the attempt to simply maximize the effectiveness of all the channels, usually at the expense of certain channels in that roach.

A4.2. Do biasing.

All previous steps can take place in the 1-hour prep before your allotted observing time. That is, before you are given control of the GBT. However, biasing must be done on blank sky, therefore you must have control of the telescope for this and all subsequent steps (else just seeing the subreflector/ground spill over). You must be on your own account, logged onto titania or ariel, and also have permission to be in the gateway from the operator.

A4.2.1. Navigate to the proper directory to run biasing:

cd /home/gbtlogs/Rcvr_MBA1_5tuning/detbias

A4.2.2. Configure the appropriate bash profile:

source /home/gbt/gbt.bash

source /home/gbt/sparrow/sparrow.bash

A4.2.3. Run the bias script:

python new_detbiasV3.py <filename>

Where filename is AGBTsemester_projectCode_session. They are typically referred to as det bias files, as det bias is a shortened way of referring to the determined bias. You will see the speed of data coming going quickly and ‘Det Bias’ (in Misc tab) changing. After waiting a while (5 min or more), you will get a set of graphs.

A good set of biases will look like this:

You will get 4 sets of graphs like this, one for each roach. This one is for roach D, or roach 4, as shown in the title. See https://safe.nrao.edu/wiki/bin/view/GB/Pennarray/TuningResults for more examples of ok and bad detbias plots.

The solid black lines indicate the AI-decided detbias for each channel. It’s okay to see some of the lines reversed in direction (like in detectors 56 to 59 in this example) however something is wrong with that detector when it doesn’t have that general shape (such as in detector 20-23, or 60-63). Having a couple bad detectors isn’t unheard of, it’s more bothersome if a large percentage of detectors don’t look right. The main thing that you are looking to evaluate is do a majority of the detectors have a good calculated value. If a majority of them are not good, you’ll want to enter a bias value in by hand. Think about it like this, for each detector, do you want to accept the bias values that it is finding or put in by hand something around 1.2.

A4.2.4. Once the plots have been come up and you have inspected them, close plots.

A4.2.5. If you are happy with the plots and values, in the terminal, enter Y to send bias to roaches and anything else to ignore calculated values.

A4.2.6. Make note of what the calculated values are by checking the Bias values in Misc!!! Then if the case that the manager crashes, you know what values to re-enter (see A5).

What is biasing? Biasing is finding the voltage that puts the TES detectors on the transition from superconducting to normal, the point at which the resistance of the superconductor is changing with temperature (and makes a good thermometer) for measuring the power landing on the bolometer.

A5: Entering biases manually

If you have a short observing session, you can manually enter the biases to save some time. To do this, unlock the manager and going one roach at a time, set the DetBias to 5.0, press enter, wait until the blue box shows a DetBias of 5.0, and then enter 1.2 (check with an experienced M2 team member as to what the current reliable value is) into the DetBias. Repeat this step for all 4 of the roaches.

If the manager crashed and you need to re-enter the values that were previously calculated, follow the same process but put in your recorded values.

A6: Crash Mitigation – Restart the Manager

A6.1. Ask the operator to restart the MUSTANG manager using TaskMaster, even if you’ve been told how to do this yourself. Restarting machines through TaskMaster is a responsibility that is supposed to only be held by the operator.

A6.2. When the operator has told you that they have restarted the M2 manager, to your Cleo Mustang Manager screen and in the drop down menu go to Managers→Off and then again to click Managers->On to to turn the manager off and back on. You’ll then want to re-check the daily cycle to make sure that it is turned off.

A6.3. If you’re re-starting the manager before biasing, then you’re done restarting the manager. If you’ve already biased, re-check that the det-biases are what you expected them to be (detailed in step 11), and check that the dataXinit buttons are on (detailed in step 11)

B: Observing

---

B1. Preparation

B1.1. Calibrator and Source Preparation – several hours before.

You are expected to have your pointing calibrators, flux calibrators, OOF sources, and science sources planned out at least a few hours before the time of your observation. You can use CLEO’s Scheduler and Skyview to do this (See our tutorial videos on CLEO).

For flux calibrators, find those that are closest to your source. You can use planets or any of the ALMA grid cals listed in the following catalog: /users/penarray/Public/Catalogs/alma_gridcal.cat

You will need to observe at least one of these during your observing session to ensure flux calibration. Preferably 2-3, but if you are ok with a 10-20% error in your flux measurement 1 calibrator is ok.

For OOF sources, it is efficient to use these flux calibrators as your first OOF source of the night. For OOF sources, a general guide is that you want a bright source that is > 1 Jy and 25 < elevation < 60. Planets, if available, are often preferred as flux calibrators (especially Mars, Uranus, or Neptune), though potentially only Uranus is also a preferred OOF target. A more intricate way of thinking about an OOF source is to consider the elevation of your science target: if it will be “low” (average observing elevation is ~35 or less), or “high” (average observing elevation ~60 or higher) then one would prefer to OOF on a source with a similar elevation. If the science target is in between, then the OOF elevation will be less important.

For pointing calibrators, you can find suitable calibrators by going to the Scheduler & SkyView in the upper right-hand corner click “Catalog…” → “Add/Select/DeSelect Catalogs …” → “mustang_pointing” → “Apply”.

The goal is to find a calibrator that is 10-15 deg from your target and > 0.5 Jy (though if have good weather a better choice is something close that is 0.1 Jy). To find a source that is > 0.5 Jy, in the Scheduler & SkyView, go to the box in the right-hand corner that says “Source Intensity Range” and in the “Min” box put 0.5 and hit enter. Now load your science source catalog, enter the time you will be observing in the “UT Date and Time” box, and find a source that is showing and is 10-15 deg from your target.

For your science source catalog, you will either need to make one or a M2 team member has already made one for you.

In summary, you will need to determine:

• Flux calibrators
• Pointing calibrators
• OOF source

B1.2: Observing Scripts – several hours before.

Template observing scripts are located in:

/users/penarray/Public/m2_template_scripts/

If you are creating the scripts for the first time for your project, you will want to copy (a) the scripts 1_m2setup, 2_m2oof, 3_m2quickDaisyOOF, and 4_m2quickDaisyPC and (b) one of the 5_ scripts for your science targets (the radius of the daisy will depend on your science – reach out to the M2 instrument team for guidance). The scripts m2quickDaisy and skydip are extra but can be of use. Read the README for instructions on editing these scripts once you have them in your project directory.

B1.3: Observing Notes.

During observing, you are expected to edit the MUSTANG observing run notes wiki and take notes of what’s occurred throughout the night on this wiki. Create a new page and entry at the bottom of the page by clicking “Edit Wiki text” and follow the naming convention of entry above <AGBTSemester_project-code_session-number>.

If you don’t have permissions yet to edit the wiki and are observing, you can take notes in a text document and email them to the MUSTANG team afterwards to upload to the wiki for you.

B1.4. Observing Prep – shortly before observations.

B1.4.1. VNC.

Open and connect to VNC session on titania or ariel via FastX or tunneling+VNC viewer.

For FastX, choose to start an XFCE session on titania or ariel. For tunneling+VNC viewer:
1. Login to GBO $ssh ude.o1714003201arn.b1714003201g.hss1714003201@eman1714003201resu1714003201
2. $ssh titania (or ariel)
3. Start vnc session
$vncserver -your_screen_geometry
This will give you your session number..
4. Open a new terminal on your computer
ssh -N -C -L 590#:titania.gbt.nrao.edu:590# ude.o1714003201arn.b1714003201g.hss1714003201@eman1714003201resu1714003201
Where # is the number you were given in step 3. After entering your password, it will just sit there and that’s good.
5. User your vnc viewer on your computer to view your session.

B1.4.2. Astrid.

Open an Astrid session and navigate to your corresponding MUSTANG2 project. The MUSTANG2 instrument team should have already populated your Astrid area with appropriate scripts.

B1.4.3. CLEO.

The following are suggested CLEO windows to have open during observing:

• Launch → Receivers → Mustang2 → select (by clicking on the gray box next to the titles) the following to be plotted so that you can monitor them: PT Fridge 1 & 2, Array, HE4 Fridge 1 & 2 Charcoal, He3 Charcoal, He4 Fridge 1 & 2 Evap, and He3 Evap. These thermometers are of interest because they can indicate that things are wrong before they affect the array, or help diagnose what is wrong and how to fix it if the array temperature starts to go up.  Sometimes a cryocycle gets started by accident – in which case if you are looking at the charcoal you can hit abort quickly and no damage is done. Other times a helium4 might run out and that can pull up the array temperature – not much you can do but often you can still collect some good data for a while.
• Launch → Status
• Launch → Antenna
• Launch → Observer Tools → Scheduler & Skyview → click on “Real-time mode” and load in the mustang_pointing and your science target catalogs

B1.5. Observing Pre-flight Checklist

• Make/Find source catalog
• Choose flux calibrator/OOF source
• Choose pointing calibrator
• Make observing log
• Open VNC session with Astrid & Cleo

B1.6. GBO Internet Issues

The internet at GBO can be spotty sometimes. Specifically there are days that the internet goes down for 30-60 seconds at a time quite often. Are you having issues with FastX or your VNC being really laggy? Check this status page (https://status.gb.nrao.edu/) to see the status of the ssh gateways. Orange means that it was down for a small portion of that time slice. Red is down for a significant portion of the time span that slice of the bar covers. Each vertical bar is a day.

If you are having issues with lagginess there are two options: 1) if you are using a VNC, change your VNC to tunnel through NRAO in Charlottesville instead of Green Bank, 2) if you are using FastX you may get kicked out but you’ll just have to keep trying to reconnect.

For the VNC tunnel through Charlottesville, you can do the following.

If you cannot even connect to green bank to start your VNC session:

ssh ude.o1714003201arn.v1714003201c.hss1714003201@eman1714003201resu1714003201

ssh titania

vncserver -geometry screenxgeometry :session#

I suggest that you manually choose a higher number session to decrease the likelihood that that number is being used on either a GBO or Charlottesville computer.

Then make your tunnel

ssh -N -C -L 590#:titania.gbt.nrao.edu:590# ude.o1714003201arn.v1714003201c.hss1714003201@eman1714003201resu1714003201

And with your VNC viewer connect to localhost:590#

The writer of this guide encountered an error when trying to create a tunnel through Charlottesville “bind [::#]:590#: address already in use, channel_setup_fwd_listener_tcpip: cannot listen to port: 590#, could not request local forwarding” This was thought to mean that that # port was already taken on a Charlottesville computer so you will have to choose a different VNC session number. Thus the suggestion to manually choose a session number that is higher initially to avoid such issues.

B2. Observing Procedure

B2.1. Communicate with operator.

A few minutes before your observing start time (say 10 minutes), get on Talk & Draw, tell the operator who you are and what project you are observing for. Since you need to fill in the operator name in Astrid, ask who the operator is (though you can just check here: https://dss.gb.nrao.edu/ops/search/current). And ask how the weather is; it is useful to get a local perspective on the weather.

B2.2. Zero the Zernike Thermal Coefficients.

It is good idea to ask the operator to zero the zernike thermal coefficients before you start (and they may ask you before you ask them). If you start with thermal zernike values a long way from optimal then it is possible for OOF to fail to converge on a good solution.  MUSTANG2 has a different focus to other receivers so if the dish is adjusted for them it could be possible that the 3 mustang scans do not cover the correct range in focus positions.  It does not take any longer to do OOF if you zero things at the start so why not make sure the dish is close enough to its final shape.

B2.3. Fill in info.

In Astrid under ObservationManagement, go to the Run tab and fill in the Observer and Operator information.

B2.4. Take control.

Once the member of the M2 instrument team has finished biasing and the operator tells you are in the gateway/gives you the go ahead, in Astrid > File > Real time mode … > work online with control of the telescope.

B2.5. Configure.

Run the 1_m2setup script in Astrid.

B2.6. OOF.

Make sure that you have changed mySrc in 2_m2oof and run the 2_m2OOF script in Astrid.

For the first OOF of the night, you need to have calSeq=True so that a skydip is done as a part of OOFing process. An OOF will take ~20 minutes to run. Check the OOF results in Astrid > DataDisplay > OOF and re-rerun if necessary. For M2, we typically apply the z5 corrections. When the corrections are available, press the green button that reads “After selecting the Zernike solution above, click this green button to send the solutions to the telescope.” Note: sometimes OOF may time out and you will get a red screen if this happens.

Note: While your OOF is running, it is a good time to:

• write the weather conditions from the GbtStatus tab in Astrid in the log (Pyrgeometer – if working, Temperature, Humidity, IR Cloud Cover, and Wind Velocity).
• start the m2gui that is used to check M2 data while observing and check the skydip and that you can see the OOF images
  • To start the m2gui execute the following commands in a terminal window. Note: the user should make the following call from a directory where they have write permission (i.e. *don’t cd to ~penarray/Public/) when starting IDL.
    • ~penarray/Public/startm2idl
    • m2gui
  • see section B3 for using the GUI

B2.7. Quick daisy on OOF source.

Run the 2_m2quickDaisyOOF script on your OOF/calibrator source (it’s best if you can make your OOF source and your calibrator source the same). Using the m2gui calculate the beam shape (WidthA & WidthB) and peak of the source (Peak_Height) and record these values in the observing log.

B2.8. Quick daisy on pointing calibrator.

Run the 3_m2quickDaisyPC script on your pointing source. Calculate the beam shape (WidthA & WidthB) and peak of the source (Peak_Height) using the m2gui (see Section B3) and record these values in the observing log. Also check the time streams.

Note: during this initial data acquisition (and to some extent, throughout the night) check your M2 Cleo screen that you opened up in step E1.4.3, and make sure that the numbers are continuing to change with time (if so, the boxes will mostly be blue) if they stop (indicated when the boxes turn lavender) then the Mustang2 manager has crashed, and you’ll need to reset it (detailed in Section C).

B2.9. Take science data.

Take ~30 minutes of science data followed by a quick daisy on your pointing calibrator. Often this is accomplished by running the 5_science_rX scheduling blocks in Astrid as many times as needed (or other types of scripts). Each individual “beauty” scan is ~8-9 minutes in length. So if you are submitting individual beauty scans (which 5_science_rX are), you can submit 4 of the science scripts in a row followed by your pointing calibrator scan. You should check the timestreams of these data to make sure that you are getting good data (see B3.5 for more details).

Note: If you try to look at the science data in the m2gui, make sure you choose the “faint science” option under “source type.”

What is science_r2p5 and science_r3?

Science_r2p5 and science_r3 are the science scans of the observation. The difference between the two is the radius size of the scans in arcminutes (one is 2.5′ and one is 3′ respectively). If you only see science scans, unlabeled otherwise, then they are likely 3′ in diameter. Legacy M2 scripts will have labels like beauty_r3.

B2.10. Continue to take science data.

Continue to do ~30 minutes of science data followed by a quick daisy on the pointing calibrator for the rest of the night. Monitor the beam size (WidthA and WidthB) and the Peak_Height using the m2gui to determine if you need to OOF again.

When to OOF?

If the new Peak_Height is down by more than ~15%, or if WidthA and WidthB become very different from one another (indicating that the beam has become overly elliptical) you’ll want to do an OOF. Optional:If you don’t have much observing time left, once the PeakHeight is down by more than 15%, instead of redoing the OOF scan, you can do another m2QuickDaisy on the pointing source to be sure that it is that low, and then do two more Beauty scans until the PeakHeight has gone down by another 15% (so a cumulative 30%).

B2.11. Be aware – Issue with quadrant detector.

In early 2023 it was discovered that over the past year or two the quadrant detector sometimes isn’t work and doesn’t write files to /home/gbtdata/project_code_sesion/QuadrantDetector/ as we expect. The GUI now has a feature that if it detects that the quadrant detector files are not being written an warning box will pop up.

If this happens during observing, press ok and ask the operator to restart the quadrant detector manager.

B3. Checking data with the m2gui

Again to open up the m2gui

$~penarray/Public/startm2idl

$m2gui

After you have opened the m2gui follow these steps to check the tipping scan, monitor the beam shape (width, widthA, widthB) and peak of calibrators (Peak_Height), or to just check the data.

B3.1 Start-up/Check Tipping Scan

B3.1.1. Go online. Click the “online” button

B3.1.2. Select tipping scan. Under Calibration, click “Select Tip Scan” and choose the most recent scan number from the bottom labeled “tip” under “scan type.” This should be from the beginning of the 4 OOF scans.

B3.1.3. Inspect plots. Many plots will pop up – one for each roach showing the results of the tipping scan for each roach. You can click out of these once they finish unless you are particularly curious about specific roaches. After these plots have been produced, you will see a graph to the right in the main gui window, showing the results of the tip scan – each roach is plotted in black with a fit in green. Check to make sure that it looks reasonable.

Example: A good tipping scan. Below is an example of a reasonable tipping scan. The black lines (one for each roach) should be fairly free of wiggles and the green line (which is the fit) should follow the black lines fairly closely.

Example: A bad tipping scan. Below is what a bad tipping scan looks like:

If the tipping scan doesn’t look right (a lot of wiggles), try running the “skydip” script in Astrid. This reruns the tipping scan without having to redo the whole OOF. If it still looks bad, check the weather conditions in CLEO. The weather might not be good enough to observe. You can also call one of the M2 instrument team and get their advice.

What is a skydip? And what are the plots that we looking at? A skydip is a flat field. If you look at the detector bias curves some are inverted and even those with the same sign will have a different response to bias. We use the fact that the atmosphere is not transparent and has a (1-exp(-Tau))/cos(elevation) dependance.  With a fair guess of the opacity, you can do a fit on each detector to get them roughly Kelvin_RJ.  These calibrations are used to make maps of known sources and the results scaled to bring them to the correct amplitude.

B3.1.4. Check the number of live detectors. At this stage, check the number of live detectors, as well as throughout the night. Record this in the observing log.

In the image below, you can see where to check the number of live detectors:

Generally it’s good to have 170+ live detectors, however it can sometimes be as low as 160 if the tuning step didn’t go very well. If you see this number as low as the 150s or 140s (especially if it’s lower than that, which it shouldn’t be) be sure to contact a M2 team member. You can also try re-tuning (see section A) and hope that that fixes it.

If the tipping scan looks good, continue forward.

B3.2. Checking Calibrator/Beam Parameters

B3.2.1. Make map. To make a map of a calibrator, after you have run a m2quickDaisy on a source, click “Update Scan List” to find the source scan number of the source you just observed then set the “Scan Numbers” to that scan number. Then set “Source Type” to “Calibrator,” and then click “Make Map”.

This will open up an image of the daisy map that you selected.

The map should look something like this:

The maps that the M2 team makes are called daisy scans. This is because they loop many times around a central point, looking somewhat like daisy petals. This emphasizes exposure time on the center of the map, with less exposure on the outside edges of the map, making the center of the map more accurately calibrated. They then use the outside of the map to calibrate the sky temperature and remove these effects in the center of the daisy in later post-processing.

What you see at this stage is an image of the daisy scan. In the center is your calibrator source, visible because it is a bright source. Later, when looking at daisy scans of your science source, it’s very likely that you will only see a flat map in the center because it’s so much more faint.

The units of the color-coding of this map are in Kelvin of the forward beam. The forward beam is calibrated for the estimated sky temperature at that elevation that we gleaned from our tipping scan earlier on in the night. Therefore, the forward beam temperature should hover around zero if everything is calibrated correctly.

The lines drawn on the map designate the beam path of the GBT on the sky relative to your source. As you can see, each loop begins at the source, extends out, and then returns to the source. This is done throughout the space around your source. Because every loop returns to your source, this results in a higher exposure time on your source relative to the rest of the sky. However, because the units are in Kelvin of the forward beam, this does not mean a higher temperature, but instead simply less noise in the map.

B3.2.2. Fit Map. Click “Fit Map”

This will produce the following plots in the gui.

Once you “Fit Map”, the fit parameters will be printed out in your terminal.

The Floating underflow error you see is not a concern.

B3.2.3. Record values. Write down the Peak_Height, WidthA, and WidthB to compare to later pointing scans to monitor the beam and decide if you need to re-OOF.

B3.3. Checking Science Scans

If you would like to make a map of of a science scan(s), you can do so by following the same steps as making a map of a calibrator but under “Source Type” you need to select “Faint Science.”

It is useful to check the time streams of your science scans during observing. You do this by making a science map then selecting the “Show Time Stream” button underneath the “Fit Map” button.

You can add several science scans together by putting them all separated by commas in the scan list.

B3.4. Showing the Time Streams – Diagnostic for Bad Weather

It is highly recommended that you check the time streams (checking how the sky temperature is changing over time) by clicking the “show time stream” button after making your map.

Here’s where the button is:

This is what a calibrator time stream in good weather will look like (the bumps are the source):

B3.5. When is the weather too bad for observing?

There are several diagnostics to use in tandem for when the weather is too bad for observing: 1) the skydip – if it is really wiggly that’s bad, 2) time streams on faint science are wiggly, and 3) the beam.

Good Weather Examples for Comparison

Skydip in good weather:

Calibrator (quick daisy) time streams in good weather:

Faint science time streams (a cluster) in good weather:

Bad Weather Examples

A bad weather skydip:

Calibrator time streams in bad weather (still looks similar to calibrator in good weather):

Faint science time streams in bad weather (very wiggly in comparison to the good weather flat time streams):

Note, you can compare the time streams of multiple scans by putting several scans separated by commas where you input the scan number to make maps.

General Advice for Determining “Bad Weather“

Once you have some indication of bad weather, you will want to make an educated guess as to what the trajectory of the weather/data is in order to determine whether or not to keep observing or give up the time. There are many tools that you can use to an assessment of this trajectory. Consider, do the following suggest that the remainder of your scans would be scientifically useful? (this can be used as a checklist of sorts)

1. Time streams
   • Check the time streams of the science scans as laid out above in B3.4. Are they wiggly? How wiggly?
   • How many “bad” science scans have there been in a row?
2. Skydip(s)
   • How does the first skydip of night look? How wiggly is it?
   • If you are seeing indications of bad weather and you decide to OOF again one could add a skydip in to test the weather (calSeq=True).
   • One could even do a one off skydip.
3. Beam
   • Has the beam been deteriorating?
4. Weather forecast
   • Check https://www.gb.nrao.edu/~rmaddale/Weather/AllOverviews.html.
   • Check another reputable weather forecaster (Weather underground, weather.forcast.gov, Windy, etc.)
5. Direct communication with the operator
   • Ask the operator what the weather is like. Since you asked at the beginning of the observation you have one data point.
   • This also serves as a way to keep the operator in the loop and aware of a potentially imminent decision to relinquish telescope control.

Note that the observer should reach out to the operator once the concern of bad weather is identified to let them know that the weather is a concern. This could be as early as the first bad scan (time streams, whether a science scan or those from a skydip). A good practice is that if there are two consecutive scans with bad time streams, the operator should be notified and consulted at this point. That doesn’t mean a decision needs to be made this early on, but it lays the groundwork so that both parties are aware of a potentially imminent decision to relinquish telescope control. If the observer has doubts, reach out to an M2 team member after a second bad scan.

A few data/weather trajectories are as follows:

• Improve
  • Is it a one off? As in its just a cloud passing by?
  • Is the or will the weather improve?
• Stay the same. Is the weather staying bad and not improving?
• Get worse. Is the trajectory getting worse and worse?

You will need to monitoring the situation over time and over multiple scans in order to make a guess about the trajectory of the data. One note is the it is usually never sufficient to come across one bad scan and call it quits. There is usually always some nebulous time span (~half hour to an hour) to determine that things are bad and staying bad. If you think the weather will improve and the improvement should happen soon and give ample time for valuable science scans, then the suggestion is to try to endure the bad weather. However, for weather staying the same and getting worse, the advice is to rely on the other metrics to make a determination, except for the case that the operator identifies clear precipitation with no expectation for improvement. At that point, one can give up the time promptly if it’s heavily raining or snowing.

When making a judgment call as to whether to give up the time due to bad weather, consider the following cases:

• How much time is left? If there is not much time left it is less likely that the weather will change.
• Are you observing a faint target? If you give up amount of time you have left, will that amount of time you have left make a difference for your science?
• How much time has been observed for the project and how much time is left in the project? We ask for a factor of 2 of overheads so maybe there is time to tolerate bad weather.

Something to note is that ~30 minutes is a rough minimum amount of time to relinquish control, but the operator will need some time to prepare a backup project so this is why it is good to keep in touch with the operator throughout this process. So the general advice is that if you give up the time near the end of an observation, the minimum time left in an observing session would be ~45 minutes. Another thing to note is that the flip side of overheads (i.e. maybe the project can tolerate bad weather) is that if you are observing the last session (using up all awarded time), any rescheduled observing would all go to overheads. If it’s not the last session, then the advice is to give up the remainder of time for bad weather (if all bad-weather items are checked).

Again, when in doubt you can always call an M2 team member to help you make the call of whether or not to give up the time.

B3.6. m2 GUI Hangs

If your m2GUI is hanging (won’t quit) do the following:

ps -u

Find the PID of startm2gui and idl and kill both.

kill -9 PID

B4. Changing M2 Projects/Second M2 Project of the Night

B4.1. Make sure tuning data is linked to the second project. If you are observing for an M2 project that is not the first M2 project of the night then before observing you will need to create a link for the tuning so that OOF & data reduction can find the right tuning. If notice is given the person tuning can put a second argument (the second project code and session) into the tuning command separated by a comma. But if that is not done, you will have to manually create a symbolic link. Before you begin observing for the second project, login to egret as lmonctrl and execute the following commands:

ssh ude.o1714003201arn.t1714003201bg.te1714003201rge@l1714003201rtcno1714003201ml1714003201

cd /home/gbtlogs/Rcvr_MBA1_5tuning/

ln -s <old_project_session> <new_project_session>

Where the old project session is the full name of the previous M2 project and the new project session is the second M2 project of the night that you are observing for. Be very careful to put in the right project and session ID or this step will not work and you won’t get any data. You can ask the previous observer for the old project session ID, or look for it by typing:

ls -ltr /home/gbtdata/

Then the last modified file will tell you what the most recent project ID was.

B4.2. Reset scan number to 1. You can ask the operator to restart the scan number at 1. But you need to do this BEFORE ANY data is taken. And sometimes it is just easier to not deal with this at all and just know the scan number that you will be starting at.

B4.3. Run m2setup. When the observing time for the second project starts, you need run m2setup in Astrid again. This is already outlined in the directions, but some people think they can skip it when changing from another MUSTANG run. This is not the case. It’s very important to still run m2setup at the beginning of your session.

B4.4. Skydip/OOF.

You can possibly possibly skip OOFing at the beginning of this second project. You can ask the previous observer when they last did an OOF and what the progression of the beam was.

If you need to re-OOf, make sure that calSeq=True to get a skydip.

If you do not need to re-OOF, do a stand-alone skydip and change myAz to the Azimuth of whatever your first source will be (calibrator, etc.). The telescope will slew to that Az.

B4.5. Flux calibrator. You’ll also want to still observe your flux calibrator using the m2quickdaisy script. This is another thing people think they can skip, but it makes reduction later more difficult. Check the beam with this flux calibrator.

C. Observing Troubleshooting

C1. MUSTANG-2 Manager

Sometimes the MUSTANG2 manager refuses to start – you try to start it and you get a failure every time (using TaskMaster or asking the operator to do this for you). 

The solution is to log onto egret, shut the computer down, log onto the iboot bar and power off egret and the housekeeping. Leave it off for 30 seconds then turn these back on. Egret may take a while to reboot but once it does you should be able to restart the manager – assuming this works you should also make sure to press the “reset heater card” button on the manager twice.

D: Closing up for the night

---

D1. Go offline

In Astrid, go from working online to working offline by going to > File > Real time mode … > work offline.

D2. Automatic Shutdown

For the shutdown process you can either do this (a) automatically or (b) manually (see Section C3). For automatic shutdown, run the following script:
$/users/penarray/Public/stopMUSTANG.bash

$cd /users/penarray/Public

$./stopMUSTANG.bash

D3. Manual Shutdown

D3.1. Set detector biases to zero

To set the biases to zero, go to the Mustang Manager in CLEO and click on the miscellaneous tab. Then in the top middle, you will see 4 rows of Det Bias 1-4, corresponding to the 4 roaches.

Unlock the manager and going one roach at a time, type 0 in the left DetBias box, press enter, and wait until the blue box (right DetBias box) shows a DetBias of 0. (Roaches 1-4 correspond to roaches A-D). Repeat this step for all 4 of the roaches.

D3.2. Turn data transmission off

In your Mustang2 Cleo screen, turn the DataXinit off for all four roaches. You will need to be in gateway AND unlock both the unlock and advanced features unlock buttons to do this.

D3.3. Turn off components

In VNC session, Gg to http://mustangboot.gbt.nrao.edu and turn off the roaches, HEMTs, and Function Generator by checking those three boxes then go to left of the screen and click ‘Off’ (gray button).

D3.4. Turn on daily cycle

Back on M2 CLEO window > Housekeeping (unlock) > recheck daily cycle to be on and put autocycle trigger to HE4 (means that if either of the He4 fridges run out it starts a cycle). 

Set the “daily cycle time” = 0.65 of a day in UT. This is the time of day that the daily cycle starts measured in fraction of a day (UT).  0.65 is a nice balance between ensuring the cycle is over by the time any observations are likely to come up, yet not so early that there is no time to work with the receiver in the morning.

D4. Kill VNC session

Either kill your FastX session or your VNC session via the terminal.

Congratulations! You’re all done! Happy Observing!