Final Post!

Wow, where have ten weeks gone? It seems like yesterday that I was just starting out, and didn't even know what a stealth galaxy was...

It's been incredible, learning so much so quickly. I feel extremely grateful for the opportunity to work for Beth and with the other people in lab. Research is cool! I know that sounds strange, but I may not have been able to say that at the beginning of the summer. It's been great working out problems without the structure of the school year, and getting to focus on the actual science.

Now, I also feel deeply invested in this project. I hope that I'm able to continue working with it, and I hope that others will be able to work with it as well. I tried to leave excessive documentation of my work so anyone can simply pick up where I left off...

For now, I'm off to Hawaii for the semester! Lot's of Astronomy out there, and I can't wait to see what it's like. (Be sure to check out updates on Astronoblog!)

Thanks, Summer. It's been great.

One more day and much to do...

With one work day left, there's a ton to be done and tied up before I get out of here. Here's my to-do list as given by Beth this morning:

1. Run a BIG simulation of 4000 fake galaxies
2. Create a code that visualizes detectability as a function of galaxy size and total absolute magnitude
3. Run a few mini-tests to make sure that the results from my simulation match up with the results from Shane Walsh's simulation in "The Invisibles" paper.

I've made progress on the second two tremendously since morning, but they won't be done until I have more time to spend with them tomorrow. The big simulation will be run overnight-- It should take about 15 hours.

But there are updates, too! Beth found a bug in my old OLD code that generated the number of stars in a galaxy given a total absolute magnitude. It was only producing about half the number of stars it should have, thus making WAY too stealthy. Now, the detections are stronger. (Maybe too strong?) Beth says it will be necessary to go over every line of code before making any weighty conclusions from this, but if this many galaxies are truly being detected (~150/400), then we can say that Andromeda XIX - type galaxies do not exist around the Milky Way because they would have been detected by now...

All for now. Tomorrow's going to be a big day

Back on track... (?)

After a great deal of time, I realize that my concerns from yesterday were incredibly naive. I was concerned because my detections were no longer strong, and I thought there was an invisible bug that magically appeared in my code.

I realize now, though, that this was never the case, and my "working" code was not as strong as I thought. The detections are weak, which is why we were originally going to change the smoothing filter. The only reason I got a chunk of strong detections at the end of last week was because I ran a larger simulation. These galaxies, which are between 1-2 kpc in half-light radius, are incredibly stealthy, and thus hard to detect given the original search parameters. So, when I run small simulations with only 5-10 repetitions, hardly anything, (or nothing at all), pops out.

In other words:

CONCLUSION: In order to truly test the detectability of AndXIX-type galaxies around the Milky Way, the search parameters need to be changed. This needs to be done carefully, however, because when we tried this last week, nothing changed.

I will not detail the amount of stupid things I did this morning to come to this conclusion. Let's just say when you svn "revert", you loose a lot of stuff. However, I think it was good for this to happen, as frustrating as it was, because now I have a deeper understanding of the search algorithm and how it works.

It's like when your toys come to life when you leave the room...

For whatever reason, my simulation no longer works. On Friday, I made all kinds of cool and interesting adjustments to my Search code that allowed each galaxy to have a range of scale sizes and absolute magnitudes, not just the one specific to AndXIX. It worked out great, and I made some beautiful figures with the output files on the data that matched.

Today, I was tweaking the code SO slightly, and now the search code won't detect anything anymore. I tried going back to the original version from this morning, and no luck. It won't work. Something must have been changed, but where?? In order to proceed, I need to learn how to go back to previous versions on our version-controlled svn directory.

This is all too bad, too, because the small change I'm making is going to make the search and match process a lot more exact. Instead of having all of the detections compared to the inserted galaxies, only the detections from a given run will be compared to the galaxies of that same run. That way, we know the detections are really being matched to a single galaxy. I'm eager to do this, and would like to get past this other problem...

Tomorrow, I suppose... Only 4 days left!

Serious Simulations

Today was the day to up the ante on this simulation. Now, instead of generating, inserting, and searching for 40 stealth galaxies in a set of SDSS data, we're working with FOUR HUNDRED simulated galaxies!

VERY cool to run a big simulation like that, but it takes over an hour and a half... so, I've also been working on other things, like combining the code that generates, the code that inserts, and the code that searches into ONE GIANT code. I spent quite a while not only making that functional, but then also cleaning it up to make it as succinct as possible. I've also, y'know, been adding to my LaTeX document (... what else is new?)

So what are the results from this giant test? Well, while the code did a great job finding these stealths, the detections weren't good enough. All but one of the detections were classified as "weak," meaning that they would be indistinguishable from other overdensities in a given sky survey. So, the next step will be to tweak the search parameters to change the size of the smoothing filter and hopefully make these detections stronger. Mimi has been working on this aspect of the project, and it will be nice to collaborate with her more again...

Six days left! I can't believe it...

more detection tests

Not a ton of new stuff, unfortunately, considering it's been a week, but progress is being made.

Earlier this week, I finished up some RCS2 tests, and looked at the results when 100% of the background was placed with the input stealths. Depending on the detectability threshold used during SPHEREMATCH, the algorithm found anywhere between 23 and 31 out of the 40 stealths... not too shabby.

Yesterday, I worked to generalize all the code for SDSS and clean up the repository so that all SDSS and RCS2 code were in separate subdirectories. I was simultaneously working to run similar detectability tests with the SDSS data when Beth realized that the data we got was faulty. Instead of giving nice, clean, rectangular strips, it gave curved bits that were hard to analyze. So, when I ran the tests, nothing popped out.

Now, I have a tiny chunk of SDSS data to play with, but something's still wrong. I've run many tests this morning, and even when I shrink the "stealths" to a concentrated area (half-light radius of 100pc), and lighten the background to 1/100 of the full dataset, hardly any detections are found, and all detections are placed in the "weak" category. There must be a bug somewhere, but I've been reading and re-reading all my code, and can't seem to find the problem. Maybe this little SDSS dataset isnt' much better? It's not a prefect rectangle, but instead looks like a staircase within the box. I wonder if this is the issue...

Either way, there's a week and half left, and this project will soon be coming to a close! My goal for the end is to have the code be fully debugged and generalized so it can be used by anyone who picks up this project, (myself included?). It would also be nice to have some semi-concrete results from these detectability tests so we can know how to proceed. Finally, it's important for me to have a fully updated LaTeX document, so I can have an available summary of my work from this summer.

Big Day

The day started out with an email from Beth with the above subject line, and a HUGE to-do list. Needless to say, it's been an incredibly productive day.

Here's what happened:

1. Running the debugged (!!!) Search Algorithm: Beth debugged that code like a total pro yesterday, and this morning, the Search Algorithm was working exactly as it should. I ran some tests to make sure this was SUPER true, which it was.

2. Matching detections to input galaxies: I added to an output analysis file that looks at the detections and how they match with the inserted fakes. Now, not only does this file tell us which fakes were detected, but it also outputs a data structure with information on those detections. Info includes: distance to detected galaxy, distance modulus filter used to detect galaxy, peak signal, area of detection [sq arcmin], and area of galaxy at half-light radius [sq arcmin].

3. Analyzing output data with figures: With this information, I began making plots that analyzed this data. As the fakes become stealthier and the data more robust, this code will easily adapt.

4. Making stealthier galaxies: We started out with galaxies with half-light radii of 200 pc, which isn't very stealthy at all, but was good while writing code and running tests. Now, we're using galaxies with half-light radii of 1683 pc -- the size of And XIX. Now detectability will no longer be as certain.

5. Odds and ends. I edited some code which output the average number of stars per degree during the search process. I cleaned up a few files. I made a directory for my SDSS files...

Up next: Making galaxies even stealthier. Or, rather, making the background more realistic. Right now, we're placing these stealths in only 10% of the RCS2 background data. With 100% of the data, these simulated galaxies will be MUCH harder to detect.

And that's all until next week! Tomorrow the gang goes rafting!

A big update for little progress

I'm afraid I haven't updated in over a week because the process I've been going through has been extremely slow-going.

After the data was inserted into the RCS2 background last week, I've worked to run Beth's search algorithm on this data. But something's not working right. So, mostly I've been running tests, and debugging small things that I understand. Meanwhile, Beth has been working with me to try to figure out what the bigger problem is.

Here's how we know something's wrong: we run the search code, and when it detects something, it spits out information about that detection to an output file, (RA, Dec, distance modulus...). But the detections have been funky. Either they output nonsensical values, or output the same values over and over, or output values that do not relate to our generated fakes, (this is the latest problem). The code takes forever to run, and so its been a process of trying to debug, waiting a while, and seeing that it's still not right.

So, to maintain my sanity, I've been trying to do other things to keep busy. Yesterday I converted my code to use SDSS data as background, instead of RCS2. Today, I'm working on adding to my LaTeX document with updates from the past few weeks. It's pretty crazy to read about what I was doing 4 weeks ago, and realizing how far I've come...

Hopefully this search code debugging business will straighten itself out soon. Ultimately, we'd like to use it to test the detectability of Stealths, and figure out which galaxies we could be missing completely with current methods of detection.

Simulated Stealths Hidden!

Code written, and now the simulated stealths are "stuffed" into the RCS2 background! I even had some figures to bring to group meeting today, showing the CMD and positions for one "layer" of the simulation. The position graph pointed out a flaw in my coding, and I was able to correct it later this afternoon.

Hooray! SO, NEXT STEP: tweaking the code so it stuffs in 1 dwarf at a time, instead of 5. Figure out how long that takes. THEN: Run Beth's search algorithm, and compare output to input to see which stealths pop out, and which are too stealthy to be caught! (i.e., finding a possible detectability limit.) ALONG THE WAY: output data on distance, and central RAs and DECs of each galaxy into an ascii file.

This is quite great progress from one week ago. I'd like to think I'm getting better (good, even?) at complex code writing.

Dwarf Simulation Complete!

Hooray! Now I have the r and g magnitudes, magnitude errors, and positions of all stars in 50 simulated dwarfs with the same absolute magnitude and scale size as Andromeda XIX!

I was able to finish the code and debug by the time Beth came to lab this afternoon, which felt like a huge success, given how stuck I was this morning.

NEXT STEP: putting these fakes into RCS2 (and eventually SDSS) data! I've started the code for that, and hope to complete it tomorrow morning.

Still tugging away

Still working on complex coding. Right now, I'm concentrating on creating a file that simulates 50 fakes-- 10 each at 5 different distances. More FOR loops, and FOR loops inside FOR loops, and IF statements, and stuffing all of that into a data structure...

I will admit, I am feeling a little lost. This morning, I've been trying to debug old problems from Firday, as well as adding new layers onto the process. It's getting frustrating, because I'm losing sight of what I'm actually trying to do. (Maybe blogging all of this out will help.)

So, the steps for this process are as follows. My confusion comes when trying to think about the ORDER these steps need to fall in:

- using previously written code, simulate magnitudes (in both r and g band), magnitude errors, and x&y positions from the center of the fake (in units of arcmin) for each individual star in each simulated dwarf.

- For the above, simulate over 5 distance categories: 50, 100, 200, 300, and 500 kpc.

- For the above, simulate 10 times for each distance category

- Take data from all 50 simulated fake dwarf galaxies, and stuff into fits binary table. Every star should also have a reference number 1-50 that associates it with its galaxy.

...Yikes...