Flattening the baseline

Hiya folks,

The next step in data processing which isn't really necessary, but is helpful and fun to do, is to flatten out the baseline on the spectrum. I managed to use my fitting program as a base to fit, bit by bit, a nice line to the spectrum and then subtract it to flatten the whole thing out better. It works far better than I could have imagined, with one major flaw. Allow me to illustrate...




The top is a small section of the unprocessed spectrum, focusing in on two peaks. The second is the same section after being run through my flattening program. As you can see, I successfully removed all of the wonky wobbly gunk and have instead just a mostly flat line with a little bit of noise (more obvious, as I'm zoomed in quite a bit more) and the two lines. BUT, the two peaks have been horribly malformed.

At the moment, I have a little bit in the program that causes the least squares fitting for each tiny section to ignore those points which are more than 0.01 different from the average value of each section, and this has in fact improved the graph. But it's still pretty fugly. I want the peaks to look like peaks so that the autopeakfitting of Grams can do all of the picking work for me, instead of me manually fitting each one. As it is, though, this just won't do.

But, since trying to get this program working is a lot more fun than fitting peaks, I shall perservere. If anyone has some suggestions, feel free to comment. Otherwise... well, feel free to comment regardless.

I need more Webcomics

My friends... I have discovered something terrible.

Brain Fist will be ending in a matter of weeks! Soon, my regular comic repertoire will be cut down even more than it has been already! First, ALILBTDII went on hiatus, then Pixel went on hiatus, and now Brain Fist is leaving us (As indicated by the news post here). All of my favourite webcomics are drifting away into oblivion, their strength in numbers reduced by half, leaving only PDH, Giantitp, and the PBF on my regular list of webcomics. Lets not forget older classics like Sexy Losers and Latex Blue, but I don't dare link to those.

I need three good webcomics to fill the void in my heart. Go and search, my minions!

VICTORY

fuzz@vaiojello:~/javastuff/physics/scans$ ls data > what
fuzz@vaiojello:~/javastuff/physics/scans$ java quadfit what
06182007_001.txt 1 of 228
06182007_002.txt 2 of 228
06182007_003.txt 3 of 228
06182007_004.txt 4 of 228
06182007_005.txt 5 of 228
06182007_006.txt 6 of 228
06182007_007.txt 7 of 228
06182007_008.txt 8 of 228
06182007_009.txt 9 of 228
06182007_010.txt 10 of 228
06182007_011.txt 11 of 228
06182007_012.txt 12 of 228
06182007_013.txt 13 of 228
06182007_014.txt 14 of 228
06182007_015.txt 15 of 228
06182007_016.txt 16 of 228
06182007_017.txt 17 of 228
06182007_018.txt 18 of 228
06182007_019.txt 19 of 228
06182007_020.txt 20 of 228
06182007_021.txt 21 of 228
06182007_022.txt 22 of 228
06182007_023.txt 23 of 228
06182007_024.txt 24 of 228
06182007_025.txt 25 of 228
06182007_026.txt 26 of 228
06182007_027.txt 27 of 228
06182007_028.txt 28 of 228
06182007_029.txt 29 of 228
06182007_030.txt 30 of 228
06182007_031.txt 31 of 228
06182007_032.txt 32 of 228
06182007_033.txt 33 of 228
06182007_034.txt 34 of 228
06182007_035.txt 35 of 228
06182007_036.txt 36 of 228
06182007_037.txt 37 of 228
06182007_038.txt 38 of 228
06182007_039.txt 39 of 228
06182007_040.txt 40 of 228
06182007_041.txt 41 of 228
06182007_042.txt 42 of 228
06182007_043.txt 43 of 228
06182007_044.txt 44 of 228
06182007_045.txt 45 of 228
06192007_001.txt 46 of 228
06192007_002.txt 47 of 228
06192007_003.txt 48 of 228
06192007_004.txt 49 of 228
06192007_005.txt 50 of 228
06192007_006.txt 51 of 228
06192007_007.txt 52 of 228
06192007_008.txt 53 of 228
06192007_009.txt 54 of 228
06192007_010.txt 55 of 228
06192007_011.txt 56 of 228
06192007_012.txt 57 of 228
06192007_013.txt 58 of 228
06192007_014.txt 59 of 228
06192007_015.txt 60 of 228
06192007_016.txt 61 of 228
06192007_017.txt 62 of 228
06192007_018.txt 63 of 228
06192007_019.txt 64 of 228
06192007_020.txt 65 of 228
06192007_021.txt 66 of 228
06192007_022.txt 67 of 228
06192007_023.txt 68 of 228
06192007_024.txt 69 of 228
06192007_025.txt 70 of 228
06192007_026.txt 71 of 228
06192007_027.txt 72 of 228
06192007_028.txt 73 of 228
06192007_029.txt 74 of 228
06192007_030.txt 75 of 228
06192007_031.txt 76 of 228
06192007_032.txt 77 of 228
06192007_033.txt 78 of 228
06192007_034.txt 79 of 228
06192007_035.txt 80 of 228
06192007_036.txt 81 of 228
06192007_037.txt 82 of 228
06192007_038.txt 83 of 228
06192007_039.txt 84 of 228
06192007_040.txt 85 of 228
06192007_041.txt 86 of 228
06192007_042.txt 87 of 228
06192007_043.txt 88 of 228
06192007_044.txt 89 of 228
06192007_045.txt 90 of 228
06192007_046.txt 91 of 228
06192007_047.txt 92 of 228
06192007_048.txt 93 of 228
06192007_049.txt 94 of 228
06192007_050.txt 95 of 228
06192007_051.txt 96 of 228
06192007_052.txt 97 of 228
06192007_053.txt 98 of 228
06192007_054.txt 99 of 228
06192007_055.txt 100 of 228
06192007_056.txt 101 of 228
06192007_057.txt 102 of 228
06192007_058.txt 103 of 228
06192007_059.txt 104 of 228
06192007_060.txt 105 of 228
06192007_061.txt 106 of 228
06192007_062.txt 107 of 228
06192007_063.txt 108 of 228
06192007_064.txt 109 of 228
06192007_065.txt 110 of 228
06192007_066.txt 111 of 228
06192007_067.txt 112 of 228
06192007_068.txt 113 of 228
06192007_069.txt 114 of 228
06192007_070.txt 115 of 228
06192007_071.txt 116 of 228
06192007_072.txt 117 of 228
06192007_073.txt 118 of 228
06192007_074.txt 119 of 228
06192007_075.txt 120 of 228
06192007_076.txt 121 of 228
06192007_077.txt 122 of 228
06192007_078.txt 123 of 228
06192007_079.txt 124 of 228
06192007_080.txt 125 of 228
06192007_081.txt 126 of 228
06192007_082.txt 127 of 228
06192007_083.txt 128 of 228
06192007_084.txt 129 of 228
06192007_085.txt 130 of 228
06192007_086.txt 131 of 228
06192007_087.txt 132 of 228
06192007_088.txt 133 of 228
06192007_089.txt 134 of 228
06192007_090.txt 135 of 228
06192007_091.txt 136 of 228
06192007_092.txt 137 of 228
06202007_001.txt 138 of 228
06202007_002.txt 139 of 228
06202007_003.txt 140 of 228
06202007_004.txt 141 of 228
06202007_005.txt 142 of 228
06202007_006.txt 143 of 228
06202007_007.txt 144 of 228
06202007_008.txt 145 of 228
06202007_009.txt 146 of 228
06202007_010.txt 147 of 228
06202007_011.txt 148 of 228
06202007_012.txt 149 of 228
06202007_013.txt 150 of 228
06202007_014.txt 151 of 228
06202007_015.txt 152 of 228
06202007_016.txt 153 of 228
06202007_017.txt 154 of 228
06202007_018.txt 155 of 228
06202007_019.txt 156 of 228
06202007_020.txt 157 of 228
06202007_021.txt 158 of 228
06202007_022.txt 159 of 228
06202007_023.txt 160 of 228
06202007_024.txt 161 of 228
06202007_025.txt 162 of 228
06202007_026.txt 163 of 228
06202007_027.txt 164 of 228
06202007_028.txt 165 of 228
06202007_029.txt 166 of 228
06202007_030.txt 167 of 228
06202007_031.txt 168 of 228
06202007_032.txt 169 of 228
06202007_033.txt 170 of 228
06202007_034.txt 171 of 228
06202007_035.txt 172 of 228
06202007_036.txt 173 of 228
06202007_037.txt 174 of 228
06202007_038.txt 175 of 228
06202007_039.txt 176 of 228
06202007_040.txt 177 of 228
06202007_041.txt 178 of 228
06202007_042.txt 179 of 228
06202007_043.txt 180 of 228
06202007_044.txt 181 of 228
06202007_045.txt 182 of 228
06202007_046.txt 183 of 228
06202007_047.txt 184 of 228
06202007_048.txt 185 of 228
06202007_049.txt 186 of 228
06202007_050.txt 187 of 228
06202007_051.txt 188 of 228
06202007_052.txt 189 of 228
06202007_053.txt 190 of 228
06202007_054.txt 191 of 228
06202007_055.txt 192 of 228
06202007_056.txt 193 of 228
06202007_057.txt 194 of 228
06202007_058.txt 195 of 228
06202007_059.txt 196 of 228
06202007_060.txt 197 of 228
06202007_061.txt 198 of 228
06202007_062.txt 199 of 228
06202007_063.txt 200 of 228
06202007_064.txt 201 of 228
06202007_065.txt 202 of 228
06202007_066.txt 203 of 228
06202007_067.txt 204 of 228
06202007_068.txt 205 of 228
06202007_069.txt 206 of 228
06202007_070.txt 207 of 228
06202007_071.txt 208 of 228
06202007_072.txt 209 of 228
06202007_073.txt 210 of 228
06202007_074.txt 211 of 228
06202007_075.txt 212 of 228
06202007_076.txt 213 of 228
06202007_077.txt 214 of 228
06202007_078.txt 215 of 228
06202007_079.txt 216 of 228
06202007_080.txt 217 of 228
06202007_081.txt 218 of 228
06202007_082.txt 219 of 228
06202007_083.txt 220 of 228
06202007_084.txt 221 of 228
06202007_085.txt 222 of 228
06202007_086.txt 223 of 228
06202007_087.txt 224 of 228
06202007_088.txt 225 of 228
06202007_089.txt 226 of 228
06202007_090.txt 227 of 228
06202007_091.txt 228 of 228
:D

The resulting fits are an average of about 0.006 off from the original data points. A good sign, considering the original data has a resolution of exactly 0.01. If I'm feeling bored tonight, I may try to make the fit even better (Maybe a quartic fit, versus the present cubic fit).

Btw, the emoticon is actual output. I was too lazy to come up with real output messages.

I ultimately went with a combination of the matrix and the easy methods. I discovered that the easy method and the matrix method are identical after a certain point. The easy method was easier to solve afterwards (though lacking in decent notation) while the matrix method, I found, was easier to get to that point in the first place. The result... Some nicely fit data.

I didn't come up with a decent method to automatically find the entire list of the contents of the directory containing the datafiles, so I used as input a file containing the output from the "ls" command. It works well enough.

By the way, like every piece of programming I'm making from here on in, this program is available to anyone who wants it, although you'll need to make some personal alterations as many things are hardcoded in that wouldn't be in a more general purpose program. Just drop me a line! By the way, as a java program it's cross platform, so this includes YOU Windows readers!

Good night.

Data Processing

Last week was spent at work collecting data with the ECTDL (External Cavity Tunable Diode Laser). I've collected about 250 or so sets of data, each spanning about 1.1 cm^-1 (A unit proportional to frequency, obtained by dividing Hz [1/s] by the speed of light in cm/s). And now the truly fun part... Data processing! Allow me to illustrate with a sample...

PZT wavelength Signal
3.000 6305.410 -0.090851
2.975 6305.410 -0.090966
2.950 6305.410 -0.090874
2.925 6305.410 -0.090867
2.900 6305.410 -0.090645
2.875 6305.410 -0.090958
2.850 6305.420 -0.091202
2.825 6305.420 -0.090951
2.800 6305.420 -0.090905
2.775 6305.420 -0.090813
2.750 6305.430 -0.090684
2.725 6305.430 -0.090584
2.700 6305.430 -0.090493
2.675 6305.430 -0.090577
2.650 6305.430 -0.090538
2.625 6305.440 -0.090546
2.600 6305.440 -0.090554
2.575 6305.440 -0.090523
2.550 6305.440 -0.090599
2.525 6305.450 -0.090599
2.500 6305.450 -0.090577
2.475 6305.460 -0.090577
2.450 6305.450 -0.090622
2.425 6305.460 -0.090752
2.400 6305.460 -0.090798
2.375 6305.460 -0.090920
2.350 6305.470 -0.090905
2.325 6305.470 -0.090867
2.300 6305.470 -0.090783
2.275 6305.480 -0.090584
2.250 6305.480 -0.090386
2.225 6305.480 -0.090493
2.200 6305.480 -0.090691
2.175 6305.490 -0.090844
2.150 6305.490 -0.090783
2.125 6305.490 -0.090714
2.100 6305.490 -0.090462
2.075 6305.500 -0.090493
2.050 6305.500 -0.090699
2.025 6305.510 -0.090867
2.000 6305.510 -0.091011
1.975 6305.510 -0.090607
1.950 6305.510 -0.090561
1.925 6305.520 -0.090905
1.900 6305.520 -0.091019
1.875 6305.520 -0.090874
1.850 6305.520 -0.090607
1.825 6305.530 -0.090996

This is a single page from a set of data. The actual range of PZT (Peizoelectric Tuning) voltage ranges from 3 volts to -3 volts. The signal is a measure of the light intensity. It is in an arbitrary unit and is mostly only useful when examining relative intensities. The wavelength is ACTUALLY the frequency, in cm^-1 units.

Anyway, the first thing you should notice is that the wavenumber readings are not very precise. The second thing you should notice is that the relationship between the wavenumber and voltage is not linear. And the third thing you should notice is that sometimes the wavenumber reading jumps back and forth. I need to compensate for this, somehow.

So what do we do when we have a whole lot of data that is not exactly precise but should still follow a nice, nonlinear trend? Why, we do a quadratic or a cubic regression of some sort. Unfortunantly, I can't just plug it into my TI-83 calculator and do it on there, especially considering there's about 250 sets of data of 241 datapoints each. Things will get a little bit disgusting really really fast.

I could copy and paste all of it directly into Excel... But a) The files arn't quite in the correct format for that and b) there's still 250 sets of data and that would be way too time consuming.

SO... I need to figure out how to fit data manually, without the luxury of calculators or applets on the internet or Excel. I need to get down to the bare bones of how it's done, and then write my own program to go through all 250 sets and re-save them as files in a nicer format that can hopefully be copied to Excel a lot better.

Do I have any leads? You betcha. I have two leads, to be precise. An easy way, and a hard way.

The easy way involves various summations, partial derivatives (A form of derivatives that will be taught in second year calculus. It is used in equations involving more than one variable. Essentially, you differentiate with respect to one variable and treat the others exactly like they were constants), followed by a nice easy linear system to solve.

The hard way involves matrices, and doing some awkwards things with them that I don't quite understand... But the harder method is apparantly more adaptable to other kinds of fitting lines than the former, and will also force me to learn more about linear algebra, something which will prove extremely useful in the long run. So... I'd better get cracking, cuz I'm going for the hard way.

I'll need to learn how to do matrices in java, and also I'll need to program some methods to deal with all of the useful operations that can be done with matrices.

If anyone has any wisdom to share on this topic, please give me a yell, or leave a comment!

Have a good one.

Happy Father's day to me!

Today, I have become a father.

That's right! I've completed the first version of Hal the irc bot. Written in java using the Pircbot api, Hal is a chatting bot that stores words said to it in a small database, notes what words can lead to others, what words can start sentences, and what words can end sentences, and then constructs randomized sentences based on this knowledge. I think Hal describes his operation best...

Hal : There man fucking awesome i think i'm notice he's doing here (second to fuck are you after this room is a file containing predefined stuff when you never have and by idling while people talk

There you have it, folks. We have now amongst us a budding young intellect. He's mildly incoherent, but that can be considered a sign of genius. He's also a bit incoporial... He must get that from his mother.

Happy father's day to me! I'm out.

Near death, and rebirth

I forgot the username and password for this blog, and for a month, it seemed all was lost. Then I discovered I could just have all of that sent to me by e-mail, and now the blog is saved.

Today was my last day of work at Staples Business Depot. I am now free from the wretched clutches of retail, and I am no longer obligated to talk about Microsoft products as though they are anything other than highway robbery, or amoral in licensing methodology.

On my home computer, I use Linux, Firefox, Pidgin, XChat, OpenOffice, and Wine. With the powers of these combined, I can and do do everything I want and need from my computing experience, and these are all totally free programs.

What did I do at Staples?

I lied to people. I told them that windows was nice and efficient (It's annoying and a resource hog), and well worth the couple hundred bucks to get it(Linux is better, more secure, more fun, and it's free). I told them that Microsoft Office was of a very good value (Open Office does the exact same stuff, and it's free). I told them that this one version of Norton Antivirus can be installed on, wait for it, 3 computers!! (All free software can be installed however many times you want)!

But no longer. I can now regain what was lost... What was taken by me by the necessity of work and of supporting my education. From here on forward, I shall be completely honest. Afterall, this is the internet! What else would I be?

This is the most important part of todays post:

With my return to honest society at hand, I request the readers feel free to take advantage of this, and ask me any question you can think of. I promise I'll answer it truthfully... Albeit not necessarily completely. Go!