NoSQL vs. Relational

Well, first of all, I suppose I need to have at least one blog entry per year, so this one just makes it. For the last few months, I've been putting more and more data structures into MongoDB instead of MySQL, and I just put a task into the backlog to move everything from MySQL to Mongo. I've worked with relational databases since 1989. Moving to an RDBMS from the indexed filesystems on the mainframe was a huge leap forward. It represented the start of the shift from overnight batch processing toward realtime data. For those twenty years I've been reverential toward the database component of the apps I've architected. I've spent significant upfront time putting together highly-normalized layouts with the goal of limiting schema changes as the product concept matures. Having had to do database conversions of various types, I've always preferred code conversions, so the idea is to overdesign the database as much as possible, and let the code catch up. So, last year the talk around NoSQL really starts to heat up. It just doesn't hit home for me, though. Why should I change my strategy of a highly organized data structure and switch to a primitive key-store with no JOINs, no referential integrity, and no strong entity-attribute model? Sure, I can see some nice two-dimensional structures which would benefit from a high-speed data store, but that's what I use memcached for, so what's the big deal? I decided to attend MongoNYC last May. It was convenient, it was cheap, so, why not? I can't remember which session I was sitting in. But during that session, it hit me on the head like a licensing renewal bill from Oracle. This is an Agile development tool. If one of the important ideas behind Agile is to get a meaningful demoable feature in front of a customer or feature owner as soon as possible, then what am I doing spending all this upfront time designing a database with a dozen tables just to represent the customer contact who receives our invoice? Why can't the Customer record (document!) just have a list of contacts? I don't need a Structured Query language to assemble all the pieces of the document because I never tore the document into pieces in the first place. I can start writing the code, dump the data into a document, and get my prototype up and running. So fast forward 6 months, and NoSQL gets more and more ink every day. Some well-respected MySQL people are still asking the wrong question. I know that performance has been a potent motivator behind the growth of MongoDB and the other NoSQL implementations, but the big deal here, folks, is development. For the last decade the true costs of running software have been tipping drastically towards product development costs and away from hardware and systems software costs. Unless of course, you're Twitter, and you want to handle 50 billion text messages per minute. But for those of us just trying to build useful apps, it's pretty cheap to throw money at performance issues. Don't get me wrong, I love the speed that Mongo gives me. But what's going to hit my bottom line is that I can get features out the door without an environmental impact statement for the database. And, finally, I'm just having fun using MongoDB. Yup, I'm still in this for the kicks.

Posted by on 31 December 2010 | 9:37 am

Apropos of Nothing, my $0.02 on Detroit

The "big 3" automakers are no longer the real US auto production base. Honda, Nissan, Toyota (and their "hidden" brands) make cars for the US market here, and their sales are a significant part of domestic auto production. The big 3 probably make more US-sold cars in Mexico and Canada then the Japanese make in the US. And yes, I know that "make" doesn't mean anything when parts and intermediate assemblies criss-cross the world several times before final assembly. The UAW is terrified of the manufacturers seeking bankruptcy protection because it's the closest the mfrs will ever get to rewriting work rules, pay scales and no-show job requirements. And all of those factors have one underlying concept: dues collection (and that cash doesn't go to retiree health benefits, either). If Ford, Chrysler, GM "go away", then their brands, intellectual property (think of the hundreds of thousands of patents they hold), usable production capacity, and employable, productive people (minus their union contracts) will be snapped up at auction by the hedge funds that still have cash (yup there are a few, and I bet you could raise 4 billion in about 20 minutes to go shopping at a Detroit fire sale) or other capital-flush enterprises. Christ, if someone can invest $37million in a blog (Huffington Post), there must be capital for an automaker. Isn't a hedge fund or Japanese manufacturer-led bailout at $0.25 on the dollar better than a runaway train government bailout at $5 on the dollar (how much actually has to be collected from us for each dollar paid out in a bailout)? Which will result in more "real" production of US autos and a better workforce and supply chain? Yes, hedge funds are corrupt and leaky, just like a Federal agency, but they are much more efficient at it, and they didn't point a gun at anyone's head to get the money. Finally, cars made by Detroit suck. Bottom line, they fall apart, they burn gas like there's a royalty arrangement involved, they lack features, and they're just not as much fun to drive. If a non-Detroit car sucks, it's likely that the suck factors will be addressed in the near future, OR, it's OK, because you still got value for money. Which products should the market encourage? Bzzt. Sorry. Too late. The market spoke long ago. Thanks for listening. When I try to talk to my cats about this, they just get up and walk away. Come on! Let's hear how utterly, completely, and thoroughly wrong I am! Speak up!

Posted by on 12 December 2008 | 3:29 pm

A chance to make the commuter tax productive

Mayor Bloomberg has used the financial crisis as an opportunity to once again bring up the issue of a commuter tax. The State Assembly (read Sheldon Silver, who represents a district in NYC) is willing to consider the idea. The State Senate (read Dean Skelos, who represents a Long Island district) appears unwilling to even discuss it. I guess that leaves David Patterson as the tie-breaker. While drastic spending cuts would be preferable to any tax increase, maybe we can come up with a commuter tax that isn't a tax per se. Let's collect the commuter tax at the bridges, tunnels, and to a much lesser extent, on the suburban rail and bus lines. I've mentioned this before, but let's trot it out again. Close the free Manhattan crossings inbound to private automobile traffic from 5AM to 3PM on weekdays. Force drivers in their own cars to pay to get into the city during working hours. Increase the inbound toll on the remaining crossings to $40 during those times. This should be the main source of the commuter tax, and at the same time, create an incentive towards mass transit instead of the disincentive that exists now. Eliminate on-street parking in Manhattan (below 135th Street) for private automobiles from 7AM to 8PM. Get the drivers who are willing to pay the higher toll into parking garages. Then hit them with an additional 10% sales tax on top of the existing sales tax. Manhattan residents who park monthly get a 100% exemption. Add a 10% surcharge onto MTA and New Jersey Transit suburban line tickets coming into New York during those times. This should make up for the loss in tolls due to the increased prices, but not enough to offset the incentive. Eliminate the outbound tolls. Get the vehicles out of Manhattan as quickly as possible. The benefits to all this: Reduced traffic, which means lower costs for the movement of people and goods, cleaner air (reducing the cost of failing to meet federal pollution standards), reduced road maintenance costs, and best of all, fewer single-passenger SUVs blowing their horns at red lights. I just loooove talking to myself.

Posted by on 26 September 2008 | 9:17 am

New Hardware Requirements For the Next Version of Windows?

Microsoft and Cray have announced a new supercomputer that runs Windows. I think this is a pretty good indicator of the minimum hardware requirements for the next version of Windows.

Posted by on 16 September 2008 | 2:25 pm

Read the RFCs! That's why they're there!

Once again, I've been told that an IP address can only have one PTR resource record. We got a new Internet connection in the office, and I requested 2 PTR records for an IP address on that connection to provide reverse address lookup. Can't have email rejected for lack of a reverse IP entry. After being ignored by Support for 2 days, I sent the request to the Account Manager, who said the request was rejected because I had 2 hostnames for one IP address, and that's not allowed. I've heard this almost every time I've set up a new Internet connection since 1996. I can't believe there are techs who still think this is true. From RFC2181 ( 10.2. PTR records Confusion about canonical names has lead to a belief that a PTR record should have exactly one RR in its RRSet. This is incorrect, the relevant section of RFC1034 (section 3.6.2) indicates that the value of a PTR record should be a canonical name. That is, it should not be an alias. There is no implication in that section that only one PTR record is permitted for a name. No such restriction should be inferred. Let's go folks. Just because Jon is dead, it doesn't mean that RFCs don't matter anymore.

Posted by on 9 April 2008 | 9:20 am

Kit Kat Needs You!

From : Calling all men! We want YOU - this kitty seems to respond more openly to gentlemen than to women. KitKat has been waiting for a new home for far too long ? since March 2007! Kit Kat often gets overlooked ? while she is a bit sad and lonely here at Bideawee, her previous foster parent described her as a ?mushy lap cat.? She loves having her head and chin scratched, and watching the world go by from a window perch in one of our cat ?resorts.? Kit Kat is a mellow 4 year old that was rescued from the street. Won't you please give this lonely girl a loving, forever home?

Posted by on 16 January 2008 | 10:10 am

Hiring and Applicants from Mars (or they might as well be)

Well, it's time for me to start hiring again. And slowly, despite years of experience, I convince myself that it's OK to post jobs on craigslist and dice and to allow the job posting pages on our website to be indexed and scraped. And I hold my breath. Because I don't have a database backend for the application process yet (number 43 on the list of projects that were due yesterday). Because my Inbox is about to be flooded with applications. Some of them might actually be from people with a glancing, occasional relationship to the requirements in the posting. Some of them might even be within commuting distance of our offices, although most will be from a minimum of 4000 miles away. Please, please don't start thinking that this is some anti-immigration rant. The only person I ever met whose family has been in this country for more than 150 years was Sue S., and I think they were the advance scouting party for the Mayflower. Her family had houses (there were a bunch of them after 350 years in the same town) with 5 foot high ceilings. Immigrants are the core, the backbone, the vitality of this country. If we want to end up like Europe, as a bunch of mouldering, inbred, half-wits waiting for Doomsday, then the current trend against immigrants is a brilliant move. Sorry, I go off like that sometimes. Anyway, it's not about where they're from, it's about where they are, and whether that location makes it even slightly possible that they could show up in the office for more than 2 hours a day a couple times per week. Oh, yes, I'm a dinosaur, a relic of ages past. I have heard that with this modern Internet thing, nobody needs to go to work anymore. We can all sit at home, jacked into a computer and it's like the whole world is one giant virtual office. What a load of crap. Aside from the fact that I don't want to create a race of eyeball-radiated, anti-social, speechless robots who never have direct human contact, people work better when they are in somewhat close proximity. They talk, they enjoy each other's company, they gain a real sense of community, and are motivated to consider themselves part of the group. Gee, there I go again. The job posting says we want people with multiple years of commercial, hands-on PHP and MySQL experience to work full-time, on-site in our Manhattan office. So naturally, I get resumes from Romania, Dallas (not sure which is farther away), India, and Canada from people with no experience in programming, or extensive experience in VB and Access. Some of these are readable, thoughtful applications. I try to turn them down politely. Most however, are illiterate (aside from the language differences in, say, Texas), and don't come close to responding to the instructions in the posting, or have '' as the From: address. They're just awful as attempts to get me interested in the human behind the email. It's very frustrating. It shows how incredibly broken the whole hiring process is. This should be a time when you meet people who will be an asset to your organization, where you make the process of selling your company as a place to work a joy and an exciting experience for the applicant. Wait. You don't think that the hiring process is a selling process? You sit back in your web and wait for your victims to beg appropriately for the privilege of working for you? Wow. First day on the job must be a blast with all those beaten, humiliated new employees shuffling in to their drab workplace, dreading first contact with the Dark Lord of hiring. The person/company/extra-terrestrial invading army who can fix this horrible, demeaning, counter-productive morass will change the world. Too bad I won't live to see it.

Posted by on 22 March 2007 | 7:32 am

The Ambassador of Lindy Hop

A new book is due out this May chronicling the life and career of Frankie Manning, the "inventor" of the air-step in the Lindy Hop. Manning was a dancer at the Savoy Ballroom in Harlem in the early 30's. He toured with a dance troupe and appeared in films. When the popularity of swing faded, the troupe disbanded and Manning went to work at the Postal Service. A revival of swing in the 80's brought Manning back into the dance scene as a teacher and choreographer, where he won a Tony for Black and Blue. The book is written by Manning with his collaborator, Cynthia Millman. Cynthia has a passion for researching Jazz dance and music. From archival records and interviews with the originators, she has amassed a wealth of knowledge, as well as a superb collection of photographs, film clips and memorabilia. Cynthia has written extensively about dance, including articles for Oxford University Press's International Encyclopedia of Dance, and has served as consultant for numerous documentaries including the PBS Swinging with Duke. Through her presentations and demonstrations, she has also shared her passion directly with audiences in both the U.S. and Europe. Cynthia also happens to be the extraordinary librarian at the Town School in Manhattan. Buy it here: Powell's Barnes and Noble

Posted by on 9 March 2007 | 9:06 am

The PATH and the Second Avenue Subway

When the Hudson and Manhattan Railroad was laying out where its tracks and stations would go, the plans included a connection to the East Side via Astor Place and then on to Grand Central Station. This segment of the lines was never built, although a short stub of maybe 50 feet was built just east of the 9th Street Station. Now comes the Second Avenue Subway with construction contracts about to be awarded. How about completing Mr. McAdoo's plans and building the extension out to Astor Place with transfers to the number 6 line and then Second Avenue? That's a pretty short line from 6th Avenue to 4th Avenue and 2nd Avenue. Even the Port Authority should be able to manage a project like that.

Posted by on 2 March 2007 | 10:08 am

I came across recently. I like it. They spider sites and pick just the About Us page. Since it's a Wiki, you can also edit your page. So if the spider picks up the wrong About Us page, or there's more that you want to say, sign in and make it happen. Here are my favorite pages: (of course) LinkExperts Link building ad network. The LinkExperts blog. Cowgirl CreameryMmmmmmm. Look around at your planet. BollywoodSARGAM The intersection of East and West. Got a site? Go see if you're listed.

Posted by on 25 September 2006 | 4:22 pm

How not to improve a grocery store

I went to the Gristede's supermarket (well, supermarket as it's defined in Manhattan) the other morning because we were out of milk. It was too early for the D'Agostinos to be open. I haven't been going to Gristede's recently because I found the store dirty, and the employees rude, but especially after one visit where I asked the manager to turn off the music on the PA. It was the worst kind of gangsta rap, including lyrics about killing policemen. As the manager finally rang up my items (all three of them), I saw a circular on the wall announcing a company contest. Apparently, the corporate headquarters finally got tired of all the complaints about the stores. There were at least 8 different categories, including cleanliness, employee attitude, and produce quality. The contest was to reward the store manager whose unit improved the most in all the categories. Wow. This just seemed wrong on so many levels. It may sound good to reward a store for improving in all these areas, but think about it. The manager (and, by extension, the district manager) who allowed this store to become the undesirable place which prompted so many complaints, would be rewarded and praised for bringing it back to the minimum acceptable level it should have been all along. Also, this is basically a sales contest. The behavior driven by sales contests evaporates the instant that contest ends, or it participants realize that they can't win. So, once this effort is finished, the stores will stop being cleaned, the employees will go back to insulting the customers (after they're finished ignoring them or discussing last night's drunken screw-ups), and the produce will stay on the shelves longer. If Gristede's really wants to change this behavior, then the managers who were directly or ultimately responsible for the operations of these units should be demoted or fired. This is not about underperforming, this is about not doing their job, plain and simple. This goes all the way to the top. Maybe Mr. Catsimidis should spend more time in the stores and less time running for mayor. If you want someone to excel at their job, make them understand that's what their job is about, day in and day out. Contests are for one-time jolts. Store cleanliness and employee attitude are not subjects for one-off and forget it contests.

Posted by on 23 August 2006 | 8:06 am

Traveler's Insurance Gets Security Wrong, Wrong, Wrong

Thanks to Joel Spolsky for the scan of the Traveler's Insurance ad in Inc. magazine: Forget the geek stereotype stuff. Like the rest of humanity, you can find techies who fit stereotypes and techies who don't. What matters here is that Traveler's is promulgating the idea that external threats are where organizations should direct the bulk of their security efforts (and, of course, increased insurance spending). WRONG!! The threats come from the inside, people! Not the outside. Laptops gone missing (for some reason loaded with sensitive or confidential data), disgruntled employees, gossips, the complete and utter inability to secure media transformations (paper to disk to screen and back to paper again...), MS Outlook, security policies which focus on making legitimate tasks hard by forcing employees to jump through hoop after hoop of red tape, MS SQL Server, being unable to flexibly apply policies instead of using broad brushes to make everyone's like more difficult, and on, and on... Whew! I feel better now. You can put away the needle with the tranquilizer. OK, yes, there are external threats, but they are much more readily identifiable, and easier to secure against. There's also a slight matter of exactly what and how the insurer is actually underwriting. Read the contract really carefully, and you will find that unless you've been documenting every last sub-atomic particle of how you secure your premises, your systems, your operations, and your data, and that those efforts conform to some bizarre idea of security developed by an insurance industry task force, you ain't gonna see dollar one from that policy. I am gonna get me a pair of red socks, though.

Posted by on 30 July 2006 | 7:58 am

What is wrong with you people??

Back in February I told you about World Community Grid. I explained how the computing power that sits almost 90% unused on the PCs you have at home, at work, and in school could be used to save lives. And you haven't done a thing about it! Since May, the total amount of computing time per day at World Community Grid has plummeted from 130 years to less than 90 years. It should be increasing, not decreasing. I'll say it again. There are lives at stake here. This is something that costs you nothing, nothing! It is the greatest tool to assist scientific research in history. That's no exaggeration. This is as if every researcher in the world was all of a sudden given their own university with 10,000 research assistants. The knowledge needed to understand disease and develop effective treatments and PREVENTION lies in the ridiculously overpowered machine that you bought to read email or make your mind-numbing Powerpoint presentations. You're not sharing it. Remember, sharing is good! Seriously, join this project. If you don't like what World Community Grid is doing, go to the BOINC project at Berkeley. Their software is being used on a lot of different efforts. Find something that suits you and join in.

Posted by on 26 July 2006 | 7:50 am

Who is Linkie-Winkie and why should I care?

Have you heard of linkie winkie? No?, well neither have I. Everybody's talking about it. See if your site has been linkie-winkie'd. Nope, I'm not telling. You have to figure this linkie winkie thing out for yourself.

Posted by on 24 July 2006 | 9:47 am

Kids say the darndest things

Came across this as I was going over some old email. My son was reading "The Pushcart War" by Jean Merrill (1964). My wife was talking to him one evening as they discussed the project he was going to do on this book. She had sent this out in an email to some friends and family. Last night Phillip and I were talking about "The Pushcart War", the book he's doing his project on. It's a satire about war and human weakness. Our conversation led to many directions, and we (he, mostly) talked bout the evil that happens in the world, religion, and his take on our president (!) At some point he said, "A learned man is a worried man". "Why do you say that?" I asked. "Well, a learned man knows what's happening in the world and he knows about people who don't have enough to eat, and about countries where there is war and places where they have bad governments. And he wants to help them. But he can't help them all because he is just one person, so he worries." "If you have your eyes open wide you see everything in the world and that is too much for a person. If you have your eyes open medium wide, maybe you see your neighborhood, your country“. ”And if you have your eyes open just a little bit you see your family, your friends and your school, and that's something you can help make better." He's 10. Maybe he should be writing this blog.

Posted by on 21 July 2006 | 4:25 pm